TLB flushing call on Windows

nt!KiRetireDpcList+0xd7

nt!KxRetireDpcList+0x5 (TrapFrame @ fffff800`cc332e70)

nt!KiDispatchInterruptContinue

nt!KiDpcInterrupt+0xca (TrapFrame @ ffffd000`a9b34d90)

nt!MiFlushTbList+0x20c

nt!MiDeleteSystemPagableVm+0x4d9

nt!MiPurgeSpecialPoolPaged+0x18

nt!MmFreeSpecialPool+0x3cf

nt!ExDeferredFreePool+0x677

nt!VerifierExFreePoolWithTag+0x44

What a BSOD!

APC_INDEX_MISMATCH (1)
This is a kernel internal error. The most common reason to see this
bugcheck is when a filesystem or a driver has a mismatched number of
calls to disable and re-enable APCs. The key data item is the
Thread->CombinedApcDisable field. This consists of two separate 16-bit
fields, the SpecialApcDisable and the KernelApcDisable. A negative value
of either indicates that a driver has disabled special or normal APCs
(respectively) without re-enabling them; a positive value indicates that
a driver has enabled special or normal APCs (respectively) too many times.
Arguments:
Arg1: 00007ffeb44461b4, Address of system call function or worker routine
Arg2: 0000000000000000, Thread->ApcStateIndex
Arg3: 000000000000ffff, (Thread->SpecialApcDisable << 16) | Thread->KernelApcDisable
Arg4: ffffc6816b407b80, Call type (0 - system call, 1 - worker routine)

Current process when closing a kernel handle.

If you call PsGetCurrentProcess() in a filter or driver when processing IRP_MJ_CLEANUP for a kernel handle the system process is returned as NtClose() calls KeStackAttachProcess() if the handle belongs to a system process kernel table.

2: kd> !thread ffffc5006e6da080

THREAD ffffc5006e6da080  Cid 1588.05ec  Teb: 00000000002aa000 Win32Thread: 0000000000000000 WAIT: (WrResource) KernelMode Non-Alertable

    ffffc5006be8eb70  SynchronizationEvent

IRP List:

    ffffc5006e2ba140: (0006,04c0) Flags: 00000404  Mdl: 00000000

    ffffc50075b8aae0: (0006,0118) Flags: 00060000  Mdl: 00000000

Not impersonating

DeviceMap                 ffffd58256416bd0

Owning Process            ffffc500733ff080       Image:         XXXXXXXX

Attached Process          ffffc5006b8b66c0       Image:        System

Microsoft Security Essentials content scan callback to the service.

Below is a stack when a MSE file system filter(WdFilter.sys) called a service(MsMpEng.exe) to perform file content scan on file open.


00 nt!KiSwapContext

01 nt!KiSwapThread
02 nt!KiCommitThreadWait
03 nt!KeWaitForMultipleObjects
04 nt!FsRtlCancellableWaitForMultipleObjects
05 FLTMGR!FltSendMessage
06 WdFilter!MpScanFile
07 WdFilter!MpAmPostCreate
08 WdFilter!MpPostCreate
09 FLTMGR!FltpPerformPostCallbacks
0a FLTMGR!FltpPassThroughCompletionWorker
0b FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted
0c FLTMGR!FltpCreate
16 nt!IopParseDevice
17 nt!ObpLookupObjectName
18 nt!ObOpenObjectByNameEx
19 nt!IopCreateFile
1a nt!NtCreateFile
1b nt!KiSystemServiceCopyEnd
1c ntdll!NtCreateFile

In response the service sent an IOCTL to the filter to create a section( i.e. a mapped file) for data scan

0b mup!MupStateMachine
0c mup!MupFsControl
0d FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted
0e FLTMGR!FltPerformSynchronousIo
0f FLTMGR!IssueControlOperation
10 FLTMGR!FltFsControlFile
11 FLTMGR!FltpSetPurgeFailureMode
12 FLTMGR!FltCreateSectionForDataScan
13 WdFilter!MpCreateSection
14 WdFilter!MpMessage
15 FLTMGR!FltpFilterMessage
16 FLTMGR!FltpMsgDispatch
17 FLTMGR!FltpDispatch
21 nt!IopSynchronousServiceTail
22 nt!IopXxxControlFile
23 nt!NtDeviceIoControlFile
24 nt!KiSystemServiceCopyEnd
25 ntdll!NtDeviceIoControlFile

Setting Irp->UserIosb for unsuccessful requests.

Just for the record.

Irp->IoStatus is not copied to Irp->UserIosb by the special kernel mode APC , i.e.  IopCompleteRequest, on Irp completion if NT_ERROR(Irp->IoStatus.Status) is true and the Irp is synchronous or has not been made pending. This is important when returning any information in Irp->IoStatus.Information for unsuccessful requests when Irp->Flags doesn't have the IRP_BUFFERED_IO flag set. To indicate that the data has not been returned and provide an additional information in Irp->UserIosb.Information use a special status like STATUS_BUFFER_OVERFLOW which is not an error code.  If the IRP_BUFFERED_IO flag is set you can't use the Information field for an unsuccessful request as the system will try to copy data from Irp->AssociatedIrp.SystemBuffer to Irp->UserBuffer in case of NT_ERROR(Irp->IoStatus.Status) is not being true.

ExInterlockedPopEntrySList processing by scheduler.

I believe this topic on ExInterlockedPopEntrySList might be interesting for Windows drivers developers.

Safety of using ExInterlockedPopEntrySList

The question was

To my knowledge, pre-Windows 8 x64 implementations of SList use 9-bit sequence numbers in the SLIST_HEADER. This means that 512 operations can complete concurrently (without progress from particular thread) until an ABA problem potentially manifests. I wonder whether, depending on the number of threads and physical cores, this couldn't plausibly occur. To further complicate, the kernel could run on a vcpu, creating time discontinuities. I would like to ask: 1. Does the Windows scheduler protect against ABA by, e.g., restarting interlocked operation upon preemption? 2. Is there some protection against hypervisor interference? 3. In the light of the above concerns, is SList on a pre-Windows 8 x64 deployment really safe for all workloads? I would have speculated that per-thread kernel allocator behavior was factored in for the ABA avoidance, but the primitives are in the Win32 API as well and any driver can employ custom pool allocator.
My answer was

I looked at the code again and found that interrupt processing code has a fixup for SList . There is a routine KiCheckForSListAddress. This routine is called at DISPATCH_LEVEL before returning from an interrupt and it fixes the EIP(RIP for x64) of a trap frame to restart SList pop operation if interrupt happened inside ExInterlockedPopEntrySList. So when an interrupt processing code returns execution to an interrupted code the code resumes at the beginning of ExInterlockedPopEntrySList ( namely ExpInterlockedPopEntrySListResume ). kd&gt; uf KiCheckForSListAddress nt!KiCheckForSListAddress: 82acbdf1 0fb7416c movzx eax,word ptr [ecx+6Ch] 82acbdf5 8b5168 mov edx,dword ptr [ecx+68h] 82acbdf8 6683f808 cmp ax,8 82acbdfc 7511 jne nt!KiCheckForSListAddress+0x1e (82acbe0f) Branch nt!KiCheckForSListAddress+0xd: 82acbdfe b8f4dda882 mov eax,offset nt!ExpInterlockedPopEntrySListResume (82a8ddf4) 82acbe03 3bd0 cmp edx,eax 82acbe05 7222 jb nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x16: 82acbe07 81fa1fdea882 cmp edx,offset nt!ExpInterlockedPopEntrySListEnd (82a8de1f) 82acbe0d eb15 jmp nt!KiCheckForSListAddress+0x33 (82acbe24) Branch nt!KiCheckForSListAddress+0x1e: 82acbe0f 6683f81b cmp ax,1Bh 82acbe13 7514 jne nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x24: 82acbe15 a1ac69bb82 mov eax,dword ptr [nt!KeUserPopEntrySListResume (82bb69ac)] 82acbe1a 3bd0 cmp edx,eax 82acbe1c 720b jb nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x2d: 82acbe1e 3b15a469bb82 cmp edx,dword ptr [nt!KeUserPopEntrySListEnd (82bb69a4)] nt!KiCheckForSListAddress+0x33: 82acbe24 7703 ja nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x35: 82acbe26 894168 mov dword ptr [ecx+68h],eax nt!KiCheckForSListAddress+0x38: 82acbe29 c3 ret Branch

Waiting for concurrent page fault completion

An interesting call stack when a thread waits in a page fault for another thread completing paging data from a file

00 nt!KiSwapContext
01 nt!KiSwapThread
02 nt!KiCommitThreadWait
03 nt!KeWaitForSingleObject
04 nt!MiWaitForCollidedFaultComplete
05 nt!MiResolveTransitionFault
06 nt!MiResolveProtoPteFault
07 nt!MiDispatchFault
08 nt!MmAccessFault
09 nt!KiPageFault
0a nt!memcpy
0b nt!CcCopyBytesToUserBuffer
0c nt!CcMapAndCopyFromCache
0d nt!CcCopyReadEx
0e nt!CcCopyRead
0f nt!FsRtlCopyRead
10 ***
11 ***
12 ***
13 nt!NtReadFile
14 nt!KiSystemServiceCopyEnd

FileObjects and SectionObjectPointer in Windows.

Just for the record.

FileObject->SectionObjectPointer is allocated and set by a file system driver but the structure is managed by the Memory Manager (Mm). SectionObjectPointer is shared between all file objects for the same data stream.

FileObject->SectionObjectPointer->DataSectionObject and FileObject->SectionObjectPointer->ImageSectionObject contain address of ControlArea for data and image.

ControlArea deletion is synchronized by ControlArea->WaitingForDeletion and ControlArea->u.Flags.BeingDeleted. WaitingForDeletion points to a structure with notification event and a reference counter.

All functions that might destroy control area take SectionObjectPointer as a parameter. These functions acquire a global lock then check that ControlArea is not NULL. If control area exists ControlArea->u.Flags.BeingDeleted is checked and if it is set a function waits on WaitingForDeletion event with incremented reference counter so the event is deleted when the last waiting thread exit from a waiting state and the reference counter drops to zero. A call to MiCleanSection set SectionObjectPointer->DataSectionObject  and  SectionObjectPointer->ImageSectionObject  to NULL. This call is synchronized with ControlArea->u.Flags.BeingDeleted.

The functions that might delete control area include MmFlushImageSection and CcPurgeCacheSection. That means that it is safe to provide SectionObjectPointer to these functions without synchronizing with file objects deletion. It is even possible to call this functions with a SectionObjectPointer when all related file objects have been deleted or have IopDeleteFile being called for them which might happen in IRP_MJ_PNP processing path.

File mapping and FILE_OBJECT in Windows

There is a WinDBG command !ca that shows file mapping related information. I will show how to get this file mapping information for a file object ( FILE_OBJECT type) by a direct access to structures.

The core of file mapping( and file data caching that uses file mapping ) is SEGMENT object and CONTROL_AREA structures. SEGMENT object contains a pointer to an array of Prototype PTEs ( ProtoPTE ) of _MMPTE_PROTOTYPE type. Each ProtoPTE points to a related physical page if the page is valid. When a file mapping is created the related  virtual memory range PTEs( Page Table Entries ) have the invalid bit set and point to Prototype PTEs. When a corresponding virtual address is accessed a page fault happens, the page fault handler follows a link to ProtoPTE and fixes process PTE to point to a real page. That allows all processes to share the same physical pages for the same file memory mapping. The physical page might need to be allocated and data read in from a file if this has not been done before, after that the page is shared between all processes mapping the file.

FILE_OBJECT has SectionObjectPointer field which is set by a file system driver (FSD) but all its fields are initialized by Memory Manager(CC) and Cache Manager(CC). SectionObjectPointer is of _SECTION_OBJECT_POINTERS type with DataSectionObject field pointing to a CONTROL_AREA structure that in turn points to a SEGMENT object. CONTROL_AREA has a _SUBSECTION structure following it at the tail, all subsequent _SUBSECTION structures are linked by NextSubsection  pointer. Each _SUBSECTION has SubsectionBase field that points to a related ProtoPTEs array.

Below all these structures for a real file object are printed from WinDBG.

0: kd> ??FileObject

struct _FILE_OBJECT * 0x8750ef80

   +0x000 Type             : 0n5

   +0x002 Size             : 0n128

   +0x004 DeviceObject     : 0x879c9030 _DEVICE_OBJECT

   +0x008 Vpb              : 0x879d5888 _VPB

   +0x00c FsContext        : 0x87bdde68 Void

   +0x010 FsContext2       : 0x863cc188 Void

   +0x014 SectionObjectPointer : 0x87bddea8 _SECTION_OBJECT_POINTERS

   +0x018 PrivateCacheMap  : 0x869acf90 Void

   +0x01c FinalStatus      : 0n0

   +0x020 RelatedFileObject : (null) 

   +0x024 LockOperation    : 0 ''

   +0x025 DeletePending    : 0 ''

   +0x026 ReadAccess       : 0 ''

   +0x027 WriteAccess      : 0 ''

   +0x028 DeleteAccess     : 0 ''

   +0x029 SharedRead       : 0 ''

   +0x02a SharedWrite      : 0 ''

   +0x02b SharedDelete     : 0 ''

   +0x02c Flags            : 0xc0012

   +0x030 FileName         : _UNICODE_STRING "\Sample Pictures\Chrysanthemum.jpg"

   +0x038 CurrentByteOffset : _LARGE_INTEGER 0x11000

   +0x040 Waiters          : 0

   +0x044 Busy             : 1

   +0x048 LastLock         : (null) 

   +0x04c Lock             : _KEVENT

   +0x05c Event            : _KEVENT

   +0x06c CompletionContext : (null) 

   +0x070 IrpListLock      : 0

   +0x074 IrpList          : _LIST_ENTRY [ 0x8750eff4 - 0x8750eff4 ]

   +0x07c FileObjectExtension : 0x8774e950 Void

0: kd> ??FileObject->SectionObjectPointer

struct _SECTION_OBJECT_POINTERS * 0x87bddea8

   +0x000 DataSectionObject : 0x863c1758 Void

   +0x004 SharedCacheMap   : 0x869acea0 Void

   +0x008 ImageSectionObject : (null) 

0: kd> dt nt!_CONTROL_AREA 0x863c1758 

   +0x000 Segment          : 0xaeb311a8 _SEGMENT

   +0x004 DereferenceList  : _LIST_ENTRY [ 0x0 - 0x0 ]

   +0x00c NumberOfSectionReferences : 1

   +0x010 NumberOfPfnReferences : 0x40

   +0x014 NumberOfMappedViews : 1

   +0x018 NumberOfUserReferences : 0

   +0x01c u                : <unnamed-tag>

   +0x020 FlushInProgressCount : 0

   +0x024 FilePointer      : _EX_FAST_REF

   +0x028 ControlAreaLock  : 0n0

   +0x02c ModifiedWriteCount : 0

   +0x02c StartingFrame    : 0

   +0x030 WaitingForDeletion : (null) 

   +0x034 u2               : <unnamed-tag>

   +0x040 LockedPages      : 0n1

   +0x048 ViewList         : _LIST_ENTRY [ 0x86a3a898 - 0x86a3a898 ]

0: kd> ??sizeof(nt!_CONTROL_AREA)

unsigned int 0x50

0: kd> dt nt!_SUBSECTION 0x863c1758+0x50

   +0x000 ControlArea      : 0x863c1758 _CONTROL_AREA

   +0x004 SubsectionBase   : 0xa8e9b008 _MMPTE

   +0x008 NextSubsection   : (null) 

   +0x00c PtesInSubsection : 0x40

   +0x010 UnusedPtes       : 0

   +0x010 GlobalPerSessionHead : (null) 

   +0x014 u                : <unnamed-tag>

   +0x018 StartingSector   : 0

   +0x01c NumberOfFullSectors : 0x40

0: kd> dt nt!_SEGMENT  0xaeb311a8 

   +0x000 ControlArea      : 0x863c1758 _CONTROL_AREA

   +0x004 TotalNumberOfPtes : 0x40

   +0x008 SegmentFlags     : _SEGMENT_FLAGS

   +0x00c NumberOfCommittedPages : 0

   +0x010 SizeOfSegment    : 0x40000

   +0x018 ExtendInfo       : (null) 

   +0x018 BasedAddress     : (null) 

   +0x01c SegmentLock      : _EX_PUSH_LOCK

   +0x020 u1               : <unnamed-tag>

   +0x024 u2               : <unnamed-tag>

   +0x028 PrototypePte     : 0xa8fe97e8 _MMPTE

   +0x030 ThePtes          : [1] _MMPTE

1: kd> dt nt!_MMPTE .
   +0x000 u                :
      +0x000 Long             : Uint8B
      +0x000 VolatileLong     : Uint8B
      +0x000 HighLow          : _MMPTE_HIGHLOW
      +0x000 Flush            : _HARDWARE_PTE
      +0x000 Hard             : _MMPTE_HARDWARE
      +0x000 Proto            : _MMPTE_PROTOTYPE
      +0x000 Soft             : _MMPTE_SOFTWARE
      +0x000 TimeStamp        : _MMPTE_TIMESTAMP
      +0x000 Trans            : _MMPTE_TRANSITION
      +0x000 Subsect          : _MMPTE_SUBSECTION
      +0x000 List             : _MMPTE_LIST

1: kd> dt nt!_MMPTE_PROTOTYPE
   +0x000 Valid            : Pos 0, 1 Bit
   +0x000 Unused0          : Pos 1, 7 Bits
   +0x000 ReadOnly         : Pos 8, 1 Bit
   +0x000 Unused1          : Pos 9, 1 Bit
   +0x000 Prototype        : Pos 10, 1 Bit
   +0x000 Protection       : Pos 11, 5 Bits
   +0x000 Unused           : Pos 16, 16 Bits
   +0x000 ProtoAddress     : Pos 32, 32 Bits

Caching and file object reference in Windows.

This is how the last reference to a file object backing cached file data is being released by the kernel. In that case this was a network filesystem

19 nt!IofCallDriver
1a mup!MupiCallUncProvider
1b mup!MupStateMachine
1c mup!MupClose
1d nt!IofCallDriver
1e nt!IopDeleteFilec
1f nt!ObpRemoveObjectRoutine
20 nt!ObfDereferenceObjectWithTag
21 nt!ObfDereferenceObject
22 nt!CcDeleteSharedCacheMap
23 nt!CcWriteBehind
24 nt!CcWorkerThread
25 nt!ExpWorkerThread
26 nt!PspSystemThreadStartup
27 nt!KiThreadStartup

How handles are closed on process termination in Windows

Just for curiosity. A call stack when handles are closed on process termination

00 nt!ObpDecrementHandleCount
01 nt!ObpCloseHandleTableEntry
02 nt!ExSweepHandleTable
03 nt!ObKillProcess
04 nt!PspExitThread
05 nt!PsExitSpecialApc
06 nt!KiDeliverApc
07 nt!KiServiceExit
08 ntdll!KiFastSystemCallRet
09 ntdll!ZwWaitForWorkViaWorkerFactory
0a ntdll!TppWorkerThread
0b KERNEL32!BaseThreadInitThunk
0c ntdll!__RtlUserThreadStart
0d ntdll!_RtlUserThreadStart

ZwQuerySystemInformation fails for SystemSessionProcessesInformation(53) when called from a driver

The following kernel mode code will always fail with STATUS_ACCESS_DENIED ( C0000005 ) error if used with a well known definition for SYSTEM_SESSION_PROCESS_INFORMATION.

typedef struct _SYSTEM_SESSION_PROCESS_INFORMATION {
    ULONG SessionId;
    ULONG SizeOfBuf;
    PVOID Buffer;
} SYSTEM_SESSION_PROCESS_INFORMATION, *PSYSTEM_SESSION_PROCESS_INFORMATION;

SYSTEM_SESSION_PROCESS_INFORMATION     Info;

Info.SessionId = SessionId;
Info.Buffer = Buffer; // a buffer allocated in the system space
Info.SizeOfBuf = SizeOfBuf;

RC = ZwQuerySystemInformation( SystemSessionProcessesInformation, &Info, sizeof(Info), &ReturnedLength );


I disassembled the sequence of calls until an error was returned. The reason for failure is that the definition for SYSTEM_SESSION_PROCESS_INFORMATION has probably changed starting from Vista. The kernel checks the size of the structure. The size is a third parameter for ZwQuerySystemInformation. If the size is 0x10(on 64 bit system) ExpQuerySystemInformation calls ProbeForWrite for Info.Buffer regardless of the previous mode ( in this case the previous mode was KernelMode ). Obviously the system allows to use the old definition only for user mode code as ProbeForWrite always throws an exception ( SEH ) when called with a kernel mode address as a parameter.

Below is a call stack when ProbeForWrite is called

nt!ProbeForWrite
nt!ExpQuerySystemInformation
nt!NtQuerySystemInformation
nt!KiSystemServiceCopyEnd
nt!KiServiceLinkage
<a call to ZwQuerySystemInformation from a kernel mode driver>

What happens with outstanding IRPs when a process terminates.

When Windows kernel terminates a process it inserts APC in each thread, in turn this APC calls PspExitThread that calls IoCancelThreadIo to cancel if possible all outstanding IRPs by calling IoCancelIrp and waits for IRP cancelation or completion. A thread waits only for IRPs that have been associated with a thread by calling IopQueueThreadIrp that adds IRP in a list of IRPs associated with a thread, the list head is IrpList field of the ETHREAD structure.

A thread will be blocked until any of the two conditions takes place

 - IrpList becomes empty, that means all outstanding IRPs completed in a normal way or were cancelled

 - 5 minutes timeout expired, in that case IopDisassociateThreadIrp is called to perform IRPs disassociation by removing them from IrpList and setting IRP->Tail.Overlay.Thread to NULL

Below is a call stack for a terminating thread with four outstanding IRPs ( marked yellow ).

        THREAD 870788e8  Cid 1100.0f5c  Teb: 7ffab000 Win32Thread: 00000000 WAIT: (DelayExecution) KernelMode Non-Alertable
            86d29460  SynchronizationEvent
        IRP List:
            87305dc8: (0006,0100) Flags: 00060a00  Mdl: 00000000
            86ecd6f8: (0006,0100) Flags: 00060a00  Mdl: 00000000
            862fd7b8: (0006,0100) Flags: 00060a00  Mdl: 00000000
            87221d80: (0006,0100) Flags: 00060a00  Mdl: 00000000
        Not impersonating
        DeviceMap                 975b0820
        Owning Process            8514a030       Image:         explorer.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      22339          Ticks: 1 (0:00:00:00.015)
        Context Switch Count      2734           IdealProcessor: 1          
        UserTime                  00:00:00.000
        KernelTime                00:00:00.031
        Win32 Start Address 0x769842ed
        Stack Init a86bfed0 Current a86bfa38 Base a86c0000 Limit a86bd000 Call 0
        Priority 10 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 2
        ChildEBP RetAddr
        a86bfa50 82ad269d nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
        a86bfa88 82ad14f7 nt!KiSwapThread+0x266
        a86bfab0 82ad11d5 nt!KiCommitThreadWait+0x1df
        a86bfb0c 82cb9171 nt!KeDelayExecutionThread+0x2aa
        a86bfb40 82cbe519 nt!IoCancelThreadIo+0x70
        a86bfbb4 82cd2051 nt!PspExitThread+0x48e
        a86bfbcc 82b058c0 nt!PsExitSpecialApc+0x22
        a86bfc1c 82a922a4 nt!KiDeliverApc+0x28b
        a86bfc1c 778770b4 nt!KiServiceExit+0x64 (FPO: [0,3] TrapFrame @ a86bfc34)
        074fe3e4 00000000 ntdll!KiFastSystemCallRet (FPO: [0,0,0])

A main process thread waits for child threads termination.

        THREAD 86e307f0  Cid 1100.1104  Teb: 7ffdf000 Win32Thread: fe9c8a88 WAIT: (Executive) KernelMode Non-Alertable
            870788e8  Thread
        Not impersonating
        DeviceMap                 975b0820
        Owning Process            8514a030       Image:         explorer.exe
        Attached Process          N/A            Image:         N/A
        Wait Start TickCount      21665          Ticks: 675 (0:00:00:10.530)
        Context Switch Count      15565          IdealProcessor: 1          
        UserTime                  00:00:00.249
        KernelTime                00:00:00.530
        Win32 Start Address 0x00e50efa
        Stack Init a87b5ed0 Current a87b5a38 Base a87b6000 Limit a87b3000 Call 4f0
        Priority 12 BasePriority 8 UnusualBoost 0 ForegroundBoost 2 IoPriority 2 PagePriority 5

        ChildEBP RetAddr
        a87b5a50 82ad269d nt!KiSwapContext+0x26 (FPO: [Uses EBP] [0,0,4])
        a87b5a88 82ad14f7 nt!KiSwapThread+0x266
        a87b5ab0 82acb0cf nt!KiCommitThreadWait+0x1df
        a87b5b2c 82cbe28e nt!KeWaitForSingleObject+0x393
        a87b5bb4 82cd2051 nt!PspExitThread+0x203
        a87b5bcc 82b058c0 nt!PsExitSpecialApc+0x22
        a87b5c1c 82a922a4 nt!KiDeliverApc+0x28b
        a87b5c1c 77876fc0 nt!KiServiceExit+0x64 (FPO: [0,3] TrapFrame @ a87b5c34)
        000ffb18 00000000 ntdll!KiUserCallbackDispatcher (FPO: [0,0,0])

A case of successful registration of an incorrectly defined file system minifilter.

 An interesting observation. If you forget to terminate FLT_OPERATION_REGISTRATION array with IRP_MJ_OPERATION_END then no instances will be attached but a minifilter is successfully registered and InstanceSetup callback is called. No any error is reported. Just yet another case of a closed source Microsoft subsystem with inconsistent behavior when you can spent hours chasing a bug by trial and error approach instead of looking at source code.

Unsafe use of FltGetFileNameInformationUnsafe

  FltGetFileNameInformationUnsafe name is spot on. It is very unsafe and dangerous function but not in the way mentioned in WDK. The documentation misses yet another dangerous case when it should not be used. This case is related to file system isolation filters( for more information on such filters follow this link https://www.osronline.com/article.cfm?article=560 ). I would state this additional condition as

  - A filter driver must not use FltGetFileNameInformationUnsafe for file objects it never observed. A filter observed a file object if it was called at least once with this file object for any operation and it never received IRP_MJ_CLOSE for this file object or observed a failed IRP_MJ_CREATE. This condition guaranties that a file object is initialized by a filter of file system driver that is beneath a filter calling FltGetFileNameInformationUnsafe . 

  When the above condition is violated by any file system filter a filter might call an underlying file system with a file object this file system never initialized as the file object was initialized by a file system filter that is above a filter that has called FltGetFileNameInformationUnsafe . At first glance it seems that in that case an upper filter must isolate the underlying filters from such file object. This is true, but there is a degenerate case when a filter can gain access to file objects that it should never see. This is a case of a process creation callback when a file object for executable file is provided as a parameter to a callback and this file object might have been initialized by a file system filter above one that registered this callback. Below is a call stack from the system where Microsoft Windows Defender file system minifilter used FltGetFileNameInformationUnsafe from a process creation callback to query a file name for a file object initialized by a file system filter attached to MiniFilter Manager ( aka fltmgr.sys ). 

Ntfs!NtfsCommonQueryInformation+0xa7

Ntfs!NtfsFsdDispatchSwitch+0xd3

Ntfs!NtfsFsdDispatchWait+0x47

nt!IovCallDriver+0x3cd

fltmgr!FltpQueryInformationFile+0x10e

fltmgr!QueryStandardLinkInformation+0x4d

fltmgr!FltpSetStreamListStandardInformationFlags+0x7a

fltmgr!FltpGetFileNameInformation+0x796

fltmgr!FltGetFileNameInformationUnsafe+0x71

WdFilter!MpGetImageNormalizedName+0x51

WdFilter!MpCreateProcessNotifyRoutineEx+0x77

nt!PspInsertThread+0x7a7

nt!NtCreateUserProcess+0x806

nt!KiSystemServiceCopyEnd+0x13

As a result NTFS crashed when it received an IRP with a file object initialized by a file system filter attached to the top of the drivers stack. The obvious solution is to use the old good ObQueryNameString as it has been done for the last 25 years by the kernel itself, for example in NtQueryVirtualMemory when processing a memory region supported by a mapped file. In that case an IRP is created and sent from the very top of a drivers stack so an isolation filter has a chance to intercept this IRP and complete it.

When are object callbacks being called?

The object callbacks registered with ObRegisterCallbacks are called from ObpCreateHandle, e.g.

01 ffffd000`98d1c4b0 fffff802`9eaa8b96 nt!ObpCallPreOperationCallbacks+0x16a
02 ffffd000`98d1c540 fffff802`9eb0e6cd nt!ObpCreateHandle+0xa76
03 ffffd000`98d1c780 fffff802`9eb0fde4 nt!PsOpenProcess+0x5cd
04 ffffd000`98d1cac0 fffff802`9e7df863 nt!NtOpenProcess+0x24
05 ffffd000`98d1cb00 00007ffc`1113375a nt!KiSystemServiceCopyEnd+0x13
06 00000062`ae57e3d8 00007ffc`0e5e59f4 ntdll!NtOpenProcess+0xa

Kernel Mode Debugging Windows IoT on Raspberry Pi 2 with WinDBG

  I already described two ways for kernel mode debugging Windows IoT for Intel Galileo( via JTAG and WinDBG ), Windows IoT for Intel Galileo is still built around Windows 8.1 not 10( at least at the time of writing ).

 In case of Raspberry PI Windows IoT is built on Windows 10 kernel. Raspberry Pi2 doesn't have easily available JTAG port, there are speculations that JTAG is available via some GPIO pins but to prove this I need the board schematics which has not been available at the time of writing.

 Anyway, this is Windows so WinDBG over serial port, USB or network should work. There are three GPIO pins dedicated to UART communication on Raspberry Pi 2 - GPIO(6) is GND, GPIO(8) is Txd, GPIO(10) is RxD.  There is a guide from Microsoft for Raspberry Pi2 kernel mode debugging - Conect Windows 10 IoT Core to WINDBG - Connecting to a Raspberry Pi 2 (RPi2) . Unfortunately it contains an error in the wiring that showed with TxD connected to TxD, RxD connected to RxD, instead TxD-RxD and TxD-RxD , i.e. the wiring as shown by Microsoft works only if null modem cable is used but doesn't work with standard FTDI connector. Also, you need a decent serial port adapter which is able to communicate at a speed at least 1 Mbaud as Raspberry PI 2 UART communicates at 921600 baud, I would recommend to buy an original FTDI TTL-232R 3.3 V cable or similar as easily available cheap USB2Serial cables are not able to communicate at the speed over 115200 baud. 

  Connect the wires like this:

I used a ribbon cable from a cobbler kit to connect FTDI 232R via male jumper cables, if you have male-female cables your can connect them directly to GPIO pins.

 Configure the board for kernel mode debugging via bcdedit, run WinDBG and establish a kernel mode debugging via COM port at  921600 baud .

 Below is a quote from the Microsoft tutorial

Start of the quote.

• Start your RPi2 and connect to it using PowerShell (you can find PowerShell instructions here  http://ms-iot.github.io/content/win10/samples/PowerShell.htm )
• Configure your RPi2, by changing the bcd settings like this:

    [192.168.0.243]: PS C:\> bcdedit -store C:\EFIESP\efi\Microsoft\Boot\bcd -dbgsettings serial
  
    [192.168.0.243]: PS C:\> bcdedit -store C:\EFIESP\efi\Microsoft\Boot\bcd -debug on
  

• From your development machine, open the device manager and find the COM port your converter is using.
• From your development machine, start WINDBG with the you provided and the key that was generated in the previous step:

    "C:\Program Files (x86)\Debugging Tools for Windows (x86)\windbg.exe" -k com:port=<PORT>,baud=921600

End of the quote.

  When executing bcdedit to activate kernel mode debugging do not provide the port speed as 921600, just leave it blank, as this value is considered as invalid by bcdedit, Raspberry Pi 2 UART by default communicates at 921600 baud and it looks like it is not possible to change it ( at least at the time of writing ).

Below is an example of WinDBG session for Windows IoT on Raspberry Pi 2, note the ARM instructions at the bottom

Appendix A.

  There is an easy way to test a serial connection with Raspberry Pi 2 when WinDBG remains silent and doesn't connect. Close WinDBG, run PuTTY , open COM port at 921600 baud, power on the board and watch the output. If there is a clear meaningful text like at the picture below then the connection is OK, if there is some output but there is no meaningful text then the serial adapter unable to communicate at 921600 baud, if there is no output at all then the wiring is wrong ( most probably you connected TxD to TxD instead RxD )

Appendix B

 Question - Can I damage a serial adapter by a wrong wiring?

 Answer -   No, normally you can't cause any damage by wrong wiring. But refrain from disconnecting or reconnecting wires when the board is powered.

Getting an object type on Windows 10 Technical Preview Build 10074

   Windows 10 Technical Preview Build 10074 came with a surprise. A bit of history - Windows 7 introduced a new way for retrieving an object type by object address, the object type pointer Type in OBJECT_HEADER was replaced with the TypeIndex which is an index in ObTypeIndexTable, this saved 3 ( 32 bit) or 7 (on 64 bit) bytes compared to a pointer. Windows 10 Build 10074 added a new feature, the TypeIndex value is not an index but a result of a binary operation between an index in ObTypeIndexTable, the second lowest byte of the object address and a value from ObHeaderCookie. The actual reason of this is not yet clear for me but it looks like an attempt to reduce an inter CPU cache coherency traffic by spreading the ObTypeIndexTable to contain copies of the object types and multiplexing access based on the object address. The exported ObGetObjectType function can be used to retrieve an object type address. Lets take a look on ObGetObjectType.

nt!ObGetObjectType:
lea         rax,[rcx-30h]
movzx   ecx,byte ptr [rcx-18h]
shr        rax,8
movzx   eax,al
xor        rax,rcx
movzx   ecx,byte ptr [nt!ObHeaderCookie (fffff802`eae3d42c)]
xor        rax,rcx
lea         rcx,[nt!ObTypeIndexTable (fffff802`eae3d8e0)]
mov       rax,qword ptr [rcx+rax*8]
ret

which can be written in C as ( where XOR(a,b) is a^b )

POBJECT_TYPE
ObGetObjectType( __in PVOID Object )
{
POBJECT_HEADER   Header = GET_OBJECT_HEADER( Object );
UCHAR    Index = XOR( Header->TypeIndex, (UCHAR)(Header>>8) );
       UCHAR    Cookie= *(PUCHAR)ObHeaderCookie;

        return  ObTypeIndexTable[ XOR(Index, Cookie) ];
}

JTAG: Windows IoT debugging on Intel Galileo 2

As it was promised I will outline the steps to debug Windows IoT kernel running on Intel Galileo using JTAG interface .  Why does somebody need this? JTAG interface gives more power over hardware and provides deeper insight into hardware state.

  For debugging over JTAG using Windows host you will need

    - a hardware JTAG debugger, I use OLIMEX ARM-USB-OCD-H
    - an adapter from JTAG debugger to 10 pin connector, e.g. ARM-JTAG-20-10
    - OpenOCD to communicate with JTAG debugger and provide a debug server connection for GDB

    - MinGW for GDB

After everything have been assembled it looks like this
  


Then you need to start OpenOCD from cmd prompt by executing the following command
openocd-x64-0.9.0-dev.exe -f interface/ftdi/olimex-arm-usb-ocd-h.cfg  -f target/quark_x10xx.cfg

For example this is an output om my PC

this provides you with a GDB server on the port 3333 and telnet connection with the OpenOCD on the port 4444. So you can connect GDB to the server by providing a command target remote localhost:3333 to GDB, for example

 
you can also connect to OpenOCD via your favorite telnet client, mine is PuTTY

IRP dispatching in WDF

Just for the record. A call stack for PnP IRP dispatching by WDF.

Wdf01000!FxPkgPnp::Dispatch
Wdf01000!FxDevice::DispatchPreprocessedIrp
Wdf01000!imp_WdfDeviceWdmDispatchPreprocessedIrp
msisadrv!MsIsaPnPIrpPreProcessingCallback
Wdf01000!FxDevice::DispatchWithLock
nt!IovCallDriver
nt!IopSynchronousCall
nt!IopQueryLegacyBusInformation
nt!PipCallDriverAddDevice
nt!PipProcessDevNodeTree
nt!PnpDeviceActionWorker
nt!PnpRequestDeviceAction
nt!IopInitializeBootDrivers
nt!IoInitSystem
nt!Phase1InitializationDiscard
nt!Phase1Initialization
nt!PspSystemThreadStartup
nt!KiStartSystemThread