ExInterlockedPopEntrySList processing by scheduler.

I believe this topic on ExInterlockedPopEntrySList might be interesting for Windows drivers developers.

Safety of using ExInterlockedPopEntrySList

The question was

To my knowledge, pre-Windows 8 x64 implementations of SList use 9-bit sequence numbers in the SLIST_HEADER. This means that 512 operations can complete concurrently (without progress from particular thread) until an ABA problem potentially manifests. I wonder whether, depending on the number of threads and physical cores, this couldn't plausibly occur. To further complicate, the kernel could run on a vcpu, creating time discontinuities. I would like to ask: 1. Does the Windows scheduler protect against ABA by, e.g., restarting interlocked operation upon preemption? 2. Is there some protection against hypervisor interference? 3. In the light of the above concerns, is SList on a pre-Windows 8 x64 deployment really safe for all workloads? I would have speculated that per-thread kernel allocator behavior was factored in for the ABA avoidance, but the primitives are in the Win32 API as well and any driver can employ custom pool allocator.
My answer was

I looked at the code again and found that interrupt processing code has a fixup for SList . There is a routine KiCheckForSListAddress. This routine is called at DISPATCH_LEVEL before returning from an interrupt and it fixes the EIP(RIP for x64) of a trap frame to restart SList pop operation if interrupt happened inside ExInterlockedPopEntrySList. So when an interrupt processing code returns execution to an interrupted code the code resumes at the beginning of ExInterlockedPopEntrySList ( namely ExpInterlockedPopEntrySListResume ). kd> uf KiCheckForSListAddress nt!KiCheckForSListAddress: 82acbdf1 0fb7416c movzx eax,word ptr [ecx+6Ch] 82acbdf5 8b5168 mov edx,dword ptr [ecx+68h] 82acbdf8 6683f808 cmp ax,8 82acbdfc 7511 jne nt!KiCheckForSListAddress+0x1e (82acbe0f) Branch nt!KiCheckForSListAddress+0xd: 82acbdfe b8f4dda882 mov eax,offset nt!ExpInterlockedPopEntrySListResume (82a8ddf4) 82acbe03 3bd0 cmp edx,eax 82acbe05 7222 jb nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x16: 82acbe07 81fa1fdea882 cmp edx,offset nt!ExpInterlockedPopEntrySListEnd (82a8de1f) 82acbe0d eb15 jmp nt!KiCheckForSListAddress+0x33 (82acbe24) Branch nt!KiCheckForSListAddress+0x1e: 82acbe0f 6683f81b cmp ax,1Bh 82acbe13 7514 jne nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x24: 82acbe15 a1ac69bb82 mov eax,dword ptr [nt!KeUserPopEntrySListResume (82bb69ac)] 82acbe1a 3bd0 cmp edx,eax 82acbe1c 720b jb nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x2d: 82acbe1e 3b15a469bb82 cmp edx,dword ptr [nt!KeUserPopEntrySListEnd (82bb69a4)] nt!KiCheckForSListAddress+0x33: 82acbe24 7703 ja nt!KiCheckForSListAddress+0x38 (82acbe29) Branch nt!KiCheckForSListAddress+0x35: 82acbe26 894168 mov dword ptr [ecx+68h],eax nt!KiCheckForSListAddress+0x38: 82acbe29 c3 ret Branch

Kernel Mode Debugging Windows IoT on Raspberry Pi 2 with WinDBG

  I already described two ways for kernel mode debugging Windows IoT for Intel Galileo( via JTAG and WinDBG ), Windows IoT for Intel Galileo is still built around Windows 8.1 not 10( at least at the time of writing ).

 In case of Raspberry PI Windows IoT is built on Windows 10 kernel. Raspberry Pi2 doesn't have easily available JTAG port, there are speculations that JTAG is available via some GPIO pins but to prove this I need the board schematics which has not been available at the time of writing.

 Anyway, this is Windows so WinDBG over serial port, USB or network should work. There are three GPIO pins dedicated to UART communication on Raspberry Pi 2 - GPIO(6) is GND, GPIO(8) is Txd, GPIO(10) is RxD.  There is a guide from Microsoft for Raspberry Pi2 kernel mode debugging - Conect Windows 10 IoT Core to WINDBG - Connecting to a Raspberry Pi 2 (RPi2) . Unfortunately it contains an error in the wiring that showed with TxD connected to TxD, RxD connected to RxD, instead TxD-RxD and TxD-RxD , i.e. the wiring as shown by Microsoft works only if null modem cable is used but doesn't work with standard FTDI connector. Also, you need a decent serial port adapter which is able to communicate at a speed at least 1 Mbaud as Raspberry PI 2 UART communicates at 921600 baud, I would recommend to buy an original FTDI TTL-232R 3.3 V cable or similar as easily available cheap USB2Serial cables are not able to communicate at the speed over 115200 baud. 

  Connect the wires like this:

I used a ribbon cable from a cobbler kit to connect FTDI 232R via male jumper cables, if you have male-female cables your can connect them directly to GPIO pins.

 Configure the board for kernel mode debugging via bcdedit, run WinDBG and establish a kernel mode debugging via COM port at  921600 baud .

 Below is a quote from the Microsoft tutorial

Start of the quote.

• Start your RPi2 and connect to it using PowerShell (you can find PowerShell instructions here  http://ms-iot.github.io/content/win10/samples/PowerShell.htm )
• Configure your RPi2, by changing the bcd settings like this:

    [192.168.0.243]: PS C:\> bcdedit -store C:\EFIESP\efi\Microsoft\Boot\bcd -dbgsettings serial
  
    [192.168.0.243]: PS C:\> bcdedit -store C:\EFIESP\efi\Microsoft\Boot\bcd -debug on
  

• From your development machine, open the device manager and find the COM port your converter is using.
• From your development machine, start WINDBG with the you provided and the key that was generated in the previous step:

    "C:\Program Files (x86)\Debugging Tools for Windows (x86)\windbg.exe" -k com:port=<PORT>,baud=921600

End of the quote.

  When executing bcdedit to activate kernel mode debugging do not provide the port speed as 921600, just leave it blank, as this value is considered as invalid by bcdedit, Raspberry Pi 2 UART by default communicates at 921600 baud and it looks like it is not possible to change it ( at least at the time of writing ).

Below is an example of WinDBG session for Windows IoT on Raspberry Pi 2, note the ARM instructions at the bottom

Appendix A.

  There is an easy way to test a serial connection with Raspberry Pi 2 when WinDBG remains silent and doesn't connect. Close WinDBG, run PuTTY , open COM port at 921600 baud, power on the board and watch the output. If there is a clear meaningful text like at the picture below then the connection is OK, if there is some output but there is no meaningful text then the serial adapter unable to communicate at 921600 baud, if there is no output at all then the wiring is wrong ( most probably you connected TxD to TxD instead RxD )

Appendix B

 Question - Can I damage a serial adapter by a wrong wiring?

 Answer -   No, normally you can't cause any damage by wrong wiring. But refrain from disconnecting or reconnecting wires when the board is powered.

WinDBG: Kernel Debugging Windows for IoT on Intel Galileo 2

When I read about Windows IoT my first question was  "Can I debug it in the kernel mode?". It happened "yes I can". Officially there is no information about kernel debugging. But Windows IoT is a subset of Windows 10 so theoretically it looked as possible.

 There were two options

   - debugging via JTAG

   - debugging via WinDBG

I will elaborate on JTAG debugging later. Now let's talk about WinDBG options and it happened that Microsoft left the door open , the image of Windows IoT released in November 2014 for Intel Galileo 2 had kernel debugging enabled via serial port at a speed 115200 bps .

This is a picture of a board with an attached serial-to-USB converter. 

Be cautious as Intel Galileo 2 uses an Arduino style pinout that differs from a standard FTDI adapters. Also, Intel Galileo 2 uses 3.3v TTL logic for serial port communication, while some FTDI adapters have 5v TTL logic, do not confuse it with 5v VOUT which is not connected to anything on Intel Galileo 2 . 

I use a USB Serial Adapter from Freetronics, which has 3.3-5v switch and Arduino pinout.

After setting everything WinDBG shows a familiar output and you can break into the kernel, though there are no symbol files on the Microsoft symbol files server as Microsoft did not suppose that anybody outside MS would perform kernel mode debugging for Windows IoT.

For example a list of drivers and kernel modules reported by WinDBG

kd> lm n t
start    end        module name
77450000 775b5000   ntdll    ntdll.dll    Fri Nov 14 19:41:52 2014 (5466CB80)
8043d000 80487000   CLFS     CLFS.SYS     Fri Nov 14 19:39:46 2014 (5466CB02)
80487000 804a2000   tm       tm.sys       Fri Nov 14 17:30:37 2014 (5466ACBD)
804a2000 804b5000   PSHED    PSHED.dll    Fri Nov 14 20:24:25 2014 (5466D579)
804b5000 804be000   BOOTVID  BOOTVID.dll  Fri Nov 14 19:40:17 2014 (5466CB21)
804be000 804c7000   ksecext  ksecext.sys  Fri Nov 14 19:40:14 2014 (5466CB1E)
804c7000 80541000   CI       CI.dll       Fri Nov 14 19:37:50 2014 (5466CA8E)
80541000 80572000   msrpc    msrpc.sys    Fri Nov 14 19:39:02 2014 (5466CAD6)
80572000 805aa000   pci      pci.sys      Fri Nov 14 19:38:15 2014 (5466CAA7)
805aa000 805dc000   sdbus    sdbus.sys    Fri Nov 14 19:39:00 2014 (5466CAD4)
80a8a000 80a93000   kdcom    kdcom.dll    Fri Nov 14 19:40:18 2014 (5466CB22)
8161b000 81674000   hal      halmacpi.dll Fri Nov 14 19:40:32 2014 (5466CB30)
81674000 81c22000   nt       ntkrpamp.exe Fri Nov 14 17:36:32 2014 (5466AE20)
81e00000 81e11000   mup      mup.sys      Fri Nov 14 19:40:18 2014 (5466CB22)
81e11000 81e19000   minvol   minvol.sys   Fri Nov 14 19:40:05 2014 (5466CB15)
81e19000 81e30000   disk     disk.sys     Fri Nov 14 19:39:39 2014 (5466CAFB)
81e33000 81e79000   fltmgr   fltmgr.sys   Fri Nov 14 19:40:05 2014 (5466CB15)
81e79000 81e8b000   fileinfo fileinfo.sys Fri Nov 14 19:38:31 2014 (5466CAB7)
81e8b000 81e9e000   WimFsf   WimFsf.sys   Fri Nov 14 19:38:57 2014 (5466CAD1)
81e9e000 81ecc000   fastfat  fastfat.sys  Fri Nov 14 19:40:10 2014 (5466CB1A)
81ecc000 81ee2000   ksecdd   ksecdd.sys   Fri Nov 14 19:39:08 2014 (5466CADC)
81ee2000 81efe000   usbccgp  usbccgp.sys  Fri Nov 14 19:37:55 2014 (5466CA93)
81efe000 81f08000   USBD     USBD.SYS     Fri Nov 14 19:40:11 2014 (5466CB1B)
81f08000 81f5e000   usbhub   usbhub.sys   Fri Nov 14 19:38:43 2014 (5466CAC3)
81f5e000 81f72000   usbehci  usbehci.sys  Fri Nov 14 19:39:04 2014 (5466CAD8)
81f72000 81fd3000   USBPORT  USBPORT.SYS  Fri Nov 14 19:39:42 2014 (5466CAFE)
81fd3000 81fe1000   pcw      pcw.sys      Fri Nov 14 17:30:36 2014 (5466ACBC)
81fe1000 81fff000   USBSTOR  USBSTOR.SYS  Fri Nov 14 19:37:52 2014 (5466CA90)
82000000 82014000   partmgr  partmgr.sys  Fri Nov 14 19:40:03 2014 (5466CB13)
82018000 820f2000   ndis     ndis.sys     Fri Nov 14 19:38:11 2014 (5466CAA3)
820f2000 82146000   NETIO    NETIO.SYS    Fri Nov 14 19:37:24 2014 (5466CA74)
82146000 8216d000   ksecpkg  ksecpkg.sys  Fri Nov 14 19:37:22 2014 (5466CA72)
8216d000 82180000   wfplwfs  wfplwfs.sys  Fri Nov 14 19:36:35 2014 (5466CA43)
82180000 821c8000   fwpkclnt fwpkclnt.sys Fri Nov 14 19:36:53 2014 (5466CA55)
821c8000 821d5000   condrv   condrv.sys   Fri Nov 14 19:40:07 2014 (5466CB17)
821d5000 821dc400   vmstorfl vmstorfl.sys Fri Nov 14 19:37:01 2014 (5466CA5D)
821dd000 821eaa00   vmbkmcl  vmbkmcl.sys  Fri Nov 14 19:38:37 2014 (5466CABD)
821eb000 821fd000   sdstor   sdstor.sys   Fri Nov 14 19:39:19 2014 (5466CAE7)
8221b000 82266000   CLASSPNP CLASSPNP.SYS Fri Nov 14 17:30:57 2014 (5466ACD1)
82289000 82293000   Fs_Rec   Fs_Rec.SYS   Fri Nov 14 17:30:36 2014 (5466ACBC)
82293000 8229b000   Null     Null.SYS     Fri Nov 14 19:40:13 2014 (5466CB1D)
8229b000 822ab000   BasicDisplay BasicDisplay.sys Fri Nov 14 19:39:16 2014 (5466CAE4)
822ab000 822b8000   watchdog watchdog.sys Fri Nov 14 19:39:37 2014 (5466CAF9)
822b8000 823e1000   dxgkrnl  dxgkrnl.sys  Fri Nov 14 19:37:36 2014 (5466CA80)
84600000 84611000   volmgr   volmgr.sys   Fri Nov 14 19:39:39 2014 (5466CAFB)
84611000 84627000   mountmgr mountmgr.sys Fri Nov 14 19:39:48 2014 (5466CB04)
84628000 846be000   Wdf01000 Wdf01000.sys Fri Nov 14 19:38:59 2014 (5466CAD3)
846be000 846cc000   WDFLDR   WDFLDR.SYS   Fri Nov 14 19:38:43 2014 (5466CAC3)
846cc000 846dd000   acpiex   acpiex.sys   Fri Nov 14 19:37:36 2014 (5466CA80)
846dd000 846e7000   WppRecorder WppRecorder.sys Fri Nov 14 19:39:30 2014 (5466CAF2)
846e7000 84752000   ACPI     ACPI.sys     Fri Nov 14 19:39:10 2014 (5466CADE)
84752000 8475b000   WMILIB   WMILIB.SYS   Fri Nov 14 19:40:12 2014 (5466CB1C)
8475b000 847d0000   cng      cng.sys      Fri Nov 14 19:37:40 2014 (5466CA84)
847d0000 847d8000   msisadrv msisadrv.sys Fri Nov 14 19:38:48 2014 (5466CAC8)
847d8000 847e3000   vdrvroot vdrvroot.sys Fri Nov 14 19:38:36 2014 (5466CABC)
847e3000 847fb000   pdc      pdc.sys      Fri Nov 14 17:30:38 2014 (5466ACBE)
8700f000 87060000   dxgmms1  dxgmms1.sys  Fri Nov 14 19:37:18 2014 (5466CA6E)
87060000 8706c000   BasicRender BasicRender.sys Fri Nov 14 19:39:04 2014 (5466CAD8)
8706c000 8707c000   Npfs     Npfs.SYS     Fri Nov 14 19:40:15 2014 (5466CB1F)
8707c000 87087000   Msfs     Msfs.SYS     Fri Nov 14 19:40:14 2014 (5466CB1E)
87087000 870a1000   tdx      tdx.sys      Fri Nov 14 19:36:46 2014 (5466CA4E)
870a1000 87118000   afd      afd.sys      Fri Nov 14 19:36:56 2014 (5466CA58)
87118000 8716d000   rdbss    rdbss.sys    Fri Nov 14 19:37:30 2014 (5466CA7A)
8716d000 87178000   npsvctrig npsvctrig.sys Fri Nov 14 19:38:19 2014 (5466CAAB)
87178000 87195000   dfsc     dfsc.sys     Fri Nov 14 19:37:59 2014 (5466CA97)
87195000 871ae000   intelppm intelppm.sys Fri Nov 14 17:30:38 2014 (5466ACBE)
871ae000 871d1000   quarkserial quarkserial.sys Mon Mar 17 15:47:17 2014 (53277B75)
871d1000 871dc000   usbohci  usbohci.sys  Fri Nov 14 19:39:14 2014 (5466CAE2)
871dc000 871e6000   stmac6x  stmac6x.sys  Fri Nov 14 19:38:22 2014 (5466CAAE)
871e6000 871ee000   dmap     dmap.sys     Fri Nov 14 19:38:03 2014 (5466CA9B)
871ee000 871f6000   quarklgpio quarklgpio.sys Fri Nov 14 19:38:04 2014 (5466CA9C)
87200000 8720c000   nsiproxy nsiproxy.sys Fri Nov 14 19:36:39 2014 (5466CA47)
87213000 873ea000   tcpip    tcpip.sys    Fri Nov 14 19:39:10 2014 (5466CADE)
873ea000 873f6000   TDI      TDI.SYS      Fri Nov 14 19:38:38 2014 (5466CABE)
873f6000 87400000   kdnic    kdnic.sys    Fri Nov 14 19:38:13 2014 (5466CAA5)

Unloaded modules:
82266000 82289000   cdrom.sys
    Timestamp: unavailable (00000000)
    Checksum:  00000000
    ImageSize:  00023000