Linux kernel debugging with GDB: getting a task running on a CPU

The current task is saved in per-cpu space for x86-64 and is accessed through the gs  register at current_task offset as

mov    %gs:0xd440,%rdx

(gdb) p/x &current_task
$63 = 0xd440
(gdb) p/x __per_cpu_offset[0]
$64 = 0xffff88001fc00000
(gdb) x/gx 0xffff88001fc00000+0xd440
0xffff88001fc0d440: 0xffff88001dea6a00
(gdb) p/d ((struct task_struct*)0xffff88001dea6a00)->pid
$67 = 243
(gdb) p/x ((struct task_struct*)0xffff88001dea6a00)->mm
$69 = 0xffff88001d1bc800
(gdb) p/x ((struct task_struct*)0xffff88001dea6a00)->active_mm
$70 = 0xffff88001d1bc800
(gdb) p/x __per_cpu_offset[2]
$73 = 0xffff88001fd00000
(gdb) x/gx 0xffff88001fd00000+0xd440
0xffff88001fd0d440: 0xffff88001f240000
(gdb) p/x ((struct task_struct*)0xffff88001f240000)->pid
$74 = 0x1
(gdb) lx-ps
0xffffffff81e104c0 <init_task> 0 swapper/0
0xffff88001f240000 1 systemd
0xffff88001f240d40 2 kthreadd
0xffff88001f2427c0 4 kworker/0:0H
0xffff88001f244240 6 mm_percpu_wq
0xffff88001f244f80 7 ksoftirqd/0
0xffff88001f245cc0 8 rcu_sched
0xffff88001f246a00 9 rcu_bh
0xffff88001f298000 10 migration/0
0xffff88001f298d40 11 watchdog/0
0xffff88001f29c240 12 cpuhp/0
0xffff88001f29cf80 13 cpuhp/1
0xffff88001f29dcc0 14 watchdog/1
0xffff88001f29ea00 15 migration/1
0xffff88001f2c8000 16 ksoftirqd/1
0xffff88001f2c9a80 18 kworker/1:0H
0xffff88001f2ca7c0 19 cpuhp/2
0xffff88001f2cb500 20 watchdog/2
0xffff88001f2cc240 21 migration/2
0xffff88001f2ccf80 22 ksoftirqd/2
0xffff88001f2cea00 24 kworker/2:0H
0xffff88001f310000 25 cpuhp/3
0xffff88001f310d40 26 watchdog/3
0xffff88001f311a80 27 migration/3
0xffff88001f3127c0 28 ksoftirqd/3
0xffff88001f314240 30 kworker/3:0H
0xffff88001f314f80 31 kdevtmpfs
0xffff88001f315cc0 32 netns
0xffff88001dc28000 34 khungtaskd
0xffff88001dc28d40 35 oom_reaper
0xffff88001dc29a80 36 writeback
0xffff88001dc2a7c0 37 kcompactd0
0xffff88001dc2b500 38 ksmd
0xffff88001dc2c240 39 crypto
0xffff88001dc2cf80 40 kintegrityd
0xffff88001dc2dcc0 41 bioset
0xffff88001dc2ea00 42 kblockd
0xffff88001dcd8000 43 ata_sff
0xffff88001dcd8d40 44 md
0xffff88001dcd9a80 45 edac-poller
0xffff88001dcda7c0 46 devfreq_wq
0xffff88001dcdb500 47 watchdogd
0xffff88001dcdc240 48 kworker/1:1
0xffff88001dcdcf80 49 kworker/2:1
0xffff88001dcddcc0 50 kworker/3:1
0xffff88001ddf8000 52 kauditd
0xffff88001ddf8d40 53 kswapd0
0xffff88001ddf9a80 54 bioset
0xffff88001ddfa7c0 55 ecryptfs-kthrea
0xffff88001dff0d40 72 kthrotld
0xffff88001dff1a80 73 acpi_thermal_pm
0xffff88001dff27c0 74 bioset
0xffff88001dff3500 75 bioset
0xffff88001dff4240 76 bioset
0xffff88001dff4f80 77 bioset
0xffff88001dff5cc0 78 bioset
0xffff88001dff6a00 79 bioset
0xffff88001dff0000 80 bioset
0xffff88001d660000 81 bioset
0xffff88001d660d40 82 scsi_eh_0
0xffff88001d661a80 83 scsi_tmf_0
0xffff88001d6627c0 84 scsi_eh_1
0xffff88001d663500 85 scsi_tmf_1
0xffff88001d718d40 91 ipv6_addrconf
0xffff88001d71dcc0 104 charger_manager
0xffff88001d71a7c0 105 bioset
0xffff88001d71ea00 106 bioset
0xffff88001d71c240 107 bioset
0xffff88001d719a80 110 jbd2/sda-8
0xffff88001d718000 111 ext4-rsv-conver
0xffff88001ddfdcc0 123 kworker/1:1H
0xffff88001ddfcf80 124 kworker/2:1H
0xffff88001ddfc240 127 kworker/0:1H
0xffff88001f350d40 135 kworker/3:2
0xffff88001dea4240 137 kworker/1:2
0xffff88001d0b0d40 140 systemd-journal
0xffff88001dea27c0 142 kworker/2:2
0xffff88001dea0d40 146 kworker/0:3
0xffff88001ded6a00 153 systemd-udevd
0xffff88001dea0000 156 kworker/3:1H
0xffff88001dea5cc0 227 cron
0xffff88001dea1a80 229 rsyslogd
0xffff88001ded0d40 235 in:imuxsock
0xffff88001ded0000 236 in:imklog
0xffff88001ded27c0 237 rs:main Q:Reg
0xffff88001ded1a80 233 agetty
0xffff88001c5d8d40 234 login
0xffff88001dea6a00 243 bash
0xffff88001dea3500 248 kworker/u8:2
0xffff88001c5d9a80 251 kworker/0:1
0xffff88001c5dc240 445 kworker/u8:1
0xffff88001c5ddcc0 452 kworker/u8:0

Unwinding a kernel mode stack for exception in Linux.

Generally GDB is unable to unwind a kernel call stack with an exception frame on it. The unwinding stops on an exception processing. For example

(gdb) bt
#0  delay_tsc (__loops=5241148) at ../arch/x86/lib/delay.c:78
#1  0xffffffff8134b532 in __delay (loops=<optimised out>) at ../arch/x86/lib/delay.c:160
#2  __const_udelay (xloops=<optimised out>) at ../arch/x86/lib/delay.c:174
#3  0xffffffff81132620 in panic (fmt=<optimised out>) at ../kernel/panic.c:297
#4  0xffffffff8101f080 in oops_end (flags=70, regs=0xffffc90000227c18, signr=9) at ../arch/x86/kernel/dumpstack.c:235
#5  0xffffffff8104d1c7 in no_context (regs=0xffffc90000227c18, error_code=0, address=8, signal=<optimised out>, si_code=<optimised out>) at ../arch/x86/mm/fault.c:867
#6  0xffffffff8104d456 in __bad_area_nosemaphore (regs=0xffffc90000227c18, error_code=0, address=8, vma=<optimised out>, si_code=196609) at ../arch/x86/mm/fault.c:953
#7  0xffffffff8104d59f in bad_area_nosemaphore (regs=<optimised out>, error_code=<optimised out>, address=<optimised out>, vma=<optimised out>) at ../arch/x86/mm/fault.c:960
#8  0xffffffff8104d8e7 in __do_page_fault (regs=0xffffc90000227c18, error_code=0, address=8) at ../arch/x86/mm/fault.c:1387
#9  0xffffffff8104dccc in do_page_fault (regs=<optimised out>, error_code=<optimised out>) at ../arch/x86/mm/fault.c:1508
#10 0xffffffff8193ecd2 in page_fault () at ../arch/x86/entry/entry_64.S:1005
#11 0xffff88001decdb40 in ?? ()
#12 0xffff88001c02ae48 in ?? ()
#13 0xffffc90000227d98 in ?? ()
#14 0xffff88001d733000 in ?? ()
#15 0xffffc90000227cf0 in ?? ()
#16 0xffff88001d733000 in ?? ()
#17 0xffff88001cd02510 in ?? ()
#18 0xffffc90000227e18 in ?? ()
#19 0x0000000000000000 in ?? ()

Linux has a structure struct pt_regs to save thread context state. A pointer to this strucrue is provided to an exception processing routine and contains a context of a thread when an exception happened. Using register values from this structure a call stack at the moment of exception can be captured with GDB.

(gdb) f 8
#8  0xffffffff8104d8e7 in __do_page_fault (regs=0xffffc90000227c18, error_code=0, address=8) at ../arch/x86/mm/fault.c:1387
1387    bad_area_nosemaphore(regs, error_code, address, NULL);

Having a valid regs pointer set register values.

(gdb) p/x *regs
$2 = {r15 = 0xffff88001decdb40, r14 = 0xffff88001c02ae48, r13 = 0xffffc90000227d98, r12 = 0xffff88001d733000, bp = 0xffffc90000227cf0, bx = 0xffff88001d733000, r11 = 0xffff88001cd02510, 
  r10 = 0xffffc90000227e18, r9 = 0x0, r8 = 0xffff88001fc9d180, ax = 0x0, cx = 0x0, dx = 0x1000, si = 0xffff88001d733000, di = 0xffffc90000227d00, orig_ax = 0xffffffffffffffff, ip = 0xffffffff811aa30c, 
  cs = 0x10, flags = 0x246, sp = 0xffffc90000227cc8, ss = 0x18}
(gdb) set $rsp=0xffffc90000227cc8
(gdb) set $rip=0xffffffff811aa30c.
(gdb) set $rbp=0xffffc90000227cf0
(gdb) set $rbx=0xffff88001d733000
(gdb) set $r15=0xffff88001decdb40
(gdb) set $r14=0xffff88001c02ae48
(gdb) set $r13=0xffffc90000227d98
(gdb) set $r12=0xffff88001d733000
(gdb) set $r11=0xffff88001cd02510
(gdb) set $r10=0xffffc90000227e18
(gdb) set $r9=0
(gdb) set $rsi=0xffff88001d733000
(gdb) set $rdi=0xffffc90000227d00

Now a call stack at the momemnt of exception can be examined.

(gdb) bt
#0  __read_once_size (size=<optimised out>, res=<optimised out>, p=<optimised out>) at ../include/linux/compiler.h:254
#1  __read_seqcount_begin (s=<optimised out>) at ../include/linux/seqlock.h:112
#2  raw_read_seqcount_begin (s=<optimised out>) at ../include/linux/seqlock.h:147
#3  read_seqcount_begin (s=<optimised out>) at ../include/linux/seqlock.h:164
#4  get_fs_root_rcu (root=<optimised out>, fs=<optimised out>) at ../fs/dcache.c:3222
#5  d_path (path=0xffffc90000227d00, buf=0xffff88001d733000 "", buflen=4096) at ../fs/dcache.c:3265
#6  0xffffffffc0000076 in redirfs_get_filename ()
#7  0xffffffffc0014121 in dummyflt_release (context=<optimised out>, args=0xffffc90000227d98) at /work/redirfs/src/dummyflt/dummyflt.c:104
#8  0xffffffffc000892e in rfs_precall_flts ()
#9  0xffffffffc0002a42 in rfs_release ()
#10 0xffffffff81193a7a in __fput (file=0xffff88001cd02500) at ../fs/file_table.c:209
#11 0xffffffff81193bb9 in ____fput (work=<optimised out>) at ../fs/file_table.c:245
#12 0xffffffff810758b9 in task_work_run () at ../kernel/task_work.c:116
#13 0xffffffff8105da35 in exit_task_work (task=<optimised out>) at ../include/linux/task_work.h:21
#14 do_exit (code=<optimised out>) at ../kernel/exit.c:878
#15 0xffffffff8105f14e in do_group_exit (exit_code=0) at ../kernel/exit.c:982
#16 0xffffffff8105f1bf in SYSC_exit_group (error_code=<optimised out>) at ../kernel/exit.c:993
#17 SyS_exit_group (error_code=<optimised out>) at ../kernel/exit.c:991
#18 0xffffffff8193d060 in entry_SYSCALL_64 () at ../arch/x86/entry/entry_64.S:203
#19 0x0000000000000000 in ?? ()




This is how Linux kernel debugging looks like

Linux dynamic linking for elf binary execution

Follow this link https://github.com/slavaim/linux-notes/blob/master/dynamic-linking.md where I used GDB to show how dynamic linking on first function invocation is implemented in Linux.

(gdb) bt
#0  strcmp () at ../sysdeps/x86_64/multiarch/../strcmp.S:142
#1  0x00007ffff7deb1d4 in _dl_name_match_p (name=0x7ffff7bd83fe "libc.so.6", map=<optimised out>) at dl-misc.c:295
#2  0x00007ffff7de402f in do_lookup_x (new_hash=new_hash@entry=2090629905, old_hash=old_hash@entry=0x7fffffffdcb0, 
    result=result@entry=0x7fffffffdcc0, scope=<optimised out>, i=<optimised out>, i@entry=0, flags=flags@entry=1, skip=skip@entry=0x0, 
    undef_map=undef_map@entry=0x7ffff7ff9520) at dl-lookup.c:462
#3  0x00007ffff7de4961 in _dl_lookup_symbol_x (undef_name=0x7ffff7bd83f9 "puts", undef_map=0x7ffff7ff9520, ref=ref@entry=0x7fffffffdd78, 
    symbol_scope=0x7ffff7ff9878, version=0x7ffff7fcf030, type_class=type_class@entry=1, flags=1, skip_map=skip_map@entry=0x0)
    at dl-lookup.c:737
#4  0x00007ffff7de9527 in _dl_fixup (l=<optimised out>, reloc_arg=<optimised out>) at ../elf/dl-runtime.c:111
#5  0x00007ffff7df04d5 in _dl_runtime_resolve () at ../sysdeps/x86_64/dl-trampoline.S:45
#6  0x00007ffff7bd86c5 in foo () from /work/test/1/shared.o
#7  0x00000000004006f5 in main () at test.c:8

Debugging Linux kernel on Intel Quark X100 .

For source level debugging I use OLIMEX JTAG . This is a fast guide how to configure it on Linux machine.  A more detailed description can be found in Intel documentation Source Level Debug using OpenOCD/GDB/Eclipse on Intel ® QuarkTM SoC X1000

Compile and install OpenOCD with Quark support.

$ git clone git://git.code.sf.net/p/openocd/code openocd-code
$ cd openocd-code
$ git branch quark v0.8.0
$ git checkout quark
$ ./bootstrap
$ ./configure --enable-ftdi
$ make
$ sudo make install

By default OpenOCD is installed in the following folders

/usr/local/bin/openocd
/usr/local/share/openocd/scripts

Connect an OLIMEX JTAG to a board and your machine

Intel Galileo with OLIMEX JTAG

Power on a board and start an OpenOCD session for Quark X1000. This will start a GDB server on 3333 port.

$ openocd -f ./interface/ftdi/olimex-arm-usb-ocd-h.cfg -f target/quark_x10xx.cfg

If OpenOCD managed to locate a JTAG and connect to a board the output will be

Start GDB and attach to a GDB server started by OpenOCD. Provide GDB with a path to a kernel compiled with debug symbols.

$ gdb
(gdb) target remote localhost:3333
(gdb) monitor halt
(gdb) symbol-file <PATH TO LINUX SOURCE>/vmlinux
(gdb) c

Ctrl-C will break kernel execution into the debugger.

JTAG: Windows IoT debugging on Intel Galileo 2

As it was promised I will outline the steps to debug Windows IoT kernel running on Intel Galileo using JTAG interface .  Why does somebody need this? JTAG interface gives more power over hardware and provides deeper insight into hardware state.

  For debugging over JTAG using Windows host you will need

    - a hardware JTAG debugger, I use OLIMEX ARM-USB-OCD-H
    - an adapter from JTAG debugger to 10 pin connector, e.g. ARM-JTAG-20-10
    - OpenOCD to communicate with JTAG debugger and provide a debug server connection for GDB

    - MinGW for GDB

After everything have been assembled it looks like this
  


Then you need to start OpenOCD from cmd prompt by executing the following command
openocd-x64-0.9.0-dev.exe -f interface/ftdi/olimex-arm-usb-ocd-h.cfg  -f target/quark_x10xx.cfg

For example this is an output om my PC

this provides you with a GDB server on the port 3333 and telnet connection with the OpenOCD on the port 4444. So you can connect GDB to the server by providing a command target remote localhost:3333 to GDB, for example

 
you can also connect to OpenOCD via your favorite telnet client, mine is PuTTY