RISC-V GNU tool chain and relocation sections in shared and static libraries.

The library code

const
unsigned char __clz_tab[] =
{
  0,1,2,2,3,3,3,3,4,4,4,4,4,4,4,4,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,5,
  6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,6,
  7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,
  7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,
  8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,
  8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,
  8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,
  8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,
};

long foo(long x)
{
    return __clz_tab[x % (sizeof(__clz_tab)/sizeof(__clz_tab[0]))];
}

Build the shared(.so) and static(.o) library.

slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-gcc  -shared -ffreestanding -nostdlib -fPIC  -o relocation.so relocation.c
slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-gcc  -c -ffreestanding -nostdlib -fPIC  -o relocation.o relocation.c

The shared library has a .got (Global Offset Table) section to reference the __clz_tab array. The .got section is adjusted by the loader with the relocation data from the .rela.dyn section.

slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-objdump -hr relocation.so

relocation.so:     file format elf64-littleriscv

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .hash         00000030  00000000000000e8  00000000000000e8  000000e8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  1 .dynsym       000000a8  0000000000000118  0000000000000118  00000118  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  2 .dynstr       00000027  00000000000001c0  00000000000001c0  000001c0  2**0
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  3 .rela.dyn     00000018  00000000000001e8  00000000000001e8  000001e8  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .text         00000038  0000000000000200  0000000000000200  00000200  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  5 .rodata       00000100  0000000000000238  0000000000000238  00000238  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  6 .dynamic      000000e0  0000000000001338  0000000000001338  00000338  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  7 .got          00000020  0000000000001418  0000000000001418  00000418  2**3
                  CONTENTS, ALLOC, LOAD, DATA
  8 .comment      00000011  0000000000000000  0000000000000000  00000438  2**0
                  CONTENTS, READONLY
                  
slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-objdump --disassemble relocation.so

relocation.so:     file format elf64-littleriscv


Disassembly of section .text:

0000000000000200 <foo>:
 200: fe010113           addi sp,sp,-32
 204: 00813c23           sd s0,24(sp)
 208: 02010413           addi s0,sp,32
 20c: fea43423           sd a0,-24(s0)
 210: fe843783           ld a5,-24(s0)
 214: 0ff7f793           andi a5,a5,255
 218: 00001717           auipc a4,0x1
 21c: 21873703           ld a4,536(a4) # 1430 <_GLOBAL_OFFSET_TABLE_+0x8>
 220: 00f707b3           add a5,a4,a5
 224: 0007c783           lbu a5,0(a5)
 228: 00078513           mv a0,a5
 22c: 01813403           ld s0,24(sp)
 230: 02010113           addi sp,sp,32
 234: 00008067           ret
 
slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-readelf -r  relocation.so

Relocation section '.rela.dyn' at offset 0x1e8 contains 1 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000001430  000300000002 R_RISCV_64        0000000000000238 __clz_tab + 0
slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-objdump -hr relocation.o

relocation.o:     file format elf64-littleriscv

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         00000038  0000000000000000  0000000000000000  00000040  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data         00000000  0000000000000000  0000000000000000  00000078  2**0
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000000  0000000000000000  0000000000000000  00000078  2**0
                  ALLOC
  3 .rodata       00000100  0000000000000000  0000000000000000  00000078  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .comment      00000012  0000000000000000  0000000000000000  00000178  2**0
                  CONTENTS, READONLY
RELOCATION RECORDS FOR [.text]:
OFFSET           TYPE              VALUE 
0000000000000018 R_RISCV_GOT_HI20  __clz_tab
000000000000001c R_RISCV_PCREL_LO12_I  .L0 

The static library also has a .got (Global Offset Table) section to reference the __clz_tab array. The .got section is adjusted by the loader with the relocation data from the .rela.dyn section.

slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-objdump -hr relocation.o

relocation.o:     file format elf64-littleriscv

Sections:
Idx Name          Size      VMA               LMA               File off  Algn
  0 .text         00000038  0000000000000000  0000000000000000  00000040  2**2
                  CONTENTS, ALLOC, LOAD, RELOC, READONLY, CODE
  1 .data         00000000  0000000000000000  0000000000000000  00000078  2**0
                  CONTENTS, ALLOC, LOAD, DATA
  2 .bss          00000000  0000000000000000  0000000000000000  00000078  2**0
                  ALLOC
  3 .rodata       00000100  0000000000000000  0000000000000000  00000078  2**3
                  CONTENTS, ALLOC, LOAD, READONLY, DATA
  4 .comment      00000012  0000000000000000  0000000000000000  00000178  2**0
                  CONTENTS, READONLY
RELOCATION RECORDS FOR [.text]:
OFFSET           TYPE              VALUE 
0000000000000018 R_RISCV_GOT_HI20  __clz_tab
000000000000001c R_RISCV_PCREL_LO12_I  .L0 

slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-objdump --disassemble relocation.o

relocation.o:     file format elf64-littleriscv


Disassembly of section .text:

0000000000000000 <foo>:
   0: fe010113           addi sp,sp,-32
   4: 00813c23           sd s0,24(sp)
   8: 02010413           addi s0,sp,32
   c: fea43423           sd a0,-24(s0)
  10: fe843783           ld a5,-24(s0)
  14: 0ff7f793           andi a5,a5,255

0000000000000018 <.L0 >:
  18: 00000717           auipc a4,0x0
  1c: 00073703           ld a4,0(a4) # 18 <.L0 >
  20: 00f707b3           add a5,a4,a5
  24: 0007c783           lbu a5,0(a5)
  28: 00078513           mv a0,a5
  2c: 01813403           ld s0,24(sp)
  30: 02010113           addi sp,sp,32
  34: 00008067           ret
  
slava@slava-P34V2:/work/risc-v/musl-riscv/tmp$ riscv64-unknown-linux-gnu-readelf -r  relocation.o

Relocation section '.rela.text' at offset 0x2a8 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000018  000800000014 R_RISCV_GOT_HI20  0000000000000000 __clz_tab + 0
00000000001c  000600000018 R_RISCV_PCREL_LO1 0000000000000018 .L0  + 0

Magenta RISC-V is running in kernel mode with threads.

Great news!

Magenta RISC-V port ( riscv-magenta ) reached a milestone. The kernel is running and switching between kernel threads.

RISC-V Magenta kernel manual stack unwinding

A manual stack unwinding when handle_exception sets a call frame

(gdb) bt
#0  0xffffffff8002e0c4 in _panic (caller=0xffffffff80002554 <do_trap_insn_misaligned+52>, frame=0xffffffff80040de8 <init_thread_union+7656>, fmt=0xffffffff800343c8 "%s unimplemented\n")
    at kernel/lib/debug/debug.c:32
#1  0xffffffff80002490 in do_trap_error (regs=0xffffffff80040e08 <init_thread_union+7688>, signo=7, code=1, addr=18446744071562076884, str=0xffffffff80034420 "Oops - instruction address misaligned")
    at kernel/arch/riscv/traps.c:52
#2  0xffffffff80002554 in do_trap_insn_misaligned (regs=0xffffffff80040e08 <init_thread_union+7688>) at kernel/arch/riscv/traps.c:67
#3  0xffffffff8000027c in handle_exception () at kernel/arch/riscv/rv64/exception.S:221
Backtrace stopped: frame did not save the PC

A frame stack has the following layout

struct frame{
    coid* caller_s0; // $s0
    void* caller_pc; // $ra
}

the current frame pointer is saved in the s0 register.

(gdb) p/x $s0
$1 = 0xffffffff80040d88
(gdb) x/2xg 0xffffffff80040d88-16
0xffffffff80040d78 <init_thread_union+7544>: 0xffffffff80040de8 0xffffffff80002490
(gdb) x/2xg 0xffffffff80040de8-16
0xffffffff80040dd8 <init_thread_union+7640>: 0xffffffff80040e08 0xffffffff80002554
(gdb) x/2xg 0xffffffff80040e08-16
0xffffffff80040df8 <init_thread_union+7672>: 0xffffffff80040f30 0xffffffff8000027c
(gdb) x/2xg 0xffffffff80040f30-16
0xffffffff80040f20 <init_thread_union+7968>: 0xffffffff80040f70 0xffffffff800022d4
(gdb) x/2xg 0xffffffff80040f70-16
0xffffffff80040f60 <init_thread_union+8032>: 0xffffffff80040fb0 0xffffffff80008f20
(gdb) x/2xg 0xffffffff80040fb0-16
0xffffffff80040fa0 <init_thread_union+8096>: 0xffffffff80040fe0 0xffffffff80008ff8
(gdb) x/5i 0xffffffff8000027c
   0xffffffff8000027c <handle_exception+352>: ld s1,256(sp)
   0xffffffff80000280 <handle_exception+356>: csrci sstatus,2
   0xffffffff80000284 <handle_exception+360>: andi s1,s1,256
   0xffffffff80000288 <handle_exception+364>: bnez s1,0xffffffff80000294 <handle_exception+376>
   0xffffffff8000028c <handle_exception+368>: addi s1,sp,280
(gdb) x/5i 0xffffffff800022d4
   0xffffffff800022d4 <arch_thread_construct_first+36>: sw a5,0(a4)
   0xffffffff800022d8 <arch_thread_construct_first+40>: jal ra,0xffffffff800021e4 <current_thread_info>
   0xffffffff800022dc <arch_thread_construct_first+44>: sd a0,-40(s0)
   0xffffffff800022e0 <arch_thread_construct_first+48>: ld a5,-40(s0)
   0xffffffff800022e4 <arch_thread_construct_first+52>: ld a4,-56(s0)
(gdb) x/5i 0xffffffff80008f20
   0xffffffff80008f20 <thread_construct_first+188>: addi a5,s0,-48
   0xffffffff80008f24 <thread_construct_first+192>: li a2,0
   0xffffffff80008f28 <thread_construct_first+196>: mv a1,a5
   0xffffffff80008f2c <thread_construct_first+200>: lui a5,0x80045
   0xffffffff80008f30 <thread_construct_first+204>: addi a0,a5,-1232
(gdb) x/5i 0xffffffff80008ff8
   0xffffffff80008ff8 <thread_init_early+112>: jal ra,0xffffffff80007678 <sched_init_early>
   0xffffffff80008ffc <thread_init_early+116>: nop
   0xffffffff80009000 <thread_init_early+120>: ld ra,40(sp)
   0xffffffff80009004 <thread_init_early+124>: ld s0,32(sp)
   0xffffffff80009008 <thread_init_early+128>: ld s1,24(sp)

GDB can not unwind after handle_exception as it is unable to verify that the handle_exception frame is valid, the handle_exception function has been written on assembler with a prolog that restores sp from a scratch register instead of a frame initalization. I added a frame pointer saving for handle_exception after the stack pointer restoration ( https://github.com/slavaim/riscv-magenta/blob/riscv/kernel/arch/riscv/rv64/exception.S ).

    /*a call frame to facilitate with debugging*/
    .macro SET_GDB_FRAME
    addi    sp, sp, -2*SZREG /* allocate the frame */
    REG_S   s0, 0(sp)        /* get the frame pointer at the exception moment */
    csrr    s0, sepc         /* get the exception PC */
    REG_S   s0, SZREG(sp)    /* set the exception PC as $ra for the frame */
    addi    s0, sp, 2*SZREG  /* set s0 to the current frame pointer */
    .endm

    .macro DEL_GDB_FRAME
    REG_L   s0, 0(sp)        /* restore the caller frame pointer */
    addi    sp, sp, 2*SZREG  /* restore the stack pointer */
    .endm

From the frame before the exception handler we see that the arch_thread_construct_first raised the exception. We need to examine this function prologue to get the offset to the bootom of the stack from the frame address.

(gdb) x/10i arch_thread_construct_first
   0xffffffff800022b0 <arch_thread_construct_first>: addi sp,sp,-64
   0xffffffff800022b4 <arch_thread_construct_first+4>: sd ra,56(sp)
   0xffffffff800022b8 <arch_thread_construct_first+8>: sd s0,48(sp)
   0xffffffff800022bc <arch_thread_construct_first+12>: sd s1,40(sp)
   0xffffffff800022c0 <arch_thread_construct_first+16>: addi s0,sp,64
   0xffffffff800022c4 <arch_thread_construct_first+20>: mv s1,ra
   0xffffffff800022c8 <arch_thread_construct_first+24>: sd a0,-56(s0)
   0xffffffff800022cc <arch_thread_construct_first+28>: li a4,0
   0xffffffff800022d0 <arch_thread_construct_first+32>: li a5,1
=> 0xffffffff800022d4 <arch_thread_construct_first+36>: sw a5,0(a4)

Now the $pc, $sp and $s0 (a frame pointer ) registers can be set to unwind the stack before handle_exception was called by a CPU. 64 bytes was subtracted from the s0 register value to get the sp register value according to the function prologue displayed above.

(gdb) set $pc=0xffffffff800022d4
(gdb) set $sp=0xffffffff80040f70-64
(gdb) set $s0=0xffffffff80040f70
(gdb) bt
#0  0xffffffff800022d4 in arch_thread_construct_first (t=0xffffffff800426d8 <idle_threads>) at kernel/arch/riscv/thread.c:34
#1  0xffffffff80008f20 in thread_construct_first (t=0xffffffff800426d8 <idle_threads>, name=0xffffffff80035368 "bootstrap") at kernel/kernel/thread.c:1016
#2  0xffffffff80008ff8 in thread_init_early () at kernel/kernel/thread.c:1037
#3  0xffffffff80003ec4 in lk_main () at kernel/top/main.c:53
#4  0xffffffff8000146c in _riscv_start () at kernel/arch/riscv/rv64/start.S:42
Backtrace stopped: frame did not save the PC