The document discusses porting Generic KASAN (Kernel Address SANitizer) to RISC-V Linux. It explains that Generic KASAN uses 1/8th of kernel memory for shadow memory and how to decide the shadow memory addresses for RISC-V. It also describes initializing the shadow memory mapping in early stage and mapping uncheckable areas to the early shadow page. The key steps are translating memory addresses to shadow addresses, allocating physical space for shadow memory, and poisoning the shadow memory. Patches for porting Generic KASAN to RISC-V were submitted to the Linux kernel mailing list.

1. Experience on porting
HIGHMEM to 32bit RISC-V Linux
Eric Lin
2020.8.2

2. About me
• 2016 ~ 2018 NCKU
– MS
• 2018.12 ~ 2020.7 Andes technology
– Software engineer in Linux kernel team
• Experience
– RISC-V Linux kernel
– Device driver
– U-BOOT
• Email:dslin1010@gmail.com

3. Outline
• Why need highmem?
• Porting highmem
• An end to high memory?

4. Why need high memory ?
• Earliest days, kernel maintained ′direct map ′ to map all physical memory in
kernel space.
– It easy for the kernel to manipulate any page in the system
• 32bit platform only have 4GB virtual address space
– To reduce TLB flush cost between kernel and user space.
– Split 4G address space to 1:3 ( 1GB => kernel, 3GB => user )
• With direct map, kernel can only map 1GB physical memory.
1G
3G
1G
0x00000000
0xC0000000
0xFFFFFFFF
Physical memory
User
kernel
(direct map)
va_pa_offset

5. Why need high memory? (cont.)
• Reserved ~896MB for linear mapping (direct map) => low memory
• > 896MB virtual address space :
– Temporary mapping
• PKMAP => kmap() 、VMALLOC => vmalloc()、vmap()
– Permanent mapping
• FIXMAP => DTB、early_ioremap()
(ZONE_NORMAL)
low mem
3G
1G
Physical memory > 1Guser
kernel
(ZONE_HIGHMEM)
high mem
linear mapping
( direct map )
896MB
0xC0000000
VMALLOC
PKMAP
FIXMAP
896MB
i386
va_pa_offset

6. Porting highmem
• Decide RV32 linux memory layout
– Refer other architecture (arm, x86, nds32)
– (option) Move VMALLOC、FIXMAP..etc after PAGE_OFFSET
• Leave user-process more address space.
• Add a PKMAP region in virtual address space
– for kmap()
• Temporary mapping for a single page
• alloc_page(__GFP_HIGHMEM) from ZONE_HIGHMEM
• Create a page table for pkmap
• Add memory slots in FIXMAP for kmap_atomic()
• Add architecture kmap() and kmap_atomic()

7. 7
• 64bit platform needn’t highmem => kernel have 128GB address space.
• After porting highmem, we would like RV32 linux memory layout as below:
RISC-V 5.6 Linux Memory layout
linear mapping
(direct map)
PKMAP
kernel
user 3G
VMALLOC
FIXMAP
Reserved
VMEMMAP
PCI_IO
linear mapping
(direct map)kernel
user
VMALLOC
FIXMAP
VMEMMAP
PCI_IO
0xffffffe0_00000000
0xffffffff_ffffffff0xffffffff
0xc0000000
0x00000000 0x00000000_00000000
128 GB
16777215 TB
PAGE_OFFSETPAGE_OFFSET
RV64RV32

8. Move memory layout
(arch/riscv/include/asm/pgtable.h)
+#define VMALLOC_SIZE (SZ_128M)
+/* Reserve 4MB from top of RAM to align with PGDIR_SIZE */
+#define VMALLOC_END (0xffc00000UL)
+#define VMALLOC_START (VMALLOC_END - VMALLOC_SIZE)
…..
51 #define VMEMMAP_END (VMALLOC_START - 1)
52 #define VMEMMAP_START (VMALLOC_START - VMEMMAP_SIZE)
+#ifdef CONFIG_HIGHMEM
+/* Set LOWMEM_END alignment with PGDIR_SIZE */
+#define LOWMEM_END (ALIGN_DOWN(PKMAP_BASE, SZ_4M))
+#define LOWMEM_SIZE (LOWMEM_END - PAGE_OFFSET)
+#endif /* CONFIG_HIGHMEM */
#define TASK_SIZE PAGE_OFFSET
---------------------------
(arch/riscv/include/asm/highmem.h)
+#define PKMAP_BASE (FIXADDR_START - SZ_2M)
linear mapping
(direct map)
PKMAP 2M
user 3G
VMALLOC 128M
FIXMAP 4M
Reserved 4M
VMEMMAP 16M
PCI_IO 16M
0xffffffff
0xc0000000
0x00000000
LOWMEM_END
• Locate VMALLOC_END and LOWMEM_END
• Must Reserved 4MB from top of RAM to
align with PGDIR_SIZE
• Add a new region for PKMAP
VMALLOC_END
TASK_SIZE

9. Porting highmem
setup_bootm()
– max_low_pfn => the end of low_memory
– max_pfn => the end of physical memory
(arch/riscv/mm/init.c)
150 void __init setup_bootmem(void)
151 {
…
188 #ifdef CONFIG_HIGHMEM
189 max_low_pfn = (PFN_DOWN(__pa(LOWMEM_END)));
190 max_pfn = PFN_DOWN(memblock_end_of_DRAM());
191 memblock_set_current_limit(__pa(LOWMEM_END));
192 #else
low mem
Physical memory
high mem
max_low_pfn
max_pfn
27 static void __init zone_sizes_init(void)
28 {
…….
34 #endif
35 max_zone_pfns[ZONE_NORMAL] = max_low_pfn;
36 #ifdef CONFIG_HIGHMEM
37 max_zone_pfns[ZONE_HIGHMEM] = max_pfn;
38 #endif
39 free_area_init_nodes(max_zone_pfns);
40 }

10. Porting highmem (cont.)
• Prepare a page table and set it to swapper_pg_dir
– swapper_pg_dir is a page directory pointer for kernel.
• 32 bit use 2 level page table (RISC-V)
pte entry
pgd
pmd (pkmap_p )
PAGE
swapper_pg_dir
Physical memory

11. Porting highmem (cont.)
+static void __init pkmap_init(void)
+{
.
+ /*
+ * Permanent kmaps:
+ */
+ vaddr = PKMAP_BASE;
+
+ pgd = swapper_pg_dir + pgd_index(vaddr);
+ p4d = p4d_offset(pgd, vaddr);
+ pud = pud_offset(p4d, vaddr);
+ pmd = pmd_offset(pud, vaddr);
+ pkmap_p = (pte_t *)__va(memblock_phys_alloc(PAGE_SIZE, PAGE_SIZE));
…….
+ memset(pkmap_p, 0, PAGE_SIZE);
+ pfn = PFN_DOWN(__pa(pkmap_p));
+ set_pmd(pmd, __pmd((pfn << _PAGE_PFN_SHIFT) |
+ pgprot_val(__pgprot(_PAGE_TABLE))));
+
+ /* Adjust pkmap page table base */
+ pkmap_page_table = pkmap_p + pte_index(vaddr);
start_kernel()
-> setup_arch
-> paging_init
->pkmap_init
• Add new function pkmap_init() for creating pkmap page table

12. Porting highmem (cont.)
//arch/riscv/include/asm/fixmap.h
enum fixed_addresses {
FIX_PTE,
FIX_PMD,
FIX_EARLYCON_MEM_BASE,
+#ifdef CONFIG_HIGHMEM
+ FIX_KMAP_RESERVED,
+ FIX_KMAP_BEGIN,
+ FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_TYPE_NR * NR_CPUS),
+#endif
+ __end_of_fixed_addresses,
};
23 #define FIXADDR_TOP (PKMAP_BASE)
24 #define FIXADDR_SIZE ((__end_of_fixed_addresses) << PAGE_SHIFT)
25 #define FIXADDR_START (FIXADDR_TOP - FIXADDR_SIZE)
FIXADDR_TOP
FIXADDR_START
(__end_of_fixed_address)
FIX_KMAP_BEGIN
4K
4K
4K
4K
FIX_KMAP_END
kmap_atomic
• Add memory slots
in FIXMAP

13. After porting
• If success, you will see …

14. After porting
• If fail, you will see …
????

15. An end to high memory?
• Upstream my first kernel patch, but …
• Arnd Bergmann (Linaro) reply …
– I would much prefer to not see highmem added to new architectures
at all if possible, see https://lwn.net/Articles/813201/
• Weiner like to improve memory-reclaim performance
– Inode-cache shrinking vs. highmem
• Inodes, being kernel data structures => low memory
• page-cache pages => can be placed in high memory
• With a large number of one-byte files on a 7G machine, it invoke inode
shrinker to reclaim inode with populated page cache. It can drop gigabytes of
hot and active page cache.
• Linus Torvalds say …

16. Reference
• https://www.kernel.org/doc/Documentation/vm/highmem.txt
• https://lwn.net/Articles/813201/
• https://lkml.org/lkml/2020/4/2/253

17. Port Generic KASAN on RISC V
胡峻銘(Nick Hu)

18. About me
• 2015 ~ 2017 in NCTU
– MS
• Work in Andes technology
– Software engineer in Linux kernel team
– 2017.12 ~ now
• Experience
– Linux kernel
• NDS32
– Perf, suspend to ram/standby, …
• RISCV
– Perf, KASAN, …
– drivers

19. Outline
• What is KASAN?
• How to use KASAN?
• How KASAN works?
• How to port Generic KASAN on RISC V
• Patch to upstream

20. What is KASAN
• Kernel Address SANitizer (KASAN)
• Dynamic memory error detector
– Out of bound
– Use after free
• It can detect
– Global variable
– Stack
– heap
• Work with slub/slab memory allocator

21. How to use KASAN
• Select CONFIG_KASAN
• Need GCC version 4.9.2 or later
– Compile-time instrumentation
• Choose
– CONFIG_KASAN_OUTLINE or
– CONFIG_KASAN_INLINE

22. How KASAN works
• Use the shadow memory to mark the status of
memory
– Use one byte shadow memory marks each 8 bytes
memory
– Use magic number fill the shadow memory
3Shadow memory
8 bytes memory

23. How KASAN works
• Compiler would insert the check code for
load/store instruction
– Example:
ffffffe0000022f4: 41e080e7 jalr 1054(ra) # ffffffe00036770e <__asan_load1>
ffffffe0000022f8: 00094783 lbu a5,0(s2)
ffffffe0000022fc: 00190b13 addi s6,s2,1
ffffffe000002300: cbc9 beqz a5,ffffffe000002392 <mount_block_root+0x148>
ffffffe000002302: 01879963 bne a5,s8,ffffffe000002314 <mount_block_root+0xca>
ffffffe000002306: 854a mv a0,s2
ffffffe000002308: 00365097 auipc ra,0x365
ffffffe00000230c: 450080e7 jalr 1104(ra) # ffffffe000367758 <__asan_store1>
ffffffe000002310: fe0b0fa3 sb zero,-1(s6)

24. How to port Generic KASAN on RISCV
• In Documentation/dev-tools/kasan.rst
– Generic KASAN dedicates 1/8th of kernel memory to
its shadow memory
• KASAN_SHADOW_SCALE_SHIFT=3
• Decide shadow memory address
– KASAN_SHADOW_SIZE
• (UL(1) << (38 - KASAN_SHADOW_SCALE_SHIFT))
– KASAN_SHADOW_START
• 0xffffffc000000000 /* 2^64 - 2^38 */
– KASAN_SHADOW_END
• (KASAN_SHADOW_START + KASAN_SHADOW_SIZE)

25. How to port Generic KASAN on RISCV
• Translate the memory address to the corresponding
shadow address
– kasan_mem_to_shadow()
– KASAN_SHADOW_OFFSET
• shadow_addr – (addr >>
KASAN_SHADOW_SCALE_SHIFT)
• (KASAN_SHADOW_END –
(1ULL <<(64 - KASAN_SHADOW_SCALE_SHIFT)))
static inline void *kasan_mem_to_shadow(const void *addr)
{
return (void *)((unsigned long)addr >> KASAN_SHADOW_SCALE_SHIFT)
+ KASAN_SHADOW_OFFSET;
}

26. How to port Generic KASAN on RISCV
• kasan_early_init()
– Init the mapping for shadow memory in early
stage
• All maps to ‘kasan_early_shadow_page’
• kasan_init()
– Mapping the memory area which don’t need
check to ‘kasan_early_shadow_page’
– Allocate physical space for shadow memory
• KASAN can poison the shadow memory

27. Patch to upstream
• V1
– https://lkml.org/lkml/2019/8/7/135
• V4
– https://lkml.org/lkml/2019/10/27/814

28. END