The document provides a detailed guide on x86_64 hardware, including setup commands, installation tips for various Linux components, and performance testing strategies. It covers kernel updates, PCIe configurations, and NVMe SSD management with various commands and tools. Additionally, it includes benchmarks for performance evaluation of different storage solutions and configurations.

1. x86_64 Hardware Deep dive
6th May, 2020 SAKURA Internet, Inc. Research Center SR / Naoto MATSUMOTO
(C) Copyright 1996-2020 SAKURA Internet Inc

2. x86_64 Hardware Deep dive cheatsheet
2
# uname -sr; tail -1 /etc/lsb-release
Linux 4.15.0-72-generic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
Setup commands
# apt update
# apt upgrade -y
# apt install -y hwloc hwloc-nox
# tar xzvf ./mlc_v3.7.tgz (Download from Intel.com)
# update-pciids
Usage commands
# lscpu
# lshw
# lstopo
# lspci –tv
# Linux/mlc
SOURCE: SAKURA Internet Research Center (2020/05)

3. GCC 9.2 Install TIPS
3
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-headers-5.3.0-050300rc4_...201908111734_all.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-headers-5.3.0-050300rc4-...201908111734_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-modules-5.3.0-050300rc4-...201908111734_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.3-rc4/linux-image-unsigned-5.3.0-050300rc4-...11734_amd64.deb
# dpkg -i linux-headers-5.3.0-050300rc4_5.3.0-050300rc4.201908111734_all.deb
# dpkg -i linux-headers-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb
# dpkg -i linux-modules-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb
# dpkg -i linux-image-unsigned-5.3.0-050300rc4-generic_5.3.0-050300rc4.201908111734_amd64.deb
# sync; sync; sync; reboot
# uname -sr; tail -1 /etc/lsb-release
Linux 5.3.0-050300rc4-generic
DISTRIB_DESCRIPTION="Ubuntu 19.04"
# apt install build-essential libgmp-dev libmpfr-dev libmpc-dev zlib1g-dev numactl -y
# wget http://ftp.tsukuba.wide.ad.jp/software/gcc/releases/gcc-9.2.0/gcc-9.2.0.tar.gz
# tar xzvf ./gcc-9.2.0.tar.gz
# ../configure --prefix=/usr --disable-multilib --with-system-zlib --enable-languages=c,c++,d,fortran,go,objc,obj-c++
# make -j12
# make install
# gcc --version
gcc (GCC) 9.2.0
SOURCE: SAKURA Internet Research Center (2020/05)

4. PCIe P2P DMA Install TIPS
4
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-headers-5.2.0-050200rc6...201906222033_all.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-headers-5.2.0-050200rc6...201906222033_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-modules-5.2.0-050200rc6...201906222033_amd64.deb
# wget https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.2-rc6/linux-image-unsigned-5.2.0-050200rc6...22033_amd64.deb
# dpkg -i linux-headers-5.2.0-050200rc6_5.2.0-050200rc6.201906222033_all.deb
# dpkg -i linux-headers-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb
# dpkg -i linux-modules-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb
# dpkg -i linux-image-unsigned-5.2.0-050200rc6-generic_5.2.0-050200rc6.201906222033_amd64.deb
# sync; sync; sync; reboot
# apt install -y nvme-cli libbobcat-dev
# git clone http://www.github.com/sbates130272/p2pmem-test
# cd p2pmem-test/
# ./p2pmem-test /dev/nvme0n1 /dev/nvme1n1 /dev/p2pmem0 -s 1 -4k --check
Copy one 4KB chunk from /dev/nvme0n1 to /dev/nvme0n1 via the memory
exposed by /dev/p2pmem0. Perform a check on the data (i.e. write know
data to /dev/nvme0n1 and validate that by reading /dev/nvme1n1 after
the p2pmem based transfer).
https://github.com/sbates130272/donard (DRAFT)
# cd /nvme_donard
# cd /libdonard
# ./waf
# cd /libdonard/build/speed
# dd if=/dev/zero of=/<nvme_drive>/test1.dat bs=1K count=100K
# ./nvme2gpu_read -b 128M -D /<nvme_drive>/test1.dat
# ./nvme2gpu_read -b 128M =/<nvme_drive>/test1.dat
SOURCE: SAKURA Internet Research Center (2020/05)

5. PCIe 4.0 NVMe SSD(M.2) RAID0 TIPS
5
# lscpu
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 5 3600 6-Core Processor
# apt install nvme-cli fio -y
# update-pciids
# lspci -tv
# lspci -vv
# nvme list
Node Model
------------ ---------------------------
/dev/nvme0n1 GIGABYTE GP-ASM2NE6100TTTD
/dev/nvme1n1 GIGABYTE GP-ASM2NE6100TTTD
:
/dev/nvme3n1 Force MP600
# mdadm --create /dev/md0 --raid-devices=3 --level=0 /dev/nvme[013]n1
# mdadm --detail /dev/md0
Number Major Minor RaidDevice State
0 259 0 0 active sync /dev/nvme0n1
1 259 1 1 active sync /dev/nvme1n1
2 259 5 2 active sync /dev/nvme3n1
# mkfs.ext4 /dev/md0
# mkdir /mnt/gen4
# mount /dev/md0 /mnt/gen4/
# fio --directory=/mnt/gen4/ --bs=4k --rw=read --size=20G --numjobs=16 --group_reporting --name=test
read: IOPS=2553k, BW=9973MiB/s (10.5GB/s) ...
# lsmem
Total online memory: 16G
SOURCE: SAKURA Internet Research Center (2020/05)

6. PCIe 4.0 NVMe SSD(M.2) SMART-LOG TIPS
6
# uname -sr; tail -1 /etc/lsb-release
Linux 4.15.0-58-generic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
# nvme list
Node SN Model Format FW Rev
---------------- -------------------- ---------------------------------------- ---------------- --------
/dev/nvme0n1 SN192608967864 GIGABYTE GP-ASM2NE6100TTTD 512 B + 0 B EGFM11.0
/dev/nvme1n1 SN192608967963 GIGABYTE GP-ASM2NE6100TTTD 512 B + 0 B EGFM11.0
/dev/nvme2n1 1924822900012855002D Force MP600 512 B + 0 B EGFM11.0
/dev/nvme3n1 PHBT7160022A032E INTEL MEMPEK1W032GA 512 B + 0 B K3110300
# nvme smart-log /dev/nvme2n1
Smart Log for NVME device:nvme2n1 namespace-id:ffffffff
critical_warning : 0
temperature : 36 C
available_spare : 100%
available_spare_threshold : 5%
percentage_used : 0%
data_units_read : 4,493,212
data_units_written : 1,336,869
host_read_commands : 41,019,727
host_write_commands : 4,432,398
controller_busy_time : 22
power_cycles : 78
power_on_hours : 44
unsafe_shutdowns : 45
media_errors : 0
num_err_log_entries : 98
Warning Temperature Time : 0
SOURCE: SAKURA Internet Research Center (2020/05)

7. Null block device High Performance I/O TIPS
7
# apt update
# apt upgrade; sync; sync; sync; reboot
# uname -sr; tail -1 /etc/lsb-release
Linux 4.15.0-58-generic
DISTRIB_DESCRIPTION="Ubuntu 18.04.3 LTS"
# lscpu
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 5 3600 6-Core Processor
# rmmod null_blk
# modprobe null_blk nr_devices=8
# fio --bs=4k --size=128G --rw=read --numjobs=8 --runtime=40 --group_reporting=1
--name=nullb0 --filename=/dev/nullb0
--name=nullb1 --filename=/dev/nullb1
--name=nullb2 --filename=/dev/nullb2
--name=nullb3 --filename=/dev/nullb3
--name=nullb4 --filename=/dev/nullb4
--name=nullb5 --filename=/dev/nullb5
--name=nullb6 --filename=/dev/nullb6
--name=nullb7 --filename=/dev/nullb7
:
read: IOPS=7201k, BW=27.5GiB/s (29.5GB/s)(384GiB/13980msec)
# mount -t tmpfs -o size=8g tmpfs /mnt/ram
# fio --bs=4k --size=500m --rw=read --numjobs=8 --runtime=40 --group_reporting=1
--name=tmpfs --directory=/mnt/ram/
:
read: IOPS=6169k, BW=23.5GiB/s (25.3GB/s)(6000MiB/249msec)
SOURCE: SAKURA Internet Research Center (2020/05)