The document discusses OpenPOWER for data centers. It begins with an overview of the POWER8 CPU architecture, including its multi-core design, simultaneous multi-threading capabilities, large caches, and cryptographic accelerators. It then covers the reliability, availability and serviceability features of OpenPOWER systems, such as error correction and retry mechanisms. The document concludes with a discussion of the open source firmware in OpenPOWER systems, including the multi-step boot process involving the Self Boot Engine, HostBoot, On-Chip Controller, Skiboot and Petitboot.

1. OpenPOWER for the data center
@wefinet – Werner Fischer, Thomas-Krenn.AG
Image wheel loader (c) by Liebherr

2. Have you ever used POWER arch?

3. Have you ever used >1 CPU core?

4. Image (c) by Liebherr

5. _ Architecture
_ Performance
_ Open Source Firmware
OpenPOWER for the data center

6. _ Architecture
_ Overview
_ RAS – Reliability, Availability, Serviceability
_ Performance
_ Open Source Firmware
OpenPOWER for the data center

7. Performance
Optimization
With
Enhanced
RISC

8. POWER8 CPU

9. _ Multi-Core: 8, 10, or 12 Cores
_ 8x Simultaneous Multi-Threading
_ 64, 80, or 96 Threads per CPU!
POWER8 CPU
POWERful Cores

10. _ 8x Simultaneous Multi-Threading
_ POWER8 can do SMT-1 -2 -4 -8
_ Each Core has dedicated resources for
up to 4 threads!
_ SMT-4 sometimes superior to SMT-8
POWER8 CPU
POWERful Cores

11. POWER8 CPU
POWERful Cores
tk@ubuntu­16­04:~$ sudo lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 64
On­line CPU(s) list: 0­63
Thread(s) per core: 8
Core(s) per socket: 8
Socket(s): 1
NUMA node(s): 1
Model: 2.0 (pvr 004d 0200)
Model name: POWER8 (raw), altivec supported
CPU max MHz: 3857.0000
CPU min MHz: 2061.0000
L1d cache: 64K
L1i cache: 32K
L2 cache: 512K
L3 cache: 8192K
NUMA node0 CPU(s): 0­63

12. Linux Performance Profiling
and Monitoring
@wefinet – Werner Fischer,
Thomas-Krenn.AG
Image http://de.freeimages.com/photo/jaguar-xjr-1537119

13. http://www.brendangregg.com/blog/2017-05-09/cpu-utilization-is-wrong.html

14. _ L1 Cache:
_ 64KB Data / Core
_ 32KB Instr. / Core
_ L2 Cache: 512KB / Core
_ L3 Cache: 8MB / Core
_ L4 Cache in Memory Buffers
POWER8 CPU
POWERful Caches

15. POWER8 CPU _ 1) per Chip: „2nd gen Nest Accelerator
Complex (NX)“
_ Symmetric Crypto
_ Compression Engine
_ Random Number Generator
_ 2) per Core
_ Symmetric Crypto
_ Cyclic Redundancy Check
POWERful Accelerators

16. _ 1) per Chip: NX
_ Symmetric Crypto
_ 2x 842 (De-)Compression Engine
_ Random Number Generator
_ → Available via Bare-metal Linux
Source: http://de.slideshare.net/sebastienchabrolles/enabling-power-8-advanced-features-on-linux
POWERful Accelerators

17. POWER8 CPU _ 2) per Core
_ Symmetric Crypto
_ Cyclic Redundancy Check
POWERful Accelerators

18. _ OpenSSL >= 1.0.2
_ SLES 12 SP2
_ RHEL 7.41
_ Ubuntu 15.10
_ Debian 9.02
Source: http://de.slideshare.net/sebastienchabrolles/enabling-power-8-advanced-features-on-linux
1) https://bugzilla.redhat.com/show_bug.cgi?id=1276310
2) https://packages.debian.org/stretch/openssl
Crypto Accelerators

19. Source: http://de.slideshare.net/sebastienchabrolles/enabling-power-8-advanced-features-on-linux
Crypto Accelerators
without
acceleration:
190 seconds
with
acceleration:
23 seconds
→ 8,2x faster!

20. Source: http://de.slideshare.net/sebastienchabrolles/enabling-power-8-advanced-features-on-linux
1) https://www.thomas-krenn.com/de/wiki/Linux_Kernel_Versionen#Kernel_3.11
2) https://www.thomas-krenn.com/de/wiki/Linux_Kernel_Versionen#Kernel_4.2
Compression Engine
_ zswap since
Kernel 3.111
_ POWER HW Support
(NX-842) since
Kernel 4.22

21. CPU Overview
POWER8 Xeon 2600 v4
Cores 8 / 10 / 12 4 … 22
Multithreading 8x SMT 2x HT
Threads 64 / 80 / 96 8 … 44
L1 Cache (Instructions) 32 KB 32 KB
L1 Cache (Data) 64 KB 32 KB
L2 Cache 512 KB / Core
(4 / 5 / 6 MB)
256 KB / Core
(1 … 5,5 MB)
L3 Cache 8 MB / Core
(64 / 80 / 96 MB)
2,5 MB / Core
( 10 … 55 MB)
L4 Cache (Centaur Chips)
Idle Energy Optimization no yes

22. _ Architecture
_ Overview
_ RAS – Reliability, Availability, Serviceability
_ Performance
_ Open Source Firmware
OpenPOWER for the data center

23. Reliability
Availability
Serviceability

24. _ CPU address- and data-paths protected
_ parity or error-correcting codes (ECC)
_ Example: ECC error on a L2 cache line
→ line gets deactivated
„Everything has checksums“

25. „Processor Instruction Retry“
_ Within CPU core, soft error events might occur
_ Event can be detected before a failing
instruction is completed?
_ → CPU can try the operation again
_ Detected faults on the memory bus (CPU ↔
DIMM) can be tried again

26. „Enhanced PCIe error handling“
_ Outage/Crash of a PCIe card
→ Recovery Mechanism
_ Re-Init during runtime
_ Server Reboot NOT necessary

27. _ Architecture
_ Overview
_ RAS – Reliability, Availability, Serviceability
_ Performance
_ Open Source Firmware
OpenPOWER for the data center

28. Image wheel loader (c) by Liebherr

30. _ SMT-8 enables more parallel transactions
_ large caches minimize latency:
_ CPU can store data and do operations
before beginning writing to the memory
subsystem
_ high memory bandwidth (up to 230 GB/s
per CPU)
_ optimizations for the POWER Platform
http://de.slideshare.net/MariaDB/ibm-linux-onpowerch (Slide 17)

31. http://blog.2ndquadrant.com/more-power-for-postgresql/
https://www.postgresql.org/about/news/1705/
http://akorotkov.github.io/blog/2016/05/09/scalability-towards-millions-tps/

32. Test Environment 1/2
_ OpenPOWER single CPU 8 Core, 64 Threads 3,32 GHz
_ 512 GB RAM
_ MegaRAID SAS 9361-8i
_ 2x SSD 500 GB
_ Debian Jessie/pgapt
_ bare metal

33. Test Environment 2/2
_ 4-socket E850/POWER8 8 Cores, 3,7 GHz
_ 128 GB RAM
_ Ubuntu LTS 16.04, LPAR/PowerVM

34. PostgreSQL 9.4 vs. 9.5 vs. 9.6 mixed read/write performance
pgbench on OpenPOWER

35. PostgreSQL 9.5 vs. 9.6 read (SELECT) only
pgbench on OpenPOWER

36. Readonly scaling OpenPOWER vs. E850

37. Readonly scaling simple SELECTs

38. Readonly scaling 8 core vs. 28 cores
software = bottleneck
(not hardware ;-)

39. PostgreSQL vs. POWER8 SMT

40. Conclusions
_ „OpenPOWER is a very interesting platform to build an open
source stack with PostgreSQL starting with the firmware“
_ „POWER servers can consolidate Linux and Unix platform
solutions with high speed PostgreSQL“
_ „[Power]KVM as a virtualization alternative“

41. _ Architecture
_ Overview
_ RAS – Reliability, Availability, Serviceability
_ Performance
_ Open Source Firmware
OpenPOWER for the data center

42. Remember
21st
Oct 2014
?

44. *
) Independent BIOS Vendors
**
) Original Equipment Manufacturers
https://www.mitre.org/publications/technical-papers/presentation-extreme-privilege-escalation-on-windows-8uefi-systems
http://www.kb.cert.org/vuls/id/552286
22.11.2013 & 04.12.2013
Mitre informs
Intel and US CERT
Case# 14-2221
Intel contacts
IBVs*
& OEMs**
,
updates reference
code in May 2014
22.07.2014
CERT VU#552286
Note to vendors
07.08.2014
CERT VU#552286
published
21.10.2014
press reports
05.11.2014
still no update
from many vendors

45. *
) Independent BIOS Vendors
**
) Original Equipment Manufacturers
https://www.mitre.org/publications/technical-papers/presentation-extreme-privilege-escalation-on-windows-8uefi-systems
http://www.kb.cert.org/vuls/id/552286
22.11.2013 & 04.12.2013
Mitre informs
Intel and US CERT
Case# 14-2221
Intel contacts
IBVs*
& OEMs**
,
updates reference
code in May 2014
22.07.2014
CERT VU#552286
Note to vendors
07.08.2014
CERT VU#552286
published
21.10.2014
press reports
05.11.2014
still no update
from many vendors
1 Year (!)

47. … “no comment to
this topic” ...
“... they were surprised
by the request
or could not
answer so far ...”

49. Remember
1
st
May 2017
?

52. My Laptop waits for a BIOS update, too

53. Firmware within a Server

54. BMC
Intel
ME
UEFI
Firmware
CPU
Microcode
HDD SSD
RAIDController
NIC
GPU
Firmware within a Server

55. BMC
Intel
ME
UEFI
Firmware
CPU
Microcode
HDD SSD
Firmware within a Server
RAIDController
NIC
GPU

57. … ...

58. ?

59. .com/open-power

61. _ Self Boot Engine (SBE, ISTEPs 1-4)
_ HostBoot (ISTEP 5-21), z.B. ECC leeren
_ microkernel, has userspace
_ CPU bus init, memory init, core init,
_ On Chip Controller (OCC) – hard hw limits
_ PowerPC 405 core, hat eigenes Realtime-OS
_ SkiBoot (OPAL)
_ Linux / Petitboot (Bootloader)
_ Betriebssystem (Linux)
(ISTEP = IPL Step)
(IPL = Initial Program Load)

62. WOW - That's a lot
_ Yes, it's a lot.
_ ~600k unique LOC
_ ~24 million LOC from elsewhere (e.g. Linux, toolchain, libc,
ncruses, lvm, busybox etc)
_ A LOT of things happen before your computer is a computer
_ before the OS runs
_ before the screen works

63. How can they maintain this?
_ Only maintain what they HAVE TO
_ Take everything else from upstream
_ e.g. „Interface to choose the boot media“
_ Option 1: develop own interface,
own method to detect discs, PCI, ...
→ a lot of work!

65. https://www.thomas-krenn.com/en/wiki/Linux_Storage_Stack_Diagram

66. How can they maintain this?
_ Only maintain what they HAVE TO
_ Take everything else from upstream
_ e.g. „Interface to choose the boot media“
_ Option 1: develop own interface,
own method to detect discs, PCI, ...
→ a lot of work!
_ Option 2: Linux is already there,
it has disc drivers, display drivers, ...

67. Was they HAVE TO
_ POWER specific
_ Hostboot
_ OCC
_ Skiboot (OPAL)
_ Generic
_ Petitboot
_ Op-build (build infrastructure) → buildroot → openwrt
_ Flash Manipulating Utilities

68. Was they don't have to
_ Linux (→ use upstream)
_ Userspace for Petitboot (→ taken from buildroot)
_ Build tooling (→ use buildroot)
_ Contributions UPSTREAMs first
_ local patch only when they HAVE TO
_ e.g. 8 Patches for Upstream Linux
(including „-openpower“ for the version string ;-)

69. Development process
_ Hostboot
_ github issues / pull requests
_ most dev done internally before chip exists
_ OCC
_ dev done internally (IBM)
_ Skiboot
_ mailing lists / github
_ Petitboot
_ mailing lists / github
_ op-build
_ github issues / pull requests

70. Interaction with UPSTREAM
_ Linux
_ Buildroot
_ Toolchain
_ POWER specific user space
_ other userspace components

71. Bild: https://github.com/open-power/docs/blob/master/hostboot/HostBoot_PG.md
Self
Boot
Engine
Hostboot
On Chip Controller & Skiboot/OPAL Linux/Petitboot
Linux/OS
BMC /
OpenBMC

72. 2.27425|ERRL|Dumping errors reported prior to registration
4.04949|Ignoring boot flags, incorrect version 0x0 ( Self Boot Engine)→
4.41091|ISTEP 6. 3 ( Hostboot bis ISTEP 21)→
5.04767|ISTEP 6. 4
5.18871|ISTEP 6. 5
10.40495|HWAS|PRESENT> DIMM[03]=A0A00000A0A00000
10.40497|HWAS|PRESENT> Membuf[04]=CC00000000000000
10.40498|HWAS|PRESENT> Proc[05]=8000000000000000
20.00800|ISTEP 6. 6
[…]
63.04461|htmgt|OCCs are now running in ACTIVE state
70.65864|ISTEP 21. 2
70.79438|ISTEP 21. 3
[37559728088,5] SkiBoot skiboot­5.1.5­2f88f6b starting...
[37559733441,5] initial console log level: memory 7, driver 5
[37559736244,6] CPU: P8 generation processor(max 8 threads/core)
[…]
[5453441264,5] IPMI: Resetting boot count on successful boot
[…]
[7551914988,5] INIT: Starting kernel at 0x20010000, fdt at 0x3054e5f0 (size 0x11309)
[7671453338,3] OPAL: Trying a CPU re­init with flags: 0x2
[ 0.000000] opal: OPAL V3 detected !
[ 0.000000] Using PowerNV machine description
[ 0.000000] Page sizes from device­tree:

73. [ 0.000000] Linux version 4.2.1­openpower2 (cyril@cyril­server) […]
[…]
[ 3.769043] SCSI subsystem initialized
[…]
[ 51.397327] NET: Registered protocol family 17
[ 51.397439] Key type dns_resolver registered
TYAN TN71­BP012 PNOR V1.01 ( Petitboot)→
──────────────────────────────────────────────────────────────────────────────
[Disk: sda2 / 3a71c024­b5ea­4031­98df­1c99beabbe1c]
Ubuntu, with Linux 4.4.0­36­generic (recovery mode)
Ubuntu, with Linux 4.4.0­36­generic
Ubuntu
[Disk: sdb2 / 538e8e0d­84cb­4522­86c3­f2d7431923f6]
Red Hat Enterprise Linux Server (0­rescue­ab029427b3fb4b87b79d81fba04c510c)
Red Hat Enterprise Linux Server (3.10.0­327.el7.ppc64) 7.2 (Maipo)
System information
System configuration
Rescan devices
Retrieve config from URL
*Exit to shell
──────────────────────────────────────────────────────────────────────────────
Enter=accept, e=edit, n=new, x=exit, h=help

74. Self Boot Engine (SBE)
Hostboot
On Chip Controller (OCC)
Skiboot
Linux
Petitboot

75. Self Boot Engine (SBE)
Hostboot
On Chip Controller (OCC)
Skiboot
Linux
Petitboot
→
op-build

76. $ git clone ­­recursive https://github.com/open­power/op­build.git
$ cd op­build
$ . op­build­env
$ op­build habanero_defconfig
$ op­build
barreleye
firestone
garrison
habanero
palmetto
romulus (POWER9)
witherspoon
zaius (POWER9)

77. How long does this take?

78. How long does this take?

79. How long does this take?

80. tk@tpt:~/git$ git clone ­­recursive https://github.com/open­power/op­build.git
tk@tpt:~/git$ cd op­build
tk@tpt:~/git/op­build$ . op­build­env
tk@tpt:~/git/op­build$ op­build habanero_defconfig
tk@tpt:~/git/op­build$ time op­build
make: Entering directory '/home/tk/git/op­build/buildroot'
[…]
33554432 bytes (34 MB, 32 MiB) copied, 0,0308641 s, 1,1 GB/s
[…]
real 67m8.767s
user 158m48.436s
sys 9m31.988s
tk@tpt:~/git/op­build$

81. $ sudo ./pflash ­e ­p /home/tk/habanero_update.pnor
About to erase 0x00000000..0x02000000 !
WARNING ! This will modify your HOST flash chip content !
Enter "yes" to confirm:yes
Erasing...
[==================================================] 100% ETA:0s
About to program "/home/tk/habanero_update.pnor" at 0x00000000..0x02000000 !
Programming & Verifying...
[==================================================] 100% ETA:0s
tk@ubuntu­16­04:~/skiboot/external/pflash$

82. ?

83. OpenBMC?
Image source: http://de.freeimages.com/photo/hourglass-1239654

85. BMC(IPMI Baseboard
Management Controller)

86. Chassis board
Motherboard
Processor
board
Memory
board
Baseboard
Management
Controller
(BMC)
System bus
NVS Storage
SDR
SEL
FRU
Chassis
mgmt.
(Satellite
Controller)
Sensors & Controls
Fan sensor
Temp. sensor
Power control
Reset control
…
FRU
Temp. s.
FRU
private mgmt. busses
IPMB
M/B
Serial
Controller
Serial
Port
Sharing
BMC
Serial
Controller
Serial/Modem
interface
LAN
interface
Serial
Connector
LAN
Connector
PCI mgmt. bus
Network
(LAN)
Controller
Remote Mmgt. Card
(KVM over IP, ...)
Auxillary
IPMB Connector
ICMB
ICMB
bridge
System
interface
Redundant Power
board
FRU Temp.
sensor
…
FRU
Access
as root
user
Access with
username & password
(UDP port 623)

88. 230.000 1u server
→ 10.223,5 m height
(Mount Everest 8.848 m)

89. IPMI auth
Type “None”
IPMI Cipher 0
IPMI RAKP+
Dump Hashes

90. _ update BMC firmware
_ use separate LAN – dedicated NIC or at least VLAN
_ deactivate unused services
_ use secure passwords AND usernames for IPMI
_ monitor with user rights ONLY
_ configure BMC firewall (if present)
_ EOL: flash BMC firmware or destroy mainboard
7 Tips for Tip 2 „Secure BMC“

91. 7 Tips for Tip 2 „Secure BMC“

92. 7 Tips for Tip 2 „Secure BMC“
...

93. 7 Tips for Tip 2 „Secure BMC“
https://www.thomas-krenn.com/de/tkmag/expertentipps/7-punkte-fuer-mehr-sicherheit-im-umgang-mit-ipmi/

94. OpenBMC – a Free Software BMC
_ REST-like API for Automatisierung
_ supports SSH to host console
_ runs „latest stable“ Linux Kernel:
_ Security fixes
_ current drivers
_ new Kernel functions
_ project started by Facebook

95. systemd
Flash LEDs Sensors
Host
Control
SSH
Python
Rocket
u-boot + Linux Kernel
Phosphor
Quelle: „OpenBMC: Boot your server with Python“, Joel Stanley, IBM OzLabs Adelaide, PyCon Australia 2016
https://2016.pycon-au.org/schedule/87/view_talk
https://www.youtube.com/watch?v=XrFaLnjOxQA

96. OpenBMC
_ Modern Kernel
_ Up-to-date Userspace
_ security patches
_ Make it „look good“
_ Make it „work well“ (REST, SSH)

97. Customize OpenBMC
_ Examples:
_ Authentication Model
_ REST API
_ App Integration
_ SDK available

98. Future plans
_ „Support the Server you have at Home“
_ Web Interface
_ Secure Boot, Trusted Boot, Sicherheit-Features
(lock-down Userspace)
_ Upstream of all components (Linux, u-boot, ...)
_ More Hardware with OpenBMC support
_ Redfish Support (check my last year's talk ;-)
_ more projects → see Raptor Engineering

100. Security
advantages
Image source: http://de.freeimages.com/photo/mobile-security-1242424

101. Firmware within a Server
BMC
Intel
ME
UEFI
Firmware
CPU
Microcode
HDD SSD
RAIDController
NIC
GPU

102. Firmware within a Server
OpenBMC
OpenPOWER
PNOR
HDD SSD
RAIDController
NIC
GPU

105. answers by three
developers
within 24h
(from three
different time
zones ;-)

106. Open Source code
on GitHub
You can compile
firmware yourself
→ Fast updates in case of
security issues
Open Source beginning
from the 1st
init up to DB
Direct contact to
developers

109. TombolaWin a Low EnergyServer / SSD / Laptop bagDrawing at 4:30 PM
(last coffee break)