Florian Feicht explains what are cgroups, how to configurate them, how to implement them and how to use them in resource management.

1. BASLE BERN BRUGG DÜSSELDORF FRANKFURT A.M. FREIBURG I.BR. GENEVA
HAMBURG COPENHAGEN LAUSANNE MUNICH STUTTGART VIENNA ZURICH
cgroups im Einsatz
Resource Management mal andersrum
Florian Feicht
Consultant

2. Florian Feicht
cgroups im Einsatz2 26.09.2016
Consultant and Trainer at Trivadis GmbH
Focus on:
– Oracle Database High Availability
– Database Administration
– Backup/Recovery
– Upgrades/Migrations
– Linux Administration

3. Agenda
cgroups im Einsatz3 26.09.2016
1. Resource Management
2. What are cgroups?
3. cgroups in init.d and systemd environments
4. Configuration
5. Implementation
6. Use Cases
7. Conclusion

4. cgroups im Einsatz4 26.09.2016
Resource Management

5. Resource Management
cgroups im Einsatz5 26.09.2016
Preserved resources
Operational stability
Reduce of costs
Different possibilities:
– Virtualization
– Application side solutions (Resource Manager)
– …
– cgroups

6. cgroups im Einsatz6 26.09.2016
What are cgroups?

7. What are cgroups?
cgroups im Einsatz7 26.09.2016
Linux Kernel feature (since 2.6.24) to limit resources
Process groups
Based on „everything is a file“ unix paradigma
sched_setaffinity Check cgroup
root Unlimited
cgroup
Check cpuset
configuration
mbind,
set_mempolicy
Check cgroup
root Unlimited
cgroups
Check cpuset
configuration

8. What are cgroups?
cgroups im Einsatz8 26.09.2016
Consists of resource controller
Base technology for Linux Container systems
Resource
Controller
cgroups

9. cgroups im Einsatz9 26.09.2016
cgroups and init.d

10. Structure
cgroups im Einsatz10 26.09.2016
Process orientated
No cgroup mount per default
libcgroup is available for administration
[root@doagdb-ol6 ~]# lssubsys -am
cpu
memory
devices
net_cls
cpuset,cpuacct,freezer,blkio,perf_event /mnt/cgroup

11. Creation
cgroups im Einsatz11 26.09.2016
libcgroup
– cgconfig
– cgcreate
– etc
Manual configuration
mount {
cpuset = /cgroup/cpuset;
cpu = /cgroup/cpu;
}
[root@doagdb-ol6 cpuset]# mkdir db1
[root@doagdb-ol6 cpuset]# ll db1/
-rw-r--r--. 1 root root 0 Apr 19 13:44 cpuset.cpu_exclusive
-rw-r--r--. 1 root root 0 Apr 19 13:44 cpuset.cpus
-rw-r--r--. 1 root root 0 Apr 19 13:44 cpuset.mems
-rw-r--r--. 1 root root 0 Apr 19 13:44 tasks

12. cgroups im Einsatz12 26.09.2016
cgroups and systemd

13. Structure
cgroups im Einsatz13 26.09.2016
Essential component
Moved from process to application layer
Administration with built-in systemd tools
libcgroup is deprecated

14. Structure
cgroups im Einsatz14 26.09.2016
system.slice
ora_db2.service ora_pmon_DB2
sshd.service
user.slice
user_<id>.slice session_1.scope
sshd: root
-bash
user_id<id>.slice session_3.scope
sshd: oracle
ora_pmon_DB1

15. systemd service configuration
cgroups im Einsatz15 26.09.2016
root@doagdb-cgrp-ol7 ~]# systemctl status oracle_db2.service
● oracle_db1.service - Oracle services started by Trivadis
BasEnv ...
CGroup: /system.slice/ora_db2.service
system.slice ora_db2.service ora_pmon_DB2
systemctl daemon-reload
systemctl enable oracle_db2.service
systemctl start oracle_db2.service

16. Creation
cgroups im Einsatz16 26.09.2016
Manual configuration
– start in user.slice
– Check startup
systemd
– No further configuration necessary
– Implementation is not complete
user.slice user_id<id>.slice session_3.scope ora_pmon_DB1
system.slice ora_db2.service ora_pmon_DB2

17. cgroups im Einsatz17 26.09.2016
cgroups configuration

18. libcgroup
cgroups im Einsatz18 26.09.2016
Check existing subsystems
– lssubsys -am
Configuration:
– cgcreate
– cgset
– cgsnapshot
Resource Controller cgroup
cgset -r cpuset.cpus=1 db1
zugeordnete CPU

19. Manual configuration
cgroups im Einsatz19 26.09.2016
Limitation of resources
Creation of subgroups under depending resource controller
– „Files“ will be created automatically
mkdir /sys/fs/cgroup/cpuset/db1
echo 0 > /sys/fs/cgroup/cpuset/db1/cpuset.cpus
echo 0 > /sys/fs/cgroup/cpuset/db1/cpuset.mems

20. systemd
cgroups im Einsatz20 26.09.2016
Resource Controller:
– CPU
– Memory
– I/O
– Devices
Configuration using systemctl
systemctl set-property unit-name property=value
system.slice ora_db2.service ora_pmon_DB2

21. Monitoring I
cgroups im Einsatz21 26.09.2016
/proc/
ps
ps -ax --format args,cgroup
ora_pmon_DB2 1:name=systemd:/system.slice/oracle_db2.service
ora_psp0_DB2 1:name=systemd:/system.slice/oracle_db2.service
cat /proc/2911/cgroup
10:perf_event,blkio,freezer,cpuacct,cpuset:/tvd12cdb

22. Monitoring II
cgroups im Einsatz22 26.09.2016
Only in systemd environments:
– systemd-cgtop
– systemd-cgls
Path Tasks %CPU Memory Input/s Output/s
/system.slice/oracle_db2.service 42 3.2 1.6G 190.8K 31.8K
├─1 /usr/lib/systemd/systemd --switched-root --system --deserialize 21
├─user.slice
│ └─user-54321.slice
│ └─session-1.scope
│ ├─2846 -bash
└─system.slice
├─oracle_db2.service
│ ├─2575 ora_pmon_DB2

23. cgroups im Einsatz23 26.09.2016
How to configure cgroups in Oracle
database environment

24. Database start in a cgroup
cgroups im Einsatz24 26.09.2016
cgexec
cgexec -g cpuset,cpuacct,blkio:tvd12cdb sqlplus / as sysdba
Redirect Shell
systemd Service
echo $$ > /mnt/cgroup/tvd12cdb/tasks
init.ora parameter
– Only with manual configuration

25. Configuration proposal by Oracle I
cgroups im Einsatz25 26.09.2016
MOS: Doc ID 1585184.1 and Doc ID 1928328.1
Provided configuration script: setup_processor_group.sh
– manual cgroup mount will be done
– Resource controller: cpuset
– Persistence is ensured

26. Configuration proposal by Oracle II
cgroups im Einsatz26 26.09.2016
Change of init.ora Parameter
– Restart necessary
Only on CDB level
Entry in alert.log
Instance started in processor group tvd12cdb (NUMA Nodes: 0
CPUs: 0)
ALTER SYSTEM SET processor_group_name='tvd12cdb' SCOPE=SPFILE;

27. Configuration proposal by Oracle III
cgroups im Einsatz27 26.09.2016
Advantages
– Easy naming
– Same functionality in init.d and
systemd environments
– cgroup usage entry in alert.log
– cpuset is enabled by default
– Permission for oracle user is set
Disadvantages
– cgroup necessary for DB startup
– Further resource controller have to be
mounted manually
– Check of the script is very limited

28. Further configuration possibilities
cgroups im Einsatz28 26.09.2016
systemd configuration
– No need to set init parameter
– No cpuset controller possible
Manual configuration
– Init parameter necessary
– Flexible resource controller
– Persistence has to be configured manually

29. Necessary DB parameter
cgroups im Einsatz29 26.09.2016
No special settings necessary
Ensure correct cgroup for server processes
One listener per database is recommended
– Check startup configuration
– DEDICATED_THROUGH_BROKER_listener_name
ALTER SYSTEM SET dedicated_server=TRUE;

30. cgroups im Einsatz30 26.09.2016
Implementation

31. Manual configuration: test database
cgroups im Einsatz31 26.09.2016
Requirements
– Usage of just one CPU
– Maximum of 10MB/s IO read und write
echo 1 > cpuset.cpus
echo "252:0 10485760" > blkio.throttle.read_bps_device
echo "252:0 10485760" > blkio.throttle.write_bps_device

32. Manual configuration: test database
cgroups im Einsatz32 26.09.2016
Comparison of AWR reports
– Unlimited:
– Limited:

33. systemd
cgroups im Einsatz33 26.09.2016
BlkIO controller has to be activated
Resource controller cpuset is not available
systemctl set-property oracle_db2.service CPUAccounting=TRUE
systemctl set-property oracle_db2.service CPUShares=500
systemctl set-property oracle_db2.service BlockIOAccounting=TRUE
systemctl set-property oracle_db2.service
BlockIOWriteBandwidth="/dev/sda 10M"

34. cgroups im Einsatz34 26.09.2016
Use Cases

35. Use Cases
cgroups im Einsatz35 26.09.2016
Database cloud environments
– Service classes by defined cgroups
– Increased security level by device mapping
Test and production database on the same hardware
Resource management in Standard Edition
Assigment of CPU and memory segments (NUMA Systeme)
I/O limitation

36. cgroups im Einsatz36 26.09.2016
Conclusion

37. Conclusion
cgroups im Einsatz37 26.09.2016
cgroups are:
– Easy to configure
– flexible
– „Licensefree“
Be careful:
– Concept necessary
– Ensure cgroup availability
– Transparent for the database
– Oracle license
Simple resource management for
Oracle databases on Linux

38. Any Question…?
Florian Feicht
Consultant
florian.feicht@trivadis.com
26.09.2016 SSD in database environments38
@FlorianFeicht

39. Session Feedback – now
TechEvent September 201639 09.09.2016
Please use the Trivadis Events Mobile App to give session feedback
Use "My schedule" if you registered for this session
Otherwise use "Agenda" and the search function
If the mobile App does not work (or if you have a Windows Phone) use your Mobile
Browser
– URL: http://trivadis.quickmobileplatform.eu/
– Username: <your_loginname> (like svv)
– Password: sent by mail...