This document discusses Checkpoint/Restore In Userspace (CRIU), a tool for live migration of processes and containers. CRIU works by dumping the memory, file descriptors, and other process state of a running process, then restoring it elsewhere. This allows live migration of processes between systems for purposes like load balancing, maintenance, and high performance computing. The document provides details on how CRIU works, its uses cases, limitations, and how to install and use it on Red Hat Enterprise Linux 7.

1. RED HAT ENTERPRISE LINUX 71
Checkpoint Restore In Userspace
SONG, CHANGAN(Leo)
APAC Technical Account Manager,
Customer Experience & Engagement,
Strategic Customer Engagement, Red Hat

2. RED HAT ENTERPRISE LINUX 72
• 프로세스의 현재 상태 저장
• 이전 상태 복원 기능 (checkpoint 전으로 )
• Checkpoint 된 프로세스의 모든 정보는 하나이상의 이미지 파일로 저장
됨
( 저장정보 : memory pages, file descriptors, inter-process
communication, and so on)
• 같은 시스템 또는 다른 시스템에 프로세스 복원
• 컨테이너 라이브 마이그레이션 같은 용도로 사용됨
• RHEL7.3 에 패키지가 포함 (criu 2.3)
• Tech-preview 기능으로 등록
CRIU
Checkpoint / Restore In User space
https://access.redhat.com/articles/2455211

3. RED HAT ENTERPRISE LINUX 73
CRIU
how does it works?
Kernel objects Process tree
criu
Image files
Namespaces
Files
Sockets
Pipes
001101
101010
110001
011010
000011
010101
001101
101010
110001
011010
000011
010101
001101
101010
110001
011010
000011
010101
001101
101010
110001
011010
000011
010101
001101
101010
110001
011010
000011
010101
001101
101010
110001
011010
000011
010101

4. RED HAT ENTERPRISE LINUX 74
CRIU
how does it works?
Kernel interfaces
Dump Restore
syscalls
netlink
/proc/
ptrace

5. RED HAT ENTERPRISE LINUX 75
CRIU
Dump
 Parasite code
 Receive file descriptors
 Dump memory content
 Prctl(), sigaction, pending signals, timers, etc.
 Ptrace
 freeze processes
 Inject a parasite code
 Netlink
 Get information about sockets, netns
 Procfs
/proc/PID/maps, /proc/PID/map_files/, /proc/PID/status,
/proc/PID/mountinfo

6. RED HAT ENTERPRISE LINUX 76
CRIU
Restore
 Collect shared objects
 Restore name-spaces
 Create a process tree
 Restore SID, PGID
 Restore objects, which should be inherited
 Files, sockets, pipes, ...
 Restore per-task properties.
 Restore memory
 Call sigreturn
 Awesome
Namespaces
Processes

7. RED HAT ENTERPRISE LINUX 77
CRIU
Interest moment
 How to restore shared objects?
 Send file descriptors via unix sockets
 Map files from /proc/self/map_files/ for restoring anon shared mappings
 How to restore memory mappings on the correct places?
 Map a new code block and a stack
 Unmap crtools' mappings
 Remap task's mappings on the correct places
 How to resume a process?
 Create a signal frame
 Call sigreturn()

8. RED HAT ENTERPRISE LINUX 78
• HPC 환경을 위해 개발
• 하나의 어플리케이션이 수백 , 수천 코어에 분산되어 실행되는 환경에
적합
• 특히 어플리케이션이 실패할 경우 , 전체 CPU 사용된 것이 쓸모없게
되고 데이터도 손실되는 약점을 CRIU 로 해소
• 어플리케이션과의 호환성 검토 필요
• 초기에는 관심받지 못하다가 container migration 으로 각광
CRIU
Birth of CR

9. RED HAT ENTERPRISE LINUX 79
• Inter-process-communication(IPC) 을 이용하여 checkpoint /restore
동작이 가능 .
• 항상 부모 프로세서와 모든 자식 프로세서 checkpoint/restores 에 대
해서 가능 .
• PID 항상 같아야 하며 , 시스템에서 이미 사용하는 PID 가 있는 경
우 , CRIU 를 이용한 프로세스 복구 단계에서 실패 .
CRIU
Limitations
https://criu.org/What_cannot_be_checkpointed

10. RED HAT ENTERPRISE LINUX 710
Live migration
CRIU
Host A Host B
Shared FS
Pre-migrate memory
with memory tracker
http://criu.org/P.Haul

11. RED HAT ENTERPRISE LINUX 711
Load balancing on cluster
CRIU
Host A
Host C
Host B

12. RED HAT ENTERPRISE LINUX 712
Power saving on cluster
CRIU
Host A
Host C
Host B

13. RED HAT ENTERPRISE LINUX 713
Node maintenance
CRIU
Host A Host B

14. RED HAT ENTERPRISE LINUX 714
Kernel upgrade w/p reboot
CRIU
Host
Kernel A
Kexec
Kernel B

15. RED HAT ENTERPRISE LINUX 715
Slow services startup
CRIU
time# service foo start
Service readiness
Spawn process
Load config
Top-up caches
Initialize resource pools
Ready
T
100%

16. RED HAT ENTERPRISE LINUX 716
Slow services startup
CRIU
time
Tt < T
Ready
Spawn process
100%
Service readiness
# service foo restore

17. RED HAT ENTERPRISE LINUX 717
Periodic snapshots
CRIU
time
Memory tracker helps
to keep images smaller

18. RED HAT ENTERPRISE LINUX 718
HPC
CRIU
time
Power
failure
0% 20% 40% 60% 60%

19. RED HAT ENTERPRISE LINUX 719
Advanced debugging
CRIU
Production Host
Application
in trouble
Developer Host
Debugger

20. RED HAT ENTERPRISE LINUX 720
Advanced testing
CRIU
...
New test
or
new hardware
?

21. RED HAT ENTERPRISE LINUX 721
Installation ciru package on RHEL7
CRIU
# yum install criu -y
...
Dependencies Resolved
=============================================================================================
Package Arch Version Repository Size
=============================================================================================
Installing:
criu x86_64 2.3-2.el7 rhel-7-server-rpms 349 k
Installing for dependencies:
protobuf-c x86_64 1.0.2-3.el7 rhel-7-server-rpms 28 k
…
# ldd `which criu`
linux-vdso.so.1 => (0x00007ffed554d000)
librt.so.1 => /lib64/librt.so.1 (0x00007f5fd0faf000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f5fd0d93000)
libprotobuf-c.so.1 => /lib64/libprotobuf-c.so.1 (0x00007f5fd0b89000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f5fd0985000)
libnl-3.so.200 => /lib64/libnl-3.so.200 (0x00007f5fd0764000)
libc.so.6 => /lib64/libc.so.6 (0x00007f5fd03a2000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5fd11bd000)
libm.so.6 => /lib64/libm.so.6 (0x00007f5fd00a0000)

22. RED HAT ENTERPRISE LINUX 722
1) criu on command
CRIU
How to Use
2) criu in runc
- restore checkpoint container
http://rhelblog.redhat.com/2016/12/08/container-live-migration-using-runc-and-criu/
# criu --help
Usage:
criu dump|pre-dump -t PID [<options>]
criu restore [<options>]
criu check [--feature FEAT]
criu exec -p PID <syscall-string>
criu page-server
criu service [<options>]
criu dedup
...
# runc checkpoint <container name>
For example,
# runc checkpoint rhel7-httpd
# runc restore -d <container name>
For example,
# runc restore -d rhel7-httpd
- store checkpoint container

23. RED HAT ENTERPRISE LINUX 723
Demo in runc
CRIU

24. RED HAT ENTERPRISE LINUX 724
Runc
CRIU
criu can now be used for following applications running in a Red Hat Enterprise
Linux 7 runc container:

vsftpd

apache httpd

sendmail

postgresql

mongodb

mariadb

mysql

tomcat

dnsmasq

25. RED HAT ENTERPRISE LINUX 725

26. RED HAT ENTERPRISE LINUX 726
THANK YOU