Presentation covers various cyber security aspects that are stands behind the AAA-Level game projects. And what is most important it covers a practically proven way to provision own data (game services) in 22 geographical locations in 22 minutes, using opensource solution - OpenNebula and it's DDC features. During this 22 minutes you receive fully distributed mesh infrastructure, located in 22 different geo locations (datacenters) provisioned using only bare metal hardware servers, with preconfigured GNU/Linux OS and preconfigured VM on top of each server. Each server has own control server in own region with backconect to 'mother' server in central location with High Availability configured, own network segments in each datacenter, elastic IP's, Backend Transfer Facilities, Local BGP.

1. Disaggregated Data Centers on a shoulders of
OpenNebula
Dmytro Korzhevin

2. About the speaker

3. About the speaker
Crytek Chief Information Security Officer,
Head of Crytek CERT (crytek.com)

4. About the speaker
eLearnSecurity: eCPPT, eMAPT, eNDP, eWDP, eJPT
EC-Council: LPT (Master), CSCU, CND, CEH, CEH (Master), CEH (Practical), ECIH, ECSA, EC-Council E|CND Item Writer,
EC-Council E|CIH Review Board member, ECSA Item Writers Group
Linux Foundation: LFCSA, LFCE
Hewlett-Packard: HP ATA Architect, HP ATA Servers and Storage, HP ATA Designing and Deploying Cloud Solutions, HP ATA
Devices, HP ATA Networks
Symantec Hacker Academy: Hacking, Client Attacks, Digital Forensics, Pen Test, Debugging, Web App Attacks, Network
Attacks, Pen Test Management, Server Attacks.
PentesterLab: Intercept Bage, White Bage, Serialize Badge, Capture-The-Flag Badge
NATO Cooperative Cyber Defence Centre of Excelence (Tallin Estonia): Rapid Reaction Expert Training, Satellite Operations,
European Security and Defence Policy (ESDP), Strategic Communications, Critical Infrastructure Awareness, Information
Security, Digital Communications, Cyber Defence.
USDHS: Offensive and Defensive Network Operations, Linux Operating System Security, Threat Hunting Teams, Cloud
Computing Security, CISM 2013, CDM, Cyber Risk Management, Cyber Security Investigations, ISACA Certified Information
Systems Auditor (CISA) Prep, (ISC)2 (TM) CISSP (R) Prep, Penetration Testing, Securing Infrastructure Devices, Securing the
Network Perimeter.
Canonical: Ubuntu System Builder (2008)
ISACA: CSX
ISO/IEC: 27001:2013, 19011:2011

5. COMPANY OVERVIEW

6. FACTS
Crytek is a leading, internationally operating developer and
publisher of video games
Known for world class IPs and products such as the original Far Cry, the Crysis
franchise, Ryse: Son of Rome and game–service Warface and HUNT SHOWDOWN
All Crytek games are built with the proprietary game development solution
CRYENGINE®
CRYENGINE is perfect for rich VR worlds and the new hardware is now capable of
bringing our ideas to life.

7. Crytek Games

8. CRYENGINE®
is Crytek’s key differentiator for success
World leading game development software for sophisticated computer and video games
Highest graphics quality and unique Realtime-3D-Technology
Innovation leadership as a result of 15 years of development know-how
Licensed by numerous third-party game developers and publishers
Sole integrated all-in-one solution for games on platforms of the current and future generation:
CRYENGINE

9. CRYENGINE

10. ● https://www.cryengine.com/
● https://youtu.be/GN5c3B6RqaI
● CRYENGINE 5.6 Tech Trailer
● https://www.youtube.com/watch?v=ObAqK8a-W9w
Showcase

11. https://github.com/crytek
https://github.com/CRYTEK/CRYENGINE
CRYENGINE

12. Game Approaches

13. Game Approaches and tools
● Visual Studio
● .NET
● mono
● dotnetcore
● perforce
IMPORTANT: CPU cores usage and HT

14. Crytek Approach

15. Crytek Approach
● dotnetcore - official
● Minimize attack surface from beginning
● IntelliTrace, software transactional memory (STM) and Pex
● Isolation and White Box Unit Testing
● Workflow - CERT

16. Showcase

17. Behind the Game

18. Behind the game - OS
● Linux OS Standardization (according to req)
● Additional security configuration for repository signatures
● LVM configuration - different schemes per server purpose
● FDE / Partition encryption
● Ulimits settings
● Kernel / Network stack tuning
● CPU and IO schedulers patches and tuning
Nice to read about: oomd, earlyoom, nohang

19. Behind the game - OS
● Spectre / Meltdown mitigations (retpoline)
● Latest CPU microcode
● Kernel mitigations
● GCC (fstack-clash-protection | mindirect-branch)
● Userspace (qemu / libvirt)

20. Behind the game - OS
Linux Security Modules (LSM)
AppArmor |
SELinux |
TOMOYO
LoadPin
Smack
Yama
SafeSetID

21. Monitoring
● Zabbix + Zabbix proxy + zabbix.dll (server integration)
● Zabbix autodiscovery for every HW server
● Vulns - CVE across installed packets - integration with Zabbix
● Kibana (ELK)
● Graphana
● Monit
● Graphite
● Graylog

22. Monitoring 2
● rsyslog (official repos, not distro)
● Logwatch
● Gitlab for all configuration files (both game and /etc)
● cachet (for status page)

23. HW / Net capacity tracking
● OpenDCIM - racks map and interconnection
● IPAM - IP Address Management
● Eramba - GRC (+compliance)

24. Compliance
● DISA STIG’s
● NIST SP (800x)
● SCAP / OpenSCAP

25. Access
● freeIPA
● Only SSH keys (elliptic curve)

26. Security (SOC and CSIRT / CERT)
● Wazuh
● Samhain HIDS
● Prelude
● GRR (Remote Live Forensics For Incident Response)
● Red ELK
● TheHive
● Chef InSpec

30. Network and network services
● DNScrypt
● NtopNG / Suricata
● iperf points
● PerfSonar “measurement island”
● NDT and speedtest
● ipsec (StrongSwan ESP) + hardware acceleration
● P2P (torrent)

31. Datacenter APIs

32. Datacenter API
How datacenter API should be provided (via official libraries):
● CLI
● Python
● Ruby
● Node.js
● PHP
● Go, etc...

33. Own integration
Something like:
● Flask, Flask-RESTPlus and Swagger UI

34. Datacenter API
Some unusual ways to use API:
● curl (testing only)
● Burpsuite / ZAP
● Metasploit module to interract with API

35. Datacenter Evaluation

36. Datacenter Evaluation
● PRICING QUESTIONS
● LOCATION QUESTIONS
● SPACE QUESTIONS
● NETWORK QUESTIONS
● POWER QUESTIONS
● COOLING QUESTIONS
● SECURITY QUESTIONS
● SUPPORT QUESTIONS
● CUSTOMER DEPLOYMENT QUESTIONS
● SERVICE LEVEL AGREEMENT QUESTIONS

37. Datacenter Evaluation
● ISO9001:2008, for quality management systems;
● ISO27001:2013, for information security;
● ISO14001:2004, for sustainability;
● PCI DSS 3.0, for information security for online payment;
● ISAE 3402 (comparable to SSAE 16) Type II, for service organization controls (SOC)
reports;
● IX Certified Data Center; for carrier-neutral colocation and interconnection.
● SAS 70 (Type 1 / Type 2)
● SSAE 16 (Type 1 / Type 2)
● SOC 1 / SOC 2 (Type 1 / Type 2) / SOC 3

38. Locations

39. Locations
Right near IX-points (AMS IX + Evoswitch DC as example)
Reliable datacenters
Close to users

40. Locations - Packet

41. Opennebula DDC
(Disaggregated Data Centers)

42. Opennebula DDC
A solution for:
1. Scalability (elasticity) problems
2. Human / configuration errors
3. Time save (big amount of data + configuration at once)
4. P2P
5. Best alternative for cold racks

43. Opennebula DDC
Scalability types:
1. Predictable (Events)
2. Mixed or Emergency
3. Unpredictable

44. Opennebula DDC
About predictable scalability
Metrics, Agreements, Formulas, ELK, Graphana, ingame analytics

45. Opennebula DDC
Mixed / Emergency scalability
Outages, including unplanned + urgent updates

46. Opennebula DDC
Unpredictable scalability
Fast grows and significant exceedances of expected statistical data

48. Provision
● Oneprovision
● Provision templates (YAML)
● IPAM Driver

49. Behind the game

50. Behind the game

51. Behind the game

52. Behind the game

53. Thank You!