OpenStack Ironic: From Zero to “Pro”.pdf

1. OPENSTACK IRONIC: FROM ZERO TO “PRO”
Aris Cahyadi Risdianto
Singapore Institute of Technology (SIT)
Huang Kang
Individual Technical Contributor

2. Former Engineer & ICT professional for
8 years and involved in the design/
implementation/maintenance of ICT
infrastructure for enterprise and service
provider. Working on research and
education of ICT infrastructure for over
12 years by building labs and testbeds
leveraging open source software and
project collaboration.
ONF* Ambassador & Steering Team, Speaker
at ONF Summit, KubeCon, FOSSASIA Summit,
IEEE/ACM Conferences, IETF/APNIC/APAN
Fellow, APNIC Community Trainer, and
OpenInfra/CNCF ID Day Organizer.
*) Now is part of LF

3. Disclaimer!
● This talk is our personal experiences in our previous role
● Nothing to do with current company or role
● We are not talking on behalf of our previous company
● It may not be a good example/experience!
● We are open to any discussion!
●

4. Day Zero to Production Day

5. Overview
● We ran our own private cloud infrastructure
● It used only for cybersecurity research and education (e.g.,
hacking, pentesting, cyber-range, DDoS, …)
● Our hardware spreaded over seven (7) of racks
● Hosted in a limited and protected hosting data center
● Consists of around 200 - 300 servers (i.e., low-end, mid-end, and
high-end models)

6. Challenges
● Proprietary or highly-customized “open-source” baremetal
provisioning system
● A problematic operational issues, inconsistent solutions
● Must be upgraded to OpenStack, but needs to keep same UI
● Old baremetal servers ( > 7 years old)
● No professional services, must be in-house develop

7. Helps are on the way!
● OpenStack docs is good, but it is very long/complex for Ironic
● Online discussion forum is very old and tight to the HW
● Example deployment configuration is not complete (only snapshot)
● Professional consultant offers help but we can’t utilize them
● Online assistance but unwilling/unable to access/check

8. Chameleon is Coming!
Chameleon Testbed Team
● Kate Keahey
● Jason Anderson
● Mike Sherman
● Others
https://www.chameleoncloud.org/

9. Progressing Better Than Nothing
● To install Chameleon-in-a-box to provision a single BM → 6 months
● To install a base openstack cluster with 3 controller, 2 compute, and
1 network for VM provisioning → 6 months
● To get first multiple baremetal provisioned → 6 months
● To integrate the backend and frontend → 6 months
● To test and provide user trial → 3 months

10. Solving the Technical Puzzles

11. Installation: DevStack to Chameleon
● DevStack is fast & easy → only work with emulated baremetal (VM with fake IPMI)
● Manual or script-based installation is possible → highly customized and high
error-prone possibility
● Kolla-ansible is fast and scalable → issue with compatibility and stability for recurrence
installation or update
● Juju + MAAS is easy and scalable → too many components and works only Ubuntu
● Chameleon-in-a-box works for baremetal physical hardware → not scalable (only in a
single machine)
● Chameleon-in-a-box for cluster is used → too many additional (unused) packages, but
got “unofficial” support :)
SOLVING THE TECHNICAL PUZZLES

12. Integration: Frontend + BackEnd + API + Infrastructure
Infrastructure
OpenStack API Server
Backend Server
Front End (Web UI) Front End (Apps)
Nova Neutron Cinder Ironic Glance
OpenStack API
Backend API (Proprietary)
Existing Platform
New CLI (Customized)*
SOLVING THE TECHNICAL PUZZLES

13. Configuration: Kolla vs Chameleon YAML config
SOLVING THE TECHNICAL PUZZLES
Hundreds line codes vs few line codes!

14. User Experience: “Experiment” to “Stack”
Source: https://ncl.sg/tutorials
Stack Project/tenant State
Stack Detail
Stack Action
(Start/Suspend/Delete)
SOLVING THE TECHNICAL PUZZLES

15. Automation: From NS2 to Heat Template
SOLVING THE TECHNICAL PUZZLES
● Different format but the same concept (node, image, network, …)
● No conversion tool is available, but it is not huge task

16. Lessons Learned

17. Networking is “Not” Working
● It combines of virtual and physical
networking configuration: user space
switch and physical port
● Different solutions: creating virtual ports
for the bridge, use virtual/physical router,
or totally “flat” network
● Be mindful other features such as DHCP
relay, TFTP over HTTP, …
● Issue with multiple NICs baremetal →
select and activate one MAC address
DHCP/TFTP
Container
OVS SW
Container
Control
ETH
BM
Bridge
BM
ETH
Baremetal Physical
Network
OpenStack Services
Network
LESSON LEARNED

18. P@$$w0rd is not Acceptable for Baremetal
● Be mindful when enrolling the baremetal from OS CLI/API:
○ The IPMI password is typed in clear text
○ A special characters for IPMI password must be type with single quote
● Be mindful when configuring IPMI password in the BIOS
○ The IPMI password must be typed in (unable to copy and paste)
○ Some of the old IPMI didn’t accept a specific characters
○ Ensure the validity period of the IPMI password (new IPMI)
LESSON LEARNED

19. To Build an Image != Use Built-in Image
● Two images are used in Ironic: deploy image and user image
● Deploy image is straightforward and no need to be customized
● Different built tools are available to build user image → many different
options and configurations
● Some tools guide are outdated and not valid → official diskimage-builder
is better but need to find correct options
# ubuntu baremetal cloud-init dhcp-all-interfaces grub2 devuser
LESSON LEARNED

20. BAREMETAL_CUSTOM is not Customizable
● Writing of the name is important → must be exact and in capital
● Specification options or metadatas are not important to match the
hardware resources
LESSON LEARNED

21. Others Things
● User image deployment: iSCSI vs Direct Deploy
● Baremetal cleaning: skip if not necessary or exclude large disk
● OpenStack cluster: rely on database cluster
LESSON LEARNED

22. Acknowledgement
Aris Cahyadi Risdianto
Team Leader
https://www.linkedin.com/in/ariscahyadirisdianto/
Huang Kang
OpenStack + System Engineer
https://www.linkedin.com/in/kang-huang-19b929289/
Felix Wong
Infrastructure/HW Engineer
https://www.linkedin.com/in/felixwongcw/
Thazin Hlaing
Backend Developer/API Engineer
https://www.linkedin.com/in/thazin-hlaing-1a372a7a/
Deepshikha
Front-end Developer
https://www.linkedin.com/in/deepshikha-121009/

23. Thank You!
ariscahyadi.risdianto@singaporetech.edu.sg