GoDaddy adopted OpenStack in late 2013 and has since scaled it significantly. They now run over 16,000 VMs and 2000 hypervisors on OpenStack. GoDaddy has focused on automating packaging, deployment, and operations through tools like Kolla, Ansible, and continuous integration/deployment. While the scale provides challenges, GoDaddy remains engaged in the community through contributions and the #openstack-operators channel. Areas for future work include further automation, self-healing, and improved APIs.

1. OpenStack at Scale
February 9th, 2017
GoDaddy's OpenStack Journey
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.

2. Where we started
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
• OpenStack (late 2013 – now())
• Strategic bet on open source!
• A nice, active, larger and bountiful community
• Deploying started soon thereafter
• Missing some key functionality (but getting better)

3. Our clouds
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
*2x the above (for public cloud and private cloud)

4. Our size
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
VMs HVs
16,000+ 2000+
VMs HVs
1,900+ 260+
~2017
~2013
(vmware)

5. Our layout
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
Highlights
• One of the first adopters of nova cells v1

6. Our journey (so far)
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
• Packaging
• Deployment
• Scale
• Operational best practices
• Keeping involved and engaged
• (and more)

7. Packaging
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
Old: partially hand generated rpms
• Error prone and fragile
New: machine generated docker images
• Github integrated
• Slack integrated
• Auto-built via jenkins jobs
• Auto-pushed to artifactory
• Auto-pulled from thereafter for deployment
+ + + +
Kolla
+

8. Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
Nifffty!

9. Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.

10. Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.

11. Deployment
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
Old: hand-driven puppet with a sprinkling of ansible
• Peer review and pray
New: moving to continuous automation (and eventual CD)
• Takes advantage of docker images to mitigate fragility
• Interactions audited and tracked by command and control
• Eliminates the (faulty) person-behind-the-keyboard
• Heavily ansible based (also slack integrated)
• Interacts with tool agnostic cloud topology (yaml) format
+ + + +

12. Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.

13. Cloud at scale - lessons learned
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
Beginners:
• Devstack
Intermediate (but still easy):
• Operating a small cloud (less than 100 nodes)
• Deploying a small cloud (even by hand/high touch)
Advanced (not easy):
• Operating multiple clouds (multiple thousands of nodes/VMs)
• Hand-crafted deployment no longer ‘just works’
• Transitioning from high touch to low touch is required
• Being involved (upstream) while doing this
Transition from high touch to low touch – the earlier the better

14. Operational best practices
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
• Do not create distribution packages
• Containerization even used minimally is worth it
• Attend the #openstack-operators meetings 
• Incorporate reoccurring questions into the product
• Education - ephemeral nature of the cloud
• Balance downstream and upstream carefully

15. Community involvement
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
Not so easy staying involved while building operations/automation/deployments.
But we manage!
0
20
40
60
80
100
120
140
160
GoDaddy Commits
0
50
100
150
200
250
300
350
400
GoDaddy Reviews

16. Things we want to work on
Copyright© 2016 GoDaddy Inc. · 14455 N. Hayden Road Scottsdale, Arizona 85260 (480) 505-8800 · All Rights Reserved.
• Less complicated upgrades & deployments
• Self-healing
• Scale & performance
• Less complicated networking/compute/… APIs
• Better resource cleanup
• Discoverable API endpoints
• (and more, partner with us!)

17. We are hiring!!!
We are hiring!!!
x.co/openjobs
For OpenStack:
• Neutron + SDN experts
• Nova experts
• Engineers passionate about automating operations
Mailto: recruiting@godaddy.com
Find: Stephen Hoffman (in the back)

18. 2017
© GoDaddy Inc.
Thanks for coming!

19. We are hiring!!!