Bloated Chefs: A Tale of Gluttony, & the Path to Enlightenment - ChefConf 2015

•

0 likes•363 views

Evan Gilman presented on optimizing Chef resources at PagerDuty that had led to bloated Chef runs. As PagerDuty grew, Chef runs experienced CPU spikes, long run times, and out of memory errors. To address this, they measured Chef run times and resource usage, identified inefficiencies like full node searches and redundant API calls, and implemented optimizations like partial searches, result memoization, and centralized API access. These changes reduced Chef run memory usage by 88% and time by 84%, allowing faster convergence and less resource intensive runs.

Technology

4/3/15
@evan2645
EVAN GILMAN
Bloated Chefs
A Tale of Gluttony, and the Path to Enlightenment

4/3/15
Agenda
BLOATED CHEFS
1. Chef resources in use at PD
2. Problems encountered as we grew
3. Measuring chef-client run
4. How we fixed it
5. How fast is it now?

4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs

4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run

4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes

4/3/15
As we grew…
BLOATED CHEFS
• CPU spikes during chef-client runs
• Awkward pauses at the beginning of the run
• chef-client run took several minutes
• chef-client OOM

4/3/15
Measuring Run Time
BLOATED CHEFS
https://github.com/joemiller/chef-handler-profiler

4/3/15
Measuring Resources
BLOATED CHEFS
• Total number of resources per run, by type
• Number of updated resources per run, by type

4/3/15
Measuring Memory
BLOATED CHEFS
• Gather proc stats with sys-proctable
• Gather GC stats
• Can be emitted as statsd

4/3/15BLOATED CHEFS
WHAT WE FOUND
AND
WHAT WE DID

4/3/15
Step-through Searches
BLOATED CHEFS

4/3/15
Step-through Searches
BLOATED CHEFS
From this

4/3/15
Step-through Searches
BLOATED CHEFS
From this
To this

4/3/15
Step-through Searches
BLOATED CHEFS
417MB -> 190MB

4/3/15
Step-through Searches
BLOATED CHEFS
417MB -> 190MB
~54%

4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results

4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled

4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node

4/3/15
Partial Searches
BLOATED CHEFS
• Provide hash map of desired results
• Minimizes volume of node data returned/handled
• hash2node
• Two searches touched

4/3/15
Result Memoization
BLOATED CHEFS
• Common search data

4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
• API-backed LWRP’s

4/3/15
Result Memoization
BLOATED CHEFS
• Common search data
• API-backed LWRP’s
• Can be generalized

4/3/15
API Tarpitting
BLOATED CHEFS
Centralize calls

4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards

4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage

4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage
• Multiple package resources (Chef 12)

4/3/15
Other Nasties
BLOATED CHEFS
• Too many conditional guards
• tmpfs storage
• Multiple package resources (Chef 12)
Six seconds for twelve packages

4/3/15
Memory Saved
BLOATED CHEFS
Before:
After:

4/3/15
Memory Saved
BLOATED CHEFS
Before: ~500MB
After:

4/3/15
Memory Saved
BLOATED CHEFS
Before: ~500MB
After: ~60MB

4/3/15
Memory Saved
BLOATED CHEFS
Before: ~500MB
After: ~60MB
88% less memory!

4/3/15
Seconds Saved
BLOATED CHEFS
Before:
After:

4/3/15
Seconds Saved
BLOATED CHEFS
Before: ~180s/run
After:

4/3/15
Seconds Saved
BLOATED CHEFS
Before: ~180s/run
After: ~30s/run

4/3/15
Seconds Saved
BLOATED CHEFS
Before: ~180s/run
After: ~30s/run
~84% faster!

Viewers also liked

CV-Zhengxia QIN_EZhengxia QIN

SlidesRobby Alvaro

USCBC 2016 Board Priorities Statement A4 – ChineseEllen Huber

2015 SCUBA ResumeTony Delia

USCBC 2016 Board Priorities Statement – ChineseEllen Huber

Tic en educacionAmparo Cuadros Bustamante

Ii peter 1 5 7 commentaryGLENN PEASE

Spotlight - Global 2015Stacey Staum

Spotlight - UH Ultimate 2015Stacey Staum

Sumansuman pal

Viewers also liked (10)

CV-Zhengxia QIN_E

Slides

USCBC 2016 Board Priorities Statement A4 – Chinese

2015 SCUBA Resume

USCBC 2016 Board Priorities Statement – Chinese

Tic en educacion

Ii peter 1 5 7 commentary

Spotlight - Global 2015

Spotlight - UH Ultimate 2015

Suman

Recently uploaded

"ML in Production",Oleksandr BaganFwdays

Bun (KitWorks Team Study 노별마루 발표 2024.4.22)Wonjun Hwang

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...Patryk Bandurski

AI as an Interface for Commercial BuildingsMemoori

Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxnull - The Open Security Community

DMCC Future of Trade Web3 - Special EditionDubai Multi Commodity Centre

Beyond Boundaries: Leveraging No-Code Solutions for Industry InnovationSafe Software

"Debugging python applications inside k8s environment", Andrii SoldatenkoFwdays

Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed

Install Stable Diffusion in windows machinePadma Pradeep

Streamlining Python Development: A Guide to a Modern Project SetupFlorian Wilhelm

costume and set research powerpoint presentationphoebematthew05

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays

Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity

Key Features Of Token Development (1).pptxLBM Solutions

Gen AI in Business - Global Trends Report 2024.pdfAddepto

SQL Database Design For Developers at php[tek] 2024Scott Keck-Warren

WordPress Websites for Engineers: Elevate Your Brandgvaughan

Human Factors of XR: Using Human Factors to Design XR SystemsMark Billinghurst

Recently uploaded (20)

"ML in Production",Oleksandr Bagan

Bun (KitWorks Team Study 노별마루 발표 2024.4.22)

Integration and Automation in Practice: CI/CD in Mule Integration and Automat...

AI as an Interface for Commercial Buildings

Nell’iperspazio con Rocket: il Framework Web di Rust!

E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx

DMCC Future of Trade Web3 - Special Edition

Beyond Boundaries: Leveraging No-Code Solutions for Industry Innovation

"Debugging python applications inside k8s environment", Andrii Soldatenko

Scanning the Internet for External Cloud Exposures via SSL Certs

Install Stable Diffusion in windows machine

Streamlining Python Development: A Guide to a Modern Project Setup

costume and set research powerpoint presentation

"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...

Dev Dives: Streamline document processing with UiPath Studio Web

Key Features Of Token Development (1).pptx

Gen AI in Business - Global Trends Report 2024.pdf

SQL Database Design For Developers at php[tek] 2024

WordPress Websites for Engineers: Elevate Your Brand

Human Factors of XR: Using Human Factors to Design XR Systems

Bloated Chefs: A Tale of Gluttony, & the Path to Enlightenment - ChefConf 2015

1. 4/3/15 @evan2645 EVAN GILMAN Bloated Chefs A Tale of Gluttony, and the Path to Enlightenment

2. 4/3/15BLOATED CHEFS

3. 4/3/15 Agenda BLOATED CHEFS 1. Chef resources in use at PD 2. Problems encountered as we grew 3. Measuring chef-client run 4. How we fixed it 5. How fast is it now?

4. 4/3/15BLOATED CHEFS CHEF @ PAGERDUTY

5. 4/3/15BLOATED CHEFS PD CHEF RESOURCES

6. 4/3/15 pd_iptables BLOATED CHEFS

7. 4/3/15 pd-ipsec::policies BLOATED CHEFS

8. 4/3/15 sumo_source BLOATED CHEFS

9. 4/3/15 pd_datadog_alert BLOATED CHEFS

10. 4/3/15BLOATED CHEFS

11. 4/3/15BLOATED CHEFS ALL WAS NOT WELL

12. 4/3/15 As we grew… BLOATED CHEFS

13. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs

14. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run

15. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run • chef-client run took several minutes

16. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run • chef-client run took several minutes • chef-client OOM

17. 4/3/15 As we grew… BLOATED CHEFS • CPU spikes during chef-client runs • Awkward pauses at the beginning of the run • chef-client run took several minutes • chef-client OOM

18. 4/3/15BLOATED CHEFS

19. 4/3/15BLOATED CHEFS

20. 4/3/15BLOATED CHEFS MEASURING

21. 4/3/15 Measuring Run Time BLOATED CHEFS

22. 4/3/15 Measuring Run Time BLOATED CHEFS https://github.com/joemiller/chef-handler-profiler

23. 4/3/15 Measuring Resources BLOATED CHEFS • Total number of resources per run, by type • Number of updated resources per run, by type

24. 4/3/15 Measuring Memory BLOATED CHEFS • Gather proc stats with sys-proctable • Gather GC stats • Can be emitted as statsd

25. 4/3/15 Measuring Memory BLOATED CHEFS

26. 4/3/15BLOATED CHEFS WHAT WE FOUND AND WHAT WE DID

27. 4/3/15 Step-through Searches BLOATED CHEFS

28. 4/3/15 Step-through Searches BLOATED CHEFS From this

29. 4/3/15 Step-through Searches BLOATED CHEFS From this To this

30. 4/3/15 Step-through Searches BLOATED CHEFS 417MB -> 190MB

31. 4/3/15 Step-through Searches BLOATED CHEFS 417MB -> 190MB ~54%

32. 4/3/15 Partial Searches BLOATED CHEFS

33. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results

34. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled

35. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node

36. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node • Two searches touched

37. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node • Two searches touched 90s -> 60s

38. 4/3/15 Partial Searches BLOATED CHEFS • Provide hash map of desired results • Minimizes volume of node data returned/handled • hash2node • Two searches touched 90s -> 60s 30%

39. 4/3/15 Result Memoization BLOATED CHEFS

40. 4/3/15 Result Memoization BLOATED CHEFS • Common search data

41. 4/3/15 Result Memoization BLOATED CHEFS • Common search data • API-backed LWRP’s

42. 4/3/15 Result Memoization BLOATED CHEFS • Common search data • API-backed LWRP’s • Can be generalized

43. 4/3/15 Result Memoization BLOATED CHEFS • Common search data • API-backed LWRP’s • Can be generalized

44. 4/3/15 API Tarpitting BLOATED CHEFS

45. 4/3/15 API Tarpitting BLOATED CHEFS Centralize calls

46. 4/3/15BLOATED CHEFS OTHER NASTIES

47. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards

48. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards • tmpfs storage

49. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards • tmpfs storage • Multiple package resources (Chef 12)

50. 4/3/15 Other Nasties BLOATED CHEFS • Too many conditional guards • tmpfs storage • Multiple package resources (Chef 12) Six seconds for twelve packages

51. 4/3/15BLOATED CHEFS BEFORE/AFTER

52. 4/3/15 Memory Saved BLOATED CHEFS Before: After:

53. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After:

54. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After: ~60MB

55. 4/3/15 Memory Saved BLOATED CHEFS Before: ~500MB After: ~60MB 88% less memory!

56. 4/3/15 Seconds Saved BLOATED CHEFS Before: After:

57. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After:

58. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After: ~30s/run

59. 4/3/15 Seconds Saved BLOATED CHEFS Before: ~180s/run After: ~30s/run ~84% faster!

60. 4/3/15BLOATED CHEFS FREEDOM

61. 4/3/15 Thank you. @evan2645 EVAN GILMAN