Hammer of the Sysadmins
Menu de jour1. Intro2. Problem statement3. Brief therapy4. Used tools5. Live demo6. Autoscaling
Intro• ~50 servers• 3 teams (customer,dev,ops)• Unpredictable heavy load (req/s andtraffic)• Heterogenous infrastructure (...
Heavy load?
Brief therapy• Consistent configuration• Track all changes• Manage changes
Used tools• Puppet• Redmine• Git• Bash scripts
Live demo• http://www.joj.sk• joj-web@bonet.sk• https://redmine.joj.sk
Questions?peter.mihalik@bonet.sk
…ouch and we are hiring• job@bonetics.com (for developers)• job@bonet.sk (for sysadmins)
Autoscaling• Vmware clean system snapshot• Provisioning profile for each node type• Zabbix collecting performance data• Za...
Upcoming SlideShare
Loading in …5
×

Peter Mihalik: Puppet

717 views

Published on

Hammer of the Sysadmins based on puppet, git and redmine. Managing complex infrastructure producing 1/3 of slovak peering traffic in peaks. joj.sk, noviny.sk, huste.tv, tipsportextraliga.sk, prohokej.sk

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
717
On SlideShare
0
From Embeds
0
Number of Embeds
31
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Introduction to customer’s infrastructureShort overview of most critical issuesWhat we want to achieve in new infrastructureWhich tools were used to implement these solutionsLive demonstrationIf we have some extra time, we could drop a word about autoscaling techniques
  • TV JOJ is one of the two biggest commercial TV stations in slovakia broadcasting 3 channelsThey’re running a lot of web sites for channels, own TV shows, sports (streaming slovakicehockey play-off for example)and online news paper site ‘noviny.sk’You could predict traffic for sport events, but it’s much harder for news.And of course, we are running lot of technologies in our app stack.
  • This is traffic generated by whole infrastructure ontuesday
  • We need consistent configuration across all servers (if you have 5 balanced web servers, you definitely need to have same config on all of them)There is strong requirement to track all changes in infrastructure and applicationProper tools for change requests, configuration reviews and approvals are priceless when your team and stack becomes bigger.
  • Puppet– You definitely need somebody with object oriented design experience in your team- The common mistake is defining processes instead of proposed system state. You should think about where you want to be instead of how to get there (that’s puppet’s work).Redmine – issue tracking, integrated with version control (code, config, one place) - code review plugin to add more orgaiozational hierarchy in productionGit – comments with redmine issue references / fixesBash – master files deployment automation
  • Here we have some security breach and leaking some top secret information about unnamned celebrityWe should report it to our sysadmin teamAnd now we should assign it to proper sysadmin and apply some hotfix through apache configuration change.
  • Just a brief overview, think about it 
  • Peter Mihalik: Puppet

    1. 1. Hammer of the Sysadmins
    2. 2. Menu de jour1. Intro2. Problem statement3. Brief therapy4. Used tools5. Live demo6. Autoscaling
    3. 3. Intro• ~50 servers• 3 teams (customer,dev,ops)• Unpredictable heavy load (req/s andtraffic)• Heterogenous infrastructure (apache,lighttpd, node-js, solr, redis, percona,mms-rtmpd, LVS, keepalived, etc.)
    4. 4. Heavy load?
    5. 5. Brief therapy• Consistent configuration• Track all changes• Manage changes
    6. 6. Used tools• Puppet• Redmine• Git• Bash scripts
    7. 7. Live demo• http://www.joj.sk• joj-web@bonet.sk• https://redmine.joj.sk
    8. 8. Questions?peter.mihalik@bonet.sk
    9. 9. …ouch and we are hiring• job@bonetics.com (for developers)• job@bonet.sk (for sysadmins)
    10. 10. Autoscaling• Vmware clean system snapshot• Provisioning profile for each node type• Zabbix collecting performance data• Zabbix trigger for new server deployment• 10s of new webservers in 3 minutes at yourservice

    ×