Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From snowflakes to a common automated platform

977 views

Published on

Wonga has been around for many years and it has gone through massive growth periods in the different regions they operate.

As a result, each engineering team did their best to keep going but that approach is not sustainable when you want to create an automated platform that should allow you to easily launch new financial products anywhere in the globe.

Since the beginning of 2016, we have been revamping how we automate things at Wonga, trying to set the foundations for such a platform, with emphasis on simple, pragmatic and efficient tooling that allows us to operate in a hybrid infrastructure (AWS and VMWare Vsphere) with both Windows and Linux servers.

There is still a long way ahead but this talk will be about where we were, where we want to be, tactical steps to start the journey and why we chose our current tools.

Published in: Engineering
  • Be the first to comment

From snowflakes to a common automated platform

  1. 1. FROM SNOWFLAKES TO A COMMON AUTOMATED PLATFORM Ricard Clau John Paul Newman
  2. 2. WHAT DO WE DO? • Automation group function in Wonga • Small team servicing X engineers in 5 locations • CI / CD pipelines, Logging / Monitoring, infrastructure provisioning, config management, … • Most of the team is quite new to the company
  3. 3. A BIT OF WONGA HISTORY • Started in 2007, DevOps was not even a thing! • Regions expansion, acquisitions, … • Massive growth, engineers did their best to keep up • Regulations happened, FCA approval needed • Massive turnover, knowledge lost, lack of docs…
  4. 4. CURRENT PROBLEMS • Snowflake servers, many attempts in the past failed • No unified processes in the group • Not great monitoring dashboards • No DevOps culture, we are seen as a service team • Sometimes, all these are great excuses
  5. 5. WANTTO ACHIEVE • Build / Provision servers & infrastructure from code • Needs to work for both Windows and Linux • Hybrid cloud (AWS) / datacenter (vSphere) • Simple, pragmatic and efficient tools • Progressive introduction of new tooling
  6. 6. INITIAL ROADMAP • Pick tools to build / automate everything • Rationalise CI / CD tooling • Plan a progressive migration to the AWS cloud • Rationalise logging / monitoring infrastructure • Build platform capabilities that can be shared
  7. 7. CI Jenkins Team City ThoughtWorks GoCD
  8. 8. CI Jenkins Team City ThoughtWorks GoCD
  9. 9. JENKINS • Hundreds of plugins and documentation • Job configuration from code via Jenkins 
 Job Builder (JJB) or Wonga's own JJB Ruby DSL* • Free! Allowing each team to have their own self- managed server and agents * https://github.com/wongatech/wongatech.github.io/blob/master/_drafts/jenkins-job-builder-ruby-dsl.md
  10. 10. SCM Gerrit GitLab GitHub
  11. 11. SCM Gerrit GitLab GitHub
  12. 12. GERRIT • Review and CI validation processes • Supports replication for DR • LDAP backed authentication • Integrates with internal tools, like JIRA • Detailed ACLs and nice project structure
  13. 13. GITHUB • Nice UI and developers familiarity • Hooks integration • Debatable Pull Requests model • Delegate DR, HA, etc… to Github • Has source code based wiki
  14. 14. SCM Puppet Labs Opscode Chef Ansible
  15. 15. SCM Puppet Labs Opscode Chef Ansible
  16. 16. ANSIBLE STRENGTHS • Easy learning curve • Agentless but you can also do ansible-pull • Plays nicely with running Windows servers • Decent community roles in Ansible Galaxy
  17. 17. ANSIBLE ISSUES • Ansible 2.0 is still a bit buggy • You always need a Linux control machine • Less flexible than Chef or Puppet (unless you write your own modules…) • Variable quality in Ansible Galaxy
  18. 18. MONITORING
  19. 19. MONITORING
  20. 20. ELK STACKVS SPLUNK • Decent in-house Splunk experience • Splunk dashboards still a bit better than Kibana • Logstash needs to configure GROK, Splunk can mostly guess itself • Still experimenting with ELK for our own stuff
  21. 21. INFLUXDATA • A platform for collecting time-series data • Model system metrics and business metrics • We use the Telegraf agent to send metrics, InfluxDB to store data and Grafana dashboards • Need to explore Kapacitor for monitoring
  22. 22. INFLUXDATA CONCERNS • Experimental support for Windows • Still 0.12 at the moment. Breaking API changes • Many people get confused about time-series data • InfluxDB cluster not free anymore
  23. 23. INFRASTRUCTURE
  24. 24. PACKER STRENGTHS • Works nicely with both Windows and Linux • Plays nicely with AWS and VMWare • Easy to share provisioning scripts • Easier to understand than Config Management tools (Chef, Puppet or Ansible)
  25. 25. PACKER CAVEATS • Need to be very prescriptive or the number of templates can get out of hand quickly • Hard to go with a DRY approach • Often not much benefit in Linux systems vs provisioning tools on startup
  26. 26. TERRAFORM STRENGTHS • Plays nicely with AWS and has some initial support for vSphere (actively developed) • Has a nice pluggable providers system to automate virtually everything… if you know Go • No real cloud agnostic competitor
  27. 27. TERRAFORM ISSUES • Not great documentation and error messages • Some providers don´t have nice update support • Tricky to store state files • Terraform modules are still a bit hacky • Relatively immature overall
  28. 28. SOME SUCCESS! • Tools decided, good engagement in the team • Building Packer AMIs andVMWare templates • Some services already fully managed by Ansible • Many servers rebuilt from config management • Small Terraform setups working
  29. 29. THE (NEAR) FUTURE • Consul for Service Discovery and Config storage • Better secrets / keys management (Vault) • Start the Prod migration to AWS (some components already running in PreProd) • Improve the current successes and think platform
  30. 30. BABY STEPS
  31. 31. A LONG JOURNEY AHEAD
  32. 32. QUESTIONS? • BTW… incidentally… we are hiring! • Come talk to us! • Thank you for listening!

×