Recommended
Recommended
More Related Content
Similar to freebeersessions #26 Scaling Up and Out Using Open Source at Netstar
Similar to freebeersessions #26 Scaling Up and Out Using Open Source at Netstar (20)
Recently uploaded
Recently uploaded (20)
freebeersessions #26 Scaling Up and Out Using Open Source at Netstar
- 1. Scaling Up and Out using Open Source at Netstar
- 2. About me…
- 3. About me…
- 4. About me… • I’m a proud father of my 9 year old daughter, Hannah. • I’ve been in the technology industry for 20 years. • Of that 20 years, almost 13 years have been at Netstar. • My background is software development; .NET, JS, Python, R, Scala, Java. • My interests outside of Technology are reading of which I enjoy mostly psychology, philosophy and anthropology.
- 5. Disclaimer… • All of the views or opinions that I’m about to present are mine and mine alone and do not represent the view or opinions of any entity whatsoever with which I have been, am now or will be affiliated. • In essence, if you’re offended by anything I present or say there’s only one person to blame: • Me • Oh, and if I get the pronunciation of some of these components/technologies wrong don’t laugh, they’re not real words
- 6. About Netstar… • Serving South Africans for 24 years. • Over 650k vehicles – 900k installed units – combination of RF & GPS/GSM • Employs 900 people (over 100 in IT, Technical and R&D) • Entered Commercial Fleet market in 2004 • Provide traffic and data services to various GPS and vehicle OEM’s since 2009 • Sub-Saharan Africa, East Africa and soon in West Africa • Australia and other international territories Normal marketing…
- 8. About Netstar… • We’re a technology company… • Delivering IOT solutions to our customers… • Logistics • Insurance • Consumers We’ve been doing IOT for 24 years… What we actually are!
- 9. Changing landscape… The Netstar open source journey started around 2013 when we realized a fundamental shift in one major factor: • Customers need for more and real-time data • From a location update on an asset every 5 minutes we’ve had to reduce that 15 seconds. • We’ve got customers that now require 1 second GPS resolution with 15 accelerometer data points per second. • We went from 350 message per second in 2013 to 3500 message per second in 2018 with a 75% growth in signal rates over the last year. The changes in the customer landscape
- 10. Changing landscape… This as facilitated by changes in our landscape: • Reduction in GSM data costs • Reduction in GSM device costs • Proliferation of IOT devices The changes in the technology landscape
- 11. Changing landscape… Technology limitations – • Using traditional RDBMS did not cut it • Limitations in scalability both vertically and horizontally were a huge problem. • Traditional SAN disks were also an issue , we were making large investments into specialised SAN technologies such as XIOtech. Data Center implementation costs – • In 2013 we built a 2nd on-prem data centre in our office park at huge cost but that provided us with on campus HA. Licensing fees • RDBMS licensing fees were also increasing and the number of licenses was becoming untenable. There were challenges with the changing landscapes: Our costs were linearly related to the data rate and storage costs, we needed another way.
- 12. What did we do?
- 13. What did we do?
- 14. How? Evolving architecture Instead of architecture we decided on principals • DevOps – at least the CI/CD portion of it. • Evolutionary architecture – Martin Fowler • Micro-services – S.O.L.I.D. • Event based – Martin Fowler • Polyglot • Stateless • Resilient So how did we resolve this rising costs issue? This was our manifesto…
- 15. How? Enablers There were basically 2 enablers for this architecture…. The 2 enablers for this were… • Docker? Containerization • Docker made their containerization platform available in 2014. • Containers gave use the flexibility of CI and CD and scale • Stephan Fabel (Product Manager at Canonical) is quoted as saying “The containerization craze shows no signs of abating” • Google? and Orchestration. • Google open sourced Kubernetes • Fantastic for the orchestration and automation of many OPS functions • Kubernetes seems to be taking the lead as the orchestrator of choice. • With the advent of Helm and HelmCharts deploying “infrastructure components has become much easier than ever.
- 16. How? Initial deployment Our first deployment of this architecture was purely a IaaS in our local data centers. • 5 node deployment • CoreOs • Kubernetes 1.5.2 • DevOps tool sets Enterprises are wary of the use of Open Source without support and backing by a major player. Initial deployment Cloud is Software Defined Infrastructure, use Software to define it.
- 17. How? DevOps DevOps Open Source Toolset
- 18. How? Cassandra We chose Cassandra as our store data storage platform. • Facebook? Cassandra. • Deployed as containers in Kubernetes. • Highly redundant and scalable. • Started with a 3 node cluster with 1.5TB. • Currently sitting at 12 node cluster with 24TB. • We’ve benchmarked our cluster at about 32000 write ops per second which reading at 1000 ops per second. Our always on data store In the three years that our cluster has been running we’ve had one outage C* of 4 hours.
- 19. How? Kafka and RabbitMQ At the moment we’re using 2 event bus technologies • LinkedIn? Kafka • Kafka provides us with broker less event bus for high performance writes and reads. • 3 node Kafka cluster deployed. • Benchmarked at 100000 writes and reads per second. • Pivotal? RabbitMQ • RabbitMQ provides us with traditional broker based event bus (pub sub with routing keys). • 3 node RabbitMQ cluster deployed. • Benchmarked at 10000 messages per second. Our always on event bus Using fit-for-purpose toolsets
- 20. How? Spark, Scala and R. Spark and R are our tools of choice for extracting, analyzing and visualizing our data from Cassandra • Spark • Massively parallel execution for ETL and analytics. • 6 node spark cluster integrated with Kubernetes. • Mostly Scala scripts and submitted jars. • Extract to Cassandra and RDBMS. • R (or arrr!) • Currently we’re using R to build our ML models from our data sets. • Applying them to streams from Kafka using SparkR. Our ETL, Analytics and Stream Analytics toolset Machine Learning and AI are huge buzzwords but applying them in production is another thing all together.
- 21. Difficulties… • Enterprise mindset • Open source is something that some geek is doing afterhours. • Were do we get support for these technologies? • Culture • Getting more developers to drink the cool-aide. And there were many… The challenge is to change perspective.
- 22. THANK YOU!
Editor's Notes
- Hi, I’m Quintin de Kok (not the one you’re thinking of, Cloud Solutions Archtect at Netstar) and I’m here to talk to you about Netstar’s journey to the cloud.
- I’m a proud father of my 9 year old daughter, Hannah. I’ve been in the technology industry for 20 years. Of that 20 years, almost 13 years have been at Netstar. My background is software development; .NET, JS, Python, R, Scala, Java. My interests outside of Technology are reading of which I enjoy mostly psychology, philosophy and anthropology.
- All of the views or opinions that I’m about to present are mine and mine alone and do not represent the view or opinions of any entity whatsoever with which I have been, am now or will be affiliated. In essence, if you’re offended by anything I present or say there’s only one person to blame: Me Oh, and if I get the pronunciation of some of these components/technologies wrong don’t laugh
- Serving South Africans for 24 years. Over 650k vehicles – 900k installed units – combination of RF & GPS/GSM Employs 900 people (over 100 in IT, Technical and R&D) Entered Commercial Fleet market in 2004 Provide traffic and data services to various GPS and vehicle OEM’s since 2009 Sub-Saharan Africa, East Africa and soon in West Africa Australia and other international territories
- This is the perception in the market.
- We’re a technology company… Delivering IOT solutions to our customers… Logistics Insurance Consumers We’ve been doing IOT for 24 years…
- Our open journey started in 2013 when we realized a fundamental shift in a few factors: Customers need for More and Real-time data – Where our commercial customers were happy with a location update on an asset every 5 minutes we’ve had to reduce that 15 seconds. Where our insurance customers were happy with a data at a 1 minute resolution we’ve got customers that now require 1 second GPS resolution with 15 accelerometer data points per second. We went from 350 message per second in 2013 to 3500 message per second in 2018 with a 75% growth in signal rates over the last 6 months. The graph on the right shows the growth in our telematics line of products over the last 4 years
- Reduction in GSM data costs – Through the use of specialised IOT GSM networks has also reduced the cost of GSM data by 30%. Reduction in GSM device costs – New consumer telematic GSM based product in which we reduced the cost of the device by 50% New commercial fleet telematics GSM device where we also reduced the costs the device by 50% Proliferation of IOT devices Here are some examples, our new bicycle recovery unit, our wireless IOT gateway and our miniature asset tracking device.
- There were multiple problems with the changing customer and technology landscapes: Technology limitations – Using traditional RDBMS did not cut it Limitations in scalability both vertically and horizontally were a huge problem. Traditional SAN disks were also an issue Had to make large investments into specialised SAN technologies such as XIOtech. Technology limitations – Using traditional RDBMS did not cut it Limitations in scalability both vertically and horizontally were a huge problem. Traditional SAN disks were also an issue Had to make large investments into specialised SAN technologies such as XIOtech. Data Center implementation costs – In 2013 we built a 2nd on-prem data centre in our office park at huge cost. Provided HA Licensing fees RDBMS licensing fees were also increasing and the number of licenses was becoming unbearable. Our costs were linearly related to the data rate and storage costs, we needed another way.
- We had to change our perspective. From a bunch of hanging bats to a bunch of hanging bats…
- We had to change our perspective. From a bunch of hanging bats to a bunch of hanging bats…
- DevOps – Evolutionary Architecture – Micro-services – Single responsibility principle[6] a class should have only a single responsibility (i.e. changes to only one part of the software's specification should be able to affect the specification of the class). Open/closed principle[7] "software entities … should be open for extension, but closed for modification." Liskov substitution principle[8] "objects in a program should be replaceable with instances of their subtypes without altering the correctness of that program." See also design by contract. Interface segregation principle[9] "many client-specific interfaces are better than one general-purpose interface."[4] Dependency inversion principle[10] one should "depend upon abstractions, [not] concretions."[4] Event based – Also a micro-services architecture approach that basically promotes using events on an event bus rather than direct coupling of APIs on micro-service. Polyglot – The premise here is that any team can code in any language and use any data store as long as the CI and CD components for the code and store are automated. Stateless – Any micro-service should be stateless and where they have state
- The 2 enablers for this were… Docker? Containerization Docker made their containerization platform available in 2014. Containers gave use the flexibility of CI and CD and scale Stephan Fabel (Product Manager at Canonical) is quoted as saying “The containerization craze shows no signs of abating” Google? and Orchestration. Google open sourced Kubernetes Fantatic for the orchestration and automation of many OPS functions – allowing auto scaling, ease of scaling and recovery. Kubernetes seems to be taking the lead as the orchestrator of choice with AWS, Azure and many other cloud providers offering it as a service.
- Initial – Our first deployment of this architecture was purely a IaaS in our local data centers. 5 node deployment CoreOs Kubernetes 1.5.2 DevOps tool sets This was a major effort and I tweeted my sentiments on this late last year, “Installing Kubernetes is like a temperamental partner, “you said wat? I’m not talking to you anymore!”. Enterprises are wary of Open source - This is where Azure with the PaaS and Partner, through the Azure Marketplace, offerings have been a great asset. It takes less than 1 day for us to spin up a new environment on Azure Container Services so deployment time for our environment has been drastically reduced, from 1 month to 1 day. One of the takeaways we had here was that, because the Kubenetes is cloud and is basically SDI (Software Defined Infrastructure), why not write software to define your infrastructure. This is where Linux bash became a great tool for us for reproducibility and documentation.
- Our CI/CD toolset was the first to be deployed in docker on Kubernetes as a start to our cluster journey. Git - where our source, Dockerfiles and YAML for Kubernetes (both dev and prod) are configured by the developer. Jenkins - Once checked into development branch or merged into master Jenkins will build and test the micro-service or library. 3. Jenkins will then push the library to Jforg Artifactory which will then either present the repository or kick off the Docker build. Dockers are then kept in our own docker repository. Artifcatory will then also kick off the kubectl scripts to update Kubernetes with the latest build of the micro-service. We use Prometheus and Grafana (nice templates and Daemon sets for Kubernetes) to do continuous Application Performance Monitoring. Which is fed-back to the developers to begin the process again.
- Facebook? Cassandra. Deployed as containers in Kubernetes. – We were definitely one of the first to do this but our architecture principals forced us to make this sacrifice (and it was) but we’re reaping the benefits now. Highly redundant and scalable – Data Centre and Rack aware we were able to utilise the campus HA with 3 factor replication. Started with a 3 node cluster with 1.5TB. Currently sitting at 12 node cluster with 24TB. We’ve benchmarked our cluster at about 32000 write ops per second which reading at 1000 ops per second. In the three years that our cluster has been running we’ve had one outage C* of 4 hours (due to my favourite thing – Certificates)
- LinkedIn? Kafka Kafka provides us with broker less event bus for high performance writes and reads. 3 node Kafka cluster deployed. Benchmarked at 100000 writes and reads per second. Pivotal? RabbitMQ RabbitMQ provides us with traditional broker based event bus (pub sub with routing keys). 3 node RabbitMQ cluster deployed. Benchmarked at 10000 messages per second. Using fit-for-purpose toolsets where necissary
- Spark Massively parallel execution for ETL and analytics. 6 node spark cluster integrated with Kubernetes. Mostly Scala scripts and submitted jars. Extract to Cassandrs and RDBMS. R (or arrr!) Currently we’re using R to build our ML models from our data sets. Applying them to streams from Kafka using SparkR. Machine Learning and AI are huge buzzwords but applying them in production is another thing all together.
- Enterprise mindset Open source is something that some geek is doing afterhours. – This is the mindset of many of the CTOs and MDs out there, but it’s if you look at the companies that are outsourcing some of these techs, they’re massive but interestingly enough their core business is not technology but rather marketing and the likes. Were do we get support for these technologies? Databricks – supporting Spark and pushing it’s use in the cloud, natively in Azure. DataStax – providing curated versions of Cassandra and 3rd line support for major issues and open sourcing drivers. Confluent – providing support for Kafka and open sourcing drivers. Microsoft – Their involvement in open source cannot be underestimated, buying GitHub and appointing a Linux guy as the CEO and appointing one of the founding members of Kubernetes, Brendan Burns.. Culture – I think Candice hit this one on the head, one of the most difficult challenges has been getting our own developers to drink the cool-aid. The challenge is to change perspective.
- Thank you, if you’d like to ask any questions feel free to google “quintin de kok”, no, don’t do that , you’ll get a picture of this guy