Scylla Summit 2018: GE Predix - Industrial-Strength IoT at Scale
•Download as PPTX, PDF•
2 likes•5,268 views
Scalability and performance are paramount for building an industrial IoT platform. At General Electric we have been using multiple data platforms to support our customers. Our time-series platform collects information from edge devices to offer data scientists, developers and controls engineers with a secured platform to analyze their workload. We used Cassandra as our system of record for several years. In the last 12 months we have shifted to Scylla. Scylla's ability to consolidate nodes has enabled us to reduce our cluster footprint by over 60%. The cost savings on operation and stability of Scylla enables us to support more customers, increase our up-time and lower the cost of operations. In this session, we describe the migration process we followed when moving from Cassandra to Scylla, while maintaining our operations and onboarding new customers.
Recommended
Recommended
More Related Content
More from ScyllaDB
More from ScyllaDB (20)
Recently uploaded
Recently uploaded (20)
Scylla Summit 2018: GE Predix - Industrial-Strength IoT at Scale
- 1. GE Digital on Scylla Venkatesh Sivasubramanian Senior Director, Data and Analytics Services, GE Digital Arvind Singh Engineer, Data Services, GE Digital
- 2. Presenter Bios Venkatesh Sivasubramanian Senior Director, Data and Analytics Services, GE Digital Venkat drives the architecture and development of Data and Analytics Services for Predix, an Industrial IoT platform. Arvind Singh Senior Staff Engineer, Data Services at GE Digital Solving performance and scalability problems gives him a high! He’s done just that at Cisco, Argus and now GE.
- 3. The GE That’s Not Obvious
- 4. GE RenewablesLargest and the most powerful offshore wind turbines
- 5. GE Renewables Better models to avoid unplanned downtimes
- 7. Big rotating things GE Power
- 8. Atlanta Monitoring and Diagnostics GE Power
- 9. Every 2 seconds a GE Powered aircraft takes off GE Aviation
- 10. Creating impossibly complex parts at scale GE Aviation
- 11. GE Predix ▪ The world’s largest Industrial IoT Platform ▪ Cross-industries data collection, analytics and alerts generation
- 12. GE Predix - Data Fabric
- 13. Requirements for Building the Largest IIoT Platform in the World ▪ Sustain more than 1TB/day of data ▪ Absorb 10’s of billions of records per day ▪ Support millions of parallel analytics transactions ▪ Must keep sub-second response time, end-to-end
- 14. What Wasn’t Working ▪ Node storage limitation cause node sprawl ▪ Increasing and decreasing clusters became a challenge ▪ Application latency numbers are on the edge of the SLA envelope ▪ Managing large number of nodes required increasing our talent pool
- 15. Evaluation ▪ Started with a fairly small cluster of 5 nodes ▪ AWS based deployment, using Scylla AMIs ▪ Directed our application to Scylla, same API, no code change ▪ Node consolidation works: same performance, fewer nodes ▪ Testing takes time, prepare a team and test plan ahead of time ▪ Use Scylla Slack channel for help and questions
- 16. Our Data Model ▪ Time-series data model ▪ Sensor information inserted on a per tenant record ▪ Resulting in → A heavy single table • Tenant info • Tag info • Metrics info ▪ We bucket data on time intervals • Helps with partition size ▪ Rollups based on time-intervals are kept in database ▪ Different data attrition policies
- 17. How it Looks from the Application Side ▪ Our performance cluster setup Component Quantity Resource Type ScyllaDB nodes 9 I3.8Xlarge TSQS nodes 5 I3.2Xlarge Kafka and ZooKeeper Nodes 7 C4.8Xlarge Go Pipeline 32 CloudFoundry instances
- 18. Ingestion Performance Results ▪ ~2.5B data points per hour → ~700K data points per second
- 19. Query Performance Results ▪ ~325M Queries per hour → ~90K queries per second
- 20. Migrating to Scylla ▪ Online operations must continue ▪ Used Spark to migrate existing data from Cassandra to Scylla ▪ Collaborate with Scylla team to optimize data model ▪ Reducing cluster footprints helps us reduce operational overhead
- 21. Migrating Using Spark ▪ Operations continue as normal during migration ▪ Data model optimization is possible during migration process
- 22. Shrink the Cluster ▪ Same or higher workload with fewer nodes Cassandra 75 i3.2xl Node reduction Lower cost Reduced admin overhead Scylla 15 i3.8xl
- 23. Scylla Deployments ▪ Replacement for Cassandra clusters ▪ Various clusters, at different size • Build a tool for the task ▪ Reduced clusters footprint by >60% ▪ Stable, predictable performance over time Result: We can serve more users, at lower operational cost
- 24. Conclusions, or What We Care About ▪ Scalability is paramount ▪ Database stability ▪ Performance predictability
- 25. Thank You Any Questions ? Please stay in touch VenkateshS@ge.com & arvind.singh1@ge.com
Editor's Notes
- Introduction - Venkat and Arvind How we deal with Time Series in the IIoT world Kinds of problems we face Exploration of ScyllaDB And the migration Strategy we employed
- GE isn’t just an appliance or lighting company.
- GE Renewables manufacture and service some of the largest and powerful offshore wind turbines. The picture show Hallide-X: Standing 260 meters tall from its heel to blade tips — more than half the height of the Empire State Building — with blades that each extend 107 meters, the turbine will generate 12 megawatts (16K European homes) Wind is depending
- To successfully Better models of the entire site leads to better operations and avoid unplanned downtime https://www.youtube.com/watch?v=IGDASBgrIRc In Switzerland over 60% of the country’s electricity is generated by water.
- Using drones to map the area around a hydro dam in Linthal, Switzerland the team was able to develop a way to store potential energy in an elevated basin And visualize their entire operations finding creative ways to store excess solar and wind energy in the form of pumped water can be reprocessed by hydro to provide power during low periods of solar/wind https://www.youtube.com/watch?v=IGDASBgrIRc
- GE manufactures massive turbines used to generate electricity, Gas or Steam powered Example produces enough electricity to power 500k homes GE equipment generates 30% the worlds power 80% of power moved throughout north america is controlled by GE systems A power plant in France using GE’s HA gas turbine won the Guinness World Record for net efficiency at 62.22 percent in 2016.
- Atlanta Monitoring & Diagnostic Center >200GW capacity, >1500 GE turbines 1 million sensors attached to machines send 200 billion data points per day Today and every day we will run over 900,000 analytics on the world’s largest fleet of gas turbines under management across 60 countries, covering 350 million people. https://www.ge.com/reports/knowledge-power-takes-software/
- state of the art jet engines every 2 seconds there is a commercial airplane with a GE engine taking off about 2200 aircraft are concurrently in the air, each carrying approximately 300 passengers, or a city of 660k in the air the most efficient jet engine GE-NX long haul flight ceramic matrix composite fan blades https://www.ge.com/reports/the-art-of-engineering-the-worlds-largest-jet-engine-shows-off-composite-curves/ additive metal manufacturing https://www.ge.com/reports/treat-avgeeks-inside-look-ges-3d-printed-aircraft-engine/ The 3D printed nozzle combined all 20 parts into a single unit, but it also weighed 25 percent less https://www.ge.com/reports/epiphany-disruption-ge-additive-chief-explains-3d-printing-will-upend-manufacturing/ Ultra-long-haul flights to become more fuel efficient: perth to London 17h20m https://www.ge.com/reports/flight-fancy-qantas-jet-flies-non-stop-australia-london-first-time-powered-ge/ 20% less fuel vs planes of similar size. ~$1.6M per plane per year
- 30,000 fuel nozzles have been 3D printed for its LEAP engines. Under the additive manufacturing method, the number of parts in a single fuel nozzle tip was reduced from about 20 pieces previously welded together to one whole piece. https://www.3dprintingmedia.network/ge-aviation-already-3d-printed-30000-fuel-nozzles-for-its-leap-engine/ --- The ATP’s cubist-looking fuel heater is honeycombed with tiny complex passages. GE will 3D print it. Image credit: Tomas Kellner/GE Reports https://www.ge.com/reports/treat-avgeeks-inside-look-ges-3d-printed-aircraft-engine/
- GE Predix: The world’s leading IOT platform, helps monitor oil rigs, gas/wind turbines, nuclear reactors, escalators/elevators & flying airplanes, issue timely alerts and recommend actions. Predix not only saves its customers dollars, it helps save lives! One of those times, when you say every second counts, it literally means that! For Predix, the scalability and performance is mission critical.
- Data Fabric offers a way to ingest at scale, enable data and analytical processing without knowing the The world’s leading IOT platform, helps monitor oil rigs, gas/wind turbines, nuclear reactors, escalators/elevators & flying airplanes, issue timely alerts and recommend actions. Predix not only saves its customers dollars, it helps save lives! One of those times, when you say every second counts, it literally means that! For Predix, the scalability and performance is mission critical.
- We build a system to last. Our IoT system have to