Scalability and performance are paramount for building an industrial IoT platform. At General Electric we have been using multiple data platforms to support our customers. Our time-series platform collects information from edge devices to offer data scientists, developers and controls engineers with a secured platform to analyze their workload.
We used Cassandra as our system of record for several years. In the last 12 months we have shifted to Scylla. Scylla's ability to consolidate nodes has enabled us to reduce our cluster footprint by over 60%. The cost savings on operation and stability of Scylla enables us to support more customers, increase our up-time and lower the cost of operations.
In this session, we describe the migration process we followed when moving from Cassandra to Scylla, while maintaining our operations and onboarding new customers.
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Scylla Summit 2018: GE Predix - Industrial-Strength IoT at Scale
1. GE Digital on Scylla
Venkatesh Sivasubramanian
Senior Director, Data and Analytics Services, GE Digital
Arvind Singh
Engineer, Data Services, GE Digital
2. Presenter Bios
Venkatesh Sivasubramanian
Senior Director, Data and Analytics Services, GE Digital
Venkat drives the architecture and development of Data
and Analytics Services for Predix, an Industrial IoT platform.
Arvind Singh
Senior Staff Engineer, Data Services at GE Digital
Solving performance and scalability problems gives him a
high! He’s done just that at Cisco, Argus and now GE.
13. Requirements for Building the Largest
IIoT Platform in the World
▪ Sustain more than 1TB/day of data
▪ Absorb 10’s of billions of records per day
▪ Support millions of parallel analytics transactions
▪ Must keep sub-second response time, end-to-end
14. What Wasn’t Working
▪ Node storage limitation cause node sprawl
▪ Increasing and decreasing clusters became a challenge
▪ Application latency numbers are on the edge of the SLA
envelope
▪ Managing large number of nodes required increasing
our talent pool
15. Evaluation
▪ Started with a fairly small cluster of 5 nodes
▪ AWS based deployment, using Scylla AMIs
▪ Directed our application to Scylla, same API, no code change
▪ Node consolidation works: same performance, fewer nodes
▪ Testing takes time, prepare a team and test plan ahead of time
▪ Use Scylla Slack channel for help and questions
16. Our Data Model
▪ Time-series data model
▪ Sensor information inserted on a per tenant record
▪ Resulting in → A heavy single table
• Tenant info
• Tag info
• Metrics info
▪ We bucket data on time intervals
• Helps with partition size
▪ Rollups based on time-intervals are kept in database
▪ Different data attrition policies
17. How it Looks from the Application Side
▪ Our performance cluster setup
Component Quantity Resource Type
ScyllaDB nodes 9 I3.8Xlarge
TSQS nodes 5 I3.2Xlarge
Kafka and ZooKeeper Nodes 7 C4.8Xlarge
Go Pipeline 32 CloudFoundry instances
20. Migrating to Scylla
▪ Online operations must continue
▪ Used Spark to migrate existing data from Cassandra to Scylla
▪ Collaborate with Scylla team to optimize data model
▪ Reducing cluster footprints helps us reduce operational
overhead
21. Migrating Using Spark
▪ Operations continue as normal during migration
▪ Data model optimization is possible during migration process
22. Shrink the Cluster
▪ Same or higher workload with fewer nodes
Cassandra
75 i3.2xl
Node reduction
Lower cost
Reduced admin
overhead
Scylla
15 i3.8xl
23. Scylla Deployments
▪ Replacement for Cassandra clusters
▪ Various clusters, at different size
• Build a tool for the task
▪ Reduced clusters footprint by >60%
▪ Stable, predictable performance over time
Result: We can serve more users, at lower operational cost
24. Conclusions, or What We Care About
▪ Scalability is paramount
▪ Database stability
▪ Performance predictability
Introduction - Venkat and Arvind
How we deal with Time Series in the IIoT world
Kinds of problems we face
Exploration of ScyllaDB
And the migration Strategy we employed
GE isn’t just an appliance or lighting company.
GE Renewables manufacture and service some of the largest and powerful offshore wind turbines.
The picture show Hallide-X: Standing 260 meters tall from its heel to blade tips — more than half the height of the Empire State Building — with blades that each extend 107 meters, the turbine will generate 12 megawatts (16K European homes)
Wind is depending
To successfully
Better models of the entire site leads to better operations and avoid unplanned downtime
https://www.youtube.com/watch?v=IGDASBgrIRc
In Switzerland over 60% of the country’s electricity is generated by water.
Using drones to map the area around a hydro dam in Linthal, Switzerland the team was able to develop a way to store potential energy in an elevated basin
And visualize their entire operations
finding creative ways to store excess solar and wind energy in the form of pumped water
can be reprocessed by hydro to provide power during low periods of solar/wind
https://www.youtube.com/watch?v=IGDASBgrIRc
GE manufactures massive turbines used to generate electricity, Gas or Steam powered
Example produces enough electricity to power 500k homes
GE equipment generates 30% the worlds power
80% of power moved throughout north america is controlled by GE systems
A power plant in France using GE’s HA gas turbine won the Guinness World Record for net efficiency at 62.22 percent in 2016.
Atlanta Monitoring & Diagnostic Center >200GW capacity, >1500 GE turbines
1 million sensors attached to machines send 200 billion data points per day
Today and every day we will run over 900,000 analytics on the world’s largest fleet of gas turbines under management across 60 countries, covering 350 million people.
https://www.ge.com/reports/knowledge-power-takes-software/
state of the art jet engines
every 2 seconds there is a commercial airplane with a GE engine taking off
about 2200 aircraft are concurrently in the air, each carrying approximately 300 passengers, or a city of 660k in the air
the most efficient jet engine GE-NX long haul flight
ceramic matrix composite fan blades
https://www.ge.com/reports/the-art-of-engineering-the-worlds-largest-jet-engine-shows-off-composite-curves/
additive metal manufacturing
https://www.ge.com/reports/treat-avgeeks-inside-look-ges-3d-printed-aircraft-engine/
The 3D printed nozzle combined all 20 parts into a single unit, but it also weighed 25 percent less
https://www.ge.com/reports/epiphany-disruption-ge-additive-chief-explains-3d-printing-will-upend-manufacturing/
Ultra-long-haul flights to become more fuel efficient: perth to London 17h20m
https://www.ge.com/reports/flight-fancy-qantas-jet-flies-non-stop-australia-london-first-time-powered-ge/
20% less fuel vs planes of similar size. ~$1.6M per plane per year
30,000 fuel nozzles have been 3D printed for its LEAP engines.
Under the additive manufacturing method, the number of parts in a single fuel nozzle tip was reduced from about 20 pieces previously welded together to one whole piece.
https://www.3dprintingmedia.network/ge-aviation-already-3d-printed-30000-fuel-nozzles-for-its-leap-engine/
---
The ATP’s cubist-looking fuel heater is honeycombed with tiny complex passages. GE will 3D print it. Image credit: Tomas Kellner/GE Reports
https://www.ge.com/reports/treat-avgeeks-inside-look-ges-3d-printed-aircraft-engine/
GE Predix: The world’s leading IOT platform, helps monitor oil rigs, gas/wind turbines, nuclear reactors, escalators/elevators & flying airplanes, issue timely alerts and recommend actions. Predix not only saves its customers dollars, it helps save lives! One of those times, when you say every second counts, it literally means that! For Predix, the scalability and performance is mission critical.
Data Fabric offers a way to ingest at scale, enable data and analytical processing without knowing the The world’s leading IOT platform, helps monitor oil rigs, gas/wind turbines, nuclear reactors, escalators/elevators & flying airplanes, issue timely alerts and recommend actions. Predix not only saves its customers dollars, it helps save lives! One of those times, when you say every second counts, it literally means that! For Predix, the scalability and performance is mission critical.