As the dangers of global climate change multiply, utility companies seek methods to reduce carbon emissions, such as integrating renewable and sustainable energy sources like wind, solar, and hydroelectric power. Renewable energy not only has the power to improve climate conditions, it also encourages economic growth. By combining advances in sensor technology with machine learning algorithms and environmental data, utility companies can monitor energy sources in real time to make faster decisions and speed innovation.
In this session, Nikita Shamgunov, CTO and co-founder of MemSQL, will conduct a live demonstration based on real-time data from 2 million sensors on 197,000 wind turbines installed on wind farms around the world. This Internet of Things (IoT) simulation explores the ways utility companies can integrate new data pipelines into established infrastructure. Attendees will learn how to deploy this breakthrough technology composed of Apache Kafka, a real-time message queue; Streamliner, an integrated Apache Spark solution; MemSQL Ops, a cluster management and monitoring interface; and a set of simulated data producers written in Python. By applying machine learning to analyze millions of data points in real time, the data pipeline predicts and visualizes health of wind farms at global scale. This architecture propels innovation in the energy industry and is replicable across other IoT applications including smart cities, connected cars, and digital healthcare.
2. Topics
• The On-Demand Economy
• From In-Memory Compu8ng to In-Memory Databases
• Renewable Energy and PowerStream
• Demo and Q&A
(c) Nikita Shamgunov and MemSQL
11. Achieving sub 100 ms latency
• Real-'me monitoring and analy'cs on streaming video
• Proac'vely diagnose issues in real-'me
• Deliver be9er viewer experience
(c) Nikita Shamgunov and MemSQL
12. Massive Ingest AND Analy1cs
• Instant accuracy to the latest repin
• Build real-5me analy5c applica5ons
• 1 GB/sec totaling 72 TB/day
(c) Nikita Shamgunov and MemSQL
16. In-Memory Databases...
• Use memory instead of disk
• Do not (need to) save data on disk
(c) Nikita Shamgunov and MemSQL
17. In-Memory Databases...
• Use memory instead of disk
• Do not (need to) save data on disk
(c) Nikita Shamgunov and MemSQL
18. In-Memory Databases...
• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
(c) Nikita Shamgunov and MemSQL
19. In-Memory Databases...
• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
(c) Nikita Shamgunov and MemSQL
20. In-Memory Databases...
• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
Well, some)mes...
(c) Nikita Shamgunov and MemSQL
23. In-Memory Databases
• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
(c) Nikita Shamgunov and MemSQL
24. In-Memory Databases
• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
(c) Nikita Shamgunov and MemSQL
25. In-Memory Databases
• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
• Tend to be distributed systems
(c) Nikita Shamgunov and MemSQL
26. In-Memory Databases
• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
• Tend to be distributed systems
• Have a different set of boClenecks
(c) Nikita Shamgunov and MemSQL
28. All database workloads will be
running on in-memory databases
(c) Nikita Shamgunov and MemSQL
29. Why?
• Memory is ge,ng cheaper (about 40% every year)
(c) Nikita Shamgunov and MemSQL
30. Why?
• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new
tape, etc)
(c) Nikita Shamgunov and MemSQL
31. Why?
• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new
tape, etc)
• In-memory databases leverage SSD (no random writes)
(c) Nikita Shamgunov and MemSQL
32. Why?
• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new
tape, etc)
• In-memory databases leverage SSD (no random writes)
• NVRAM is coming (and could be cheaper than SSD)
(c) Nikita Shamgunov and MemSQL
33. Why?
• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new
tape, etc)
• In-memory databases leverage SSD (no random writes)
• NVRAM is coming (and could be cheaper than SSD)
In-memory databases are tuned to
modern hardware and modern workloads
(c) Nikita Shamgunov and MemSQL
47. Demo Sequence
• Powerstream user interface
• Showcase largest windfarms
• Real-8me simula8ons
• Witness live opera8ons
• Ease of new pipeline setup
• Ka>a subscrip8on
(c) Nikita Shamgunov and MemSQL
48. Enabling predic.ve analy.cs
• Use exis(ng models from SAS
• Create models in Spark MLlib
• Predic(ve scoring as part of the pipeline
(c) Nikita Shamgunov and MemSQL
49. From a Real-Time Dashboard to Predic5ve
Applica5ons
(c) Nikita Shamgunov and MemSQL