A general purpose system for storing and analysing real-time time-series data

1. Project Phoenix Stephen Rose Ashok Subramanian MongoUK March 21, 2011

2. Background Architecture Data-store Selection Implementation

3. What is Phoenix? “a platform that allows data to be transferred seamlessly to the people who want it, presented in the way they want to see it, regardless to the source, type or location of the inputs”

4. What sort of data? Time-series Sensor readings External data feeds Meta data Adding context Credits: http://commons.wikimedia.org/wiki/File:CPI-plot.png

5. What is Phoenix? “a web basedplatform capable of ingesting, combining, processing and distributing data - in real time”

6. Target Markets Smaller, cheaper, better connected Credits: http://www.flickr.com/photos/opensourceway/4370251113/ Credits: http://www.flickr.com/photos/juggerphotos/4562680976/in/photostream/

7. Why develop a new platform? No existing platforms for the use cases Typical solutions use Excel, SQL cannot cope with amount of data no real-time capability

8. Real-time Analysis COMPLEX EVENT PROCESSING STREAM COMPUTING Reporting Processing Big Data Target Customer BUSINESS INTELLIGENCE BATCHPROCESSING Batch Analysis

9. Real-time Analysis COMPLEX EVENT PROCESSING STREAM COMPUTING PHOENIXPLATFORM Reporting Processing Big Data Target Customer BUSINESS INTELLIGENCE BATCHPROCESSING Batch Analysis

10. Project Timelines 2010 Research Market Customers Design Architecture Data model Development 4 developers Rapid! Release Version 1 Phoenix local Q1 Q2 Q3 Q4

11. Background Architecture Data-store Selection Implementation

12. Data Model Credits: http://www.flickr.com/photos/nazly/5502884394/sizes/z/

13. Data Highways and Lanes Credits: http://www.flickr.com/photos/mhzmaster/965947834/sizes/z/

14. LATITUDE MY CAR LONGITUDE MPG MPG GFORCES GFORCES PASSENGER HEART RATE LATITUDE MY WIFE’S CAR LONGITUDE MPG MPG TEMPERATURE TEMPERATURE

15. Markers Credits: http://www.flickr.com/photos/rebelxt/5284536592/sizes/z/in/photostream/

16. STEVE’S WIFE DRIVING STEVE DRIVING LATITUDE MY CAR LONGITUDE MPG MPG G FORCE G FORCE

17. Architecture Phoenix Customer APPLICATION STREAM PROCESSOR ALGORITHMS DATASTORE DATA FEEDS

18. Offline and Cloud Credits: http://www.flickr.com/photos/noii/2327713880/

19. Background Architecture Data Store Selection Implementation

20. Selection Criteria Local and Cloud Use cases – time-series and marker Performance Query Community Maintenance Deployment

21. 3 different categories

22. Data Stores Evaluated

23. Background Architecture Data Store Selection Implementation

24. Data Stores Phoenix Customer APPLICATION STREAM PROCESSOR ALGORITHMS DATASTORE DATA FEEDS TIME SERIES MARKERS

25. Marker Store T I M E FROM FROM TO L I N K E D H I G H W A Y S

26. Marker Store { "fromId" : ”Athlete1", "toId" : ”Ergometer", "startTime" : 1, "endTime" : 50 } { "fromId" : ”Athlete2", "toId" : ”Ergometer", "startTime" : 60, "endTime" : null } L I N K E D H I G H W A Y S

27. Marker Store { ”deviceId" : ”1", ”calibrationType" : ”sensor1", ”validFrom" : 1, ”validTo" : null “calibrations" : [ {”expected”: 10.0, “actual”: 12.9 }, {”expected”: 50.0, “actual”: 61.7 } ] } C A L I B R A T I O N M A R K E R S

28. Marker Store

29. Marker Store Stack Http JbossNettyWebserver JSON Java

30. Marker Store API JSON POST to /linkedhighway/create GET from /linkedhighway/findactive/{fromId} POST to /calibration/create GET from /calibration/{markerId}

31. Phoenix Data Store Ergometer Force Reading (Newtons) {X1, Y1} {Xn, Yn} Distance Travel (Metres) {X1, Y1} {Xn, Yn}

32. Phoenix Data Store { “_id”: “HandleForce”, “xDomainType”: “SINT64”, “xUnits”: “ms”, “yDomainType”: “DOUBLE”, “yUnits”: “N” } L A N E M E T A D A T A

33. Phoenix Data Store { ”_id" : 1, ”y" : 0 } { “_id”: 2, “y”: 0.5 } { “_id”: 3, “y”: 0.707 } { “_id”: 4, “y”: 0.866 } { “_id”: 5, “y”: 1 }… T I M E S E R I E S

34. Phoenix Data Store

35. Phoenix Data Store Stack Tcp JbossNetty Server – NIO Google Protocol Buffers Java

36. Phoenix Data Store API Commit connect() openLane(…) persist(x, y) disconnect() Query connect() submitQuery(…) returns a Cursor disconnect() J A V A C L I E N T

37. Performance

38. Phoenix Local AMD Athlon II 2.8GHz 64-bit 640GB 2.5” Hard Drive SATA 5400 rpm 2 X 2GB DDR3 Ubuntu 10.10 EXT4

39. Issues Faced No Journaling in Mongo 1.6 (Fixed in 1.8) Compression DB repair did not work once – unclean shutdown Assertion failure magic == 0x41424344 db/pdfile.h

40. Summary MongoDB - Does what it says on the tin

41. Thank You Questions? stephen.rose@mclaren.com ashok.subramanian@thoughtworks.com