Project Phoenix<br />Stephen Rose<br />Ashok Subramanian<br />MongoUK<br />March 21, 2011<br />
Background<br />Architecture<br />Data-store Selection<br />Implementation<br />
What is Phoenix?<br />	“a platform that allows data to be transferred seamlessly to the people who want it, presented in t...
What sort of data?<br />Time-series<br />Sensor readings<br />External data feeds<br />Meta data<br />Adding context<br />...
What is Phoenix?<br />	“a web basedplatform capable of ingesting, combining, processing and distributing data - in real ti...
Target Markets<br />Smaller, cheaper, better connected<br />Credits: http://www.flickr.com/photos/opensourceway/4370251113...
Why develop a new platform?<br />No existing platforms for the use cases<br />Typical solutions use Excel, SQL<br />cannot...
Real-time Analysis<br />COMPLEX EVENT PROCESSING<br />STREAM<br />COMPUTING<br />Reporting<br />Processing<br />Big Data<b...
Real-time Analysis<br />COMPLEX EVENT PROCESSING<br />STREAM<br />COMPUTING<br />PHOENIXPLATFORM<br />Reporting<br />Proce...
Project Timelines<br />2010			                            <br />Research<br />Market<br />Customers<br />Design<br />Archi...
Background<br />Architecture<br />Data-store Selection<br />Implementation<br />
Data Model<br />Credits: http://www.flickr.com/photos/nazly/5502884394/sizes/z/<br />
Data Highways and Lanes<br />Credits: http://www.flickr.com/photos/mhzmaster/965947834/sizes/z/<br />
LATITUDE<br />MY CAR<br />LONGITUDE<br />MPG<br />MPG<br />GFORCES<br />GFORCES<br />PASSENGER HEART RATE<br />LATITUDE<br...
Markers<br />Credits: http://www.flickr.com/photos/rebelxt/5284536592/sizes/z/in/photostream/<br />
STEVE’S WIFE DRIVING<br />STEVE DRIVING<br />LATITUDE<br />MY CAR<br />LONGITUDE<br />MPG<br />MPG<br />G FORCE<br />G FOR...
Architecture<br />Phoenix  		   					Customer<br />APPLICATION<br />STREAM PROCESSOR<br />ALGORITHMS<br />DATASTORE<br />D...
Offline and Cloud<br />Credits: http://www.flickr.com/photos/noii/2327713880/ <br />
Background<br />Architecture<br />Data Store Selection<br />Implementation<br />
Selection Criteria<br />Local and Cloud<br />Use cases – time-series and marker<br />Performance<br />Query<br />Community...
3 different categories<br />
Data Stores Evaluated<br />
Background<br />Architecture<br />Data Store Selection<br />Implementation<br />
Data Stores<br />Phoenix  Customer<br />APPLICATION<br />STREAM PROCESSOR<br />ALGORITHMS<br />DATASTORE<br />DATA FEEDS<b...
Marker Store<br />T I M E<br />FROM<br />FROM<br />TO<br />L I N K E D		H I G H W A Y S<br />
Marker Store<br />{	"fromId" : ”Athlete1",<br />	"toId" : ”Ergometer",<br />	"startTime" : 1,<br />	"endTime" : 50<br />}<...
Marker Store<br />{	”deviceId" : ”1",<br />	”calibrationType" : ”sensor1",<br />	”validFrom" : 1,<br />	”validTo" : null<b...
Marker Store<br />
Marker Store Stack<br />Http<br />JbossNettyWebserver<br />JSON<br />Java<br />
Marker Store API<br />JSON<br />POST to /linkedhighway/create<br />GET from /linkedhighway/findactive/{fromId}<br />POST t...
Phoenix Data Store<br />Ergometer<br />Force Reading (Newtons)<br />{X1, Y1}<br />{Xn, Yn}<br />Distance Travel (Metres)<b...
Phoenix Data Store<br />{	“_id”: “HandleForce”,<br />	“xDomainType”: “SINT64”,<br />	“xUnits”:	“ms”,<br />	“yDomainType”:	...
Phoenix Data Store<br />{	”_id" : 1,<br />	”y" : 0<br />}<br />{	“_id”: 2,<br />	“y”: 0.5<br />}<br />{	“_id”: 3,<br />	“y...
Phoenix Data Store<br />
Phoenix Data Store Stack<br />Tcp<br />JbossNetty Server – NIO<br />Google Protocol Buffers<br />Java<br />
Phoenix Data Store API<br />Commit<br />connect()<br />openLane(…)<br />persist(x, y)<br />disconnect()<br />Query<br />co...
Performance<br />
Phoenix Local<br />AMD Athlon II 2.8GHz 64-bit<br />640GB 2.5” Hard Drive SATA 5400 rpm<br />2 X 2GB DDR3<br />Ubuntu 10.1...
Issues Faced<br />No Journaling in Mongo 1.6 (Fixed in 1.8)<br />Compression<br />DB repair did not work once – unclean sh...
Summary<br />MongoDB - Does what it says on the tin<br />
Thank You<br />Questions?<br />stephen.rose@mclaren.com<br />ashok.subramanian@thoughtworks.com<br />
Upcoming SlideShare
Loading in …5
×

Project Phoenix - MongoUK 2011

3,513 views

Published on

A general purpose system for storing and analysing real-time time-series data

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,513
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
5
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • generic
  • Increased connectivityCheaper, smaller sensorsNew computing techniques
  • Each sensor manufacturer creates a custom tool to view data from that sensor
  • Relative performance of the different types of data stores with BerkeleyDB as the baseline to benchmark against
  • Project Phoenix - MongoUK 2011

    1. 1. Project Phoenix<br />Stephen Rose<br />Ashok Subramanian<br />MongoUK<br />March 21, 2011<br />
    2. 2. Background<br />Architecture<br />Data-store Selection<br />Implementation<br />
    3. 3. What is Phoenix?<br /> “a platform that allows data to be transferred seamlessly to the people who want it, presented in the way they want to see it, regardless to the source, type or location of the inputs”<br />
    4. 4. What sort of data?<br />Time-series<br />Sensor readings<br />External data feeds<br />Meta data<br />Adding context<br />Credits: http://commons.wikimedia.org/wiki/File:CPI-plot.png<br />
    5. 5. What is Phoenix?<br /> “a web basedplatform capable of ingesting, combining, processing and distributing data - in real time”<br />
    6. 6. Target Markets<br />Smaller, cheaper, better connected<br />Credits: http://www.flickr.com/photos/opensourceway/4370251113/<br />Credits: http://www.flickr.com/photos/juggerphotos/4562680976/in/photostream/<br />
    7. 7. Why develop a new platform?<br />No existing platforms for the use cases<br />Typical solutions use Excel, SQL<br />cannot cope with amount of data<br />no real-time capability<br />
    8. 8. Real-time Analysis<br />COMPLEX EVENT PROCESSING<br />STREAM<br />COMPUTING<br />Reporting<br />Processing<br />Big Data<br />Target Customer<br />BUSINESS INTELLIGENCE<br />BATCHPROCESSING<br />Batch Analysis<br />
    9. 9. Real-time Analysis<br />COMPLEX EVENT PROCESSING<br />STREAM<br />COMPUTING<br />PHOENIXPLATFORM<br />Reporting<br />Processing<br />Big Data<br />Target Customer<br />BUSINESS INTELLIGENCE<br />BATCHPROCESSING<br />Batch Analysis<br />
    10. 10. Project Timelines<br />2010 <br />Research<br />Market<br />Customers<br />Design<br />Architecture<br />Data model<br />Development<br />4 developers<br />Rapid!<br />Release<br />Version 1<br />Phoenix local<br />Q1 Q2 Q3 Q4 <br />
    11. 11. Background<br />Architecture<br />Data-store Selection<br />Implementation<br />
    12. 12. Data Model<br />Credits: http://www.flickr.com/photos/nazly/5502884394/sizes/z/<br />
    13. 13. Data Highways and Lanes<br />Credits: http://www.flickr.com/photos/mhzmaster/965947834/sizes/z/<br />
    14. 14. LATITUDE<br />MY CAR<br />LONGITUDE<br />MPG<br />MPG<br />GFORCES<br />GFORCES<br />PASSENGER HEART RATE<br />LATITUDE<br />MY WIFE’S CAR<br />LONGITUDE<br />MPG<br />MPG<br />TEMPERATURE<br />TEMPERATURE<br />
    15. 15. Markers<br />Credits: http://www.flickr.com/photos/rebelxt/5284536592/sizes/z/in/photostream/<br />
    16. 16. STEVE’S WIFE DRIVING<br />STEVE DRIVING<br />LATITUDE<br />MY CAR<br />LONGITUDE<br />MPG<br />MPG<br />G FORCE<br />G FORCE<br />
    17. 17. Architecture<br />Phoenix Customer<br />APPLICATION<br />STREAM PROCESSOR<br />ALGORITHMS<br />DATASTORE<br />DATA FEEDS<br />
    18. 18. Offline and Cloud<br />Credits: http://www.flickr.com/photos/noii/2327713880/ <br />
    19. 19. Background<br />Architecture<br />Data Store Selection<br />Implementation<br />
    20. 20. Selection Criteria<br />Local and Cloud<br />Use cases – time-series and marker<br />Performance<br />Query<br />Community<br />Maintenance<br />Deployment<br />
    21. 21. 3 different categories<br />
    22. 22. Data Stores Evaluated<br />
    23. 23. Background<br />Architecture<br />Data Store Selection<br />Implementation<br />
    24. 24. Data Stores<br />Phoenix Customer<br />APPLICATION<br />STREAM PROCESSOR<br />ALGORITHMS<br />DATASTORE<br />DATA FEEDS<br />TIME SERIES<br />MARKERS<br />
    25. 25. Marker Store<br />T I M E<br />FROM<br />FROM<br />TO<br />L I N K E D H I G H W A Y S<br />
    26. 26. Marker Store<br />{ "fromId" : ”Athlete1",<br /> "toId" : ”Ergometer",<br /> "startTime" : 1,<br /> "endTime" : 50<br />}<br />{ "fromId" : ”Athlete2",<br /> "toId" : ”Ergometer",<br /> "startTime" : 60,<br /> "endTime" : null<br />}<br />L I N K E D H I G H W A Y S<br />
    27. 27. Marker Store<br />{ ”deviceId" : ”1",<br /> ”calibrationType" : ”sensor1",<br /> ”validFrom" : 1,<br /> ”validTo" : null<br /> “calibrations" : <br /> [ <br /> {”expected”: 10.0,<br /> “actual”: 12.9<br /> },<br /> {”expected”: 50.0,<br /> “actual”: 61.7<br /> }<br /> ]<br />}<br />C A L I B R A T I O N M A R K E R S<br />
    28. 28. Marker Store<br />
    29. 29. Marker Store Stack<br />Http<br />JbossNettyWebserver<br />JSON<br />Java<br />
    30. 30. Marker Store API<br />JSON<br />POST to /linkedhighway/create<br />GET from /linkedhighway/findactive/{fromId}<br />POST to /calibration/create<br />GET from /calibration/{markerId}<br />
    31. 31. Phoenix Data Store<br />Ergometer<br />Force Reading (Newtons)<br />{X1, Y1}<br />{Xn, Yn}<br />Distance Travel (Metres)<br />{X1, Y1}<br />{Xn, Yn}<br />
    32. 32. Phoenix Data Store<br />{ “_id”: “HandleForce”,<br /> “xDomainType”: “SINT64”,<br /> “xUnits”: “ms”,<br /> “yDomainType”: “DOUBLE”,<br /> “yUnits”: “N”<br />}<br />L A N E M E T A D A T A<br />
    33. 33. Phoenix Data Store<br />{ ”_id" : 1,<br /> ”y" : 0<br />}<br />{ “_id”: 2,<br /> “y”: 0.5<br />}<br />{ “_id”: 3,<br /> “y”: 0.707<br />}<br />{ “_id”: 4,<br /> “y”: 0.866<br />}<br />{ “_id”: 5,<br /> “y”: 1<br />}…<br />T I M E S E R I E S<br />
    34. 34. Phoenix Data Store<br />
    35. 35. Phoenix Data Store Stack<br />Tcp<br />JbossNetty Server – NIO<br />Google Protocol Buffers<br />Java<br />
    36. 36. Phoenix Data Store API<br />Commit<br />connect()<br />openLane(…)<br />persist(x, y)<br />disconnect()<br />Query<br />connect()<br />submitQuery(…) returns a Cursor<br />disconnect()<br />J A V A C L I E N T<br />
    37. 37. Performance<br />
    38. 38. Phoenix Local<br />AMD Athlon II 2.8GHz 64-bit<br />640GB 2.5” Hard Drive SATA 5400 rpm<br />2 X 2GB DDR3<br />Ubuntu 10.10<br />EXT4<br />
    39. 39. Issues Faced<br />No Journaling in Mongo 1.6 (Fixed in 1.8)<br />Compression<br />DB repair did not work once – unclean shutdown<br />Assertion failure magic == 0x41424344 db/pdfile.h<br />
    40. 40. Summary<br />MongoDB - Does what it says on the tin<br />
    41. 41. Thank You<br />Questions?<br />stephen.rose@mclaren.com<br />ashok.subramanian@thoughtworks.com<br />

    ×