• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Project Phoenix - MongoUK 2011
 

Project Phoenix - MongoUK 2011

on

  • 2,851 views

A general purpose system for storing and analysing real-time time-series data

A general purpose system for storing and analysing real-time time-series data

Statistics

Views

Total Views
2,851
Views on SlideShare
2,849
Embed Views
2

Actions

Likes
2
Downloads
1
Comments
0

2 Embeds 2

https://si0.twimg.com 1
https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • generic
  • Increased connectivityCheaper, smaller sensorsNew computing techniques
  • Each sensor manufacturer creates a custom tool to view data from that sensor
  • Relative performance of the different types of data stores with BerkeleyDB as the baseline to benchmark against

Project Phoenix - MongoUK 2011 Project Phoenix - MongoUK 2011 Presentation Transcript

  • Project Phoenix
    Stephen Rose
    Ashok Subramanian
    MongoUK
    March 21, 2011
  • Background
    Architecture
    Data-store Selection
    Implementation
  • What is Phoenix?
    “a platform that allows data to be transferred seamlessly to the people who want it, presented in the way they want to see it, regardless to the source, type or location of the inputs”
  • What sort of data?
    Time-series
    Sensor readings
    External data feeds
    Meta data
    Adding context
    Credits: http://commons.wikimedia.org/wiki/File:CPI-plot.png
  • What is Phoenix?
    “a web basedplatform capable of ingesting, combining, processing and distributing data - in real time”
  • Target Markets
    Smaller, cheaper, better connected
    Credits: http://www.flickr.com/photos/opensourceway/4370251113/
    Credits: http://www.flickr.com/photos/juggerphotos/4562680976/in/photostream/
  • Why develop a new platform?
    No existing platforms for the use cases
    Typical solutions use Excel, SQL
    cannot cope with amount of data
    no real-time capability
  • Real-time Analysis
    COMPLEX EVENT PROCESSING
    STREAM
    COMPUTING
    Reporting
    Processing
    Big Data
    Target Customer
    BUSINESS INTELLIGENCE
    BATCHPROCESSING
    Batch Analysis
  • Real-time Analysis
    COMPLEX EVENT PROCESSING
    STREAM
    COMPUTING
    PHOENIXPLATFORM
    Reporting
    Processing
    Big Data
    Target Customer
    BUSINESS INTELLIGENCE
    BATCHPROCESSING
    Batch Analysis
  • Project Timelines
    2010
    Research
    Market
    Customers
    Design
    Architecture
    Data model
    Development
    4 developers
    Rapid!
    Release
    Version 1
    Phoenix local
    Q1 Q2 Q3 Q4
  • Background
    Architecture
    Data-store Selection
    Implementation
  • Data Model
    Credits: http://www.flickr.com/photos/nazly/5502884394/sizes/z/
  • Data Highways and Lanes
    Credits: http://www.flickr.com/photos/mhzmaster/965947834/sizes/z/
  • LATITUDE
    MY CAR
    LONGITUDE
    MPG
    MPG
    GFORCES
    GFORCES
    PASSENGER HEART RATE
    LATITUDE
    MY WIFE’S CAR
    LONGITUDE
    MPG
    MPG
    TEMPERATURE
    TEMPERATURE
  • Markers
    Credits: http://www.flickr.com/photos/rebelxt/5284536592/sizes/z/in/photostream/
  • STEVE’S WIFE DRIVING
    STEVE DRIVING
    LATITUDE
    MY CAR
    LONGITUDE
    MPG
    MPG
    G FORCE
    G FORCE
  • Architecture
    Phoenix Customer
    APPLICATION
    STREAM PROCESSOR
    ALGORITHMS
    DATASTORE
    DATA FEEDS
  • Offline and Cloud
    Credits: http://www.flickr.com/photos/noii/2327713880/
  • Background
    Architecture
    Data Store Selection
    Implementation
  • Selection Criteria
    Local and Cloud
    Use cases – time-series and marker
    Performance
    Query
    Community
    Maintenance
    Deployment
  • 3 different categories
  • Data Stores Evaluated
  • Background
    Architecture
    Data Store Selection
    Implementation
  • Data Stores
    Phoenix Customer
    APPLICATION
    STREAM PROCESSOR
    ALGORITHMS
    DATASTORE
    DATA FEEDS
    TIME SERIES
    MARKERS
  • Marker Store
    T I M E
    FROM
    FROM
    TO
    L I N K E D H I G H W A Y S
  • Marker Store
    { "fromId" : ”Athlete1",
    "toId" : ”Ergometer",
    "startTime" : 1,
    "endTime" : 50
    }
    { "fromId" : ”Athlete2",
    "toId" : ”Ergometer",
    "startTime" : 60,
    "endTime" : null
    }
    L I N K E D H I G H W A Y S
  • Marker Store
    { ”deviceId" : ”1",
    ”calibrationType" : ”sensor1",
    ”validFrom" : 1,
    ”validTo" : null
    “calibrations" :
    [
    {”expected”: 10.0,
    “actual”: 12.9
    },
    {”expected”: 50.0,
    “actual”: 61.7
    }
    ]
    }
    C A L I B R A T I O N M A R K E R S
  • Marker Store
  • Marker Store Stack
    Http
    JbossNettyWebserver
    JSON
    Java
  • Marker Store API
    JSON
    POST to /linkedhighway/create
    GET from /linkedhighway/findactive/{fromId}
    POST to /calibration/create
    GET from /calibration/{markerId}
  • Phoenix Data Store
    Ergometer
    Force Reading (Newtons)
    {X1, Y1}
    {Xn, Yn}
    Distance Travel (Metres)
    {X1, Y1}
    {Xn, Yn}
  • Phoenix Data Store
    { “_id”: “HandleForce”,
    “xDomainType”: “SINT64”,
    “xUnits”: “ms”,
    “yDomainType”: “DOUBLE”,
    “yUnits”: “N”
    }
    L A N E M E T A D A T A
  • Phoenix Data Store
    { ”_id" : 1,
    ”y" : 0
    }
    { “_id”: 2,
    “y”: 0.5
    }
    { “_id”: 3,
    “y”: 0.707
    }
    { “_id”: 4,
    “y”: 0.866
    }
    { “_id”: 5,
    “y”: 1
    }…
    T I M E S E R I E S
  • Phoenix Data Store
  • Phoenix Data Store Stack
    Tcp
    JbossNetty Server – NIO
    Google Protocol Buffers
    Java
  • Phoenix Data Store API
    Commit
    connect()
    openLane(…)
    persist(x, y)
    disconnect()
    Query
    connect()
    submitQuery(…) returns a Cursor
    disconnect()
    J A V A C L I E N T
  • Performance
  • Phoenix Local
    AMD Athlon II 2.8GHz 64-bit
    640GB 2.5” Hard Drive SATA 5400 rpm
    2 X 2GB DDR3
    Ubuntu 10.10
    EXT4
  • Issues Faced
    No Journaling in Mongo 1.6 (Fixed in 1.8)
    Compression
    DB repair did not work once – unclean shutdown
    Assertion failure magic == 0x41424344 db/pdfile.h
  • Summary
    MongoDB - Does what it says on the tin
  • Thank You
    Questions?
    stephen.rose@mclaren.com
    ashok.subramanian@thoughtworks.com