Con8862 no sql, json and time series data

1,924 views
1,745 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,924
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
26
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Con8862 no sql, json and time series data

  1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.1
  2. 2. NoSQL , JSON and TimeSeries Data - CON8862 Anuj Sahni, Principal Product Manager Ashok Holla, Senior Sales Consultant
  3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3 Safe Harbor The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
  4. 4. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.4 Agenda  Why JSON ?  Its role in NoSQL and Big Data  NoSQL Use Cases  Oracle NoSQL DB overview  Architecture  Data Modeling using JSON schema  Time Series case study and demo
  5. 5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5 Brief History of data Interchange  10 years ago XML was primary data exchange format – Vast improvement over SGML (Standard Generalized Markup Language) – Enabled people to exchange documents across HTTP Language of Internet
  6. 6. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.6 Brief History of data Interchange (cont…)  Last few years, a bold transformation happening in world of data interchange – JSON (JavaScript Object Notation) emerged as an alternative to XML  looks more data-structure like,  Light weight/bandwidth-non-intensive,  language independent, Language of Internet
  7. 7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7 What made JSON Cool ?  API – No business value gain operating in silos – REST replacing SOAP as data transfer protocol  The Internet of Things – “JSON better adapted to devices with limited capabilities”  Full-stack JavaScript – JavaScript is new hot – Node.js gone mainstream  Big Data
  8. 8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8 Thoughts Things Processes Newer Challenges
  9. 9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9 Get Fast Answers to New Questions What is Big data ?
  10. 10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10 NoSQL and Big Data Where did it come from? SQL JDBC, ODBC General Purpose Managed Schemas Security, Backups Analytics … Distributed Processing Distributed, Replicated File System Driver Application NoSQL databases Flexible Schemas Sharded, Replicated Database High Speed, Simple Ops More Flexible Schema Management Globally Distributed, “Always On” data Competitive Advantages of “Fast Data” Lower TCO, commodity HW scale-out
  11. 11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11 Kinds of NoSQL Database Based on Storage Model Key Value Columnar Document Graph Oracle NoSQL DB Riak Dynamo Voldemort Cassandra HBase MongoDB CouchBase Neo4J GraphDB
  12. 12. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12 Few common characteristics  They all store data in de-normalized fashion – Don’t support PK-> FK relationship (no cascade deletes) – Don’t support complex Joins – Don’t manage constraints (application’s job)  Maintains data relationships using JSON/BSON representation – Simple lookup operations by primary keys are generally enough In NoSQL Databases
  13. 13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13 Agenda  Why JSON ?  Its role in NoSQL and Big Data  NoSQL Use Cases  Oracle NoSQL DB overview  Architecture  Data Modeling using JSON schema  Time Series case study and demo
  14. 14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14 PointFlexibility Point Web-Scale Transaction Processing High velocity, volume, variety, low information density data capture Web browsing, Shopping Carts, CDR processing, Sensor data, Stock data Web-Scale Personalization Guaranteed low latency lookups for end-customers Advertising, Product Recommendations, Online Catalogs, Social Media, Profile Management, Personalization Real-Time Event Processing Real time events trigger rule that perform low latency lookups Medical Monitoring, Factory Automation, Oil & Gas, Geo- location NoSQL Database Use Case Summary
  15. 15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15 Use Case – Online Social Gaming  Problem – Very low latency requirements – Player movement must feel like a real time operation, while being tracked on the server – Extreme data velocity – Popular games, large scale user base (Farmville boasts 80 million active users) – Highly available – 24/7 sites – Write heavy workloads  Solution – Where to use a NoSQL Database? – Player interaction data store – Database to track player movement and game interaction – Game play statistics – Per player usage statistics – Persistent chat store – For games that allow player communication via chat, the NoSQL database is used as a persistent message store (auditing and COPA compliance) Web Scale Personalization
  16. 16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16 Use Case – Online Social Gaming Web Scale Personalization JSON maps well with HTML5/JS based Client Player interaction stored as simple JSON events Easy to store entire player state reliably NoSQLDBDriver Application Shard 2 Shard N Shard 1 Schema Evolution for agile development Transparent load balancing On-demand cluster expansion REST
  17. 17. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.17 Use Case – Smart Meter Analytics  Problem – Real Time Access to time-series data – identity patterns hidden within the terabytes of data  Identify theft,  predict system failures ,  better manage operations – Large volume of machine data – 1000 fold increase in data collection  Write heavy workloads – Highly available – 24/7 sites  Solution – Where to use a NoSQL Database? – Fast time based queries – Daily, weekly, yearly consumption trends. – Flexible data model – can adapt to various kind of sensors or changes in sensor format – Scalable and Reliable performance – Easy to scale as workload increases because of more sensors Web Scale Transactions
  18. 18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18 Smart Meter Analytics Web Scale Transactions Analytic Engine User Interface Export back to Smart Grid NoSQLDBDriver Application Shard 2 Shard N Shard 1 Time Series Data Data coming in many different format Data coming fast Horizontally scalable database JSON Easy to model keys for time bases queries
  19. 19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19 Agenda  Why JSON ?  Its role in NoSQL and Big Data  Introduction to NoSQL  Oracle NoSQL DB overview  Architecture  Data Modeling using JSON schema  Time Series case study and demo
  20. 20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20  Flexible Key-Value Data Model  ACID transactions  Horizontally Scalable  Highly Available  Elastic Configuration  Simple administration  Intelligent Driver  Commercial grade software and support Features Oracle NoSQL Database Application Storage Nodes Datacenter B Storage Nodes Datacenter A Application NoSQL DB Driver Application NoSQL DB Driver Application Java SE 6 (JDK 1.6.0 u25)+; Solaris or Linux Scalable, Highly Available, Key-Value Database
  21. 21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21 Simple Data Model  Simple, flexible key-value paradigm  Simple operations – read/insert/update/delete, RMW support  Ordered scan of key ranges  Unordered scan of all data (non-transactional)  Streaming API for LOBs  Java and C APIs  Key-value pairs
  22. 22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22  Compact, highly efficient serialization  Synergy with Hadoop  Supports serialization from and to JSON strings  Bindings from serialized formats to language constructs  Easy to use mechanism for schema evolution  Schema definition tracked with schema ID of data writer  Schema versions can be opaque to readers Oracle NoSQL – Why AVRO?
  23. 23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23 JSON support in Oracle NoSQL DB Schema Creation
  24. 24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24 JSON support in Oracle NoSQL DB (cont…) Data Manipulation
  25. 25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25 JSON support in Oracle NoSQL DB (cont…) Schema Evolution
  26. 26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26 JSON support in Oracle NoSQL DB (cont…) Agile development
  27. 27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27 Time Series Analysis using Oracle NoSQL database
  28. 28. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.28 Big Data Challenge  High velocity of stock tick data generated at a massive volume each day  Millions of customers trading stocks on- line based on the current value and the historical trends.  How do you ensure these buy/sell transactions happens in real time?  How do you store large volume of tick data in a quick and consistent manner? – So you can run the trend analysis on the time series data. UC - Stock Tick Analysis
  29. 29. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.29 Introduction – Oracle Investments Inc.  Goal – Ensure fast and consistent storage of large volume of tick data – Deliver a platform where customers can:  Analyze historical trends by plotting time-series data  Trade stocks in real time by placing buy/sell orders  Challenge – 100k’s customers trading concurrently. – 10k’s of ticks each second – Real time response required for order execution
  30. 30. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.30 Introduction – Oracle Investments Inc.  Value – Real time:  Data capture and random access to time sensitive data  Order processing – Automatic data partitioning for easy scalability – Lowest $/ops for a consistent storage – Highly Available system for 24/7 operation
  31. 31. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.31 Big Data Appliance Oracle NoSQL DB Oracle NoSQL DB as Data Store NoSQL Driver Read, Update  Stores all key interactions required to drive application. For example:  User Profile  Stock Tick Data  Transaction information  Why Oracle NoSQL Database?  Highly Scalable  Extremely performant  Super low-latency
  32. 32. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.32 Architecture Business Layer Big Data Appliance Oracle NoSQL DB NoSQL Driver Stock Tick Generator Tick data every 5 seconds Trading Read, Update Analytics Time Series Analysis Order requests User Profile Updates Order processing Historical Data
  33. 33. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.33 NoSQL Data Representation {"lasttimestamp":"1367251181219“,"lastclose":"45.17","lasthigh":"32.25","lastlow":"31.94","lastopen":"32.21","lastvolume":"26" , "stockArray": [ {"timestamp":"1367245601219“, "value":"32.21“, "open":"32.23“, "high":"32.27“, "low":"32.14“,"volume":"4700"} {"timestamp":"1367245602220“, "value":"32.15“, "open":"32.14“, "high":"32.27“, "low":"32.09“,"volume":"4000"} {"timestamp":"1367245603319“, "value":"32.11“, "open":"32.09“, "high":"32.27“, "low":"32.14“,"volume":"3500“} ] } Denormalized Data
  34. 34. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.34 NoSQL Data Representation  Get Stock Information for a Company – Key = /SYMBOL/YYYY-MM/-/DD – (Ex: /ORCL/2013-05/-/02 , Stock data for 2nd of May 13)  Write = put(key, value)  Read = get(key)
  35. 35. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.35 Demo
  36. 36. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.36 Oracle NoSQL DB Resources  Support via OTN forums and Oracle Support process  OTN Forum: – Forum Home » Big Data » NoSQL Database – forums.oracle.com/forums/forum.jspa?forumID=1388  Oracle.com: – www.oracle.com/us/products/database/nosql/overview/index.html  OTN (including documentation and download): – www.oracle.com/technetwork/products/nosqldb/overview/index.html Support
  37. 37. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.37 Oracle NoSQL DB Resources  On OTN and in download – docs.oracle.com/cd/NOSQL/html/index.html  Getting Started Guides  Programmatic API  Installation & Release Notes  FAQ Documentation
  38. 38. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.38 References  Why JSON will continue to push XML out of the picture  Interactive Matter Lab  Architecting the Internet of Things(p. 102)
  39. 39. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.39
  40. 40. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40 Appendix
  41. 41. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.41  Configurable Durability per operation  Configurable Consistency per operation  ACID by default  Transaction scope is single API call  Records share same major key  Multiple operations supported Greater Flexibility Features – Configurable ACID Transactions
  42. 42. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.42 Developing Applications Data Modeling Subject IDQualifier – Value Type Range Binary JSON/Avro RDF Triples Tables/Rows Queries sensors by section Pressure Sensor Sensor IDLeft Front What a List The Value Value Types = A specific sensor All measures of a sensor A range of sensor measures Major Key // / / / / Minor Key CT 123-PS3234LF – psi, irate Timestamp Array of Int=/ / / / / /
  43. 43. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.43  Increase Data Capacity – Add more storage nodes – New shards automatically created  Increase Data Throughput – More shards = better write throughput – More replicas/shard = better read throughput On Demand Elasticity NoSQL DB Driver Application Master Replica Replica StorageNode StorageNode StorageNode Shard-1 Master Replica Replica Shard-2 On-Demand Cluster Expansion
  44. 44. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.44  Supports heterogeneous storage topology  Replicas move from over-utilized to under-utilized storage nodes  Number of shards and replication factor remain unchanged Improve Performance Rebalance an Unbalanced Store Storage Node 1 Storage Node 2 Storage Node 3 Represents a data fragment
  45. 45. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.45 What’s Really Important? Technical Feature Importance Why Storage Model Not really Will merge over time Specific Features Somewhat Application requirements? Performance Somewhat Rapid changes, YMWV Integration Critical Long term, Repetitive cost Reliability/Support Critical Early products, Product direction Predictability Critical Production reqs & SLAs
  46. 46. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.46  Query NoSQL data from Oracle Database  Access NoSQL data from Hadoop for DW and analytics  Share data with Coherence for extensible in-memory cache grid  Persist history & event streams for processing with OEP  Store & query RDF data using Oracle RDF for NoSQL Integration Oracle NoSQL Database: Integrated out of the box
  47. 47. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.47 Reliability & Support  Decades of widespread, reliable deployment experience  15+ years of mission-critical non-relational database technology  Oracle Support available for both Enterprise and Community Edition Oracle NoSQL Database: Enterprise-Grade Software & Support
  48. 48. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.48  Automatic election of new Master  Rejoining nodes automatically synchronize with the Master  Isolated nodes can still service reads  All nodes are symmetric Automatic Failover Highly Reliable Replication factor = 5 Rep Node Master Rep Node Replica Rep Node Replica Rep Node Replica Rep Node Replica New Master

×