Solution Use Case Demo: The Power of Relationships in Your Big Data

2,225 views

Published on

In this security solution demo, we have integrated Oracle NoSQL DB with InfiniteGraph to demonstrate the power of using the right tools for the solution. By integrating the key value technology of Oracle with the InfiniteGraph distributed graph database, we are able to create new views of existing Call Detail Record (CDR) details to enable discovery of connections, paths and behaviors that may otherwise be missed.

Discover how to add value to your existing Big Data to increase revenues and performance!

Published in: Technology, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,225
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Solution Use Case Demo: The Power of Relationships in Your Big Data

  1. 1. 1 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  2. 2. NoSQL Databases Robert Greene, Anuj Sahni, Product Management
  3. 3. The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 3 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  4. 4. NoSQL – in the eyes of the Oracle Fundamentals Key-Value Graph 4 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  5. 5. NoSQL – in the eyes of the Oracle Design drivers Systems Friendly Usability Scalability Predictably Responsive 5 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  6. 6. Systems Friendly Useful Services Eventing Cache  Data Collaborations – Orchestrating NoSQL – Monitoring – Process and/or Data movement 6 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. HDFS RDBMS
  7. 7. NoSQL DB and Event Processing Features  OEP for event-driven & streaming applications  Oracle NoSQL DB accessed from KV Cartridge  NoSQL DB data directly accessible via CQL queries 7 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  8. 8. 8 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  9. 9. 9 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  10. 10. Integration with RDF RDF Use Cases Semantic Metadata Layer Text Mining & Entity Analytics • Unified content metadata for federated resources • Validate semantic and structural consistency  Find related content & relations by navigating connected entities  “Reason” across entities Social Media Analysis 10  Analyze social relations using curated metadata - Blogs, wikis, video - Calendars, IM, voice Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  11. 11. Commodity Servers Processing Application NoSQL DB Driver Real Time Access Big Data Acquisition Servers 11 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Data Connector
  12. 12. Engineered Systems Processing Application NoSQL DB Driver Real Time Access Big Data Acquisition Batch Processing HDFS, Hadoop, CDH Map Reduce ORCH ‐ Stats Map Reduce Pig ‐ Sessionize Map Reduce Hive ‐ Activities Big Data Appliance 12 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Data Connector
  13. 13. Usability– Major/Minor KV Paradigm User mailbox data Key Space : Value : { /user/id/-/folder/inbox/arrival date /user/id/-/folder/deleted/arrival date “name" : "Email", "namespace" : "com.companyX.email.avro", "type" : "record", "fields": [ {"name": "from", "type": "string", "default": ""}, {"name": "to", "type": "string", "default": ""}, {"name": "sender", "type": "string", "default": ""}, {"name": "cc", "type": "string", "default": ""}, {"name": "subject", "type": "string", "default": ""}, {“name”: “msgBody”, “type”: “string”, “default”: “”} 13 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. ]}
  14. 14. Dev Usability– Modeled as Nested Tables User mailbox data Parent Table Name: User Primary Key Major Key UserID Major Key Inherited from parent table Child Table Name: Folder Primary Key UserID 14 Folder Name Copyright © 2013, Oracle and/or its affiliates. All rights reserved. “Value” Arrival Date From To Sender CC Subject Msg Body
  15. 15. Admin Usability - Enterprise Manager Integration Standard Protocol Support  SNMP (Enterprise Edition Only) – – – Storage node SNMP agent – admin and rep nodes Performance metrics – Via polling – Service change events – Via traps or polling –  MIBs in SNMPv2 format Notifications (Traps) – Latency and throughput limits OEM Integrated for holistic RDBMS / NoSQL Monitoring 15 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  16. 16. Admin Usability – Online Rolling Upgrades Ever tried to upgrade a 200 node system while it’s active? Online Rolling Upgrade  We did do it!  Admin commands available to describe safe upgrade order  Scripted available hands-free upgrade experience  Read/Write availability throughout the upgrade process Time to Upgrade (min) What’s the Big Deal 17.5 15 12.5 10 7.5 5 2.5 0 72 (24x3) 144 (48x3) 216 (72x3) Total Nodes (Shards x Rep. Factor) 16 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  17. 17. Scalability Architecture – Applications View Application  Elastic Shards NoSQL Driver (split, add, contract) Partition Partition Partition M M M R R R R R R  Writes to elected node Store 17 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.  Reads from any node in system
  18. 18. Scalability– Elastic Cluster Expansion Application Smart Topology  Increase Data Capacity – Add more storage nodes – New shards automatically created  Increase Data Throughput – More shards = better write throughput – More replicas/shard = better read NoSQL DB Driver Master Master Replica Replica Replica Replica Shard-1 Shard-2 throughput StorageNode 18 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. StorageNode StorageNode
  19. 19. Scalability - Rebalance an Unbalanced Store Storage Node 1 Storage Node 2 Storage Node 3 Improve Performance  Supports heterogeneous storage topology  Replicas move from overutilized to under-utilized storage nodes  Number of shards and replication factor remain unchanged Represents a vnode 19 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  20. 20. YCSB – Benchmark Commodity Servers What’s the Big Deal  1.6 billion records  94K insert/sec  25K read/update/sec  Low latency  Linear scalability 20 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  21. 21. YCSB - Benchmarking with SSD’s • Twitter sees ~500M tweets/day • • This is 350K a minute Capture twitter activity with 3 commodity servers • 1.25M ops/sec 2 billion records • 2 TB of data • 95% read, 5% update • Low latency, High Scalability • 1,400,000 4 1,200,000 1,000,000 3 800,000 2 600,000 400,000 1 200,000 0 0 6 (2x3) 12 (4x3) 24 (8x3) 30 (10x3) Cluster Size ( 3->15 servers ) Throughput (ops/sec) Read Latency (ms) 21 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Write Latency (ms) Average Latency (ms) What’s the Big Deal Throughput (ops/sec) Mixed Throughput
  22. 22. YCSB – Benchmarking Predictability Oracle NoSQL Database: Designed for Predictability 95/5 Read/Update Throughput 70,000 7 60,000 6 50,000 5 40,000 4 30,000 3 20,000 2 10,000 1 0 144 (48x3) 20% 40% 60% 80% 216 (72x3) 216* (72x3) Nodes (Shards x RF) Throughput (ops/sec) 22 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Read Latency (ms) Update Latency (ms) Average Latency (ms) 8 0 Throughput (ops/sec) 80,000
  23. 23. Oracle NoSQL Database Scalable, Highly Available, Key-Value Database Application Application NoSQL DB Driver Features Application Application NoSQL DB Driver  Simple Key-Value Data Model  Horizontally Scalable  Highly Available  Elastic Configuration  ACID Transactions  Simple administration  Transparent load balancing  Commercial grade software and support  Secondary Indexing  Authentication / SSL Security  Data Center support Storage Nodes 23 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Storage Nodes Datacenter A Datacenter B
  24. 24. Oracle NoSQL Database Subscription Model New business-friendly support service  Oracle NoSQL Database Community Edition – Open Source AGPL Edition  Support is now available for Community Edition – Price is $2,000/year per server – No upfront license fee – Provides for full Oracle support policy response – Purchase online via the Oracle Store  Offers affordable support option for startups  Provides Oracle expertise for production deployment 24 Copyright © 2013, Oracle and/or its affiliates. All rights reserved. The Store https://shop.oracle.com/
  25. 25. Oracle NoSQL DB Licensing Community -or- Enterprise Edition  Enterprise Edition – Closed Source. Standard Oracle License.  Community Edition has all of the basic functionality and APIs. Gets you started.  Enterprise Edition for large, production, multi-data center, Oracle integrationcentric customers and/or non-GPL compliant customers. 25 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  26. 26. Oracle NoSQL DB Resources  NoSQL DB Use Cases, White Papers, Data Sheets, Benchmarks http://www.oracle.com/technetwork/products/nosqldb/overview/index.html  NoSQL DB Documentation http://www.oracle.com/technetwork/products/nosqldb/documentation/index.html  NoSQL DB Downloads http://www.oracle.com/technetwork/products/nosqldb/downloads/index.html  NoSQL DB OTN Forum http://forums.oracle.com/forums/forum.jspa?forumID=1388  OU Training Classes http://bit.ly/V5qbmY  Great Application Development article on OTN: http://www.oracle.com/technetwork/articles/bigdata/oracle-nosqldb-appdev-1891870.html 26 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  27. 27. 27 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.
  28. 28. 28 Copyright © 2013, Oracle and/or its affiliates. All rights reserved.

×