Informix NoSQL & Hybrid SQL detailed deep dive

2. Speaker Introduction John F. Miller III Lead Architect, STSM Keshava Murthy NoSQL & SQL Architect, STSM Jef Treece Product Manager

3. Overview Business phenomenon of NoSQL − Value: Open Source vs Enterprise Class What is NoSQL, JSON and BSON? − Quick overview − IBM/Mongo Partnership New Informix NoSQL/JSON Capabilities − Solution Highlights Mongo Compatibility − The Power of a Hybrid Solution − Analytics Demo Deep Dive − − 3 Putting together an application Seeing it live NewSQL

5. Informix NoSQL Capabilities Key-value Stores – Hashed Key value model – Capable of massive data storage Column Family Stores – Keys that point to multiple columns and column families Document Data Base – JSON / BSON Formatted documents – Collections of Documents with Key-value stored versions Graph Databases – – – – Nodes / relationship between nodes Properties of nodes Data values Supports SPARQL ? © 2013 IBM Corporation

6. Business Value of Informix NoSQL Increases business opportunity through capture and analysis of “interactive / interaction data” for a full understanding of customers and markets Lowers business barriers and reduces IT costs through Rapid Application Deployment Increases business ability to respond to volume changes through Scalable / extensible architecture Lowers IT overhead through use of Low Cost Commodity Hardware Changes business view to “what is happening” versus “what happened” © 2013 IBM Corporation

7. Use Cases for Informix NoSQL Session Store – High volume ingestion / low latency access requirements – Linkage to a specific user provides immediate recognition of customer preferences – Session restore after “break” is immediate to last functions performed increase customer confidence and connection to their desire(s) User Profile Store – Customer Profiles, orders and shipment status is immediate and searchable – Fast access for Authentication and preferences – Historical access and click streams increase personalization & targeting Content and Metadata Store – Document database for digital content, articles, instruction manuals – Key-value linkages provide linkage to customer profiles and history – Multiple image/data type support provide fast access to different data type © 2013 IBM Corporation

8. Use Case for Informix NoSQL (cont) Mobile Apps – Ability to store content and user data in schema-less formats increase development and deployment speed and changes to existing apps – Supports multi-format data storage associated with differing device types – Scalable storage provides for document, image, text oriented storage and retrieval of apps data / information Third Party Aggregation – High speed ingestion of data feed from multiple sources: • Retail store loyalty programs, social media, marketing information, purchase histories by groups, person, industries – Ease of format management and analysis High Availability Cache – – – – Storage for popular / frequent search results Session information High Frequency Ads tailored for User Profiles, Locations, searches Dual function cache for data store and fast response time for retrieval © 2013 IBM Corporation

9. Informix NoSQL Use Case (cont) Globally Distributed Data Repository – Scalable nodes and access across distributed systems – Location affinity for workload optimization – Capture of differing formats and languages to provide global and discrete views of the business and analytics of searches, purchases, etc. eCommerce – Ability to handle purchase spikes / peak time through scalable system architecture and low cost commodity hardware – Fast / rapid application deployment through schema-less development model – Ability to combine transaction data with user action reduces cost of upgrading business apps for transaction management Social Media & Gaming – Rapid app development , deployment, and change implementation increase ability to grow customer base and respond to trends – or create them – Fast access to user profile for authentication and preferences / historical information – Real time Analytics and trend identification © 2013 IBM Corporation

10. Informix NoSQL Use Cases (cont) Ad Targeting – Fast access to user profile and histories permit well managed Ad placement increasing revenue and buy decision – Click history and Pyschographic based suggestions – Ability to ingest other feeds and quickly relate to a specific customer • Loyalty programs • Social Media • Associations (LinkedIn, Dating sites, Industry Groups etc.) © 2013 IBM Corporation

11. Informix and MongoDB Have Free Editions Editions MongoDB Free Developer Innovator-C Basic For Purchase 11 Informix Express, Workgroup, Advanced Workgroup, Enterprise, Advanced Enterprise Standard, Enterprise

12. MongoDB Subscriptions Basic Standard Enterprise Edition MongoDB MongoDB MongoDB Enterprise Support 9am-9pm local, M-F 24x7x365 24x7x365 License AGPL Commercial Commercial Emergency Patches Not Included Included Included Price $2,500 / Server / Year $5,000 / Server / Year $7,500 / Server / Year Additional monthly charges for backup services. Subscription information obtained from 10Gen site, June 26, 2013. 12

13. Price Point Comparison Estimate, 3-year cost Dual Core Intel Nehalem Innovator-C Express Workgroup (4 core, 8 GB, 2 nodes) (16 core, 16 GB, unlimited nodes) Product Cost $0 $8,540 $19,740 $1,680 Included Included Support Renewal Year 2 $1,680 $1,708 $3,948 Support Renewal Year 3 $1,680 $1,708 $3,948 Total $5,040 $11,956 $27,636 Support Subscription Year 1 24 x 7 x 365 Production System Down Development Call Emergency Patches Free Upgrades MongoDB Enterprise, 3-year cost: $22,500 Subscription information obtained from 10Gen site, June 26, 2013. Retail U.S. prices subject to change, valid as of June 26, 2013. 13

15. Informix NoSQL Key Messages Mobile computing represents one of the greatest opportunities for organizations to expand their business, requiring mobile apps to have access to all relevant data DB2 and Informix have embraced the JSON standard and MongoDB API to give mobile app developers the ability to combine critical data managed by DB2 or Informix with the next generation mobile apps DB2 JSON and Informix JSON let developers leverage the abilities of relational DBMS together with document store systems 15 IBM Confidential © 2013 IBM Corporation

16. DB2/Informix JSON for Install Base Clients Leverage all relevant enterprise data for your mobile app development The Offer Client Value Any for-purchase edition of One solution for relational and non-relational data • Latest version of DB2 LUW 10.5 • Latest version of Informix 12.1 • Relational tables and NoSQL collections co-exist in the same database Includes modern interface with JSON /BSON support • Allows joins between NoSQL and Relational tables • Lets DB2/Informix be compatible with existing MongoDB apps with minimal/no application changes • Ability to handle a mix of structured and unstructured data • Joins utilize indexes on both relational and NoSQL Brings Enterprise level performance, reliability, and security to mobile apps • Includes ACID compliance Includes built-in online backup and restore capabilities • Flexible schema allows rapid delivery of apps • Compatible with MongoDB programming interfaces Automatic Document Compression (only if they buy Advanced Enterprise or add-on compression with Enterprise Edition?) 16 IBM Confidential © 2013 IBM Corporation

17. DB2 JSON Go-to-Market Timeline Phase 1 – June through August 15 Phase 2 – eGA – Q4 Timing Target Phase 3 - 2014 Jan 1, 2014 – TBD Targeted clients/partners Opportunistic white space Install Base – Top xxxx accounts Key partners TBD Opportunistic white space DB2 Trial code with Tech Preview Content Tech preview on dW A series of how-to articles on dW: Part 1, Part 2, Part 3, Part 4. Video: DB2 JSON Overview Video: Getting started Update NoSQL product page with JSON content Re-promote DB2 Trial code IBM Champions for blogs dW DM for JSON Community covreing both DB2 and Informix Who Deliverables Events 17 MongoDB World Tech Bits and Tech Talks Developer Days DB2 Night Show via BP Leverage existing 10Gen event calendar to insert content (these are potential – not confirmed) IBM Confidential On-going Connect with AIM, Rational for joint presence at 3rd conferences and events. © 2013 IBM Corporation

18. Informix JSON Go-to-Market Timeline Phase 1 - EVP Phase 2 – eGA – Q4 Phase 3 - 2014 Timing Aug. 5, 2013 – Sep. 13, 2013 Sep. 13, 2013 – Dec. 31, 2013 Jan 1, 2014 – TBD Target • • • • • Who Confirmed EVP participants Begooden BP Moukouri BP Consult-IX BP ADT BP Openbet Cust Technicolor White space cust Targeted clients/partners Opportunistic white space Install Base – Top xxxx accounts Key partners TBD Opportunistic white space Top xxxx accounts Key partners TBD (criteria to be determined for each) TBD Potential EVP participants Nuvento White space cust SRCE BP Cisco OEM cust Zebra Fish BP Deliverables Events 18 NoSQL Client Business Value Deck NoSQLTechnical White-paper 12.10.xC2 Release White Paper Developer Works article on NoSQL Client Updated NoSQL demo • • Content on Informix marketing page with re-direct to DevWorks page Promote Informix trial code? Chat with the Labs • • IOD sessions IOD Lab IBM Confidential Connect with AIM, Rational for joint presence at 3rd conferences and events. © 2013 IBM Corporation

19. IOD 2013 Roadmap: Extend Database Application Versatility with NoSQL Session No. Session Title IPT-2059 JSON Document Support in IBM DB2 Schema Flexibility in a Trusted Database Environment IPT-2909 Embracing NoSQL: IBM and the Community IPT-3149 Big Data, Hadoop, and NoSQL: A Crash Course for a Fearless IBM DB2 Practitioner IDB-1943 Firefighter Safety Enhancement through NoSQL Information Integration Architectures with IBM DB2 Graph Store IDB-2075 Competitive Product Delivery with IBM DB2 JSON IDB-1878 Agile Product Development Using IBM DB2 with JSON IDX-3691 Get the Best of Both Worlds: Bring NoSQL to Your SQL Database IDX-3722 Balancing Big Data Workloads with NoSQL Auto-Sharding IDX-2551 "Developing ""Hybrid"" NoSQL/SQL Applications using JSON/BSON with Informix" IPT-3070 Extending Information Management Solutions with NoSQL Capabilities--XML, RDF and JSON IDZ-2599 NoSQL and DB2 for z/OS? Receiving Agility from a Trusted Enterprise Database © 2013 IBM Corporation

21. Some Typical NoSQL Use Cases - Mostly Interactive Web/Mobile Online/Mobile Gaming − − − − − Leaderboard (high score table) management Dynamic placement of visual elements Game object management Persisting game/user state information Persisting user generated data (e.g. drawings) Display Advertising on Web Sites − − Ad Serving: match content with profile and present Real-time bidding: match cookie profile with ad inventory, obtain bids, and present ad Communications − Device provisioning Social Networking/Online Communities 21 E-commerce/Social Commerce – Storing frequently changing product catalogs Logging/message passing – Drop Copy service in Financial Services (streaming copies of trade execution messages into (for example) a risk or back office system) Dynamic Content Management and Publishing (News & Media) – Store content from distributed authors, with fast retrieval and placement – Manage changing layouts and user generated content

23. NOSQL, JSON AND BSON OVERVIEW Technical Opportunities/ Motivation What are NoSQL Databases? Quick overview of JSON What is sharding?

24. New Era in Application Requirements Store data from web/mobile application in their native form − New web applications use JSON for storing and exchanging information − Very lightweight – write more efficient applications − It is also the preferred data format for mobile application back-ends Move from development to production in no time! − Ability to create and deploy flexible JSON schema − Gives power to application developers by reducing dependency on IT − Ideal for agile, rapid development and continuous integration

25. Why NoSQL? Non-traditional data management requirements driven by Web 2.0 companies − Document stores, key-value stores, graph and columnar dbms The Three Vs: − Velocity – high frequency of data arrivals − Volume – BigData − Variability – unstructured data, flexibility in schema design New data interchange formats – like JSON (JavaScript Object Notation) and BSON (Binary JSON) Scale-out requirements across heterogeneous environment – Cloud computing 25

26. What is a NoSQL Database? Not Only SQL or NOt allowing SQL A non-relational database management systems − Does not require a fixed schema − Avoids join operations − Scales horizontally − No ACID (eventually consistent) Good with distributing data and fast application development Provides a mechanism for storage and retrieval of data while providing horizontal scaling.

27. IBM Use Case Characteristics for JSON

28. Why Most Commercial Relational Databases cannot meet these Requirements

29. NoSQL Database Philosophy Differences No ACID – No ACID (Atomicity, Consistency, Isolation, Durability) – An eventual consistence model No Joins – Generally single row/document lookups Flexible Schema – Rigid format 29 © 2013 IBM Corporation

30. Partnership with IBM and MongoDB MongoDB and IBM announced a partnership in June 2013 There are many common use cases of interest addressed by the partnership − − − − Accessing JSON Data in DB2, Informix MongoDB using JSON query Schema-less JSON Data for variety of applications Making JSON data available for variety of applications Securing JSON Data IBM and MongoDB are collaborating in 3 areas: − − − Open Governance: Standards and Open Source Technology areas of mutual interest Products and solutions

31. Basic Translation Terms/Concepts Mongo/NoSQL Terms Traditional SQL Terms Database Database Collection Table Document Row Field Column Collection Table Name John 31 Value Document 28 Scott Key 21 Tim {"name":"John","age":21} {"name":"Tim","age":28} {"name":"Scott","age":30} Age 30 Row

32. JSON Details JSON Syntax Rules – – – – – JSON syntax is a subset of the JavaScript object notation syntax: Data is in name/value pairs Data is separated by commas Curly braces hold objects Square brackets hold arrays JSON Name/Value Pairs – JSON data is written as key/value pairs. – A key/value pair consists of a field name (in double quotes), followed by a colon, followed by a value: "name":"John Miller" The 6 types of JSON Values: – – – – – – 32 A number (integer or floating point) A string (in double quotes) A Boolean (true or false) An array (in square brackets) An object (in curly brackets) Null

33. Example of Supported JSON Types Example of each JSON type Mongo specific JSON types in blue – date { "string":"John", "number":123.45, "boolean":true, "array":[ "a", "b", "c" ], "object: { "str":"Miller", "num":711 }, "value": NULL, "date": ISODate("2013-10-01T00:33:14.000Z") }

34. JSON: JavaScript Object Notation What is JSON? − JSON is lightweight text-data interchange format − JSON is language independent − JSON is "self-describing" and easy to understand JSON is syntax for storing and exchanging text information much like XML. However, JSON is smaller than XML, and faster and easier to parse. { "name":"John Miller", "age":21, "count":27, "employees": [ { "f_name":"John" , "l_name":"Doe" }, { "f_name":"Anna","m_name" : "Marie","l_name":"Smith" }, { "f_name":"Peter" , "l_name":"Jones" } ] } BSON is a binary form of JSON. 34 © 2013 IBM Corporation

35. The Power of JSON Drives Flexible Schema JSON key value pair enables a flexible schema

36. Data Access Examples Relational Representation LName FName Address Miller John 123 Blazer St JSON Representation JSON_string = ‘{“LName”:”Miller”,”FName”,”John”,”Address”:”123 Blazer St”}’; Javascript data access var JSONelems = JSON.parse( JSON_string ) f_name = JSONelems.FNname; l_name = JSONelems.FName; l_addr = JSONelems.Address;

37. Simple Code Example db.posts.insert( ‘{“author”:”John”, “date”,”2013-04-20”,”post”,”mypost Creates the database “db” if it does not exists Creates the collection “posts” if the it does not exists Insert a record into a blog post by user John db.posts.find ( ‘{ “author”:”John” }’ ) Retrieve all posts by user John ”}’ )

38. Dynamic Elasticity Rapid horizontal scalability − Ability for the application to grow by adding low cost hardware to the solution − Ability to add or delete nodes dynamically − Ability rebalance the data dynamically Application transparent elasticity

39. Why Scale Out Instead of Up? Scaling Out − Adding more servers with less processors and RAM − Advantages Startup costs much less Can grow instep with the application Individual servers cost less − Several less expensive server rather than fewer high cost servers − Redundant servers cost more Greater chance of isolating catastrophic failures Scaling Up − Adding more processors and RAM to a single server − Advantages Less power consumption than running multiple servers Less infrastructure (network, licensing,..)

40. Difference between Sharding Data VS Replication Shard Key state= “CA” Shard Key state= “WA” Sharding Each node hold a portion of the data • Hash • Expression Data is copied to all nodes Actions are shipped to applicable nodes 40 Same data on each node Inserted data is placed on the correct node Shard Key state= “OR” Replication Work on local copy and modification are propagated

41. Motivation of Sharding Synergistic with the application strategy Start small and grow with commodity hardware Sharding is not

42. Sharding is not for Data Availability Sharding is for growth, not availability Redundancy of a node provides high availability for the data − Both Mongo and Informix allow for multiple redundant nodes − Mongo refers to this as Replica Sets and the additional nodes slaves − Informix refers to this as MACH, and additional nodes secondary With Informix the secondary server can: − Provide high availability − Scale out Execute select Allow Insert/Update/Deletes on the secondary servers Share Disks with the master/primary node

43. Basic Data Distribution/Replication Terms Term Description Informix Term Shard A single node or a group of nodes holding the same data (replica set) Instance Replica Set A collection of nodes contain the same data MACH Cluster Shard Key The field that dictates the distribution of the documents. Must always exist in a document. Shard Key Sharded Cluster A group shards were each shard contains a portion of the data. Grid/Region Slave A server which contains a second copy of the data for read only processing. Secondary Server Remote Secondary 43

44. Sharding is not for Data Availibility Shard Key state= “CA” Shard Key state= “WA” Shard Key state= “OR” 44

45. NEW INFORMIX NOSQL/JSON CAPABILITIES

46. IBM Use Case Characteristics for JSON

47. High Level Solution 47

48. Flexible Schema, Native JSON and BSON Provide native JSON & BSON support in the Informix Database Server Two new built in “first-class” data types called JSON and BSON Support for MongoDB client side APIs throughout the entire database Enhance horizontal scaling by enabling sharding on all database objects and models Adaptive default system initialization − − 48 Up and running in seconds Adapts to computer an environment

49. Two New Data Types JSON and BSON Native JSON and BSON data types Index support for NoSQL data types Native operators and comparator functions allow for direct manipulation of the BSON data type Database Server seamlessly converts to and from JSON BSON Character data 49 JSON

50. JSON and BSON Data Type Details Row locks the individual BSON/JSON document − MongoDB must lock the database Bigger documents – 2GB maximum size − MongoDB caps at 16MB Ability to compress documents − MongoDB currently not available Ability to intelligently cache commonly used documents − MongoDB currently not available

51. Client Applications Applications MongoDB native Client New Wire Protocol Listener supports existing MongoDB drivers Connect to MongoDB or Informix with same application! MongoDB driver MongoDB Wire Protocol IBM JDBC NoSQL Driver Wire Protocol Listener MongoDB web browser MongoDB shell Mobile Informix DB

52. MongoDB Application Compatibly Ability to use any of the MongodB client drivers and frameworks against the Informix Database Server − Little to no change required when running MongoDB programs − Informix listens on the same default port as mongo, no need to change. Leverage the different programming languages available − Language examples C, C#, Erlang, JAVA, node.js, Perl, Python, Ruby Mongo Action Description db.customer.insert( { name: “John", age: 21 } ) Insert into database “db” the customer collection the associated document. db.customer.find( {age: { $gt:21 } } ) Retrieve from database “db” the customer collection all documents with the age greater than 21. 52

53. Fix up JSON Basic Mongo Operations Conceptual Operations Mongo Action db.customer.insert( { name: “John", age: 21 } ) CREATE DATABASE if not exist db CREATE TABLE if not exist customer INSERT INTO customer VALUES ( { “name”:”John”,”age:21”} ) db.customer.find() SELECT bson_doc FROM customer db.customer.find( {age: { $gt:21 } } ) SELECT * FROM customer WHERE age > 21 db.customer.drop() DROP TABLE customer db.customer.ensureIndex( { name : 1, age : -1 } ) CREATE INDEX idx_1 on customer(name , age DESC) db.customer.remove( {age: { $gt:21 } } ) DELETE FROM customer where age > 21 db.customer.update( { age: { $gt: 20 } }, { $set: { status: “Drink" } }, { multi: true } ) 53 Traditional SQL Action UPDATE customer SET bson_doc_field (status) = { “status”:“Drink" } WHERE age > 20

54. Scaling Out Using Sharded Queries Shard Key state= “CA” Shard Key state= “WA” Find sold cars for all states 1. Request data from local shard 2. Automatically sends request to other shards requesting data 3. Returns results to client Shard Key state= “OR” 54 © 2013 IBM Corporation

55. Scaling Out Using Sharded Inserts Shard Key state= “CA” Shard Key state= “WA” Row state = “OR” 1. Insert row sent to your local shard 2. Automatically forward the data to the proper shard Shard Key state= “OR” 55 © 2013 IBM Corporation

56. Scaling Out Using Sharded Delete Shard Key state= “CA” Shard Key state= “WA” Delete state = “OR” AND Dnum = “123123” 1. Delete condition sent to local shard 2. Local shard determine which shard(s) to forward the operation Shard Key state= “OR” 56 3. Execute the operation © 2013 IBM Corporation

57. Scaling Out Using Sharded Update Shard Key state= “CA” Shard Key state= “WA” Row state = “OR” Row state = “OR” 1. Insert a row on your local shard 2. Automatically forward the data to the proper shard Shard Key state= “OR” 57 © 2013 IBM Corporation

58. Scaling Out Adding a Shard Shard Key state= “CA” Shard Key state= “WA” Command Add Shard “NV” 1. Send command to local node 2. New shard dynamically added, data re-distributed (if required) Shard Key state= “OR” 58 Shard Key state= “NV” © 2013 IBM Corporation

59. Sharding with Hash Hash based sharding simplifies the partitioning of data across the shards Benefits − No data layout planning is required − Adding additional nodes is online and dynamic Cons − Adding additional node requires data to be moved Data automatically broken in pieces

60. Scaling Out with Hash Sharding - Insert Shard Key HASH( gambler_ID ) Shard Key HASH( gambler_ID ) Row Gambler_ID =“11” 1. Insert row sent to your local shard 2. Local shard determine which shard(s) to forward the insert 3. Execute the operation Shard Key HASH( gambler_ID ) 60 © 2013 IBM Corporation

61. Move to END Informix NoSQL Cluster Architecture Overview Shard Key state= “CA” Shard Key state= “OR” 61 two independent copies of the data, but three servers to share the workload (two servers share the same disks). Read/Write activity supported on all servers independent copies of two the data and two servers to share the workload. Read/Write activity supported on all servers three independent copies of the data, but four servers to share the workload (two servers share the same disk). Read/Write activity supported on all servers Shard Key state= “WA” © 2013 IBM Corporation

62. SHARDING NEEDED HERE

63. INFORMIX NOSQL DEEP DIVE INTO IMPLEMENTATION

64. Focus Areas. Applications need easier way to persist objects. − Object-relational layers like hibernate has performance & functional issues. − Support Mongo API and its eco system Flexible Schema: Changing schema in RDBMS is a significant operation, especially in production. − Need exclusive access to table & application downtime − Fast changing schema & sparse data has issues with fixed schema. -- Implement JSON (BSON) type, data store & indexing Scale Out: While RDBMS has solutions for cluster and MPP, features enabling application development and deployment in cloud+MPP architecture is limited. − Use Informix flexible grid & replication to shard tables; − Enhance query processing for distributed queries

65. Client Applications New Wire Protocol Listener supports existing MongoDB drivers Connect to MongoDB or Informix with same application! MongoDB native Client application MongoDB MongoDB driver IBM JDBC NoSQL Driver Wire Protocol Listener MongoDB web browser application Informix NoSQL Cluster MongoDB shell IBM Wire Protocol Listener logic shared with DB2 and Extreme Scale 65 © 2013 IBM Corporation

66. Informix for SQL and NoSQL Applications NoSQL APP IBM Wire Listener IBM Wire Listener JDBC connections JDBC connections SQL Apps + Tools SQL Apps + Tools SQL Drivers ODBC/JDBC NoSQL APP SQL Drivers ODBC/JDBC IBM Wire Listener JDBC connections Informix Dynamic Server (shard 1) Informix Dynamic Query processing Distributed Queries JSON Tables Collections Server (shard…n) IDXs IDXs JSON Tables Collections Tables Tables Tables IDXs Logs Enterprise replication + Flexible Grid Tables IDXs ER + Grid Logs

67. Informix for SQL and NoSQL Applications Mongo Application SQL Apps + Tools NoSQL/BSON SQL IBM Wire Listener SQL Drivers ODBC/JDBC JDBC connections SQL + BSON SQL Informix Dynamic Server (shard 1) Query processing JSON Tables Collections Distributed Queries SELECT Informix Dynamic Server (shard…n) IDXs IDXs JSON Tables Collections Tables Tables Tables IDXs Logs Enterprise replication + Flexible Grid INSERT UPDATE DELETE Tables IDXs ER + Grid Logs

68. NoSQL Feature Checklist.

72. NoSQL Feature Informix Implementation 1. Flexible Schema Use BSON and JSON data type. Complete row is stored in a single column; BSON, JSON are multi-rep types and can store up to 2GB. 2. Accessing KV pairs within JSON. Translate the Mongo queries into SQL expressions to access key-value pairs. Informix has added expressions to extract specific KV pairs. Select bson_new(data, “{id:1, name:1, addr:1}”) from tab; 3. Indexing Support standard B-tree index via functional indexing. create index itid on t(bson_value_int(data, “{id:1}”); Create index itnamestate on t(bson_value_lvarchar(data, “{name,1}”), bson_value_lvarchar(data, “{city,1}”); Informix also supports indexing bsons keys with different data types. 4. Sharding Supports range & hash based sharding.Informix has built-in technology for replication. Create identical tables in multiple nodes. Add meta data for partitioning the data across based on range or hash expressions. 5. SELECT Limited support now. Mongo API helps by disallowing joins. The translated query on a single table is transformed into federated UNION ALL query; includes shard elimination.

73. NoSQL Feature Informix Implementation 6. Updates (single node) INSERT: Simple direct insert. DELETE: DELETE statement with WHERE bson_extract() > ?; or bson_value..() > ? UPDATE: bson_update(bson, “update expr”) will return a new bson after applying the bson expression. Simple updates to non_sharded tables will be direct UPDATE statement in a single transaction. UPDATE tab bsoncol = bson_update(bsoncol, “expr”) where 7. Updates (sharded env) INSERT – All rows are inserted to LOCAL shard, replication threads will read logs and replicate to the right shard and delete from local node (log only inserts underway). DELETE – Do the local delete and replicate the “delete statements” to target node in the background UPDATE – Slow update via select-delete-insert. 8. Transaction Each statement is a distinct transaction in a single node environment. The data and the operation is replicated via enterprise replication. 9. Isolation levels. NoSQL session can use any of Informix isolation levels. -- Examples from Mongo applications. 10. Locking Application control only on the node they’re connected to. Standard Informix locking semantics will apply when data is replicated and applied on the target shard. -- Verify.. What’s the default locking level? -- Any option to change it? Just ONCONFIG??

74. Feature Informix Implementation 11. Hybrid access Mongo APIs can simply access Informix tables, views, virtual tables as if they’re JSON collections. db.foo.find(); foo can be JSON collection, relational table, a view or a virtual table on top of timeseries or Websphere MQ. From MongoAPI to relational tables. 12. Hybrid access 1. Directly get binary BSON or cast to JSON to get in textual form. From SQL to JSON data 2. Use expressions to extract to extract specific key-value pairs. 3. To be done: NoSQL collections will only have one BSON object in the table. We can “imply” the expressions when the SQL refers to a column. SELECT t.c1, t.c2 from t; SELECT bson_extract(t.data, “{c1:1}”), bson_extract(t.data, “{c2:1}”) from t; So, that’s twelve steps for NoSQL Anonymous!

75. 1. Flexible Schema Clients exchange BSON document with the server both for queries and data. Thus, BSON becomes a fundamental data type. The explicit key-value pairs withing the JSON/BSON document will be roughly equivalent to columns in relational tables. However, there are differences! − The type of the KV data encoded within BSON is determined by the client − Server is unaware of data type of each KV pair at table definition time. − No guarantees that data type for each key will remain consistent in the collection. − The keys in the BSON document can be arbitrary; − While customers exploit flexible schema, they’re unlikely to create a single collection and dump everything under the sun into that collection. − Due to the limitations of Mongo/NoSQL API, customers typically denormalize the tables (customer will have customer+customer addr + customer demographics/etc) to avoid joins.

76. 1. Flexible Schema – Informix Implementation • Informix has a new data type BSON to store the data. • Informix also has a JSON data type to convert between binary and text form. • BSON and JSON are abstract data types (like spatial, etc). • BSON and JSON multi-representational types. •Objects up to 4K is stored in data pages. •Larger objects (up to 2GB) are stored out of row, in BLOBs. •MongoDB limits objects to 16MB. •This is all seamless and transparent to applications.

77. 1. Flexible Schema – Informix Implementation CREATE TABLE customer (data BSON); • BSON is the binary represenation of JSON. •It has length and types of the key-value pairs in JSON. • MongoDB drivers send and receive in BSON form.

78. 2. Accessing KV pairs within JSON. • We’ll have number of extract expressions/functions •Expressions returning base type bson_value_bigint(BSON, “key”); bson_value_lvarchar(bsoncol, “key.key2”); bson_value_date(bsoncol, “key.key2.key3”); bson_value_timestamp(bsoncol, “key”) bson_value_double(bsoncol, “key”); bson_value_boolean(bsoncol, “key”); bson_value_array(bsoncol, “key”); bson_keys_exist(bsoncol, “key”); Bson_value_document(bsoncol, “key”) Bson_value_binary(bsonol, “key”) Bson_value_objectid(bsoncol, “key”) •Expression returning BSON subset. Used for bson indices. bson_extract(bsoncol, “projection specification”) •Expressions to project out of SELECT statement. bson_new(bsoncol, “{key1:1, key2:1, key3:1}”); bson_new(bsoncol, “{key5:0}”);

79. 2. Accessing KV pairs within JSON. Mongo Query SQL Query db.customers.find(); SELECT SKIP ? Data::bson db.customers.find({},{num:1,name:1}) ; SELECT SKIP ? bson_new( data, '{ "num" : 1.0 , "name" : 1.0}')::bson FROM customers db.customers.find({}, {_id:0,num:1,name:1}); SELECT SKIP ? bson_new( data, '{_id:0.0, "num" : 1.0 , "name" : 1.0}')::bson FROM customers db.customers.find({status:”A”}) SELECT SKIP ? data FROM customers WHERE bson_extract(data, ‘status') = “A” db.customers.find({status:”A”}, {_id:0,num:1,name:1}); FROM customers SELECT SKIP ? bson_new( data, '{ "_id" : 0.0 , "num" : 1.0 , "name" : 1.0}')::bson FROM customers WHERE bson_extract(data, 'name') = “A”

80. 3. Indexing • • • • • Supports B-Tree indexes on any key-value pairs. Indices could be on simple basic type (int, decimal) or BSON Indices could be created on BSON and use BSON type comparison Listener translates ensureIndex() to CREATE INDEX Listener translates dropIndex() to DROP INDEX Mongo Query SQL Query db.customers.ensureIndex({orderD ate:1}) CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" : "newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name" : "x_1" , "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64 db.customers.ensureIndex({orderD ate:1, zip:-1}) CREATE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC, bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0 , "$extract"] , "c2" : [ -1.0 , "$extract"]} , "name" : "c1_1_c2__1" , "index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64 db.customers.ensureIndex({orderD ate:1}, {unique:true) CREATE UNIQUE INDEX IF NOT EXISTS v_c1_1_c2__1 ON v (bson_extract(data,'c1') ASC, bson_extract(data,'c2') DESC) using bson (ns='{ "name" : "newdb.v.$c1_1_c2__1"}', idx='{ "ns" : "newdb.v" , "key" : { "c1" : [ 1.0 , "$extract"] , "c2" : [ -1.0 , "$extract"]} , "name" :"c1_1_c2__1" , "unique" : true , "index" : "v_c1_1_c2__1"}') EXTENT SIZE 64 NEXT SIZE 64

81. 3. Indexing db.w.find({x:1,z:44},{x:1,y:1,z:1}) Translate to: SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1} FROM w WHERE ( bson_extract(data, 'x') = '{ "x" : 1.0 }'::json::bson ) AND ( bson_extract(data, 'z') = '{ "z" : 44.0 }'::json::bson) Estimated Cost: 2 Estimated # of Rows Returned: 1 1) keshav.w: SEQUENTIAL SCAN Filters: (informix.equal(informix.bson_extract(keshav.w.data ,'z' ),UDT ) AND informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT))

82. 3. Indexing •Functional Index is built on bson expressions CREATE INDEX IF NOT EXISTS w_x_1 ON w (bson_extract(data,'x') ASC) using bson (ns='{ "name" : "newdb.w.$x_1"}', idx='{ "ns" : "newdb.w" , "key" : {"x" : [ 1.0 , "$extract"]} , "name" : "x_1" , "index" : "w_x_1"}') EXTENT SIZE 64 NEXT SIZE 64 •Listener is aware of the available index and therefore generates right predicates. db.w.find({x:1}); gets translated to SELECT SKIP ? data FROM w WHERE bson_extract(data, 'x') = ?

83. 3. Indexing db.w.find({x:5,z:5}, {x:1,y:1,z:1}) Translates to: SELECT bson_new( data, '{ "x" : 1.0 , "y" : 1.0 , "z" : 1 .0}')::bson FROM w WHERE ( bson_extract(data, 'x') = '{ "x" : 5.0 }'::json::bson ) AND ( bson_extract(data, 'z') = '{ "z" : 5.0 }'::json::bson ) Estimated Cost: 11 Estimated # of Rows Returned: 1 1) keshav.w: INDEX PATH Filters: informix.equal(informix.bson_extract(keshav.w.data ,'z' ), UDT ) (1) Index Name: keshav.w_x_1 Index Keys: informix.bson_extract(data,'x') (Serial, fragments: ALL) Lower Index Filter: informix.equal(informix.bson_extract(keshav.w.data ,'x' ),UDT )

85. 4. Mongo Sharding

86. 4. Mongodb SHARDING (roughly) Shard a single table by range or hashing. Mongos will direct the INSERT to target shard. Mongos tries to eliminate shards for update, delete, selects as well. FIND (SELECT) can happen ONLY a SINGLE table. It also works as coordinator for multi-node ops. Once a row is inserted to a shard, it remains there despite any key update. No transactional support on multi-node updates. − Each document update is unto its own.

87. 4. Informix Sharding App Server App Server App Server App Server Mongo Driver Mongo Driver Mongo Driver Mongo Driver Listener Listener Listener Listener JDBC JDBC JDBC JDBC Customer/1 Customer/2 Customer/3 Customer/4 Customer/5 Customer/6 Sales/1 Sales/2 Sales/3 Sales/4 Sales/5 Sales/6 Location/1 Location/2 Location/3 Location/4 Location/5 Location/6 Informix/2 Informix/3 Informix/1 Informix/4 Informix/5 Enterprise repliation + Flexible Grid Informix/6

88. 4. Informix sharding + High Availability App Server App Server App Server App Server Mongo Driver Mongo Driver Mongo Driver Mongo Driver Listener Listener Listener Listener JDBC JDBC JDBC JDBC Informix/1 Primary Informix/2 Primary Informix/3 Primary Informix/4 Primary Informix/5 Primary Informix/6 Primary Informix/1 SDS/HDR Informix/2 SDS/HDR Informix/3 SDS/HDR Informix/4 SDS/HDR Informix/5 SDS/HDR Informix/6 SDS/HDR Informix/1 RSS Informix/2 RSS Informix/3 RSS Informix/4 RSS Informix/5 RSS Informix/6 RSS Enterprise repliation + Flexible Grid

89. 4. Sharding – Informix Implementation •Shard a single table by range or hashing. cdr define shard myshard mydb:usr1.mytab –type=delete –key=”bson_get(bsoncol, ‘STATE’)” –stragety=expression versionCol=version servA “in (‘TX’, ‘OK’)” servB “in (‘NY’,’NJ’) “ servC “in (‘AL’,’KS’) “ servD remainder cdr define shard myshard mydb:usr1.mytab –type=delete –key=state –stragety=hash --versionCol=version servA servB servC servD •Shard a single table by range or hashing. <<Mongo syntax>>

90. 4. Sharding – Informix Implementation •Shard a single table by range or hashing. •Sharding is transparent to application. •Each CRUD statement will only touch a single table. •Limitation of MongoDB…Makes it easier for Informix. •Lack of joins is a big limitation for SQL applications. •Lacks transactional support for distributed update.

91. 4. Informix Sharding Identical table is created on each node and meta data is replicated on each node. Schema based replication is the foundation for our sharding. CRUD operations can go to any node. − We’ll use replication and other techniques to reflect the data in the target node, eventually. − Right now, replication is asynchronous. − Informix has synchronous replication not used for sharding now.

92. 5. SELECT on sharded tables •The query can be submitted to any of the nodes via the listener. •That node acts as the “coordinator” for the distributed query. •It also does the node elimination based on the query predicate. •After that, the query is transformed to UNION ALL query SELECT SKIP ? bson_new( data, '{"_id":0.0 ,"num":1.0 ,"name" : 1.0}')::bson FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or bson_extract(data, 'name') = “X” is transformed into: SELECT SKIP ? bson_new( data, '{"_id":0.0 FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or UNION ALL SELECT SKIP ? bson_new( data, '{"_id":0.0 FROM customers@rsys1:db WHERE bson_extract(data, 'name') = “A” or ,"num":1.0 ,"name" : 1.0}')::bson bson_extract(data, 'name') = “X” ,"num":1.0 ,"name" : 1.0}')::bson bson_extract(data, 'name') = “X”

93. 6. INSERT: Single node •If necessary, automatically create database & table (collection) on the application INSERT •Collections: CREATE TABLE t(a GUID, d BSON); •GUID column is needed to ensure unique row across the SHARDs. Also used as PK for replication. •Client application inserts JSON, client API converts this into BSON, generates _id (object id) if necessary & sends to server over JDBC/ODBC. •Server saves the data into this table as BSON, with an automatically generated GUID

94. 6. DELETE: Single

Informix NoSQL & Hybrid SQL detailed deep dive

Informix NoSQL & Hybrid SQL detailed deep dive

Recommended

More Related Content

What's hot

What's hot(20)

Similar to Informix NoSQL & Hybrid SQL detailed deep dive

Similar to Informix NoSQL & Hybrid SQL detailed deep dive(20)

More from Keshav Murthy

More from Keshav Murthy(20)

Recently uploaded

Recently uploaded(20)

Informix NoSQL & Hybrid SQL detailed deep dive