Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building node.js applications with Database Jones


Published on

Silicon Valley CodeCamp 2015 presentation on Database Jones

Published in: Software
  • Be the first to comment

Building node.js applications with Database Jones

  1. 1. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Database Jones ! J.D. Duncan, ! Craig Russell, 1 1
  2. 2. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. J.D. ! Senior Software Engineer, MySQL Cluster, at Oracle ! Former Unix Sysadmin & web developer ! MySQL AB (2004) - Sun (2008) - Oracle (2011) ! Projects ! Database Jones, NDB Memcache, mod_ndb 2 2
  3. 3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Craig ! Architect, Oracle Corp. ! Java Data Objects JSR-243 Specification Lead ! Projects ! Database Jones connector for MySQL Cluster ! Cluster/J Java Connector for MySQL Cluster ! Secretary, Apache Software Foundation 3 3
  4. 4. Why Database Jones? •Node.JS • Highly scalable Javascript platform for web services •MySQL Cluster • Highly scalable database engine •Database Jones • Highly scalable API for database access from Node.JS 4
  5. 5. MySQL Cluster Overview 5
  6. 6. MySQL Cluster Architecture MySQL C 6
  7. 7. Database Jones •Fast, easy database API for Node.js •JavaScript for user API and common operations •Two adapters currently shipping • NDB: Native adapter to MySQL Cluster C++ API • MySQL: Adapter to node-mysql (third party tool) • Open architecture allows other adapters •Most operations are asynchronous 7
  8. 8. Data Model •Two models are supported: • Operations using table names return plain objects • {"id": 554, "first_name": "Roy", "last_name": "Raye"} • Operations using JavaScript constructors return instances of JavaScript classes 8
  9. 9. Object Mapping •Table mapping defines the relationship between • Tables and columns • Objects and fields • Foreign keys and object relationships •Table mapping is flexible • Change field names • Provide transformations of data • Database format to JavaScript format • e.g. Date types, True/False, custom formats 9
  10. 10. Mapped Table Formats •Classic • Fields are stored in individual columns •Serialized • Objects are serialized into JSON and stored in a JSON column •Hybrid • Some fields are stored in their own column • All other fields are serialized 10
  11. 11. Classic Format •Most appropriate for existing tables/schema •Schema is defined outside the application •Compose complex objects via database joins 11
  12. 12. Serialized •Most appropriate for “schema-less” designs •Limit query capability (we’re working on it) •Limited join capability (we’re working on this too) 12
  13. 13. Hybrid •Combines best features of serialized and classic •Some fields are stored in their own column • object composition via joins •All other fields are stored serialized • Object composition via serialization •Hybrid tables can store arbitrarily complex objects efficiently 13
  14. 14. Database Jones Operations •Metadata (non-transactional) • List tables • Get metadata for a table •CRUD (transactional) • Insert (save) • Find by key (includes joins) • Delete by key • Update by key •Query (transactional) • Complex query criteria 14
  15. 15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Install & Demo 15
  16. 16. Install from Github ✦ git clone ✦ git config --global --add core.symlinks true You may need to allow git to make symlinks: 16
  17. 17. Components The user API DB Service Providers Network Configuration A directory of symlinks Sample Application code Unified debugging library for C++ and JS Benchmark 17
  18. 18. Architecture Jones Application Code DB Service Provider Jones API Jones SPI Database Database Network Protocol 18
  19. 19. Using the DB Service Providers •MySQL • npm install mysql •NDB • cd jones-ndb • node configure 19
  20. 20. Sample Twitter-like Application •In samples/tweet •5 tables 20
  21. 21. Contents of samples/tweet 4 simple API demos 2 SQL scripts Main Application (Command Line & HTTP) Demos of tweet.js 21
  22. 22. tweet.js Usage: node tweet {options} {command} {command arguments} -a <adapter>: run using the named adapter (default: ndb) -h or --help: print this message -d or --debug: set the debug flag --detail: set the detail debug flag -df <file>: enable debug output from <file> -E or --deployment <name>: use deployment <name> (default: te COMMANDS: put user <user_name> << JSON Extra Fields >> get user <user_name> delete user <user_name> post tweet <author> << Message >> get tweet <tweet_id> delete tweet <tweet_id> put follow <user_follower> <user_followed> get followers <user_name> get following <user_name> get tweets-by <user_name> get tweets-at <user_name> get tweets-about <hashtag> get tweets-recent <count> start server <server_port_number> 22
  23. 23. Author table: Hybrid data model CREATE TABLE author ( user_name varchar(20) CHARACTER SET UTF16LE not null, full_name varchar(250), tweet_count int unsigned not null default 0, SPARSE_FIELDS varchar(4000) CHARACTER SET utf8, PRIMARY KEY(user_name) ) ; “Catch All” JSON Column from create_tweet_tables.sql 23
  24. 24. Hybrid Insert into Author node tweet put user caligula '{ "full_name": "Gaius Julius Caesar Germanicus" , "profile_text": "I am your little boot!" }' user_name full_name SPARSE_FIELDS from 24
  25. 25. API Essentials •SessionFactory is a heavyweight master connection •Session is a lightweight pooled connection for a user • e.g. for an HTTP request • a session has a single active transaction •Session provides APIs for database operations • Example: use session.find() to find a single row of data 25
  26. 26. API Sample application: find.js /* This script shows an example find() operation using a table name and primary key, and working with promises. For a similar example using callbacks rather than promises, see insert.js */ "use strict"; var jones = require("database-jones"); 26
  27. 27. find.js: Configure a database connection /* new ConnectionProperties(adapter, deployment) The first argument names a database backend, e.g. "ndb", "mysql", etc. The second argument names a "deployment" defined in a jones_deployments.js file. (A default file can be found two directories up from here). jones_deployments.js is the preferred place to customize the host, username, password, and other parameters of the database connection. */ var connectionProperties = new jones.ConnectionProperties("mysql", "test"); 27
  28. 28. find.js: Process the command line /* node find.js table_name primary_key_value argv[0] argv[1] argv[2] argv[3] */ if (process.argv.length !== 4) { console.log("Usage: node find <table> <key>n"); process.exit(1); } var table_name = process.argv[2], find_key = process.argv[3]; 28
  29. 29. find.js: Run the database operation /* This version of openSession() takes one argument and returns a promise. The argument is the set of connection properties obtained above. Once the session is open, use it to find an object. find() is a Jones API call that takes a primary key or unique key and, on success, returns *only one object*. */ jones.openSession(connectionProperties). then(function(session) { return session.find(table_name, find_key); }). then(console.log, console.trace). // log the result or error then(jones.closeAllOpenSessionFactories); // disconnect 29
  30. 30. Many varieties of find() The first argument can be: • Table Name • JavaScript constructor that has been mapped • Projection describing a structure of related objects 30
  31. 31. Many varieties of find() The second argument can be: • String or Number 1-part primary key lookup • { key_field_name : value } Primary key or any unique index • { key_part_1 : value1, key_part_2: value2 } Multi-part primary key or unique index 31
  32. 32. Many varieties of find() • The optional third argument can be a callback • Any extra arguments after the third will be supplied to the callback (for additional context) • find() always returns a promise (even if you don’t use the promise) 32
  33. 33. insert.js: Callback Style /* This script shows an example persist() operation using a table name and primary key, and working with callbacks. */ function disconnectAndExit(status) { jones.closeAllOpenSessionFactories(function() { process.exit(status); }); } /* handleError() exits if "error" is set, or otherwise simply returns. */ function handleError(error) { if(error) { console.trace(error); disconnectAndExit(1); } } 33
  34. 34. insert.js: Run the operation /* This version of openSession() takes three arguments: ConnectionProperties A table name, which will be validated upon connecting A callback which will receive (error, session) */ jones.openSession(connectionProperties, table_name, function(err, session) { handleError(err); /* The callback for persist() only gets one argument */ session.persist(table_name, object, function(err) { handleError(err); console.log("Inserted: ", object); disconnectAndExit(0); }); }); 34
  35. 35. scan.js: Query /* This script provides an example of the Jones Query API. In this example, we query the tweet table for posts by a particular author, and apply a sort and limit on the query. */ "use strict"; var jones = require("database-jones"); if (process.argv.length < 3 ) { console.log("usage: node scan <author> [limit] [order]"); process.exit(1); } 35
  36. 36. scan.js: Query Parameters // node scan.js <author> [limit] [order] var connectionProperties = new jones.ConnectionProperties("ndb", "test"), queryTerm = process.argv[2], limit = Number(process.argv[3]) || 20, order = (process.argv[4] == "asc" ? "asc" : "desc"); 36
  37. 37. scan.js: Query Operation jones.openSession(connectionProperties). then(function(session) { return session.createQuery("tweet"); }). then(function(query) { /* Here we can define query conditions. For more details see API-documentation/Query */ query.where(query.author_user_name.eq(queryTerm)); /* Then execute the query, using limit & order parameters. */ return query.execute({ "limit" : limit, "order" : order }); }). then(console.log, console.trace). // log the result or error then(jones.closeAllOpenSessionFactories); // disconnect 37
  38. 38. Session Methods ✦find() ✦remove() ✦persist() ✦update() ✦save() ✦load() Metadata Operations Key Operations ✦listTables() ✦getTableMetadata() Query Operations ✦createQuery() Others ✦currentTransaction() ✦close() ✦createBatch() ✦setPartitionKey() 38
  39. 39. Batch: powerful grouping of “mixed” key operations batch = session.createBatch(); tag = tags.hash.pop(); // # hashtags while(tag !== undefined) { tagEntry = new HashtagEntry(tag, tweet); batch.persist(tagEntry); tag = tags.hash.pop(); } return batch.execute(); ✦batch.find() ✦batch.remove() ✦batch.persist() ✦batch.update() ✦ ✦batch.load() 39
  40. 40. Example from tweet.js /* Insert a tweet. - Start a transaction. - Persist the tweet & get its auto-increment id. - Create & persist #hashtag & @mention records (all in a single batch). - Increment the author's tweet count. - Then commit the transaction. */ function InsertTweetOperation(params, data) { [ ... ] session.currentTransaction().begin(); session.persist(tweet). then(function() { return session.find(Author, authorName);}). then(incrementTweetCount). then(createTagEntries). then(commitOnSuccess, rollbackOnError). then(function() {return tweet;}). then(this.setResult). then(this.onComplete, this.onError); 40
  41. 41. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Query & Projection APIs 41
  42. 42. Query API •Session is Query factory using asynchronous api •Query has a filter based on the mapped object •Filter has comparators • eq, ne, gt, ge, le, lt, between, isNull, isNotNull •Filter has boolean operations • and, or, not, andNot, orNot •Query execution is asynchronous •Filter determines query strategy • Primary/unique key lookup; index scan; table scan •Properties govern query execution •Results are given in callback 42
  43. 43. Query Comparators •Comparators compare properties to parameters •Query Domain Type property names correspond to Constructor field names (properties) •Parameters are created by name • qdt.param('date_low') •Properties are referenced by field name in Constructor • qdt.date_created •Comparators are properties of qdt properties •'date_low')); •Comparators return predicates 43
  44. 44. Query Operators •Predicates are used as query filters via where function •Predicates are results of comparators or operators •Operators combine predicates: and, or, andNot, orNot, not • var predicate1 ='date_low')); • var predicate2 ='date_high')); • var predicate = predicate1.and(predicate2); • var predicate = predicate3.andNot(predicate4); • qdt.where(predicate) 44
  45. 45. Query Execution •var promise = query.execute(parameters, callback); •parameters: regular javascript object • skip: number of results to skip over • limit: number of results to return • order: 'asc' or 'desc' • user-specified parameters defined by q.param •callback(err, results) •results: result[ ], possibly no elements 45
  46. 46. Query Example session.createQuery(Employee). then(function(query) { return query.where( execute({limit: 20})}). then(function(results) { results.forEach(function(result) { console.log('Name:',, 'age:', result.age); }) }). then(closeSession, reportError); 46
  47. 47. Projection •Composition of complex objects by joins • one to one, one to many, many to one, many to many •Define relationships (bidirectional) in table mapping •Choose fields to select in Projection •Choose relationships to join in Projection •find() can take a Projection as the first argument 47
  48. 48. Projection API •Projection constructor defines the domain object •Fields are added to Projection •Relationships to other Projections are added to Projection •Loops are not supported •Arbitrary nesting depth 48
  49. 49. join.js "use strict"; var jones = require("database-jones"); /* Constructors for application objects */ function Author() { } function Tweet() { } 49
  50. 50. join.js: TableMapping for Author /* TableMappings describe the structure of the data. */ var authorMapping = new jones.TableMapping("author"); authorMapping.applyToClass(Author); authorMapping.mapSparseFields("SPARSE_FIELDS"); authorMapping.mapOneToMany( { fieldName: "tweets", // field in the Author object target: Tweet, // mapped constructor targetField: "author" // target join field }); 50
  51. 51. join.js: TableMapping for Tweet /* TableMappings describe the structure of the data. */ var tweetMapping = new jones.TableMapping("tweet"); tweetMapping.applyToClass(Tweet); tweetMapping.mapManyToOne( { fieldName: "author", // field in the Tweet object target: Author, // mapped constructor foreignKey: "author_fk" // SQL foreign key relationship }); 51
  52. 52. join.js: Projection /* Projections describe the structure to be returned from find(). */ var tweetProjection = new jones.Projection(Tweet); tweetProjection.addFields(["id","message","date_created"]); var authorProjection = new jones.Projection(Author); authorProjection.addRelationship("tweets", tweetProjection); authorProjection.addFields(["user_name","full_name"]); 52
  53. 53. join.js: Run the database operation /* The rest of this example looks like find.js, only using find() with a projection rather than a table name. */ jones.openSession(new jones.ConnectionProperties("mysql", "test")). then(function(session) { return session.find(authorProjection, find_key); }). then(console.log, console.trace). // log the result or error then(jones.closeAllOpenSessionFactories). // disconnect then(process.exit, console.trace); 53
  54. 54. join.js: Result $ node join.js nero { user_name: 'nero', full_name: 'Lucius Domitius Ahenobarus', tweets: [ { id: 3, message: 'I love to sing!', date_created: Thu Oct 01 2015 16:09:46 GMT-0700 (PDT) }, { id: 4, message: 'I am the best #poet and the best #gladiator!', date_created: Thu Oct 01 2015 16:09:46 GMT-0700 (PDT) } ] } 54
  55. 55. Copyright © 2013, Oracle and/or its affiliates. All rights reserved. Thank you Code Camp! ✦ git clone 55