Using Spring with NoSQL databases (SpringOne China 2012)

5,312 views

Published on

A look at how Spring Data simplifies the development of applications that use NOSQL databases such as Redis and MongoDB.

Published in: Technology

Using Spring with NoSQL databases (SpringOne China 2012)

  1. 1. Using Spring with NoSQL databasesChris Richardson,Author of POJOs in Action, Founder of the original CloudFoundry.com @crichardson chris.richardson@springsource.com http://plainoldobjects.com/
  2. 2. Presentation goal NoSQL databases: what, why and how How Spring Data simplifies thedevelopment of NoSQL applications
  3. 3. About Chris
  4. 4. (About Chris)
  5. 5. About Chris()
  6. 6. About Chris
  7. 7. About Chrishttp://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/
  8. 8. vmc push About-Chris Developer AdvocateSignup at http://cloudfoundry.com
  9. 9. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  10. 10. Relational databases are great...• SQL • Well supported • High-level • JDBC • Sorting • Hibernate/JPA • Aggregation • Spring• ACID semantics • Well understood • Developers • Operators
  11. 11. ... but they have limitations• Object/relational impedance mismatch• Complicated to map rich domain model to relational schema• Difficult to handle semi-structured data, e.g. varying attributes• Schema changes• Extremely difficult/impossible to scale• Poor performance for some use cases
  12. 12. Solution: Spend Money ORhttp://upload.wikimedia.org/wikipedia/commons/e/e5/Rising_Sun_Yacht.JPG• Hire more DevOps• Use application-level sharding• Build your own middleware• … http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#
  13. 13. Solution: Use NoSQL Benefits• Higher performance• Higher scalability• Richer data-model• Schema-less Drawbacks • Limited transactions • Relaxed consistency • Unconstrained data
  14. 14. Growing in popularity…
  15. 15. But don’t get too excited
  16. 16. Solution: Use NewSQL• Relational databases with SQL and ACID transactions AND• New and improved architecture - designed for modern hardware• Radically better scalability and performance• NewSQL vendors: Gemfire/SQLFire, VoltDB, ...
  17. 17. Future = multi-paradigm data storage for enterprise applications IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg
  18. 18. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  19. 19. Redis• Advanced key-value store K1 V1• Written in C K2 V2• Very fast, e.g. 100K reqs/sec• Optional persistence ... ...• Transactions with optimistic locking• Master-slave replication• Sharding using client-side consistent hashing
  20. 20. Using Redis (via CLI)Datatypes: redis 127.0.0.1:6379> set foo 1 OK•Strings redis 127.0.0.1:6379> get foo•Hashes "1"•Maps redis 127.0.0.1:6379> sadd myset a•Lists (integer) 1•Sets redis 127.0.0.1:6379> sadd myset b (integer) 1•Sorted sets redis 127.0.0.1:6379> smembers myset 1) "a" 2) "b" redis 127.0.0.1:6379> srem myset a (integer) 1 redis 127.0.0.1:6379> smembers myset 1) "b"
  21. 21. Redis use cases• Replacement for Memcached • Handling tasks that overload an RDBMS • Session state • Hit counts - INCR • Cache of data retrieved from system of • Most recent N items - LPUSH and LTRIM record (SOR) • Randomly selecting an item –• Replica of SOR for queries needing high- SRANDMEMBER performance • Queuing – Lists with LPOP, RPUSH, …. • High score tables – Sorted sets and ZINCRBY • …
  22. 22. MongoDB• Document-oriented database • Geospatial queries • JSON-style documents: objects, lists, • Grid FS provides file storage primitives • Very fast, asynchronous writes • Schema-less • Highly scalable and available• Transaction = update of a single document• Rich query language for dynamic queries
  23. 23. Data model = Binary JSON documents Server Database: Food To Go Collection: Restaurants { "name" : "TGI Fridays", "type" : ”American", "serviceArea" : [ "94619", "94618" ], Sequence of "openingHours" : [ bytes on disk { è fast i/o "dayOfWeek" : "Wednesday", "open" : 1730, "close" : 2230 } ], "_id" : ObjectId("4bddc2f49d1505567c6220a0") }
  24. 24. MongoDB CLI> r = {name: Ajanta}> db.restaurants.save(r)> r{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r = db.restaurants.findOne({name:"Ajanta"}){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r.type= "Indian”> db.restaurants.save(r)> db.restaurants.update({name:"Ajanta"}, {$set: {name:"Ajanta Restaurant"}, $push: { menuItems: {name: "Chicken Vindaloo"}}})> db.restaurants.find(){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "menuItems" : [ { "name" : "Chicken Vindaloo" } ], "name" : "Ajanta Restaurant", "type" : "Indian" }> db.restaurants.remove(r.id)
  25. 25. MongoDB query by example{ serviceArea:"94619", Find a restaurant that openingHours: { serves the 94619 zip $elemMatch : { code and is open at 6pm "dayOfWeek" : "Monday", "open": {$lte: 1800}, on a Monday "close": {$gte: 1800} } }}DBCursor cursor = collection.find(qbeObject);while (cursor.hasNext()) { DBObject o = cursor.next(); … }
  26. 26. Scaling MongoDB Shard 1 Shard 2 Mongod Mongod (replica) (replica) Mongod Mongod Mongod Mongod (master) (replica) (master) (replica)Config Server Mongos mongod Mongos mongod Client mongod
  27. 27. MongoDB use cases• Use cases • Who is using it? • High volume writes • Shutterfly, Foursquare • Complex data • Bit.ly Intuit • Semi-structured data • SourceForge, NY Times • GILT Groupe, Evite, • SugarCRM
  28. 28. Other NoSQL databasesType ExamplesExtensible columns/Column-oriented Hbase e SimpleDB, DynamoDB r it avo Cassandra r f youGraph Neo4j t t ouKey-value I lef Voldemort, Riak ifDocument So rry CouchDb http://nosql-database.org/ lists 122+ NoSQL databases
  29. 29. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  30. 30. Spring Data is here to help For NoSQL databaseshttp://www.springsource.org/spring-data
  31. 31. Spring Data sub-projects• Relational • QueryDSL • JPA • Big Data • JDBC Extensions • Hadoop• NoSQL • HDFS and M/R • Redis • Hive • Mongo • Pig • HBase • Cascading • Neo4j • Splunk • Gemfire • Access • Lucene • REST
  32. 32. What you get• Template classes that hide the boilerplate code• Auto-generated (generic) repositories for some NOSQL databases• Java NoSQL mapping• Cross Store Persistence• Support in Roo
  33. 33. Get the book!
  34. 34. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  35. 35. Redis challenges• Connection management: need to get and reliably close connections• Data mapping: application objects Redis binary/strings• Multiple client libraries with gratuitously different APIs
  36. 36. Spring Data for Redis• Low-level - RedisConnection(Factory) • Connection management • Supports Jedis, Jredis, Rjc and Srp • Pluggable Java binary conversion • Insulates client code from underlying • Support classes: library • Collections-backed by RedisTemplate• High-level - RedisTemplate • Atomic Counters • Builds on RedisConnection(Factory) • Support for Redis pub/sub
  37. 37. Low-level API = RedisConnection(Factory)
  38. 38. Using RedisConnectionFactorypublic class LowLevelRedisTest { @Autowired private RedisConnectionFactory redisConnectionFactory; @Test public void testLowLevel() { Library independent code J RedisConnection con = null; try { con = redisConnectionFactory.getConnection(); byte[] key = "foo".getBytes(); Ugly byte arrays L byte[] value = "bar".getBytes(); con.set(key, value); byte[] retrievedValue = con.get(key); Assert.assertArrayEquals(value, retrievedValue); } finally { if (con != null) { con.close(); } Need to clean up L } }
  39. 39. Configuring RedisConnectionFactory@Configurationpublic class RedisConfiguration { @Value("${databaseHostName}") protected String databaseHostName; @Bean public RedisConnectionFactory jedisConnectionFactory() { JedisConnectionFactory factory = new JedisConnectionFactory(); factory.setHostName(databaseHostName); factory.setPort(6379); factory.setUsePool(true); return factory; }}
  40. 40. High-level API = RedisTemplate• Builds on RedisConnection(Factory) • Maps Redis exceptions DataAccessException• Analogous to JdbcTemplate • StringRedisTemplate• Parameterized type • Extends RedisTemplate<String, • K - Key type String> • V – Value type • Keys and values are Strings• Handles Java Key/Value Redis byte[]
  41. 41. Using StringRedisTemplatepublic class RedisTemplateTest { @Autowired private StringRedisTemplate stringRedisTemplate; Returns KV type specific interface @Test public void testGetAndSet() { stringRedisTemplate.opsForValue().set("foo", "bar"); assertEquals("bar", stringRedisTemplate.opsForValue().get("foo")); } @Test Converts between Strings and byte[] public void testHashOps() { stringRedisTemplate.opsForHash().put("myHash", "myKey", "value"); assertEquals("value", stringRedisTemplate.opsForHash().get("myHash", "myKey")); assertEquals(Collections.singleton("myKey"), stringRedisTemplate.opsForHash().keys("myHash")); assertEquals(Collections.singletonMap("myKey", "value"), stringRedisTemplate.opsForHash().entries("myHash")); }
  42. 42. Configuring StringRedisTemplate@Configurationpublic class RedisConfiguration { @Bean public RedisConnectionFactory jedisConnectionFactory() { … } @Bean public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory factory) { StringRedisTemplate template = new StringRedisTemplate(); template.setConnectionFactory(factory); return template; }}
  43. 43. RedisTemplate: Java objects binary data• DefaultSerializer - defaults to JdkSerializationRedisSerializer• KeySerializer• ValueSerializer• HashKeySerializer• HashValueSerializer
  44. 44. StringRedisTemplate uses StringRedisSerializer• Stores keys and values as Strings
  45. 45. Register serializers to override the default behavior Converted to JSON by RedisTemplate
  46. 46. Redis-backed Collections The key@Testpublic void testRedisSet() { Set<String> mySet = new DefaultRedisSet<String>("mySet", stringRedisTemplate); Assert.assertTrue(mySet.isEmpty()); mySet.add("a"); assertEquals(Collections.singleton("a"), stringRedisTemplate.opsForSet().members("mySet"));}
  47. 47. Redis Atomic Counters
  48. 48. Redis Pub/Sub - Consumer public class MyRedisSubscriber {@Configuration String message;public class RedisConfiguration { void handleMessage(String text) { @Bean this.message = text; public MyRedisSubscriber myListener() { return new MyRedisSubscriber(); } } } @Bean public RedisMessageListenerContainer redisMessageListenerContainer( RedisConnectionFactory redisConnectionFactory, MyRedisSubscriber myListener) { RedisMessageListenerContainer c = new RedisMessageListenerContainer(); c.setConnectionFactory(redisConnectionFactory); c.addMessageListener(new MessageListenerAdapter(myListener), new ChannelTopic("myChannel")); return c; }
  49. 49. Redis Pub/Sub - Producerpublic class RedisPubSubTest { @Autowired private StringRedisTemplate stringRedisTemplate; @Autowired private MyRedisSubscriber myListener; @Test public void testPubSub() throws InterruptedException { … stringRedisTemplate.convertAndSend("myChannel", "hello"); TimeUnit.SECONDS.sleep(4); Assert.assertEquals("hello", myListener.message); }}
  50. 50. Redis caching support KVs = <prefix + K, V>Template needs to (de)serialize K and V Sorted set of all keys for clear()
  51. 51. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  52. 52. MongoDB API usage patterns• Create and store Mongo singleton • Queries• Externalized server host, port etc. • Construct query object• Inserts/Updates • mongo.getDatabase(…).getCollection(…) • Map application POJO DBObject • Iterate through Cursor • Map DBObject application POJO • mongo.getDatabase(…).getCollection(…) • Partial document updates • Configure asynchronous vs. synchronous writes Higher-level than JDBC but still repetitive, …
  53. 53. Spring Data - MongoDB• MongoTemplate • Map-Reduce integration• Query, Criteria, and Update DSLs • GridFS support• Generic repositories• Querydsl integration• Cross-store persistence• GeoSpatial integration
  54. 54. MongoTemplate MongoTemplate POJO ó DBObject databaseNameSimplifies data access userId mapping PasswordTranslates exceptions defaultCollectionName writeConcern writeResultChecking save() insert() remove() <<interface>> updateFirst() MongoConvertor findOne() write(Object, DBObject) find() read(Class, DBObject) … uses Mongo MongoMapping (Java Driver class) Converter
  55. 55. Example entitypublic class Restaurant { private String id; public class MenuItem { private String name; private String name; private double price; private List<MenuItem> menuItems; public MenuItem(String name, double price) { public Restaurant(String name) { this.name = name; this.name = name; this.price = price; } … } ........} Spring Data uses fields and non- default constructors
  56. 56. Example data access code@Repositorypublic class RestaurantRepository { @Autowired private MongoTemplate mongoTemplate; public void add(Restaurant restaurant) { mongoTemplate.save(restaurant); } public List<Restaurant> findRestaurantsByName(String restaurantName) { return mongoTemplate.find( query(where("name").is(restaurantName)), Restaurant.class); }
  57. 57. A document in the restaurant collection{ "_id" : ObjectId("4d977f55d3fe3119c904e026"), "_class" : "net.chrisrichardson.mongodb.example.mongotemplate.Restaurant", "name" : "Ajanta" "menuItems" : [ { "name" : "Tandoori Portobello Mushrooms", "price" : 5.5 }, { "name" : "Duck Curry Kerala", "price" : 15 } ]}
  58. 58. Spring MongoDB Example - Config 1@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration { private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase; private @Value("#{mongoDbProperties.host}") Defines a String mongoDbHost; MongoTemplate public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); } @Override <beans> protected String getDatabaseName() { <context:annotation-config/> return mongoDbDatabase; } <context:component-scan base-package="net.chrisrichardson.mongodb.example"/>... <util:properties id="mongoDbProperties" location="mongodb.properties"/>} External Config </beans>mongodb.properties: databaseName=demo1 host=192.168.253.150
  59. 59. Spring MongoDB Example - Config 2<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate"> <constructor-arg ref="mongoFactory"/></bean><mongo:db-factory id="mongoFactory" host= "#{mongoDbProperties.host}" dbname="#{mongoDbProperties.databaseName}" /><util:properties id="mongoDbProperties" location="mongodb.properties"/>
  60. 60. Update example@Repositorypublic class RestaurantRepository { public void addMenuItem(String restaurantId, MenuItem newMenuItem) { mongoTemplate.updateFirst( query(where("_id").is(new ObjectId(restaurantId))), new Update().push("menuItems", newMenuItem), Restaurant.class);} Atomic, in-place update of document
  61. 61. Geospatial example 1 case class FriendRecord(id : String, name : String,@Component location : Point)class MongoFriendService extends FriendService { @Autowired var mongoTemplate: MongoTemplate = _ Collection name @PostConstruct def createGeoIndex { val dbo = new BasicDBObject dbo.put("location", "2d") mongoTemplate.getCollection("friendRecord").ensureIndex(dbo) } Create geospatial 2d index
  62. 62. Geospatial example 2 - finding nearby@Componentclass MongoFriendService extends FriendService { override def findNearbyFriends(request: NearbyFriendsRequest) = { val location = new Point(request.longitude, request.latitude) val distance = new Distance(3, Metrics.MILES) val query = NearQuery.near(location).maxDistance(distance) val result = mongoTemplate.geoNear(query, classOf[FriendRecord]) val nearby = result.getContent.map(_.getContent) FindNearbyFriendsResponse(nearby.map(f => FriendInfo(f.name, f.id))) }
  63. 63. Callbacks – access driver API with exception translation Exceptions are translated@Testpublic void testDbCallback() { Restaurant ajanta = makeAjantaRestaurant(); restaurantRepository.add(ajanta); assertCollectionExists("restaurants2");} private Void assertCollectionExists(final String collectionName) { return mongoTemplate.execute(new DbCallback<Void>(){ @Override public Void doInDB(DB db) { Set<String> collectionNames = db.getCollectionNames(); Assert.assertTrue("Missing from " + collectionNames, collectionNames.contains(collectionName)); return null; }}); }
  64. 64. Defining a Mongo Generic Repositorypublic class Person { private ObjectId id; private String firstname; private String lastname;… getters and setters}interface PersonRepository extends MongoRepository<Person, ObjectId> { List<Person> findByLastname(String lastName);}Person p = new Person("John", "Doe");personRepository.save(p);Person p2 = personRepository.findOne(p.getId());List<Person> johnDoes = personRepository.findByLastname("Doe");assertEquals(1, johnDoes.size());
  65. 65. Mongo Repository configuration<bean><mongo:repositories base-package="net.chrisrichardson.mongodb.example.mongorepository" mongo-template-ref="mongoTemplate" /></beans> Scans classpath looking for subtypes of MongoRepository in the base package
  66. 66. Richer mapping@Document(collection=”people”) Annotations define mapping: @Document, @Id, @Indexed,public class Person { @PersistanceConstructor, @CompoundIndex, @DBRef, @GeoSpatialIndexed, @Value @Id private ObjectId id; Map fields instead of properties no getters or setters private String firstname; required @Indexed private String lastname; Non-default constructor Index generation @PersistenceConstructor public Person(String firstname, String lastname) { this.firstname = firstname; this.lastname = lastname; }….}
  67. 67. Richer mapping configuration@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration { private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;private @Value("#{mongoDbProperties.host}")String mongoDbHost; Defines MongoTemplate bean@Overridepublic Mongo mongo() throws Exception { return new Mongo(mongoDbHost);}@Override Configures classpath scanningpublic String getDatabaseName() { return mongoDbDatabase;} @Override public String getMappingBasePackage() { return Person.class.getPackage().getName(); }}
  68. 68. Support for the QueryDSL project Generated from Type-safe domain model class composable queriesQPerson person = QPerson.person;Predicate predicate = person.homeAddress.street1.eq("1 High Street") .and(person.firstname.eq("John"))List<Person> people = personRepository.findAll(predicate);assertEquals(1, people.size());assertPersonEquals(p, people.get(0));
  69. 69. Cross-store/polyglot persistence Person person = new Person(…);@Entity entityManager.persist(person);public class Person { // In Database Person p2 = entityManager.find(…) @Id private Long id; private String firstname; private String lastname;// In MongoDB@RelatedDocument private Address address; { "_id" : ObjectId(”….."), "_entity_id" : NumberLong(1), "_entity_class" : "net.. Person", "_entity_field_name" : "address", "zip" : "94611", "street1" : "1 High Street", …}
  70. 70. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  71. 71. Using Mongo and Redis with Cloud Foundry• Create a Mongo or Redis service• Bind the service to your application• Access the service • Via auto-reconfiguration • Using <cloud:*/> namespace
  72. 72. Creating a Redis Server
  73. 73. Deploying a Redis application
  74. 74. Redis bean definitions
  75. 75. USING THE APPLICATION
  76. 76. About <cloud:redis-connection-factory/><cloud:redis-connection-factory id="redisConnectionFactory" Use when multiple service-name="redis1" services are bound/>
  77. 77. Deploying a Mongo application
  78. 78. MongoDB bean definitions
  79. 79. Using the Mongo Application
  80. 80. About <cloud:mongo-db-factory/> Use when multiple services are bound<cloud:mongo-db-factory id="mongoFactory" service-name="mongo1" > <cloud:mongo-options connections-per-host="..." ] max-wait-time="..." /></cloud:mongo-db-factory>
  81. 81. NoSQL and Caldecott• Caldecott let’s you tunnel to a NoSQL service• Use Redis CLI • Explore database, adhoc operations • ...• Use Mongo CLI etc • Explore database, adhoc operations • Mongo dump/restore • ...
  82. 82. Summary• NoSQL databases sometimes offer a combination of: • Higher scalability and performance • Schema less, richer data models• Spring Data simplifies the development of NoSQL applications• Cloud Foundry supports Mongo and Redis
  83. 83. @crichardson chris.richardson@springsource.com http://plainoldobjects.com Questions?Sign up for CloudFoundry.com
  84. 84. Cloud Foundry 启动营在www.cloudfoundry.com注册账号并成功上传应用程序,即可于12月8日中午后凭账号ID和应用URL到签到处换取Cloud Foundry主题卫衣一件。84
  85. 85. iPhone5 等你拿第二天大会结束前,请不要提前离 ,将填写完整的意见反馈表投到签到处的抽奖箱内,即可参与“iPhone5”抽奖活动。85
  86. 86. Birds of a Feather 专家面对面所有讲师都会在课程结束后,到紫兰厅与来宾讨论课程上的问题86

×