Your SlideShare is downloading. ×
Using Spring with NoSQL databases (SpringOne China 2012)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Using Spring with NoSQL databases (SpringOne China 2012)

3,788
views

Published on

A look at how Spring Data simplifies the development of applications that use NOSQL databases such as Redis and MongoDB.

A look at how Spring Data simplifies the development of applications that use NOSQL databases such as Redis and MongoDB.

Published in: Technology

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,788
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
75
Comments
0
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Using Spring with NoSQL databasesChris Richardson,Author of POJOs in Action, Founder of the original CloudFoundry.com @crichardson chris.richardson@springsource.com http://plainoldobjects.com/
  • 2. Presentation goal NoSQL databases: what, why and how How Spring Data simplifies thedevelopment of NoSQL applications
  • 3. About Chris
  • 4. (About Chris)
  • 5. About Chris()
  • 6. About Chris
  • 7. About Chrishttp://www.theregister.co.uk/2009/08/19/springsource_cloud_foundry/
  • 8. vmc push About-Chris Developer AdvocateSignup at http://cloudfoundry.com
  • 9. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  • 10. Relational databases are great...• SQL • Well supported • High-level • JDBC • Sorting • Hibernate/JPA • Aggregation • Spring• ACID semantics • Well understood • Developers • Operators
  • 11. ... but they have limitations• Object/relational impedance mismatch• Complicated to map rich domain model to relational schema• Difficult to handle semi-structured data, e.g. varying attributes• Schema changes• Extremely difficult/impossible to scale• Poor performance for some use cases
  • 12. Solution: Spend Money ORhttp://upload.wikimedia.org/wikipedia/commons/e/e5/Rising_Sun_Yacht.JPG• Hire more DevOps• Use application-level sharding• Build your own middleware• … http://www.trekbikes.com/us/en/bikes/road/race_performance/madone_5_series/madone_5_2/#
  • 13. Solution: Use NoSQL Benefits• Higher performance• Higher scalability• Richer data-model• Schema-less Drawbacks • Limited transactions • Relaxed consistency • Unconstrained data
  • 14. Growing in popularity…
  • 15. But don’t get too excited
  • 16. Solution: Use NewSQL• Relational databases with SQL and ACID transactions AND• New and improved architecture - designed for modern hardware• Radically better scalability and performance• NewSQL vendors: Gemfire/SQLFire, VoltDB, ...
  • 17. Future = multi-paradigm data storage for enterprise applications IEEE Software Sept/October 2010 - Debasish Ghosh / Twitter @debasishg
  • 18. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  • 19. Redis• Advanced key-value store K1 V1• Written in C K2 V2• Very fast, e.g. 100K reqs/sec• Optional persistence ... ...• Transactions with optimistic locking• Master-slave replication• Sharding using client-side consistent hashing
  • 20. Using Redis (via CLI)Datatypes: redis 127.0.0.1:6379> set foo 1 OK•Strings redis 127.0.0.1:6379> get foo•Hashes "1"•Maps redis 127.0.0.1:6379> sadd myset a•Lists (integer) 1•Sets redis 127.0.0.1:6379> sadd myset b (integer) 1•Sorted sets redis 127.0.0.1:6379> smembers myset 1) "a" 2) "b" redis 127.0.0.1:6379> srem myset a (integer) 1 redis 127.0.0.1:6379> smembers myset 1) "b"
  • 21. Redis use cases• Replacement for Memcached • Handling tasks that overload an RDBMS • Session state • Hit counts - INCR • Cache of data retrieved from system of • Most recent N items - LPUSH and LTRIM record (SOR) • Randomly selecting an item –• Replica of SOR for queries needing high- SRANDMEMBER performance • Queuing – Lists with LPOP, RPUSH, …. • High score tables – Sorted sets and ZINCRBY • …
  • 22. MongoDB• Document-oriented database • Geospatial queries • JSON-style documents: objects, lists, • Grid FS provides file storage primitives • Very fast, asynchronous writes • Schema-less • Highly scalable and available• Transaction = update of a single document• Rich query language for dynamic queries
  • 23. Data model = Binary JSON documents Server Database: Food To Go Collection: Restaurants { "name" : "TGI Fridays", "type" : ”American", "serviceArea" : [ "94619", "94618" ], Sequence of "openingHours" : [ bytes on disk { è fast i/o "dayOfWeek" : "Wednesday", "open" : 1730, "close" : 2230 } ], "_id" : ObjectId("4bddc2f49d1505567c6220a0") }
  • 24. MongoDB CLI> r = {name: Ajanta}> db.restaurants.save(r)> r{ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r = db.restaurants.findOne({name:"Ajanta"}){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "name" : "Ajanta" }> r.type= "Indian”> db.restaurants.save(r)> db.restaurants.update({name:"Ajanta"}, {$set: {name:"Ajanta Restaurant"}, $push: { menuItems: {name: "Chicken Vindaloo"}}})> db.restaurants.find(){ "_id" : ObjectId("4e555dd9646e338dca11710c"), "menuItems" : [ { "name" : "Chicken Vindaloo" } ], "name" : "Ajanta Restaurant", "type" : "Indian" }> db.restaurants.remove(r.id)
  • 25. MongoDB query by example{ serviceArea:"94619", Find a restaurant that openingHours: { serves the 94619 zip $elemMatch : { code and is open at 6pm "dayOfWeek" : "Monday", "open": {$lte: 1800}, on a Monday "close": {$gte: 1800} } }}DBCursor cursor = collection.find(qbeObject);while (cursor.hasNext()) { DBObject o = cursor.next(); … }
  • 26. Scaling MongoDB Shard 1 Shard 2 Mongod Mongod (replica) (replica) Mongod Mongod Mongod Mongod (master) (replica) (master) (replica)Config Server Mongos mongod Mongos mongod Client mongod
  • 27. MongoDB use cases• Use cases • Who is using it? • High volume writes • Shutterfly, Foursquare • Complex data • Bit.ly Intuit • Semi-structured data • SourceForge, NY Times • GILT Groupe, Evite, • SugarCRM
  • 28. Other NoSQL databasesType ExamplesExtensible columns/Column-oriented Hbase e SimpleDB, DynamoDB r it avo Cassandra r f youGraph Neo4j t t ouKey-value I lef Voldemort, Riak ifDocument So rry CouchDb http://nosql-database.org/ lists 122+ NoSQL databases
  • 29. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  • 30. Spring Data is here to help For NoSQL databaseshttp://www.springsource.org/spring-data
  • 31. Spring Data sub-projects• Relational • QueryDSL • JPA • Big Data • JDBC Extensions • Hadoop• NoSQL • HDFS and M/R • Redis • Hive • Mongo • Pig • HBase • Cascading • Neo4j • Splunk • Gemfire • Access • Lucene • REST
  • 32. What you get• Template classes that hide the boilerplate code• Auto-generated (generic) repositories for some NOSQL databases• Java NoSQL mapping• Cross Store Persistence• Support in Roo
  • 33. Get the book!
  • 34. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  • 35. Redis challenges• Connection management: need to get and reliably close connections• Data mapping: application objects Redis binary/strings• Multiple client libraries with gratuitously different APIs
  • 36. Spring Data for Redis• Low-level - RedisConnection(Factory) • Connection management • Supports Jedis, Jredis, Rjc and Srp • Pluggable Java binary conversion • Insulates client code from underlying • Support classes: library • Collections-backed by RedisTemplate• High-level - RedisTemplate • Atomic Counters • Builds on RedisConnection(Factory) • Support for Redis pub/sub
  • 37. Low-level API = RedisConnection(Factory)
  • 38. Using RedisConnectionFactorypublic class LowLevelRedisTest { @Autowired private RedisConnectionFactory redisConnectionFactory; @Test public void testLowLevel() { Library independent code J RedisConnection con = null; try { con = redisConnectionFactory.getConnection(); byte[] key = "foo".getBytes(); Ugly byte arrays L byte[] value = "bar".getBytes(); con.set(key, value); byte[] retrievedValue = con.get(key); Assert.assertArrayEquals(value, retrievedValue); } finally { if (con != null) { con.close(); } Need to clean up L } }
  • 39. Configuring RedisConnectionFactory@Configurationpublic class RedisConfiguration { @Value("${databaseHostName}") protected String databaseHostName; @Bean public RedisConnectionFactory jedisConnectionFactory() { JedisConnectionFactory factory = new JedisConnectionFactory(); factory.setHostName(databaseHostName); factory.setPort(6379); factory.setUsePool(true); return factory; }}
  • 40. High-level API = RedisTemplate• Builds on RedisConnection(Factory) • Maps Redis exceptions DataAccessException• Analogous to JdbcTemplate • StringRedisTemplate• Parameterized type • Extends RedisTemplate<String, • K - Key type String> • V – Value type • Keys and values are Strings• Handles Java Key/Value Redis byte[]
  • 41. Using StringRedisTemplatepublic class RedisTemplateTest { @Autowired private StringRedisTemplate stringRedisTemplate; Returns KV type specific interface @Test public void testGetAndSet() { stringRedisTemplate.opsForValue().set("foo", "bar"); assertEquals("bar", stringRedisTemplate.opsForValue().get("foo")); } @Test Converts between Strings and byte[] public void testHashOps() { stringRedisTemplate.opsForHash().put("myHash", "myKey", "value"); assertEquals("value", stringRedisTemplate.opsForHash().get("myHash", "myKey")); assertEquals(Collections.singleton("myKey"), stringRedisTemplate.opsForHash().keys("myHash")); assertEquals(Collections.singletonMap("myKey", "value"), stringRedisTemplate.opsForHash().entries("myHash")); }
  • 42. Configuring StringRedisTemplate@Configurationpublic class RedisConfiguration { @Bean public RedisConnectionFactory jedisConnectionFactory() { … } @Bean public StringRedisTemplate stringRedisTemplate(RedisConnectionFactory factory) { StringRedisTemplate template = new StringRedisTemplate(); template.setConnectionFactory(factory); return template; }}
  • 43. RedisTemplate: Java objects binary data• DefaultSerializer - defaults to JdkSerializationRedisSerializer• KeySerializer• ValueSerializer• HashKeySerializer• HashValueSerializer
  • 44. StringRedisTemplate uses StringRedisSerializer• Stores keys and values as Strings
  • 45. Register serializers to override the default behavior Converted to JSON by RedisTemplate
  • 46. Redis-backed Collections The key@Testpublic void testRedisSet() { Set<String> mySet = new DefaultRedisSet<String>("mySet", stringRedisTemplate); Assert.assertTrue(mySet.isEmpty()); mySet.add("a"); assertEquals(Collections.singleton("a"), stringRedisTemplate.opsForSet().members("mySet"));}
  • 47. Redis Atomic Counters
  • 48. Redis Pub/Sub - Consumer public class MyRedisSubscriber {@Configuration String message;public class RedisConfiguration { void handleMessage(String text) { @Bean this.message = text; public MyRedisSubscriber myListener() { return new MyRedisSubscriber(); } } } @Bean public RedisMessageListenerContainer redisMessageListenerContainer( RedisConnectionFactory redisConnectionFactory, MyRedisSubscriber myListener) { RedisMessageListenerContainer c = new RedisMessageListenerContainer(); c.setConnectionFactory(redisConnectionFactory); c.addMessageListener(new MessageListenerAdapter(myListener), new ChannelTopic("myChannel")); return c; }
  • 49. Redis Pub/Sub - Producerpublic class RedisPubSubTest { @Autowired private StringRedisTemplate stringRedisTemplate; @Autowired private MyRedisSubscriber myListener; @Test public void testPubSub() throws InterruptedException { … stringRedisTemplate.convertAndSend("myChannel", "hello"); TimeUnit.SECONDS.sleep(4); Assert.assertEquals("hello", myListener.message); }}
  • 50. Redis caching support KVs = <prefix + K, V>Template needs to (de)serialize K and V Sorted set of all keys for clear()
  • 51. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  • 52. MongoDB API usage patterns• Create and store Mongo singleton • Queries• Externalized server host, port etc. • Construct query object• Inserts/Updates • mongo.getDatabase(…).getCollection(…) • Map application POJO DBObject • Iterate through Cursor • Map DBObject application POJO • mongo.getDatabase(…).getCollection(…) • Partial document updates • Configure asynchronous vs. synchronous writes Higher-level than JDBC but still repetitive, …
  • 53. Spring Data - MongoDB• MongoTemplate • Map-Reduce integration• Query, Criteria, and Update DSLs • GridFS support• Generic repositories• Querydsl integration• Cross-store persistence• GeoSpatial integration
  • 54. MongoTemplate MongoTemplate POJO ó DBObject databaseNameSimplifies data access userId mapping PasswordTranslates exceptions defaultCollectionName writeConcern writeResultChecking save() insert() remove() <<interface>> updateFirst() MongoConvertor findOne() write(Object, DBObject) find() read(Class, DBObject) … uses Mongo MongoMapping (Java Driver class) Converter
  • 55. Example entitypublic class Restaurant { private String id; public class MenuItem { private String name; private String name; private double price; private List<MenuItem> menuItems; public MenuItem(String name, double price) { public Restaurant(String name) { this.name = name; this.name = name; this.price = price; } … } ........} Spring Data uses fields and non- default constructors
  • 56. Example data access code@Repositorypublic class RestaurantRepository { @Autowired private MongoTemplate mongoTemplate; public void add(Restaurant restaurant) { mongoTemplate.save(restaurant); } public List<Restaurant> findRestaurantsByName(String restaurantName) { return mongoTemplate.find( query(where("name").is(restaurantName)), Restaurant.class); }
  • 57. A document in the restaurant collection{ "_id" : ObjectId("4d977f55d3fe3119c904e026"), "_class" : "net.chrisrichardson.mongodb.example.mongotemplate.Restaurant", "name" : "Ajanta" "menuItems" : [ { "name" : "Tandoori Portobello Mushrooms", "price" : 5.5 }, { "name" : "Duck Curry Kerala", "price" : 15 } ]}
  • 58. Spring MongoDB Example - Config 1@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration { private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase; private @Value("#{mongoDbProperties.host}") Defines a String mongoDbHost; MongoTemplate public Mongo mongo() throws Exception { return new Mongo(mongoDbHost); } @Override <beans> protected String getDatabaseName() { <context:annotation-config/> return mongoDbDatabase; } <context:component-scan base-package="net.chrisrichardson.mongodb.example"/>... <util:properties id="mongoDbProperties" location="mongodb.properties"/>} External Config </beans>mongodb.properties: databaseName=demo1 host=192.168.253.150
  • 59. Spring MongoDB Example - Config 2<bean id="mongoTemplate" class="org.springframework.data.mongodb.core.MongoTemplate"> <constructor-arg ref="mongoFactory"/></bean><mongo:db-factory id="mongoFactory" host= "#{mongoDbProperties.host}" dbname="#{mongoDbProperties.databaseName}" /><util:properties id="mongoDbProperties" location="mongodb.properties"/>
  • 60. Update example@Repositorypublic class RestaurantRepository { public void addMenuItem(String restaurantId, MenuItem newMenuItem) { mongoTemplate.updateFirst( query(where("_id").is(new ObjectId(restaurantId))), new Update().push("menuItems", newMenuItem), Restaurant.class);} Atomic, in-place update of document
  • 61. Geospatial example 1 case class FriendRecord(id : String, name : String,@Component location : Point)class MongoFriendService extends FriendService { @Autowired var mongoTemplate: MongoTemplate = _ Collection name @PostConstruct def createGeoIndex { val dbo = new BasicDBObject dbo.put("location", "2d") mongoTemplate.getCollection("friendRecord").ensureIndex(dbo) } Create geospatial 2d index
  • 62. Geospatial example 2 - finding nearby@Componentclass MongoFriendService extends FriendService { override def findNearbyFriends(request: NearbyFriendsRequest) = { val location = new Point(request.longitude, request.latitude) val distance = new Distance(3, Metrics.MILES) val query = NearQuery.near(location).maxDistance(distance) val result = mongoTemplate.geoNear(query, classOf[FriendRecord]) val nearby = result.getContent.map(_.getContent) FindNearbyFriendsResponse(nearby.map(f => FriendInfo(f.name, f.id))) }
  • 63. Callbacks – access driver API with exception translation Exceptions are translated@Testpublic void testDbCallback() { Restaurant ajanta = makeAjantaRestaurant(); restaurantRepository.add(ajanta); assertCollectionExists("restaurants2");} private Void assertCollectionExists(final String collectionName) { return mongoTemplate.execute(new DbCallback<Void>(){ @Override public Void doInDB(DB db) { Set<String> collectionNames = db.getCollectionNames(); Assert.assertTrue("Missing from " + collectionNames, collectionNames.contains(collectionName)); return null; }}); }
  • 64. Defining a Mongo Generic Repositorypublic class Person { private ObjectId id; private String firstname; private String lastname;… getters and setters}interface PersonRepository extends MongoRepository<Person, ObjectId> { List<Person> findByLastname(String lastName);}Person p = new Person("John", "Doe");personRepository.save(p);Person p2 = personRepository.findOne(p.getId());List<Person> johnDoes = personRepository.findByLastname("Doe");assertEquals(1, johnDoes.size());
  • 65. Mongo Repository configuration<bean><mongo:repositories base-package="net.chrisrichardson.mongodb.example.mongorepository" mongo-template-ref="mongoTemplate" /></beans> Scans classpath looking for subtypes of MongoRepository in the base package
  • 66. Richer mapping@Document(collection=”people”) Annotations define mapping: @Document, @Id, @Indexed,public class Person { @PersistanceConstructor, @CompoundIndex, @DBRef, @GeoSpatialIndexed, @Value @Id private ObjectId id; Map fields instead of properties no getters or setters private String firstname; required @Indexed private String lastname; Non-default constructor Index generation @PersistenceConstructor public Person(String firstname, String lastname) { this.firstname = firstname; this.lastname = lastname; }….}
  • 67. Richer mapping configuration@Configurationpublic class MongoExampleConfig extends AbstractMongoConfiguration { private @Value("#{mongoDbProperties.databaseName}") String mongoDbDatabase;private @Value("#{mongoDbProperties.host}")String mongoDbHost; Defines MongoTemplate bean@Overridepublic Mongo mongo() throws Exception { return new Mongo(mongoDbHost);}@Override Configures classpath scanningpublic String getDatabaseName() { return mongoDbDatabase;} @Override public String getMappingBasePackage() { return Person.class.getPackage().getName(); }}
  • 68. Support for the QueryDSL project Generated from Type-safe domain model class composable queriesQPerson person = QPerson.person;Predicate predicate = person.homeAddress.street1.eq("1 High Street") .and(person.firstname.eq("John"))List<Person> people = personRepository.findAll(predicate);assertEquals(1, people.size());assertPersonEquals(p, people.get(0));
  • 69. Cross-store/polyglot persistence Person person = new Person(…);@Entity entityManager.persist(person);public class Person { // In Database Person p2 = entityManager.find(…) @Id private Long id; private String firstname; private String lastname;// In MongoDB@RelatedDocument private Address address; { "_id" : ObjectId(”….."), "_entity_id" : NumberLong(1), "_entity_class" : "net.. Person", "_entity_field_name" : "address", "zip" : "94611", "street1" : "1 High Street", …}
  • 70. Agenda• Why NoSQL?• Overview of NoSQL databases• Introduction to Spring Data• Using Spring Data for Redis• Using Spring Data for Mongo• Deploying on Cloud Foundry
  • 71. Using Mongo and Redis with Cloud Foundry• Create a Mongo or Redis service• Bind the service to your application• Access the service • Via auto-reconfiguration • Using <cloud:*/> namespace
  • 72. Creating a Redis Server
  • 73. Deploying a Redis application
  • 74. Redis bean definitions
  • 75. USING THE APPLICATION
  • 76. About <cloud:redis-connection-factory/><cloud:redis-connection-factory id="redisConnectionFactory" Use when multiple service-name="redis1" services are bound/>
  • 77. Deploying a Mongo application
  • 78. MongoDB bean definitions
  • 79. Using the Mongo Application
  • 80. About <cloud:mongo-db-factory/> Use when multiple services are bound<cloud:mongo-db-factory id="mongoFactory" service-name="mongo1" > <cloud:mongo-options connections-per-host="..." ] max-wait-time="..." /></cloud:mongo-db-factory>
  • 81. NoSQL and Caldecott• Caldecott let’s you tunnel to a NoSQL service• Use Redis CLI • Explore database, adhoc operations • ...• Use Mongo CLI etc • Explore database, adhoc operations • Mongo dump/restore • ...
  • 82. Summary• NoSQL databases sometimes offer a combination of: • Higher scalability and performance • Schema less, richer data models• Spring Data simplifies the development of NoSQL applications• Cloud Foundry supports Mongo and Redis
  • 83. @crichardson chris.richardson@springsource.com http://plainoldobjects.com Questions?Sign up for CloudFoundry.com
  • 84. Cloud Foundry 启动营在www.cloudfoundry.com注册账号并成功上传应用程序,即可于12月8日中午后凭账号ID和应用URL到签到处换取Cloud Foundry主题卫衣一件。84
  • 85. iPhone5 等你拿第二天大会结束前,请不要提前离 ,将填写完整的意见反馈表投到签到处的抽奖箱内,即可参与“iPhone5”抽奖活动。85
  • 86. Birds of a Feather 专家面对面所有讲师都会在课程结束后,到紫兰厅与来宾讨论课程上的问题86