Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

BreizhCamp 2013 - Pimp my backend


Published on

Our presentation at BreizhCamp 2013

Published in: Technology
  • Be the first to comment

  • Be the first to like this

BreizhCamp 2013 - Pimp my backend

  1. 1. By @FinistSeb and @LostInBrittany
  2. 2. Sebastien Lambour Le développeur de l'Est le plus à l'Ouest ! Java & C++ Developer, coding addict, continuous intégration ayatollah ● Senior Software Engineer currently, Crédit Mutuel Arkea ● FinistJug actif member BreizhBeans community and commiter +sebastien.lambour @FinistSeb
  3. 3. Horacio Gonzalez Spaniard lost in Brittany, Java developer, dreamer and all-around geek ● Senior Software Engineer at Crédit Mutuel Arkea ○ Currently... ● Leader of the FinistJUG and the BreizhBeans community +Horacio.Gonzalez @LostInBrittany
  4. 4. BreizhBeans Developer community at the far end of French Brittany @BreizhBeans
  5. 5. Why ? I help to create a startup on social translation, would you code with me ? Great, I do the backend! How many members? Some 30.000 - 50.000 will be fine! Ouch, is it not too small? I like big toys!!!! Nexus 4 Me too! Let's think big then OK, it will rock !
  6. 6. Backends They often look like that !
  7. 7. Transforming it is HARD ! Next challenge : Granite aircraft !!
  8. 8. A sexy backend? Let's groom it! Light weight Data Driven Java Cloud Ready OK, go around... Test driven Scalable
  9. 9. Following the beaten track Brainless route JSF 2.0 Hibernate Spring CRUD (Create, Read, Update,Delete) SQL JPA CXF The usual suspects...
  10. 10. Booooooring... And ineffective Let's use it!
  11. 11. Design choices Because we need some solid roots
  12. 12. Core principles What do we want our backend to be? ● Scalability ○ ● Adaptability ○ ● Because you want to be sure that your code works! NoSQL (meaning Not Only SQL) ○ ● Because your business evolves, and your backend need to evolve with it! Testability ○ ● Because traffic growth should be a blessing, not a problem! Because you need not only tables and foreign keys, but non-SQL stores, search indexes, caches... Productivity ○ Because the other 4 doesn't matter if you need to spend 4 working days to modify a service...
  13. 13. A first choice : Apache Thrift « Software framework for scalable and performant crosslanguage services development » ● ● ● ● ● Interface description language Multi-language code generation Serialisation/deserialisation RPC Framework High performance ○ On transport layer ○ On data manipulation
  14. 14. Thrift, it's cool Thrift boots productivity!
  15. 15. A second choice : MongoDB ● We needed a NO-SQL store for documents ● We wanted to use Cassandra ○ Cassandra & Thrift, a love story ● We wanted a PaaS-ready backend ○ Neither Cloudbees nor Cloudfoundry supported Cassandra ○ But they supported MongoDB ● We loved MongoDB ○ Document DB are great for document oriented applications
  16. 16. ThriftMongoBridge Because we wanted both Thrift and MongoDB
  17. 17. Thrift rocks and works But... How to put Thrift objects into MongoDB ?
  18. 18. First bad idea! Use Spring Data or Morphia Thrift object mapping is bad !
  19. 19. Damned! First bad idea!
  20. 20. Second bad idea ! Use the Mongo Driver JSON parsing Deserialisation will be a bit hard...
  21. 21. Damned ! Second bad idea!
  22. 22. Thrift isn't cool anymore ? We need a new idea !
  23. 23. Thrift, it's cool ! But we had work to do... Write a TBSONProtocol ! The ThriftMongoBridge is born !
  24. 24. What does it do ? POJO generated by the Thrift compiler public class AnotherThrift implements org.apache.thrift.TBase... { ... public String anotherString; public int anotherInteger; Thrift IDL struct AnotherThrift { 1:string anotherString, 2:i32 anotherInteger, } Compile to ... public static final Map<_Fields, org.apache.thrift.meta_data. FieldMetaData> metaDataMap; .... public void read(org.apache.thrift.protocol.TProtocol iprot)... ... public void write(org.apache.thrift.protocol.TProtocol oprot) ......; } TBSonSerializer TBSonProtocol Write To BSON TBSonDeSerializer Read from BSON {anotherString:"value", anotherInteger:69}
  25. 25. Really simple to use Too simple ? AnotherThrift anotherThrift = new AnotherThrift(); anotherThrift.setAnotherString("value"); anotherThrift.setAnotherInteger(69); // serialize into DBObject DBObject dbObject = ThriftMongoHelper.thrift2DBObject(anotherThrift); {anotherString:"value", anotherInteger:69}
  26. 26. How it works write(protocol) Thrift object generated TBSONSerializer TBase getResult sentense directives (begin / end) push Data (read / write) pop Struct context DBObject Context Stack TBSONProtocol DBObject (BSON) MongoDB object
  27. 27. The BreizhCamp demo Because you want to see some code, don't you?
  28. 28. The BreizhCamp demo Given this Thrift model & service struct BreizhCampEvent { 1:i64 date, 2:string startTime, 3:string endTime, 4:com.breizhbeans.backend.thrift.enums.model.EventTypes type, 5:string room, 6:string resume, 7:string details, 8:list<string> speakers 9:list<com.breizhbeans.backend.thrift.enums.model.Track> tracks, } service ProgramService { com.....GetProgramResponse getProgram(1:com.....GetProgramRequest request) throws (1:com....TechnicalException texp, 2:com...FunctionalException fexp) } struct GetProgramRequest { 1:com.breizhbeans.backend.thrift.enums.model.Track track, 2:com.breizhbeans.backend.thrift.enums.model.EventTypes type, 3:string room, 4:list<string> speakers, } struct GetProgramResponse { 1:list<com.breizhbeans.backend.thrift.model.BreizhCampEvent> events, }
  29. 29. The BreizhCamp demo First : Write a test ● One loader by collection ● One goal : Have Human readable test ;-) ● Manage automatically data insertion before test and clean up after test Application code to test @Test public void testGetEvents() throws Exception { List<BreizhCampEvent> actualEvents = programDao.getEvent(); List<BreizhCampEvent> expectedEvents = new ArrayList<BreizhCampEvent>(); expectedEvents.add(getEventPimpMyBackend()); Assert.assertEquals(expectedEvents, actualEvents); } Expected data building (used by the test and the data loader) Standard JUnit assertion
  30. 30. The BreizhCamp demo Write a DAO public List<BreizhCampEvent> getEvent() throws TechnicalException { DBCursor cursor = null; try { List<BreizhCampEvent> events = new ArrayList<BreizhCampEvent>(); DBCollection collection = db.getCollection(collectionName); DBObject query = new BasicDBObject(); // execute the query retrieve DB collection cursor = collection.find(query); while(cursor.hasNext()) { Build and execute the Query events.add( (BreizhCampEvent) ThriftMongoHelper.DBObject2Thrift(; } return events; } catch (Exception e) { throw new H4TTechnicalException("Unexpected error", e, null); Deserialize Thrift objects } finally { cursor.close(); } Error handling } Never forget it ;-)
  31. 31. The BreizhCamp demo Can I update thrift Object by code? @Override public BreizhCampEvent updateRoom(final BreizhCampEvent event) throws TechnicalException { try { BasicDBObject updateQuery = new BasicDBObject(); Mongo Update Query updateQuery.append("$set", new BasicDBObject() .append(getFields(BreizhCampEvent._Fields.ROOM), event.getRoom())); db.getCollection(collectionName) .update(getPrimaryKeyRequest(event), updateQuery) .getLastError() Thrift structure mapping .throwOnError(); return event; } catch (Exception e) { Query execution throw new H4TTechnicalException("Unexpected error", e, ObjectUtils.toString(event)); } } Error handling
  32. 32. The BreizhCamp demo Can I update thrift Object by script? Mongo DB Bson writed : > db.program.find(); { "_id" : "id=10", "date" : NumberLong("-56433978000000"), "startTime" : "14:45", "endTime" : "15:40", "type" : 4, "room" : "Bréhat", "resume" : "Pimp my backend!", "details" : "Le frontend, le frontend...... ...", "speakers" : [ "Sébastien Lambour", "Horacio Gonzalez" ], "tracks" : [ 7 ], "id" : 10, "classname" : "com.breizhbeans.backend.thrift.model. BreizhCampEvent" } Update the field room :> db.program.update(); { "id": 10}, { $set: { "room": "Ouessant" } }
  33. 33. And the perfs? Because speed matters!
  34. 34. Talking about perfs The candidates are ... Spring Data Morphia And... ThriftMongoBridge, of course
  35. 35. Spring Data vs ThriftMongoBridge ● Write and Read 500 objects into MongoDB. ● Each document is composed with a List<> and a Set<> of 500 objects. ● The document have a JSON size of 91613 bytes Who will win ?
  36. 36. Code ThriftMongoBridge First // Get a collection DBCollection collection = db.getCollection("dummyColl"); Write // Serialize the Thrift object DBObject dbObject = ThriftMongoHelper.thrift2DBObject(thriftObject); // Put the document collection.insert(dbObject); Read DBCursor cursorDoc = collection.find(); while (cursorDoc.hasNext()) { DBObject dbObject =; ThriftObject thirftObject = (ThriftObject)ThriftMongoHelper.DBObject2Thrift(dbObject); }
  37. 37. Code Spring Data First // Retrieve the MongoOperations bean MongoOperations mongoOperation = (MongoOperations) ctx.getBean("mongoTemplate"); Write // Serialize the Thrift object, "springObject"); Read List<SpringObject> listUser = mongoOperation.findAll(SpringObject.class, "springObject");
  38. 38. Time to run !
  39. 39. Thrift over MongoDB rocks! We need a logo for our OpenSource library... Available now : 2.0
  40. 40. A storage library a backend is not, young padawan Because we need more things
  41. 41. At the beginning... Pretty layers, neatly isolated, like a Care Bears' birthday cake
  42. 42. Later in the project Like a cake after an earthquake
  43. 43. And in the end... As pretty as a British pudding... and almost as tasty
  44. 44. Be Layer less Now build the Backend ! le b ha c rea un ● Each package have a goal ● NO package assembly rules ● Improve the complexity detection by dependencies analysis Goal : Fight against the "God classes" anti-pattern m ea dr
  45. 45. Thrift services Prevent protocol rupture, have a stable services signature. com.h4t.CheckAccessResponse checkAccess (1:com.h4t.CheckAccessRequest request) throws (1:com.h4t.TechnicalException texp, 2:com.h4t.FunctionalException fexp) stable method signature
  46. 46. Integrate this into the backend Thrift IFace implementation Request / Response decapsulation + log & monitoring Core Business code Thrift object read write ThriftMongoHelper Mongo Opérations Query DBObject DBObject Mongo Driver
  47. 47. Inversion of control Keep simple too ! Prefer static dependencies injection. Time to do the choice or
  48. 48. Inversion of control Why Because ? and ○ binding of the MongoDB service by the cloud ○ no production configuration required ○ only one war
  49. 49. Testability Because without tests you need faith
  50. 50. Test strategy No compromise! The rules are simple... no exception!
  51. 51. TDD is my religion 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Execute test without setup Execute test with your build tool Execute test into your IDE No injection into test framework Reuse only application injection Keep it simple (repeat it 7 times) Have a loose coupling (test the smallest parts) Test must be deterministic and reproducible Do pair review with the tests If the complexity is growing up refactor now ... 11. If a rule goes wrong, rewrite it !
  52. 52. From concept to reality Load thrift object with preset datas into MongoDB Get expected Thift object with preset datas Collection Loader Thrift DataSet ThriftMongoTestTools Get a Thrift object from MongoDB related to a unique key
  53. 53. Tests have goals DAO : Validate queries and set up the test data loaders IFACE : Functional tests SERVICE : Test a business centric service TU : Unit test not related to the business (test first approach)
  54. 54. Tests are Data Driven @Before public void setUp() throws Exception { Inject Data into MongoDB eventLoader.load(getEventPimpMyBackend()); } private BreizhCampEvent getEventPimpMyBackend() throws Exception { String details = "Le frontend, le frontend... ....."; BreizhCampEvent event = eventLoader.getExpected(10, "14/06/2013", "14:45", "15: 40", EventTypes.CONF, "Bréhat", "Pimp my backend!", details); event = eventLoader.addSpeakers(event, "Sébastien Lambour", "Horacio Gonzalez"); event = eventLoader.addTracks(event, Track.CLOUD); return event; } Build test Data with java code, files, BDD or the MongoThrift Test tools (not opensourced yet ;-) )
  55. 55. What's next? Because our work is not completed... yet!
  56. 56. Why integrate our lib ? #Pragmatism ● ● ● ● Productivity oriented Fully testable Container less (no servlet dependencies) Open Source : Apache 2.0 The ThriftMongoBackend Core model will born soon ! With a name: TINAF (ThriftMongoBackend Is Not A Framework)
  57. 57. What's next ? ● monitoring Thrift services in the cloud ● load tests with vmware & 10gen ● contribute TINAF ● Develop a thrift service valve to ensure brandwitdth for critical services
  58. 58. Thank you !