By @FinistSeb and @LostInBrittany
Sebastien Lambour
Le développeur de l'Est le plus à l'Ouest !
Java & C++ Developer, coding addict,
continuous intégration ...
Horacio Gonzalez
Spaniard lost in Brittany, Java developer,
dreamer and all-around geek
● Senior Software Engineer at Créd...
BreizhBeans

Developer community at the far end
of French Brittany

http://breizhbeans.org/
@BreizhBeans
Why ?

I help to create a startup
on social translation, would
you code with me ?

Great, I do the backend!
How many membe...
Backends
They often look like that !
Transforming it is HARD !
Next challenge :
Granite aircraft !!
A sexy backend?
Let's groom it!

Light weight

Data Driven

Java
Cloud Ready

OK, go around...

Test
driven
Scalable
Following the beaten track
Brainless route
JSF 2.0

Hibernate
Spring CRUD
(Create, Read, Update,Delete)

SQL

JPA
CXF

The...
Booooooring...
And ineffective

Let's use it!
Design choices
Because we need some solid roots
Core principles
What do we want our backend to be?
●

Scalability
○

●

Adaptability
○

●

Because you want to be sure tha...
A first choice :
Apache Thrift
« Software framework for scalable and performant crosslanguage services development »
●
●
●...
Thrift, it's cool
Thrift boots productivity!
A second choice :
MongoDB
● We needed a NO-SQL store for documents
● We wanted to use Cassandra
○ Cassandra & Thrift, a lo...
ThriftMongoBridge
Because we wanted both
Thrift and MongoDB
Thrift rocks and works
But...
How to put Thrift objects into MongoDB ?
First bad idea!
Use Spring Data or Morphia

Thrift object mapping is bad !
Damned!

First bad idea!
Second bad idea !
Use the Mongo Driver JSON parsing

Deserialisation will be a bit hard...
Damned !

Second bad idea!
Thrift isn't cool anymore ?
We need a new idea !
Thrift, it's cool !
But we had work to do...
Write a TBSONProtocol !

The ThriftMongoBridge is born !
What does it do ?
POJO generated by the Thrift compiler
public class AnotherThrift implements org.apache.thrift.TBase... {...
Really simple to use
Too simple ?
AnotherThrift anotherThrift = new AnotherThrift();
anotherThrift.setAnotherString("value...
How it works
write(protocol)

Thrift object generated

TBSONSerializer

TBase

getResult
sentense directives
(begin / end)...
The BreizhCamp demo
Because you want to see some code,
don't you?
The BreizhCamp demo
Given this Thrift model & service
struct BreizhCampEvent {
1:i64 date,
2:string startTime,
3:string en...
The BreizhCamp demo
First : Write a test

● One loader by collection
● One goal : Have Human readable test ;-)
● Manage au...
The BreizhCamp demo
Write a DAO
public List<BreizhCampEvent> getEvent() throws TechnicalException {
DBCursor cursor = null...
The BreizhCamp demo
Can I update thrift Object by code?
@Override
public BreizhCampEvent updateRoom(final BreizhCampEvent ...
The BreizhCamp demo
Can I update thrift Object by script?
Mongo DB Bson writed : > db.program.find();
{ "_id" : "id=10", "...
And the perfs?
Because speed matters!
Talking about perfs
The candidates are ...

Spring Data
Morphia
And...
ThriftMongoBridge, of course
Spring Data vs
ThriftMongoBridge
● Write and Read 500 objects into MongoDB.
● Each document is composed with a List<> and ...
Code
ThriftMongoBridge
First
// Get a collection
DBCollection collection = db.getCollection("dummyColl");

Write
// Serial...
Code
Spring Data
First
// Retrieve the MongoOperations bean
MongoOperations mongoOperation =
(MongoOperations) ctx.getBean...
Time to run !
Thrift over MongoDB rocks!
We need a logo for our OpenSource library...

Available now :

2.0
https://github.com/BreizhBea...
A storage library
a backend is not,
young padawan
Because we need more things
At the beginning...
Pretty layers, neatly isolated,
like a Care Bears' birthday cake
Later in the project
Like a cake after an earthquake
And in the end...
As pretty as a British pudding...
and almost as tasty
Be Layer less
Now build the Backend !

le

b
ha
c
rea

un

● Each package have a goal
● NO package assembly rules
● Improv...
Thrift services
Prevent protocol rupture, have a stable services signature.
com.h4t.CheckAccessResponse checkAccess
(1:com...
Integrate this into
the backend
Thrift IFace implementation
Request / Response decapsulation + log & monitoring

Core Busi...
Inversion of control
Keep simple too !
Prefer static dependencies injection.
Time to do the choice
or
Inversion of control
Why
Because

?
and

○ binding of the MongoDB service by the cloud
○ no production configuration requi...
Testability
Because without tests
you need faith
Test strategy
No compromise!
The rules are simple... no exception!
TDD is my religion
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.

Execute test without setup
Execute test with your build tool
Execute te...
From concept to reality
Load thrift object
with preset datas
into MongoDB

Get expected
Thift object
with preset
datas

Co...
Tests have goals
DAO : Validate queries and set up the test data loaders
IFACE : Functional tests
SERVICE : Test a busines...
Tests are Data Driven
@Before
public void setUp() throws Exception {

Inject Data into MongoDB

eventLoader.load(getEventP...
What's next?
Because our work is not completed... yet!
Why integrate our lib ?
#Pragmatism
●
●
●
●

Productivity oriented
Fully testable
Container less (no servlet dependencies)...
What's next ?
● monitoring Thrift services in the cloud
● load tests with vmware & 10gen
● contribute TINAF
● Develop a th...
Thank you !
Upcoming SlideShare
Loading in …5
×

BreizhCamp 2013 - Pimp my backend

483 views
401 views

Published on

Our presentation at BreizhCamp 2013

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
483
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

BreizhCamp 2013 - Pimp my backend

  1. 1. By @FinistSeb and @LostInBrittany
  2. 2. Sebastien Lambour Le développeur de l'Est le plus à l'Ouest ! Java & C++ Developer, coding addict, continuous intégration ayatollah ● Senior Software Engineer currently, Crédit Mutuel Arkea ● FinistJug actif member BreizhBeans community and commiter sebastien.lambour@gmail.com +sebastien.lambour @FinistSeb
  3. 3. Horacio Gonzalez Spaniard lost in Brittany, Java developer, dreamer and all-around geek ● Senior Software Engineer at Crédit Mutuel Arkea ○ Currently... ● Leader of the FinistJUG and the BreizhBeans community http://lostinbrittany.org/ +Horacio.Gonzalez @LostInBrittany
  4. 4. BreizhBeans Developer community at the far end of French Brittany http://breizhbeans.org/ @BreizhBeans
  5. 5. Why ? I help to create a startup on social translation, would you code with me ? Great, I do the backend! How many members? Some 30.000 - 50.000 will be fine! Ouch, is it not too small? I like big toys!!!! Nexus 4 Me too! Let's think big then OK, it will rock !
  6. 6. Backends They often look like that !
  7. 7. Transforming it is HARD ! Next challenge : Granite aircraft !!
  8. 8. A sexy backend? Let's groom it! Light weight Data Driven Java Cloud Ready OK, go around... Test driven Scalable
  9. 9. Following the beaten track Brainless route JSF 2.0 Hibernate Spring CRUD (Create, Read, Update,Delete) SQL JPA CXF The usual suspects...
  10. 10. Booooooring... And ineffective Let's use it!
  11. 11. Design choices Because we need some solid roots
  12. 12. Core principles What do we want our backend to be? ● Scalability ○ ● Adaptability ○ ● Because you want to be sure that your code works! NoSQL (meaning Not Only SQL) ○ ● Because your business evolves, and your backend need to evolve with it! Testability ○ ● Because traffic growth should be a blessing, not a problem! Because you need not only tables and foreign keys, but non-SQL stores, search indexes, caches... Productivity ○ Because the other 4 doesn't matter if you need to spend 4 working days to modify a service...
  13. 13. A first choice : Apache Thrift « Software framework for scalable and performant crosslanguage services development » ● ● ● ● ● Interface description language Multi-language code generation Serialisation/deserialisation RPC Framework High performance ○ On transport layer ○ On data manipulation
  14. 14. Thrift, it's cool Thrift boots productivity!
  15. 15. A second choice : MongoDB ● We needed a NO-SQL store for documents ● We wanted to use Cassandra ○ Cassandra & Thrift, a love story ● We wanted a PaaS-ready backend ○ Neither Cloudbees nor Cloudfoundry supported Cassandra ○ But they supported MongoDB ● We loved MongoDB ○ Document DB are great for document oriented applications
  16. 16. ThriftMongoBridge Because we wanted both Thrift and MongoDB
  17. 17. Thrift rocks and works But... How to put Thrift objects into MongoDB ?
  18. 18. First bad idea! Use Spring Data or Morphia Thrift object mapping is bad !
  19. 19. Damned! First bad idea!
  20. 20. Second bad idea ! Use the Mongo Driver JSON parsing Deserialisation will be a bit hard...
  21. 21. Damned ! Second bad idea!
  22. 22. Thrift isn't cool anymore ? We need a new idea !
  23. 23. Thrift, it's cool ! But we had work to do... Write a TBSONProtocol ! The ThriftMongoBridge is born !
  24. 24. What does it do ? POJO generated by the Thrift compiler public class AnotherThrift implements org.apache.thrift.TBase... { ... public String anotherString; public int anotherInteger; Thrift IDL struct AnotherThrift { 1:string anotherString, 2:i32 anotherInteger, } Compile to ... public static final Map<_Fields, org.apache.thrift.meta_data. FieldMetaData> metaDataMap; .... public void read(org.apache.thrift.protocol.TProtocol iprot)... ... public void write(org.apache.thrift.protocol.TProtocol oprot) ......; } TBSonSerializer TBSonProtocol Write To BSON TBSonDeSerializer Read from BSON {anotherString:"value", anotherInteger:69}
  25. 25. Really simple to use Too simple ? AnotherThrift anotherThrift = new AnotherThrift(); anotherThrift.setAnotherString("value"); anotherThrift.setAnotherInteger(69); // serialize into DBObject DBObject dbObject = ThriftMongoHelper.thrift2DBObject(anotherThrift); {anotherString:"value", anotherInteger:69}
  26. 26. How it works write(protocol) Thrift object generated TBSONSerializer TBase getResult sentense directives (begin / end) push Data (read / write) pop Struct context DBObject Context Stack TBSONProtocol DBObject (BSON) MongoDB object
  27. 27. The BreizhCamp demo Because you want to see some code, don't you?
  28. 28. The BreizhCamp demo Given this Thrift model & service struct BreizhCampEvent { 1:i64 date, 2:string startTime, 3:string endTime, 4:com.breizhbeans.backend.thrift.enums.model.EventTypes type, 5:string room, 6:string resume, 7:string details, 8:list<string> speakers 9:list<com.breizhbeans.backend.thrift.enums.model.Track> tracks, } service ProgramService { com.....GetProgramResponse getProgram(1:com.....GetProgramRequest request) throws (1:com....TechnicalException texp, 2:com...FunctionalException fexp) } struct GetProgramRequest { 1:com.breizhbeans.backend.thrift.enums.model.Track track, 2:com.breizhbeans.backend.thrift.enums.model.EventTypes type, 3:string room, 4:list<string> speakers, } struct GetProgramResponse { 1:list<com.breizhbeans.backend.thrift.model.BreizhCampEvent> events, }
  29. 29. The BreizhCamp demo First : Write a test ● One loader by collection ● One goal : Have Human readable test ;-) ● Manage automatically data insertion before test and clean up after test Application code to test @Test public void testGetEvents() throws Exception { List<BreizhCampEvent> actualEvents = programDao.getEvent(); List<BreizhCampEvent> expectedEvents = new ArrayList<BreizhCampEvent>(); expectedEvents.add(getEventPimpMyBackend()); Assert.assertEquals(expectedEvents, actualEvents); } Expected data building (used by the test and the data loader) Standard JUnit assertion
  30. 30. The BreizhCamp demo Write a DAO public List<BreizhCampEvent> getEvent() throws TechnicalException { DBCursor cursor = null; try { List<BreizhCampEvent> events = new ArrayList<BreizhCampEvent>(); DBCollection collection = db.getCollection(collectionName); DBObject query = new BasicDBObject(); // execute the query retrieve DB collection cursor = collection.find(query); while(cursor.hasNext()) { Build and execute the Query events.add( (BreizhCampEvent) ThriftMongoHelper.DBObject2Thrift(cursor.next())); } return events; } catch (Exception e) { throw new H4TTechnicalException("Unexpected error", e, null); Deserialize Thrift objects } finally { cursor.close(); } Error handling } Never forget it ;-)
  31. 31. The BreizhCamp demo Can I update thrift Object by code? @Override public BreizhCampEvent updateRoom(final BreizhCampEvent event) throws TechnicalException { try { BasicDBObject updateQuery = new BasicDBObject(); Mongo Update Query updateQuery.append("$set", new BasicDBObject() .append(getFields(BreizhCampEvent._Fields.ROOM), event.getRoom())); db.getCollection(collectionName) .update(getPrimaryKeyRequest(event), updateQuery) .getLastError() Thrift structure mapping .throwOnError(); return event; } catch (Exception e) { Query execution throw new H4TTechnicalException("Unexpected error", e, ObjectUtils.toString(event)); } } Error handling
  32. 32. The BreizhCamp demo Can I update thrift Object by script? Mongo DB Bson writed : > db.program.find(); { "_id" : "id=10", "date" : NumberLong("-56433978000000"), "startTime" : "14:45", "endTime" : "15:40", "type" : 4, "room" : "Bréhat", "resume" : "Pimp my backend!", "details" : "Le frontend, le frontend...... ...", "speakers" : [ "Sébastien Lambour", "Horacio Gonzalez" ], "tracks" : [ 7 ], "id" : 10, "classname" : "com.breizhbeans.backend.thrift.model. BreizhCampEvent" } Update the field room :> db.program.update(); { "id": 10}, { $set: { "room": "Ouessant" } }
  33. 33. And the perfs? Because speed matters!
  34. 34. Talking about perfs The candidates are ... Spring Data Morphia And... ThriftMongoBridge, of course
  35. 35. Spring Data vs ThriftMongoBridge ● Write and Read 500 objects into MongoDB. ● Each document is composed with a List<> and a Set<> of 500 objects. ● The document have a JSON size of 91613 bytes Who will win ?
  36. 36. Code ThriftMongoBridge First // Get a collection DBCollection collection = db.getCollection("dummyColl"); Write // Serialize the Thrift object DBObject dbObject = ThriftMongoHelper.thrift2DBObject(thriftObject); // Put the document collection.insert(dbObject); Read DBCursor cursorDoc = collection.find(); while (cursorDoc.hasNext()) { DBObject dbObject = cursorDoc.next(); ThriftObject thirftObject = (ThriftObject)ThriftMongoHelper.DBObject2Thrift(dbObject); }
  37. 37. Code Spring Data First // Retrieve the MongoOperations bean MongoOperations mongoOperation = (MongoOperations) ctx.getBean("mongoTemplate"); Write // Serialize the Thrift object mongoOperation.save(springObject, "springObject"); Read List<SpringObject> listUser = mongoOperation.findAll(SpringObject.class, "springObject");
  38. 38. Time to run !
  39. 39. Thrift over MongoDB rocks! We need a logo for our OpenSource library... Available now : 2.0 https://github.com/BreizhBeans/ThriftTools
  40. 40. A storage library a backend is not, young padawan Because we need more things
  41. 41. At the beginning... Pretty layers, neatly isolated, like a Care Bears' birthday cake
  42. 42. Later in the project Like a cake after an earthquake
  43. 43. And in the end... As pretty as a British pudding... and almost as tasty
  44. 44. Be Layer less Now build the Backend ! le b ha c rea un ● Each package have a goal ● NO package assembly rules ● Improve the complexity detection by dependencies analysis Goal : Fight against the "God classes" anti-pattern m ea dr
  45. 45. Thrift services Prevent protocol rupture, have a stable services signature. com.h4t.CheckAccessResponse checkAccess (1:com.h4t.CheckAccessRequest request) throws (1:com.h4t.TechnicalException texp, 2:com.h4t.FunctionalException fexp) stable method signature
  46. 46. Integrate this into the backend Thrift IFace implementation Request / Response decapsulation + log & monitoring Core Business code Thrift object read write ThriftMongoHelper Mongo Opérations Query DBObject DBObject Mongo Driver
  47. 47. Inversion of control Keep simple too ! Prefer static dependencies injection. Time to do the choice or
  48. 48. Inversion of control Why Because ? and ○ binding of the MongoDB service by the cloud ○ no production configuration required ○ only one war
  49. 49. Testability Because without tests you need faith
  50. 50. Test strategy No compromise! The rules are simple... no exception!
  51. 51. TDD is my religion 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Execute test without setup Execute test with your build tool Execute test into your IDE No injection into test framework Reuse only application injection Keep it simple (repeat it 7 times) Have a loose coupling (test the smallest parts) Test must be deterministic and reproducible Do pair review with the tests If the complexity is growing up refactor now ... 11. If a rule goes wrong, rewrite it !
  52. 52. From concept to reality Load thrift object with preset datas into MongoDB Get expected Thift object with preset datas Collection Loader Thrift DataSet ThriftMongoTestTools Get a Thrift object from MongoDB related to a unique key
  53. 53. Tests have goals DAO : Validate queries and set up the test data loaders IFACE : Functional tests SERVICE : Test a business centric service TU : Unit test not related to the business (test first approach)
  54. 54. Tests are Data Driven @Before public void setUp() throws Exception { Inject Data into MongoDB eventLoader.load(getEventPimpMyBackend()); } private BreizhCampEvent getEventPimpMyBackend() throws Exception { String details = "Le frontend, le frontend... ....."; BreizhCampEvent event = eventLoader.getExpected(10, "14/06/2013", "14:45", "15: 40", EventTypes.CONF, "Bréhat", "Pimp my backend!", details); event = eventLoader.addSpeakers(event, "Sébastien Lambour", "Horacio Gonzalez"); event = eventLoader.addTracks(event, Track.CLOUD); return event; } Build test Data with java code, files, BDD or the MongoThrift Test tools (not opensourced yet ;-) )
  55. 55. What's next? Because our work is not completed... yet!
  56. 56. Why integrate our lib ? #Pragmatism ● ● ● ● Productivity oriented Fully testable Container less (no servlet dependencies) Open Source : Apache 2.0 The ThriftMongoBackend Core model will born soon ! With a name: TINAF (ThriftMongoBackend Is Not A Framework)
  57. 57. What's next ? ● monitoring Thrift services in the cloud ● load tests with vmware & 10gen ● contribute TINAF ● Develop a thrift service valve to ensure brandwitdth for critical services
  58. 58. Thank you !

×