Bavarian government runs a document template application (RTF or ODF with Groovy, Python, Ruby or Tcl as scripting language) serving different government offices. Having complex and hierarchical data structures to organize the templates, MongoDB was selected to replace the Oracle-based persistence layer. This presentation is about the improvements they have achieved with the migration to MongoDB, problems they had to solve underway and unit testing of the persistence layer in order to keep their quality level. Presentation slides by Christian Brensing, Senior Developer at Rechenzentrum Süd, shown at Munich MongoDB User Group Meetup on 18th September 2012

1. MongoDB@Bayern
Migration from RDBMS, Problems, Unit-Testing

2. Christian Brensing
Bayerisches Landesamt für
Statistik und Datenverarbeitung
Rechenzentrum Süd

3. • Template-Processor (~ JSP)
• ODF and RTF
• PDF / PostScript postprocessing
• Groovy, Python, Ruby, Tcl
• In Production using RDBMS
since 2008
• Central HR system (SAP) uses
BayText to create output

4. Client for developing the templates (BayText-IDE)

5. ODF-Template using Ruby

6. Generated document

7. 2 Million
Generated documents per year

8. Tiny dataset
50,000 records and 10 GB BLOB

9. a small system after all

10. So why migrate?

11. Because we can

12. developers.pop()
Smaller teams

13. Easy to learn
Read one book and you're done

14. Simple mapping

15. permissions templates documents
codes groups document_usages
folder_configurations requests folders
nodes structures template_tags
dynafields components roles_roles
roles bundles scripts

16. nodes
roles
properties
GridFS

17. {!
"_id" : ObjectId(),!
"_type" : "Document",!
"updated_at" : ISODate("2007-12-06T10:04:00.543Z"),!
"updated_by" : "maggie",!
"version" : 1,!
"path" : "foo.bar.Document",!
"description" : "",!
"language" : "ruby",!
"format" : "odf",!
"autoload_uplevel" : -1,!
"requests" : [!
{!
"name" : "Test",!
"description" : "foo", !
"xml" : zlib("<?xml ...?>")!
}, !
... !
],!
"structure" : [!
{ !
"alias" : "some alias",!
"component_path" : "foo.bar.Document$Document",!
"parent", -1 !
},!
... !
], !
} !
"tags" : ["foo", "bar", "baz"]!
}

18. public class RoleReadConverter implements Converter<DBObject, Role> {!
@Override!
public Role convert(DBObject source) {!
Role role = new Role();!
role.setName((String) source.get("name"));!
...!
return role;!
}!
}!
!
!
public class RoleWriteConverter implements Converter<Role, DBObject> {!
@Override!
public DBObject convert(Role source) {!
return new BasicDBObject()!
.append("name", role.getName())!
.append(...);!
}!
}!
!
!
// Convert a DBObject to an Entity!
conversionService.convert(collection.findOne(...));
Easy mapping using Spring-ConversionService

19. Performance
up to 10x

20. ORM no more
The MongoDB driver is all you need

21. Obstacles?

22. Mentality

23. Referential Integrity
Multi-Document-Update

24. {!
"_id" : ObjectId(),!
"_type" : "Folder", !
"updated_at" : ISODate("2007-12-06T10:04:00.543Z"),!
"updated_by" : "maggie",!
"version" : 1,!
"path" : "foo.bar.Folder",!
"description" : "bla",!
"bundles" : [!
... !
]!
"document_usages" : [!
{!
"name" : "foo",!
"description" : "bla", !
"exec_order" : 5, !
"print_copies" : 2,!
"type": "FAIR_COPY",!
"document_path" : "a.b.Document",!
"bundle" : "bar"!
},!
...!
}
Linking documents via a path instead of ObjectId

25. But paths are mutable!
All references must be updated

26. Approach
• Pseudo-Transaction with an update locking
mechanism in a custom oplog collection
• Periodical repair jobs finishing cancelled
operations
• Extended interpretation of eventual consistency

27. Unit-Tests
Continuous Integration

28. F.I.R.S.T.

29. Isolated
Tests are using a unique DB per JVM

30. @Configuration!
@Profile("test")!
public class TestSpringConfiguration extends SpringConfiguration {!
@Override!
public void init() {!
setTestProperties();!
!
super.init();!
!
// Test-DB löschen!
db().dropDatabase();!
}!
!
private void setTestProperties() {!
String dbname = env.getProperty("db.name", "baytext_test_" + getSimpleUsername());!
String host = env.getProperty("db.host", "mongodb-dev.db.rz-sued.bybn.de");!
String port = env.getProperty("db.port", "27016");!
!
cmProperties.setProperty("servers", String.format("%s:%s", host, port));!
cmProperties.setProperty("name", dbname);!
}!
!
// OS-Username without prefix (e.g. maggie instead of lfstad-maggie)!
private static String getSimpleUsername() {!
String username = SystemUtils.USER_NAME;!
int indexOfDash = username.indexOf('-');!
return indexOfDash != -1 ? username.substring(indexOfDash + 1) : username;!
}!
}
Test-ApplicationContext

31. Repeatable
Collections are purged before each test method

32. public class MongoTestExcecutionListener extends AbstractTestExecutionListener {!
@Override!
public void beforeTestMethod(TestContext testContext) throws Exception {!
purgeCollections();!
}!
!
private void purgeCollections() {!
DB db = MongoDBHolder.getDB();!
for (String collectionName : db.getCollectionNames()) {!
if (collectionName.startsWith("fs.") || collectionName.startsWith("system.")) {!
continue;!
}!
DBCollection collection = db.getCollection(collectionName);!
if (!collection.isCapped()) {!
collection.remove(new BasicDBObject());!
}!
}!
}!
}
Spring-TestExecutionListener

33. @RunWith(SpringJUnit4ClassRunner.class)!
@ActiveProfiles({"test"})!
@ContextConfiguration(classes = TestSpringConfiguration.class)!
@TestExecutionListeners({!
DependencyInjectionTestExecutionListener.class,!
DirtiesContextTestExecutionListener.class,!
MongoTestExcecutionListener.class!
})!
public abstract class MongoTestSupport {!
}!
!
!
!
public class DocumentRepositoryTest extends MongoTestSupport {!
}!
Test base class

34. 95% Coverage

35. Summary
• Developing the data layer is fun again
• Reduced complexity
• Flat learning curve compared to SQL/ORM
• use_mongo() unless transactions_required

36. Thank you!