NoSQL: Delivering scale without complexity

1. NoSQL:
Delivering scale without complexity
Chris Smith
Chief Architect Companies House

2. O!nce Upon
A Time....

3. 1985 - 2008 ICL Mainframe

4. 1997 Companies House First Website

5. Website Mainframe

6. 2004
Web enabled filing of information

7. Data Model
Website required data storage

8. Fetch data
from Mainframe
Store in
website DB
Populate
HTML form
Submit data
back to DB
When complete,
send to Mainframe
User
iterate

9. Directors
Secretaries
Shareholders
Addresses
Shares

10. First steps
Design a data model

11. NORMALISATION

12. Data Modelling
Companies House data is not normalised

13. Observation:
A normalised model
increased software complexity

14. What about performance?

15. Normalised model:
Multiple inserts and joins per entity

16. Core System
HTML form
XML
DB Tables
Director
Person
Address
Address
Iterate
XML
DB Tables
Director
Person
Address
Address
Website

17. Observation:
Companies with large datasets
experienced poor performance

18. Our first Lesson
Understand your data

19. Complexity
Multiple websites and API’s

20. Companies House
3rd Party
Systems
Direct WebFiling WebCheck XML Gateway
Core Registry Additional Resources

21. Where did this lead us?

22. It’s all about data
Information is everything

23. It’s about customers
What do customers need from the data?
How do they use it?

24. Second Lesson
Understand your use cases

25. What did we do?

26. Expose our data
First design a use case API
JSON, REST

27. Data Modelling
Model to satisfy the API

28. Challenge convention
Duplicate data where required

29. Single website uses API
Same use cases across channels

30. Choose a “database”
Relational didn’t work
Need an object store

31. Other criteria
High performance and dynamic
queries on rapidly changing data

32. Why MongoDB?
Met all our needs
Agile, highly scalable and resilient
Objects map neatly to language types

33. Design for performance
Pre-load MongoDB with API data

34. Reduce complexity
Data stored exactly as exposed by API

35. Data mirror
MongoDB mirrors core system data

36. Synchronisation
Oracle triggers drive delta-update process
Core system
Oracle triggers
JSON API
Information change
Delta update
system
MongoDB
Collections
HTTP
Queue

37. Handling deviation
Function to re-sync any data on demand

38. Simplicity
Single website and API

39. Companies House
3rd Party
Systems
Direct WebFiling WebCheck XML Gateway
Core Registry Additional Resources

40. Website
JSON API
3rd Party
Systems
Core Registry Additional Resources

41. Maintainability
Schemas restrict flexibility

42. Customer benefits
Reliable, open access to data suiting need
Open RESTfull JSON API

43. Developer benefits
Reduced complexity and learning curve,
flexibility

44. Service benefits
Highly available and horizontally scalable
Highly maintainable

45. MongoDB infrastructure
Three replica set, SSD and spindle shards

46. System infrastructure
Mesos/Marathon nodes on
virtualised blade infrastructure
NetApp storage

47. Software technologies
MongoDB
ElasticSearch
Mesos
Marathon
AWS
Kibana
Fluentd
Perl
Scala
Java
nodejs
Mojolicious
compass
swagger
statd
git
Ansible
Jenkins
HAProxy
nginx
Linux

48. Questions?