The document outlines the development of a microservices-based ERP system using Node.js and MongoDB, detailing key topics such as service architecture, performance metrics, and security considerations. It discusses engineering decisions, service auto-discovery, and features of APIs, as well as the importance of properly managing databases and data relationships. The author reflects on lessons learned, achieving a throughput of 330 transactions per second with a focus on scalability and efficient resource management.

1. Building a Micro-services based
ERP system
Using Node.js and MongoDB
Jerry M. Reghunadh
Senior Architect
http://capiot.com

2. { “title” :“Topics” }
• Building our services - Decision process, Modeling, Referential Integrity
• Enabling service auto-discovery
• Validating request, response and data
• Performance and Throughput
• Our learnings – What we did right and what we did not.

3. {“short_intro”: “ERP”}
Core-Masters
Category Brand
Product Deal
Inventory and Warehouse
Warehouses Racks
BinsStock Adjustments
Finance and Commissions
AccountsPayments
ReceiptsCommissions
Third-party integrations
Wallet Recharge
Money Transfer Bill pay
Billing and Order Management
Orders Promotions
User Management
OTPAccess control
Logistics and Returns
RoutingReturns
Servicing

4. APIs are for the greater good!
“APIs are so
overrated”
President Frank Underwood
#FU2016

5. {“apis”:“very chatty!”}
Deal
Brand
Category
Product
1
2
3
4
1
2
3
4

6. { “topic”:“Engineering Decisions”}

8. Low resource footprint

9. Low resource footprint Elasticity
• pm2
• Docker

10. Low resource footprint Elasticity
• pm2
• Docker
Monolingual
.js & json

12. Documents
JSON

13. Documents
JSON
Isn’t my data relational?
Why not SQL?

14. Documents
JSON
Flexible-Schema

15. Documents
JSON
Flexible-Schema Clustering + Sharding

16. { “topic”:“Architecture”}
Cache
Micro-services
Logger
Big data store
Gateway

17. { “topic”:“Architecture”}
Redis
Node.js + Mongo
Filebeat
Elasticsearch
Node.js
Portal (Angular)
API Gateway

18. { “topic”:“APIs”}
Service 1
Service 2
Service N
...
Node.js
Express.js
Swagger.io
Mongoose.js
Passport.js

19. { “api.framework”: “Swagger.io”}
Refer Open API Specification (OAS) https://github.com/OAI/OpenAPI-Specification

20. { “odm”:“Mongoosejs”}
mongoosejs.com
Application layer
• Enforces schema
• Pre- and Post- hooks
• Promises

21. {“library”: “Swagger-mongoose-crud}
https://github.com/capiotsoftware/swagger-mongoose-crud
Create
Update
Delete
List
Get
Bulk Update
(and more)

22. MongoDB features as APIs
Create
Update
Delete
List
Get
GET http://localhost:8080/api/category/v1
PUT http://localhost:8080/api/category/v1/C2
GET http://localhost:8080/api/category/v1/C2
DELETE http://localhost:8080/api/category/v1/C2
POST http://localhost:8080/api/category/v1
db.categories
.find({name:{$regex: /as/}},{id:1, name:1})
.limit(10)
.skip(0)
.sort({name:-1})

23. {“question”: “How to auto-discover services?”}
Gateway
Alice
Bob

24. {“question”: “How to auto-discover services?”}
Gateway
Bob?
Alice
Bob
I want
Alice
Alice?

25. {“solution”: “IP-based config”}
Gateway
Alice
Bob
I want
Alice
Problem: Non-scalable!

26. {“solution”: “Look-up service”}
Gateway
Alice
Bob
I want
Alice
• Dynamic lookup solution
• Lightweight service
• Fast fetch
Cache
https://github.com/capiotsoftware/puttu-redis

27. {“issue”: “Security”}
?
• Censorship
• Privileged access

28. {“issue”: “Security”}
✓
• Censorship
• Privileged access
HTTPs
RBAC:
Role Based
Access
Control

29. {“rbac.define”: “How to set permissions”}
Service
name
Attributes
Roles

30. ERP Revisited
APIs
Adding features
Security
Scaling and Performance
Logging and tracingCache
Micro-services
Logger
Big data store
Gateways

31. {“role”: “MongoDB”}
• Graphs $graphLookup
• Finding parent child relationships
• Finding optimal logistics route from route definitions, limited by hop count

32. {“role”: “MongoDB”}
• Geospatial queries
• Finding nearby hops to create routes.
• Offloaded most operations on the DB.
• Counters using $inc()
• Aggregate
• Graph lookup
• Auto expiring documents.

33. {“topic”:“Throughput and performance”}

34. {“topic”:“Throughput and performance”}
What we achieved: 330 tps
o 4Core 28GB
o 42 services on one machine
…

35. {“topic”:“Throughput and performance”}
What we achieved: 330 tps
o 4Core 28GB
o 42 services on one machine
…
Isn’t that low?
Not really!
Existing system: 150 tps
Projected: 300 tps in 2 years

36. {“topic”:“Throughput and performance”}
What we achieved: 330 tps
o 4Core 28GB
o 42 services on one machine
…
Isn’t that low?
Not really!
Existing system: 150 tps
Projected: 300 tps in 2 years
Consider other matrices
§ Talent acquisition
§ Feature turn around time
§ Agility of platform

37. {“question”: “What if I’m looking for more?”}
Might not be the right tool

38. {“question”: “What if I’m looking for more?”}
Might not be the right tool
Alternatives

39. {“question”: “What if I’m looking for more?”}
Might not be the right tool
Alternatives
Request: GET /
Response: Test response
9k tps
15k tps

40. {“question”: “What if I’m looking for more?”}
Might not be the right tool
Alternatives
Request: GET /
Response: Test response
9k tps
15k tps
What about DB?

41. {“question”: “What if I’m looking for more?”}
Might not be the right tool
Alternatives
Request: GET /
Response: Test response
9k tps
15k tps
What about DB?
+
5000 writes per secondHFT Solution

42. {“reflections”: “What we did right”}
ü Consistent with separation of collections
ü Know your DB
ü Create re-usable libraries/frameworks
ü Handle errors early

43. {“reflections”: “If time travel was possible…”}
o Understand: Write or Read intensive
o HTTP calls are expensive.
o Think of transactional data

44. Thankyou