Sarine's big data journey by Rostislav Aronov

1. Shaping diamonds with Elastic
Rostislav Aronov, System Architect, Sarine

2. • Rostislav Aronov
• System Architect @ Sarine
• More then 25 years in SW industry
• From coder via DBA, IT, DevOps to architect
• Currently leading key projects at Sarine -
• Cloud connected planning application
• Sarine Profile
• Others
About Me
On Time, On quality, On budget!

3. • Public company – Singapore exchange
• Market share at manufacturing – more than 80%
• Millions of the world’s diamonds are touched by Sarine technology
• CUT grading – ALL major labs – NGTC, GIA, AGS, HRD, IGI…
• Number of employees – 600 World Wide
• Video - https://youtu.be/0PL9BtO-xl4
About Sarine Technologies

4. About SarineTechnologies

5. About SarineTechnologies
Mining & Rough
Trade
Manufacturing
Gemological
Reporting
Polish Trade Retail

8. About SarineTechnologies
Perfect Stone
High Clarity, Colour and Cut
grades Low Clarity Low Colour Low Cut
Sarine ProfileTM

9. About SarineTechnologies
Sarine Planning application

10. • Stand-alone (disconnected) desktop application
• No usage statistics
• No customer feedback
• Support done by screen sharing
• Application/Client Security only
• Applanders - Escaping Appland!
About SarineTechnologies

11. Sarine – Cloud connected application
Work station (client)
Sarine Application
Back End Service for Data
processing
Data Store
SQL-RDS/DynamoDB/S3
Elasticsearch
Logstash
Kibana
Lambda Statistic collector

12. • Application centric logs – requests and results statistic data
• .Net Services activity logs – log4net info and error level logs
• Client sense logs – requests round trip statistics
• Lambda activity logs – info and errors
Sarine – Cloud connected application

13. • CEO – Now this is 21st century!
• PM –We have valuable feedback
• CC – It is easy to help our customers
• DevOps –We know how to monitor and maintain the production system
• Dev –We have cool technology
• And more…
Elastic and Sarine - BusinessValue

14. • Elasticsearch cluster with 3 nodes
• 125 indices (daily, weekly, monthly)
• 268 shards
• 400 millions docs
• 1 terabyte data
• And counting…
Elastic – How we use it?

15. • Shield
• Marvel
• Watcher
• Head
• Kopf
• CB – XDCR
• Cloud – AWS for S3 backups
• Curator
Elastic – How we use it?

16. • DataZone
• POC
• Architecture and Sizing
• First implementation
• Team education
• Elastic subscription
• Fine tuning and best practices recommendation
• Production monitoring, 24/7 support
• Team education
Partners for our Big Data Journey

17. • More Sarine projects on elastic
• Sarine Profile
• Main data store – Couchbase
• Main log/history system
• Main search system
• Main Data visualization system
• Automatic tests
• Results collection
• Data visualization, reports
• Main monitoring tool
• Alerts based on watchers
• Main dashboard
Elastic – Is a Strategic Partner

18. • Use elastic for real-time data collection, analysis and visualization
• Use elastic for super fast search as a part of web and desktop application
• Plan index structure (shards, periodic, size)
• Plan the cluster size, use at least 3 nodes to avoid split brain
• Design index mapping carefully. Use templates, analyzers according to your
needs. Default setting are good enough for general case only!
• Educate you team, it is new and not intuitive for most people technology
• Use partners and consulting at least at the begging
• Re-indexing is expensive. Avoid it.
Elastic – Recommendations

19. Q & A
Sarine - Shaping diamonds with Elastic

20. • Careers at Sarine
• Client Ninja / Developer for PresentationTeam
• Full stack Developers for Cloud Services & PresentationTeams
• Test Engineer to lead QA in Cloud Services team
Sarine - Shaping diamonds with Elastic

21. Thank You!