Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real-world Experiences in Scala

Real-world projects implemented using Scala, Play, Mongo.

  • Login to see the comments

Real-world Experiences in Scala

  1. 1. Real-world Experiences in Scala How I found happiness by Amir Karimi | www.dev-frame.com Aug 2015
  2. 2. Philosophy of Life It's about the journey not the destination If you want to be happy
  3. 3. Geeks Philosophy It's about the tools not the work If you want to be a happy developer
  4. 4. My Philosophy I'm happy with Linux and Scala for now
  5. 5. Bio
  6. 6. The Past ● Fanatic C# Developer for 10 years ● Lived in MS echo chamber ● .NET CF, WinForms, WebForms, MVC, WF, WCF, WPF, Silverlight, LINQ, EF, MS SQL, VS, etc. ● Love Linux: in my dreams I was developing on my own Linux machine!
  7. 7. Converting; Why? ● Tired of MS echo chamber ● Feel lonely ● Love Linux and Open Source ● To run a huge private server with less than 2GB of RAM – Clean installation of Ubuntu server 14.04 takes less than 50MB of RAM – Windows Server 2012 Essentials needs at least 2048MB? ● To observe copyright and licensing (VS, Windows Server/Client, SQL Server, etc.)
  8. 8. My Ideal Programming Language ● Open Source ● Cross platform (Practically) ● Ready to use with mature ecosystem (Practical) ● Strongly-type / Type-safe ● Lambdas / Closures => Functional (I was thinking about LINQ) ● Object Oriented (Interfaces, Generics, ...) ● Modern (Not Verbose: Type inference, Auto Getter/Setter, …)
  9. 9. Available ones ● C# - Cross Platform (Practically) ● Python / Ruby / JS - Strongly-type / Type-safe ● Java - Modern (Not Verbose: Type inference, Auto Getter/Setter, …) and didn't support lambdas at the time ● Go - Ready to use with mature ecosystem | Object Oriented (Interfaces, Generics, …) ● Scala - What?
  10. 10. Scala Features ● Type inference ● Powerful Syntax ● Functional ● DSL ● Immutable Structures ● Case Classes ● Pattern Matching ● Implicits ● Macros ● Powerful Type System and Standard Library ● etc.
  11. 11. Demo Scala Features
  12. 12. Extra Bonus ● Moore's law dying and the need of concurrent programming – Facts ● First 3GHz single core CPU at 2002 ● It's 13 years and there are still just 3GHz CPUs ● But with more cores (your cellphone would have 4) ● Thread blocking is very expensive – Scala and other functional programming languages shine here – Akka is another bonus ● Somebody created a Redis clone on top of Akka (CurioDB) ● Big Data – Spark is on top of Scala and Akka (100x faster than Hadoop)
  13. 13. My New Ecosystem ● Scala – The only available language which meet my needs ● Play Framework – Mature, Type-safe (even templates), High Velocity, Async, Iteratees, Simple to Use ● MongoDB – Open Source, Fast, Distributed, Schema Free, Not good for everything ● ReactiveMongo – Blazingly Fast, Async, Iteratees
  14. 14. Demo Play Framework
  15. 15. Demo MongoDB
  16. 16. Deployment Requirement ● Development Machine – JDK – Activator (Play) – Git (to push the binaries to the server) – Bash ● Production Server – JRE (I don't use this server as build machine) – Nginx (as front-end if you have more than one web apps) – Virtualmin ● DNS ● Main Server ● Webmail – MongoDB – Git (as binary repository for publishing process)
  17. 17. Deployment Process Development Machine Production Server Source Bin Bare Git Repo “/var/webapps/[app] service play-[app] restart
  18. 18. Show Me the Code Deployment
  19. 19. Real-world Projects Tehran International Book Fair ● Website ● Publisher Registration System ● Student Coupon Selling System
  20. 20. Real-world Projects ● Tehran International Book Fair (TIBF) Website – Up to 40,000 unique visitors daily – 2,287 publishers – 361,788 books ● TIBF Publisher Registration System – Registration process of TIBF – About 24,000,000,000 IRR ≈ 700,000$ total transactions in three weeks ● TIBF Student Coupon Selling System – With a quota system based on provinces – More than 500 request/sec – About 87,000,000,000 IRR ≈ 2,600,000$ total transactions in three weeks
  21. 21. Challenges ● TIBF Website – Large number of RPS on book search page ● Users can find their favorite book location ● TIBF Registration System – Complicated process on top of NoSQL database ● TIBF Student Coupon Selling System – Large number of RPS – Large number of financial transactions – Integration with Bank Summary: ● High availability ● Large number of RPS ● Complicated processes on top of NoSQL ● One small server for all of them – 6 cores – 6 GB of RAM – Non-SSD hard disk
  22. 22. Solutions – High Velocity ● Play is a high velocity Framework by default ● Using async/non-blocking data-access library: ReactiveMongo ● Futures and Promises ● Iteratees (for rendering huge data with minimum resource usage: RAM or CPU) ● Everything is perfect; even didn't need caching!
  23. 23. Solutions – NoSQL ● Manage relationships asynchronously – Get fair information → Get registration information → Get basic information → … ● All non-block – Repeat data ● Fairs in which the exhibitor is registered ● Registered exhibitors for each fair – Search ● Generate keywords on CUD, don't use RegEx ● NoSQL is not suitable for a system with complex relationships ● Reports are painful but amazingly fast – Need a job to collect data into another collection – For real-time reports, data should be collected on CUD events
  24. 24. Solutions - Integration ● Scala is on top of JVM with a huge high quality libraries – I needed Rijndael-256 which is not common and is not implemented in Java standard libraries ● Bouncycastle ● Also used – Play's WS as web service client – Scalaxb to generate WSDL classes (they didn't support REST) – Again everything is async
  25. 25. Libraries and Tools - 100% Free● OS – Linux (server and development) ● IDE and Tools – Activator – Scala IDE – RoboMongo – Git ● DB – MongoDB ● Front-end Server – Nginx ● Framework Libraries – Play Framework – ReactiveMongo – Macwire (compile-time DI) – Silhouette (authentication) – Scrimage (image resizing) – Persianutils by Bahman Movaghar – Bouncycastle – Xcala; my own application library – etc.
  26. 26. Well Done! ● Common – Full-stack development – Full async architecture (top to bottom) – Very good RAM usage except for exporting Excel files ● TIBF Website – Less than 100ms responses on peak times ● TIBF Registration System – Most of the publishers were happy – Most of the operators were happy – I was agile to implement real-time requests in minutes or hours ● TIBF Coupon System – Boom! Things didn't go well – Although about 5,000 successful purchases have been done in a couple of hours...
  27. 27. Failure ● After lunching the TIBF coupon website, all websites seemed down ● but the system load was OK! ● Incomplete payments started to show up – Verified payments without having invoices marked as paid – Double payments ● Stress – Loneliness – Failure
  28. 28. Failure I Failed :((((( ● Bad feeling about Scala ● Feels like someone is pouring ice water on my head ● A weird kind of relief ● Everything got quiet
  29. 29. Didn't Give Up ● Took a break, got relaxed and focused ● Checked the bandwidth ● Checked the server disk speed ● Checked the logs – Nginx: Too many open files ● Checked the connections – netstat -na | grep :80 | wc -l ● About 4,000 open connections! ● Google – Default OS limitation for open files is 4,000 – There is a simple solution for “Too many open files” – It was a victory! I could have reach to the OS limits on a small server
  30. 30. Fix ● Add “fs.file-max” to “/etc/sysctl.conf” ● Set soft and hard limits in “/etc/security/limits.conf” for Nginx and the user by which Play apps are running ● Set “worker_rlimit_nofile” in “/usr/local/nginx/conf/nginx.conf” ● The server begin responding but the RAM was still too low – Exporting Excel files for operators eats the RAM – The laptop I used for this presentation has more RAM than the server did (6GB) 6GB 3M$
  31. 31. Final Relief ● They gave me a giant server – 120 Core – 128 GB of RAM ● I used at most lest than 1% of this server resources! ● Although again I increased the default OS limitation on number of open files ● And their network got a serious problem
  32. 32. Servers; Old / New
  33. 33. Bonus Demo Iteratees in Action
  34. 34. Bonus Demo Speed in Action
  35. 35. Thank you Build amazing things with your lovely tools Iran-SUG => Iranian Scala User Group Amir Karimi <a.karimi.k@gmail.com> @4m1rk www.dev-frame.com

×