Talk about the joureny of small tech team re-designing SmartUp.io from scratch, and the technical paths from MVP to Production. High level overview of architecture and tech stack decisions, best-practices and culture.

2. From prototype
to production
The journey of re-designing SmartUp.io

3. ● Extrovert geek and tech lover
● Joined SmartUp.io @ Q2 2016
● Adores Linux, OSS, is a general minimalist
● Mining the bits in the startup & product
mines, but been through the valleys of
outsourcing & consultancy scene as well
About Me - whoami
Mate Lang
CTO @ SmartUp.io

4. About SmartUp.io - Who we are
● Startup company with a mobile-first,
gamified, social micro-learning SaaS
○ Initial goal is to help entrepreneurs launch and
take their startup companies to success
without bothering VCs with the exact same
questions
○ Found out, we have a much wider use case:
reach, engage, train & inspire communities to
facilitate learning
● Got a lot of attention from clients and
investors

5. SmartUp.io - Motivation
● Deloitte predicts huge
disruption opportunity
in corporate learning
sector

6. SmartUp.io - Motivation
● Deloitte predicts huge
disruption opportunity
in corporate learning
sector
Company’s average
net promoter score
for their LMS?

7. SmartUp.io - Motivation
● Deloitte predicts huge
disruption opportunity
in corporate learning
sector
Company’s average
net promoter score
for their LMS?
-8

8. What is this talk about
● How shoud I (the geek) attack the problem domain to be successful?
● How did we decompose and redesign a “complex” (a.k.a messy) monolith into
maintainable microservices
● How did we cope NFRs like security and performance

9. Fact #1 - In software the only constant thing is change

10. Fact #1 - In software the only constant thing is change
Fact #2 - Geek is the new sexy ;)

11. Market driven evolution
● Initial product is simple media consumption
○ Users learn by consuming content and competing with each other
○ SmartUp content team writes content on protected administration webapp
○ Users are on the same platform, without isolation
● ACME Inc comes along with proposal to use the platform internally for learning
○ Users are isolated for ACME Inc and FooBar Inc (multi-tenancy)
○ Company needs to be able to write their own content on their “slice” of the platform
● Introduce Communities = isolated instance of our platform suitable to serve the
needs for business consumers

12. Version 1 (Wondeer)

13. Market driven evolution
What we had What we wanted to sell

14. But most importantly...

15. Market driven evolution
This is what we had
under the hood

16. Plan is to rewrite from scratch
Team’s estimation of backlog = ~ 7 months

17. Plan is to rewrite from scratch
Actual delivery time = 7 months 10 months

18. Version 2 (Brontocorn)

19. Cool :)
Let’s rewrite...

20. Where could it go
wrong?

21. Where could it go
wrong?
Right from architecting

23. Understanding your domain
● Clients use a “smart hack” on V1 to obtain a predefined order of their published
learning material
● By default the platform facilitates “feed-like” behaviour
○ no explicit ordering
○ potentially infinite list of items
● The right question to ask as an engineer
Why do our client would ever
want that?

24. Understanding your domain
● Clients want a course like structure
● That is substantially different than our individual publishing model
○ The content is designed from the beginning in an ordered fashion
○ The completion should be in an ordered fashion
○ Analytics should be able to correlate between completion records
○ It deserves it’s own management

25. Understanding your domain
● Clients want a course like structure
● That is substantially different than our individual publishing model
○ The content is designed from the beginning in an ordered fashion
○ The completion should be in an ordered fashion
○ Analytics should be able to correlate between completion records
○ It deserves it’s own management
VS

26. Let’s get geeky

27. Geek team shop list for the re-write
● Scalable and maintainable codebase on all platforms
● Needs to be microservice architecture, because everything is easier with them
(this is a fat lie)
● Automate everything that is possible to automate
● Detailed and helpful documentation
● Needs to ship in a continuous fashion with Docker because Docker is cool (it
actually is)
● Needs to have Infrastructure-as-Code
● Needs to use managed solutions
● Web client and mobile apps useable by your grandma

28. Tech stack

29. Microservices - the why
Google Trends - search for term “microservice”

30. Microservices - the why
● Haters gonna hate, but there is undeniable interest & adoption in
engineering-led companies
● Microservices take common clean code (SOLID) concerns to system level
○ SRP - a service should be concerned about a single coherent domain
○ OCP - extending behavior done through encapsulating with higher level services.
Change in remote context should not produce change in a given service.
○ ISP - introducing edge services - specialized backends for clients
● The above seems like common sense, but engineers do fail in designing such
systems

31. Microservices - the reason we fail
“Often stepping back
you see more, don’t you?”
David Hockney
painter, draughtsman, printmaker, stage designer and photographer

32. Microservices - my fast service design test
● If you want to know whether you have (not) designed it correctly (false
positives may appear) fill in the following test
Test for your service design
1. Imagine you have to open source your service. Are you able to do so without
doing code changes, but staying useful to an engineer outside your business
domain? …………………………………
------------------------------ END OF TEST ------------------------------

33. Let’s see a concrete example

34. Meet the
“Leaderboard Service*”
*(the tech scene ran out of deity names to name services after,
so no Zeus or Hydra for you)

35. The leaderboard - proposition
● Service responsible to manage leaderboards
● A leaderboard is an ordered list of players associated with a score
● It needs to allow the near real-time update of such boards, based on individual
score change events
● Allows the player to check the “transaction history” not just the aggregated
state
● It allows the fast retrieval of a certain segment of the board
○ Top X
○ Around X ( X-10, X, X+10)

36. Something
like this...

37. But a bit more modern...

38. Like this

39. The leaderboard - collecting points
Card
Points
Card

40. The leaderboard - under the hood
public interface LeaderboardService {
LeaderboardCreationResponseDto createLeaderboard(LeaderboardCreationRequestDto requestDto);
LeaderboardDto retrieveTopNLeaderboard(String leaderboardId, long size);
LeaderboardDto retrieveAroundNLeaderboard(String leaderboardId, String playerId, long size);
PlayerDto retrievePlayer(String leaderboardId, String playerId);
void updatePlayerScore(String leaderboardId, String playerId, long score);
}
● Note that this interface is agnostic of all SmartUp related logic
● Could be reused in any situation where you want to represent entities in a
sorted order by score
● No matter what the player entity is, it’s created lazily when you first upgrade it’s
score
● Currently used to incentivize consumption, but can be applied to groups of
players, content creators, etc.

41. The leaderboard - under the hood

42. ● Due to event sourcing we can always reconstruct state in case of failure
● Redis is in-memory. HA or not, should it ever go down, an update of state would
effectively restore our read-optimized model
● Our number of Score Processors scale with the data volume
● Concurrent state update correctness guaranteed by DynamoDB Stream shards
● In case of processor failure, upon service restoration the unprocessed events
would get picked up, all in a couple minutes (up to a couple weeks of staying
behind)
The leaderboard - the gain

43. The leaderboard

44. The leaderboard
QC: PASS

45. Another example...

46. Meet the Content Service

47. The Content Service - The proposition
A service that handles the creation and modification of versioned learning material.
Also enables versions to be instantiated and completed by consumers.
Encapsulates both structural and behavioural functionalities, like:
- Structural
- Question text
- Limited number of answers
- Solution explanation
- Behavioural
- Single-choice
- Multiple-choice

48. The Content Service - Under the hood

49. The Content Service - the gain
● Due to the Context being an abstract entity we can support lots of use cases for
consumption rules & resilient to change
○ Sharing of consumption record
○ You can re-do a content in certain circumstances (e.g. exam mode has separate context)
● Feedback from our Head Of Content after going live
“Fast, smooth and easy. And really fast. And damn, this thing's fast...”
● Design enables easy clean-up should we ever do so.
For now we store every change a content creator made.
“Because I can” - Dr. Bob Kelso, Scrubs

50. Tuning your
microservices, cause

51. Tuning your
microservices, cause
with high throughput
comes high latency

52. Tuning your microservices
● Simple Operation: Check users
credentials and request JWT Token
● Initial results: not too bad
● P95 responds in 701 ms

53. Tuning your microservices
● Scale it up
○ 1 OAuth Service
○ 2 User Services
● P95 in ~35k ms
● Almost all requests respond after 1
second
● 40% requests FAILED

54. THAT DOES NOT MAKE SENSE!

55. Tuning your microservices
● Found out there is no connection HTTP pooling -> TCP handshake penalty
● Update Spring Cloud to Edgware
● Set correct timeouts for Ribbon and Hystrix
● Reduce (yes, reduce) Tomcat resources
○ Max-Threads
■ The maximum number of request processing threads to be created
○ Max-Connections
■ The maximum number of connections that the server will accept and
process at any given time
○ Accept-Count
■ The maximum queue length for incoming connection requests when all
possible request processing threads are in use

56. Tuning your microservices
● 0 Failed Requests
● Mean Req/s: ~50
● P95 latency: 148 ms

57. Tuning your microservices

58. Sometimes

59. If microservices have
not solved your
engineering problems...

60. Try hiring a
“DevOps Engineer”

63. As per Wikipedia
DevOps =
a software engineering practice that aims at
unifying software development (Dev) and
software operation (Ops).

64. Automate
EVERYTHING!

65. Bronicorn Release
● Went live October 3, 0600 RO time
● Development environment was used for previous 10 months
● No other environment due to cost reasons
● On the day of release we created
○ Staging (Acceptance Testing) environment
○ Production

66. Bronicorn Release
● Went live October 3, 0600 RO time
● Development environment was used for previous 10 months
● No other environment due to cost reasons
● On the day of release we created
○ Staging (Acceptance Testing) environment
○ Production
45 minutes difference between
deployments (5 minutes active)

67. How is that possible?

68. How is that possible?
Infrastructure-as-Code

69. Infrastructure as Code
Definition
Infrastructure as code (IaC) is the process of managing and provisioning computer data
centers through machine-readable definition files, rather than physical hardware
configuration or interactive configuration tools. (as per Wikipedia)
● Just a Bunch of shell scripts
● Modern provisioning tools like: Chef, Puppet, Ansible
● Cloud-ready IoC management: Cloudformation, HashiCorp Terraform

70. Meet Terraform
● DSL based using HCL (HashiCorp Configuration Language)
● Module oriented
● Manages dependencies between resources (e.g. DNS depends on IP)
● Nice interpolation syntax
● Natively manages multiple environments through configuration (e.g. instance
types differ from env to env)
● Workflow = Plan > Review > Apply
● Configurable state backends
● From V0.10 supports pluggable “providers” (e.g. AWS, GCP)

71. Meet Terraform
Src: Terraform Homepage

72. Our way of Terraforming
● 4 AWS VPCs
○ Services VPC (For maintenance, and unified connection to other VPCs)
○ SmartUp VPCs (e.g. Dev, Stg, Prod)
● Each service owns its own module along with dependencies
Eg: Leaderboard:
○ Redis
○ Queues
○ DynamoDB Tables & Streams
○ etc
● Peering module to connect Services <-> SmartUp
● Using encrypted S3 for safe state storage

73. Managing service configuration
● Services pick up their configuration exclusively in runtime
● No mvn package -Pdev|stg|prod
● Build one artifact (docker) and use it everywhere
● Consul as service discovery and configuration storage
● Terraform injects properties into Consul upon execution
● No configuration done in YML
● Using Spring Cloud Config to pick these values up from Consul upon startup
and checking periodically for changes

74. Managing service configuration

75. Managing service configuration

76. Each team is responsible for
their delivery process

77. Each team is responsible for
their delivery process
From design to production

78. Let’s put it to production
● Loads of deploys in a geek’s life, better make it simple
● A good pipeline will
○ Provide fast feedback before PR integration (build, test & check infra dependencies)
○ Deploy ASAP changes to dev (fail-fast)
○ Streamline production releases so they prevent human error
○ Clear separation between steps
● Preferably define the whole pipeline using code
● Decided to use CircleCI - YAML based Workflows

79. Kudos to the team

80. Thank you for your
kind attention!
Mate Lang
CTO @ smartup.io
mate@smartup.io
twitter: @langmate
medium: @matelang