While there are resources on why other companies have made the transition to SOA, and end-game service boundaries may be clear, there are few resources on how to make progress towards SOA without sinking an entire team or stopping concurrent product development. Faced with a growing codebase, slower tests and performance issues, we've extracted services from our large Rails app the way we do everything else: iteratively. I’ll walk through what we did right, and what we almost did wrong. Presented at RailsConf 2014.

1. Iterative Service
Oriented
Architecture
Eric Saxby
@sax @ecdysone @sax

2. From thisFrom this

3. To thisTo this

4. Why should you listen to me?
Proprietary and
Confidential

5. Why should you listen to me?
Proprietary and
Confidential
■ I have many leather bound books

6. Why should you listen to me?
Proprietary and
Confidential
■ I have many leather bound books
■ My apartment smells of rich mahogany

7. Why should you listen to me?
Proprietary and
Confidential
■ Application developer
Many various technologies. Rails for ~6 years."
■ Recent focus has been operational"
■ Such acronyms, so buzzword
TDD, Agile, DevOps, SOA, Cloud, IaaS
■ Broken a surprising amount of things in a short time"
■ Work at Wanelo

8. Wanelo? Whaaa?

9. Wanelo? Whaaa?
■ Social network for shopping
If you buy things on the Internet, you are our demographic

10. Wanelo? Whaaa?
■ Social network for shopping
If you buy things on the Internet, you are our demographic
■ Many millions of users

11. Wanelo? Whaaa?
■ Social network for shopping
If you buy things on the Internet, you are our demographic
■ Many millions of users
■ Databases with billions of records

13. More importantly

14. More importantly
■ Iterated from one large Rails codebase to services

15. More importantly
■ Iterated from one large Rails codebase to services
■ Open source as much as we can

16. More importantly
■ Iterated from one large Rails codebase to services
■ Open source as much as we can
■ Scaled Ruby where people say Ruby can’t be scaled

17. More importantly
■ Iterated from one large Rails codebase to services
■ Open source as much as we can
■ Scaled Ruby where people say Ruby can’t be scaled
■ Very small team

18. But are you really here for success?
“Good judgment comes from experience. Experience
comes from bad judgment.”
— Proverb"
— Theo Schlossnagle"
http://www.infoq.com/presentations/Scalable-Internet-Architectures

19. “The Anti-Pattern Goldmine”

20. ■ That I may or may not have worked for"
■ One giant, tangled Rails 2 codebase
our vendored code had vendored code!
■ 30 engineers racing to commit code as fast as possible"
■ Success == large project launched on specific date
A completely hypothetical company

21. Releases a mess
■ Code branches for months at a time"
■ Monthly releases turn to weekly releases"
■ Change list distributed and explained to executives"
■ Multi-hour deploys"
■ Annual code freeze means 3 month code bonanza

22. Over time, things get worse…

23. Over time, things get worse…
■ Rough estimates == Firm deadlines

24. Over time, things get worse…
■ Rough estimates == Firm deadlines
■ Shortcuts only way to meet deadlines

25. Over time, things get worse…
■ Rough estimates == Firm deadlines
■ Shortcuts only way to meet deadlines
■ As soon as projects finished, teams split to meet new
deadlines

26. Over time, things get worse…
■ Rough estimates == Firm deadlines
■ Shortcuts only way to meet deadlines
■ As soon as projects finished, teams split to meet new
deadlines
■ Estimates based on worst-case timelines…
never worst-case enough

27. Technical DebtTechnical Debt

28. Technical DebtTechnical Debt

29. What did I think I learned?
■ When programming is not fun, THERE IS A PROBLEM"
■ SOA IS SOLUTION TO ALL OUR PROBLEMS"
■ DEVOPS IS THE ANSWER

30. Now for a role change
■ I move into the operations team
renamed to “not the operations team”
BECAUSE DEVOPS"
■ Working on systems automation"
■ Engineering: SOA is the only way forward"
■ Product: SOA is that thing getting in the way of cranking out
features

31. If at first you can’t succeed

32. If at first you can’t succeed
■ Take a firm stance

33. If at first you can’t succeed
■ Take a firm stance
■ Take some deep breaths

34. If at first you can’t succeed
■ Take a firm stance
■ Take some deep breaths
■ Cry like a baby

37. New features would be services, or not built
■ New login feature as separate application"
■ Data service to manage
Enterprise software home-grown Rails app

38. “This will be hard and we’ll screw up,
but we have no other option”

40. Okay, so what is SOA again?
One site composed of multiple applications
Internal External
UIAPI only
HTTP
Synchronous
TCP
Asynchronous

41. And why would I want this?
■ Scale organization"
■ Each team manages their own code"
■ Each team deploys their own code"
■ Outsource some code without giving keys to the kingdom

42. And why would I want this?
■ Scale codebase"
■ Performance tightly coupled to workload"
■ Hide complexity behind clean interfaces"
■ Use tools best suited to task at hand

43. And why would I want this?
■ Scale tests"
■ Small codebase == small test suite"
■ but… costly learning curve to test service boundaries

44. At some point, the cost of not doing
services outweighs the cost of doing
services

45. Ok… back to our protagonists
■ Data service 9 months "
■ Complex state transitions between applications"
■ No major data problems"
■ Dependent applications ambiguously successful

46. EngineeringEngineering
We didn’t break anything!We didn’t break anything!

47. ProductProduct
9 months??????9 months??????

48. Ok, ok, ok, but what about?…
■ 3 months
2-4 engineers + the “DevOps” team"
■ Staging servers available in 2-3 weeks
2 months later… no one had deployed to them!
■ Released with feature flag
Extra few weeks to break it in production

49. But then…
■ Worked exactly as intended"
■ Very resilient to failures!
■ User metrics generally successful

50. EngineeringEngineering
Success!!!1!!Success!!!1!!

51. ProductProduct
3 months??????????3 months??????????

52. How many engineers does it take to
Service Oriented Architecture?

53. What is success?
■ Engineering: Success"
■ Product: Failure!
■ Who wins?

54. Nobody

55. But let’s go deeper…
■ Why did we need SOA?"
■ Engineering did not trust product"
■ Engineering did not trust that we could fix
the shortcuts required to do our job

56. And even deeper…
■ Product did not trust Engineering to deliver
on promises"
■ Product accountable for features, not for
quality

57. Trust

58. So what did we learn?

59. So what did we learn?
Fuck it, SOA FROM THE BEGINNING next time

60. So what did we learn?
Fuck it, SOA FROM THE BEGINNING next time
X

61. Agile is more than just sprinting
■ Iterate, using data to guide next iteration"
■ Deploy soon, to start collecting data"
■ Red — Green — Refactor"
■ SOA will not solve organizational problems

62. SOA does not “fix” your
code quality

63. SOA is a tool to help with
our technological pain points

64. Iteration is how we create
and deploy changes

65. Iteration
■ Small changes deployed quickly"
■ Feature flags"
■ Each step prioritized and completed when
necessary"
■ But when SOA?

66. Performance

67. Code complexity will be less
than not doing SOA

68. New code completely unrelated
to old code

69. With the caveat…
■ May go into old codebase short-term"
■ Deploy today is better than deploy next
week"
■ Requires TRUST that this will be fixed later

71. So… Performance

72. ■ Site getting slower"
■ DBs becoming I/O bound on disk"
■ User growth/activity is crazy, and increasing

73. ■ One table completely outstripping others"
■ Reads from table killing caches for others"
■ We’re in the cloud, so max resource limits

74. ■ Site will stop working, soon"
■ Don’t want to stop feature development"
■ 10 engineers

75. Step 1: Isolate the data

76. Remove ActiveRecord relations
class Product < ActiveRecord::Base
has_many :saves
end

77. Remove ActiveRecord relations
class Product < ActiveRecord::Base
def saves
Save.where(product_id: self.id)
end
end

78. Pretend it’s in a different database
class Save < ActiveRecord::Base
establish_connection "saves_#{Rails.env}"
end
production:
database: production_db
host: 10.0.0.100
saves_production:
database: production_db
host: 10.0.0.100

79. ■ Each change deployable"
■ Tests + staging servers will show where
things break"
■ Watch out for DB connection count!

80. Switch to using different database
■ Create read-only replica"
■ Site maintenance mode"
■ Push new database.yml pointing to replica"
■ Promote replica to be master"
■ Restart unicorns"
■ Disable maintenance

81. ■ Performance may now be fine"
■ Next step can come later, whenever
necessary or convenient

82. Step 2: Isolate the interface

83. ■ How many ways do you access table?"
■ What is long term growth?"
■ What are long term goals?

84. How would you shard if you could shard?
■ Wanelo == Saves"
■ Saves are viewed by"
■ Product"
■ User

85. Refactor to create an API
class Product < ActiveRecord::Base
def saves
Save.where(product_id: self.id)
end
end

86. Refactor to create an API
class Product < ActiveRecord::Base
def saves
Save.by_product(self.id)
end
end

87. Refactor to reduce the API
■ Remove redundancy"
■ Add tests"
■ You will break things very soon!

88. Step 3: Extract DB adapter

89. Move finders into a module or base class
class Save
include SavesClient
end

90. module SavesClient
def self.included(other)
other.send(:attr_accessor, :id, :product_id)
other.extend ClientClassMethods
end
!
module ClientClassMethods
def by_product(*args)
adapter.new(self).by_product(*args)
end
!
def adapter
@adapter ||= SavesClient::DbAdapter
end
end
end

91. ■ Finders call an adapter"
■ Adapter wraps database calls

92. Why an adapter?
■ Adapter is your feature flag"
■ Deployable as in-place refactor"
■ Can be replaced later with different adapters

93. module SavesClient
class DbAdapter
def self.close_connections
# Check in the database connection,
# since we're shutting down this thread
SavesService::Save.clear_active_connections!
end
!
def by_product(*args)
relation :by_product, *args
end
!
def relation(method, *args)
SavesClient::AdapterRelation.new(self,
SavesService::Save.send(method, *args))
end
end
end

94. ■ Because THREADS"
■ State changes should always return a new instance"
■ #all, #first, #etc are called on Relation instance"
■ Relation will delegate to adapter
def relation(method, *args)
SavesClient::AdapterRelation.new(self,
SavesService::Save.send(method, *args))
end
Adapter methods return a Relation

95. ■ ActiveRecord model, which moves into the client"
■ Insert your favorite DB adapter here"
■ Important part is that this is hidden from the
including class
def relation(method, *args)
SavesClient::AdapterRelation.new(self,
SavesService::Save.send(method, *args))
end
Scope of query is called on…

96. Save.by_product
DBAdapter
AdapterRelation.new

97. relation.all
adapter.all
ActiveRecord::Save Postgres
Instantiate client instance(s)

98. Critical to deploy at this step
■ 100% guaranteed to make mistakes"
■ 100% guaranteed to have missed something"
■ Cost of fixing this is low at this step

99. Step 4: Launch service app

100. Insert your framework of choice
■ HTTP? JSON? MessagePack?"
■ What is easiest to develop?"
■ Transport mechanism maps to an adapter in
the client"
■ Can be replaced later

101. brief interlude

102. Q: Why should you have deployed
by now?

103. A: Because the client is much
more complex than the server

104. So………..

105. Your bugs are in the client

106. Server code will be dictated by the client
■ By service launch time, whiteboard diagram
will have changed"
■ Understanding will have changed"
■ Don’t write your server before you understand
the needs of the client

107. So, back to the server
■ We used Sinatra + Oj"
■ Sinatra is awesome!

108. Step 5: Use the service

109. Adapter is your feature flag
switch between DB and service"
=="
switch between DB and HTTP adapters

110. Save.by_product
HTTPAdapter
AdapterRelation.new

111. relation.all
adapter.all
HTTPClient JSON to Hash
Instantiate client instance(s)

112. Retrospective

113. ■ Isolate data"
■ Isolate interface"
■ Extract DB adapter into gem"
■ Launch service"
■ Switch to service adapter

114. Tests?

115. Integration vs Unit
■ Spin up server, a la Sunspot?"
■ “Fake” adapter"
■ In-memory store (Array)"
■ Need unit + integration"
■ Can delete redundant tests later

116. Development
■ Need to actually run service"
■ Failures are in interactions"
■ Foreman + Subcontractor"
■ Each app needs isolated Bundler env"
■ Don’t mix dependencies

117. But what about a new app?

118. How can we iterate on something
that doesn’t exist?

119. ■ Deploy as early as possible"
■ Chef/Puppet/Salt/Capistrano/CFEngine"
■ Focus on being able to change quickly"
■ Isolate and understand differences

120. Feature flags are more than on/off
■ Off"
■ Asynchronously write"
■ On

121. Integrate deep before wide
■ Code path from beginning to end"
■ Edge cases/Errors will bubble up"
■ Avoid complexity or cleverness in design"
■ It will come on its own

122. Takeaways
■ Barriers to SOA are barriers to all development"
■ Figure out how to iterate"
■ Hexagonal architecture can bubble up from
organization’s needs"
■ Priorities/needs change. Facilitate this.

123. Thanks!
sax ecdysone sax
especially to everyone involved in this work, of which I was one
small part