Running stateful services is difficult, upgrading under those services is even more difficult. The presentation explores some approaches to move forward while hosting stateful services

1. Upgrading under the
weight of all that
state
Quinton Anderson

2. Context

3. Canonical
Model

4. Source
Source
Source
Source
Raw
Data
Business
Data
Access
Layer
Access
Layer
Access
Layer
Access
Layer

5. Load
Balancer
//TODO
Function
Cntrl-V
Scaling

7. Downstream
systems
• Specialised
management
systems
• Reporting Systems
• Product
management
Channel &
product
systems
Master Data
Management
Hadoop
• Leverage all data & reduce
integration costs
• Comprehensive dataset –
internal & external, realtime &
batch, structured & unstructured
• Advanced analytics / machine
learning
Group Data
Warehouse
• Understand our business
• Accurate, conformed, and
reconciled data
• Access layer to support BI &
reporting
BI/Reporting
• User facing tools
• Regulatory reporting
• Dishoarding
• Self service BI for the
masses
Customer record &
insights
All data
Price,
conversation,
credit dec.
etc.
Financial Data
Subset of
data
User
access
Information for
people
Core Financial
Systems and
functions
• P&L
• Recon
• General Ledger
• Etc…
Closed loop,
automated ‘decisions’
Decisioning
• Personalise/optimise decisions,
maximise customer value
• E.g. price, credit decision, next
conversation, experience
Core information repositories
Analytics applications
Other systems

8. Channels
Hadoop
Rules
Serving and decisioning
Analytic
Records
Systems Of Record
Core
Banking
Payments
Event Processor
Raw Data
Derived Data
Feature Store
Event Store
Scoring
Machine
Learning
www
Event Streams
Customer
Information
data loaded
Data
analysed &
processed
Insights &
events
captured
Integration API/Service Discovery

9. > 4000 Daily Batch Jobs

10. > 6 PB of State and growing

11. Hbase,
Cassandra,
HDFS,
Influx,
Elastic Search,
Kafka,
Etcd,
Zookeeper
OpenStack Swift

12. Oracle,
MySQL,
Postgres

13. Hundreds of services

14. MR1,
MR2,
Spark,
Akka

15. Dev,
Test,
Staging,
Prod 1,
Prod 2,
Etc…

16. == Complexity

17. Imperative:

21. Culture

24. Architecture

25. Immutable

26. Someone else’s computer

27. State Locality

28. Workload non-locality

29. Flexible over optimal

30. Practically, it is a closed system

31. State management is my problem

32. All abstractions are leaky

33. Repo(s) CI/CD Apps
Docker Calico
Mesos Yarn
Spark, MR, Impala, etc
Marathon,
Chronos, Cassandra, etc
CI/CD
CI/CD
Repo(s)
Repo(s)
Open
Stack
Nova
Nova/Ironic
OS
KVM
OS
Firmware + Hardware + Tags

34. Strategies

35. Outsource the problem, and tool away the
resulting issues

36. Delete it, tool away the resulting issues

37. Be stateless, tool away the resulting issues

38. Implement some patterns, incrementally
optimise. Tool away the resulting issues

39. Excess Capacity

40. Patterns

41. Consumer
Router
DB
Old Old
Web
App
DB
Web
App

42. Consumer
Router
DB
Old Old
Web
App
DB
Web
App

43. L4
HAProxy
Old Old Old Old

44. L4
HAProxy
Old Old Old Old New

45. L4
HAProxy
Old Old Old Old New

46. L4
HAProxy
Old Old Old Old New

47. L4
HAProxy
Old Old Old New New

48. L4
HAProxy
Old Old New New New

49. L4
HAProxy
Old New New New New

50. L4
HAProxy
New New New New New

51. == Incrementally accept risk

52. In place upgrade

53. Stateful

55. CAP, PACELC

56. Data models

57. Atomicity

58. Access patterns

59. Implementation approaches = ??

60. Upgrade Duration
O(N)

61. for node in nodes:
if info[node]['instance']:
if Status(node).run().wait() == AVAILBLE_FOR_MAINTENANCE:
MaintenanceMode(node).run().wait()
Upgrade(node).run().wait()
Health = HealthTests(node).run.wait()
UpdateStatus(node, health).run.wait()

62. all_good = True
host = self.cdh.get_host(self.host_map[self.node_name])
if host.healthSummary != 'GOOD':
all_good = False
# Look up the host by its roles
for c in self.cdh.get_all_clusters():
for s in c.get_all_services():
for r in s.get_all_roles():
h = r.hostRef
if h.hostId == self.host_map[self.node_name]:
if r.healthSummary != 'GOOD':
all_good = False
return all_good

63. O(log N)

65. nodeComputation = for {
_ <- Status(node)
_ <- MaintenanceMode(_,node)
_ <- Upgrade(node)
nodeResult <- HealthTests(node)
} yield nodeResult
upgrade = for {
node <- group
comp <- nodeComputation(node)
} yield comp.exec
groups.map(upgrade)

66. Repo(s) CI/CD Apps
Docker Calico
Mesos Yarn
Spark, MR, Impala, etc
Marathon,
Chronos, Cassandra, etc
CI/CD
CI/CD
Repo(s)
Repo(s)
Open
Stack
Nova
Nova/Ironic
OS
KVM
OS
Firmware + Hardware + Tags

67. Workflow

68. Jenkins
Environment
Branch PR
Merge
Dev
Deploy
Master
Deploy
Test
Change
Plan

69. clusters:
green-cluster:
dns:
nameservers:
- x.x.x.x
data_domain: *.*.*
etcd:
token: green-cluster
masters:
able:
provision_id: 1
lan:
-
mac: 0c:c4:7a:c1:2e:92
ip: 1.1.11.151/24
vlan: 11
gateway: 1.1.1.1
ironic_id: a7af76ad-6583-4209-ba5f-cf1477b6405e
flavor: ramish-baremetal-flavor2
image: *mesos-master-green
theta:
provision_id: 2
lan:
-
mac: 0c:c4:7a:a9:04:0c
ip: 1.1.11.53/24
vlan: 11
gateway: 1.1.1.1
ironic_id: 8ff1fd1c-4893-11e6-a447-2f366077ca0e
flavor: ramish-baremetal-flavor2
image: *mesos-master-green
tobias:
provision_id: 3
lan:
-
mac: 0c:c4:7a:a8:f6:ac
ip: 1.11.11.52/24
vlan: 11
gateway: 1.1.1.1
ironic_id: c89fdd08-232c-40fe-b965-49fc3e4dcba7
flavor: ramish-baremetal-flavor2
image: *mesos-master-green

71. Recommendations

72. Instrument as much of deployment and
provisioning as you can

73. Optimise incrementally, learn the
right hard lessons

74. Allow for manual intervention, but
attack it aggressively

75. Encourage your people to intervene

76. Prevent Pets

77. Spend more time on testing