Key points
Organisation
» In the teams
Spotify's squads: the team knows about their services
» Between the teams
Responsibility boundaries, communication, shared goals
Key points
Deployment pipeline
» New services
Create new services within hours
» Continuous Delivery
Small and fast deployments
» Delivery as a Service
The teams can change the technology stack themselves
Key points
Monitoring and debugging
» Pro-active monitoring
health-checks
» Active error logging and alterting
know what is going on
» Debugging after incident
tracing between services
Key points
Fault tolerance
» Perfection, hopes or embracing faults
Because that will happen
» Architecture
A server dies
» Application
A service is down or throwing exceptions
Retry
$waitStrategy = new CountLimited(
new ExponentialBackOff(
new SleepWaiter(),
1
),
10
);
$runner = new RetryOperationRunner(
$decoratedRunner,
$waitStrategy
);
Fallback
$staticResultFactory = new StaticResultFactory([]);
$runner = new FallbackOperationRunner(
$decoratedRunner,
$staticResultFactory
);
// ...
$messages = $runner->run($operation);
Rate limiting
$rateLimit = new LeakyBucket(
new InMemoryStorage(),
new TimeRate(10, TimeRate::PER_SECOND)
);
$runner = new RateLimitedOperationRunner(
$decoratedRunner,
$rateLimit,
new SleepWaiter()
);
Buffered (ie postponed runs)
$buffer = new InMemoryOperationBuffer();
$bufferedRunner = new BufferedOperationRunner($runner, $buffer);
// Run operations
$bufferedRunner->run($firstOperation);
$bufferedRunner->run($secondOperation);
// ...
// ...
$results = $bufferedRunner->runBufferedOperations();
Get runners from YAML
tolerance:
operation_runners:
default:
retry:
# ...
callback:
# ...
buffered:
# ...
fallback:
# ...
Buffered runner
A lot of operations don't have to be executed just now.
$runner->run(new Callback(function() {
$this->pushMyBusinessMetricToElasticSearch();
}));
» Run operations on kernel.terminate event
Wrap your services into operations
just by using a tolerance.operation_wrapper tag
<services>
<service id="app.comments_client" class="AppCommentsHttpClient">
<tag name="tolerance.operation_wrapper"
methods="getLastComments"
runner="tolerance.operation_runner.default" />
</service>
</services>
Use your service as usual
$comments = $this->client->getLastComments();