Pie is the fastest, easiest, and most fun chat app you’ll ever use for work. In this presentation we reveal how Pie uses Amazon Web Services and open source software tools to power its realtime chat platform.

1. Pie AWS

2. About me
• Kuan-Yen Heng (Chris)
• Software Engineer at Pie (pie.co)
• Primarily backend and DevOps
• kuanyen@pie.co
• https://github.com/gigablah
• @gigablah

3. About Pie
• Chat for work
• Multi device sync
(web, iOS, Android)
• Rich media integration
• We build Pie using Pie!

4. Requirements
• Realtime websocket messaging
• Horizontal scalability with autoscaling
• Load balancing across availability zones
• Job queue / background worker system
• Rapid develop-build-test-deploy cycle
• Zero downtime deployment with rollback

5. Technologies used
TERRAFORM

6. Architecture
A
B
C
api
worker

7. Infrastructure as code
• We use Terraform to define and manage our
staging and production clusters
• AWS resources (VPC, Security Groups,
Launch Configurations, Autoscaling Groups,
Instances, Load Balancers) configured in HCL
• Version control for your infrastructure
• Separate planning and execution phases

8. Dependency graph

10. Cons
• Terraform is not yet mature
• Not all AWS resources and parameters
supported
• Currently not possible to port in existing
infrastructure
• Considering AWS CloudFormation

11. App/service distribution

12. Docker workflow
docker push
docker pull
registry
FROM debian:wheezy
MAINTAINER blah <me@blah.co>
RUN apt-get install rabbitmq-server
EXPOSE 5672 15672
ENTRYPOINT ["/bin/bash", "-c"]
CMD ["/usr/sbin/rabbitmq-server"]
Dockerfile
docker build
docker tag
image
docker run
container
docker commit

13. Why containers?
• Lightweight, fast startup compared to VMs
• Repeatable, consistent builds
• Dependency isolation
• Pristine host OS; only Docker installed
• Homogenous hosts, easier management
• “Servers as cattle”

14. Container orchestration

15. CoreOS: fleetd + etcd
fleetd
etcd
fleetd
etcd
fleetctl
fleetd
etcd
systemd systemd
docker docker

16. Scheduling units
• Basically writing systemd units
• Fleet specific metadata [X-Fleet]
• Schedule global units, specify constraints and
dependencies, restart policies
• Deploy units based on machine fleet
metadata, e.g. role=api and role=worker

17. MachineMetadata=role=api
Global=true
Global=true
single instance

18. > fleetctl list-units
UNIT MACHINE ACTIVE SUB
api-discovery@master_123.service 75e1c8bd.../10.0.10.xxx active running
api-discovery@master_123.service f54a4d78.../10.0.11.xxx active running
api-discovery@master_123.service 320af1d0.../10.0.12.xxx active running
api-proxy.service 75e1c8bd.../10.0.10.xxx active running
api-proxy.service f54a4d78.../10.0.11.xxx active running
api-proxy.service 320af1d0.../10.0.12.xxx active running
api@master_123.service 75e1c8bd.../10.0.10.xxx active running
api@master_123.service f54a4d78.../10.0.11.xxx active running
api@master_123.service 320af1d0.../10.0.12.xxx active running
logspout.service 75e1c8bd.../10.0.10.xxx active running
logspout.service 17291bf6.../10.0.11.xxx active running
logspout.service 320af1d0.../10.0.12.xxx active running
logspout.service e1c8ca4c.../10.0.10.xxx active running
logspout.service f54a4d78.../10.0.11.xxx active running
logspout.service d28b5a20.../10.0.12.xxx active running
logspout.service db206400.../10.0.10.xxx active running
rabbitmq.service e1c8ca4c.../10.0.10.xxx active running
rabbitmq.service 17291bf6.../10.0.11.xxx active running
rabbitmq.service d28b5a20.../10.0.12.xxx active running
worker@master.service e1c8ca4c.../10.0.10.xxx active running
worker@master.service 17291bf6.../10.0.11.xxx active running
worker@master.service d28b5a20.../10.0.12.xxx active running

19. CI/CD pipeline

20. bastion
registry
build
api
cluster
api
api
api
worker
cluster
worker
worker
worker

21. Test in Docker container
Push to registry

22. bastion
registry
cluster
hubotdeploy
“hubot deploy api:master”
pull deploy container
fleetetcd docker
dockerfleetetcd
pull api container

23. Logging and monitoring

24. collector
container
gliderlabs/logspout
Amazon
CloudWatch

25. monitoring / metrics
container
etsy/statsd
datadog/docker-dd-agent
scoutapp/docker-scout
logentries/docker-logentries
Amazon
CloudWatch

26. What’s next?

27. Amazon ECS
• CoreOS: too many moving parts?
• fleet and etcd still evolving
• Problems with btrfs
• Studying a move to Amazon Linux with ECS

28. ECS parallels
• The ECS agent container takes the place of
fleetd
• Cluster and task management through the
AWS CLI
• Task definitions in JSON
• ECS handles container lifecycle; in fleet unit
files you still have to manage your containers

29. Quick demo

30. Thank you
kuanyen@pie.co
https://github.com/gigablah
@gigablah