From Node Interactive North America 2016. In this talk, Ross Kukulinski, a NodeJS Evangelist, container enthusiast, and NodeSource technical product manager will share his cloud-native implementation of the Internet Archive’s Wayback Machine, including a live demonstration. Ross will share his goals, architecture, and technology stack choices to implement a scalable, containerized, microservice implementation of the Wayback Machine using Node.JS, Docker, and Kubernetes. Topics covered in the talk will include:- Architecture- Scaling- Monitoring- CI/CD

1. © 2016 NodeSource1

2. © 2016 NodeSource2

3. © 2016 NodeSource3
Welcome
Ross Kukulinski
Currently PM @ NodeSource
Node.js Evangelism WG
Introduction to CoreOS - O’Reilly Media
@RossKukulinski on Twitter/GitHub

4. © 2016 NodeSource4
What does “Cloud Native” mean?

5. © 2016 NodeSource
Cloud Native
5
At its root, Cloud Native is structuring
teams, culture and technology to utilize
automation and architectures to manage
complexity and unlock velocity.
Joe Beda - @jbeda

6. © 2016 NodeSource6

7. © 2016 NodeSource7

8. © 2016 NodeSource8

9. © 2016 NodeSource9

10. © 2016 NodeSource
Internet Archive
10
SIMPLE DESIG N GOALS
• The system should use only commodity equipment
• The system should not rely on commercial software
• The system should not require a PhD to implement or maintain
• The system should be as simple as possible

11. © 2016 NodeSource11

12. © 2016 NodeSource12

13. © 2016 NodeSource13
http://cloudytimemachine.com

14. © 2016 NodeSource
Buzzword Bingo!
14
Current implementation
• React & webpack
• Public-facing REST API (Swagger Documentation)
• Node.js microservice architecture
• Message queues for internal APIs
• 100% Containerized from day 1 - Docker & Kubernetes
• Cloud Storage APIs
• RethinkDB
• CI/CD using CircleCI

15. © 2016 NodeSource15

16. © 2016 NodeSource
Lessons Learned
16
• It’s fun to experiment recklessly
• Kubernetes is great for developers
• Design docs for discussion & reference materials
• Ship early & ship often
• Containers scale independent of your cloud provider
• Microservices shift complexity around, don’t resolve it
• Documenting & versioning streaming APIs is difficult

17. © 2016 NodeSource
Open source tooling
17
• Containerizing stateful services is hard!
• github.com/rosskukulinski/kubernetes-rethinkdb-cluster
• It’s ok to have opinions
• npmjs.com/package/k8s-scripts

18. © 2016 NodeSource
Future
18
What’s next?
• More tests… always more tests…
• Ephemeral environments for PR testing
• Language agnostic message bus
• Replace Redis with NATS.io
• Streaming API documentation & validation

19. © 2016 NodeSource19

20. © 2016 NodeSource
Resources
20
Jaffe, E. & Kirkpatrick, S. (2009). Architecture of the internet archive. Proceeedings of SYSTOR 2009: The
Israeli Experimental Systems Conference on - SYSTOR ’09. doi:10.1145/1534530.1534545
http://web.archive.org/web/20140828081906/http://iwaw.europarchive.org/07/IWAW2007_tofel.pdf
http://archive-access.sourceforge.net/projects/wayback/
http://www.motherjones.com/media/2014/05/internet-archive-wayback-machine-brewster-kahle
https://blog.archive.org
https://sourceforge.net/projects/archive-crawler

21. Thank you.
Ross Kukulinski
ross@kukulinski.com
@RossKukulinski