Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Deploying 3 times a day without any downtimeAlessandro Nadalin - VP Tech @ Namshi.com Rocket Tech Summit 2015 - September ...
When was the last time that *.namshi.com was down?
I don’t know
I should know
Probably now.
SOA
SOA ~40 services
SOA ~40 services docker
SOA ~40 services docker heavy on JS
SOA ~40 services docker heavy on JS devoted to experimenting
SOA ~40 services docker heavy on JS devoted to experimenting Result?
A bit of history...
“When are you guys gonna deploy it?”
“In 2 weeks” (2012)
“Sometime this week” (2013)
“Possibly today” (2014)
Now.
This is how it works @ Namshi
We don’t like centralization
Single Point of Failure
On error, rollback all the things!
Black sheep syndrome
Ship it
Don’t wait tomorrow for what you can get today
Result?
Microservices
microservice-based teams
~40 services * ⅔ people = ~100 engineers
~15 people rotate, assume different roles
project-based teams
quality over quantity
“adding manpower to a late project makes it later” https://goo.gl/Bc0uBf
Communication overhead takes time to get used to the whole thing
keep things small, flat, plain and simple
means small teams, simple software
What is the simplest way to get devs their development environment?
FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js
FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js Docker
How long does it take to get started with any of the microservices?
git clone git@github.com:namshi/stuff.git cd stuff echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
git clone git@github.com:namshi/stuff.git (20s) cd stuff echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-comp...
git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docke...
git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docke...
git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docke...
~ (master ✔) docker images | grep watchdog watchdog_web [...] 38.74 MB
Make it so that it takes <2m to start hacking on your services
Make it simple.
How do we get this stuff to production?
FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js
FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js It’s that simple.
Ok, but how do I get to deploy the container?
There’s a bot for everything
@c3po deploy app1 patch-1 to staging
All it takes is a 100 locs or so
Is it really without a downtime?
We’re actually down every time we deploy
it’s just that no one notices
keep a secondary setup up&running
deploy there (with downtime), switch the LB
http://martinfowler.com/bliki/BlueGreenDeployment.html
Everything seems simple, but how do you glue all of this?
github.com/namshi/roger
github.com/namshi/roger
We build following few principles:
Resilient
"Resilience is accepting your [...] reality" https://en.wikipedia.org/wiki/Elizabeth_Edwards
Robust
"Be conservative in what you send, be liberal in what you accept" https://en.wikipedia.org/wiki/Robustness_principle
Simple
"Innovation is saying no to 1,000 things" http://www.goodreads.com/quotes/629613-people-think-focus-means-saying-yes-to-th...
Safe
"Each calling function must check non-void function return values, and the validity of parameters must be checked inside e...
Pragmatic
"Each calling function must check non-void function return values, and the validity of parameters must be checked inside e...
How does the typical workflow look like?
spend time spec-ing things out before accepting them in the backlog
“if it’s not simple enough, it’s wrong”
+1 from a lead, less formal review from peers
“more deletions, less additions”
“we process 5 builds in parallel, don’t piss others off”
“don’t test everything, just what you fear”
Then, Go Live.
Takeaways?
break everything down
MVA
MVA novice engineers are the ones who can’t abstract
MVA novice engineers are the ones who can’t abstract senior engineers are the ones who can abstract
MVA novice engineers are the ones who can’t abstract senior engineers are the ones who can abstract great engineers are th...
heterogeneous, committed team
avalanche of go-live
failure is always an option
Dystopian architecture
Dystopian architecture http://www.infoq.com/news/2013/05/dystopia-as-a-service
Challenges?
queues of deployments
parallel deployments based on cookie
everything is new
use the internal infra as a testing platform
manual QA
simplify the dangerous stuff, think 3ice before touching
think mobile
Desktop is the new Blackberry
Whether you are running everything through an isomorphic JavaScript framework
every day https://eng.asana.com/2014/02/scaling-asana-com/ & https://asana.com/luna
or fatass ruby frontends
30 times a day https://github.com/blog/1241-deploying-at-github
or you’re just that crazy
every 11.6 seconds https://news.ycombinator.com/item?id=2971521
(in 2011) https://news.ycombinator.com/item?id=2971521
Roll it out live.
github.com/odino twitter.com/_odino_ odino.org VP Technology github.com/namshi twitter.com/TechNamshi tech.namshi.com
CREDITS https://www.flickr.com/photos/keoni101/5145003933/in/photolist-8QDuqn-9onbtL-98NqNJ-bpre1J-bCm9ov-5nspQa-gFXppe-7R...
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin
Upcoming SlideShare
Loading in …5
×

Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin

1,636 views

Published on

A look at how we try to make our architecture robust, resilient and fun to work with: Namshi is not github or spotify but... ...imitation is the sincerest form of flattery!

Published in: Technology
no profile picture user

  • Be the first to comment

Show More

Deploying 3 times a day without a downtime @ Rocket Tech Summit in Berlin

  1. 1. Deploying 3 times a day without any downtimeAlessandro Nadalin - VP Tech @ Namshi.com Rocket Tech Summit 2015 - September 2015 - Berlin
  2. 2. When was the last time that *.namshi.com was down?
  3. 3. I don’t know
  4. 4. I should know
  5. 5. Probably now.
  6. 6. SOA
  7. 7. SOA ~40 services
  8. 8. SOA ~40 services docker
  9. 9. SOA ~40 services docker heavy on JS
  10. 10. SOA ~40 services docker heavy on JS devoted to experimenting
  11. 11. SOA ~40 services docker heavy on JS devoted to experimenting Result?
  12. 12. A bit of history...
  13. 13. “When are you guys gonna deploy it?”
  14. 14. “In 2 weeks” (2012)
  15. 15. “Sometime this week” (2013)
  16. 16. “Possibly today” (2014)
  17. 17. Now.
  18. 18. This is how it works @ Namshi
  19. 19. We don’t like centralization
  20. 20. Single Point of Failure
  21. 21. On error, rollback all the things!
  22. 22. Black sheep syndrome
  23. 23. Ship it
  24. 24. Don’t wait tomorrow for what you can get today
  25. 25. Result?
  26. 26. Microservices
  27. 27. microservice-based teams
  28. 28. ~40 services * ⅔ people = ~100 engineers
  29. 29. ~15 people rotate, assume different roles
  30. 30. project-based teams
  31. 31. quality over quantity
  32. 32. “adding manpower to a late project makes it later” https://goo.gl/Bc0uBf
  33. 33. Communication overhead takes time to get used to the whole thing
  34. 34. keep things small, flat, plain and simple
  35. 35. means small teams, simple software
  36. 36. What is the simplest way to get devs their development environment?
  37. 37. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js
  38. 38. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js Docker
  39. 39. How long does it take to get started with any of the microservices?
  40. 40. git clone git@github.com:namshi/stuff.git cd stuff echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
  41. 41. git clone git@github.com:namshi/stuff.git (20s) cd stuff echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
  42. 42. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts docker-compose up
  43. 43. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docker-compose up
  44. 44. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docker-compose up (depends on the image)
  45. 45. git clone git@github.com:namshi/stuff.git (20s) cd stuff (...) echo “127.0.0.1 stuff.namshi.com” >> /etc/hosts (...) docker-compose up (~5s with alpine)
  46. 46. ~ (master ✔) docker images | grep watchdog watchdog_web [...] 38.74 MB
  47. 47. Make it so that it takes <2m to start hacking on your services
  48. 48. Make it simple.
  49. 49. How do we get this stuff to production?
  50. 50. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js
  51. 51. FROM alpine RUN apk add --update nodejs COPY . /src CMD node /src/index.js It’s that simple.
  52. 52. Ok, but how do I get to deploy the container?
  53. 53. There’s a bot for everything
  54. 54. @c3po deploy app1 patch-1 to staging
  55. 55. All it takes is a 100 locs or so
  56. 56. Is it really without a downtime?
  57. 57. We’re actually down every time we deploy
  58. 58. it’s just that no one notices
  59. 59. keep a secondary setup up&running
  60. 60. deploy there (with downtime), switch the LB
  61. 61. http://martinfowler.com/bliki/BlueGreenDeployment.html
  62. 62. Everything seems simple, but how do you glue all of this?
  63. 63. github.com/namshi/roger
  64. 64. github.com/namshi/roger
  65. 65. We build following few principles:
  66. 66. Resilient
  67. 67. "Resilience is accepting your [...] reality" https://en.wikipedia.org/wiki/Elizabeth_Edwards
  68. 68. Robust
  69. 69. "Be conservative in what you send, be liberal in what you accept" https://en.wikipedia.org/wiki/Robustness_principle
  70. 70. Simple
  71. 71. "Innovation is saying no to 1,000 things" http://www.goodreads.com/quotes/629613-people-think-focus-means-saying-yes-to-the- thing-you-ve
  72. 72. Safe
  73. 73. "Each calling function must check non-void function return values, and the validity of parameters must be checked inside each function." http://sdtimes.com/nasas-10-rules-developing-safety-critical-code/
  74. 74. Pragmatic
  75. 75. "Each calling function must check non-void function return values, and the validity of parameters must be checked inside each function, unless you are 1000% freaking sure" http://sdtimes.com/nasas-10-rules-developing-safety-critical-code/
  76. 76. How does the typical workflow look like?
  77. 77. spend time spec-ing things out before accepting them in the backlog
  78. 78. “if it’s not simple enough, it’s wrong”
  79. 79. +1 from a lead, less formal review from peers
  80. 80. “more deletions, less additions”
  81. 81. “we process 5 builds in parallel, don’t piss others off”
  82. 82. “don’t test everything, just what you fear”
  83. 83. Then, Go Live.
  84. 84. Takeaways?
  85. 85. break everything down
  86. 86. MVA
  87. 87. MVA novice engineers are the ones who can’t abstract
  88. 88. MVA novice engineers are the ones who can’t abstract senior engineers are the ones who can abstract
  89. 89. MVA novice engineers are the ones who can’t abstract senior engineers are the ones who can abstract great engineers are the ones who know when to abstract
  90. 90. heterogeneous, committed team
  91. 91. avalanche of go-live
  92. 92. failure is always an option
  93. 93. Dystopian architecture
  94. 94. Dystopian architecture http://www.infoq.com/news/2013/05/dystopia-as-a-service
  95. 95. Challenges?
  96. 96. queues of deployments
  97. 97. parallel deployments based on cookie
  98. 98. everything is new
  99. 99. use the internal infra as a testing platform
  100. 100. manual QA
  101. 101. simplify the dangerous stuff, think 3ice before touching
  102. 102. think mobile
  103. 103. Desktop is the new Blackberry
  104. 104. Whether you are running everything through an isomorphic JavaScript framework
  105. 105. every day https://eng.asana.com/2014/02/scaling-asana-com/ & https://asana.com/luna
  106. 106. or fatass ruby frontends
  107. 107. 30 times a day https://github.com/blog/1241-deploying-at-github
  108. 108. or you’re just that crazy
  109. 109. every 11.6 seconds https://news.ycombinator.com/item?id=2971521
  110. 110. (in 2011) https://news.ycombinator.com/item?id=2971521
  111. 111. Roll it out live.
  112. 112. github.com/odino twitter.com/_odino_ odino.org VP Technology github.com/namshi twitter.com/TechNamshi tech.namshi.com
  113. 113. CREDITS https://www.flickr.com/photos/keoni101/5145003933/in/photolist-8QDuqn-9onbtL-98NqNJ-bpre1J-bCm9ov-5nspQa-gFXppe-7RZ4xk-8Ct1ds-7KgJJQ-bWMzAU-9BiH5F-bprhCU-bCmcZK-bprhuE- nXqr1M-bCm9BF-9JcPvT-bprdBS-bCm8oM-gGBZfK-5iatLr-8EZw4m-8UHQhk-8SCpW4-qP9K5T-agZDMz-eP4mF-jSRwF-k5MQvX-8mfGks-fZ3Ybf-fZ3YdE-fZ3Yvo-fZ3Vc8-fZ3LjY-acpv98-TXHG-k5SvYL- 8EVHuL-k5TK4G-k5PakZ-xtWznE-7jUCPz-ptnrW5-9EKb64-549pn1-nWGvvK-5nwJRW-bbBnct https://www.flickr.com/photos/kodomut/6736910191/in/photolist-bgjqFZ-oK6wi7-cAPsgw-du2eXy-fPYjvm-Ldmf-8UJyvL-cjuPWh-7ouhnG-3jFShG-7cDUKd-5RGCpJ-okMaW-8XNkq-kZXpCS-4zLS4V- bpq2Be-6gCEDA-nPDiLh-8CsTbz-8UFvqD-eUNqSW-kBNAV-8s5ooq-xroyz-qrWR6-nEoXY-2aNAQA-oBKJZZ-4EYx1j-czxuvm-8UJz39-6vYKE8-aahUEo-8UFwz2-epWYsQ-9tkdUT-ejb2fA-9bGFZ8-sDNq9j- tNcTSP-6PQjU-kutg5F-b1qbNa-axLrfZ-tNcDh6-emc9G6-6XpcCq-8UFvKg-9tkecD https://www.flickr.com/photos/jfgornet/9782373576/in/photolist-fUrdRo-iiRme5-3dphWV-qbi7CS-295V7V-oVYVkK-whiv-5bSqL6-bCAzX7-e43Tvu-5bXgrN-9Po1MM-9PrdzW-eVaqAj-aDdYbB-cryeCf- d2W3u-fUr7Ca-8QPJin-bRvh2i-7bHRv8-9jDPRs-ARgb7-unsLMg-5Rmbqk-gpzfbq-4uS6uw-s2nje-v2HTyQ-7mKESL-e1RHJu-7oAes-8iLVD3-dq3xC9-67yUML-dq3nBr-dq3xiq-viLYuJ-uni1uh-dq3nFK- bpC4Cd-crycDs-vjCJN4-v2RhyV-9PqRLW-8ZMuCv-dq3xqh-dRi3hi-wmRFPW-8yVu3e https://www.flickr.com/photos/bill-fellow/4059471685/in/photolist-7bHRv8-9jDPRs-ARgb7-unsLMg-5Rmbqk-gpzfbq-4uS6uw-s2nje-v2HTyQ-7mKESL-e1RHJu-7oAes-8iLVD3-dq3xC9-67yUML- dq3nBr-dq3xiq-viLYuJ-uni1uh-dq3nFK-bpC4Cd-crycDs-vjCJN4-v2RhyV-9PqRLW-8ZMuCv-dq3xqh-dRi3hi-wmRFPW-8yVu3e-v2qEcV-vjCGGv-rXSAuw-2sAvon-qmT7s4-9YWFmv-4sJryS-9zMU3w- 67rfWu-67mARP-67qDSS-ebSx8M-2URQyo-3Aphtk-6W3md6-5W36P7-4hkUXk-crye1L-295V8t-jeVgze https://www.flickr.com/photos/alan-light/7615187330/in/photolist-cAVPTh-hBX91g-hcpiXt-6n2KSB-7BcSDW-6ZZBB2-7BcSv3-9KdE9P-hBXcre-MAxEu-Lwj7H-bCgeBe-euumXn-cBWjH-a8NebL- 5ZooTX-9KdE4T-pMk9k9-oD6BGT-ZqpRg-cURCRh-52xgrx-8p2uw8-9BPa4-518mMG-662bdz-xGaDRX-2zSHXX-5SxTvs-8K1pKQ-53xG2q-53truc-9s2uWR-cHDiXs-8rad18-dFFXGM-9s36vn-9s5TGu- fEdhRE-5j7ekh-5j7eiW-6ii1k3-9mQH9Q-9KdEea-6HriJ1-6VZwQT-87sR7j-5aTwhd-bFzizn-59Dkdi https://www.flickr.com/photos/virtualsugar/372136216/in/photolist-yTi8E-maMzdd-8RFFez-bEQKGX-fEAEJ3-t5s62j-29mCH-dykJ1U-qgFzoc-ffh4SP-5mTEZY-cvijZ7-xJ38eN-dHxCmf-qwZcFJ- mQ9EYH-ciUqAG-6VUTzH-9zRGmF-9KeU3d-2DVBG-3e8A-9yvWyr-bHqKkp-cc7knY-92u8oY-iyeiSA-4TUSm1-4ZD3vw-qrjaAz-4dSJ4e-buvXYL-rwXX4Z-9NUE1F-5rwYT5-rxpxx1-oDCcdr-viNKVU- 5oM9mP-ax7WJ4-hLCvT-nwfVm-dyff3V-DykQR-by9ag-7DzVio-63NxSa-9Lm6rU-9hme9H-9Lm5Ej https://www.flickr.com/photos/greenoid/16365496287/in/photolist-qWarT6-5N947S-bDBRvz-51uAdq-5ZEPea-dZbtit-atypb1-fnuXsK-cSbmLN-6nPMgk-cHGG4A-8Lr7o3-8Lo4CK-6n2W2z-bQmB9K- 5x2oRB-8Lr8MW-8Lr8cE-seCNBZ-5dRyGJ-im4teH-9yxT46-9yATJE-7dWhf1-24J4S-bBoVQo-jgHyNo-cs1Q47-p7EHQp-9irsTV-9ktqn1-9HmoJD-y4fGn-fpF8U8-85R4Yj-76jHPe-8ByuRV-4ZwkyA-67DsR2- b27Jzc-9sfnyg-6nfwZm-8E3w4T-7Di8BH-7Di8ot-7Di8bp-bLdvw-ekLdGQ-jEWJfL-icwf4n https://www.flickr.com/photos/aigle_dore/15571693270/in/photolist-pJ1ZUb-8omGZn-pjwoda-pHZeyg-7sMyvU-5QjZw-k253ZL-4Jhwqr-vmPmUH-oXoNz5-awif3w-9jKjEN-ea6V4H-91qx9Q-qUPGF2- 7SrRAw-rWqAho-5HFn2P-wEp54m-3dTtJR-9om4e5-2NENJw-a3yfbr-of4Vwg-p9xCZs-54e3RM-wxbkv8-kjQiui-rWyKW8-b2Jq4R-wG699M-iiEtfF-oRPrpe-vXGiLo-boRv1i-wxweWU-6qpJih-hZ2n8J- cW8Kfs-xLBstF-tKKQUm-5PPqVj-8omhkX-ppMdyd-aCzjww-dHmpqy-se2nf2-dGka9m-bVSXic-dfJeU7 https://www.flickr.com/photos/intvgene/370973576/in/photolist-yMkw9-2auo-97ji6E-4wuofP-tGWXu6-59ktec-59pCrN-59pBRW-5VxirZ-9U3veb-8sWpQY-iSEFkV-9cfBr2-9boLH5-6mSB7s-aKpAmx- 4fUsNL-7vEVv7-8BPU1e-5hC1EA-4j1h7w-9aTAwj-7vEVHQ-87V3bY-cee2wo-3KDNbn-9SiXWP-7ASTkd-4jpvQH-7vB7fR-59pGDh-jagpmg-59pEN7-8nDt9B-6xFmn-ruwDs-hfwXP-4qBiH6-59pHBJ-aCvLMi- 59kxJk-5B3jzN-4qB8yx-7TBNEZ-59pKod-ehoEq3-okMnQt-4fUzTo-7vB7oB-6b4BVY https://www.flickr.com/photos/levork/2869703078/in/photolist-5nzYKh-4iSxJx-h84kt-7zQ7Zc-96Bkfr-4DB4yG-9vNc4G-8z9rd4-jb4gtt-xj8dC-p2EwTb-7Qbfeu-7Jqm5K-4UAU46-k2MVwr-xj8mr- 97mizy-enB6mH-enDw5D-enE9AK-96VRju-7ZC3Mu-9Ty6t2-c3fj5-aB3JJg-aB6gM3-aB6ePw-6HvAUJ-6HrxUc-3aQjbL-bY4Vv3-6gwRWM-86uABy-6HvAN7-pE83F3-7HNQgW-7HNQ25-7HJUhk-dtQ5L6- dtQ5y4-dtQ5g2-dtQ5M8-dtQ5yZ-dtVDAy-dtQ5uV-dtQ5gv-dtVDLQ-dtVDu1-dtVDdm-dtQ5AP https://www.flickr.com/photos/bribri/267441683/in/photolist-pCH6v-rLg5vA-fJc7bv-fJfSdF-fJxnbQ-fJfNri-fJxm33-6mNbXo-cYAyXs-4qTrcR-9gTBHx-6xTonW-fJtGmq-3UM5cm-5MquU1-uqnsNy- 5EGt56-dQBQtP-emiLxz-74hbx5-igpt93-pwFrQh-pv3MGX-ofL8zQ-6RDxtD-dMY3D4-r6xmyT-doxPLP-v5VCax-5z5qJV-p1u3gM-r3WFZr-bxVboc-9AXWjP-pMu5wR-awx1qC-4ZoEL1-9QeBRK-tZ8Ndq- 9RhoRv-6mNaDj-6mHYKp-6mJ2ip-6mNaTU-6mHYhk-fJc7qV-7JtbPS-bEsPJC-6TqzCL-4aGX1s https://www.flickr.com/photos/leehaywood/4215672566/in/photolist-7qwqzL-612xaV-xrbJu9-r2EX8b-o4ZJ4i-5Ydqw4-eimaUN-7KF23a-csn2TY-83RpRh-earXnG-8noT5N-pN3mtW-8QU4ii-pDJzKg- 6NroHq-6nTq7H-6ypk1S-dNxLKB-93a3ez-93d9xS-5uChhT-ATm2Y-8CgeyZ-o87Kh-db38BP-3zS9vf-qj6ZnP-93d9cE-93a25B-cB9MEA-2FJcT-dTUsFR-6JG6He-6vhRC-939YAZ-4Q4Rta-37H1HU-8r8Aaj- 8YffL4-6CVtJ6-aCjbRF-aeujs6-fvL94X-8F8azb-8S6Axf-5ZhqeG-ib6Z8-8yhgcf-939ZaD https://www.flickr.com/photos/michaelpardo/16239379663/in/photolist-qK24Nx-5zNShb-7hdF1N-aCGDqT-afSi2z-oF4Q25-4vg5r4-53Zde-5ZSEaZ-45ykWT-hXsmSY-5gAZhi-eBWbNx-kdhPYV- e6Z13z-9Qf8CN-4JNbcv-4TLGoA-9BV2oD-k54mxX-nGTp9Q-6Bh5W8-iYfjhp-a1rCX-qHHLFw-7xJmdm-6cnNX3-6rg4Hy-4oiMrb-dWy8JX-5MsJqJ-g8PYx8-oVfk37-oSLAA2-6hSNgn-5yXjAn-3eWKix- c5L67s-65vYNC-2kbdGp-6RjTi4-33gRyW-aqCdny-9JvstH-nJFecd-6Ky3Ep-duWt8G-d2XwjL-52iHYy-ouB4oh https://www.flickr.com/photos/by-sgillies/4612006879/in/photolist-82xJXZ-cH76wU-cH74UW-cH7241-7NavgS-qSR3Xh-7J5ZjK-4no65H-digoTb-rrf9DT-iyh6mh-nHwfDd-dS9xSd-nRiqpa-7Nav6q- 5Xetug-fL5h6a-cH7ZhL-fL5gK2-cH81Cu-fLmU6E-pXevyv-cH83gf-k5XBHG-k5XCGW-uTyZeN-G9G6P-o82Pzi-pXG2eb-fQAYgs-pV9mkH-4jJBCj-6RTT4E-9MMPxt-bEny6x-7y2oog-6RBMg4-oJ5d1-82xJYP- boW8ch-8mLDsb-7N6vaZ-9ZADoF-hhHCC-9Zu1rK-5CAxDp-pXew48-8zWQaY-8zTHCt-pVshzw https://www.flickr.com/photos/neontommy/8195166220/in/photolist-dubnpj-39syS9-5o5PPy-j7D7EJ-6w1VMD-8CPRdi-ijya6J-j7C6kS-am7F3K-nCWmUT-5o5PsS-j7zRvk-ik38xz-fJ7fv7-39sG4U- pgQ9Vq-pgQsL3-j7AFHU-fHPqjn-78k5jh-fJ79DU-2Z5exe-2Z9xEo-2Z4ZZ4-78k5ts-fHPGkD-5mmzdQ-du5KYH-d8oyUJ-j7Ekx7-j7DfTj-fJ7aXL-dsvB5q-gSsvhS-jvpHqe-78k4yQ-fJ7bc5-kpH4fu-fJ7fbq- wae8P6-5seA8D-2Z9FLS-fJ7eFG-ik2B4j-5nXctH-pzdFgF-78k5bW-nXdjwF-8BkoQ5-8Bko8y https://www.flickr.com/photos/usaghumphreys/8123584654/in/photolist-dnRuGo-dnRChm-dnRwnQ-dnRjct-dnRpVD-dnRoMt-dnRtjG-dnRmdM-dnRzdJ-ciiL4u-5fHLEC-d4FjFC-d4Fpz3-d4FqtL- d4FnAE-d4FmsG-oYAmF3-eQNeGZ-2kEMeF-75qwdj-ebyApV-wLi3HY-bfRnrT-d6mRoo-c3xXnN-bfRnmF-tCgNcj-ucvFdf-eh4VY3-oYACLu-oKxku2-2JYUEY-2JYVbC-w94YoH-vccJTi-pGTvix-4PuaWC- 9Roc2Y-d7tX4A-w25JNv-dmGuP6-cLDcUd-brBPoP-dgpf8B-eSZMX9-edMWRi-q53feK-waq2BV-cK6QBL-2VRKtL https://www.flickr.com/photos/twid/3013680713/in/photolist-5AiUhi-5Aobef-5AoaZj-qj1xqY-oGXJb-LJsXv-8hNkFU-aqzAxZ-pn5z1W-6G2dgg-5p3Xf7-pjmPrf-c6GraN-bCbSK5-5p3XmY-6G4BYN- 6ABLFC-8342MV-6p1CvA-nLZJK8-5sbZyn-bz6bV3-snq1XE-7MZgN6-pncatE-c5Abhy-4FVwG9-a3Np9x-qJLptH-oXsh3G-bDVGiR-6oWAog-4igtMe-8jAq4w-97MixC-6AUZt-47yNA5-9bcWfH-7CFnNH- 4XnWfX-6ZYojh-2yTdoG-qMQvWw-63yGHT-92FCN4-4FVwcy-fpg8-pEEow9-bE9fDc-uw1FLn https://www.flickr.com/photos/robin1966/16252834424/in/photolist-qLd2rb-71XJX3-oFh9Lf-pbGmj7-9piRsv-wV3Voc-dHydjE-97ZEyz-8NEx3U-ntssed-49mxGt-dHydTq-9kbHQZ-vCShQ-gs9VG3- 97Zy1r-cDwCAS-4UDXk1-82WxrU-89ubgX-dHsMCB-k28MG3-bA2uTS-cZ9Ktw-cXmS4G-dHebrV-dedzHL-cXgdzY-jHn9PM-ekKpLa-97ZyPa-bEWeyt-acAuAB-9kbJNP-cXmmXS-cXfDbh-cYBCQy-goTmeg- dtG73y-cXjfMq-oehG4N-4YUVVD-nYQ2NE-82TpLP-9teNoy-9ZBiFi-cXgwFN-cYRXwh-khZCKr-cXfRBw https://www.flickr.com/photos/hktang/4243300265/in/photolist-7sY2kp-3aFY7-anLPHW-iUCPSN-riy1sB-9Qk6W7-foYszt-5gvyUL-dVy7PS-98FH5a-iKHZLF-51mAYh-5gqShD-ksPKNd-r1qxFW- 375zuh-pGvQbe-7x3sBg-pPnsBG-4nkFqL-cPrJzS-9t5oE8-vc5KA-reiprC-vTcSxB-5V9pju-7HHHCa-7vXv6S-9ywJev-nPf5ar-6zveL1-93ysED-4MpKK9-2VUyZL-9xMCJe-dSwkqa-7bmxL7-u9RHxu-51mAxC- ffDKR8-pPh5H-r1QJ1N-XP4Qt-7hLVq7-9R8Dv4-hNGJ65-fLBHyT-hNGTwh-9DpTpt-4LCh5p https://www.flickr.com/photos/gsfc/12209193235/in/photolist-jATiMD-eWpyyD-aG3Pcc-aUux8-5Uc9Fn-eWB8XC-2BQcU-qiyn4c-7PGYNQ-phJDQk-eWpHhp-cSiy4U-axXAYt-7BRK6G-7PDTLx-7Q5tpT- 2BQeH-aqCa22-7Ptm4u-pdYMRC-do2QEF-5stXWy-6y4QUE-5spAvg-aqCa32-dr1P3R-c3ocB5-bizfM-7Q1xwZ-4CViS4-9y3cgk-dx76a9-7PDXj6-bCRXLB-86mdyj-axZEky-7PDS94-oY1BkX-axZxvE-8K8Sw7- axZMsW-8Nxttu-aqC9Vg-7dMYwz-82VGZM-Maa4f-bizgK-6FPXuX-7PXyc5-7PDV34 https://www.flickr.com/photos/peterme/1253686915/in/photolist-2UMtrD-2UMBv6-2USbfE-2URYmh-2UMBVp-2UMv54-2UMzQF-q8h49G-kJkWG-2UMMmr-2UMMRT-2US6P3-2US695-2UMGgT- 2UMxn4-2US4pf-4JNAp4-257iwn-atU3wi-8jbQxi-oMsNA-6P7otR-fhfVs7-naBjky-adVc4z-4qFFE2-pbpkgi-pQMaf8-bLANFx-5SUYtc-pQM9RH-edxG5v-bLANBX-8Lejk5-AD3JQ-q7YtFx-pkDRki-5xgGUy- 5us9C9-257jdr-drtUTS-kYEvbW-dx6D7U-X7oNE-6Rseze-b5J9GP-4Af9Wu-5SQ13B-pbmApw-naBoL9 https://www.flickr.com/photos/opensourceway/5537457201/in/photolist-9rjV72-c3vzMW-6TyZng-58xAj6-bwdC97-8fL4R6-rCE4XT-vYaSD2-7JUU7P-atCKP5-6Q7466-4boAmW-gFWgPr-93Nfb3- yUbTj-q1TobB-D9s7z-86Kywi-cGMG9S-K4mGV-bER864-8eSYcF-4bjyva-8SXnr1-wJh5Vf-vcxb7L-nmSXYZ-i6KbA-4bozRh-pU4HhE-63narx-4MjXTb-4ZAk72-47Apyk-bWhBF1-4YSqGT-7RC7eF-9fEkF3- 5uZMFQ-pkDuwi-asX4ws-514qqR-8S4Xon-HgpXC-K6KDR-7Q7xJT-ee6kCu-8272cg-edHTVF-3gNReU

×