Building and Deploying MediaSalsa, an Open Source DAM as Saas platform

2,447 views

Published on

Building and Deploying MediaSalsa an Open source Digital Asset Management platform as a Service.

My FlossUK 2014 talk

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
2,447
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
18
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Building and Deploying MediaSalsa, an Open Source DAM as Saas platform

  1. 1. Building and Deploying MediaSalsa A Digital Asset Management System as a Service Kris Buytaert @krisbuytaert Slides by Michel van de Ven and Julien Pivotto
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Some books, some papers, some blogsSome books, some papers, some blogs ● Evangelizing devopsEvangelizing devops
  3. 3. MediaMosa ● Drupal-based Digital Asset Management system ● Commissioned by SURFnet and Kennisnet ● Open Source (GPLv2), Open Standards ● Webservice oriented: REST ● Store & retrieve assets ● Manage metadata using open standards – Dublin Core, Qualified DC, IEEE/LOM, CZP – OAI-PMH, BagIt ● Transcode video, audio, images, PDF &c, stream content ● Users: Kennisnet, NIBG, UGent, UvA, TiU, RUG, Radboud, UOslo, Avans, PolitieAcademie, Acquia (NBC Sport), Cineca &c ● http://mediamosa.org
  4. 4. = MediaMosa as a Service
  5. 5. MediaSalsa infrastructure (simplified) ● For each environment (OTAP) – Backends: Core service (MediaMosa) – Frontends: Optional – Web servers – Database servers – Solr servers – Transcoding servers ● Instance @ RUG DC for NL .edu ● Instances @ Commercial DCs for non-edu
  6. 6. Culture Automation Measurement Sharing
  7. 7. Inuits ● Inuits is an Open Source company – We contribute back ● +40 people in 3 countries (BE, NL, UA) ● One language: English ● We offer – Consulting ● Development ● System Administration – and a niche Saas Platform
  8. 8. Ideal world vs budget and reality → pragmatic approach
  9. 9. Distributed team Communication = HELL How to fix it?
  10. 10. Distributed team ● Daily virtual stand-up over XMPP / Hangout ● Redmine project management – Issues, tasks, sprints – Repositories – Documentation (wiki) ● Internal mailing lists ● Jenkins notifications by mail and XMPP ● Internal training
  11. 11. Dev and Ops ● Dev ● Architecture ● Develop new features ● Write unit tests (!) ● Ops ● Infrastructure (puppet) ● CD and CI (jenkins) → Ops teach dev about monitoring and distributed services → Dev teach ops about required libraries and testing, dependencies
  12. 12. Culture Automation Measurement Sharing
  13. 13. Puppet Puppet automates all the things → mcollective orchestrates all the things
  14. 14. CD ● Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP. It means every change is proven to be deployable at any time” (@ccaum) ● Puppet code – Deployed to dev environment – Same puppet code for each environment – User-triggered deployments to UAT & Prod – Feature flags in Puppet code per environment (switchable architecture) ● Application code – Continuous integration in dev – User-triggered deployments to UAT – Deployment to prod is a business decision
  15. 15. Testing ● Developers test a lot, but – The tests don’t work – It works on my machine™ – Wrong platform – Wrong PHP version Fixed now, thanks to Jenkins! and Vagrant...
  16. 16. Version Control ● Git ● Code is under revision control – Prefer small commits – Local features branches ● Infrastructure as code → git / hiera
  17. 17. Using OS packaging system ● Consistency, security, dependencies ● Uniquely identify where files are coming from ● Source repo may not be reacheable ● Little overhead when you automate ● Configuration does not belong in a package
  18. 18. Pipelines ● A collection of jobs ● Run in sequence ● Start on checkout, end on deployment ● From the developers’ side: → Git push ← Mail with changes + link to deploy
  19. 19. Our current pipelines ● Puppetized ● Puppet code and Application code (backend and frontends) ● 1 pipeline / project / customer Defining a pipeline: salsajobs::pipeline::frontend { ’qwerty-inc’: git_repository => ’ssh://git@redmine/qwerty.git’, dashboard_view => ’mds-frontends’, target_urls => { ’uat’ => ’http://qwerty.uat.mediasalsa.eu’, ’prod’ => ’http://qwerty.mediasalsa.eu’, }, require => Salsajobs::Views::Dashboard[’mds-frontends’], }
  20. 20. Pipelines steps ● Checkout ● Syntax: php -l ● Style: Drupal Coder ● Package: FPM ● Deploy to dev environment: mcollective ● Tests in dev environment: drush run-tests ● Publish package and promote: mcollective
  21. 21. Promotion ● At the end of the pipeline ● Send a simple email ● A link to the promotion page ● The changelog ● Promotion page contains one button per environment ● Must promote to UAT before Production
  22. 22. Tools used with Jenkins ● Pulp to manage RPM repositories ● Mcollective to update packages, run drush
  23. 23. Same Pipelines, Tools, Patterns are used by both devs and ops
  24. 24. Culture Automation Measurement Sharing
  25. 25. Logstash ● Collect all the logs – Drupal logs – Apache logs – Deployment logs – System logs ● Interprete, filter and correlate them ● Logstash, ElasticSearch, Kibana, statsd, Graphite
  26. 26. Icinga ● Monitor everything – vhosts – databases – cronjobs – unit test suites
  27. 27. Graphite + gdash ● Collectd ● Monitor platform usage ● FFmpeg usage ● Number of accounts ● Pipelined !
  28. 28. Culture Automation Measurement Sharing
  29. 29. Open Source ● Mediamosa is Fully Open Source ● Lots of the PuppetCode to deploy it is ● Our passwords etc aren't
  30. 30. Experiences We have just shared our experience... But also at the NLUUG Fall Conference, DrupalCon Prague
  31. 31. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ Inuits HQInuits HQ Duboistraat 50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×