Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building and Deploying MediaSalsa, an Open Source DAM as Saas platform

2,598 views

Published on

Building and Deploying MediaSalsa an Open source Digital Asset Management platform as a Service.

My FlossUK 2014 talk

Published in: Technology

Building and Deploying MediaSalsa, an Open Source DAM as Saas platform

  1. 1. Building and Deploying MediaSalsa A Digital Asset Management System as a Service Kris Buytaert @krisbuytaert Slides by Michel van de Ven and Julien Pivotto
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Some books, some papers, some blogsSome books, some papers, some blogs ● Evangelizing devopsEvangelizing devops
  3. 3. MediaMosa ● Drupal-based Digital Asset Management system ● Commissioned by SURFnet and Kennisnet ● Open Source (GPLv2), Open Standards ● Webservice oriented: REST ● Store & retrieve assets ● Manage metadata using open standards – Dublin Core, Qualified DC, IEEE/LOM, CZP – OAI-PMH, BagIt ● Transcode video, audio, images, PDF &c, stream content ● Users: Kennisnet, NIBG, UGent, UvA, TiU, RUG, Radboud, UOslo, Avans, PolitieAcademie, Acquia (NBC Sport), Cineca &c ● http://mediamosa.org
  4. 4. = MediaMosa as a Service
  5. 5. MediaSalsa infrastructure (simplified) ● For each environment (OTAP) – Backends: Core service (MediaMosa) – Frontends: Optional – Web servers – Database servers – Solr servers – Transcoding servers ● Instance @ RUG DC for NL .edu ● Instances @ Commercial DCs for non-edu
  6. 6. Culture Automation Measurement Sharing
  7. 7. Inuits ● Inuits is an Open Source company – We contribute back ● +40 people in 3 countries (BE, NL, UA) ● One language: English ● We offer – Consulting ● Development ● System Administration – and a niche Saas Platform
  8. 8. Ideal world vs budget and reality → pragmatic approach
  9. 9. Distributed team Communication = HELL How to fix it?
  10. 10. Distributed team ● Daily virtual stand-up over XMPP / Hangout ● Redmine project management – Issues, tasks, sprints – Repositories – Documentation (wiki) ● Internal mailing lists ● Jenkins notifications by mail and XMPP ● Internal training
  11. 11. Dev and Ops ● Dev ● Architecture ● Develop new features ● Write unit tests (!) ● Ops ● Infrastructure (puppet) ● CD and CI (jenkins) → Ops teach dev about monitoring and distributed services → Dev teach ops about required libraries and testing, dependencies
  12. 12. Culture Automation Measurement Sharing
  13. 13. Puppet Puppet automates all the things → mcollective orchestrates all the things
  14. 14. CD ● Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP. It means every change is proven to be deployable at any time” (@ccaum) ● Puppet code – Deployed to dev environment – Same puppet code for each environment – User-triggered deployments to UAT & Prod – Feature flags in Puppet code per environment (switchable architecture) ● Application code – Continuous integration in dev – User-triggered deployments to UAT – Deployment to prod is a business decision
  15. 15. Testing ● Developers test a lot, but – The tests don’t work – It works on my machine™ – Wrong platform – Wrong PHP version Fixed now, thanks to Jenkins! and Vagrant...
  16. 16. Version Control ● Git ● Code is under revision control – Prefer small commits – Local features branches ● Infrastructure as code → git / hiera
  17. 17. Using OS packaging system ● Consistency, security, dependencies ● Uniquely identify where files are coming from ● Source repo may not be reacheable ● Little overhead when you automate ● Configuration does not belong in a package
  18. 18. Pipelines ● A collection of jobs ● Run in sequence ● Start on checkout, end on deployment ● From the developers’ side: → Git push ← Mail with changes + link to deploy
  19. 19. Our current pipelines ● Puppetized ● Puppet code and Application code (backend and frontends) ● 1 pipeline / project / customer Defining a pipeline: salsajobs::pipeline::frontend { ’qwerty-inc’: git_repository => ’ssh://git@redmine/qwerty.git’, dashboard_view => ’mds-frontends’, target_urls => { ’uat’ => ’http://qwerty.uat.mediasalsa.eu’, ’prod’ => ’http://qwerty.mediasalsa.eu’, }, require => Salsajobs::Views::Dashboard[’mds-frontends’], }
  20. 20. Pipelines steps ● Checkout ● Syntax: php -l ● Style: Drupal Coder ● Package: FPM ● Deploy to dev environment: mcollective ● Tests in dev environment: drush run-tests ● Publish package and promote: mcollective
  21. 21. Promotion ● At the end of the pipeline ● Send a simple email ● A link to the promotion page ● The changelog ● Promotion page contains one button per environment ● Must promote to UAT before Production
  22. 22. Tools used with Jenkins ● Pulp to manage RPM repositories ● Mcollective to update packages, run drush
  23. 23. Same Pipelines, Tools, Patterns are used by both devs and ops
  24. 24. Culture Automation Measurement Sharing
  25. 25. Logstash ● Collect all the logs – Drupal logs – Apache logs – Deployment logs – System logs ● Interprete, filter and correlate them ● Logstash, ElasticSearch, Kibana, statsd, Graphite
  26. 26. Icinga ● Monitor everything – vhosts – databases – cronjobs – unit test suites
  27. 27. Graphite + gdash ● Collectd ● Monitor platform usage ● FFmpeg usage ● Number of accounts ● Pipelined !
  28. 28. Culture Automation Measurement Sharing
  29. 29. Open Source ● Mediamosa is Fully Open Source ● Lots of the PuppetCode to deploy it is ● Our passwords etc aren't
  30. 30. Experiences We have just shared our experience... But also at the NLUUG Fall Conference, DrupalCon Prague
  31. 31. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ Inuits HQInuits HQ Duboistraat 50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×