Building and Deploying MediaSalsa
A Digital Asset Management System as a Service
Kris Buytaert
@krisbuytaert
Slides by Mic...
Kris BuytaertKris Buytaert
● I used to be a Dev,I used to be a Dev,
● Then Became an OpThen Became an Op
● Chief Trolling ...
MediaMosa
● Drupal-based Digital Asset Management system
● Commissioned by SURFnet and Kennisnet
● Open Source (GPLv2), Op...
=
MediaMosa as a Service
MediaSalsa infrastructure (simplified)
● For each environment (OTAP)
– Backends: Core service (MediaMosa)
– Frontends: Opti...
Culture
Automation
Measurement
Sharing
Inuits
● Inuits is an Open Source company
– We contribute back
● +40 people in 3 countries (BE, NL, UA)
● One language: En...
Ideal world vs budget and reality
→ pragmatic approach
Distributed team
Communication = HELL
How to fix it?
Distributed team
● Daily virtual stand-up over XMPP / Hangout
● Redmine project management
– Issues, tasks, sprints
– Repo...
Dev and Ops
● Dev
● Architecture
● Develop new features
● Write unit tests (!)
● Ops
● Infrastructure (puppet)
● CD and CI...
Culture
Automation
Measurement
Sharing
Puppet
Puppet automates all the things
→ mcollective orchestrates all the things
CD
● Continuous Delivery vs Continuous Deployment
– “Continuous Delivery doesn't mean every change is deployed to producti...
Testing
● Developers test a lot, but
– The tests don’t work
– It works on my machine™
– Wrong platform
– Wrong PHP version...
Version Control
● Git
● Code is under revision control
– Prefer small commits
– Local features branches
● Infrastructure a...
Using OS packaging system
● Consistency, security, dependencies
● Uniquely identify where files are coming from
● Source re...
Pipelines
● A collection of jobs
● Run in sequence
● Start on checkout, end on deployment
● From the developers’ side:
→ G...
Our current pipelines
● Puppetized
● Puppet code and Application code (backend and
frontends)
● 1 pipeline / project / cus...
Pipelines steps
● Checkout
● Syntax: php -l
● Style: Drupal Coder
● Package: FPM
● Deploy to dev environment: mcollective
...
Promotion
● At the end of the pipeline
● Send a simple email
● A link to the promotion page
● The changelog
● Promotion pa...
Tools used with Jenkins
● Pulp to manage RPM repositories
● Mcollective to update packages, run drush
Same Pipelines, Tools, Patterns
are used by both devs and ops
Culture
Automation
Measurement
Sharing
Logstash
● Collect all the logs
– Drupal logs
– Apache logs
– Deployment logs
– System logs
● Interprete, filter and corre...
Icinga
● Monitor everything
– vhosts
– databases
– cronjobs
– unit test suites
Graphite + gdash
● Collectd
● Monitor platform usage
● FFmpeg usage
● Number of accounts
● Pipelined !
Culture
Automation
Measurement
Sharing
Open Source
● Mediamosa is Fully Open Source
● Lots of the PuppetCode to deploy it is
● Our passwords etc aren't
Experiences
We have just shared our experience...
But also at the NLUUG Fall Conference, DrupalCon Prague
ContactContact
Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be
Further ReadingFurther Reading
@k...
Building and Deploying MediaSalsa, an Open Source DAM as Saas platform
Upcoming SlideShare
Loading in …5
×

Building and Deploying MediaSalsa, an Open Source DAM as Saas platform

2,286 views

Published on

Building and Deploying MediaSalsa an Open source Digital Asset Management platform as a Service.

My FlossUK 2014 talk

Published in: Technology
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
2,286
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
17
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

Building and Deploying MediaSalsa, an Open Source DAM as Saas platform

  1. 1. Building and Deploying MediaSalsa A Digital Asset Management System as a Service Kris Buytaert @krisbuytaert Slides by Michel van de Ven and Julien Pivotto
  2. 2. Kris BuytaertKris Buytaert ● I used to be a Dev,I used to be a Dev, ● Then Became an OpThen Became an Op ● Chief Trolling Officer and Open SourceChief Trolling Officer and Open Source Consultant @inuits.euConsultant @inuits.eu ● Everything is an effing DNS ProblemEverything is an effing DNS Problem ● Building Clouds since before the bookstoreBuilding Clouds since before the bookstore ● Some books, some papers, some blogsSome books, some papers, some blogs ● Evangelizing devopsEvangelizing devops
  3. 3. MediaMosa ● Drupal-based Digital Asset Management system ● Commissioned by SURFnet and Kennisnet ● Open Source (GPLv2), Open Standards ● Webservice oriented: REST ● Store & retrieve assets ● Manage metadata using open standards – Dublin Core, Qualified DC, IEEE/LOM, CZP – OAI-PMH, BagIt ● Transcode video, audio, images, PDF &c, stream content ● Users: Kennisnet, NIBG, UGent, UvA, TiU, RUG, Radboud, UOslo, Avans, PolitieAcademie, Acquia (NBC Sport), Cineca &c ● http://mediamosa.org
  4. 4. = MediaMosa as a Service
  5. 5. MediaSalsa infrastructure (simplified) ● For each environment (OTAP) – Backends: Core service (MediaMosa) – Frontends: Optional – Web servers – Database servers – Solr servers – Transcoding servers ● Instance @ RUG DC for NL .edu ● Instances @ Commercial DCs for non-edu
  6. 6. Culture Automation Measurement Sharing
  7. 7. Inuits ● Inuits is an Open Source company – We contribute back ● +40 people in 3 countries (BE, NL, UA) ● One language: English ● We offer – Consulting ● Development ● System Administration – and a niche Saas Platform
  8. 8. Ideal world vs budget and reality → pragmatic approach
  9. 9. Distributed team Communication = HELL How to fix it?
  10. 10. Distributed team ● Daily virtual stand-up over XMPP / Hangout ● Redmine project management – Issues, tasks, sprints – Repositories – Documentation (wiki) ● Internal mailing lists ● Jenkins notifications by mail and XMPP ● Internal training
  11. 11. Dev and Ops ● Dev ● Architecture ● Develop new features ● Write unit tests (!) ● Ops ● Infrastructure (puppet) ● CD and CI (jenkins) → Ops teach dev about monitoring and distributed services → Dev teach ops about required libraries and testing, dependencies
  12. 12. Culture Automation Measurement Sharing
  13. 13. Puppet Puppet automates all the things → mcollective orchestrates all the things
  14. 14. CD ● Continuous Delivery vs Continuous Deployment – “Continuous Delivery doesn't mean every change is deployed to production ASAP. It means every change is proven to be deployable at any time” (@ccaum) ● Puppet code – Deployed to dev environment – Same puppet code for each environment – User-triggered deployments to UAT & Prod – Feature flags in Puppet code per environment (switchable architecture) ● Application code – Continuous integration in dev – User-triggered deployments to UAT – Deployment to prod is a business decision
  15. 15. Testing ● Developers test a lot, but – The tests don’t work – It works on my machine™ – Wrong platform – Wrong PHP version Fixed now, thanks to Jenkins! and Vagrant...
  16. 16. Version Control ● Git ● Code is under revision control – Prefer small commits – Local features branches ● Infrastructure as code → git / hiera
  17. 17. Using OS packaging system ● Consistency, security, dependencies ● Uniquely identify where files are coming from ● Source repo may not be reacheable ● Little overhead when you automate ● Configuration does not belong in a package
  18. 18. Pipelines ● A collection of jobs ● Run in sequence ● Start on checkout, end on deployment ● From the developers’ side: → Git push ← Mail with changes + link to deploy
  19. 19. Our current pipelines ● Puppetized ● Puppet code and Application code (backend and frontends) ● 1 pipeline / project / customer Defining a pipeline: salsajobs::pipeline::frontend { ’qwerty-inc’: git_repository => ’ssh://git@redmine/qwerty.git’, dashboard_view => ’mds-frontends’, target_urls => { ’uat’ => ’http://qwerty.uat.mediasalsa.eu’, ’prod’ => ’http://qwerty.mediasalsa.eu’, }, require => Salsajobs::Views::Dashboard[’mds-frontends’], }
  20. 20. Pipelines steps ● Checkout ● Syntax: php -l ● Style: Drupal Coder ● Package: FPM ● Deploy to dev environment: mcollective ● Tests in dev environment: drush run-tests ● Publish package and promote: mcollective
  21. 21. Promotion ● At the end of the pipeline ● Send a simple email ● A link to the promotion page ● The changelog ● Promotion page contains one button per environment ● Must promote to UAT before Production
  22. 22. Tools used with Jenkins ● Pulp to manage RPM repositories ● Mcollective to update packages, run drush
  23. 23. Same Pipelines, Tools, Patterns are used by both devs and ops
  24. 24. Culture Automation Measurement Sharing
  25. 25. Logstash ● Collect all the logs – Drupal logs – Apache logs – Deployment logs – System logs ● Interprete, filter and correlate them ● Logstash, ElasticSearch, Kibana, statsd, Graphite
  26. 26. Icinga ● Monitor everything – vhosts – databases – cronjobs – unit test suites
  27. 27. Graphite + gdash ● Collectd ● Monitor platform usage ● FFmpeg usage ● Number of accounts ● Pipelined !
  28. 28. Culture Automation Measurement Sharing
  29. 29. Open Source ● Mediamosa is Fully Open Source ● Lots of the PuppetCode to deploy it is ● Our passwords etc aren't
  30. 30. Experiences We have just shared our experience... But also at the NLUUG Fall Conference, DrupalCon Prague
  31. 31. ContactContact Kris BuytaertKris Buytaert Kris.Buytaert@inuits.beKris.Buytaert@inuits.be Further ReadingFurther Reading @krisbuytaert@krisbuytaert http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/ http://www.inuits.be/http://www.inuits.be/ Inuits HQInuits HQ Duboistraat 50Duboistraat 50 2060 Antwerpen2060 Antwerpen BelgiumBelgium 891.514.231891.514.231 +32 475 961221+32 475 961221

×