Retour sur des conférences assistées durant DrupalCon 2016 à Dublin avec des sujets DevOps mais également Drupal 8 : docker, kubernetes, jenkins 2 pipeline, Lcache, configuration management, migrate, cache

1. DrupalCon
Dublin
2016
Capgemini - Digital Factory - Nantes

2. Yann Jajkiewicz
@yjajkiew
https://yann.me
Jérémy Greffin
@jeremygreffin
Github : jgreffin

3. DrupalCon Dublin 2016
Introduction

4. 3 joursAvec jusqu’à 11 conférences en parallèle
10 667Tasses de café
1800Personnes sur les 5 jours

6. Timeline
DevOps Drupal Echange

7. DevOps
Docker
From socketwench
https://goo.gl/RFJ6Un

8. Why Docker ?
› Containers
› Sandboxes everything
› Instructions aren’t enough
Docker vs Vagrant :
› Vagrant = multiple VM
› Docker = 1 VM for multiple containers

9. Physical hardware
Host OS
Hyperviseur
VM OS VM OS VM OS VM OS
Server
A
Server
B
Server
C
Server
D
Vagrant :

10. Physical hardware
Host OS
Container
A
Docker :
Docker utilities
Linux Kernel
Docker engine
Container
B
Container
C
Hyperviseur

11. How to Docker ?
Docker files
› Construction d’une image
› Description de l’environnement en YAML
› 1 fichier par processus
Docker compose
› Automatisation du déploiement des images
› Définition de l’architecture
› 1 commande pour tout installer

12. Docker & Drupal
Go shopping...
› Docker4Drupal
› Redis
› Memcached
› MySQL, MariaDB
› D7 & D8
› . . .
… or build your own

13. DevOps
Kubernetes
From tpryan
https://goo.gl/3k23PK

14. Problème
Gestion complexe des VMs
et des containers
Kubernetes
Solution
Kubernetes gère “magically”
différents containers au sein
de différentes VMs

15. Kubernetes : késako
› Open Source
› Started by Google
› Système d’orchestration de containers
Paradigme :
› Build scripts VS Desired state
› Children VS Employees

16. Kubernetes : paradigme
Employee
“We had a tough day, go
home and get some sleep”
Child
Go upstairs
Get undressed
Put on pajamas
Brush your teeth

17. Kubernetes : how it works
› Docker files
› Pods (1 or + containers), YAML style
› Pods controller :
› Réplication, déploiement, service
› Gestion des environnements (prod, test)
› Autoscaling
› Logging, monitoring, web interfacing, cluster

18. Docker
compose
Docker Machine
Docker
Kubernetes
Multiple
containers on
same localhost
Cluster of
container hosts
replication
orchestration
scheduling
Manage
Remote
Container
Hosts
Launch Container hosts in several clouds

19. Everything at Google runs on Containers
› Gmail, Web Search, Maps,...
› MapReduce, batch,...
› GFS, Colossus,...
› Google Platform (VMs run on containers
underneath everything)

20. DevOps
Jenkins 2
Pipeline
From aroq & shumushin
https://goo.gl/gahNsm

21. Why Jenkins ?
› Release process :
› Frequent
› Automated
› Repeatable
› Lowrisk
› Build, deploy, test, release
› Continuous delivery
› Continuous deployment

22. Jenkins : Pipeline
› Plugin
› Jenkins 1 Workflow → Jenkins 2 Pipeline
› Can survive restarts
› Pausable (for human input)
› BlueOcean Pugin → better GUI
› Jenkinsfile : container for the pipeline
› Multibranch pipelines : automatic
pipeline creation for branches

23. Jenkins : pratiques Drupal
› Ne pas commit “vendor”
› Pas de composer install ou update en PROD
Solutions :
› Build, tarball, sync (doesn’t work with Acquia)
› Target repo : commit > CI > Build > commit >
deploy

24. Jenkins : workflow
Git commit &
push
Jenkins
Pipeline
Gitlab webhook
Init Config
Build
Stage
prepare
Stage deployStage testing
Prod prepare Prod deploy Prod testing Finalize

25. Jenkins : tools

26. Jenkins : tools
› Docman - one or more source repos +
docman config > docman > target repo
› Druflow - deploy drupal-procedures
› Drush - especially drush aliases
› Behat - testing framework
› Pipeline - to orchestrate
› Pipeline libraries - make it DRY

27. Drupal
Performances

28. Redis or
Memcache
Web
Server
Web
Server
Web
Server
Web
Server
Bottleneck
Traditional caching implementation
By David Strauss

29. LCache
It combines:
› A well established architectural patterns
from modern CPU architecture
From platform.sh
https://goo.gl/1mdkVT

30. Core 0
Existing solutions: Multi-Core Processors
CPU
Core 1
CPU
L1 cache L1 cache
L2 cache

31. LCache
It combines:
› A well established architectural patterns
from modern CPU architecture
› Efficient data propagation from mysql’s
row-level replication
From platform.sh
https://goo.gl/1mdkVT
› Experience on Valhalla to provide an
efficient website object caching

32. Better
› Frequently read
› Rarely written
› Large
LCache
Worse
› Read once or not at all
› Things handleable
earlier in the stack
› Keys updated often
› Clearing 100+ keys with
a tag (because of
replication)
From platform.sh
https://goo.gl/1mdkVT

33. D8 : practical caching
3 méthodes de cache :
› Cache anonyme
› Reverse proxy caching
› Cache authentifié
From platform.sh
https://goo.gl/1mdkVT

34. D8 : practical caching
› Cache anonyme:
› Module : Internal page cache
› Activé par défaut dans D8
› Page cache middleware
› Default request policy : no CLI, no session
› Default response policy : no kill switch, no
“no_cache” route, no server errors
From platform.sh
https://goo.gl/1mdkVT

35. › Cache anonyme :
› Kill switch
Drupal::service(‘ page_cache_kill_switch ’)->trigger();
› No_cache route
system.cron:
path: ‘cron/{key}’
defaults:
_controller: ‘DrupalsystemCronController::run’
options:
no_cache: TRUE
From platform.sh
https://goo.gl/1mdkVT

36. › Reverse proxy caching :
› Varnish, CDN, ...
› Important : définir le “page maximum age”
› Définir le reverse proxy dans settings.php
$settings[‘reverse_proxy’] = TRUE;
$settings[‘reverse_proxy_addresses’] = array(‘127.0.0.1’, …);
$settings[‘reverse_proxy_header’] = ‘X_CLUSTER_CLIENT_IP;
From platform.sh
https://goo.gl/1mdkVT

37. › Cache authentifié :
› Username block depends on user logged in
› Admin links depends on permissions
› Help block depends on URL
› Solution : cache contexts
› contexts => [‘user’]
› contexts => [‘user.permissions’]
› contexts => [‘url’]
From platform.sh
https://goo.gl/1mdkVT

38. › Cache authentifié :
› How to invalidate ?
› Invalidate dependencies ?
› Solution : cache tags
› tags => [‘node_list’]
› tags => [‘user:1]
› tags => [‘node:1’]
From platform.sh
https://goo.gl/1mdkVT

39. › Cache authentifié :
› How to decide to cache ?
From platform.sh
https://goo.gl/1mdkVT

40. › Cache authentifié :
› Exemple :
› Metadata :
From platform.sh
https://goo.gl/1mdkVT

41. D8 : practical caching
› Cacheability debugging :
› Module renderviz : permet de voir les
metadata dans l’inspecteur du navigateur
From platform.sh
https://goo.gl/1mdkVT

42. Drupal
ElasticSearch

43. Drupal & search
› Core module for search
› Last significant update on Drupal 4.7

44. Search API
› Abstraction of SolR
› Configuration
› Hard to customize

45. Search
› Fast
› Text analysis
› Facets
Exemples : Wikipedia, Amazon, Github, ...

47. › Open source project
› Developer-Friendly RESTful API

48. How to use :
- Get it
- Unzip it
- Run it

49. › 1 exemple : Save requests
-> Percolator
Index content VS search query

50. › How to start ?
2 ways :
➔ Combine with search API
➔ Custom connectors with . . .
◆ Friendly API
◆ Custom code

51. https://www.drupal.org/project/elasticsearch_helper

52. Drupal
Bonnes pratiques

53. Configuration management
› Export/import configuration
› Configuration installer : installation profile, UI
› Override local configuration : settings.php
› Drush can ignore configurations about modules
› Drush_cmi_tools : use a YAML list of ignored
configurations
› Configuration split : blacklist, ignore set, entities
From pescetti, ademarco, bircher
https://goo.gl/MSjK3T

54. Configuration management
› Use Git as parallel development
› Careful about merge, workflow
› Safe : export, commit, merge, import, push
› Features for D8 :
› analyzes and automatically packages up site
configuration into a set of features
› Only for development purposes From pescetti, ademarco, bircher
https://goo.gl/MSjK3T

55. Configuration management
› Client messing with production configuration
› Lock configuration on production (settings.php)
› Export from prod and merge to dev
› Configuration split : choose what to keep
› Deploy content
› Use migrate on events
› Default_content module : any module
provides hal+json for entities
› Deploy module : stage and preview content
From pescetti, ademarco, bircher
https://goo.gl/MSjK3T

56. Configuration management : modules
› Config_installer : install site from config
› Config_readonly : lock conf via UI
› Config_update : diff between original & active
› Config_devel : helps developing config
› Config_split : import/export filtered config
› Config_tools : automatically commit config
changes to git
› Features : bundle config for re-use
› Config_sync : safely import config from
updated modules From pescetti, ademarco, bircher
https://goo.gl/MSjK3T

57. Drupal
Migration

58. Migrate with D8
1. Requirements
2. Anatomy of a migration
3. Migration Framework
4. Performance tips

59. 1. REQUIREMENTS

60. 1. Migrate is in core !
Tree new modules
› Migrate
Handle migration. Framework.
› Migrate Drupal
Contains migration from D6 & D7
› Migrate Drupal UI
The older migrate UI (new in 8.1.x)

61. 1. Migrate is in core !
How to execute migrations ?
› UI ?
› Drush command ?
Contrib to Migrate tools & Migrate Plus

62. 2. ANATOMY OF A MIGRATION

63. 2. Workflow of a migration
DestinationProcess
Process
Process
Source

64. DEFINITIONS
Yaml Files.
Custom files
2. In files
PLUGINS
PHP Files
Core or custom files
Types:
› Source
› Process
› Destination
› Builder
› ID Map

65. An easy example

66. DEFINITION
config/install/migrate_plus.migration.article_node.yml

67. PLUGINS
src/Plugin/migrate/source/ArticleNode.php

68. EXECUTION
Only with Drush8 (8.1-dev) and migrate_tools enabled

69. 3. PLUGINS

70. 3. Source
Here we tell SqlBase:
- Which Database is the Source

71. 3. Source
- And which Plugin will make the Query

72. 3. Destination
How and where to store the data
- entity:<place-here-an-entity>
Need more destination plugins?
⇒ Search for “destination:” in core

73. 3. ID MAPPING
How Migrate associates old rows with new rows

74. 3. PROCESS
How we transform each field, each file or data.
You are able to:
› Map fields: Same value as origin.
› Modify: Change or process the value.
› Add: Create new fields from other fields or calculate
these fields.

75. 3. PROCESS
Map fields. Values are equal in both sides

76. 3. PROCESS
DefaultValue. Add a default value

77. 3. PROCESS
Callable. Values are related but a process is
needed with a function

78. 3. PERFORMANCE TIPS

79. 4. Performance tips
1. Hardware improvements (ssd, cpu,..)
2. Disabling functionalities (hook, modules)
3. Incremental migration (hash)
4. Multithreaded migrations

80. 4. Performance tips
$ drush mi cm_articles_1_400 &&
drush mi cm_articles_401_800 &&

81. D8 : Bonnes pratiques en vrac
› Drush is 10x faster than composer
› Librairie “composer-patches” : appliquer des
patchs via composer
› Pre-commit hook : PHP Lint check
› PHPCBF : PHP Code Beautifier & Formatter
› Kint : debug in twig
› Smacks: organize CSS & JS
› Templating : set var in twig (no preprocess)
Bonus D7 : drush unsuck

82. MERCI !
Questions ?