[BarCamp2018][20180915][Tips for Virtual Hosting on Kubernetes]

1. BarCamp Hong Kong 2018
Tips for Virtual Hosting on Kubernetes
PantaRei Design Limited
Wong Hoi Sing, Edison
2018-09-15

2. Wong Hoi Sing, Edison
●
2005 - Drupal Developer & Contributor
– https://drupal.org/user/33940
●
2008 - HKDUG Co-founder
– https://groups.drupal.org/drupalhk
●
2010 - CEO, PantaRei Design
– hswong3i@pantarei-design.com

7. PantaRei Design
●
Everything Changes and Nothing Remains Still
●
Reinvent Enterprise with Open Source Software and Cloud Computing
●
Hong Kong based FOSS service provider
– Content Management System (CMS) with Drupal
– Cloud Hosting Solution with Amazon Web Services (AWS)
– Team collaborate solution with Atlassian
●
Business Partner with industry leaders
– 2012, AWS Consulting Partner
– 2013, Acquia Partner
– 2013, Atlassian Experts
– 2014, Rackspace Hosting Partner
●
http://pantarei-design.com

9. Outline
●
Virtual Hosting for Dummy
●
Let’s Demo with Drupal 8.6
●
Tips & Tricks
●
Roadmap
●
Q&A

10. Virtual Hosting for Dummy
●
What is Virtual Hosting?
●
How Virtual Hosting Works?
●
How to Improve it?
●
Overall Design

11. What is Virtual Hosting?
●
Virtual hosting is a method for
hosting multiple domain names
(with separate handling of each
name) on a single server (or pool
of servers)

12. How Virtual Hosting Works?
●
Single Server, Single Cluster, Single Domain
– tasksel install lamp-server
– No redundant, no scale out, no per talent security
required (well…)
– From KISS principle (Keep it simple, stupid) point of
view, it is good enough that not easy to fail
– BTW, from hosting service vendor point of view,
resource could not be fully utilized (or, overselling...)

14. How Virtual Hosting Works?
(cont.)
●
Single Server, Single Cluster, Multiple Domain
– Cpanel, Plesk, Virtualmin, etc
– No redundant, no scale out
– Shared resource, e.g. disk, database, service
– Resouce could now be utilize and overselling, but
security between each talent MUST BE very careful

19. How Virtual Hosting Works?
(cont.)
●
Multiple Server, Single Cluster, Multiple Domain
– Manage with DevOps, e.g. Chef, Puppet, Ansible, etc
– With redundant
– Difficult for implement
●
Custom deployment playbook required
– Difficult for scale out
●
Manually update the playbook
●
Deploy native package to node per OS
– Shared resource, e.g. disk, database, service
– Security between each talent MUST BE very careful

24. How Virtual Hosting Works?
(cont.)
●
Multiple Server, Multiple Cluster, Single
Domain
– With redundant
– With scale out
– As simple security management as All-in-one
– But you will need many server (!?)
– Deployment and management is difficult, too (!!)

28. How to Improve it?
●
Simple architecture deployment and management
●
Simple application installation and configuration
●
Simple centralized network storage cluster with
backup plan
●
Simple service fail detection and rescheduling
●
Simple per talent isolation and security
●
(NEW) Hot/Warm/Cold disaster recovery plan

29. Overall Design
●
Ansible: deployment management
●
Docker: application pre-packaging
●
CephFS: centralized network storage
●
Kubernetes: runtime service lifecycle
management
●
(NEW) Duplicity: backup with deduplication

33. Overall Design (cont.)
●
Each talent (i.e. domain) goes into individual
Kubernetes namespace
●
Each namespace have its own Apache, PHP, MySQL,
SSHD, etc
●
All dynamic data store outside Kubernetes, goes
into CephFS
●
(NEW) DR with CephFS snapshot + Rsync + Duplicity

34. Let’s Demo with Drupal 8.6
●
Create Kubernetes Namespace
●
Deploy Application Stack
●
Install Drupal 8.6
●
Scale It Out!

35. Create Kubernetes Namespace
●
kubectl apply -Rf 00-namespace.yml
●
kubectl get namespace
| grep hswong3i-net

37. Deploy Application Stack
●
kubectl apply -Rf .
●
kubectl -n hswong3i-net
get pod -o wide

39. Install Drupal 8.6
●
kubectl -n hswong3i-net
exec -ti sshd-66cbd7dc65-pk9v2 /bin/bash
●
cd /var/www/html/web
●
echo "<?php phpinfo(); ?>"
> phpinfo.php
●
https://hswong3i.net/phpinfo.php

42. Install Drupal 8.6 (cont.)
●
cd /var/www/html
●
wget -c
https://ftp.drupal.org/files/projects/drupal-8.6.1.tar.gz
●
tar zxf drupal-8.6.1.tar.gz
--strip-components=1
-C /var/www/html/web/
●
chown -Rf www-data:www-data /var/www/html
●
https://hswong3i.net/core/install.php

51. Scale It Out!
●
kubectl -n hswong3i-net patch deploy httpd
-p '{"spec":{"replicas":4}}'
●
kubectl -n hswong3i-net patch deploy php
-p '{"spec":{"replicas":4}}'
●
kubectl -n hswong3i-net patch sts mariadb
-p '{"spec":{"replicas":3}}'

53. Tips & Tricks
●
Ansible
●
Ceph
●
Kubernetes
●
Application Stack

54. Ansible
●
Installing Ansible is simple
– pip install ansible
●
Managing access to remote guest is simple
– Password-less SSH login by public-private key pair
●
Running command on remote guest is simple
– ansible -i guest1,guest2, -m ping
– ansible -i guest1,guest2, -m apt -a ‘name=vim state=present’
– ansible -i guest1,guest2, -m shell -a ‘uname -a’

55. Ansible (cont.)
●
Made complicated application
deployment manageable
– ceph-ansible
– kubespray
– openstack-ansible

56. Ceph
●
DON’T deploy Ceph with Kubernetes + Helm!!
– By our design Kubernetes consume Ceph as
Persistent Volumes (PV) provider
– Deploying Ceph on top of Kubernetes become a “
Chicken or the Egg” paradox
●
TRUST ME!! You will absolutely get mad when your Ceph
crashed and recovery is required :-P
– Simply deploy with ceph-ansible

57. Ceph (cont.)
●
Collocated + Bluestore
– Collocated simplify your disk partition
table
– Bluestore scale much better than
traditional XFS-based filestore
●
E.g. Backed with Aliyun EBS (max ~110MB/s), a
3 OSD CephFS could provide ~60MB/s

59. Ceph (cont.)
●
CephFS Snapshot + Duplicity
– Create a CephFS Snapshot is simple and fast (>1s)
●
cd /mnt/cephfs && mkdir -p .snap/`date +%s`
– Snapshot therefore could async backup by Duplicity as simple flattened .gz format, with
Data Deduplication, e.g.
●
Weekly full backup
●
Daily incremental backup
●
Keep maximum 4 weeks backup
– (NEW) Don’t keep snapshot after rsync
●
Multiple-Master-MDS + snapshot just get stable on Ceph 13.2.0
●
Snapshot will use up your usable space
– (NEW) Both rsync and duplcity use `mtime + size`
●
Don’t use `--size-only`: backup will ALWAYS FAIL
●
(Optional) Use `--checksum` on rsync will slow down for at least x4 times, but duplicity hard code logic with
`mtime + size`...

62. Kubernetes
●
PVC with cephfs-provisioner (or ceph-csi)
– With cephfs-provisioner we could simply integrate with Kubernets
PersistentVolumeClaims
– kubespray ready for this deployment
– Everything from Application Stack could go into CephFS, with
ReadWriteMany, e.g.
●
/root (for sshd pod, for remote management)
●
/var/www/html (for httpd/php pods, for document root)
●
/var/lib/mysql (for mysql/mariadb pods, for data store)
– Backup made simple with CephFS Snapshot + Duplicity

63. Kubernetes (cont.)
●
Network Policy with Weave
– Weave is a simple and lightweight vxlan-based CNI Network
Provider, supporting Kubernete’s Network Policy
– kubespray ready for this deployment
– We could isolate network traffic between each namespace, e.g.
●
DENY all traffic from other namespaces (i.e. default allow traffic inside
namespace)
●
ALLOW all traffic from a namespace (e.g. allow traffic from ingress-nginx
namespace)
●
ALLOW all traffic to an application (e.g. allow external access to sshd pod for
remote management)

65. Kubernetes (cont.)
●
Ingress with ingress-nginx
– ingress-nginx use Nginx as Controller for Kubernetes
Ingress Resource, supporting HTTPS /TLS with SNI (i.e.
multiple HTTPS virtual hosting with single public IP)
– kubespray ready for this deployment
– Proxy Protocol support is required when working with
AWS L4 ELB, for fetching client’s source IP address

66. Kubernetes (cont.)
●
Let’s Encrypt with cert-manager
– With cert-manager we could request TLS
certificate from Let’s Encrypt for free, with
auto renewal and ingress-nginx integration
– kubespray ready for this deployment
– Simply use http01 for varification

67. Kubernetes (cont.)
●
(NEW) StatefulSet Limitation
– Deleting and/or scaling a StatefulSet down will not
delete the volumes associated with the
StatefulSet.
– This is done to ensure data safety, which is
generally more valuable than an automatic purge
of all related StatefulSet resources
– https://kubernetes.io/docs/concepts/workloads/co
ntrollers/statefulset/#limitations

68. Kubernetes (cont.)
●
(NEW) Master-Slave Replication?
– NO, do Single-Master or Multi-
Master with load-balancing
– Simply use Deployment for Single-
Master pod automatically failover by
Kubernetes

69. Application Stack
●
MariaDB Multi-master cluster with Kubernetes peer-
finder
– When starting a new MariaDB multi-master cluster, we
need to know if “this” node should bootstrap a new
cluster, or join existing cluster
– Kubernetes peer-finder could give a hand for discovering
if any existing peers
– We build a Docker image which simplify such integration
with StatefulSet deployment

70. Application Stack (cont.)
●
MariaDB innodb_buffer_pool_size
– Simply give large enough
innodb_buffer_pool_size, so let MariaDB try
to cache all warmup-ed data from disk to
memory/swap, and async write-back to disk
– This could greatly reduce the disk I/O
bottleneck

71. Application Stack (cont.)
●
PHP cache with opcache + apcu
– With opcache enabled PHP will cache the
precomplied script bytecode in shared memory,
which greatly improve the overall performance
– With apcu enable legacy application could also
benefit with their existing cache
implementation

72. Application Stack (cont.)
●
Drupal need a lots of PHP extensions
– Official PHP Docker image comes with
almost none of Drupal required extensions
– By the way, installing those extensions
manually is VERY TIME CONSUMING!!
– We build a Docker Image which simplify such
integration...

73. Application Stack (cont.)
●
Apache cache with mod_cache_disk
– With mod_cache_disk enable Apache will try to cache any
static assets (e.g. css/js/jpg/gif/png/etc) as much as
possible
– Don’t worry to much about the disk I/O performance,
Linux kernel will automatically cache it for you into
memory/swap, if required
– We build a Docker Image which simplify such integration

74. Roadmap
●
CephFS with Kubernetes CSI
●
Logging with Fluentd, ElasticSearch, and
Kibana
●
Monitoring and Alert with Prometheus and
Grafana
●
Manage Application Stack with Helm and
Chart

75. Q&A

76. Contact Us
●
Address: Unit 326, 3/F, Building 16W, No.16
Science Park West Avenue, Hong Kong Science
Park, Shatin, N.T.
●
Phone: +852 3576 3812
●
Fax: +852 3753 3663
●
Email: sales@pantarei-design.com
●
Web: http://pantarei-design.com