Recommended
Recommended
More Related Content
What's hot
What's hot (14)
Similar to Craft CMS: Beyond the Small Business; Advanced tools and configurations
Similar to Craft CMS: Beyond the Small Business; Advanced tools and configurations (20)
Recently uploaded
Recently uploaded (20)
Craft CMS: Beyond the Small Business; Advanced tools and configurations
- 1. Beyond the Small Business; Advanced tools and configurations Nate Iler Flipbox Digital
- 3. ● We owe it to our clients ● Protect data in transit and at rest ○ SSL ○ Encryption ● Managing access/policies ● Omit sensitive data in plain text Security
- 4. ● Automate scaling ○ Up for high demand ○ Down for low demand ● Automate recovery ● Loosely coupling resources ● Monitoring Reliability
- 5. ● Experiment ● Decouple with growth ○ Database ○ App ● Appropriate Type / Size ● Tune Performance
- 6. ● Evaluate providers ● Leverage efficient services ● Automation ○ Last project as starting point ○ Autoscale ● Build -vs- Buy Cost
- 7. Infrastructures of Today ● Expensive ● Physical Hardware ● Maintenance Staff ● Single point of failure Datacenter On Premise Cloud / Iaas Infrastructure as Code
- 8. Traditional Development Flow local dev development staging UAT / ... production
- 9. Infrastructure as Code Flow local dev development / staging / production
- 10. Evolution of Local Development Cowboy Native / MAMP Virtualization Containers
- 11. ● Consistency ● Disposable ● Configurable ● Lightweight docker-compose.yml Developing (locally) with Docker
- 12. Evolution of your Infrastructure
- 13. only for you
- 15. 1,000s to less than 10,000 users
- 16. 10,000s to 500,000k users
- 17. > 500,000 users
- 18. Auto-scaling
- 19. Impact on Craft SessionsLogging Assets Queue Cache
- 20. Centralized Logging Email / SMS Analysis Log Services
- 21. Assets Amazon S3, Google Cloud Storage, Rackspace Cloud Files, DigitalOcean Spaces, fortrabbit Object Storage Storage Volume Providers <?php $bucketBaseName = 'assets.bucket'; $baseS3Url = 'https://s3.amazonaws.com/'; $baseCDNUrl = 'https://cdn.your-site.com/'; $cloudFrontId = 'XXXXXXX'; return [ '*' => [ 'gallery' => [ 'bucket' => 'test.' . $bucketBaseName, 'hasUrls' => true, 'url' => $baseS3Url . 'test.' . $bucketBaseName . '/', 'subfolder' => 'gallery', 'region' => 'us-east-1', 'expires' => '1 months' ] ], 'production' => [ 'gallery' => [ 'bucket' => $bucketBaseName, 'url' => $baseCDNUrl . '/', 'cfDistributionId' => $cloudFrontId ] ] ]; config/volumes.php
- 22. Session Management config/app.php <?php return [ 'components' => [ 'redis' => [ 'class' => yiiredisConnection::class, 'hostname' => getenv('REDIS_HOSTNAME') ], 'session' => [ 'class' => flipboxcraftsessionRedis::class, 'keyPrefix' => 'session_' ] ] ]; composer require flipboxfactory/craft-session-redis 2) 1)
- 23. Cache config/app.php <?php return [ 'components' => [ 'redis' => [ 'class' => yiiredisConnection::class, 'hostname' => getenv('REDIS_HOSTNAME') ], 'cache' => [ 'class' => yiiredisCache::class ] ] ]; ApcCache, ArrayCache, DbCache, DummyCache, FileCache, MemCache, WinCache, XCache, ZendDataCache Native Cache Providers
- 24. Queues File, Db, Redis, RabbitMQ, AMQP Interop, Beanstalk, Gearman, AWS SQS config/app.php Native Queue Providers <?php return [ 'components' => [ 'queue' => [ 'class' => yiiqueuesqsQueue::class, 'url' => getenv('SQS_URL'), 'region' => getenv('SQS_REGION') ] ] ];
- 27. Pipeline: Project PASS FAIL artifact Only perform deployment from master/develop branches git push via webhook deployment provider
- 28. Pipeline: Plugin PASS FAIL /docs Only deploy docs from master branch git push via webhook
- 30. 4) 3) 2) 1) Install Docker: https://docs.docker.com/install/ composer create-project flipboxlabs/craft <SOME_DIR> make craft-setup make test Example Project Testing starts container, begins craft setup run all tests (via codeception) create new project (forked craftcms/craft)
- 31. Thank You! We know; this topic is vast. Join us as we take a closer look at the topics covered in this presentation. (click to view this series) DevOps with Craft CMS
- 32. Resources ● 1Password ● LastPass ● Let’s Encrypt ● AWS Certificate Manager ● OpenVPN ● AWS Auto Scaling ● AWS Auto Recovery ● Service-oriented architecture
- 33. Resources ● Terraform (IaC) ● CloudFormation (IaC) ● Docker ● Docker for Craft (our starter project) ● Scaling Up to Your First 10 Million Users ● Shootout at the Local Dev Corral ● Containers on AWS ● AWS Well-Architected Framework
- 34. Resources ● Codeception Testing ● AWS Instance Types ● Splunk ● Papertrail ● Elastic (Beats, Logstash / Kibana / Elasticsearch) ● Travis CI ● Scrutinizer CI ● Jenkins
Editor's Notes
- About Me Flipbox Craft Experience (5+ years) Work with associations and membership based sites. 100k+ users / organizations Complex requirements Craft is one facet to a larger digital solution We’ve learned a lot of our clients, and I’d like to share some of that with you today. Geared towards: Teams Supporting a lot of moving parts Highly collaborative DevOps Already tech oriented Interested and want to learn more I’ll start: Infrastructure Planning Move to: Local development with containers Finish up with: CI/CD Let’s begin Analyzing the infrastructure
- What is Infrastructure? The infrastructure is everything that runs your software (Craft) App service Database server CDN I’m not able to dictate infrastructure requirements for my projects? Maybe not yet; but with a deeper understanding and a little practice you may be able to offer it as a service. You don’t want to be a system admin and manage servers? Me neither. I’ll walk though some strategies around Infrastructure as Code and automation that reduces this burden. Disclaimer You may see Amazon Web Services references (security, robust services, price) Provider agnostic Choose whatever you’re comfortable with Remember the overall principles
- Security easily overlooked an area to never compromise We owe it to our clients Clients may not prioritize security, but we should This includes Passwords Database backups Reports Excel docs What to do Use password management tool 1Password LastPass Protect data in transit and at rest All sites should be using SSL Let’s Encrypt issues free SSL Certs AWS offers free Certificate Manager Encrypt data in storage (where possible) Includes Craft Asset Volumes (certain Providers) AWS S3 Managing access/policies VPN Only allow access to resources through the VPN Non-production environments App / Database / Cache services We use OpenVPN And enforce MFA (multi-factor authentication) If we’re not on the VPN, we can’t access resources. Limit the use of API keys and tokens (if possibles) Difficult to track usage in the wild Instead, use resource roles Ex: We don’t use AWS keys Craft Asset Volumes Leverage a centralized access management tool (IAM) if possible One place to audit and control access When an team member leaves, all access is revoked Omit sensitive data in plain text In your Craft project repository remove API Keys Credentials Certificates Instead, access sensitive data through: Environmental variables A key management service
- Reliability How to handle reliability: Disruptions Failures Demand / Load changes Automate scaling High demand -> scale up Low demand -> scale down Do this without: Over provisioning resources Under provisioning resources NOTE: Many providers offer these automations to you: Ex: AWS you adjust autoscaling thresholds Automate recovery Detecting when Craft is not responding: Done through health checks (200 statuses) Perform backups Databases Applications Operating environments For the advanced: failover to geographic: availability zone regions NOTE: Many providers offer these automations to you: Ex: AWS you can indicate db backup retention Ex: AWS failed health check will auto replace instance Loosely coupling resources As Craft / data / load grows, so should the infrastructure (cover this a bit later) Aim for single responsibility resources: Ex: hey database server, store my data Could be MySQL, Postgres Monitoring Tell me when something isn’t right Slack Email SNS Automated recovery may still be happening behind the scenes. Ex: We have a Slack room for DevOps and CloudWatch notifications appear there
- Performance The ability to efficiently use resources. Experiment With pay-as-you-go, resources are cheap Benchmark / Test Simulate various loads. Learn from tests and mistakes Decouple with growth As Craft data grows use dedicated services: Results in: running smaller resources running an optimized resource; Ex Database resource Greater performance / same cost Resource Type / Sizes Not all compute instances are build the same. Ex: AWS has over 15 types of computing instances: Optimized for: Memory Accelerated Computing Storage Dense Storage (D2) High-Frequency (T2) Compute sizes also vary: 1 vCPU / .5 Gb Mem 96 vCPU / 384 Gb Mem Tune Review metrics: Provider dashboards Services like New Relic Optimize software Web Server: Nginix PHP: v7+ Craft: attend Andrews presentation (at the end of the day)
- Cost Deliver the best technical solution at the lowest price point with the least amount of effort. Evaluate Providers Such as: Google Cloud Azure Digital Ocean Heroku AWS Each have a variety of offerings Choose the best that fits Leverage efficient services Choose the most optimized for the task at hand (we talked about this prior) Only use services and features that are needed Ex: A database read replica isn’t needed on low demand apps A multi-region solution may not be needed for an intranet (corporate use only) Automate Use the last project as a starting point especially with IaC (which we’ll touch on later) Enable autoscaling Enables running the least amount of resources per the demand Build -vs- Buy How much does the SaaS offering cost: Laravel Forge ($12 / $19 / $39) DeployBot ($15 / $25 / $50) Buddy ($49/ $98) Nanobox Does it satisfy your requirements (and next 6 months)
- Infrastructures of Today Looking back: Servers were in: closets under desks in data centers Resulting in: dedicated physical hardware that’s expensive to operate has single points of failure needs staff to maintain Talk about expensive... More recently: We pushed computing to the Cloud: no longer think about hardware have endless: computing power service offerings billed on a per-usage basis Can it get any better? Perhaps. Today, Infrastructure as Code is talking hold: We get all of the benefits of the Cloud, plus: The Infrastructure is defined in configuration files: Lives next to your project code (git repo) It’s great for configuration management You can easily trace errors Revert bad configuration (to a known good state)
- Traditional Development Flow A team member: commits a new feature the code looks great they want to show the client Three environments exist: Development internal team Staging / UAT acceptance testing Production live Keep in mind: These environments are: always running pre-configured (which may have been a long time ago) slightly different have different / stale data At the end of the day, it’s difficult to: have consistency across environments perform infrastructure updates pay for resources that are rarely utilized
- Infrastructure as Code Flow A team member: commits a new feature the code looks great they want to show the client Leveraging IaC: The notation of dedicated ‘development’ / ‘staging’ environments no longer exist instead, an infrastructure is: built on-demand based on a IaC file (in the repo) close to production as possible disposable / promotable
- Evolution of local development Why do we develop locally? It’s faster It’s isolated Able to work without internet What are some challenges? How close can we mirror production? We want to remove: unexpected errors / regressions ‘it worked on my machine’ Cowboy - Coding directly on a server Pros None. Don’t do it. Native / MAMP - Web server / PHP / Database running directly on your machine Pros Able to leverage modern application development patterns Cons Difficult to mirror production environment Virtualization (Vagrant, Homestead) - Web server / PHP / Database running virtually Pros Disposable Configurable Consistent (for teams) Cons Heavy (resource / storage) Not portable Difficult to manage pre project Containers - Let’s take a deer look at containers
- Developing (locally) with Docker Docker is a container platform It’s growing in popularity and support Here are some reasons why containers are great for your Craft project Consistency Across teams Each user runs the same software. Across environments Closely mirror production resources Disposable Test various versions / upgrades Run scripts / ETL / Reports / One offs Configurable Built specifically for a project’s needs Lightweight intended to be small only contain the dependencies to perform the single task However, while working with Docker we’ve noticed: Not native to MacOS and Windows High Sierra is slow Learning curve (like anything new) Review code example (screenshot)
- Evolution of your Infrastructure As your Craft project grows so should your infrastructure. Throughout these diagrams, you’ll see: Loose coupling of resources Single responsibility of resources
- only for you Single Server all resources are shared.
- under 1000 users Web Server DB Service
- 1,000s to less than 10,000 users We added: Load Balancer Distributing load across multiple web servers Auto-Scaling Web servers High availability Greater scalability Redis Handle Session Handle Cache
- 10,000s to 500,000k users We added: Database Read Replica Taking load off master CDN Assets (css, js, uploads) Static Html Pages
- > 500k users We added: Queue External service Worker Tier Workers dedicated to processing queue
- Let’s take a look at the scalable web tier Here’s what’s happening under the hood: Instances rotate randomly A load balancer distributes to one instance
- How does this scalable/disposable infrastructures impact Craft We’ve identified the following areas that are impacted: Logging Session Management Assets (file uploads) Queue Cache
- Centralized Logging Remember: We may have multiple web servers running. By default, the logs are stored in `storage/logs` directory An unexpected event occurs. One of the first thing you probably do is go to the logs. However, logs are: Spread across multiple server Therefore you need to review logs independently (stitching them together) Gone. The server has rotated out and the logs not available Centralized logging remedy this problem. Logs are pushed to a dedicated ‘centralized’ service. You can view all of your logs in one location. Search tools Notifications Example- Our setup: Each server is configured to push logs to CloudWatch. Craft logs are also pushed to Cloudwatch (via File Beat). We search the logs via Elasticsearch / Kibana CloudWatch notifications are triggered when escalated actions are identified
- Assets Again: We may have multiple web servers running. Uploading assets to a ‘local’ volume will not work. You can: Mount a shared volume (nfs, etc) across all instances Slow Difficult to configure / manage Use an external storage provider and volume. User uploads are in a central location Image transforms are performed and uploaded Looking at the volumes config: We’re using AWS S3 as the volume / provider On production, we’re pointing to a CDN
- Session Management By default, PHP stores sessions in the file system. As a result: it would appear as you’re logged in on some requests and out on others To address this: Sessions should be managed through an external service. I recommend using Redis and Craft can handle this in two steps Require a Redis session package for Craft. Alter your app.php config file to utilize Redis.
- Cache Craft uses File system cache by default. As a result: Cache items would be specific to each server This isn’t ideal as it’s not as: It’s not as performant Breaking cache would only occur on one server To address this: Cache should be managed through an external service. Similar to sessions, I recommend using Redis: Alter your app.php config file to utilize Redis
- Queueing Craft manage queue items via the database. In scenarios where the queue is heavily utilized, using an external service might be preferred. In this case, I’ve altered app.php config file to utilize AWS SQS service. It’s important to note, rarely do you need to define alternate caching services, but it’s useful when: Adding a lot of items to the queue Intensive operations are needed
- I just covered the tools, services and principles behind running our code. But let’s switch gears now and look at how we work with the code itself. More specifically: How to get deploy code From our machines to production How to increase your confidence that something isn’t going to break. To do this, we’ll look at CI / CD.
- Continuous Integration CI is essentially an automation step to your code Great for teams Suggests small, incremental changes (over massive refactors) Allows developers to address issues sooner. Is required for CD. What’s CD? Continuous Delivery CD is essentially the process of getting the code (from CI) where it needs to go: Production UAT It supports releasing updates more frequently (rather than slow/large) updates. There’s an automation alternative to Continuous Delivery call Continuous Deployment. Continuous Deployment CD automaters the delivery. We’ll look at a couple example: Project Plugin
- Project Pipeline Begins with ‘git push’ All branches All users A hook/event notifies CI service CI Services is responsible for: Build Tests Standards Analysis (quality) The CI service will result in a pass/fail (go / stop) BEGIN CD (only on ‘master’ (or pre-defined) branches) An artifact (via build) is produced from CI. The artifact can be manually or automatically deployed.
- Plugin Pipeline An example with the plugins we write. Begins with ‘git push’ All branches All users Travis CI + Scrutinizer CI Tests Code Quality Standards On ‘master’ branch our docs (VuePress) are deployed to GitHub pages.
- I mentioned Tests in the CI process. Why test? A lot of moving features/functionality (adding enhancements disrupts this) Projects: Business critical functions (accepting payments, reporting, etc) Plugins: You’re not the only one using it Working w/ teams (we aren’t writing 100% of the code) Distribute workload across team Solo projects picking back up months after Increase confidence in the code We’re not breaking critical site aspects Hurdles Where to begin? Lowest hanging fruit Critical features/business functions Allocating time (I can knock this out in 10 minutes … nope) It takes time (and maintenance)
- How do you get started testing your projects? We forked the craft project repo and added some Codeception tests to get started. What we added: Docker containers (simple / straight forward) Unit / Functional / Acceptance tests (basic) Install Docker Docker is available for a wide range of operating systems. Create Craft Project The standard composer command; however we’re pointing to a forked repository of our own The ‘make’ commands are intended to simplify common operations. You can look at the entire command via the `Makefile` … You’ll notice we’re simply executing the php commands within the docker container `Make` is required and may need to be installed Setup Craft Starts Docker container Runs Craft Setup Run tests Starts Docker container Executes Codeception tests
- Our journey is over. We briefly touched on: Infrastructure principles (including Iac) Containers CI/CD There’s a whole lot more to dive into. We’re working on a multi-series post on our web site to go over much of this presentation in greater detail. Hopefully you’re able to look at Craft through a different lens and approach your next big project with confidence.