Disposable Environments at Scale        / or: How I Learned to Stop Worrying and Love ZFS
Eric Sproul   Build Engineer     Sysadmin    Consultant     Twitter:     @eirescot
Recipe for Success  enablement (n): the act of providing (someone)  with adequate power, means, opportunity or authority  ...
Background   Etsy is the worlds handmade marketplace             Experiencing rapid growth    Every department needed to u...
The Situation                  No data warehouse            Analytical queries against               PostgreSQL OLTP      ...
Problems           Long-running queries to OLTP             destroy web performance                Re-running reports     ...
Requirements               Relieve pressure from                  OLTP database               Make production data        ...
Solution           Create separate BI analytics DB                   Build it on ZFS
Initial Capabilities            ETL to collate table-level data              from multiple databases              Run deep...
Reaping the Benefits of ZFS                    Snapshots                  Faster backups              Simple replica creat...
Reaping the Benefits of ZFS                 Compression           Extend usable life of storage:                  PgSQL lo...
We Want More!       Monthly reports now on BI system         Occasional problems require             re-running reports   ...
We Want More!          Need to test report changes        Fine to dev with small mock-up          Staging requires somethi...
The Next Level          Disposable environments                  Run on slave replica                 R/W copy of BI data ...
Disposable Environment                  Use a non-global zone    set   zonepath=/zones/bistage    set   autoboot=true    s...
Disposable Environment            Starting state: ZFS datasets         bi01tank/pgsql/data         bi01tank/pgsql/data/91 ...
Disposable Environment                    Take snapshots         zfs snapshot -r bi01tank/pgsql/data@stage         bi01tan...
Disposable Environment                      Create clones         zfs clone <src_dataset>@stage <dst_dataset>         bi01...
Disposable Environment    Zone now sees a full, writable copy of data     Unchanged data is referenced to origin          ...
Disposable Environment   pgsql/data/91   pgsql/data/91@stage   stage/data/91                                              ...
Next-Level Results             Any report can be re-run                on the same data             Massage existing data ...
Next-Level Results      When finished with the environment:                 Shut down zone            Delete clone & origi...
Return on Investment         BI database runs on 2 machines        OLTP database lifetime extended          two years past...
Bonus!         With the same technique,            we can safely test:          PostgreSQL upgrades             Schema cha...
Thank You           ZFS, Zones and much more       are available to the community via          illumos and its distributio...
Upcoming SlideShare
Loading in …5
×

Disposable Environments at Scale

743 views

Published on

Presentation from ZFS Day, 2 October 2012 (http://zfsday.com)

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
743
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Disposable Environments at Scale

  1. 1. Disposable Environments at Scale / or: How I Learned to Stop Worrying and Love ZFS
  2. 2. Eric Sproul Build Engineer Sysadmin Consultant Twitter: @eirescot
  3. 3. Recipe for Success enablement (n): the act of providing (someone) with adequate power, means, opportunity or authority (to do something) This is a story of how ZFS enabled business success
  4. 4. Background Etsy is the worlds handmade marketplace Experiencing rapid growth Every department needed to understand how the business was evolving Asked OmniTI for help
  5. 5. The Situation No data warehouse Analytical queries against PostgreSQL OLTP Large initial size (~250 GB) Forecast to reach 1 TB+ within a year
  6. 6. Problems Long-running queries to OLTP destroy web performance Re-running reports produces different results Inflexible reporting interface
  7. 7. Requirements Relieve pressure from OLTP database Make production data continuously available Enable correlation of other sources Flexible reporting UI
  8. 8. Solution Create separate BI analytics DB Build it on ZFS
  9. 9. Initial Capabilities ETL to collate table-level data from multiple databases Run deep analytic queries without impacting website New web UI enables ad-hoc reporting
  10. 10. Reaping the Benefits of ZFS Snapshots Faster backups Simple replica creation
  11. 11. Reaping the Benefits of ZFS Compression Extend usable life of storage: PgSQL logical: 1.3T On-disk: 653G (2.0x) Intelligent resilver Shorter rebuilds == Reduced risk of data loss
  12. 12. We Want More! Monthly reports now on BI system Occasional problems require re-running reports Still get different results, same as before
  13. 13. We Want More! Need to test report changes Fine to dev with small mock-up Staging requires something that looks like production
  14. 14. The Next Level Disposable environments Run on slave replica R/W copy of BI data Discard when finished
  15. 15. Disposable Environment Use a non-global zone set zonepath=/zones/bistage set autoboot=true set limitpriv=default,dtrace_proc,dtrace_user set ip-type=shared add net set address=10.1.2.3 set physical=bnx0 end add dataset set name=bi01tank/stage end
  16. 16. Disposable Environment Starting state: ZFS datasets bi01tank/pgsql/data bi01tank/pgsql/data/91 bi01tank/pgsql/wal_archive bi01tank/pgsql/wal_archive/91
  17. 17. Disposable Environment Take snapshots zfs snapshot -r bi01tank/pgsql/data@stage bi01tank/pgsql/data@stage bi01tank/pgsql/data/91@stage bi01tank/pgsql/wal_archive@stage bi01tank/pgsql/wal_archive/91@stage
  18. 18. Disposable Environment Create clones zfs clone <src_dataset>@stage <dst_dataset> bi01tank/pgsql/data@stage bi01tank/pgsql/data/91@stage bi01tank/pgsql/wal_archive@stage bi01tank/pgsql/wal_archive/91@stage bi01tank/stage/data bi01tank/stage/data/91 bi01tank/stage/wal_archive bi01tank/stage/wal_archive/91
  19. 19. Disposable Environment Zone now sees a full, writable copy of data Unchanged data is referenced to origin Changes accumulate to clone
  20. 20. Disposable Environment pgsql/data/91 pgsql/data/91@stage stage/data/91 Unchanged data referenced from snapshot Change accounted to clone Live FS Snapshot Clone
  21. 21. Next-Level Results Any report can be re-run on the same data Massage existing data or bring more in for ad-hoc report Test changes to reports and web UI
  22. 22. Next-Level Results When finished with the environment: Shut down zone Delete clone & origin snap
  23. 23. Return on Investment BI database runs on 2 machines OLTP database lifetime extended two years past expectation Faster, more granular, and ad-hoc reporting enables better decisions by management
  24. 24. Bonus! With the same technique, we can safely test: PostgreSQL upgrades Schema changes
  25. 25. Thank You ZFS, Zones and much more are available to the community via illumos and its distributions Go forth and enable your business!

×