1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Mission:imPossibleScaling in theCloud1Saturday, Ma...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.The Core: Message Bus100% Cloud Native2Saturday, M...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What Cloud Translates To3Small to mid sizeinstance...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What Does Your Data Look Like?• Log Data Classic O...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Requirements5• No Data Loss• Ability to Pause, Pro...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Tools• Chef• Liquibase• HornetQ• Rep Mgr• Postgres...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Distributed OLTP7Disparate event types can be sent...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Vertical Sharding- Inheritance is your friend!8One...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Vertical Sharding- Inheritance is your friend!9Con...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Vertical Sharding- Inheritance is your friend!10WH...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Close The Shard11FOR tbl IN SELECT t.table_name FR...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Star & Snowflake OLAP12Because API calls can’t time...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Build Your Snowflake13• Pull Aggregates• Add/Update...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Build Your Snowflake14Driven by Sequential IDPositi...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Aggregate on Pull15CREATE TABLE source_epochAS SEL...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.The Defense16Bugs for the entire system will find t...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like17WriterWriterSaturday, May ...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like18WriterWriterOLTPTypeAOLTPT...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like19WriterWriterOLTPTypeAOLTPT...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like20WriterWriterOLAPWHType1Que...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like21WriterWriterOLAPWHType1Que...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like22WriterWriterOLAPWHType1Que...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.http://news.messagebus.com/2013/01/03/mission-impo...
1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Thank You!24Saturday, May 11, 13
Upcoming SlideShare
Loading in …5
×

Mission impossible

1,014 views

Published on

Scaling postgres in the cloud.

Published in: Spiritual, News & Politics
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,014
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
22
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Mission impossible

  1. 1. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Mission:imPossibleScaling in theCloud1Saturday, May 11, 13
  2. 2. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.The Core: Message Bus100% Cloud Native2Saturday, May 11, 13
  3. 3. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What Cloud Translates To3Small to mid sizeinstancesLack of Control OverPhysical ResourcesNo delay inexpandingLimited StorageOptionsRedundancy is EasyRedundancy isNecessarySaturday, May 11, 13
  4. 4. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What Does Your Data Look Like?• Log Data Classic OLTP• By Nature, it’s Distributed• Long & Mid Term Storage• Eventual v. Immediate Consistency• Speed of Retrieval -Classing OLAP Warehouse4Saturday, May 11, 13
  5. 5. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Requirements5• No Data Loss• Ability to Pause, Promote, and Update Seamlessly• Do Not be the Bottleneck• Allow for Out of Sequence Events• Restate and Correct AnalyticsSaturday, May 11, 13
  6. 6. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Tools• Chef• Liquibase• HornetQ• Rep Mgr• Postgres6Saturday, May 11, 13
  7. 7. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Distributed OLTP7Disparate event types can be sent to separateinstances.Within the same event, parallel writes don’t requiresynchronous multi-masters.Batch Process Writes with Copy IntoSaturday, May 11, 13
  8. 8. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Vertical Sharding- Inheritance is your friend!8One parent per event, sharded by time, allows for efficientquerying, individual backups, and general sanity.Saturday, May 11, 13
  9. 9. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Vertical Sharding- Inheritance is your friend!9Constraints: Timestamp not DateEvent ID (pkey)Explain Analyze!Saturday, May 11, 13
  10. 10. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Vertical Sharding- Inheritance is your friend!10WHILE startday <= endday endday LOOPtomorrow := startday + interval 1 day;index_name := idx_name || replace(startday::text, -, );tname := event_ || to_char(startday, YYYY_MM_DD);EXECUTE create table || quote_ident(tname) ||() inherits(event);EXECUTE ALTER TABLE || quote_ident(tname) || ADD CONSTRAINTcreatedck CHECK (created >= || quote_literal(startday) ||AND created < || quote_literal(tomorrow) ||);EXECUTE CREATE INDEX || index_name || ON ||quote_ident(tname)||(act_id) ;startday := tomorrow;END LOOP;Saturday, May 11, 13
  11. 11. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Close The Shard11FOR tbl IN SELECT t.table_name FROM information_schema.tables tWHERE LEFT(t.table_name, 6 ) = event_AND t.table_name NOT IN (SELECT table_nameFROM information_schema.constraint_column_usageWHERE constraint_name = idcheck)AND REPLACE(REPLACE(t.table_name, event_, ), _, -)::DATE < NOW()::DATELOOPEXECUTE SELECT coalesce(min(event_id), 0), coalesce(max(event_id), 0) FROM ||quote_ident(tbl) ||;INTO smin, smax;EXECUTE ALTER TABLE || quote_ident(tbl) || ADD CONSTRAINT idcheck CHECK (event_idBETWEEN || smin || AND || smax || );END LOOP;Saturday, May 11, 13
  12. 12. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Star & Snowflake OLAP12Because API calls can’t time out.Partition-able on fact, account - what works for you.Saturday, May 11, 13
  13. 13. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Build Your Snowflake13• Pull Aggregates• Add/Update base fact• Cascade Updates Through DimensionsQuartz Triggered Calls Three Phase Process:Saturday, May 11, 13
  14. 14. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Build Your Snowflake14Driven by Sequential IDPosition Lookup, Pull, & Update in a SingleTransactionSave State for Performance AnalysisSaturday, May 11, 13
  15. 15. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Aggregate on Pull15CREATE TABLE source_epochAS SELECT * FROM dblink(dbname=src host=127.0.0.2 port=5432 user=rickastleypassword=never_gonna_give_you_up,SELECT some, thing, COUNT(id),DATE_TRUNC(hour, TO_TIMESTAMP(date)),MAX(id) AS idFROM eventsWHERE id > 50823461GROUP BY 1, 2 ORDER BY id)AS t(some TEXT, thing, INT event_count INT, hour TIMESTAMP, id INT)Saturday, May 11, 13
  16. 16. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.The Defense16Bugs for the entire system will find their way throughto your warehouse. Do not always trust your data.Saturday, May 11, 13
  17. 17. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like17WriterWriterSaturday, May 11, 13
  18. 18. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like18WriterWriterOLTPTypeAOLTPTypeBOLTPTypeBSaturday, May 11, 13
  19. 19. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like19WriterWriterOLTPTypeAOLTPTypeBOLTPTypeBOLAPWHType1Saturday, May 11, 13
  20. 20. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like20WriterWriterOLAPWHType1Query HeadOLTPTypeAOLTPTypeBOLTPTypeBSaturday, May 11, 13
  21. 21. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like21WriterWriterOLAPWHType1Query HeadOLTPTypeAOLTPTypeBOLTPTypeCOLTPTypeAOLTPTypeAOLTPTypeCOLTPTypeCSaturday, May 11, 13
  22. 22. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.What It All Looks Like22WriterWriterOLAPWHType1Query HeadOLTPTypeAOLTPTypeBOLTPTypeCOLTPTypeAOLTPTypeAOLTPTypeCOLTPTypeCOLAPWHType1Query HeadSaturday, May 11, 13
  23. 23. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.http://news.messagebus.com/2013/01/03/mission-impossible/Questions?23Saturday, May 11, 13
  24. 24. 1/24/13Cloud-Native: Flexible, Redundant, & Infinitely Scalable byNature.Thank You!24Saturday, May 11, 13

×