FirehoseEngineeringdesigninghigh-volumedata collectionsystems                     Josh Berkus                  HiLoad, Mos...
Firehose Database   Applications (FDA)(1) very high volume of data    input from many producers(2) continous processing of...
Mozilla Socorro
Upwind
Fraud Detection System
Firehose Challenges
1. Volume●   100s to 1000s facts/second●   GB/hour
1. Volume●   spikes in volume●   multiple uncoorindated sources
1. Volumevolume always grows over time
2. Constant flowsince data arrives 24/7 …while the user interface can bedown, data collection can neverbe down
2. Constant flow        ●   cant stop            receiving to            processETL     ●   data can            arrive out...
3. Database size●       terabytes to petabytes    ●   lots of hardware    ●   single-node DBMSes arent enough    ●   diffi...
3. Database size●       database growth    ●   size grows quickly    ●   need to expand storage    ●   estimate target dat...
3. Database size    “We will decide on a dataretention policy when we run out         of disk space.”          – every bus...
4.
many components = many failures
4. Component failure●       all components fail    ●   or need scheduled downtime    ●   including the network●       coll...
solving firehose problems
socorro project
http://crash-stats.mozilla.com
Mozilla Socorro                          webserverscollectors                           reports             processors
socorro data volume●       3000 crashes/minute    ●   avg. size 150K●       40TB accumulated raw data    ●   500GB accumul...
dealing with volume  load balancers   collectors
dealing with volumemonitor   processors
dealing with size             data            40TB       expandible   metadata   views   500GB   fixed size
dealing with       component failureLots of hardware ...  ●   30 Hbase       ●   3 ES servers      nodes          ●   6 co...
load balancing &  redundancy    load balancers   collectors
elastic connections●       components queue their data    ●   retain it if other nodes are down●       components resume w...
elastic connections                        crash mover           collectorreciever               local file queue
server management●       puppet    ●   controls configuration of all servers    ●   makes sure servers recover    ●   allo...
Upwind
Upwind ●   speed ●   wind speed ●   heat ●   vibration ●   noise ●   direction
Upwind 1. maximize    power    generation 2. make sure    turbine isnt    damaged
dealing with volumeeach turbine:            90 to 700 facts/secondwindmills per farm:      up to 100number of farms:      ...
dealing with volume   local     historian   analytic   reports   storage               database   local     historian   an...
dealing with volume   local     historian   analytic   storage               database                                    m...
multi-tenant partitioning●       partition the whole application    ●   each customer gets their own        toolchain●    ...
dealing with:        constant flow and size             minute       hours     days     months   yearshistorian           ...
time-based rollups●       continuously accumulate        levels of rollup    ●   each is based on the level below it    ● ...
time-based rollups●       allows:    ●   very rapid summary reports for        different windows    ●   retaining differen...
firehose tips
data collection must be:   ●   continuous   ●   parallel   ●   fault-tolerant
data processing must be:   ●   continuous   ●   parallel   ●   fault-tolerant
every component     must be able to fail●   including the network●   without too much data loss●   other components must  ...
5 tools to use1.   queueing software2.   buffering techniques3.   materialized views4.   configuration management5.   comp...
4 donts1.   use cutting-edge technology2.   use untested hardware3.   run components to capacity4.   do hot patching
firehose mastered?
Contact●       Josh Berkus: josh@pgexperts.com    ●   blog: blogs.ittoolbox.com/database/soup●       PostgreSQL: www.postg...
Firehose      Engineering       designing       high-volume       data collection       systems                           ...
Firehose Database              Applications (FDA)          (1) very high volume of data              input from many produ...
Mozilla Socorro                  3Mozillas Socorro system, which collects firefox & thunderbird crash data from around the...
Upwind                                             4Upwind, which collects a huge amount of metrics from large wind turbin...
Fraud Detection System                                               5And numerous other systems, such as a fraud detectio...
Firehose Challenges                                            6regardless of how these firehose systems are used,  they s...
1. Volume     ●   100s to 1000s facts/second     ●   GB/hour                             7The first and most obvious is vo...
1. Volume     ●    spikes in volume     ●    multiple uncoorindated sources        8You also have to plan for unpredicted ...
1. Volume        volume always grows over time                                               9… there is always, always, a...
2. Constant flow       since data arrives 24/7 …       while the user interface can be       down, data collection can nev...
2. Constant flow                          ●   cant stop                              receiving to                         ...
3. Database size        ●       terabytes to petabytes            ●   lots of hardware            ●   single-node DBMSes a...
3. Database size        ●       database growth            ●   size grows quickly            ●   need to expand storage   ...
3. Database size            “We will decide on a data        retention policy when we run out                 of disk spac...
4.                                                15problem #4 is illustrated by this architecture diagram of  the Socorro...
many components          = many failures                    16… every day of the week something is going to be crashing on...
4. Component failure       ●       all components fail           ●   or need scheduled downtime           ●   including th...
solving firehose problems              18So lets talk about how a couple of different teams solved some of these firehose ...
socorro project                                               19First were going to talk about Mozillas Socorro  system. I...
http://crash-stats.mozilla.com                                                20At the end of the processing we get pretty...
21Theres quite a rich set of data there collecting everything about every aspect of whats making Firefox and other Mozilla...
Mozilla Socorro                                         webservers               collectors                               ...
23a full architecture diagram would look like this one, but  even this doesnt show all components.
socorro data volume       ●       3000 crashes/minute           ●   avg. size 150K       ●       40TB accumulated raw data...
dealing with volume                                               25                load balancers   collectorshow does Mo...
dealing with volume                                               26              monitor     processors5. the raw crashes...
dealing with size                              data                             40TB                         expandible   ...
dealing with              component failure      Lots of hardware ...         ●   30 Hbase       ●   3 ES servers         ...
load balancing &                 redundancy                                                    29                      loa...
elastic connections       ●       components queue their data           ●   retain it if other nodes are down       ●     ...
elastic connections                                crash mover                   collector        reciever               l...
server management       ●       puppet           ●   controls configuration of all servers           ●   makes sure server...
Upwind                                               33lets move on to a different team and different methods  of facing f...
Upwind                          ●    speed                          ●    wind speed                          ●    heat    ...
Upwind                         1. maximize                            power                            generation         ...
dealing with volume       each turbine:                   90 to 700 facts/second       windmills per farm:      up to 100 ...
dealing with volume                  local     historian   analytic   reports                  storage               datab...
dealing with volume                 local     historian   analytic                 storage               database         ...
multi-tenant partitioning        ●       partition the whole application            ●   each customer gets their own      ...
dealing with:               constant flow and size                    minute       hours     days     months   years      ...
time-based rollups        ●       continuously accumulate                levels of rollup            ●   each is based on ...
time-based rollups       ●       allows:           ●   very rapid summary reports for               different windows     ...
firehose tips                                               43based on my experiences with several of these  firehose proj...
data collection must be:               ●   continuous               ●   parallel               ●   fault-tolerant         ...
data processing must be:              ●   continuous              ●   parallel              ●   fault-tolerant            ...
every component              must be able to fail        ●   including the network        ●   without too much data loss  ...
5 tools to use       1.   queueing software       2.   buffering techniques       3.   materialized views       4.   confi...
4 donts        1.   use cutting-edge technology        2.   use untested hardware        3.   run components to capacity  ...
firehose mastered?                                                49so now that we know how to do it, its time for you to ...
Contact       ●       Josh Berkus: josh@pgexperts.com           ●   blog: blogs.ittoolbox.com/database/soup       ●       ...
Проектирование крупномасштабных приложений сбора данных (Josh Berkus)
Проектирование крупномасштабных приложений сбора данных (Josh Berkus)
Upcoming SlideShare
Loading in...5
×

Проектирование крупномасштабных приложений сбора данных (Josh Berkus)

902
-1

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
902
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
24
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Проектирование крупномасштабных приложений сбора данных (Josh Berkus)

  1. 1. FirehoseEngineeringdesigninghigh-volumedata collectionsystems Josh Berkus HiLoad, Moscow October 2011
  2. 2. Firehose Database Applications (FDA)(1) very high volume of data input from many producers(2) continous processing of incoming data
  3. 3. Mozilla Socorro
  4. 4. Upwind
  5. 5. Fraud Detection System
  6. 6. Firehose Challenges
  7. 7. 1. Volume● 100s to 1000s facts/second● GB/hour
  8. 8. 1. Volume● spikes in volume● multiple uncoorindated sources
  9. 9. 1. Volumevolume always grows over time
  10. 10. 2. Constant flowsince data arrives 24/7 …while the user interface can bedown, data collection can neverbe down
  11. 11. 2. Constant flow ● cant stop receiving to processETL ● data can arrive out of order
  12. 12. 3. Database size● terabytes to petabytes ● lots of hardware ● single-node DBMSes arent enough ● difficult backups, redundancy, migration ● analytics are resource-consumptive
  13. 13. 3. Database size● database growth ● size grows quickly ● need to expand storage ● estimate target data size ● create data ageing policies
  14. 14. 3. Database size “We will decide on a dataretention policy when we run out of disk space.” – every business user everywhere
  15. 15. 4.
  16. 16. many components = many failures
  17. 17. 4. Component failure● all components fail ● or need scheduled downtime ● including the network● collection must continue● collection & processing must recover
  18. 18. solving firehose problems
  19. 19. socorro project
  20. 20. http://crash-stats.mozilla.com
  21. 21. Mozilla Socorro webserverscollectors reports processors
  22. 22. socorro data volume● 3000 crashes/minute ● avg. size 150K● 40TB accumulated raw data ● 500GB accumulated metadata / reports
  23. 23. dealing with volume load balancers collectors
  24. 24. dealing with volumemonitor processors
  25. 25. dealing with size data 40TB expandible metadata views 500GB fixed size
  26. 26. dealing with component failureLots of hardware ... ● 30 Hbase ● 3 ES servers nodes ● 6 collectors ● 2 PostgreSQL ● 12 processors servers ● 8 middleware & ● 6 load web servers balancers … lots of failures
  27. 27. load balancing & redundancy load balancers collectors
  28. 28. elastic connections● components queue their data ● retain it if other nodes are down● components resume work automatically ● when other nodes come back up
  29. 29. elastic connections crash mover collectorreciever local file queue
  30. 30. server management● puppet ● controls configuration of all servers ● makes sure servers recover ● allows rapid deployment of replacement nodes
  31. 31. Upwind
  32. 32. Upwind ● speed ● wind speed ● heat ● vibration ● noise ● direction
  33. 33. Upwind 1. maximize power generation 2. make sure turbine isnt damaged
  34. 34. dealing with volumeeach turbine: 90 to 700 facts/secondwindmills per farm: up to 100number of farms: 40+est. total: 300,000 facts/second (will grow)
  35. 35. dealing with volume local historian analytic reports storage database local historian analytic reports storage database
  36. 36. dealing with volume local historian analytic storage database master database local historian analytic storage database
  37. 37. multi-tenant partitioning● partition the whole application ● each customer gets their own toolchain● allows scaling with the number of customers ● lowers efficiency ● more efficient with virtualization
  38. 38. dealing with: constant flow and size minute hours days months yearshistorian buffer table table table table minute hours days months historian buffer table table table minute hours days historian buffer table table
  39. 39. time-based rollups● continuously accumulate levels of rollup ● each is based on the level below it ● data is always appended, never updated ● small windows == small resources
  40. 40. time-based rollups● allows: ● very rapid summary reports for different windows ● retaining different summaries for different levels of time ● batch/out-of-order processing ● summarization in parallel
  41. 41. firehose tips
  42. 42. data collection must be: ● continuous ● parallel ● fault-tolerant
  43. 43. data processing must be: ● continuous ● parallel ● fault-tolerant
  44. 44. every component must be able to fail● including the network● without too much data loss● other components must continue
  45. 45. 5 tools to use1. queueing software2. buffering techniques3. materialized views4. configuration management5. comprehensive monitoring
  46. 46. 4 donts1. use cutting-edge technology2. use untested hardware3. run components to capacity4. do hot patching
  47. 47. firehose mastered?
  48. 48. Contact● Josh Berkus: josh@pgexperts.com ● blog: blogs.ittoolbox.com/database/soup● PostgreSQL: www.postgresql.org ● pgexperts: www.pgexperts.com● Upcoming Events ● PostgreSQL Europe: http://2011.pgconf.eu/ ● PostgreSQL Italy: http://2011.pgday.it/ The text and diagrams in this talk is copyright 2011 Josh Berkus and is licensed under the creative commons attribution license. Title slide image is licensed from iStockPhoto and may not be reproduced or redistributed. Socorro images are copyright 2011 Mozilla Inc.
  49. 49. Firehose Engineering designing high-volume data collection systems Josh Berkus HiLoad, Moscow October 2011 1I created this talk because, for some reason, Ive been working on a bunch of the same kind of data systems lately.
  50. 50. Firehose Database Applications (FDA) (1) very high volume of data input from many producers (2) continous processing of incoming data 2I call these “Firehose Database Applications”. Theyre characterized by two factors, and have a lot of things in common regardless of which industry theyre used in.1. These applications are focused entirely on the collection of large amounts of data coming in from many high-volume, producers, and2. they process data into some more useful form, such as summaries, continuously 24 hours a day.Some examples ...
  51. 51. Mozilla Socorro 3Mozillas Socorro system, which collects firefox & thunderbird crash data from around the world.
  52. 52. Upwind 4Upwind, which collects a huge amount of metrics from large wind turbine farms in order to monitor and analyze them.
  53. 53. Fraud Detection System 5And numerous other systems, such as a fraud detection system which receives data from over 100 different transaction processing systems and analyzes it to look for patterns which would indicate fraud.
  54. 54. Firehose Challenges 6regardless of how these firehose systems are used, they share the same four challenges, all of which need to be overcome in some way to get the system running and keep it running.
  55. 55. 1. Volume ● 100s to 1000s facts/second ● GB/hour 7The first and most obvious is volume. A firehose system is collecting many, many facts per second, which may add up to gigabytes per hour.
  56. 56. 1. Volume ● spikes in volume ● multiple uncoorindated sources 8You also have to plan for unpredicted spikes in volume which may be 10X normal.Also your data sources may come in at different rates, and with interruptions or out of order data.But, the biggest challenge of volume always is ...
  57. 57. 1. Volume volume always grows over time 9… there is always, always, always more of it than there used to be.
  58. 58. 2. Constant flow since data arrives 24/7 … while the user interface can be down, data collection can never be down 10the second challenge is dealing with the all-day, all- week, all-year nature of the incoming data.for example, it is often permissiable for the user interface or reporting system be out of service, but data collection must go on at all times without interruption. Otherwise, data would be lost.this makes for different downtime plans than standard web applications, who can have “read only downtimes”. Firehose systems cant; they must arrange for alternate storage if the primary collections is down.
  59. 59. 2. Constant flow ● cant stop receiving to process ETL ● data can arrive out of order 11it also means that conventional Extract Transform Load (ETL) approaches to data warehousing dont work. You cant wait for the down period to process the data. Instead, data must be processed continuously while still collecting.Data is also never at rest; you must be prepared for interruptions and out-of-order data.
  60. 60. 3. Database size ● terabytes to petabytes ● lots of hardware ● single-node DBMSes arent enough ● difficult backups, redundancy, migration ● analytics are resource-consumptive 12another obvious challenge is the sheer size of the data.once you start accumulating data 24x7, you will rapidly find yourself dealing with terabytes or even petabytes of data. This requires you to deal with lots of hardware and large clustered systems or server farms, including clustered or distributed databases.this makes backups and redundancy either very difficult or very expensive.performing analytics on the data and reporting on it becomes very resource-intensive and slow at large data sizes.
  61. 61. 3. Database size ● database growth ● size grows quickly ● need to expand storage ● estimate target data size ● create data ageing policies 13the database is also growing rapidly, which means that solutions which work today might not work next month. You need to estimate the target size of your data and engineer for that, not the amount of data you have now.you will also have to create a data ageing or expiration policy so that you can eventually purge data and limit the database to some finite size. this is the hardest thing to do because ...
  62. 62. 3. Database size “We will decide on a data retention policy when we run out of disk space.” – every business user everywhere 14… users never ever ever want to “throw away” data. No matter how old it is or how infrequently its accessed.This isnt special to firehose systems except that they accumulate data faster than any other kind of system.
  63. 63. 4. 15problem #4 is illustrated by this architecture diagram of the Socorro system. Firehose systems have lots and lots of components, often using dozens of servers and hundreds of application instances. In addition to cost and administrative overhead, what having that much stuff means is ...
  64. 64. many components = many failures 16… every day of the week something is going to be crashing on you. You simply cant run a system with 90+ servers and expect them to be up all the time.
  65. 65. 4. Component failure ● all components fail ● or need scheduled downtime ● including the network ● collection must continue ● collection & processing must recover 17Even when components dont fail unexpectedly, they have to be taken down for upgrades, maintenance, replacement, or migration to new locations and software. But the data collection and processing has to go on while components are unavailable or offline. Even when processing has to stop, components must recover quickly when the system is ready again, and must do so without losing data.
  66. 66. solving firehose problems 18So lets talk about how a couple of different teams solved some of these firehose problems.
  67. 67. socorro project 19First were going to talk about Mozillas Socorro system. Ill bet a lot of you have seen this Firefox crash screen. The data it produces gets sent to the socorro system at Mozilla, where its collected and processed.
  68. 68. http://crash-stats.mozilla.com 20At the end of the processing we get pretty reports like this, which let the mozilla developers know when releases are stable and ready, and where frequent crashing and hanging issues are. You can visit it right now: Its http://crash-stats.mozilla.com
  69. 69. 21Theres quite a rich set of data there collecting everything about every aspect of whats making Firefox and other Mozilla products crash, individually and in the aggregate.
  70. 70. Mozilla Socorro webservers collectors reports processors 22this is a simplified architecture diagram showing the very basic data flow of Socorro.1. Crashes come in through the internet,2. are harvested by a cluster of collectors3. raw crashes are stored in Hbase4. a cluster of processors pull raw crashes from Hbase process crashes, and write processed crashes and metadata to Hbase and Postgres5. Postgres populates views which are used to support the web interface and reports used internally by mozilla developers.
  71. 71. 23a full architecture diagram would look like this one, but even this doesnt show all components.
  72. 72. socorro data volume ● 3000 crashes/minute ● avg. size 150K ● 40TB accumulated raw data ● 500GB accumulated metadata / reports 24socorro receives up to 3000 crashes every minute from around the world. each of these crashes comes with 50 to 1000K of binary crash data which needs to be processed.within the target data lifetime of 6-12 months, socorro has accumulated nearly 40 terabytes of raw data and half a terabyte of metadata, views and reports.
  73. 73. dealing with volume 25 load balancers collectorshow does Mozilla deal with this large data volume? In one word:parallelism1. crashes come in from the internet2. they hit a set of round-robin load balancers3. the load balancers randomly assign them to one of six crash collectors4. the crash collectors receive the crash information, and then write the raw crash to a 30-server Hbase cluster.
  74. 74. dealing with volume 26 monitor processors5. the raw crashes in HBase are pulled out in parallel by twelve processor servers running 10 processor instances each.6. these processors do a lot of work to process the binary crashes and then write the processed crash results to Hbase an the processed crash metadata and summary data to Postgres.7. for scheduling and preventing duplication, the processors are coordinated by a monitor server, which runs a queue and monitoring system backed by Postgres.
  75. 75. dealing with size data 40TB expandible metadata views 500GB fixed size 27Why have both Hbase and PostgreSQL? Isnt one database enough?Well, the 30-server Hbase cluster holds 40 terabytes of data. Its good at doing this in a redundant, scalable way and is very fast for pulling individual crashes out of the billion or so which are stored there.However Hbase is very poor at ad-hoc queries, or simple browsing of data over the web. It also often requires downtime for maintenanceSo we use Postgres which holds metadata and holds summary “materialized views” which are used by the web application and the reports to present analytics to users.Postgres cant store the raw data though because it cant easily expand to the sizes needed.
  76. 76. dealing with component failure Lots of hardware ... ● 30 Hbase ● 3 ES servers nodes ● 6 collectors ● 2 PostgreSQL ● 12 processors servers ● 8 middleware & ● 6 load web servers balancers … lots of failures 28Of course, with the more than 60 servers were dealing with, something is always going down. Thats simply reality. Some of the worst failures have involved the entire network going out and all components losing communication with each other.This means that we need to be prepared to have failures and to recover from them. How do we do that?
  77. 77. load balancing & redundancy 29 load balancers collectorsOne obvious way is to have lots of redundant components for each role in the application. As we already saw, for incoming data there is more than one load balancer and are 6 collectors. This means that we can lose several machines and still be receiving user data, and that in catastrophic failure we only lose an minority of the data.
  78. 78. elastic connections ● components queue their data ● retain it if other nodes are down ● components resume work automatically ● when other nodes come back up 30A more important strategy weve learned through hard experience is the requirement to have “elastic connections”. That is, each component is prepared to lose communication with the components on either side of it. They queue data trough several mechanisms, and resume work when the next component in line comes back up.
  79. 79. elastic connections crash mover collector reciever local file queue 31For example, lets look at the collectors. They receive data from the web and write to Hbase. But its more complex than that.1. The collectors can lose communication to Hbase for a variety of reasons. It can be down for maintenance, or there can be network issues.2. The crashes are received by receiver processes3. which write them to local file storage in a primitive, filename-based queue.4. A separate “crashmover” process looks for files in the queue5. When Hbase is available, the crashmover saves the files to Hbase.
  80. 80. server management ● puppet ● controls configuration of all servers ● makes sure servers recover ● allows rapid deployment of replacement nodes 32Another thing which has been essential to managing component failure is comprehensive server management through Puppet. Having servers under central management means that we can make sure services are up and configured correctly when they are supposed to be. And it also means that if we permanently lose nodes we can replace them quickly with identically configured nodes.
  81. 81. Upwind 33lets move on to a different team and different methods of facing firehose challenges. Upwind.
  82. 82. Upwind ● speed ● wind speed ● heat ● vibration ● noise ● direction 34power-generating wind turbines have onboard computers which produce a lot of data about the status and operation of the wind turbine Upwind takes that data, collects it, summarizes it and turns it into graphical summary reports for customers who own wind farms.
  83. 83. Upwind 1. maximize power generation 2. make sure turbine isnt damaged 35the customers need this data for two reasons:1. wind turbines are very expensive and they need to make sure theyre maximizing power generation from them.2. to intervene and keep the turbines from catching fire or otherwise becoming permanently damaged.
  84. 84. dealing with volume each turbine: 90 to 700 facts/second windmills per farm: up to 100 number of farms: 40+ est. total: 300,000 facts/second (will grow) 36turbines come with a lot of different monitors, between 90 and 700 of them depending on the model. each of these monitors produces a reading every second.an individual “wind farm” can have up to 100 turbines, and Upwind expects more than 40 farms to make use of the service once its fully deployed. Plus they want to get more customers.this means an estimated 300,000 metrics per second total by next year.
  85. 85. dealing with volume local historian analytic reports storage database local historian analytic reports storage database 37Upwind chose a very different route than Mozilla for dealing with this volume. Lets explain the data flow first:1. Wind farms write metrics to local storage, which can hold data for a couple hours.2. This data is then read by an industry-specific application called the historian which interprets the raw data and stores each second of data.3. that data is then read from the historian by Postgres4. which produces nice summaries and analytics for the reports.As it turns out, data is not shared between wind farms most of the time. So Upwind can scale simply by adding a whole additional service tool chain for each windfarm it brings online.
  86. 86. dealing with volume local historian analytic storage database master database local historian analytic storage database 38In the limited cases where we need to do inter-farm reports, we will have a master Postgres node which will use pl/proxy and/or foreign data wrappers to summarize data from multiple wind farms.
  87. 87. multi-tenant partitioning ● partition the whole application ● each customer gets their own toolchain ● allows scaling with the number of customers ● lowers efficiency ● more efficient with virtualization 39this is whats known as multi-tenant partitioning, where you give each customer, or “tenant”, a full stack of their own. Its not very resource-efficient, but it does easily scale as you add customers, and eliminates the need to use complex clustered storage or databases.cloud hosting and virtualization is also making this strategy easier to manage.
  88. 88. dealing with: constant flow and size minute hours days months years historian buffer table table table table minute hours days months historian buffer table table table minute hours days 40 historian buffer table tableIn order to deal with the constant flow of data and the large database size, we continously produce a series of time-based summaries of the data:1. a buffer holding a several minutes of data. this also accounts for lag and out-of-order data.2. hour summaries 3. day summaries4. month summaries 5. annual summaries
  89. 89. time-based rollups ● continuously accumulate levels of rollup ● each is based on the level below it ● data is always appended, never updated ● small windows == small resources 41For efficiency, each level builds on the level beneath it. This means that no analytic operation ever has to work with a very large set, an so can be very fast.Even better, the time-based summaries can be run in parallel, so that we can make maximum use of the processors on the server.
  90. 90. time-based rollups ● allows: ● very rapid summary reports for different windows ● retaining different summaries for different levels of time ● batch/out-of-order processing ● summarization in parallel 42this allows us to have a set of “views” which summarize the data at the time windows users look at. This means almost “instant” reports for users.
  91. 91. firehose tips 43based on my experiences with several of these firehose projects, heres some general rules for building this kind of an application
  92. 92. data collection must be: ● continuous ● parallel ● fault-tolerant 44data collection has to be 24x7. It must be done in parallel by multiple processes/servers, and above all must be fault-tolerant.
  93. 93. data processing must be: ● continuous ● parallel ● fault-tolerant 45data processing has to be built the same way.
  94. 94. every component must be able to fail ● including the network ● without too much data loss ● other components must continue 46the lesson most people dont seem to be prepared for is that every component must be able to fail without permanently downing the system, or causing unacceptable data loss.
  95. 95. 5 tools to use 1. queueing software 2. buffering techniques 3. materialized views 4. configuration management 5. comprehensive monitoring 47heres some classes of tools you should be using in any firehose application.queuing and buffering for fault-tolerancematerialized views to deal with volume and sizeconfiguration management and comprehensive monitoring to keep the system up.
  96. 96. 4 donts 1. use cutting-edge technology 2. use untested hardware 3. run components to capacity 4. do hot patching 48some things you will be tempted to do and shouldnt* dont use cutting edge hardware or software because its not reliable and will lead to repeated unacceptable downtimes* dont put hardware into service without testing it first beause it will be unreliable or perform poorly, dragging the system down* never run components or layers of your stack at capacity, because then you have no headroom for when you have a surge in data* and never do hot patching of production services because you cant manage it and it will lead to mistakes and taking the system down.
  97. 97. firehose mastered? 49so now that we know how to do it, its time for you to build your own firehose applications. its challenging and fun.
  98. 98. Contact ● Josh Berkus: josh@pgexperts.com ● blog: blogs.ittoolbox.com/database/soup ● PostgreSQL: www.postgresql.org ● pgexperts: www.pgexperts.com ● Upcoming Events ● PostgreSQL Europe: http://2011.pgconf.eu/ ● PostgreSQL Italy: http://2011.pgday.it/ The text and diagrams in this talk is copyright 2011 Josh Berkus and is licensed under the creative commons attribution license. Title slide image is licensed from iStockPhoto and may not be reproduced or redistributed. Socorro images are copyright 2011 Mozilla Inc. 50contact and copyright information.Questions?
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×