Event counting and statistics at Finn.noAlexei Bakanovteamcore@finn.no
Finn.no Cassandra usecases● Event counting and statistics● FINNboks messaging system● IP-Geo● Your Last searches● JDBC-act...
Usecase: Event counting and statisticsAn Event:● Anything that has a timestamp on it(not continious)
Event example: Pageview of an ad
Event example: «Annonsegalleri» banner is shown
Event example: «Lagret søk» email is sent for an ad
Usecase: Event counting and statistics
Usecase: Event counting and statistics(testdata)
Usecase: Event counting and statistics“Stone age” system Old system Current systemCounter updates inWeb-app.Storing in RDB...
Usecase: Event counting and statistics. «Stone age» system«Stone age» architecture:● Syncronous counter updates● Updating ...
Usecase: Event counting and statistics. «Stone age» systemRDBSWeb server Web server Web server. . .++count++count++count
Usecase: Event counting and statistics. «Stone age» systemPros:● Real time numbersCons:● High DB commit-log write times du...
Usecase: Event counting and statistics. «Stone age» systemVery long time ago...
Usecase: Event counting and statistics. Old systemOld architecture:● Asyncronous event logging via Scribe● Saving event da...
Usecase: Event counting and statistics. Old architecturePros:● Less load on main RDBMS● Intervall based statistics● Abilit...
Usecase: Event counting and statistics.«Oppdrag»s interval-based statistics(testdata)
Usecase: Event counting and statistics. Grouped by subCategory
Usecase: Event counting and statistics. Grouped by client types
Usecase: Event counting and statistics. Grouped by referrer domain
Usecase: Event counting and statistics. Repeated views by a person
Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassa...
Usecase: Event counting and statistics. Old system. Event loggingEvent bean (Thrift IDL):struct Event {/** Event domain. T...
Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassa...
Usecase: Event counting and statistics. Old system. Raw event data● Ca 1500k events per sec to log in peak times● ByteOrde...
Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassa...
Usecase: Event counting and statistics. Old system. AggregationHadoop jobs:● Sum up events for each finnkode grouped bysub...
Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassa...
Usecase: Event counting and statistics. Old system. Aggregated dataSuperColumns for interval-based countersHOUR_2013_05_30...
Usecase: Event counting and statistics. Current system. Aggregated dataHOUR_2013_05_30_18:PageViewHOUR_2013_05_30_18:Email...
Usecase: Event counting and statistics. Current system. Aggregated data
Usecase: Event counting and statistics. Old system. DisadvantagesOld Raw event data cluster disadvantages:ByteOrderedParti...
Usecase: Event counting and statistics. Current system. Raw dataCurrent system for raw data:● Same cluster as aggregated d...
Usecase: Event counting and statistics. Current system. Raw data● CQL = SQL without JOINs, GROUPBYs and other unimportant ...
Usecase: Event counting and statistics. Current system. Raw dataCREATE TABLE events (realtb_sharded text, ← Partition keyt...
Usecase: Event counting and statistics. Current system. Raw dataHadoop data reading:1. Get a list of InputSplitsHDFS:A fil...
Usecase: Event counting and statistics. Current system. Raw dataProcess all data collected 18:00 – 19:00 30.05.2013:1. Get...
Usecase: Event counting and statistics. Current system. Raw dataProcess data of type “AD” collected 18:00 – 19:00 30.05.20...
Usecase: Event counting and statistics. Current system. Raw dataGetting InputSplits:Get Partition keys for data of type AD...
Usecase: Event counting and statistics. NextGenEvent Counting and Statistics NextGen:Ad-hoc analytics:– Apache Hive integr...
Usecase: Event counting and statistics?
Cassandra at Finn.io — May 30th 2013
Upcoming SlideShare
Loading in...5
×

Cassandra at Finn.io — May 30th 2013

360

Published on

Published in: Technology, Design
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
360
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Cassandra at Finn.io — May 30th 2013"

  1. 1. Event counting and statistics at Finn.noAlexei Bakanovteamcore@finn.no
  2. 2. Finn.no Cassandra usecases● Event counting and statistics● FINNboks messaging system● IP-Geo● Your Last searches● JDBC-activity monitoring● Scam-control activity log← Today
  3. 3. Usecase: Event counting and statisticsAn Event:● Anything that has a timestamp on it(not continious)
  4. 4. Event example: Pageview of an ad
  5. 5. Event example: «Annonsegalleri» banner is shown
  6. 6. Event example: «Lagret søk» email is sent for an ad
  7. 7. Usecase: Event counting and statistics
  8. 8. Usecase: Event counting and statistics(testdata)
  9. 9. Usecase: Event counting and statistics“Stone age” system Old system Current systemCounter updates inWeb-app.Storing in RDBMS.Raw Event data in C*(ByteOrderedPartitioner)Hadoop jobs to rollupevent dataAggregated data in C*(RandomPartitioner +SuperColumns)CQL3 for raw Event DataComposite columns forAggregated data
  10. 10. Usecase: Event counting and statistics. «Stone age» system«Stone age» architecture:● Syncronous counter updates● Updating counters inside a web-app● Using Finns main relational database as astorage for counters
  11. 11. Usecase: Event counting and statistics. «Stone age» systemRDBSWeb server Web server Web server. . .++count++count++count
  12. 12. Usecase: Event counting and statistics. «Stone age» systemPros:● Real time numbersCons:● High DB commit-log write times during peak-hours. Overall Finn performance degradation.● No interaval based statistics like daily counters,just totals
  13. 13. Usecase: Event counting and statistics. «Stone age» systemVery long time ago...
  14. 14. Usecase: Event counting and statistics. Old systemOld architecture:● Asyncronous event logging via Scribe● Saving event data in a raw unnormalized formatto C*● Hadoop jobs to sum up event counters over timeperiods● Serving aggregated statistics from C*
  15. 15. Usecase: Event counting and statistics. Old architecturePros:● Less load on main RDBMS● Intervall based statistics● Ability to re-aggregate data and get new insights● Better Command-Query separationCons:● Not real-time, although jobs run every minute
  16. 16. Usecase: Event counting and statistics.«Oppdrag»s interval-based statistics(testdata)
  17. 17. Usecase: Event counting and statistics. Grouped by subCategory
  18. 18. Usecase: Event counting and statistics. Grouped by client types
  19. 19. Usecase: Event counting and statistics. Grouped by referrer domain
  20. 20. Usecase: Event counting and statistics. Repeated views by a person
  21. 21. Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassandra1.finn.noCassandra(ByteOrderedPartitioner)Raw event datawebserver2.finn.no…..webserver3.finn.no…..Scribe daemonCassandra(RandomPartitioner)Aggregated dataHadoop aggregation job
  22. 22. Usecase: Event counting and statistics. Old system. Event loggingEvent bean (Thrift IDL):struct Event {/** Event domain. Typical AD, CV, Oppdrag */1: required string type;/** Event name. Typical PageView, EmailSent */2: optional string subCategory;/** Arbitrary key-value map with extra info like finnkode or userid */3: required map<string, string> values;}● Event bean-> binary-> base64 + timestamp-> scribe message
  23. 23. Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassandra1.finn.noCassandra(ByteOrderedPartitioner)Raw event datawebserver2.finn.no…..webserver3.finn.no…..Scribe daemonCassandra(RandomPartitioner)Aggregated dataHadoop aggregation job
  24. 24. Usecase: Event counting and statistics. Old system. Raw event data● Ca 1500k events per sec to log in peak times● ByteOrderedPartitioner for storing raw data. TTL = 3mnth(picture from datastax.com)Timestamp Eventbean1369164000 0x1f0562bda6...1369164001 0x364dd9a5a6...1369164002 0x4d96508da6...1369164003 0x64dec775a6...Sequential rowkeys = Hadoop friendlyget_range_slices(1369164000, 1369164003) toget data for Hadoop splits
  25. 25. Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassandra1.finn.noCassandra(ByteOrderedPartitioner)Raw event datawebserver2.finn.no…..webserver3.finn.no…..Scribe daemonCassandra(RandomPartitioner)Aggregated dataHadoop aggregation job
  26. 26. Usecase: Event counting and statistics. Old system. AggregationHadoop jobs:● Sum up events for each finnkode grouped bysubCategory.map(events):for event in events:finnkode = event.getValues().get(“ad.id”)subcategoryCount = (event.subcategory, 1)emit(finnkode, subcategoryCount)reduce(finnkode, subcategoryCountList):subcategoryTotals = {}for subcategory, count in subcategoryCountList:subcategoryTotals[subcategory] += countfor subcategory, count in subcategoryTotals.iteritems():incrementCassandraCounter(finnkode, subcategory, count, HOUR_somehour)incrementCassandraCounter(finnkode, subcategory, count, DAY_someday)incrementCassandraCounter(finnkode, subcategory, count, TOTAL)
  27. 27. Usecase: Event counting and statistics. Old system. Data flowwebserver1.finn.noTomcatScribe daemonasync event loggingcassandra1.finn.noCassandra(ByteOrderedPartitioner)Raw event datawebserver2.finn.no…..webserver3.finn.no…..Scribe daemonCassandra(RandomPartitioner)Aggregated dataHadoop aggregation job
  28. 28. Usecase: Event counting and statistics. Old system. Aggregated dataSuperColumns for interval-based countersHOUR_2013_05_30_18 DAY_2013_05_30 TOTALPageView Email PageView Email PageView Email3706119 10 2 20 4 50 53706052 23 3 102 4 234 10Min. time resolution QUARTER_HOURNo TTL on Counter columns, got really wide and slowhttp://www.makingitscale.com/2012/scaling-cassandra-counter-columns.html
  29. 29. Usecase: Event counting and statistics. Current system. Aggregated dataHOUR_2013_05_30_18:PageViewHOUR_2013_05_30_18:EmailDAY_2013_05_30:PageViewDAY_2013_05_30:EmailTOTAL:PageViewTOTAL:Email3706119 10 2 20 4 50 53706052 23 3 102 4 234 10HOUR_2013_05_30_18 DAY_2013_05_30 TOTALPageView Email PageView Email PageView Email3706119 10 2 20 4 50 53706052 23 3 102 4 234 10SuperColumnsCompositeColumns+ clean up jobs to remove old QUARTER_HOUR and HOUR columnsMigration from SuperColumns to Composite columns
  30. 30. Usecase: Event counting and statistics. Current system. Aggregated data
  31. 31. Usecase: Event counting and statistics. Old system. DisadvantagesOld Raw event data cluster disadvantages:ByteOrderedPartitioner: just one node at work at a time taking all loadSkinny rows: 5 billions rows for 3 month of data on each nodeExtreme unstable instances failing with OOM errorsHadoop jobs fail or hang on init stage despite of QUORUM consistencyOutdated statistics across Finn services+=
  32. 32. Usecase: Event counting and statistics. Current system. Raw dataCurrent system for raw data:● Same cluster as aggregated data, i.e.RandomPartitioner + CQL3
  33. 33. Usecase: Event counting and statistics. Current system. Raw data● CQL = SQL without JOINs, GROUPBYs and other unimportant stuff● Abstraction over physical C* storage● CQL Table transposes rows into Composite columnsTimestamp Eventbean1369164000 0x1f0562bda6...1369164001 0x364dd9a5a6...1369164002 0x4d96508da6...1369164003 0x64dec775a6...1369164000:Eventbean1369164001:Eventbean1369164002:Eventbean1369164003:Eventbean0x1f0562bda6... 0x364dd9a5a6... 0x4d96508da6... 0x64dec775a6...CQL tableUnderlying ColumnFamilyTimebucket1369164000Timebucket1369164000136916400013691640001369164000Partition key(row key)Clustering keyPartition + clustering = PRIMARY KEYOther columns are static for every PK pair
  34. 34. Usecase: Event counting and statistics. Current system. Raw dataCREATE TABLE events (realtb_sharded text, ← Partition keytype text, ← Clustering keycollected_ts timeuuid, ← Clustering keyPRIMARY KEY(realtb_sharded, type, collected_ts),key_values_json text,← Static columnreal_ts timestamp, ← Static columnreal_tb bigint, ← Secondary Indexcollected_tb bigint ← Secondary Index);“real” timestamp – Event occurred at the client“collected” timestamp – Event reached C*
  35. 35. Usecase: Event counting and statistics. Current system. Raw dataHadoop data reading:1. Get a list of InputSplitsHDFS:A file block is replicated across several machinesFileInputSplit: (“file, start, length”, IP-addresses)C*:Rows with same Partition key are replicated across several machinesEventsInputSplit: (Partition key, IP-addresses)2. InputSplit → Map-tasks on IP-addresses3. Map-task reads data based on:HDFS: “file, start, length”C*: Partition key
  36. 36. Usecase: Event counting and statistics. Current system. Raw dataProcess all data collected 18:00 – 19:00 30.05.2013:1. Get InputSplits:For minute in (18:00-19:00).getMinutes:SELECT realtb_sharded FROM events WHERE collected_tb = minutetoken(realtb_sharded) → IP-addresses2. Map-task:SELECT * FROM events WHERE realtb_sharded=17:58SELECT * FROM events WHERE realtb_sharded=17:59SELECT * FROM events WHERE realtb_sharded=18:03CREATE TABLE events (realtb_sharded text, ← Partition keytype text, ← Clustering keycollected_ts timeuuid, ← Clustering keyPRIMARY KEY(realtb_sharded, type, collected_ts),key_values_json text,← Static columnreal_ts timestamp, ← Static columnreal_tb bigint, ← Secondary Indexcollected_tb bigint ← Secondary Index);
  37. 37. Usecase: Event counting and statistics. Current system. Raw dataProcess data of type “AD” collected 18:00 – 19:00 30.05.2013:1. Get InputSplits:For minute in (18:00-19:00).getMinutes:SELECT realtb_sharded FROM events WHERE collected_tb = minuteand type = “AD”token(realtb_sharded) → IP-addresses2. Map-task:SELECT * FROM events WHERE realtb_sharded=17:58 and type = “AD”and collected_ts>minTimeuuid(“18:00”)and collected_ts<maxTimeuuid(“19:00”)CREATE TABLE events (realtb_sharded text, ← Partition keytype text, ← Clustering keycollected_ts timeuuid, ← Clustering keyPRIMARY KEY(realtb_sharded, type, collected_ts),key_values_json text,← Static columnreal_ts timestamp, ← Static columnreal_tb bigint, ← Secondary Indexcollected_tb bigint ← Secondary Index);
  38. 38. Usecase: Event counting and statistics. Current system. Raw dataGetting InputSplits:Get Partition keys for data of type AD collected during timebucket 29.05.2013 21:18 –21:19Hadoop Map-task:Get rows for a Partition key from split limiting by type and collected timestamp
  39. 39. Usecase: Event counting and statistics. NextGenEvent Counting and Statistics NextGen:Ad-hoc analytics:– Apache Hive integration (we have Pig)– Hive ODBC driver for Tableu integrationAggregation jobs:– Higher level library like Cascading or Apache Crunch than rawM/R-code.Hadoop 2
  40. 40. Usecase: Event counting and statistics?

×