Introduction to Snowplow - Big Data & Data Science Israel

  • 1,003 views
Uploaded on

Snowplow is an open source event analytics platform built on Hadoop, Amazon Kinesis & Redshift …

Snowplow is an open source event analytics platform built on Hadoop, Amazon Kinesis & Redshift

Snowplow is a web and event analytics platform with a difference: rather than tell our users how they should analyze their data, we deliver their event-level data in their own data warehouse, on their own Amazon Redshift or Postgres database, so they can analyze it any way they choose. Snowplow is used by data-savvy media, retail, and SaaS businesses to better understand their audiences and how they engage with their websites and applications.

Agenda:

1. Intro to Snowplow - why we built it, what needs does it solve
2. Current Snowplow design and architecture
3. Agile event analytics with Snowplow & Looker
4. Evolution of Snowplow - from web analytics to a business' digital nervous system with Amazon Kinesis
5. Snowplow research & roadmap - event grammars, unified logs, feedback loops

Presenter Bio:
Alex Dean is the co-founder and technical lead at Snowplow Analytics. At Snowplow Alex is responsible for Snowplow's technical architecture, stewarding the open source community and evaluating new technologies such as Amazon Kinesis. Prior to Snowplow, Alex was a partner at technology consultancy Keplar, where the idea for Snowplow was conceived. Before Keplar Alex was a Senior Engineering Manager at OpenX, the open source ad technology company. Alex lives in London, UK.

More in: Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,003
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
31
Comments
0
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Introduction to Snowplow - an open source event analytics platform Big Data & Data Science – Israel
  • 2. Agenda today 1. Introduction to Snowplow 2. Current Snowplow design and architecture 3. Agile event analytics with Snowplow & Looker 4. Evolution of Snowplow 5. Questions Many thanks for organizing to:
  • 3. Introduction to Snowplow
  • 4. Snowplow is an open-source web and event analytics platform, first version released in early 2012 • Co-founders Alex Dean and Yali Sassoon met at OpenX, the open-source ad technology business in 2008 • After leaving OpenX, Alex and Yali set up Keplar, a niche digital product and analytics consultancy • We released Snowplow as a skunkworks prototype at start of 2012: github.com/snowplow/snowplow • We started working full time on Snowplow in summer 2013
  • 5. At Keplar, we grew frustrated by significant limitations in traditional web analytics programs • Sample-based (e.g. Google Analytics) • Limited set of events e.g. page views, goals, transaction s • Limited set of ways of describing events (custom dim 1, custom dim 2…) Data collection Data processing Data access • Data is processed ‘once’ • No validation • No opportunity to reprocess e.g. following update to business rules • Data is aggregated prematurely • Only particular combinations of metrics / dimensions can be pivoted together (Google Analytics) • Only particular type of analysis are possible on different types of dimension (e.g. sProps, eVars, conversion goals in SiteCatalyst • Data is either aggregated (e.g. Google Analytics), or available as a complete log file for a fee (e.g. Adobe SiteCatalyst) • As a result, data is siloed: hard to join with other data sets
  • 6. And we saw the potential of new “big data” technologies and services to solve these problems in a scalable, low-cost manner These tools make it possible to capture, transform, store and analyse all your granular, event-level data, to you can perform any analysis Amazon EMRAmazon S3CloudFront Amazon Redshift
  • 7. We wanted to take a fresh approach to web analytics • Your own web event data -> in your own data warehouse • Your own event data model • Slice / dice and mine the data in highly bespoke ways to answer your specific business questions • Plug in the broadest possible set of analysis tools to drive value from your data Data warehouseData pipeline Analyse your data in any analysis tool
  • 8. Early on, we made a crucial decision: Snowplow should be composed of a set of loosely coupled subsystems 1. Trackers 2. Collectors 3. Enrich 4. Storage 5. AnalyticsA B C D D = Standardised data protocols Generate event data from any environment Launched with: • JavaScript tracker Log raw events from trackers Launched with: • CloudFront collector Validate and enrich raw events Launched with: • HiveQL + Java UDF- based enrichment Store enriched events ready for analysis Launched with: • Amazon S3 Analyze enriched events Launched with: • HiveQL recipes These turned out to be critical to allowing us to evolve the above stack
  • 9. Our initial skunkworks version of Snowplow – it was basic but it worked, and we started getting traction Website / webapp Snowplow data pipeline v1 (spring 2012) CloudFront- based pixel collector HiveQL + Java UDF “ETL” Amazon S3 JavaScript event tracker
  • 10. What did people start using it for? Warehousing their web event data Agile aka ad hoc analytics To enable… Marketing attribution modelling Customer lifetime value calculations Customer churn detection RTB fraud Product recommendations
  • 11. Current Snowplow design and architecture
  • 12. Our protocol-first, loosely-coupled approach made it possible to start swapping out existing components… Website / webapp Snowplow data pipeline v2 (spring 2013) CloudFront- based event collector Scalding- based enrichment JavaScript event tracker HiveQL + Java UDF “ETL” Amazon Redshift / PostgreSQL Amazon S3 or Clojure- based event collector
  • 13. Our protocol-first, loosely-coupled approach made it possible to start swapping out existing components… Website / webapp Snowplow data pipeline v2 (spring 2013) CloudFront- based event collector Scalding- based enrichment JavaScript event tracker HiveQL + Java UDF “ETL” Amazon Redshift / PostgreSQL Amazon S3 or Clojure- based event collector • Allow Snowplow users to set a third-party cookie with a user ID • Important for ad networks, widget companies, multi- domain retailers • Because Snowplow users wanted a much faster query loop than HiveQL/MapReduc e • We wanted a robust, feature-rich framework for managing validations, enrich ments etc
  • 14. What is Scalding? • Scalding is a Scala API over Cascading, the Java framework for building data processing pipelines on Hadoop: Hadoop DFS Hadoop MapReduce Cascading Hive Pig Java Scalding Cascalog PyCascading cascading. jruby
  • 15. We chose Cascading because we liked their “plumbing” abstraction over vanilla MapReduce
  • 16. Why did we choose Scalding instead of one of the other Cascading DSLs/APIs? • Lots of internal experience with Scala – could hit the ground running (only very basic awareness of Clojure when we started the project) • Scalding created and supported by Twitter, who use it throughout their organization – so we knew it was a safe long-term bet • We believe that data pipelines should be as strongly typed as possible – all the other DSLs/APIs on top of Cascading encourage dynamic typing: • Define the inputs and outputs of each of your data processing steps in an unambiguous way • Catch errors as soon as possible – and report them in a strongly typed way too
  • 17. Our “enrichment process” (formerly known as ETL) actually does two things: validation and enrichment • Our validation model looks like this: • Under the covers, we use a lot of monadic Scala (Scalaz) code Raw events “Bad” raw events + reasons why they are bad “Good” enriched events Enrichment Manager
  • 18. Adding the enrichments that web analysts expect = very important to Snowplow uptake • Web analysts are used to a very specific set of enrichments from Google Analytics, Site Catalyst etc • These enrichments have evolved over the past 15-20 years and are very domain specific: • Page querystring -> marketing campaign information (utm_ fields) • Referer data -> search engine name, country, keywords • IP address -> geographical location • Useragent -> browser, OS, computer information
  • 19. We aim to make our validation and enrichment process as modular as possible Enrichment Manager Not yet integrated • This encourages testability and re-use – also it widens the number of contributors vs this functionality being embedded in Snowplow • The Enrichment Manager uses external libraries (hosted in a Snowplow repository) which can be used in non-Snowplow projects:
  • 20. Agile event analytics with Snowplow and Looker
  • 21. Just last week we announced our official partnership with Looker • Looker is a BI visualization and data modelling startup with some cool features: 1. Slice and dice any combination of dimension and metrics 2. Quickly and easily define dimensions and metrics that are specific to your business using Looker's light-weight metadata model 3. Drill-up and drill-down to visitor-level and event-level data 4. Dashboards are a starting point for more involved analysis 5. Access your data from any application: Looker as a general purpose data server +
  • 22. Demo – first let’s look at some enriched Snowplow events in Redshift
  • 23. Demo – now let’s see how that translates into Looker
  • 24. Evolution of Snowplow
  • 25. There are three big aspects to Snowplow’s roadmap 1. Make Snowplow work as well for non-web (e.g. mobile, IoT) environments as the web 2. Make Snowplow work as well with unstructured events as it does with structured events (aka page views, ecommerce transactions etc) 3. Move Snowplow away from an S3-based data pipeline to a unified log (Kinesis/Kafka)-based data pipeline
  • 26. Snowplow is developing into an event analytics platform (not just a web analytics platform) Data warehouse Collect event data from any connected device
  • 27. So far we have open-sourced a few different trackers – with more planned JavaScript Tracker – the original No-JS aka pixel tracker Lua Tracker – for games Arduino Tracker – for the Internet of Things Python Tracker – releasing this week
  • 28. As we get further away from the web, we need to start supporting unstructured events • By unstructured events, we mean events represented as JSONs with arbitrary name: value pairs (arbitrary to Snowplow, not to the company using Snowplow!) _snaq.push(['trackUnstructEvent', 'Viewed Product', { product_id: 'ASO01043', category: 'Dresses', brand: 'ACME', returning: true, price: 49.95, sizes: ['xs', 's', 'l', 'xl', 'xxl'], available_since$dt: new Date(2013,3,7) } ]);
  • 29. Supporting structured and unstructured events is a difficult problem • Almost all of our competitors fall on one or other side of the structured- unstructured divide: Structured events (page views etc) Unstructured events (JSONs)
  • 30. We want to bridge that divide, making it so that Snowplow comes with structured events “out of the box”, but is extensible with unstructured events Structured events (page views etc) Unstructured events (JSONs)
  • 31. This is super-important to enable businesses to construct their own high-value bespoke analytics • What is the impact of different ad campaigns and creative on the way users behave, subsequently? What is the return on that ad spend? • How do visitors use social channels (Facebook / Twitter) to interact around video content? How can we predict which content will “go viral”? • How do updates to our product change the “stickiness” of our service? ARPU? Does that vary by customer segment?
  • 32. To achieve this, we are prototyping a new approach using JSON Schema, Thrift/Avro and a shredding library • We are planning to replace the existing flow with a JSON Schema-driven approach: Enrichment Manager Raw events in JSON format JSON Schema defining events Enriched events in Thrift or Arvo format Shredder 1. Define structure 2. Validate events 3. Define structure 4. Drive shredding Enriched events in TSV ready for loading into db 5. Define structure
  • 33. JSON Schema just gives us a way of representing structure – we are also evolving a grammar to represent events Subject Direct Object Indirect Object Verb Event Context Prep. Object~
  • 34. In parallel, we plan to evolve Snowplow from an event analytics platform into a “digital nervous system” for data driven companies • The event data fed into Snowplow is written into a “Unified Log” • This becomes the “single source of truth”, upstream from the datawarehouse • The same source of truth is used for real-time data processing as analytics e.g. • Product recommendations • Ad targeting • Real-time website personalisation • Systems monitoring Snowplow will drive data-driven processes as well as off- line analytics
  • 35. CLOUD VENDOR / OWN DATA CENTER Search Silo SOME LOW LATENCY LOCAL LOOPS E-comm Silo CRM SAAS VENDOR #2 Email marketing ERP Silo CMS Silo SAAS VENDOR #1 NARROW DATA SILOES Streaming APIs / web hooks Unified log Archiving Hadoop < WIDE DATA COVERAGE > < FULL DATA HISTORY > Systems monitoring Eventstream HIGH LATENCY LOW LATENCY Product rec’s Ad hoc analytics Management reporting Fraud detection Churn prevention APIs Some background on unified log based architectures
  • 36. We are part way through our Kinesis support, with additional components being released soon Scala Stream Collector Raw event stream Enrich Kinesis app Bad raw events stream Enriched event stream S3 Redshift S3 sink Kinesis app Redshift sink Kinesis app Snowplow Trackers • The parts in grey are still under development – we are working with Snowplow community members on these collaboratively
  • 37. Questions? http://snowplowanalytics.com https://github.com/snowplow/snowplow @snowplowdata To have a meeting, coffee or beer tomorrow (Monday) – @alexcrdean or alex@snowplowanalytics.com
  • 38. Useful for answering questions… Website / webapp Snowplow data pipeline v2 (spring 2013) CloudFront- based event collector Scalding- based enrichment JavaScript event tracker HiveQL + Java UDF “ETL” Amazon Redshift / PostgreSQL Amazon S3 or Clojure- based event collector
  • 39. Useful for answering questions… 1. Trackers 2. Collectors 3. Enrich 4. Storage 5. AnalyticsA B C D D = Standardised data protocols Generate event data from any environment Launched with: • JavaScript tracker Log raw events from trackers Launched with: • CloudFront collector Validate and enrich raw events Launched with: • HiveQL + Java UDF- based enrichment Store enriched events ready for analysis Launched with: • Amazon S3 Analyze enriched events Launched with: • HiveQL recipes These turned out to be critical to allowing us to evolve the above stack