Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
1
April 2016 – AWS Loft, London
Gareth Rogers, Data Engineer
2
Metail lets you try on clothes online
Discover clothes on
your body shape
Create, save outfits
and share
Shop with confi...
3
Proven impact as validated by
American business schools and A/B tests
‘‘
…customers who had access to the fitting tool a...
4
Architecture Theory
• Our architecture is modelled on Nathan Marz’s Lambda Architecture: http://lambda-
architecture.net...
5
Architecture Practice – Data Collection
6
Architecture Practice – Data Collection
New Data and Collection
• We’re using Snowplow for the initial stages of our pip...
7
Architecture Practice – Serving Layer
Serving Layer• Initially queries over Hadoop  Redshift came along 
• RedshiftSQL...
8
Architecture Practice – Batch Layer
• Daily process the raw events to create batch view
• Run using Elastic MapReduce (E...
9
Extract Transform and Load (ETL)
• Snowplow’s ETL driven by config files executed in Ruby
– Initial step executed outsid...
10
Extract Transform and Load (ETL)
• Snowplow’s ETL driven by config files executed in Ruby
– Initial step executed outsi...
11
Extract Transform and Load (ETL)
• To that cluster we add steps
• Initial step use s3distcp to aggregate the log files
...
12
Extract Transform and Load (ETL)
• Metail’s ETL is very similar to
Snowplow’s
• Use AWS’ Data Pipeline to drive the
wor...
13
Extract Transform and Load (ETL)
• Metail ETL written in
– Cascalog, logic programming over Hadoop
– Cascalog = Cascadi...
14
Extract Transform and Load (ETL)
• Soon Parkour a Clojure wrapper over Hadoop Java API
– Access to full Hadoop API with...
15
Summary
• This pipeline has been built and managed by 3-5 people
• It’s about a year and a half old and continues to ev...
Upcoming SlideShare
Loading in …5
×

Metail and Elastic MapReduce

181 views

Published on

Metail allows users to discover clothes on their body shape online with minimum measurements from the user. With your avatar you can create outfits and coupled with our size advice this gives you a confidence in the size and fit.

I'm part of the team within Metail that has built a pipeline to collection, enriched and serve data to the company and our clients, and which has been used to validate Metail's product. This talk was given at the AWS Loft in London 21st April 2016 where I gave an overview of the end-to-end pipeline and then went into detail how we're using AWS' EMR to perform a batch processing of the collected data which is then served internally with Redshift.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Metail and Elastic MapReduce

  1. 1. 1 April 2016 – AWS Loft, London Gareth Rogers, Data Engineer
  2. 2. 2 Metail lets you try on clothes online Discover clothes on your body shape Create, save outfits and share Shop with confidence of size and fit
  3. 3. 3 Proven impact as validated by American business schools and A/B tests ‘‘ …customers who had access to the fitting tool are more likely to come back to the site, and this effect is statistically significant… ‘‘ …shows approximately a 5.1 percent reduction in returns compared to the control group…In other words, providing fit information reduces average fulfilment costs” …sales for users with access to the tool were substantially higher overall - 22.32 percent larger ‘‘ Source: “The Value of Fit Information in Online Retail: Evidence from a Randomized Field Experiment” by Prof Santiago Gallino (Dartmouth College - Tuck School of Business) & Prof Antonio Moreno (Northwestern University) –Oct 21, 2015 DATA 1000+ GARMENTS POINTS3M
  4. 4. 4 Architecture Theory • Our architecture is modelled on Nathan Marz’s Lambda Architecture: http://lambda- architecture.net • Should include a speed layer to give a real time view on sampled data – We’ve not implemented this New Data Batch Layer Master dataset Serving Layer Batch views Query Query Query
  5. 5. 5 Architecture Practice – Data Collection
  6. 6. 6 Architecture Practice – Data Collection New Data and Collection • We’re using Snowplow for the initial stages of our pipeline • Using their JavaScript tracker and Cloudfront collector configuration • Tracker performs a GET request on a Cloudfront distributed image (pixel) • Query parameters of the contain the event data e.g. GET http://d2sgzneryst63x.cloudfront.net/i?e=pv&url=...&page=...&... • Cloudfront configured to log the requests to S3 • We now have our master record
  7. 7. 7 Architecture Practice – Serving Layer Serving Layer• Initially queries over Hadoop  Redshift came along  • RedshiftSQL good for small data science team! • Not so good for everyone else in the company • Introduced Looker  • Data model in SQL • Dashboards • Point and click data exploration • Permissions • Version control
  8. 8. 8 Architecture Practice – Batch Layer • Daily process the raw events to create batch view • Run using Elastic MapReduce (EMR) hosted Hadoop service in AWS • Create views of the master record through enrichment and aggregation • Populates the schema for speedy Redshift queries Batch Layer
  9. 9. 9 Extract Transform and Load (ETL) • Snowplow’s ETL driven by config files executed in Ruby – Initial step executed outside of EMR – Copy data from Cloudfront incoming log bucket to another S3 bucket for processing – Next create EMR cluster
  10. 10. 10 Extract Transform and Load (ETL) • Snowplow’s ETL driven by config files executed in Ruby – Initial step executed outside of EMR – Copy data from Cloudfront incoming log bucket to another S3 bucket for processing – Next create EMR cluster
  11. 11. 11 Extract Transform and Load (ETL) • To that cluster we add steps • Initial step use s3distcp to aggregate the log files • Snowplow’s ETL written in Scalding – Scalding = Cascading (Java higher level MapReduce libraries) in Scala – They provide a compiled JAR hosted in S3
  12. 12. 12 Extract Transform and Load (ETL) • Metail’s ETL is very similar to Snowplow’s • Use AWS’ Data Pipeline to drive the workflow – Really great to get going – But quickly hit complexity limitations
  13. 13. 13 Extract Transform and Load (ETL) • Metail ETL written in – Cascalog, logic programming over Hadoop – Cascalog = Cascading + Datalog in Clojure – Ridiculously compact and expressive – But steep learning curve and impenetrable errors
  14. 14. 14 Extract Transform and Load (ETL) • Soon Parkour a Clojure wrapper over Hadoop Java API – Access to full Hadoop API with no abstractions just more idiomatic Clojure – Learning curve is mainly Hadoop – Errors still impenetrable
  15. 15. 15 Summary • This pipeline has been built and managed by 3-5 people • It’s about a year and a half old and continues to evolve • Composed of a few different technologies and EMR used to do the batch processing • Using EMR has made cluster managing and scaling straightforward • The synergy between EMR and S3 is a powerful feature – Encourages immutable infrastructure – You don’t need your compute cluster running to hold your data!

×