This document describes Vente-Exclusive's use of Google Cloud Platform technologies like Google Cloud Storage, Cloud DataProc, BigQuery, Cloud Pub/Sub, Cloud DataFlow and App Engine for their big data analytics processes. It discusses how they leverage these services for ETL, batch and stream processing of user behavior and transactional data to power analytics dashboards and optimize various business metrics like reducing returns, improving customer insights and forecasting.

1. GCP at Vente-Exclusive
Big Data on Google Cloud

2. Flash sales?
We reserve a limited
supply of products at
a supplier
We prepare them for
selling
You buy some of the
products on offer
We order only what
was purchased
And we ship it!

3. Process: brands
Pre-Sales Marketing Sales LogisticsSourcing

4. Process: prepare the sales
Pre-Sales Marketing Sales LogisticsSourcing

5. Process: communicate
Pre-Sales Marketing Sales LogisticsSourcing

6. Process: sales started
Pre-Sales Marketing Sales LogisticsSourcing

7. Process: ship it
Pre-Sales Marketing Sales LogisticsSourcing

8. Data
● Avoiding returns
● Better product offering
● More relevant communication
● Cost efficient advertising
● Reporting to our suppliers
● Forecasting of logistics workforce
● Estimating of stock purchases
● Investigate where to diversify
● More detailed customer insights
● Improving multi-channel conversion
● Much much more …

9. Crunching Power
Dark Ages

10. Hadoop rollout

11. Database
SQL Server Cloud Storage Compute Engine

12. Hadoop: uploading denormalized
SQL Server Cloud Storage Compute Engine

13. Hadoop: nightly processing
SQL Server Cloud Storage Compute Engine

14. Hadoop: download to SQL
SQL Server Cloud Storage Compute Engine

15. Hadoop: complete setup
SQL Server Cloud Storage Compute Engine

16. Hadoop: DataProc service
SQL Server Cloud Storage DataProc

17. Pig Example
ul = LOAD '${users}' using AvroStorage();
province = LOAD 'gs://bucket/bi_ProvincePostalCode.csv'
using CSVExcelStorage() as (country_id:int,postal_code:chararray,province:chararray);
ul = foreach ul generate user_id AS user_id, ... vp_member_id as vp_member_id;
ulp = JOIN ul BY (country_id,postal_code) LEFT OUTER, province BY (country_id,postal_code) USING 'replicated';
ulp = foreach ulp generate
ISOToBQ('${year}-${month}-${day}T00:00:00Z') AS bucket_date,
user_id AS user_id, … loyalty_segment as loyalty_segment,
vp_member_id as vp_member_id;
store ulp into '${out}' using JsonStorage();

18. DataProc Console

19. Hadoop Learnings
● Cloud Storage is a life saver
○ Reliable
○ Always accessible
○ Not gone when you delete the cluster (read HDFS)
○ Well worth the speed tradeoff
● On demand cluster
○ Saves Compute cost
● DataProc
○ Cluster up in about 1 minute
○ Better per job overview and logging

20. BigQuery rollout

21. BigQuery: interactive queries

22. BigQuery: preparing for import
Cloud Storage BigQueryDataProc

23. BigQuery: importing
Cloud Storage BigQueryDataProc

24. BigQuery: exporting to storage
Cloud Storage BigQueryDataProc

25. BigQuery: reuse in Hadoop
Cloud Storage BigQueryDataProc

26. BigQuery: download to SQL
Cloud Storage BigQuerySQL Server

27. BigQuery: big picture
Cloud Storage BigQueryDataProc

28. BigQuery: connector
Cloud Storage BigQuery
DataProc

29. BigQuery Learnings
● User centric
○ Don’t have to be a Data Scientist or Engineer
○ Think more about the datasets and table layout
○ Harder to change, because it’s used throughout the company
● Cost
○ One shots are never a problem
○ Integration in incur hidden cost
● Bucketization
○ Buckets per day
○ Monthly or Years if data is too small
○ Create views for users

30. Interlude
Age of Enlightenment

31. Orchestration

32. Orchestration: foundation of big data

33. Visualisation

34. Visualisation: tableau

38. Visualisation: integration in BackOffice

39. Integration Learnings
● Making it visible for the business
● Keep an eye on hidden cost
● Training is key

40. Realtime
Space Age

41. Cloud PubSub

42. PubSub: collecting user behaviour
Cloud PubSub
App Engine

43. PubSub: publishing events on-premises
Cloud PubSub
App Engine

44. PubSub: 3th party integration
Cloud PubSub
App Engine

45. PubSub: publish via App Engine
Cloud PubSub
App Engine

46. PubSub: ingres
Cloud PubSub
App Engine

47. Cloud DataFlow

48. DataFlow: PubSub source
DataFlow
(streaming)
DataFlow (batch)
BigQuery

49. DataFlow: streaming processing
DataFlow
(streaming)
DataFlow (batch)
BigQuery

50. DataFlow: streaming BigQuery inserts
DataFlow
(streaming)
DataFlow (batch)
BigQuery

51. DataFlow: real time dashboards
DataFlow
(streaming)
DataFlow (batch)
BigQuery

52. DataFlow: ingres backup
DataFlow
(streaming)
DataFlow (batch)
BigQuery

53. DataFlow: unified model
DataFlow
(streaming)
DataFlow (batch)
BigQuery

54. DataFlow: restore or new flow
DataFlow (batch)
DataFlow
(streaming)
BigQuery

55. DataFlow: big real time picture
DataFlow (batch)
DataFlow
(streaming)
BigQuery

56. Real-time Learnings
● DataFlow is key
○ Unified model
○ Powerful semantics for streaming

57. Complete picture
Fitting it all together

58. Current State of Affairs

59. Q&A
No state secrets