The data sizes in E-commerce are quickly outpacing our ability to deal with by using standard big data tools. But we already know how to scale the technical capacity, but how do we scale the knowledge of the organization, the data security, our ability to quickly iterate and gain a positive impact from it? It all starts with having the right tools available, and ends with setting up a clear architecture, that is on one hand rigidly enforced, but still provides enough flexibility to enable creativity. In the talk we will also touch upon some key difference between the "western" big data tools and the ones created by Alibaba.

1. Dealing with small
challenges when
they become
massive
Data at scale

2. 1

3. 2

4. 3

5. 1st Task

6. 2nd Task

7. n-th Task

8. Size of the task
Difficultyofthetask

9. Size of the task
Difficultyofthetask
X X X
……

10. n-th Task

12. Complete collapse
I hate my job
Discomfort
Easy
Size of the task
Difficultyofthetask
When to give up and invest in a new
approach?

13. Re-focus from features to data platform is
needed
Tool development
Use case
1
Use case
2
Use case
3
Tool development
Use case
1
Use case
2
Use case
3

14. To put it simply…
13
Data Model

15. To put it simply…
14
Data ModelIN

16. To put it simply…
15
Data ModelIN

17. To put it simply…
16
Data ModelIN OUT

18. Possible goals for the “ideal” Data platform
17
Data
ModelIN OUT
1) Include all
data
2) Real-time
collection

19. Possible goals for the “ideal” Data platform
18
Data
ModelIN OUT
1) Usability
2) Stability
3) Efficiency
1) Include all
data
2) Real-time
collection

20. Possible goals for the “ideal” Data platform
19
Data
ModelIN OUT
1) Usability
2) Stability
3) Efficiency
1) Maximize
usage
2) Maximize
utility
1) Include all
data
2) Real-time
collection

21. What to focus on with Data Platform
development?
Sales Tracker …
Data
consumers
Business System Data
Operational Data Store
Common Data/Dimension Model
Application Data Service
(Big) Data Warehouse
FinanceDSBI …
Real-time
log collection
Real-time and Batch
versions of the data model
Standardization of CDM
with clear naming
convention
Meta-data and lineage
management tool
Compatibility of data
consumption tools with
core platform
Data staging with
connectivity to real-time
and batch platforms
Access controls on
column level
Data labeling and
anonymization,
particularly for PII data
IDE for ease of
deployment and ops
Automated Data
Quality control
Data stability
monitoring and alerting
Data transfer capacity
management tool
Computing queue and
storage management

22. 21
If you want to go fast, go alone. If you want to go far, go together.
African proverb