The document discusses reference architectures for enterprise big data use cases. It begins by providing background on how databases have scaled over time and the evolution of large-scale data processing. It then discusses the basic idea behind big data use cases, which is to use all available data regardless of structure or source. The document outlines some key requirements like fault tolerance, dynamic scaling, and processing all data types. It proposes an architectural approach using NoSQL databases and cloud computing alongside traditional data warehousing. Finally, it shares two reference architectures - the current IBM approach and a transitional approach.

1. P4.1 Reference Architectures for Enterprise Big Data Use Cases
Romeo Kienzler, Data Scientist, Advisory Architect, IBM Germany, Austria, Switzerland
IBM Center of Excellence for Data Science, Cognitive Systems and BigData
(A joint-venture between IBM Research Zurich and IBM Innovation Center DACH)

2. ● Motivation
● Use Cases
● How Databases scale
● Evolution of Large Scale Data Processing
● Requirements and Ingredients
● Architectural Proposal
Agenda

3. Motivation – the World before 2000
.coms Traditional Enterprises
MySQL, Postgres DB2, Oracle, Teradata

4. Motivation – the World after 2000
.coms Traditional Enterprises
NoSQL DB2, Oracle, Teradata

5. ● Use ALL available data independently weather it is
Use Cases – Basic Idea

6. ● Use ALL available data independently weather it is
● Inside
Use Cases – Basic Idea

7. ● Use ALL available data independently weather it is
● Inside or outside your company
Use Cases – Basic Idea

8. ● Use ALL available data independently weather it is
● Inside or outside your company
● Structured
Use Cases – Basic Idea

9. ● Use ALL available data independently weather it is
● Inside or outside your company
● Structured, semi-structured
Use Cases – Basic Idea

10. ● Use ALL available data independently weather it is
● Inside or outside your company
● Structured, semi-structured, unstructured
Use Cases – Basic Idea

11. ● Use ALL available data independently weather it is
● Inside or outside your company
● Structured, semi-structured, unstructured or binary
Use Cases – Basic Idea

12. ● Use ALL available data independently weather it is
● Inside or outside your company
● Structured, semi-structured, unstructured or binary
● At rest
Use Cases – Basic Idea

13. ● Use ALL available data independently weather it is
● Inside or outside your company
● Structured, semi-structured, unstructured or binary
● At rest or in motion
Use Cases – Basic Idea

14. ● Scale-out because Scale-up not possible
● Why?
● There is a sweet spot (global optimum) for the ideal
node size
● Determines the number of cluster nodes
● Because node price vs node size is not linear
How do Databases Scale?

15. ● To minimize overall cluster price
● Use many node because of rather small node size
● Use commodity hardware
● Fault tolerance
● Won't go into CAP theorem here → Google
● For dynamic workloads and dynamic scale-in/out
→ CLOUD
How do Databases Scale, Conclusion

16. ● Current situation in Enterprises
● BI Tools
● SQL (42%)
● R (33%)
● Python (26%)
● Excel (25%)
● Java, Ruby, C++ (17%)
● SPSS, SAS (9%)
● Current situation in .coms
● Writing MapReduce Jobs
● Using proprietary query languages
● Code everything from scratch
Current situation

17. ● Programing language implementations
● Usage of high level query languages
● Pig
● Jaql
● SQL or SQL like
● BigSQL
● HQL
● CQL
● R push back
● BigR
● Rhadoop
● BigData spread sheets
● BI push back
● SPSS push back
Evolution

18. Pig(Latin) / JAQL

19. BigSQL

20. Big Spreadsheets

21. BigR

22. SPSS

23. SPSS

24. Business Intelligence

25. ● Fault tolerance
● Dynamic and elastic scale-in and out
● Processing data of all types
● Use familiar ways of working with data
Requirements Summary

26. ● NoSQL DB
● Cloud
● Push-back from Business Applications
Ingredients

27. IBM Reference Architecture (current)
Information
Ingestion
Security and Business Continuity Management
Information
Interaction
Analytics Sources
Shared Operational Information
Traditional
Sources
3rd
Party
Applications
Governing Systems
Visualization, Data
Mining &
Exploration
Visualization, Data
Mining &
Exploration
User Reports &
Dashboards
User Reports &
Dashboards
Accelerators &
Application
Frameworks
Accelerators &
Application
Frameworks
User Guided
Applications &
Advanced
Analytics
User Guided
Applications &
Advanced
Analytics
Content
Repository
Content
Repository
Master Data
Hubs
Master Data
Hubs
Reference
Data Hubs
Reference
Data Hubs
Warehouse ZoneWarehouse Zone
Integrated Warehouse &
Marts Zone
(Guided Analytical Area)
Integrated Warehouse &
Marts Zone
(Guided Analytical Area)
Exploration Zone
(Discovery Area)
Exploration Zone
(Discovery Area)
LandingAreaZones
LandingAreaZones

28. IBM Reference Architecture (transition)

29. 15 minutes Discussion
Twitter: @romeokienzler
Email: romeo . Kienzler (at) ch.ibm.com