Enterprise Architecture in the Era of Big Data and Quantum Computing

©2014 Knowledgent Group Inc. All Rights Reserved
Pragmatic Enterprise Architecture
In the Era of Big Data and Quantum Computing

Agenda
• Bio / Who is Jim Luisi?
• WIIFM / What’s in it for me?
• Enterprise Architecture … Past… Future
• What is Big Data?
• Why has Big Data recently become more compelling?
• What architectural disciplines relate to Big Data?
• EA View of Big Data
• IT Challenge
• Big Data TPM… APM… DB Architectures… Quantum Computing
• Big Data Use Case Families
• Hardware Architecture / QC
• Software Architecture / QC
• Hypernumbers and QC in Perspective
• Big Data Technology Combination of Accelerators
• Big Data IA Foundation
• Big Data Metadata… MDM… RDM
• Big Data Ecosystem
• Questions

Bio
• 30 years
• Business
- Business owner
- Big Data in advertising industry
- Worked on business side in Wall St
• IT
- VLDB with government before Big Data
- Specialized compartments in NP-complete problem space
- Hadoop Big Data architecture with government
- Many areas of artificial intelligence are in the Big Data space
• Philosophy
- holistic view
- Enterprise Architecture perspective
• Author
- Artificial intelligence
- Enterprise architecture
Who is Jim Luisi?

• 99.9% of the Big Data mystery will be removed
• You’ll be able to engage in the conversation
• You can help transform your organization
WIIFM
What’s in it for me?

• EA was a corporate committee of generalists
• EA focused on a few parts
- Capability that answered basic questions crossing
– Applications
– Tools
– Organizational Units
EA Past
Enterprise Architecture

• EA will render a holistic view of automation
• Standards and governance will manage complexity
EA Future

• Subject matter experts will make the technology ‘D’
• SMEs will collaborate with other SMEs
• SMEs will minimize complexity
EA Future (cont’d)

• From a storage perspective
- Lots of storage
• From a distribution perspective
- Data stored in lots of places globally
• From a database design perspective
- Lots of rows
- Lots of tables
- Lots of columns
• From an algorithmic or mathematical perspective
- Lots of variables
- Lots of combinations
- Lots of permutations
- Summed up as one optimal answer among a large number of possibilities
What is Big Data?

Why Has Big Data become more compelling?
• Big Data has become
- More advanced to address a wider array of business challenges
- More friendly to developers and end users
- A low cost alternative to traditional transaction databases (OldSQL)
- A significant competitive differentiator in many industries
• Big Data can now address business use cases involving:
- Many enterprise grade requirements
- Vast volumes of structured, unstructured, and semi-structured data
- High speed and complex data transformation rates
- High velocity ingestion rates
- Large numbers of concurrent users
- Competitive hyper-real-time requirements (e.g., milliseconds, nanoseconds)

What architectural disciplines relate to Big Data?
Business
Architecture
Business Continuity
Marketing Architecture
Operations
Architecture
Disaster Recovery Architecture
Storage Architecture
Infrastructure Architecture
Network SecurityApplication
Architecture
Reporting Architecture
Integration Architecture
Application Portfolio Management
Technology Portfolio Management
Application Security
Information
Architecture
Data Architecture
Data Governance Architecture
Master Data Management (MDM &
RDM)
Metadata Management
Data Security

• Variety of Big Data
- database technologies
- hardware technologies
- applications
- reporting tools
• Variety of sources
- Open source providers
- Licensed providers
- In-house developed
- Custom built providers
EA View of Big Data

• All requiring a solid foundation
• None of which are free
• Adding to the complexity of your IT landscape
IT Challenge

• Competing open source foundations
- Apache Software Foundation (ASF)
- Free Software Foundation (FSF)
- Many individual companies
• Competing open source providers of Hadoop
- Hortonworks,
- Cloudera
- MapR
- IBM
- Microsoft
- Intel (IDH) now being sunset
Big Data TPM
Technology Portfolio Management

TPM / Competing Big Data Frameworks
• and more…

• Applications use assorted Big Data technologies
- Open source
- Licensed
- Traditional databases (OldSQL)
- Traditional Big Data technologies
- Non-traditional (NoSQL)
- OLTP (New SQL) fully ACID
- Proprietary breed
- Quantum computing
Big Data APM
Application Portfolio Management
• Approved
– Data sources
– Use cases
– Technologies
– Time period
• Approved
– Data sources
– Technologies
– Time period
• Approved
– Data sources
– Data target
– Time period

Parent Child Data
A
B C
D E X
Hierarchical
Parent First Set A Last Set A Data
A
B1 B2 B3
C
X
Network Linked List
Parent Next Set A Prior Set A First Set C Last Set C Data Data
Associative
Record A
Key Value 'Blue Eyes' 'Height '6''
Pointers Addr2 Addr3
Addr4 Addr4
Addr7 Addr10
Addr10 Addr14
Inverted List
Data
Addr1 Data
Addr2 Data
Addr3 Data
Addr4 Data
Addr5 Data
Addr6 Data
Data
Loan # Date Type Amount
PK
Data C1 C2 C3 C4
Loan # Payment # Due Date Due Amount
Data C1 P1 P2 P3
Loan
Payment Coupon
PK
Relational
Header
Row3
Row4 cntd
Db Page (4K)
Footer
Row1
Row2
Row3 cntd Row4
Free
Space
Underlying Storage Structure
Just as there are DB Architectures of OldSQL

There are database architectures of
TPM / Traditional Big Data Technologies

Microsoft HDInsight Intel Distribution (IDH)
And there are database architectures of
TPM / Non- Traditional Big Data Technologies

TPM / Newer SQL Types
NoSQL is a new way of
thinking about databases,
founded on the belief that a
relational database model
may not be the best solution
for all situations.
NewSQL is a class of
modern relational
database management
systems that seek to
provide the same
scalable performance of
NoSQL systems for online
transaction processing
workloads while still
maintaining the ACID
guarantees of a traditional
database system.
NoSQL NewSQL

What does ACID mean?
• Atomicity
- refers to all or nothing for a logical unit of work
• Consistency
- refers to adherence of data integrity rules that are enforced by the DB
• Isolation
- refers to the need to enforce a sequence of transactions when updating a
database (e.g., two purchasers both trying to purchase the last instance of an
item)
• Durability
- refers to safeguarding information once a commit has been performed to
declare successful completion of a transaction.

TPM / Proprietary Big Data Product Names
• US Government
- Hypernumbers
• Major Financial Conglomerate
- Hypercube
• High-tech Companies
- Named for each client
- Hypercompression
- LNB
- DBX / DB accelerator

TPM / Quantum Computing
• Probabilistic computing
- confidence levels
- two competing quantum computing architectures
• Gate model (aka quantum circuit)
- Shor’s algorithm (code breaking algorithm)
- present implementations have few qubits
- qubit and gate growth is linear
- prone to decoherence (quantum physics wave function collapse)
- complete error correction theory
• Adiabatic quantum computing (AQC)
- discrete combinatorial optimization problems (NP-complete)
- present implementations are 512 qubits
- qubit growth is geometric doubling every 24 months
- not prone to decoherence
- lacks complete error correction theory

What are some Big Data Use Case Families?
• Document / content management
• Online transaction processing (OLTP)
• Data warehousing
• Real-time analytics
• Batch analytics
• Geographic information systems (GIS)
• Search
• Predictive analytics
• Deterministic algorithms
• Probabilistic

Document / Content Management Use Cases
IT Documents
 presentations
 word documents
 spreadsheets
 spreadsheet applications
 MS Access applications
 standards documents
 company policies
 architectural frameworks
Candidate Requirement Types
Maximum document size
Maximum ingestion rates
Maximum access rates
Maximum and mean access speed
Maximum number of documents
Maximum total storage
Maximum concurrent users
Supported data types
Maximum number of indexes
Global accessibility
Multi-data center
Fault tolerance
Developer friendly
Business Documents
 loan applications and documents
 mortgage applications and
documents
 insurance applications and
documents
 new account forms
Customer Documents
 diplomas
 customer in
 birth certificates
 insurance policies
 records for tax preparation
MongoDB
 16 MB limit per
document
 Developer friendly
Cassandra
 2 GB limit per document*
 Few keys
Hadoop HDFS
 No practical limit per
document size
 Slow access
Basho Riak
 2 GB limit per document
 Fault tolerant
Couchbase
 20 MB limit per
document
MarkLogic
 512 MB limit per
document
 High speed access
Candidate Solutions

Online Transaction Processing Use Cases
Candidate Solutions
E-Commerce
 Global web-based transactions
 Global inventory
 Global shipping
Marketing
 RFID supply chain
 Opportunity based marketing
 Google glasses applications
Consumer Products & Services
 In-home medical care
 word documents
Governmental Capabilities
 Military logistics
 Homeland security
Financial Industry
 Global customer exposure
 Operational risk
Peak transactions per second
Maximum length of transaction
System availability
System security
High volume transaction access paths
Number of concurrent connections and sessions
Internationalization (e.g., Unicode)
Data volume requiring compression
Geospatial index support
Full text search
Index support
Sharding support
Maximum value size
Operating system
Minimum memory requirements
Real time analytics
VoltDB
 High speed in-memory
 ACID
SQLFire
 ACID
NuoDB
 ACID
Google Spanner
 Successor to BigTable
 ACID
Clustrix
 Shared-nothing
architecture
Akiban
 ACID

Data Warehousing Use Cases
Candidate
Solutions
Financial Industry
 Underwriting
Marketing
 Customer analytics
 M&A decision making
 Divestiture decision making
 Campaign management
Science Based Industry
 Pharmaceutical development
 Genetics research
 Materiel management
 Intelligence capabilities
 Human disease management
 Food supply analysis
High data ingestion rates
Large data persistence layer
Large number of concurrent users
Drill downs
Comprehensive index support
Sharding support
Robust SQL interface
Backup and restorability
Disaster recoverability
Commodity hardware
Staffing and skill availability
Trainability
Affordability
Vertica
 High speed DB layer
 High ingestion rate
 High concurrent users
 Real-time analytics
Teradata
 Widest array of connectors
Hadoop
 High ingestion capacity
 Not for real-time analytics
Greenplum
SAP Hana

Real Time Analytics Use Cases
Candidate Solutions
 Investment risk
 Operational risk
 Financial risk
 Market risk
 Credit risk
 Intelligence capabilities
 Human disease management
Marketing
 Opportunity based marketing
 Dynamic web advertising
 Regulatory exception reporting
 Operational performance
 Trading analytics
 Algorithmic trading
 Real time valuations
Financial Industry
Large data persistence layer
Real-time analytics
Large number of concurrent users
Drill downs
Comprehensive index support
Sharding support
Robust SQL interface
Backup and restorability
Disaster recoverability
Commodity hardware
Staffing and skill availability
Trainability
Affordability
Vertica
Greenplum
SAP Hana
Teradata
 Widest array of
connectors

Batch Analytics Use Cases
Candidate Solutions
Large data volumes
Validation ability
Traceability
Integratability
Maintainability
Affordability
Financial Industry
 Financial crime
 Anti-money laundering
 Sanctions
 FATCA
 Insurance fraud detection
 Back testing
 Credit risk
 Portfolio valuation
Hadoop HDFS
 MapReduce
 Large data volumes
Hadoop HBase
 MapReduce
(one HBase table as input)
Engineering
 Equipment failure forecasting
Commerce
 Collusion forecasting
 Legislative forecasting
 Regulatory forecasting
Marketing
Government al Capabilities
 Terrorist activity forecasting
 Terrorism event forecasting

Geographic Information System Use Cases
Candidate Solutions
User friendly
ACID compliant
Flexibility
Query able
Full spatial function support
Address Geocoding
 Warrant serving
 Emergency services
 Crime analysis
 Public health analysis
Linear Metric Event Modeling
 Road maintenance activities
 Roadway projects
 Traffic & safety analysis
Cartography
 Hazardous materials tracking
 Taxable asset tracking
Routing
 Evacuation planning
 Towing & plowing
 Refuse removal
 Emergency services
Topological
 Elevation data
 Orthophotography
 Hydrography
Neo4j
 Fully ACID
PostGIS
 Open source
 Geographic support
 Built on PostgreSQL
Oracle Spatial
 Spatial support in Oracle
GeoTime
 Temporal 3D visual analysis

Search Use Cases
Candidate SolutionsCandidate Requirement Types
Usability
Flexibilty
Maintainability
Search as a Service (Saas)
 Website search
Search Enabled Applications
 External data source identification
E-discovery
 Legal holds
 Investigations
Data Landscape Mapping
 Locating data cross data center
Lucidworks
 Built on Solr
 GUI for common use cases
Solr
 Flexibility
Splunk
 Machine generated output

Predictive Analytics Use Cases
Candidate Solutions
Financial Industry
 Capital markets fraud detection
 Wholesale banking fraud detection
 Retail banking fraud detection
 Insurance fraud detection
 Market risk forecasting
 Market opportunity forecasting
 Operational defect forecasting
Engineering
 Equipment failure forecasting
Commerce
 Collusion forecasting
 Regulatory shift forecasting
Marketing
 Customer LTV scoring
 Customer defection score
 Terrorist activity forecasting
 Terrorism event forecasting
Large learning sets
Supports complex models
Rapidly changing learning sets
High volume for operational deployment
Validation ability
Traceability
Integratability
Maintainability
Affordability
Fair Isaac HNC
 Large feature set
 Extensive professional
services
 High speed
 High learning rate
 Real-time deployment
SAS CEP & Statistical
Packages
 High speed
 Supports many source types
 Comprehensive feature set
Ward Systems
 Large feature set
 Extensive professional
services
 High speed
 High learning rate
 Real-time deployment

Particular Deterministic Algorithmic Use Cases
Candidate Solutions
Any Industry
 Matrix-vector multiplication
 Relational algebra
 Computing Selections
 Computing Projections
 Union, Intersection, Difference
 Grouping and aggregation
 Reducer size and replication rate
 Similarity joins
 Graph modeling
Large data volumes
Validation ability
Traceability
Integratability
Maintainability
Affordability
Hadoop HDFS
 MapReduce
Hadoop HBase
 MapReduce
IBM Netezza
 Hardware based algorithms

NP Deterministic Algorithmic Use Cases
Candidate Solutions
(Proprietary)
NP-complete problems
- reasonably testable
Massive number of tables involved in joins
Massive number of rows involved in joins
Large number of calculations
High permutation count
High combination count
Massively distributed
Super large numbers
Real-time
Hyper-numbers
Hyper-cubes
Hyper-compression
DBX
LNB
NP-hard problems
- proof is not reasonably testable

What is NP?
• NP stands for ‘non-deterministic polynomial time’
• Conventional computers can take billions or trillions of years
• NP-complete and NP-hard refers only to the ‘check ability’ of a solution in
time
• NP-complete can be checked in a reasonable length of time
• NP-hard cannot be checked in a reasonable length of time

Probabilistic Use Cases
• Cryptography (aka code breaking)
• Prime number generation
• Traveling salesmen problem
• Labeling images and objects within images
• NLP meaning extraction
• Correlations among Big Data
- genetic code correlations
• Testing a scientific hypothesis
• Machine learning for problem solving
- self-programming

Hardware architecture / AQC
• Commercially available platforms
- 128 qubit Rainier-4 (D-Wave One)
- 512 qubit Vesuvius 3 (D-Wave Two)
- 10’ black cube
- electromagnetically shielded
- digital optical cables
- closed liquid helium cooling system (20mK is 100 times colder than interstellar
space 2.75K)
- cylindrical magnetic shields <1 nanoTesla (nT)
• Each qubit has four wave function values
- ‘-1-1’, ‘+1+1’, ‘-1+1’, ‘+1-1’
• Each qubit is surrounded by switches
- over 180 Josephson junctions per qubit in 3-D space

Software Architecture / AQC
full disjunctive normal form

AQC Software - Bottom Up
• Programming the hardware
- is all about managing the switches
- to represent a Boolean expression that is to be optimized
- to allow the expression to be preloaded with data values
- to allow the variables to be set to an unknown state
- the slowest step is the annealing process
- qubit values are inspected
- solution is tested on a conventional computer
- calculation is repeated multiple times
• In quantum terms you create a mathematical formula to represent the
energy state of the system in the form of a Boolean SAT problem
• SAT were the first documented example of an NP-complete problem
where no know algorithms can solve them in a reasonable length of time
using conventional computers*

AQC Software - Bottom Up (cont’d)
• System Application Program Interface
- communicates directly with the hardware
- requires expertise in quantum mechanics mathematics
- is necessary to program only when
– developing new functions
– exploring quantum physics
– conducting QC experiments
• Complier
- does not convert a higher level language into machine language
- requires no knowledge of
– QC physics
– QC hardware
- it is layer that allows the programmer to focus on
– bit strings and Boolean mathematics

AQC Software - Top Layers
• Client libraries
- conventional programming languages
• Frameworks
- wrapped functions (aka toolkit) bundled for reuse
– supervised binary classification
– supervised multiple label assignment
– unsupervised feature learning
• Applications (top)
- this layer interacts with the end user GUI
- the GUI itself is on a conventional computer
– directly outside the cube
– on a conventional network or Intranet or
– on the Internet anywhere with access

D-Wave Customers
US Government
D-Wave’s QC is a controversial science
scale of quantum entanglement

Hypernumbers and QC
in Perspective
• AQC computing today
- Effective probabilistic approach
- Just starting to get traction
• Hypernumbers
- Highly effective deterministic approach
- Hypernumber architecture
– Deeper into the polynomial problem space
• Promise of QC
- Potentially deeper into polynomial timespace

Big Data Technology is a Combo of Accelerators
• Reduced code set
- Eliminating large amounts of DBMS code
- Eliminating large amounts of OLTP code
• Distributed processing
- Parallel processing
- Loosely or tightly coupled
• Compression
- Data encoding
- Least number of bit data functions
• Proprietary hardware
- Performing algorithms at the data persistence layer
- Massively parallel platforms, networks, etc.
- Quantum computer platforms

Big Data IA Foundation
• Information architecture
- Logical data architecture
- Physical data architecture
• Master data management
- Shared master data
- Reference data management
– Code tables
– Shared files
• Metadata management
- Business data glossary
- SDLC metadata
- Big Data metadata ecosystem

Big Data Metadata Ecosystem

Integrated MDM RDM Architecture
• RDM
- Centrally maintained for the enterprise
- Passed to application databases
- Managed in HBase for reference data lookups
• MDM
- Processed prior to Big Data deployment
– Landed
– Profiled
– Cleansed
– Standardized
– Integrated

Big Data Ecosystem Without EA
•Uncoordinated
•Inconsistent
•Complex

EA Driven Big Data Ecosystem

Questions?
• Pragmatic Enterprise Architecture: Strategies to Transform Information
Systems in the Era of Big Data and Quantum Computing
• James V Luisi / 732-740-2274
• James.Luisi@knowledgent.com
• Connect on Linked-in for new books and updates
• Available at Amazon.com

Enterprise Architecture in the Era of Big Data and Quantum Computing

More Related Content

What's hot

Viewers also liked

Similar to Enterprise Architecture in the Era of Big Data and Quantum Computing

Recently uploaded

Enterprise Architecture in the Era of Big Data and Quantum Computing