Sneak Peak into Self-Service, Cross-Enterprise, Job Scheduling with CA Workload Automation Advanced Integration for Hadoop

Sneak Peak into Self-Service, Cross-Enterprise, Job
Scheduling with CA Workload Automation Advanced
Integration for Hadoop
Beeshmanth (B) Kotamreddy
DevOps: Continuous Delivery
CA Technologies
Principal Product Manager
DO4T42T
@TwitterHandle
#CAWorld

2 © 2015 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
For Informational Purposes Only
Terms of this Presentation
© 2015 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The presentation provided at CA
World 2015 is intended for information purposes only and does not form any type of warranty. Some of the specific slides with customer
references relate to customer's specific use and experience of CA products and solutions so actual results may vary.
Certain information in this presentation may outline CA’s general product direction. This presentation shall not serve to (i) affect the rights
and/or obligations of CA or its licensees under any existing or future license agreement or services agreement relating to any CA software
product; or (ii) amend any product documentation or specifications for any CA software product. This presentation is based on current
information and resource allocations as of November 18, 2015, and is subject to change or withdrawal by CA at any time without notice. The
development, release and timing of any features or functionality described in this presentation remain at CA’s sole discretion.
Notwithstanding anything in this presentation to the contrary, upon the general availability of any future CA product release referenced in
this presentation, CA may make such release available to new licensees in the form of a regularly scheduled major product release. Such
release may be made available to licensees of the product who are active subscribers to CA maintenance and support, on a when and if-
available basis. The information in this presentation is not deemed to be incorporated into any contract.

Agenda
BIGDATA AND CHANGING CUSTOMER NEEDS
HADOOP
Q & A
BUSINESS CHALLENGES
CA WORKLOAD AUTOMATION ADVANCED INTEGRATION FOR HADOOP
DEMO
1
2
3
4
5
6

Maximize the value of Big Data
with the power of Workload
Automation
 HDFS Operations
 Pig
 Hive
 Sqoop
 Oozie Workflows
Exciting, disruptive & evolving ecosystem
"80% of customer data will be wasted due to immature enterprise data 'value chains.' “
~IDC

What is Big Data?
Datasets whose volume, velocity, variety and complexity exceed ability of commonly
used software tools to capture, process, store, manage, and analyze them.
Information Sources
MobileTransactional
Data
SearchTextsCRM, SCM,
ERP
$ € ¥
ImagesEmail Social
Media
IT Ops AudioVideo
Velocity Volume
Variety Complexity
Big
Data

Enterprises across all industries use Big Data
Enterprises require new capabilities around processing large amounts of data in a variety of different formats
 Fraud Prevention
 Trading Risks
 Customer Risk Assessment
 Call Detail Records
 Real-time bandwidth allocations
 Life time value and promotions
RETAILERS
 Customer Analytics
 Brand Sentiment Analytics
 Promotion Planning
TELCO CARRIERSBANKS
 Genomic Analysis
 Medical trial Analysis
 Hospital Diagnostics Analytics
 IOT/Smart Meter Analytics
 Energy trading and pricing risk
analytics
GOVERNMENT/
PUBLIC SECTOR
 Crime Intelligence and Prevention
 Fraud Prevention
UTILITY PROVIDERSHEALTH CARE PROVIDERS
$

What is Hadoop ?
Hadoop is… open-source software designed for
High Scalability, Fault Tolerant and Highly Distributed
Key elements:
1. Distributed processing of Big Data (e.g. MapReduce)
2. Distributed storage (Hadoop Distributed File System or HDFS)
HDFS
(Distributed Reliable Storage)
MapReduce
(Resource Management
& Data Processing)
HDFS
(Distributed Reliable Storage)
YARN
(Resource Management)
MapReduce
(Dist. Programming)
Hadoop 1.0 Hadoop 2.0
Spark
(In Memory)
HBase
(NoSQLstore)
Hive
(Query)
Pig
(Scripting)
Oozie
(Workflow)

MapReduce – Core Hadoop1
 Hadoop’s MapReduce framework involves two phases:
1. Map Phase: Distributes dataset among multiple servers and
operates on the data locally.
2. Reduce Phase: Recombines the partial results.
A distributed computing Framework

Job-1
Job-2
Job-3
Job-4
Job-5
HDFS
Data Nodes
Task Trackers
Hadoop Distributed File System (HDFS)
Self-healing, high bandwidth Clustered Storage
• Name Node - One of the
Core Hadoop services that
maintains the namespace –
knows where data is and
manages blocks on data nodes
• Data Node - serves that
actual store the data in their
local disks.
• Secondary Name Node -
performs periodic checkpoint
of primary name node to
serve as a backup in case of
failure
Slave Nodes
2
4
5
1
2
5
1
3
4
2
3
5
1
3
4
HDFS breaks incoming files into blocks and stores them
redundantly across the cluster.
Name Node (primary)
Name Node (secondary)
Master Node
Periodic Checkpoint
2

SO, YOU HAVE DATA
And you want it to help you better
understand your business, customers and
marketplace.
THAT’S WHY YOU USE HADOOP
But, extracting data insights may require you
to interface with systems outside of Hadoop.
And that isn’t always easy…

Enterprises typically have multiple scheduling engines
to manage end-to-end business processes
Companies typically interface with multiple systems such as
ERP (SAP/ Oracle etc.), databases, reporting tools, point of sale systems,
social media files etc., in addition to Hadoop
As a result, Enterprises use multiple tools to manage
their workload automation needs
Visualizing the end-to-end business workflows,
& managing dependencies across Hadoop
and non-Hadoop systems might not always be easy

Challenges
Multiple Schedulers needed to run traditional jobs and Hadoop jobs
 Hadoop jobs may not integrate into existing Workflows
Heterogeneous Environment and Tools
 Team productivity, experience, knowledge
 Placing workloads - “right place , right time”
Slow responsiveness to the business
 No central location to monitor end-end workflows

Drag and drop Hadoop jobs
into existing workflows.
Monitor traditional and
Hadoop jobs from a single
console.
Detect problems early
and resolve them quickly.
Set up automatic alerts
for critical events.
Unified visibility into your
heterogeneous and Hadoop
environments
Improved performance
and uptime through proactive
monitoring and alerts
Lower costs by eliminating the complexity of disconnected monitoring tools
BIG DATA MADE EASY with
CA Workload Automation Advanced Integration for Hadoop

Extract and
Move Input Files
Transform and
Process Input
Run Specialized
ERP jobs to
extract data
Ingest Data in
DW
Batch Ingestion
into Hadoop and
Batch Analytics
Load results into
NoSQL for
interactive queries
FTP JOB
Receive POS data
DATABASE JOB
Extract Pricing
data when
database trigger
occurs
INFORMATICA
JOB
Parse POS Files
INFORMATICA
JOB
Run SQL query to
match customer
data in SAP
SAP JOB
Run SAP Extract
Job to extract
inventory data
TERADATA JOB
Run ETL to merge
input files into
DW
SQOOP JOB
Run Sqoop jobs
to copy data into
Hadoop cluster
PIG JOB
Run pig jobs for
operational
analytics
NOSQL JOB
Interactive Search
job to run
dynamic
promotion
WorkflowWorkloadsUsecase
Extract POS,
Inventory, Price Data
Mine Customer Information and
Inventory Information from ERP
Load Data into NoSQL
and render dynamic
discounting on-demand
Perform Batch aggregation and
Machine learning for Promotion
Analytics
CA Workload Automation extends scheduling for Big Data
Retail Customer Analytics in the Application Economy

Timeline as of November, 2015
 Reduce Silos and increase efficiency with
centralized scheduling of Hadoop & Non-
Hadoop workloads through supporting
 HDFS Operations
 Pig
 Hive
 Sqoop
 Oozie Workflows
 Integrated Hadoop & Heterogeneous
environments through supporting
Cloudera and HortonWorks distributions
for Hadoop
Marquee
Features/
Business
Value
Planned Under Consideration
DELIVERED PLANNED UNDER CONSIDERATION
Product/
Releases
CA WLA Advanced Integration for Hadoop R12.0
Currently GA
Roadmap: Product Name
 Unify additional Hadoop environments
through supporting other distributions
such as IBM Big Insights, MapR, Greenplum
 Extended security through native support
for Kerberos authentication.
 Centralized Scheduling for evolving
Hadoop ecosystem with integrations to
Spark, Hbase, Flume, Kafka, Tajo etc.
CA WLA Advanced Integration for Hadoop R12.X CA WLA Advanced Integration for Hadoop RX.X

The Evolving Hadoop Ecosystem
Components Description
mahout R
Data Mining/machine learning
tools used against Hadoop data to
detect patterns and trends
Pig
Scripting language for analyzing
large datasets. Compiles to
MapReduce jobs
MapReduce YARN
Programming model for
processing large data sets. YARN
performs overall resource mgmt
Oozie
A workflow scheduler tool to
manage Hadoop MapReduce jobs
Sqoop Hive
Enable SQL for Hadoop data:
Sqoop - Data transfer between
Hadoop and structured datastores.
HIVE - data warehouse for Hadoop.
Drill - open source, low latency SQL
query engine for Hadoop and NoSQL.
Drill
ZooKeeper
Coordination of config. data,
naming and synchronization of
Hadoop projects
Components Description
BigTop
Packaging services for Hadoop
projects to ease testing and
deployment
HBase
A non-relational, distributed
database that runs on top of
HDFS
Thrift / AVRO
Schema-based data serialization
system using RPC calls
Solr
hutch
Indexing and search tools for
data stored in HDFS for Hadoop
Elasticsearch
Kafka / Flume
Collect, aggregate, and move
streaming data from multiple
sources into Hadoop
Spark
AppDev tool for Hadoop apps
combining batch, streaming, and
interactive analytics
Anbari Chukwa
Monitoring & Management of
Hadoop clusters and nodes

Influencing Our Roadmap
Winning with CA
 Submit your ideas on
communities.ca.com
 Vote & comment on ideas
that are important to you
 CA Product Management
reviews ideas and updates
status as they move
through the lifecycle
 “Currently Planned” idea
status indicates inclusion
in Agile Backlog or
Product Roadmap
Take the opportunity to influence our product development.
Help ensure that we deliver is what you need and want.
Agile Development
CA Communities Ideation
 Register to participate in:
– Live Demos/End-of-Sprint
Reviews
– Private - Members Only -
Online Community
– Pre-Release Onsite Testing
and Support (Beta)
– Upgrade Support from
SWAT Team
 How to register:
https://validate.ca.com
Customer Validation

DEMO
CA Workload Automation Advanced Integration for
Hadoop

Q & A

For More Information
To learn more, please visit:
http://cainc.to/Nv2VOe
CA World ’15

Sneak Peak into Self-Service, Cross-Enterprise, Job Scheduling with CA Workload Automation Advanced Integration for Hadoop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (7)

Similar to Sneak Peak into Self-Service, Cross-Enterprise, Job Scheduling with CA Workload Automation Advanced Integration for Hadoop

Similar to Sneak Peak into Self-Service, Cross-Enterprise, Job Scheduling with CA Workload Automation Advanced Integration for Hadoop (20)

More from CA Technologies

More from CA Technologies (20)

Recently uploaded

Recently uploaded (20)

Sneak Peak into Self-Service, Cross-Enterprise, Job Scheduling with CA Workload Automation Advanced Integration for Hadoop