Big Data LDN 2018: DATAOPS: BRINGING DEVOPS TO DATA INTEGRATION

•

0 likes•501 views

Date: 13th November 2018 Location: Data Ops Theatre Time: 11:50 - 12:20 Speaker: Girish Pancha Organisation: StreamSets About: DataOps borrows concepts from agile development to streamline the process of building, deploying and operating dataflow pipelines at scale. Putting DataOps into action requires not only the right technology, but more broadly a thoughtful approach to align the people and the process behind such an initiative. StreamSets' CEO Girish Pancha will explore the emerging trend of DataOps and discuss: • Principles and benefits of DataOps • Common DataOps use cases • Practical guidelines for putting DataOps into action • How StreamSets can help on a DataOps journey

Data & Analytics

1© StreamSets, Inc. All rights reserved.
Bringing DevOps to Data Integration
Girish Pancha | CEO

2© StreamSets, Inc. All rights reserved. 2© StreamSets, Inc. All rights reserved.
data drift (noun) — unexpected,
unannounced and unending
changes to data structure,
infrastructure, and semantics.
Data Drift is the
silent killer of
Data Analytics

3© StreamSets, Inc. All rights reserved. 3© StreamSets, Inc. All rights reserved.
Data Analytics is being transformed…
REPORTS
DATA MINING
ETL
DATA
INTEGRATION
ON-PREM,
EDW

4© StreamSets, Inc. All rights reserved. 4© StreamSets, Inc. All rights reserved.
…by AI, self-service, edge-to-cloud, need for speed
MACHINE
LEARNING
STREAM
COMPUTE
DATA PREP
CLOUD,
DATA LAKE
DATA MINING
ETL
DATA
INTEGRATION
ON-PREM,
EDW
1010101010
1010101010
REPORTS APPLICATIONSAND

5© StreamSets, Inc. All rights reserved. 5© StreamSets, Inc. All rights reserved.
DATA OPS
This digital transformation requires DataOps
REPORTS APPLICATIONSAND
MACHINE
LEARNING
STREAM
COMPUTE
DATA PREP
CLOUD,
DATA LAKE
DATA MINING
ETL
DATA
INTEGRATION
ON-PREM,
EDW
1010101010
1010101010

6© StreamSets, Inc. All rights reserved. 6© StreamSets, Inc. All rights reserved.
OLD WAY
NEW WAY
FIXED PLATFORM
APPLICATION
DOWNTIME
AD-HOC DATA
PIPELINES
OPERATIONAL
BLINDNESS
ARCHITECTURAL
AGILITY
APPLICATION
EFFICIENCY
ENGINEERING
PRODUCTIVITY
BUSINESS
CONFIDENCE

7© StreamSets, Inc. All rights reserved. 7© StreamSets, Inc. All rights reserved.
AUTOMATION
DataOps applies DevOps practices to data analytics
MONITORINGDRIFT AWARENESS

8© StreamSets, Inc. All rights reserved. 8© StreamSets, Inc. All rights reserved.
DATAOPS
MATURITY
MODEL
TRADITIONAL ANALYTICS
Fixed Platform
Application Downtime
Ad-hoc Data Pipelines
Operational Blindness
00
NOT
AW
ARE

9© StreamSets, Inc. All rights reserved. 9© StreamSets, Inc. All rights reserved.
TRADITIONAL ANALYTICS
Fixed Platform
Application Downtime
Ad-hoc Data Pipelines
Operational Blindness
01
MODERN ANALYTICS
Integration Automation
System MonitoringSYSTEM
AW
ARE
00
NOT
AW
ARE
DATAOPS
MATURITY
MODEL

10© StreamSets, Inc. All rights reserved. 10© StreamSets, Inc. All rights reserved.
TRADITIONAL ANALYTICS
Fixed Platform
Application Downtime
Ad-hoc Data Pipelines
Operational Blindness
DATAOPS
MATURITY
MODEL
02
APPLICATION
AW
ARE
MODERN ANALYTICS
Application Automation
Data Monitoring
01
SYSTEM
AW
ARE
MODERN ANALYTICS
Integration Automation
System Monitoring
00
NOT
AW
ARE

11© StreamSets, Inc. All rights reserved. 11© StreamSets, Inc. All rights reserved.
03
BUSINESSAW
ARE
MODERN ANALYTICS
Entity-Centric Automation
Decision Monitoring
DATAOPS
MATURITY
MODEL
TRADITIONAL ANALYTICS
Fixed Platform
Application Downtime
Ad-hoc Data Pipelines
Operational Blindness
02
APPLICATION
AW
ARE
MODERN ANALYTICS
Application Automation
Data Monitoring
01
SYSTEM
AW
ARE
MODERN ANALYTICS
Integration Automation
System Monitoring
00
NOT
AW
ARE

12© StreamSets, Inc. All rights reserved. 12© StreamSets, Inc. All rights reserved.
StreamSets is where DevOps meets Data Integration
MACHINE
LEARNING
STREAM
COMPUTE
DATA PREP
CLOUD,
DATA LAKE
DATA MINING
ETL
DATA
INTEGRATION
ON-PREM,
EDW
1010101010
1010101010
REPORTS APPLICATIONSAND

13© StreamSets, Inc. All rights reserved. 13© StreamSets, Inc. All rights reserved.
03
BUSINESSAW
ARE
DECOUPLED EVERYTHING, INTRUMENTED EVERYWHERE
DATAOPS
MATURITY
STARTS WITH
SMART PIPELINES
TRADITIONAL ANALYTICS
Fixed Platform
Application Downtime
Ad-hoc Data Pipelines
Operational Blindness
02
APPLICATION
AW
ARE
01
SYSTEM
AW
ARE
00
NOT
AW
ARE
MODERN ANALYTICS
Architectural Agility
MODERN ANALYTICS
Engineering Productivity
Application Efficiency
MODERN ANALYTICS
Business Confidence

15© StreamSets, Inc. All rights reserved. 15© StreamSets, Inc. All rights reserved.
Where does your DataOps journey
begin?
MACHINE LEARNING
AI & DATA SCIENCE
CLOUD DATA
WAREHOUSING
STREAMING
DATA
APPLICATIONS
© StreamSets, Inc. All rights reserved.
Data Analytics is being transformed
by AI, self-service, edge-to-cloud, need for speed

16© StreamSets, Inc. All rights reserved.
Thank you
16© StreamSets, Inc. All rights reserved.
Girish Pancha
e: girish@streamsets.com
t: @girishpancha

More from Matt Stubbs

Date: 14th November 2018 Location: Governance and MDM Theatre Time: 10:30 - 11:00 Speaker: Mike Ferguson Organisation: IBS About: For most organisations today, data complexity has increased rapidly. In the area of operations, we now have cloud and on-premises OLTP systems with customers, partners and suppliers accessing these applications via APIs and mobile apps. In the area of analytics, we now have data warehouse, data marts, big data Hadoop systems, NoSQL databases, streaming data platforms, cloud storage, cloud data warehouses, and IoT-generated data being created at the edge. Also, the number of data sources is exploding as companies ingest more and more external data such as weather and open government data. Silos have also appeared everywhere as business users are buying in self-service data preparation tools without consideration for how these tools integrate with what IT is using to integrate data. Yet new regulations are demanding that we do a better job of governing data, and business executives are demanding more agility to remain competitive in a digital economy. So how can companies remain agile, reduce cost and reduce the time-to-value when data complexity is on the up? In this session, Mike will discuss how companies can create an information supply chain to manufacture business-ready data and analytics to reduce time to value and improve agility while also getting data under control.

Big Data LDN 2018: DATA MANAGEMENT AUTOMATION AND THE INFORMATION SUPPLY CHAI...

Matt Stubbs

Big Data LDN 2018: AI VS. GDPR

Matt Stubbs

Date: 13th November 2018 Location: Governance and MDM Theatre Time: 11:50 - 12:20 Speaker: Mark Pritchard Organisation: Denodo About: Self-service analytics promises to liberate business users to perform analytics without the assistance of IT, and this in turn promises to free IT to focus on enhancing the infrastructure. Join us to learn how data virtualization will allow you to gain real-time access to enterprise-wide data and deliver self-service analytics. We will explore how you can seamlessly unify fragmented data, replace your high-maintenance and high cost data integrations with a single, low-maintenance data virtualization layer; and how you can preserve your data integrity and ensure data lineage is fully traceable.

Big Data LDN 2018: REALISING THE PROMISE OF SELF-SERVICE ANALYTICS WITH DATA ...

Matt Stubbs

Date: 13th November 2018 Location: Governance and MDM Theatre Time: 11:10 - 11:40 Organisation: TIBCO About: The big data phenomenon continues to accelerate, resulting in multiple data lakes at most organisations. However, according to Gartner, “Through 2019, 90% of the information assets from big data analytic efforts will be siloed and unusable across multiple business processes.” Are you ready to unleash this data from these silos and deliver the insights your organisation needs to drive compelling customer experiences, innovative new products and optimized operations? In this session you will learn how to apply data virtualisation to: - Access, transform and deliver data from across your lakes, clouds and other data sources - Empower a range of analytic users and tools with all the data they need - Move rapidly to a modern and flexible data architecture for the long run In addition, you will see a demonstration of data virtualisation in action.

Big Data LDN 2018: TURNING MULTIPLE DATA LAKES INTO A UNIFIED ANALYTIC DATA L...

Matt Stubbs

Big Data LDN 2018: MICROSOFT AZURE AND CLOUDERA – FLEXIBLE CLOUD, WHATEVER TH...

Matt Stubbs

Date: 14th November 2018 Location: Data-Driven Ldn Theatre Time: 12:30 - 13:00 Organisation: Cloudera About: The growth of public cloud is reinforcing the need to think more carefully about taking a consistent approach to data governance as technology teams build out a flexible and agile infrastructure to meet the demands of the business. Join this session to learn more about Cloudera's recommended approach for enterprise-grade security and governance and how to ensure a consistent framework across private, public and on-premises environments.

Big Data LDN 2018: CONSISTENT SECURITY, GOVERNANCE AND FLEXIBILITY FOR ALL WO...

Matt Stubbs

Date: 14th November 2018 Location: Data-Driven Ldn Theatre Time: 11:10 - 11:40 Organisation: Microlise About: Microlise are a leading provider of technology solutions to the transport and logistics industry worldwide. Discover how, with over 400,000 connected assets generating billions of messages a day, Microlise is evolving its platform to bring real-time analytics to its customers to improve safety, security and efficiency outcomes.

Big Data LDN 2018: MICROLISE: USING BIG DATA AND AI IN TRANSPORT AND LOGISTICS

Matt Stubbs

Date: 14th November 2018 Location: Data-Driven Ldn Theatre Time: 10:30 - 11:00 Speaker: Anna Matty Organisation: Experian About: Today there is a widespread focus on the 'how' in relation to problem solving. How can we gain better knowledge of what consumers want, or need? How can we be more efficient, reduce the cost to serve, or grow the lifetime value of a customer? But, how do you move to a place where you are not only solving a problem, you are redesigning the entire strategic potential of that problem? You are being armed with insight on what the problem is. Data and innovation offer huge potential to revolutionise all markets. There is an opportunity to be one step ahead of the need, to redesign journeys and enhance enterprise strategies. To do this you need access to the most advanced analytics but also the best quality, including variations and types of data, and then the technology that can act on this insight. Data science can present a unique opportunity for uncovered growth and accelerate your business through strategic innovation – fast. In this session you will hear more about how today's analytics can move from a single task, to an ongoing strategic opportunity. An opportunity that helps you move at the speed of the market and helps you maximise every opportunity.

Big Data LDN 2018: EXPERIAN: MAXIMISE EVERY OPPORTUNITY IN THE BIG DATA UNIVERSE

Matt Stubbs

Date: 13th November 2018 Location: Data-Driven Ldn Theatre Time: 13:10 - 13:40 Speaker: Brian Goral Organisation: Cloudera About: The field of machine learning (ML) ranges from the very practical and pragmatic to the highly theoretical and abstract. This talk describes several of the challenges facing organisations that want to leverage more of their data through ML, including some examples of the applied algorithms that are already delivering value in business contexts.

Big Data LDN 2018: A LOOK INSIDE APPLIED MACHINE LEARNING

Matt Stubbs

Date: 13th November 2018 Location: Data-Driven Ldn Theatre Time: 12:30 - 13:00 Speaker: Paul Wilkinson, Naveen Gupta Organisation: Cloudera About: Investment banks are faced with some of the toughest regulatory requirements in the world. In a market where data is increasing and changing at extraordinary rates the journey with data governance never ends. In this session, Deutsche Bank will share their journey with big data and explain some of the processes and techniques they have employed to prepare the bank for today’s challenges and tomorrow’s opportunities. Brought to you by Naveen Gupta, VP Software Engineering, Deutsche Bank and Paul Wilkinson, Principal Solutions Architect, Cloudera.

Big Data LDN 2018: DEUTSCHE BANK: THE PATH TO AUTOMATION IN A HIGHLY REGULATE...

Matt Stubbs

Date: 14th November 2018 Location: Self-Service Analytics Theatre Time: 13:50 - 14:20 Speaker: Stephanie McReynolds Organisation: Alation About: Raw data is proliferating at an enormous rate. But so are our derived data assets - hundreds of dashboards, thousands of reports, millions of transformed data sets. Self-service analytics have ensured that this noise is making it increasingly hard to understand and trust data for decision-making. This trust gap is holding your organisation back from business outcomes. European analytics leaders have found a way to close the gap between data and decision-making. From MunichRe to Pfizer and Daimler, analytics teams are adopting data catalogues for thousands of self-service analytics users. Join us in this session to hear how data catalogues that activate data by incorporating machine learning can: • Increase analyst productivity 20-40% • Boost the understanding of the nuances of data and • Establish trust in data-driven decisions with agile stewardship

Big Data LDN 2018: FROM PROLIFERATION TO PRODUCTIVITY: MACHINE LEARNING DATA ...

Matt Stubbs

Date: 13th November 2018 Location: Self-Service Analytics Theatre Time: 15:50 - 16:20 Speaker: Nishanth Kadiyala Organisation: Progress About: The exploding API economy, combined with an advanced analytics market projected to reach $30 billion by 2019, is forcing IT to expose more and more data through APIs. Business analysts, data engineers, and data scientists are still not happy because their needs never really made it into the existing API strategies. This is because most APIs are designed for application integration, but not for the data workers who are looking for APIs that facilitate direct data access to run complex analytics. Data APIs are specifically designed to provide that frictionless data access experience to support analytics across standard interoperable interfaces such as OData (REST) or ODBC/JDBC (SQL). Consider expanding your API strategy to service the developers with open analytics in this $30 billion market.

Big Data LDN 2018: DATA APIS DON’T DISCRIMINATE

Matt Stubbs

Date: 13th November 2018 Location: Self-Service Analytics Theatre Time: 14:30 - 15:00 Speaker: Zaf Khan Organisation: Arcadia Data About: The use of data lakes continue to grow, and a recent survey by Eckerson Group shows that organizations are getting real value from their deployments. However, there’s still a lot of room for improvement when it comes to giving business users access to the wealth of potential insights in the data lake. While the data management aspect has been fairly well understood over the years, the success of business intelligence (BI) and analytics on data lakes lags behind. In fact, organizations often struggle with data lakes because they are only accessible by highly-skilled data scientists and not by business users. But BI tools have been able to access data warehouses for years, so what gives? In this talk, we’ll discuss: • Why traditional BI tools are architected well for data warehouses, but not data lakes. • Why every organization should have two BI standards: one for data warehouses and one for data lakes. • Innovative capabilities provided by BI for data lakes

Big Data LDN 2018: A TALE OF TWO BI STANDARDS: DATA WAREHOUSES AND DATA LAKES

Matt Stubbs

Date: 13th November 2018 Location: Self-Service Analytics Theatre Time: 12:30 - 13:00 Speaker: Joel McKelvey Organisation: Looker About: Companies that use data well are more efficient, effective, and profitable. Unfortunately, most organizations struggle to keep up with the changing supply of data — and the growing business demands for that data. The key is to connect data supply to data users in a way that scales, supports existing workflows, and serves as a foundation for the future. This session will explore how to bring data to users where and when they need it without sacrificing data governance or unified metrics. This session will also present proven ways to build a data foundation for your organisation that can support future changes in both data supply and data demand. Specifically, attendees will discover: • The key considerations to driving the most value from data, including: self-service, governance, custom interfaces, modeling, and connections to existing business systems. • How to provide users access to data in a way that naturally fits in their existing workflows and allows users to take immediate action. • How companies like Deliveroo and King extract critical business insights from growing data and deliver those insights to their business users.

Big Data LDN 2018: FIGHTING DATA CHAOS: CONNECTING USERS TO DATA AT SCALE

Matt Stubbs

Date: 13th November 2018 Location: Self-Service Analytics Theatre Time: 11:50 - 12:20 Speaker: Matt Pepper Organisation: MIcroStretegy About: In this session, Matt Pepper will demonstrate how MicroStrategy enables an Enterprise to provide: • Governed Data Discovery at scale • Freedom for departments to build applications whilst remaining compliant • The capability to link disparate data into a cohesive platform • The right information to the right person, in the right format, at the right time

Big Data LDN 2018: INTELLIGENCE EVERYWHERE – POWER TO THE PEOPLE

Matt Stubbs

Date: 14th November 2018 Location: Data Ops Theatre Time: 14:30 - 15:00 Speaker: Nishanth Kadiyala Organisation: Progress About: Gartner predicts that insights-driven businesses will earn $1.8 trillion annually by 2021. To prepare for this monumental growth, businesses are spending significantly more time and resources to manage and analyse their enterprise data. Progress DataDirect conducts an annual survey to determine the latest trends in accessing and utilising enterprise data. This year’s survey had more than 1,400 respondents. Join Nishanth Kadiyala, Product Evangelist from Progress, as he unveils the 2018 data connectivity and analytics patterns in databases, big data, cloud, REST APIs, OData, data security, open and embedded analytics, hybrid connectivity, and more.

Big Data LDN 2018: 2018 DATA TRENDS: RESULTS FROM FIFTH ANNUAL DATA AND ANALY...

Matt Stubbs

Big Data LDN 2018: AGILE DATA MASTERING: THE RIGHT APPROACH FOR DATAOPS

Matt Stubbs

Date: 14th November 2018 Location: Data Ops Theatre Time: 11:50 - 12:20 Speaker: Marion Azoulai Organisation: TIBCO About: Data science may be “one of the sexiest jobs of the 21st Century,” but it’s likely your most valuable analytics employees are spending too much time on the most mundane tasks: prepping data for analysis. Make it easy to clean and work with data to give time back to your analytics talent so they can focus on answering questions, solving problems, and discovering opportunities to innovate. Join this session to learn practical tips and tricks to significantly reduce the time needed to transform and wrangle data and leave more time for generating insights.

Big Data LDN 2018: TIPS AND TRICKS TO WRANGLE BIG, DIRTY DATA

Matt Stubbs

Date: 14th November 2018 Location: Data Ops Theatre Time: 11:10 - 11:40 About: In this presentation, Pradeep will share examples of organizations that have realized the promise of the Operational Data Warehouse (ODW) to deliver real-time insights to drive business value. Pradeep will share examples that span financial reporting, retail, stock market and clinical trial data. Christophe Daguin will discuss how one of Expandium’s customers, who is a large mobile operator manages their Quality of Service (QoS) for 20 million subscribers who generate 5 billion transactions per day that are managed in their Actian Vector analytic database.

Big Data LDN 2018: DELIVERING ON THE OPERATIONAL DATA WAREHOUSE PROMISE

Matt Stubbs

Date: 13th November 2018 Location: Data Ops Theatre Time: 15:10 - 15:40 Speaker: Steve Keil Organisation: MariaDB About: In an increasingly commodified and digital business landscape, it’s becoming harder to develop and maintain a competitive edge. Loyalty is replaced by price, and humanity by business process abstractions. We’ve been sold the myth that big data and ‘data transformation’ will drive improved decision making – giving us an edge to compete. Yet, the reality is we’ve failed to materialise on this promise, and despite increased data collection, storage, and implementations, we’re largely unable to generate expected benefits. Find out why our strategy and approach has been wrong, and develop a new way forward for data-driven strategy, leadership, and transformation.

Big Data LDN 2018: THE HOLY GRAIL OF ANALYTICS. OUR EPIC FAIL… AND OUR REVOLU...

Matt Stubbs

More from Matt Stubbs (20)