Managing Successful Data Projects: Technology Selection and Team Building

•

2 likes•2,589 views

Recent years have seen dramatic advancements in the technologies available for managing and processing data. While these technologies provide powerful tools to build data applications, they also require new skills. Ted Malaska and Jonathan Seidman explain how to evaluate these new technologies and build teams to effectively leverage these technologies and achieve ROI with your data initiatives.

Managing Successful
Data Projects:
Technology Selection and
Team Building
Strata Data Conference, San Jose 2018
Ted Malaska | @ted_malaska
Jonathan Seidman | @jseidman

About the presenters
▪Technical Group Architect at Blizzard Entertainment
- Cloud, Build, Deployment, Data
▪Principal Solutions Architect at Cloudera
▪Big Data Architect at FINRA
▪Contributor to Apache HDFS, HBase, Flume, Avro, Pig, Spark, YARN,
Sqoop, Kudu, Kafka, …
▪Co-Author of O’Reilly’s Hadoop Application Architectures
▪O’Reilly Online Trainer Course Creator
▪Adviser to the Board of MetiStream
▪Video Gamer
Ted Malaska

About the presenters
▪Software Engineer at Cloudera
▪Co-Author of O’Reilly’s Hadoop Application
Architectures
▪Previously Technical Lead on the big data team at
Orbitz, co-founder of the Chicago Hadoop User Group
and Chicago Big Data
Jonathan Seidman

Ted Malaska & Jonathan Seidman
Foundations
forArchitecting
Data Solutions
MANAGING SUCCESSFUL DATA PROJECTS

Before we get started…
§ A story of two paths:
- Vendor
- Customer

There’s a lot on the line
§ Limited amount of time to make decisions
§ Time and resource cost money
§ Don’t want to paint yourself into a corner
§ Everyday something new comes up
§ Mistakes can be career/project damaging

Tricks to Evaluate Hype and Trends
Google Trends

Tricks to Evaluate Hype and Trends
Github activity

Tricks to Evaluate Hype and Trends
Jira Activity

Tricks to Evaluate Hype and Trends
Meetup.com

Build or buy? It depends
§ Internal culture
§ Technical religions
§ Skill sets
§ Opportunity for upside
- Incremental
- Revolutionary
- Control
§ Effectiveness of a vender

Consider buy in
§ Internal buy in
§ Allow for fail fast and restarting

How to stay focused
§ Where is the Business Value
§ All things are possible but not all things are easy
§ Stay dumb
§ You must hate the technology before you select it
§ Treat a vender as a vender not a friend
§ There must be a desire

Reducing risk
§ Interface design
§ Fail fast
§ Cloud based provisioning
§ Containers
§ Identify weak points with a passion

Instead, build well rounded teams
Sysadmins Developers Analysts Data Scientists
Other roles:
Data Protection Officer Network/Systems EngineersProduct Managers

How to find people?
Start with people you already have, but make sure you invest in
training…
§Linux, network, DBAs –> sysadmins
§Developers –> developers
- Easy if you’re at a company like Orbitz, otherwise maybe not so much.
§Analysts –> analysts
§It’s not an easy path though.
- Set goals instead of micro-managing development.
- Be prepared to iterate, don’t be afraid to fail.

Also don’t forget other teams
Communication is key
DBAs Other Project Teams

Think beyond just skills
§ Also look for complementary personalities.
§ And avoid toxic personalities.
- But what if they’re really talented?
- See above.

Customer Engagement
§ Your teams should work closely with your customers, whether they’re external or internal.

Thank you!
Ted Malaska | @ted_malaska
Jonathan Seidman | @jseidman

Federal organizations increasingly are focused on creating environments that enable more data-driven decisions. Yet ensuring that all data is considered and is current, complete, and accurate is a tall order for most. To make data analytics meaningful to support real-world transformation, agency staff need business tools that provide user-friendly dashboards, on-demand reporting, and methods to manage efficiently the rise of voluminous and varied data sets and types commonly associated with big data. In most cases, existing systems are insufficient to support these requirements. Enter the enterprise data hub (EDH), a software architecture specifically designed to be a unified platform that can economically store unlimited data and enable diverse access to it at scale. Plan to attend this discussion to understand the key considerations to making an EDH the architectural center of your agency’s modern data strategy.

Big data journey to the cloud maz chaudhri 5.30.18

Cloudera, Inc.

For most analysts, the pace of analytics and data science can be frustrating. The common waterfall approach works well for the fixed reports, but it can be a lengthy process to request additional data sets, create new reports, or serve new use cases. So it’s no surprise that organizations are looking to shift towards a self-service model, empowering business users to discover and iterate quickly. However, it’s not just about opening up this access, but also ensuring the results are accurate and trusted. When there are petabytes of data, how does a user know which tables to use and which are most relevant? How do you strike the balance between discovery and agility, while still meeting enterprise governance standards to truly get more value from your data? During this webinar, you’ll learn how to empower end-users to make self-service BI a reality within your organization while fostering governance collaboration between all data stakeholders. We’ll discuss and demo: Strategies of consolidating data across silos for fast, flexible access Enabling easy discovery and exploration, including understanding which data to trust and where to start New capabilities for intelligent query assistance as well as immediate performance optimizations and recommendations as-you-go Collaboration and access outside of just SQL for data science and beyond In addition, we will walk through best practices and considerations when developing your organizational strategy around self-service analytics, and highlight several real-world success stories from a wide range of industries. 3 things to learn: Strategies of consolidating data across silos for fast, flexible access Enabling easy discovery and exploration, including understanding which data to trust and where to start New capabilities for intelligent query assistance as well as immediate performance optimizations and recommendations as-you-go

Big data journey to the cloud rohit pujari 5.30.18

Cloudera, Inc.

PaaS or Fail: Rule the Cloud with Altus

Cloudera, Inc.

Preparing data for analysis and insights is the foundation of any data-driven exercise. Moving workloads to a PaaS, be it data engineering, analytic database, or data science requires a two step leap of faith - in trusting the public cloud, and then your PaaS vendor. In this webinar we will discuss the architecture of a PaaS solution for data management and understand the nitty gritty details of what exactly this involves with the following: An exploration of the architecture of Cloudera Altus PaaS - the industry’s first multi-function, multi-cloud data and analytic platform-as-a-service A dive into use cases and a demo of Altus The synergy between AWS and Altus to help you securely standardize on a combination of public cloud and data management 3 things to learn: An exploration of the architecture of Cloudera Altus PaaS - the industry’s first multi-function, multi-cloud data and analytic platform-as-a-service A dive into use cases and a demo of Altus The synergy between AWS and Altus to help you securely standardize on a combination of public cloud and data management

How Data Drives Business at Choice Hotels

Cloudera, Inc.

3 Things to Learn: -How data is driving digital transformation to help businesses innovate rapidly -How Choice Hotels (one of largest hoteliers) is using Cloudera Enterprise to gain meaningful insights that drive their business -How Choice Hotels has transformed business through innovative use of Apache Hadoop, Cloudera Enterprise, and deployment in the cloud — from developing customer experiences to meeting IT compliance requirements

How to Build Continuous Ingestion for the Internet of Things

Cloudera, Inc.

The Internet of Things is moving into the mainstream and this new world of data-driven products is transforming a vast number of industry sectors and technologies. However, IoT creates a new challenge: how to build and operationalize continual data ingestion from such a wide and ever-changing array of endpoints so that the data arrives consumption-ready and can drive analysis and action within the business. In this webinar, Sean Anderson from Cloudera and Kirit Busu, Director of Product Management at StreamSets, will discuss Hadoop's ecosystem and IoT capabilities and provide advice about common patterns and best practices. Using specific examples, they will demonstrate how to build and run end-to-end IOT data flows using StreamSets and Cloudera infrastructure.

Kelley Blue Book Uses Big Data to Increase User Engagement Over 100%

Cloudera, Inc.

Extreme Sports & Beyond: Exploring a new frontier in data with GoPro

Cloudera, Inc.

GoPro is a powerful global brand, thanks in large part to its innovative cameras and accessories that capture moments other cameras just miss: surfing in Maui, skiing in Tahoe, recording your child’s first steps. And today, the company is nearly as well known for its user-generated social and content networks. Join us for this special webinar hosted by Tableau, Trifacta, and Cloudera—featuring GoPro. We’ll dive into GoPro’s data strategy and architecture, from ingest and processing to data prep and reporting, all on AWS.

Enterprise Hadoop in the Cloud. In Minutes. | How to Run Cloudera Enterprise ...

Cloudera, Inc.

Hadoop Essentials -- The What, Why and How to Meet Agency Objectives

Cloudera, Inc.

This session will provide an executive overview of the Apache Hadoop ecosystem, its basic concepts, and its real-world applications. Attendees will learn how organizations worldwide are using the latest tools and strategies to harness their enterprise information to solve business problems and the types of data analysis commonly powered by Hadoop. Learn how various projects make up the Apache Hadoop ecosystem and the role each plays to improve data storage, management, interaction, and analysis. This is a valuable opportunity to gain insights into Hadoop functionality and how it can be applied to address compelling business challenges in your agency.

Risk Management for Data: Secured and Governed

Cloudera, Inc.

Cloudera Tech Day Presentation by Eddie Garcia, Chief Security Architect, Cloudera. Protecting enterprise data is an increasingly complex challenge given the diversity and sophistication of threat actors and their cyber-tactics. In this session, participants will hear a comprehensive introduction to Hadoop Security, including the “three A’s” for secure operating environments: Authentication, Authorization, and Audit. In addition, the presenter will cover strategies to orchestrate data security, encryption, and compliance, and will explain the Cloudera Security Maturity Model for Hadoop. Attendees will leave with a greater understanding of how effective INFOSEC relies on an enterprise big data governance and risk management approach.

Using Hadoop to Drive Down Fraud for Telcos

Cloudera, Inc.

Consolidate your data marts for fast, flexible analytics 5.24.18

Cloudera, Inc.

In this webinar, Cloudera and AtScale will showcase: How a company can modernize their analytic architecture to deliver flexibility and agility to more end-users. How using AtScale’s Universal Semantic layer can end the data chaos and allow business users to use the data in the modern platform. Highlight the performance of AtScale and Cloudera’s analytic database with newly completed TPC-DS standard benchmarking. Best practices for migrating from legacy appliances.

Building a Data Hub that Empowers Customer Insight (Technical Workshop)

Cloudera, Inc.

We have seen the evolution with the Bi and Data Science fields from the structured data warehouse to data lake and finally, to the data hub. This session will cover the key steps required to building a data hub, examining how best to align and engage stakeholders and develop architectural sanction to enable your organisations to realise new customer insights and better enable you to achieve business objectives.

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

Cloudera, Inc.

In this workshop, we will look outside the box and help expand the problem space to include issues you may not have thought were possible before Big Data. From Near Real Time (NRT) recommendation engines, loan applications to churn detection, Big Data is answering new questions and providing organisations with a competitive edge through revenue increase, cost savings and risk mitigation. We will take a special look at the role the Cloud can play in elevating your analytics environment. We will discuss real world examples of how Big Data answers these questions and does it at a lower cost outlay.

What’s New in Cloudera Enterprise 6.0: The Inside Scoop 6.14.18

Cloudera, Inc.

Introducing Cloudera Navigator Optimizer: Offload Assessments and Active Data...

Cloudera, Inc.

Cloudera Enterprise can be used as an adaptive, high-performance analytic database, complementing existing data warehouses by relieving the pressure of growing numbers of ETL jobs and BI analytics. But where do you get started when developing your offload strategy? How can you identify which workloads are the best fit for which system? And once you’re up and running, how can you constantly adapt to Hadoop’s changing data needs? Cloudera Navigator Optimizer eases the path for moving the right workloads to Hadoop and then actively manages data allowing you to take advantage of Hadoop’s benefits. Now generally available with the recent release of Cloudera 5.8 and a unique part of Cloudera’s analytic database solution, Navigator Optimizer gives you the workload visibility and assessments to build a predictable offload plan, adapt to evolving data and workload demands, and optimize query performance for Hadoop technologies 3 Things to Learn: Join Ewa Ding, Senior Product Manager at Cloudera, as she discusses: -An overview of Cloudera Navigator Optimizer and its key features -A live demo and key use cases of this web-based tool -What’s next for active data optimization in Hadoop

Leveraging the Cloud for Big Data Analytics 12.11.18

Cloudera, Inc.

End to End Streaming Architectures

Cloudera, Inc.

Data is being generated at a feverish pace and many businesses want all of it at their disposal to solve complex strategic problems. As decision making moves to real-time, enterprises need data ready for analysis immediately. Sean Anderson and Amandeep Khurana will discuss common pipeline trends in modern streaming architectures, Hadoop components that enable streaming capabilities, and popular use cases that are enabling the world of IOT and real-time data science.

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...

Cloudera, Inc.

You like to use R, and you need to use big data. dplyr, one of the most popular packages for R, makes it easy to query large data sets in scalable processing engines like Apache Spark and Apache Impala. But there can be pitfalls: dplyr works differently with different data sources—and those differences can bite you if you don’t know what you’re doing. Ian Cook is a data scientist, an R contributor, and a curriculum developer at Cloudera University. In this webinar, Ian will show you exactly what you need to know about sparklyr (from RStudio) and the package implyr (from Cloudera). He will show you how to write dplyr code that works across these different interfaces. And, he will solve mysteries: Do I need to know SQL to use dplyr? When is a “tbl” not a “tibble”? Why is 1 not always equal to 1? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems? 3 things to learn: Do I need to know SQL to use dplyr? When should you collect(), collapse(), and compute()? How can you use dplyr to combine data stored in different systems?

Foundations strata sf-2019_final

Jonathan Seidman

Foundations for Successful Data Projects – Strata London 2019

Jonathan Seidman

What's hot

Data Drive Applications_Webinar

Sean Spediacci

Making Self-Service BI a Reality in the Enterprise

Cloudera, Inc.

Big data journey to the cloud rohit pujari 5.30.18

Cloudera, Inc.

PaaS or Fail: Rule the Cloud with Altus

Cloudera, Inc.

How Data Drives Business at Choice Hotels

Cloudera, Inc.

How to Build Continuous Ingestion for the Internet of Things

Cloudera, Inc.

Kelley Blue Book Uses Big Data to Increase User Engagement Over 100%

Cloudera, Inc.

Extreme Sports & Beyond: Exploring a new frontier in data with GoPro

Cloudera, Inc.

Enterprise Hadoop in the Cloud. In Minutes. | How to Run Cloudera Enterprise ...

Cloudera, Inc.

Hadoop Essentials -- The What, Why and How to Meet Agency Objectives

Cloudera, Inc.

Risk Management for Data: Secured and Governed

Cloudera, Inc.

Using Hadoop to Drive Down Fraud for Telcos

Cloudera, Inc.

Consolidate your data marts for fast, flexible analytics 5.24.18

Cloudera, Inc.

Building a Data Hub that Empowers Customer Insight (Technical Workshop)

Cloudera, Inc.

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

Cloudera, Inc.

What’s New in Cloudera Enterprise 6.0: The Inside Scoop 6.14.18

Cloudera, Inc.

Introducing Cloudera Navigator Optimizer: Offload Assessments and Active Data...

Cloudera, Inc.

Leveraging the Cloud for Big Data Analytics 12.11.18

Cloudera, Inc.

End to End Streaming Architectures

Cloudera, Inc.

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...

Cloudera, Inc.

What's hot (20)

Data Drive Applications_Webinar

Making Self-Service BI a Reality in the Enterprise

Big data journey to the cloud rohit pujari 5.30.18

PaaS or Fail: Rule the Cloud with Altus

How Data Drives Business at Choice Hotels

How to Build Continuous Ingestion for the Internet of Things

Kelley Blue Book Uses Big Data to Increase User Engagement Over 100%

Extreme Sports & Beyond: Exploring a new frontier in data with GoPro

Enterprise Hadoop in the Cloud. In Minutes. | How to Run Cloudera Enterprise ...

Hadoop Essentials -- The What, Why and How to Meet Agency Objectives

Risk Management for Data: Secured and Governed

Using Hadoop to Drive Down Fraud for Telcos

Consolidate your data marts for fast, flexible analytics 5.24.18

Building a Data Hub that Empowers Customer Insight (Technical Workshop)

How Big Data Can Enable Analytics from the Cloud (Technical Workshop)

What’s New in Cloudera Enterprise 6.0: The Inside Scoop 6.14.18

Introducing Cloudera Navigator Optimizer: Offload Assessments and Active Data...

Leveraging the Cloud for Big Data Analytics 12.11.18

End to End Streaming Architectures

Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfac...

Similar to Managing Successful Data Projects: Technology Selection and Team Building

Foundations strata sf-2019_final

Jonathan Seidman

Foundations for Successful Data Projects – Strata London 2019

Jonathan Seidman

How to successfully engage enterprise software vendors – software selection

John Cachat

So many clouds - 7 things to consider when choosing your IaaS provider

Sirris

A Primer for Your Next Data Science Proof of Concept on the Cloud

Alton Alexander

Lean Analytics: How to get more out of your data science team

Digital Transformation EXPO Event Series

Lean Analytics is a set of rules to make data science more streamlined and productive. It touches on many aspects of what a data scientist should be and how a data science project should be defined to be successful. During this presentation Richard will present where data science projects go wrong, how you should think of data science projects, what constitutes success in data science and how you can measure progress. This session will be loaded with terms, stories and descriptions of project successes and failures. If you're wondering whether you're getting value out of data science, how to get more value out of it and even whether you need it then this talk is for you! What you will take away from this session Learn how to make your data science projects successful Evaluate how to track progress and report on the efficacy of data science solutions Understand the role of engineering and data scientists Understand your options for processes and software

How a global manufacturing company built a data science capability from scratch

Carlo Torniai

Transform Banking with Big Data and Automated Machine Learning 9.12.17

Cloudera, Inc.

Banks are rich in valuable data and can build and maintain a competitive advantage by identifying and executing on high-value machine learning projects leveraging the rich data available.This webinar will describe use cases fit for big data and machine learning in the banking sector (commercial, consumer, regulatory, and markets) and the impact they can have for your organization. 3 things to learn: * How to create a next generation data platform and why it is important * How to monetize big data using predictive modeling and machine learning * What is needed for automated machine learning as a sustainable, cost-effective, and efficient solution

7 things to consider when choosing your IaaS provider for ISV/SaaS

Frederik Denkens

SAP Inside Track 2018 - "Quidquid agis, prudenter agas ..." - Learnings from ...

Christian Lechner

Successfully developing Cloud Native applications is often regarded from a purely technological perspective - what design aspects have to be considered, what frameworks are best fitting and so on. However, there are many more especially non-technological factors that you have to take into account and that will influence your success. In this talk, I would like to draw your attention to these factors and share the experiences and learnings that we made when developing our first Cloud Native applications.

Holistic Product Development

Gary Pedretti

In the world of agile, there is theory and then there is practice. We like to talk about self-organizing teams, asynchronous execution, BDD, TDD, and emergent architecture. We also talk about cross-functional teams: how analysts, testers, architects, technical writers, and UX designers belong on the same team, right next to programmers. It all sounds nice in theory, but how does this work in reality? What do these people actually do? How do they interact? What does it look like? Is there really a pragmatic way to make this work? In this simulation, a cross-functional team will actually build a piece of software. Every specialist will have a hand in the process. Every specialist will also act as a generalist. Everyone will add value. And as a team, we’ll get something DONE. This is your opportunity to see agile development in practice, and to bridge the gap between what agilists say and what teams do. And it’s not as new or as difficult as you think – affinity between testers, BA’s, coders, and other team members has really been at the root of effective development practices all along. Let’s just finally acknowledge that it works, demonstrate its capabilities, and encourage it going forward. This IS agile development.

Building successful data science teams

Venkatesh Umaashankar

Speaker: Venkatesh Umaashankar LinkedIn: https://www.linkedin.com/in/venkateshumaashankar/ What will be discussed? What is Data Science? Types of data scientists What makes a Data Science Team? Who are its members? Why does a DS team need Full Stack Developer? Who should lead the DS Team Building a Data Science team in a Startup Vs Enterprise Case studies on: Evolution Of Airbnb’s DS Team How Facebook on-boards DS team and trains them Apple’s Acqui-hiring Strategy to build DS team Spotify -‘Center of Excellence’ Model Who should attend? Managers Technical Leaders who want to get started with Data Science

Data Governance in an Agile SCRUM Lean MVP World

DATAVERSITY

Most of us learned data modeling via a waterfall-driven methodology lens. Yet Agile and other modern development methods have for the most part assumed that data governance is an anti-pattern to just getting things (software) done. Well look at questions such as: •Are Agile and Data Governance Enemies? •How can we get stuff done AND get systems delivered? •And what do we do about existing systems delivered without data governance attention? We'll also look at how data modeling fits in the answers to these questions.

Profit from AI & Machine Learning: The Best Practices for People & Process

Tony Baer

Open Web Technologies and You - Durham College Student Integration Presentation

darryl_lehmann

10 bezcennych lekcji dla software developera stającego się szefem firmy

Wojciech Seliga

[Originally Polish lecture with English slides - with a few exceptions] Przez wiele lat byłem software developerem. Koncentrowałem się na kodzie, projektach software'owych oraz interakcjach w moim zespole i z klientami. Byłem pewny, że Agile rozwiązuje wszystkie problemy tego świata. Śmiałem się z komiksów Scotta Adamsa i stworzonej przez niego karykatury szefa (PHB). Życie było proste i piękne... Teraz od ponad 8 lat prowadzę firmę software'ową, którą przy blisko 90 osobach trudno już nazwać maleństwem. Sam stałem się "szefem" na pełen etat. Podczas prezentacji podzielę się z Wami różnymi doświadczeniami oraz naukami (nieraz bolesnymi) jakie wyniosłem w ostatnich latach podczas mojej stopniowej przemiany z developera/inżyniera w przedsiębiorcę i szefa firmy. O ile zapewne nie wszystkie sytuacje i wnioski mają lub mogą mieć (o ile marzysz o własnym startupie czy zespole) zastosowanie w Twoim życiu, same sobie ich uświadomienie może oszczędzić Ci w przyszłości straty mnóstwa czasu, energii i pieniędzy oraz uniknąć przykrych rozczarowań.

Data Con LA 2019 - The challenges of data science for veteran media organizat...

Data Con LA

An overview of the tools, engineering infrastructure, and decisions that we've been making in concert with our newsroom and executive leadership.- What tools might you use and why? Python, Scala, Spark, AWS, Google GCP, BigQuery, Redshift, Tableau, SQL?- What engineering tradeoffs do you have to make when building data science platforms and choosing big data tools?- How can you empower analysts, engineers, data scientists, and privacy advocates to build a data driven culture ecosystem at at any size company?- How do you recruit, train, and retain world class data talent against competitors like Netflix, Google, and Snap?

The Big Data Dream Team

Accenture Analytics

Adam Boyse

Lucia Garcia

360 digital transformation profile

Kamal Singh

Similar to Managing Successful Data Projects: Technology Selection and Team Building (20)