A Tale of a Data Driven Culture-(Gloria Lau, Timeful)
•
5 likes•2,801 views
This document summarizes a keynote talk about creating a truly data-driven culture. The speaker advocates giving all employees, not just data scientists, self-service access to analytics tools to remove friction and speed up innovation. The talk describes an experiment where non-engineers were given access to Databricks and learned to explore data on their own, forming their own insights more quickly without needing data scientists to act as "SQL monkeys". This approach focused data scientists on building data products while empowering all employees to be data-driven.
Report
Share
Report
Share
Download to read offline
Recommended
Keynote at Spark Summit
This document summarizes a talk given by Gloria Lau on creating a data-driven culture. The key points are:
1) Most people in a company want answers from data and are capable of exploring data themselves given the right tools.
2) The biggest friction to being data-driven is lack of speed when accessing and analyzing data.
3) An experiment was conducted giving all employees access to Databricks to analyze production data in real time. This removed the need for "SQL monkeys" and led to shorter innovation cycles as more people directly explored the data.
Keynote at Big Data Tech Con SF 2014
The document discusses the importance of building data products in the right order. It recommends first focusing on data infrastructure, then doing offline modeling, launching an initial online data product, and gathering user feedback. This allows issues to be identified and addressed without wasting resources on unnecessary optimizations. Two key questions are proposed: 1) how a change will impact the core user metric, and 2) how users will spend their limited time with the product. Premature optimization should be avoided, and focus should be on the critical 3% of code that matters most for users.
Join 2017_Deep Dive_To Use or Not Use PDT's
Jonathon Miller-Girvetz discusses best practices for using persistent derived tables (PDTs) in Looker. He covers when to use ephemeral derived tables versus PDTs, as well as when to move PDT logic to an ETL process. Some key points include: use PDTs for historical summaries and aggregations, but EDTs for quick or dynamic queries; consider data freshness, computational resources, and use outside of Looker when deciding between PDTs and ETL; and follow conventions like consistent naming when developing PDTs.
Wisdom of Crowds Webinar Deck
The webinar covered the benefits of Looker's approach to business intelligence (BI), which leverages in-database analytics to provide self-service BI capabilities. It discussed how Looker avoids data movement and summarization, uses a modern web architecture for accessibility from any device, and employs an intelligent modeling layer called LookML to describe data and create reusable business logic. The presentation included a demo of Looker's capabilities and concluded by thanking attendees and providing information on future webinars and accessing a free trial of Looker.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
Beyond Data Discovery: The Value Unlocked by Modern Data Modeling
In this webinar we will discuss Looker’s novel approach to data modeling and how it powers a data exploration environment with unprecedented depth and agility.
Some topics we will cover:
-A new architecture beyond direct connect
-Language-based, git-integrated data modeling
-Abstractions that make SQL more powerful and more efficient
Objectivity/DB: A Multipurpose NoSQL Database
The speakers will describe the flexible configuration possibilities that Objectivity/DB provides, with an emphasis on how best to distribute data across multiple storage nodes. The session will start by describing the distributed processing architecture of Objectivity/DB before covering the new Placement Manager features. The speakers will also describe how Objectivity/DB compares and contrasts with other NoSQL solutions.
Managing Data Science | Lessons from the Field
The document discusses best practices for managing data science teams based on lessons learned. It outlines common pitfalls such as solving the wrong problem, having the wrong tools, or results being used incorrectly. Issues include data science being different from software development and forgetting other stakeholders. Recommendations include establishing processes for the full lifecycle from ideation to monitoring, using modular systems thinking, and defining roles like data scientists, managers, and product owners to address organizational challenges. The goal is to deliver measurable, reliable, and scalable insights.
Recommended
Keynote at Spark Summit
This document summarizes a talk given by Gloria Lau on creating a data-driven culture. The key points are:
1) Most people in a company want answers from data and are capable of exploring data themselves given the right tools.
2) The biggest friction to being data-driven is lack of speed when accessing and analyzing data.
3) An experiment was conducted giving all employees access to Databricks to analyze production data in real time. This removed the need for "SQL monkeys" and led to shorter innovation cycles as more people directly explored the data.
Keynote at Big Data Tech Con SF 2014
The document discusses the importance of building data products in the right order. It recommends first focusing on data infrastructure, then doing offline modeling, launching an initial online data product, and gathering user feedback. This allows issues to be identified and addressed without wasting resources on unnecessary optimizations. Two key questions are proposed: 1) how a change will impact the core user metric, and 2) how users will spend their limited time with the product. Premature optimization should be avoided, and focus should be on the critical 3% of code that matters most for users.
Join 2017_Deep Dive_To Use or Not Use PDT's
Jonathon Miller-Girvetz discusses best practices for using persistent derived tables (PDTs) in Looker. He covers when to use ephemeral derived tables versus PDTs, as well as when to move PDT logic to an ETL process. Some key points include: use PDTs for historical summaries and aggregations, but EDTs for quick or dynamic queries; consider data freshness, computational resources, and use outside of Looker when deciding between PDTs and ETL; and follow conventions like consistent naming when developing PDTs.
Wisdom of Crowds Webinar Deck
The webinar covered the benefits of Looker's approach to business intelligence (BI), which leverages in-database analytics to provide self-service BI capabilities. It discussed how Looker avoids data movement and summarization, uses a modern web architecture for accessibility from any device, and employs an intelligent modeling layer called LookML to describe data and create reusable business logic. The presentation included a demo of Looker's capabilities and concluded by thanking attendees and providing information on future webinars and accessing a free trial of Looker.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
Beyond Data Discovery: The Value Unlocked by Modern Data Modeling
In this webinar we will discuss Looker’s novel approach to data modeling and how it powers a data exploration environment with unprecedented depth and agility.
Some topics we will cover:
-A new architecture beyond direct connect
-Language-based, git-integrated data modeling
-Abstractions that make SQL more powerful and more efficient
Objectivity/DB: A Multipurpose NoSQL Database
The speakers will describe the flexible configuration possibilities that Objectivity/DB provides, with an emphasis on how best to distribute data across multiple storage nodes. The session will start by describing the distributed processing architecture of Objectivity/DB before covering the new Placement Manager features. The speakers will also describe how Objectivity/DB compares and contrasts with other NoSQL solutions.
Managing Data Science | Lessons from the Field
The document discusses best practices for managing data science teams based on lessons learned. It outlines common pitfalls such as solving the wrong problem, having the wrong tools, or results being used incorrectly. Issues include data science being different from software development and forgetting other stakeholders. Recommendations include establishing processes for the full lifecycle from ideation to monitoring, using modular systems thinking, and defining roles like data scientists, managers, and product owners to address organizational challenges. The goal is to deliver measurable, reliable, and scalable insights.
H2O World - Collaborative, Reproducible Research with H2O - Nick Elprin
The document discusses collaborative and reproducible research using H2O. It conveys that reproducibility and collaboration are important for individual productivity, team efficiency, and generating insights. Challenges include that analytical work involves more than code, and environments can differ. The solution proposed is a central hub for running, tracking, and sharing work to provide easy access to resources and turnkey deployment while enabling version control, reproducibility, and collaboration. Containers are discussed as a way to standardize environments. Ten simple rules for reproducible research are also presented.
Big Data Thailand 2016 Meetup 1
This document discusses big data and emerging technologies. It notes that big data has moved past the peak of inflated expectations on the hype cycle and become prevalent across many industries. It also discusses citizen data science and machine learning, various cloud and big data technologies, programming languages, tools, and concepts like real-time processing, deep learning, Apache projects like Alluxio, Flink and Tajo. It concludes by questioning whether organizations should build, buy, or take a hybrid approach to their big data needs.
Harness the power of data
Harness the power of data
Now is the time to become an analytics-driven organization. Discover how.
Amazon
Data science 101 Masterclass
In this presentation, Microsoft data scientists Ben Keen and Shahzia Holtom cover an introduction to data science with respect to:
- What is a data scientist?
- What data does a data scientist need?
- AI ethics and responsibility
- What is MLOps and how does it drive value?
Building an Insight Machine - Strata DDBD 2015
First, we will explore the power of a compounding insight machine (as opposed to an ad hoc insight machine):
-Human time is focused on improving logic, rather than executing outcomes
-Less dependent on human biases or frailty
-Robust to and tested by a huge collection of scenarios
Second, we will explore the anatomy of such a machine:
-The roles you need to cast on your team and who to fill them with
-The key processes required for generating and capturing insight and, more importantly, for building upon those insights
-The technology required to enable this approach
How the economist with cloud BI and Looker have improved data-driven decision...
This session by The Economist Group, Cloud BI Ltd and Looker explores the challenges of data-driven decision making and how powerful the approach can be. Hear how the solution was implemented quickly and evolved in the cloud and the benefits of being able to see and understand customer preferences through a 360-degree view.
Leveraging the Cloud for BI Infrastructure: With Focus on and Demos of Micros...
Get a better understanding of the array of Cloud offerings for your BI infrastructure plus, a demo of spinning up an Azure instance. View the webinar recording: http://www.senturus.com/resources/microsoft-azures-value-proposition-for-bi/.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
Modernizing Architecture for a Complete Data Strategy
The document outlines a presentation about modernizing data strategies. It discusses how companies' relationships with data are changing and the business drivers for adopting big data and analytics. It then provides guidance on building a modern data strategy, emphasizing the importance of people, process, and technology. Specifically, it recommends starting with high-impact use cases, staying agile, and evolving capabilities over time to maximize value from data. The presentation also covers how Hadoop is being used for different workloads in both on-premise and cloud environments.
Microservices And Fast Data: Industry And Architecture Trends [with 451 Resea...
This document discusses trends in microservices and fast data architectures. It summarizes Lightbend's Fast Data Platform, which provides streaming engines, pluggable machine learning libraries, a reactive platform, operational tooling, and infrastructure support to enable fast data applications. The platform aims to support real-time use cases like predictive analytics, personalization, financial processes, and industrial IoT by moving from batch to streaming data processing.
Mathworks case example
Data analytics uses data to drive decision making and enable organizations to derive insights, drive business actions, and produce better outcomes. Analytics maturity ranges from descriptive analytics, which describes what happened, to predictive and prescriptive analytics. Analytic investment drives return on investment through historical analysis and online analytics. Developing Internet of Things systems is an iterative workflow where analytics answer questions and more data is taken than initially planned to enable additional insights.
Operationalizing analytics to scale
Many companies have invested time and money into building sophisticated data pipelines that can move massive amounts of data, often in real time. However, for the analyst or data scientist who builds offline models, integrating their analyses into these pipelines for operational purposes can pose a challenge.
In this slide deck, we will discuss some key technologies and workflows companies can leverage to build end-to-end solutions for automating statistical and machine learning solutions: from collection and storage to analysis and real-time predictions.
Enable Advanced Analytics with Hadoop and an Enterprise Data Hub
This document discusses enabling advanced analytics with Hadoop and an enterprise data hub. It describes current challenges around siloed data and long timelines for analytics projects. An agile analytics process is proposed using an enterprise data hub to break down data silos and deliver insights faster. Case studies are presented on how Monsanto used such a system to automate research and development decisions to reduce product development time from years to months. A second case study describes a system that analyzes mobile and social data to identify suicide risk factors in veterans in real time.
How to Build a Data-Driven Company: From Infrastructure to Insights
Companies like Buffer, SeatGeek, and Asana aren’t just talking about the value of data, they’re building data infrastructure that can actually deliver it. Join this 45-minute webinar to learn why these companies are investing in data and what you need to know to keep up.
Belladati Meetup Singapore Workshop
The meetup agenda outlines presentations on using BellaDati's agile analytics and reporting tool. The morning sessions cover creating quick reports, collaboration features, and industry-specific app templates. The afternoon advanced analytics session with Peter Fedorocko highlights BellaDati's integrated data warehouse, ETL scripts, and capabilities for analyzing Twitter, MongoDB, and other data sources.
Adoption is the only option hadoop is changing our world and changing yours f...
Adoption Is The Only Option
Hadoop Is Changing Our World And Changing Yours
Mike Gualtieri, Principal Analyst
Forrester
Self-Service Data Preparation for Tableau Users
Many users spend too much time preparing data and not enough time providing the valuable analysis. Datawatch is the fastest and easiest solution for self-service data preparation and automation to analyze your data in Tableau quickly and easily.
How to implement Hadoop successfully
This document provides tips for successfully implementing Hadoop. It recommends starting with a well-defined use case using a small team on the cloud to test feasibility. It also stresses the importance of skills training given Hadoop's complexity, and adjusting processes and solutions based on data freshness needs rather than pursuing real-time analytics. Democratizing data access through self-service tools is also highlighted to maximize insights from Hadoop implementations.
Leveraged Analytics at Scale
Most of analytics modeling work today focuses on the production of single-purpose "artisanal" models for predictions. This approach to analytics is fragile with respect to model consistency, reorganization, and resource availability. This talk will argue that instead the focus of analytics modeling should be toward the production of analytics interchangeable parts, which can be combined in creative ways to produce a wide variety of analytics results. This "nuts and bolts" approach allows analytics groups to produce results in an agile way where the time between ask and answer is determined by the right combination of analytics, rather than the modeling.
Hadoop summit socialize_v1.0
This document discusses Big Data analytics solutions from Splunk and ShareThis. It introduces Splunk, which provides a machine data platform for collecting and analyzing structured and unstructured data. ShareThis uses Splunk to analyze over 20 billion monthly ad impressions to gain insights. The document outlines how Splunk was used to build a real-time dashboard for analyzing social sharing trends across ShareThis' network in order to help their PR team. It also discusses how Splunk can provide operational analytics and insights from sources like websites, APIs, and mobile notification systems.
What makes an effective data team?
This document discusses what makes an effective data team. It begins with introductions from Alex Dean, CEO of Snowplow Analytics. It then discusses how Snowplow helps companies collect and analyze customer event data. The document outlines a hierarchy of needs for a data team, beginning with ensuring data is available and ending with data scientists doing industry-leading work. It provides advice on each level of the hierarchy to help data teams become more effective.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
Product Management in the Era of Data Science
My slide-deck from a webinar on the same topic for the Institute of Product Leadership, April 4th, 2017
What does it take to build killer products in the “AI-first” era? What makes for a great Data Science-driven product and how do great Product Managers leverage Data Science to drive value for customers? Find out how to avoid the pitfalls of hype-chasing Data Science tactics. Learn how to work with Data Science and Engineering to build a compelling product and solve real problems.
Mandar takes a practitioner’s approach to present his recipe for success for building Data Science-driven products that drive enduring value for customers.
More Related Content
What's hot
H2O World - Collaborative, Reproducible Research with H2O - Nick Elprin
The document discusses collaborative and reproducible research using H2O. It conveys that reproducibility and collaboration are important for individual productivity, team efficiency, and generating insights. Challenges include that analytical work involves more than code, and environments can differ. The solution proposed is a central hub for running, tracking, and sharing work to provide easy access to resources and turnkey deployment while enabling version control, reproducibility, and collaboration. Containers are discussed as a way to standardize environments. Ten simple rules for reproducible research are also presented.
Big Data Thailand 2016 Meetup 1
This document discusses big data and emerging technologies. It notes that big data has moved past the peak of inflated expectations on the hype cycle and become prevalent across many industries. It also discusses citizen data science and machine learning, various cloud and big data technologies, programming languages, tools, and concepts like real-time processing, deep learning, Apache projects like Alluxio, Flink and Tajo. It concludes by questioning whether organizations should build, buy, or take a hybrid approach to their big data needs.
Harness the power of data
Harness the power of data
Now is the time to become an analytics-driven organization. Discover how.
Amazon
Data science 101 Masterclass
In this presentation, Microsoft data scientists Ben Keen and Shahzia Holtom cover an introduction to data science with respect to:
- What is a data scientist?
- What data does a data scientist need?
- AI ethics and responsibility
- What is MLOps and how does it drive value?
Building an Insight Machine - Strata DDBD 2015
First, we will explore the power of a compounding insight machine (as opposed to an ad hoc insight machine):
-Human time is focused on improving logic, rather than executing outcomes
-Less dependent on human biases or frailty
-Robust to and tested by a huge collection of scenarios
Second, we will explore the anatomy of such a machine:
-The roles you need to cast on your team and who to fill them with
-The key processes required for generating and capturing insight and, more importantly, for building upon those insights
-The technology required to enable this approach
How the economist with cloud BI and Looker have improved data-driven decision...
This session by The Economist Group, Cloud BI Ltd and Looker explores the challenges of data-driven decision making and how powerful the approach can be. Hear how the solution was implemented quickly and evolved in the cloud and the benefits of being able to see and understand customer preferences through a 360-degree view.
Leveraging the Cloud for BI Infrastructure: With Focus on and Demos of Micros...
Get a better understanding of the array of Cloud offerings for your BI infrastructure plus, a demo of spinning up an Azure instance. View the webinar recording: http://www.senturus.com/resources/microsoft-azures-value-proposition-for-bi/.
Senturus, a business analytics consulting firm, has a resource library with hundreds of free recorded webinars, trainings, demos and unbiased product reviews. Take a look and share them with your colleagues and friends: http://www.senturus.com/resources/.
Modernizing Architecture for a Complete Data Strategy
The document outlines a presentation about modernizing data strategies. It discusses how companies' relationships with data are changing and the business drivers for adopting big data and analytics. It then provides guidance on building a modern data strategy, emphasizing the importance of people, process, and technology. Specifically, it recommends starting with high-impact use cases, staying agile, and evolving capabilities over time to maximize value from data. The presentation also covers how Hadoop is being used for different workloads in both on-premise and cloud environments.
Microservices And Fast Data: Industry And Architecture Trends [with 451 Resea...
This document discusses trends in microservices and fast data architectures. It summarizes Lightbend's Fast Data Platform, which provides streaming engines, pluggable machine learning libraries, a reactive platform, operational tooling, and infrastructure support to enable fast data applications. The platform aims to support real-time use cases like predictive analytics, personalization, financial processes, and industrial IoT by moving from batch to streaming data processing.
Mathworks case example
Data analytics uses data to drive decision making and enable organizations to derive insights, drive business actions, and produce better outcomes. Analytics maturity ranges from descriptive analytics, which describes what happened, to predictive and prescriptive analytics. Analytic investment drives return on investment through historical analysis and online analytics. Developing Internet of Things systems is an iterative workflow where analytics answer questions and more data is taken than initially planned to enable additional insights.
Operationalizing analytics to scale
Many companies have invested time and money into building sophisticated data pipelines that can move massive amounts of data, often in real time. However, for the analyst or data scientist who builds offline models, integrating their analyses into these pipelines for operational purposes can pose a challenge.
In this slide deck, we will discuss some key technologies and workflows companies can leverage to build end-to-end solutions for automating statistical and machine learning solutions: from collection and storage to analysis and real-time predictions.
Enable Advanced Analytics with Hadoop and an Enterprise Data Hub
This document discusses enabling advanced analytics with Hadoop and an enterprise data hub. It describes current challenges around siloed data and long timelines for analytics projects. An agile analytics process is proposed using an enterprise data hub to break down data silos and deliver insights faster. Case studies are presented on how Monsanto used such a system to automate research and development decisions to reduce product development time from years to months. A second case study describes a system that analyzes mobile and social data to identify suicide risk factors in veterans in real time.
How to Build a Data-Driven Company: From Infrastructure to Insights
Companies like Buffer, SeatGeek, and Asana aren’t just talking about the value of data, they’re building data infrastructure that can actually deliver it. Join this 45-minute webinar to learn why these companies are investing in data and what you need to know to keep up.
Belladati Meetup Singapore Workshop
The meetup agenda outlines presentations on using BellaDati's agile analytics and reporting tool. The morning sessions cover creating quick reports, collaboration features, and industry-specific app templates. The afternoon advanced analytics session with Peter Fedorocko highlights BellaDati's integrated data warehouse, ETL scripts, and capabilities for analyzing Twitter, MongoDB, and other data sources.
Adoption is the only option hadoop is changing our world and changing yours f...
Adoption Is The Only Option
Hadoop Is Changing Our World And Changing Yours
Mike Gualtieri, Principal Analyst
Forrester
Self-Service Data Preparation for Tableau Users
Many users spend too much time preparing data and not enough time providing the valuable analysis. Datawatch is the fastest and easiest solution for self-service data preparation and automation to analyze your data in Tableau quickly and easily.
How to implement Hadoop successfully
This document provides tips for successfully implementing Hadoop. It recommends starting with a well-defined use case using a small team on the cloud to test feasibility. It also stresses the importance of skills training given Hadoop's complexity, and adjusting processes and solutions based on data freshness needs rather than pursuing real-time analytics. Democratizing data access through self-service tools is also highlighted to maximize insights from Hadoop implementations.
Leveraged Analytics at Scale
Most of analytics modeling work today focuses on the production of single-purpose "artisanal" models for predictions. This approach to analytics is fragile with respect to model consistency, reorganization, and resource availability. This talk will argue that instead the focus of analytics modeling should be toward the production of analytics interchangeable parts, which can be combined in creative ways to produce a wide variety of analytics results. This "nuts and bolts" approach allows analytics groups to produce results in an agile way where the time between ask and answer is determined by the right combination of analytics, rather than the modeling.
Hadoop summit socialize_v1.0
This document discusses Big Data analytics solutions from Splunk and ShareThis. It introduces Splunk, which provides a machine data platform for collecting and analyzing structured and unstructured data. ShareThis uses Splunk to analyze over 20 billion monthly ad impressions to gain insights. The document outlines how Splunk was used to build a real-time dashboard for analyzing social sharing trends across ShareThis' network in order to help their PR team. It also discusses how Splunk can provide operational analytics and insights from sources like websites, APIs, and mobile notification systems.
What's hot (19)
H2O World - Collaborative, Reproducible Research with H2O - Nick Elprin
H2O World - Collaborative, Reproducible Research with H2O - Nick Elprin
How the economist with cloud BI and Looker have improved data-driven decision...
How the economist with cloud BI and Looker have improved data-driven decision...
Leveraging the Cloud for BI Infrastructure: With Focus on and Demos of Micros...
Leveraging the Cloud for BI Infrastructure: With Focus on and Demos of Micros...
Modernizing Architecture for a Complete Data Strategy
Modernizing Architecture for a Complete Data Strategy
Microservices And Fast Data: Industry And Architecture Trends [with 451 Resea...
Microservices And Fast Data: Industry And Architecture Trends [with 451 Resea...
Enable Advanced Analytics with Hadoop and an Enterprise Data Hub
Enable Advanced Analytics with Hadoop and an Enterprise Data Hub
How to Build a Data-Driven Company: From Infrastructure to Insights
How to Build a Data-Driven Company: From Infrastructure to Insights
Adoption is the only option hadoop is changing our world and changing yours f...
Adoption is the only option hadoop is changing our world and changing yours f...
Similar to A Tale of a Data Driven Culture-(Gloria Lau, Timeful)
What makes an effective data team?
This document discusses what makes an effective data team. It begins with introductions from Alex Dean, CEO of Snowplow Analytics. It then discusses how Snowplow helps companies collect and analyze customer event data. The document outlines a hierarchy of needs for a data team, beginning with ensuring data is available and ending with data scientists doing industry-leading work. It provides advice on each level of the hierarchy to help data teams become more effective.
Frank Bien Opening Keynote - Join 2016
Frank Bien, CEO of Looker - along with Amazon, Google and other data disrupters - discuss how innovators are deeply integrating analytics into every aspect of their businesses, from mobile to warehouse to cloud.
Frank shares Looker’s vision for the future of business intelligence and data analytics and reveal pivotal product and partnership updates.
Product Management in the Era of Data Science
My slide-deck from a webinar on the same topic for the Institute of Product Leadership, April 4th, 2017
What does it take to build killer products in the “AI-first” era? What makes for a great Data Science-driven product and how do great Product Managers leverage Data Science to drive value for customers? Find out how to avoid the pitfalls of hype-chasing Data Science tactics. Learn how to work with Data Science and Engineering to build a compelling product and solve real problems.
Mandar takes a practitioner’s approach to present his recipe for success for building Data Science-driven products that drive enduring value for customers.
GoDataDriven Giovanni Lanzani
This document discusses applied data science and machine learning. It provides an overview of machine learning concepts like learning from data and choosing the best model for a given problem. It then discusses how companies often focus more on proofs of concept and presentations rather than implementing machine learning solutions. The document outlines three hurdles to implementation: oversimplifying requirements, the "Kaggle curse" of focusing only on model accuracy, and ensuring data and skills are production-grade. It provides recommendations like having a clear business case, automating data workflows, and hiring data scientists skilled in programming for real-world impact.
Splunk/Socialize at Hadoop Summit
This document discusses Big Data analytics solutions from Splunk and ShareThis. It introduces Splunk, which provides a machine data platform for collecting and analyzing structured and unstructured data. ShareThis uses Splunk to analyze over 20 billion monthly ad impressions to gain insights. The document outlines how Splunk was used to build a real-time dashboard for analyzing social sharing trends across ShareThis' network in order to help their PR team. It also discusses how Splunk can provide operational analytics and insights from sources such as websites, APIs, and mobile notification systems.
Big Data at the Speed of Business: Lessons Learned from Leading at the Edge
How do you make big data accessible, usable and valuable for everyone? And mine your data for intelligence in minutes and hours, not weeks and months? What about getting real-time insights from your data, even before you persist and replicate it? In this talk, we’ll examine compelling, real-world examples that offer a blueprint for integrating big data technologies (Splunk, Hadoop, RDBMS, Cassandra, HBase), delivering rapid visibility and insights to IT professionals, data analysts and business users, and that accelerate the adoption of big data in the enterprise.
Giovanni Lanzani GoDataDriven
This document discusses applied data science and machine learning. It begins by introducing the author and then discusses machine learning concepts like learning from data and choosing the best predictive model. It explains that data science is about creating value from data using machine learning, analytics, and visualization. However, many companies struggle to operationalize data science projects and end up with only prototypes instead of production systems. The document outlines three common hurdles - oversimplifying requirements, focusing only on model accuracy instead of practicality, and having insufficient data engineering skills. It advocates for taking a more holistic, business-focused approach to applied data science.
Data In Action: Business Value of Data
Lecture on databases for Technology and Operations class at Pepperdine Graziado School of Business. Presented October 2023.
Putting data science in your business a first utility feedback
Abed Ajraou is a Data & Insights Director, Lead Data Scientist, Data Teacher, Data Speaker and Data Author
O365 Practical Adoption Strategies - SPS Toronto 2017
This presentation covers some strategies and tactics you can use to stack the odds in your favour for a high degree of Office 365 adoption across your organization. I’ve come up with a 3-step strategy and some specific tactics for each so your end-users can thrive in O365!
Building a Data Cloud to enable Analytics & AI-Driven Innovation - Lak Lakshm...
Learn how Google Cloud addresses the key challenges when building an Agile Data & AI platform. This lecture is important regardless of the Cloud you are (will be) using because most businesses face the same 6 challenges:
1. High-quality AI requires a lot of data
2. AI Expertise is in high demand
3. Getting the value of ML requires a modern data platform
4. Activating ML requires surfacing AI into decision UIs
5. Operationalizing ML is hard
6. State-of-the-art changes rapidly
The lecture recording with Q&A is at https://youtu.be/ntBEQdD1IeQ
PXL Data Engineering Workshop By Selligent
On 2020-12-09 Laurens Vijnck and Jonny Daenen gave a workshop at PXL.
During this session, we collectively provisioned a streaming ingestion pipeline in mere minutes. The technology stack included Pub/Sub, Dataflow, and BigQuery. Hereafter, students had the opportunity to perform interactive queries on their own real-time data to answer a series of business questions. These questions were borrowed from real-life cases that we encountered at Selligent Marketing Cloud.
Google Colab (Free Jupyter Notebooks) and Google Data Studio have proven to be excellent tools to facilitate these kinds of interactive sessions.
Office 365 - Practical Adoption Strategies - SP Unite @ Haarlem
Get some practical advice on ways to improve end-user adoption for Office 365. Why it matters and ways to get there.
SPUnite17 O365 Practical Adoption Strategies
This document summarizes Joanne Klein's presentation on Office 365 adoption. It discusses identifying stakeholders, creating an adoption team, and treating adoption as a service with three phases: preparing for adoption, training users, and ongoing support. It provides specific tactics for each phase like creating an adoption center with training resources, conducting targeted training, and designating productivity champions. The presentation emphasizes focusing on business value, changing user behavior, and measuring adoption metrics over time.
The Right Data Warehouse: Automation Now, Business Value Thereafter
The Briefing Room with Dr. Robin Bloor and WhereScape
Live Webcast on April 1, 2014
Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7b23b14b532bd7be60a70f6bd5209f03
In the Big Data shuffle, everyone is looking at Hadoop as “the answer” to collect interesting data from a new set of sources. While Hadoop has given organizations the power to gather more information assets than ever before, the question still looms: which data, regardless of source, structure, volume and all the rest, are significant for affecting business value – and how do we harness it? One effective approach is to bolster the data warehouse environment with a solution capable of integrating all the data sources, including Hadoop, and automating delivery of key information into the rights hands.
Register for this episode of The Briefing Room to hear veteran Analyst Robin Bloor as he explains how a rapidly changing information landscape impacts data management. He will be briefed by Mark Budzinski of WhereScape, who will tout his company’s data warehouse automation solutions. Budzinski will discuss how automation can be the cornerstone for closing the gap between those responsible for data management and the people driving business decisions.
Visit InsideAnlaysis.com for more information.
Three signs your architecture is too small for big data. Camp IT December 2014
Three capability gaps that a traditional business intelligence architecture has with respect to processing big data and recommended extensions to address them.
Worst Practices in Artificial Intelligence
In this talk I discuss six "worst practices" in Artificial Intelligence, so that you don't make the same mistakes as you embark on your AI and Machine Learning journey!
The full talk (in cantonese) is here: https://youtu.be/NIIztmpA6Hc?t=1172
They Said, We Said: Bridge the Communication Gap with Behavior-Driven Develop...
The document describes a presentation by Sheetal Patel of The Vanguard Group on using Behavior-Driven Development (BDD) to improve communication between business and development teams. BDD uses examples in plain language to formalize requirements and validate software behavior. The Vanguard used BDD and Cucumber to engage business, development, and testing in collaborative sessions to define stories and test scenarios upfront. This helped address miscommunications over software features and reduced unused features from 64% to a lower percentage.
Analytics-Enabled Experiences: The New Secret Weapon
Tracking and analyzing how our individual products come together has always been an elusive problem for Steelcase. Our problem can be thought of in the following way: “we know how many Lego pieces we sell, yet we don’t know what Lego set our customers buy.” The Data Science team took over this initiative, which resulted in an evolution of our analytics journey. It is a story of innovation, resilience, agility and grit.
The effects of the COVID-19 pandemic on corporate America shined the spotlight on office furniture manufacturers to solve for ways on which the office can be made safe again. The team would have never imagined how relevant our work on product application analytics would become. Product application analytics became an industry priority overnight.
The proposal presented this year is the story of how data science is helping corporations bring people back to the office and set the path to lead the reinvention of the office space.
After groundbreaking milestones to overcome technical challenges, the most important question is: What do we do with this? How do we scale this? How do we turn this opportunity into a true competitive advantage? The response: stop thinking about this work as a data science project and start to think about this as an analytics-enabled experience.
During our session we will cover the technical elements that we overcame as a team to set-up a pipeline that ingests semi-structured and unstructured data at scale, performs analytics and produces digital experiences for multiple users.
This presentation will be particularly insightful for Data Scientists, Data Engineers and analytics leaders who are seeking to better understand how to augment the value of data for their organization
From SQL to Python - A Beginner's Guide to Making the Switch
Information about moving from a career in Data Analysis to a career in Data Science, with practical steps for getting started and coding in Python.
Similar to A Tale of a Data Driven Culture-(Gloria Lau, Timeful) (20)
Big Data at the Speed of Business: Lessons Learned from Leading at the Edge
Big Data at the Speed of Business: Lessons Learned from Leading at the Edge
Putting data science in your business a first utility feedback
Putting data science in your business a first utility feedback
O365 Practical Adoption Strategies - SPS Toronto 2017
O365 Practical Adoption Strategies - SPS Toronto 2017
Building a Data Cloud to enable Analytics & AI-Driven Innovation - Lak Lakshm...
Building a Data Cloud to enable Analytics & AI-Driven Innovation - Lak Lakshm...
Office 365 - Practical Adoption Strategies - SP Unite @ Haarlem
Office 365 - Practical Adoption Strategies - SP Unite @ Haarlem
The Right Data Warehouse: Automation Now, Business Value Thereafter
The Right Data Warehouse: Automation Now, Business Value Thereafter
Three signs your architecture is too small for big data. Camp IT December 2014
Three signs your architecture is too small for big data. Camp IT December 2014
They Said, We Said: Bridge the Communication Gap with Behavior-Driven Develop...
They Said, We Said: Bridge the Communication Gap with Behavior-Driven Develop...
Analytics-Enabled Experiences: The New Secret Weapon
Analytics-Enabled Experiences: The New Secret Weapon
From SQL to Python - A Beginner's Guide to Making the Switch
From SQL to Python - A Beginner's Guide to Making the Switch
More from Spark Summit
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
In this session we will present a Configurable FPGA-Based Spark SQL Acceleration Architecture. It is target to leverage FPGA highly parallel computing capability to accelerate Spark SQL Query and for FPGA’s higher power efficiency than CPU we can lower the power consumption at the same time. The Architecture consists of SQL query decomposition algorithms, fine-grained FPGA based Engine Units which perform basic computation of sub string, arithmetic and logic operations. Using SQL query decomposition algorithm, we are able to decompose a complex SQL query into basic operations and according to their patterns each is fed into an Engine Unit. SQL Engine Units are highly configurable and can be chained together to perform complex Spark SQL queries, finally one SQL query is transformed into a Hardware Pipeline. We will present the performance benchmark results comparing the queries with FGPA-Based Spark SQL Acceleration Architecture on XEON E5 and FPGA to the ones with Spark SQL Query on XEON E5 with 10X ~ 100X improvement and we will demonstrate one SQL query workload from a real customer.
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
In this talk, we’ll present techniques for visualizing large scale machine learning systems in Spark. These are techniques that are employed by Netflix to understand and refine the machine learning models behind Netflix’s famous recommender systems that are used to personalize the Netflix experience for their 99 millions members around the world. Essential to these techniques is Vegas, a new OSS Scala library that aims to be the “missing MatPlotLib” for Spark/Scala. We’ll talk about the design of Vegas and its usage in Scala notebooks to visualize Machine Learning Models.
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
This presentation introduces how we design and implement a real-time processing platform using latest Spark Structured Streaming framework to intelligently transform the production lines in the manufacturing industry. In the traditional production line there are a variety of isolated structured, semi-structured and unstructured data, such as sensor data, machine screen output, log output, database records etc. There are two main data scenarios: 1) Picture and video data with low frequency but a large amount; 2) Continuous data with high frequency. They are not a large amount of data per unit. However the total amount of them is very large, such as vibration data used to detect the quality of the equipment. These data have the characteristics of streaming data: real-time, volatile, burst, disorder and infinity. Making effective real-time decisions to retrieve values from these data is critical to smart manufacturing. The latest Spark Structured Streaming framework greatly lowers the bar for building highly scalable and fault-tolerant streaming applications. Thanks to the Spark we are able to build a low-latency, high-throughput and reliable operation system involving data acquisition, transmission, analysis and storage. The actual user case proved that the system meets the needs of real-time decision-making. The system greatly enhance the production process of predictive fault repair and production line material tracking efficiency, and can reduce about half of the labor force for the production lines.
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
Graph is on the rise and it’s time to start learning about scalable graph analytics! In this session we will go over two Spark-based Graph Analytics frameworks: Tinkerpop and GraphFrames. While both frameworks can express very similar traversals, they have different performance characteristics and APIs. In this Deep-Dive by example presentation, we will demonstrate some common traversals and explain how, at a Spark level, each traversal is actually computed under the hood! Learn both the fluent Gremlin API as well as the powerful GraphFrame Motif api as we show examples of both simultaneously. No need to be familiar with Graphs or Spark for this presentation as we’ll be explaining everything from the ground up!
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Building accurate machine learning models has been an art of data scientists, i.e., algorithm selection, hyper parameter tuning, feature selection and so on. Recently, challenges to breakthrough this “black-arts” have got started. In cooperation with our partner, NEC Laboratories America, we have developed a Spark-based automatic predictive modeling system. The system automatically searches the best algorithm, parameters and features without any manual work. In this talk, we will share how the automation system is designed to exploit attractive advantages of Spark. The evaluation with real open data demonstrates that our system can explore hundreds of predictive models and discovers the most accurate ones in minutes on a Ultra High Density Server, which employs 272 CPU cores, 2TB memory and 17TB SSD in 3U chassis. We will also share open challenges to learn such a massive amount of models on Spark, particularly from reliability and stability standpoints. This talk will cover the presentation already shown on Spark Summit SF’17 (#SFds5) but from more technical perspective.
Apache Spark and Tensorflow as a Service with Jim Dowling
In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.
Apache Spark and Tensorflow as a Service with Jim Dowling
In Sweden, from the Rise ICE Data Center at www.hops.site, we are providing to reseachers both Spark-as-a-Service and, more recently, Tensorflow-as-a-Service as part of the Hops platform. In this talk, we examine the different ways in which Tensorflow can be included in Spark workflows, from batch to streaming to structured streaming applications. We will analyse the different frameworks for integrating Spark with Tensorflow, from Tensorframes to TensorflowOnSpark to Databrick’s Deep Learning Pipelines. We introduce the different programming models supported and highlight the importance of cluster support for managing different versions of python libraries on behalf of users. We will also present cluster management support for sharing GPUs, including Mesos and YARN (in Hops Hadoop). Finally, we will perform a live demonstration of training and inference for a TensorflowOnSpark application written on Jupyter that can read data from either HDFS or Kafka, transform the data in Spark, and train a deep neural network on Tensorflow. We will show how to debug the application using both Spark UI and Tensorboard, and how to examine logs and monitor training.
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
With the rapid growth of available datasets, it is imperative to have good tools for extracting insight from big data. The Spark ML library has excellent support for performing at-scale data processing and machine learning experiments, but more often than not, Data Scientists find themselves struggling with issues such as: low level data manipulation, lack of support for image processing, text analytics and deep learning, as well as the inability to use Spark alongside other popular machine learning libraries. To address these pain points, Microsoft recently released The Microsoft Machine Learning Library for Apache Spark (MMLSpark), an open-source machine learning library built on top of SparkML that seeks to simplify the data science process and integrate SparkML Pipelines with deep learning and computer vision libraries such as the Microsoft Cognitive Toolkit (CNTK) and OpenCV. With MMLSpark, Data Scientists can build models with 1/10th of the code through Pipeline objects that compose seamlessly with other parts of the SparkML ecosystem. In this session, we explore some of the main lessons learned from building MMLSpark. Join us if you would like to know how to extend Pipelines to ensure seamless integration with SparkML, how to auto-generate Python and R wrappers from Scala Transformers and Estimators, how to integrate and use previously non-distributed libraries in a distributed manner and how to efficiently deploy a Spark library across multiple platforms.
Next CERN Accelerator Logging Service with Jakub Wozniak
The Next Accelerator Logging Service (NXCALS) is a new Big Data project at CERN aiming to replace the existing Oracle-based service.
The main purpose of the system is to store and present Controls/Infrastructure related data gathered from thousands of devices in the whole accelerator complex.
The data is used to operate the machines, improve their performance and conduct studies for new beam types or future experiments.
During this talk, Jakub will speak about NXCALS requirements and design choices that lead to the selected architecture based on Hadoop and Spark. He will present the Ingestion API, the abstractions behind the Meta-data Service and the Spark-based Extraction API where simple changes to the schema handling greatly improved the overall usability of the system. The system itself is not CERN specific and can be of interest to other companies or institutes confronted with similar Big Data problems.
Powering a Startup with Apache Spark with Kevin Kim
In Between (A mobile App for couples, downloaded 20M in Global), from daily batch for extracting metrics, analysis and dashboard. Spark is widely used by engineers and data analysts in Between, thanks to the performance and expendability of Spark, data operating has become extremely efficient. Entire team including Biz Dev, Global Operation, Designers are enjoying data results so Spark is empowering entire company for data driven operation and thinking. Kevin, Co-founder and Data Team leader of Between will be presenting how things are going in Between. Listeners will know how small and agile team is living with data (how we build organization, culture and technical base) after this presentation.
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
As common sense would suggest, weather has a definite impact on traffic. But how much? And under what circumstances? Can we improve traffic (congestion) prediction given weather data? Predictive traffic is envisioned to significantly impact how driver’s plan their day by alerting users before they travel, find the best times to travel, and over time, learn from new IoT data such as road conditions, incidents, etc. This talk will cover the traffic prediction work conducted jointly by IBM and the traffic data provider. As a part of this work, we conducted a case study over five large metropolitans in the US, 2.58 billion traffic records and 262 million weather records, to quantify the boost in accuracy of traffic prediction using weather data. We will provide an overview of our lambda architecture with Apache Spark being used to build prediction models with weather and traffic data, and Spark Streaming used to score the model and provide real-time traffic predictions. This talk will also cover a suite of extensions to Spark to analyze geospatial and temporal patterns in traffic and weather data, as well as the suite of machine learning algorithms that were used with Spark framework. Initial results of this work were presented at the National Association of Broadcasters meeting in Las Vegas in April 2017, and there is work to scale the system to provide predictions in over a 100 cities. Audience will learn about our experience scaling using Spark in offline and streaming mode, building statistical and deep-learning pipelines with Spark, and techniques to work with geospatial and time-series data.
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
In many cases, Big Data becomes just another buzzword because of the lack of tools that can support both the technological requirements for developing and deploying of the projects and/or the fluency of communication between the different profiles of people involved in the projects.
In this talk, we will present Moriarty, a set of tools for fast prototyping of Big Data applications that can be deployed in an Apache Spark environment. These tools support the creation of Big Data workflows using the already existing functional blocks or supporting the creation of new functional blocks. The created workflow can then be deployed in a Spark infrastructure and used through a REST API.
For better understanding of Moriarty, the prototyping process and the way it hides the Spark environment to the Big Data users and developers, we will present it together with a couple of examples based on a Industry 4.0 success cases and other on a logistic success case.
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Nielsen used Databricks to test new digital advertising rating methodologies on a large scale. Databricks allowed Nielsen to run analyses on thousands of advertising campaigns using both small panel data and large production data. This identified edge cases and performance gains faster than traditional methods. Using Databricks reduced the time required to test and deploy improved rating methodologies to benefit Nielsen's clients.
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Data lineage tracking is one of the significant problems that financial institutions face when using modern big data tools. This presentation describes Spline – a data lineage tracking and visualization tool for Apache Spark. Spline captures and stores lineage information from internal Spark execution plans and visualizes it in a user-friendly manner.
Goal Based Data Production with Sim Simeonov
Since the invention of SQL and relational databases, data production has been about specifying how data is transformed through queries. While Apache Spark can certainly be used as a general distributed query engine, the power and granularity of Spark’s APIs enables a revolutionary increase in data engineering productivity: goal-based data production. Goal-based data production concerns itself with specifying WHAT the desired result is, leaving the details of HOW the result is achieved to a smart data warehouse running on top of Spark. That not only substantially increases productivity, but also significantly expands the audience that can work directly with Spark: from developers and data scientists to technical business users. With specific data and architecture patterns spanning the range from ETL to machine learning data prep and with live demos, this session will demonstrate how Spark users can gain the benefits of goal-based data production.
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Have you imagined a simple machine learning solution able to prevent revenue leakage and monitor your distributed application? To answer this question, we offer a practical and a simple machine learning solution to create an intelligent monitoring application based on simple data analysis using Apache Spark MLlib. Our application uses linear regression models to make predictions and check if the platform is experiencing any operational problems that can impact in revenue losses. The application monitor distributed systems and provides notifications stating the problem detected, that way users can operate quickly to avoid serious problems which directly impact the company’s revenue and reduce the time for action. We will present an architecture for not only a monitoring system, but also an active actor for our outages recoveries. At the end of the presentation you will have access to our training program source code and you will be able to adapt and implement in your company. This solution already helped to prevent about US$3mi in losses last year.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to use Redis with Apache Spark is a technical tutorial designed to address integrating Redis with an Apache Spark deployment to increase the performance of serving complex decision models. To set the context for the session, we start with a quick introduction to Redis and the capabilities Redis provides. We cover the basic data types provided by Redis and cover the module system. Using an ad serving use-case, we look at how Redis can improve the performance and reduce the cost of using complex ML-models in production. Attendees will be guided through the key steps of setting up and integrating Redis with Spark, including how to train a model using Spark then load and serve it using Redis, as well as how to work with the Spark Redis module. The capabilities of the Redis Machine Learning Module (redis-ml) will be discussed focusing primarily on decision trees and regression (linear and logistic) with code examples to demonstrate how to use these feature. At the end of the session, developers should feel confident building a prototype/proof-of-concept application using Redis and Spark. Attendees will understand how Redis complements Spark and how to use Redis to serve complex, ML-models with high performance.
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Here we present a general supervised framework for record deduplication and author-disambiguation via Spark. This work differentiates itself by – Application of Databricks and AWS makes this a scalable implementation. Compute resources are comparably lower than traditional legacy technology using big boxes 24/7. Scalability is crucial as Elsevier’s Scopus data, the biggest scientific abstract repository, covers roughly 250 million authorships from 70 million abstracts covering a few hundred years. – We create a fingerprint for each content by deep learning and/or word2vec algorithms to expedite pairwise similarity calculation. These encoders substantially reduce compute time while maintaining semantic similarity (unlike traditional TFIDF or predefined taxonomies). We will briefly discuss how to optimize word2vec training with high parallelization. Moreover, we show how these encoders can be used to derive a standard representation for all our entities namely such as documents, authors, users, journals, etc. This standard representation can simplify the recommendation problem into a pairwise similarity search and hence it can offer a basic recommender for cross-product applications where we may not have a dedicate recommender engine designed. – Traditional author-disambiguation or record deduplication algorithms are batch-processing with small to no training data. However, we have roughly 25 million authorships that are manually curated or corrected upon user feedback. Hence, it is crucial to maintain historical profiles and hence we have developed a machine learning implementation to deal with data streams and process them in mini batches or one document at a time. We will discuss how to measure the accuracy of such a system, how to tune it and how to process the raw data of pairwise similarity function into final clusters. Lessons learned from this talk can help all sort of companies where they want to integrate their data or deduplicate their user/customer/product databases.
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
The use of large-scale machine learning and data mining methods is becoming ubiquitous in many application domains ranging from business intelligence and bioinformatics to self-driving cars. These methods heavily rely on matrix computations, and it is hence critical to make these computations scalable and efficient. These matrix computations are often complex and involve multiple steps that need to be optimized and sequenced properly for efficient execution. This work presents new efficient and scalable matrix processing and optimization techniques based on Spark. The proposed techniques estimate the sparsity of intermediate matrix-computation results and optimize communication costs. An evaluation plan generator for complex matrix computations is introduced as well as a distributed plan optimizer that exploits dynamic cost-based analysis and rule-based heuristics The result of a matrix operation will often serve as an input to another matrix operation, thus defining the matrix data dependencies within a matrix program. The matrix query plan generator produces query execution plans that minimize memory usage and communication overhead by partitioning the matrix based on the data dependencies in the execution plan. We implemented the proposed matrix techniques inside the Spark SQL, and optimize the matrix execution plan based on Spark SQL Catalyst. We conduct case studies on a series of ML models and matrix computations with special features on different datasets. These are PageRank, GNMF, BFGS, sparse matrix chain multiplications, and a biological data analysis. The open-source library ScaLAPACK and the array-based database SciDB are used for performance evaluation. Our experiments are performed on six real-world datasets are: social network data ( e.g., soc-pokec, cit-Patents, LiveJournal), Twitter2010, Netflix recommendation data, and 1000 Genomes Project sample. Experiments demonstrate that our proposed techniques achieve up to an order-of-magnitude performance.
More from Spark Summit (20)
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim Dowling
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub Wozniak
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin Kim
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization...
Recently uploaded
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
Open Source Contributions to Postgres: The Basics POSETTE 2024
Postgres is the most advanced open-source database in the world and it's supported by a community, not a single company. So how does this work? How does code actually get into Postgres? I recently had a patch submitted and committed and I want to share what I learned in that process. I’ll give you an overview of Postgres versions and how the underlying project codebase functions. I’ll also show you the process for submitting a patch and getting that tested and committed.
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Harness the power of AI-backed reports, benchmarking and data analysis to predict trends and detect anomalies in your marketing efforts.
Peter Caputa, CEO at Databox, reveals how you can discover the strategies and tools to increase your growth rate (and margins!).
From metrics to track to data habits to pick up, enhance your reporting for powerful insights to improve your B2B tech company's marketing.
- - -
This is the webinar recording from the June 2024 HubSpot User Group (HUG) for B2B Technology USA.
Watch the video recording at https://youtu.be/5vjwGfPN9lw
Sign up for future HUG events at https://events.hubspot.com/b2b-technology-usa/
一比一原版(UO毕业证)渥太华大学毕业证如何办理
UO毕业证录取书【微信95270640】购买（渥太华大学毕业证成绩单硕士学历）Q微信95270640代办UO学历认证留信网伪造渥太华大学学位证书精仿渥太华大学本科/硕士文凭证书补办渥太华大学 diplomaoffer,Transcript购买渥太华大学毕业证成绩单购买UO假毕业证学位证书购买伪造渥太华大学文凭证书学位证书,专业办理雅思、托福成绩单，学生ID卡，在读证明，海外各大学offer录取通知书，毕业证书，成绩单，文凭等材料:1:1完美还原毕业证、offer录取通知书、学生卡等各种在读或毕业材料的防伪工艺（包括 烫金、烫银、钢印、底纹、凹凸版、水印、防伪光标、热敏防伪、文字图案浮雕，激光镭射，紫外荧光，温感光标）学校原版上有的工艺我们一样不会少，不论是老版本还是最新版本，都能保证最高程度还原，力争完美以求让所有同学都能享受到完美的品质服务。
文凭办理流程：
1客户提供办理信息：姓名生日专业学位毕业时间等（如信息不确定可以咨询顾问：微信95270640我们有专业老师帮你查询）；
2开始安排制作毕业证成绩单电子图；
3毕业证成绩单电子版做好以后发送给您确认；
4毕业证成绩单电子版您确认信息无误之后安排制作成品；
5成品做好拍照或者视频给您确认；
6快递给客户（国内顺丰国外DHLUPS等快读邮寄）。
7完成交易删除客户资料
高精端提供以下服务：
一：渥太华大学渥太华大学毕业证文凭证书全套材料从防伪到印刷水印底纹到钢印烫金
二：真实使馆认证（留学人员回国证明）使馆存档
三：真实教育部认证教育部存档教育部留服网站可查
四：留信认证留学生信息网站可查
五：与学校颁发的相关证件1:1纸质尺寸制定（定期向各大院校毕业生购买最新版本毕,业证成绩单保证您拿到的是鲁昂大学内部最新版本毕业证成绩单微信95270640）
A.为什么留学生需要操作留信认证?
留信认证全称全国留学生信息服务网认证,隶属于北京中科院。①留信认证门槛条件更低,费用更美丽,并且包过,完单周期短,效率高②留信认证虽然不能去国企,但是一般的公司都没有问题,因为国内很多公司连基本的留学生学历认证都不了解。这对于留学生来说,这就比自己光拿一个证书更有说服力,因为留学学历可以在留信网站上进行查询!
B.为什么我们提供的毕业证成绩单具有使用价值？
查询留服认证是国内鉴别留学生海外学历的唯一途径但认证只是个体行为不是所有留学生都操作所以没有办理认证的留学生的学历在国内也是查询不到的他们也仅仅只有一张文凭。所以这时候我们提供的和学校颁发的一模一样的毕业证成绩单就有了使用价值。只硕大的蛇皮袋手里拎着长铁钩正站在门口朝黑色的屋内张望不好坏人小偷山娃一怔却也灵机一动立马仰起头双手拢在嘴边朝楼上大喊：“爸爸爸——有人找——那人一听朝山娃尴尬地笑笑悻悻地走了山娃立马“嘭的一声将铁门锁死心却咚咚地乱跳当山娃跟父亲说起这事时父亲很吃惊抚摸着山娃的头说还好醒得及时要不家早被人掏空了到时连电视也没得看啰不过父亲还是夸山娃能临危不乱随机应变有胆有谋山娃笑笑说那都是书上学的看童话和小说时多
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
by
Timothy Spann
Principal Developer Advocate
https://budapestdata.hu/2024/en/
https://budapestml.hu/2024/en/
tim.spann@zilliz.com
https://www.linkedin.com/in/timothyspann/
https://x.com/paasdev
https://github.com/tspannhw
https://www.youtube.com/@flank-stack
milvus
vector database
gen ai
generative ai
deep learning
machine learning
apache nifi
apache pulsar
apache kafka
apache flink
Experts live - Improving user adoption with AI
Bekijk de slides van onze sessie Enhancing Modern Workplace Efficiency op Experts Live 2024.
The Building Blocks of QuestDB, a Time Series Database
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
End-to-end pipeline agility - Berlin Buzzwords 2024
We describe how we achieve high change agility in data engineering by eliminating the fear of breaking downstream data pipelines through end-to-end pipeline testing, and by using schema metaprogramming to safely eliminate boilerplate involved in changes that affect whole pipelines.
A quick poll on agility in changing pipelines from end to end indicated a huge span in capabilities. For the question "How long time does it take for all downstream pipelines to be adapted to an upstream change," the median response was 6 months, but some respondents could do it in less than a day. When quantitative data engineering differences between the best and worst are measured, the span is often 100x-1000x, sometimes even more.
A long time ago, we suffered at Spotify from fear of changing pipelines due to not knowing what the impact might be downstream. We made plans for a technical solution to test pipelines end-to-end to mitigate that fear, but the effort failed for cultural reasons. We eventually solved this challenge, but in a different context. In this presentation we will describe how we test full pipelines effectively by manipulating workflow orchestration, which enables us to make changes in pipelines without fear of breaking downstream.
Making schema changes that affect many jobs also involves a lot of toil and boilerplate. Using schema-on-read mitigates some of it, but has drawbacks since it makes it more difficult to detect errors early. We will describe how we have rejected this tradeoff by applying schema metaprogramming, eliminating boilerplate but keeping the protection of static typing, thereby further improving agility to quickly modify data pipelines without fear.
一比一原版(UCSF文凭证书)旧金山分校毕业证如何办理
毕业原版【微信:176555708】【(UCSF毕业证书)旧金山分校毕业证】【微信:176555708】成绩单、外壳、offer、留信学历认证（永久存档真实可查）采用学校原版纸张、特殊工艺完全按照原版一比一制作（包括：隐形水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠，文字图案浮雕，激光镭射，紫外荧光，温感，复印防伪）行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备，十五年致力于帮助留学生解决难题，业务范围有加拿大、英国、澳洲、韩国、美国、新加坡，新西兰等学历材料，包您满意。
【我们承诺采用的是学校原版纸张（纸质、底色、纹路），我们拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！】
【业务选择办理准则】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理一份就读学校的毕业证【微信176555708】文凭即可
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理一份毕业证【微信176555708】即可
三、进国企，银行，事业单位，考公务员等等，这些单位是必需要提供真实教育部认证的，办理教育部认证所需资料众多且烦琐，所有材料您都必须提供原件，我们凭借丰富的经验，快捷的绿色通道帮您快速整合材料，让您少走弯路。
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
留信网服务项目：
1、留学生专业人才库服务（留信分析）
2、国（境）学习人员提供就业推荐信服务
3、留学人员区块链存储服务
→ 【关于价格问题（保证一手价格）】
我们所定的价格是非常合理的，而且我们现在做得单子大多数都是代理和回头客户介绍的所以一般现在有新的单子 我给客户的都是第一手的代理价格，因为我想坦诚对待大家 不想跟大家在价格方面浪费时间
对于老客户或者被老客户介绍过来的朋友，我们都会适当给一些优惠。
选择实体注册公司办理，更放心，更安全！我们的承诺：客户在留信官方认证查询网站查询到认证通过结果后付款，不成功不收费！
University of New South Wales degree offer diploma Transcript
澳洲UNSW毕业证书制作新南威尔士大学假文凭定制Q微168899991做UNSW留信网教留服认证海牙认证改UNSW成绩单GPA做UNSW假学位证假文凭高仿毕业证申请新南威尔士大学University of New South Wales degree offer diploma Transcript
Intelligence supported media monitoring in veterinary medicine
Media monitoring in veterinary medicien
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
学校原件一模一样【微信：741003700 】《(英国UCA毕业证书)创意艺术大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
The Modern Marketing Reckoner (MMR) is a comprehensive resource packed with POVs from 60+ industry leaders on how AI is transforming the 4 key pillars of marketing – product, place, price and promotions.
办(uts毕业证书)悉尼科技大学毕业证学历证书原版一模一样
原版一模一样【微信：741003700 】【(uts毕业证书)悉尼科技大学毕业证学历证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
原版制作(unimelb毕业证书)墨尔本大学毕业证Offer一模一样
学校原件一模一样【微信：741003700 】《(unimelb毕业证书)墨尔本大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原。
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
【主营项目】
一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度)
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Learn SQL from basic queries to Advance queries
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Population Growth in Bataan: The effects of population growth around rural pl...
A population analysis specific to Bataan.
Recently uploaded (20)
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data Lake
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
The Building Blocks of QuestDB, a Time Series Database
The Building Blocks of QuestDB, a Time Series Database
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
Intelligence supported media monitoring in veterinary medicine
Intelligence supported media monitoring in veterinary medicine
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
A Tale of a Data Driven Culture-(Gloria Lau, Timeful)
- 1. A Tale of a Data-Driven Culture Gloria Lau VP of Data, Timeful (Acquired by Google) Keynote @ Spark Summit, 2015
- 3. This talk is • a hindsight on why Spark matters to non-engineers too
- 4. Every company aspires to be more data-driven • What is involved? • Analytics • Deep dives • Explorations • on production data real time with speed
- 5. Common implementation • Hire more data scientists, turn them into SQL monkeys
- 6. Problem • everyone should have be able to dig into data • long innovation cycle • monkeys do not last
- 7. Truly data-driven • Say NO to job descriptions • easier said than done
- 8. How to throw out job descriptions
- 9. Friction = Bad Less friction = Less bad • Most in your company want answers from data • Most in your company are capable of finding answers from data, given the right tool • Your goal is to find that tool that minimizes friction to data
- 10. Biggest friction to data • Lack of speed • Imagine your favorite search engine’s SLA is… 10 seconds
- 11. Experiment • Kill the SQL monkey by turning everyone into their own SQL monkeys • Give PMs, designers and everyone access to Databricks. • Fail fast. Learn from examples. Visualize. Inspect. Fail again. Iterate. • You have to trust your team to be capable. Give them the tool and they shall learn to be data-driven.
- 12. Experiment Results • Everyone spends time with data, forms their own insights, and no longer needs a crutch to explore • Data scientists* focus on building data products • feature engineering etc also benefits from speed • Much shorter innovation cycle. * disclaimer: very liberal definition throughout
- 13. What did we learn • Every company aspires to be data-driven. You hire SQL monkeys. Wrong. • Democratize thou data by removing friction. Speed should be top on your list. • #SQLMonkeyEveryone