Agile Data Science is a lean methodology that is adopted from Agile Software Development. At the core it centers around people, interactions, and building minimally viable products to ship fast and often to solicit customer feedback. In this presentation, I describe how this work was done in the past with examples. Get started today with our help by visiting http://www.alpinenow.com
Curious about Data Science? Self-taught on some aspects, but missing the big picture? Well, you’ve got to start somewhere and this session is the place to do it.
This session will cover, at a layman’s level, some of the basic concepts of Data Science. In a conversational format, we will discuss: What are the differences between Big Data and Data Science – and why aren’t they the same thing? What distinguishes descriptive, predictive, and prescriptive analytics? What purpose do predictive models serve in a practical context? What kinds of models are there and what do they tell us? What is the difference between supervised and unsupervised learning? What are some common pitfalls that turn good ideas into bad science?
During this session, attendees will learn the difference between k-nearest neighbor and k-means clustering, understand the reasons why we do normalize and don’t overfit, and grasp the meaning of No Free Lunch.
To be successful as a data science team, we need to continuously deliver data-driven insights and data products that generate business value. Identifying the best opportunities and building solutions that actually get used in production requires very close collaboration with business users and subject matter experts. What can we learn from agile software development methodologies, and how can we apply them to data science projects?
Is Agile Data Science just two buzzwords put together? I argue that agile is a very practical and applicable methodology, that does work well in the real world for all sorts of Analytics and Data Science workflows.
http://theinnovationenterprise.com/summits/digital-web-analytics-summit-london-2015/schedule
Not a whole lot of organization have fitted Hadoop into a Scrum Development Framework. This deck provides the expectations around what needs to be done in Hadoop in order to support Scrum.
Estamos presenciando inovações tecnológicas que possibilitam utilizar ciência dos dados sem a necessidade de antecipar grandes investimentos. Este contexto facilita a adoção de práticas e valores ágeis que encorajam a antecipação de insights e aprendizado contínuo. Nesta palestra, iremos abordar temas como times multi-funcionais, práticas ágeis de engenharia de software e desenvolvimento iterativo, incremental e colaborativo no contexto de produtos e soluções de ciência dos dados.
Curious about Data Science? Self-taught on some aspects, but missing the big picture? Well, you’ve got to start somewhere and this session is the place to do it.
This session will cover, at a layman’s level, some of the basic concepts of Data Science. In a conversational format, we will discuss: What are the differences between Big Data and Data Science – and why aren’t they the same thing? What distinguishes descriptive, predictive, and prescriptive analytics? What purpose do predictive models serve in a practical context? What kinds of models are there and what do they tell us? What is the difference between supervised and unsupervised learning? What are some common pitfalls that turn good ideas into bad science?
During this session, attendees will learn the difference between k-nearest neighbor and k-means clustering, understand the reasons why we do normalize and don’t overfit, and grasp the meaning of No Free Lunch.
To be successful as a data science team, we need to continuously deliver data-driven insights and data products that generate business value. Identifying the best opportunities and building solutions that actually get used in production requires very close collaboration with business users and subject matter experts. What can we learn from agile software development methodologies, and how can we apply them to data science projects?
Is Agile Data Science just two buzzwords put together? I argue that agile is a very practical and applicable methodology, that does work well in the real world for all sorts of Analytics and Data Science workflows.
http://theinnovationenterprise.com/summits/digital-web-analytics-summit-london-2015/schedule
Not a whole lot of organization have fitted Hadoop into a Scrum Development Framework. This deck provides the expectations around what needs to be done in Hadoop in order to support Scrum.
Estamos presenciando inovações tecnológicas que possibilitam utilizar ciência dos dados sem a necessidade de antecipar grandes investimentos. Este contexto facilita a adoção de práticas e valores ágeis que encorajam a antecipação de insights e aprendizado contínuo. Nesta palestra, iremos abordar temas como times multi-funcionais, práticas ágeis de engenharia de software e desenvolvimento iterativo, incremental e colaborativo no contexto de produtos e soluções de ciência dos dados.
Leveraging Open Source Automated Data Science ToolsDomino Data Lab
The data science process seeks to transform and empower organizations by finding and exploiting market inefficiencies and potentially hidden opportunities, but this is often an expensive, tedious process. However, many steps can be automated to provide a streamlined experience for data scientists. Eduardo Arino de la Rubia explores the tools being created by the open source community to free data scientists from tedium, enabling them to work on the high-value aspects of insight creation and impact validation.
The promise of the automated statistician is almost as old as statistics itself. From the creations of vast tables, which saved the labor of calculation, to modern tools which automatically mine datasets for correlations, there has been a considerable amount of advancement in this field. Eduardo compares and contrasts a number of open source tools, including TPOT and auto-sklearn for automated model generation and scikit-feature for feature generation and other aspects of the data science workflow, evaluates their results, and discusses their place in the modern data science workflow.
Along the way, Eduardo outlines the pitfalls of automated data science and applications of the “no free lunch” theorem and dives into alternate approaches, such as end-to-end deep learning, which seek to leverage massive-scale computing and architectures to handle automatic generation of features and advanced models.
A Practical-ish Introduction to Data ScienceMark West
In this talk I will share insights and knowledge that I have gained from building up a Data Science department from scratch. This talk will be split into three sections:
1. I'll begin by defining what Data Science is, how it is related to Machine Learning and share some tips for introducing Data Science to your organisation.
2. Next up well run through some commonly used Machine Learning algorithms used by Data Scientists, along with examples for use cases where these algorithms can be applied.
3. The final third of the talk will be a demonstration of how you can quickly get started with Data Science and Machine Learning using Python and the Open Source scikit-learn Library.
Presentation at Data ScienceTech Institute campuses, Paris and Nice, May 2016 , including Intro, Data Science History and Terms; 10 Real-World Data Science Lessons; Data Science Now: Polls & Trends; Data Science Roles; Data Science Job Trends; and Data Science Future
Intro to Data Science for Non-Data ScientistsSri Ambati
Erin LeDell and Chen Huang's presentations from the Intro to Data Science for Non-Data Scientists Meetup at H2O HQ on 08.20.15
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
A presentation delivered by Mohammed Barakat on the 2nd Jordanian Continuous Improvement Open Day in Amman. The presentation is about Data Science and was delivered on 3rd October 2015.
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Data Science London
What 'kind of things' does a data scientist do? What are the foundations and principles of data science? What is a Data Product? What does the data science process looks like? Learning from data: Data Modeling or Algorithmic Modeling? - talk by Carlos Somohano @ds_ldn at The Cloud and Big Data: HDInsight on Azure London 25/01/13
Two hour lecture I gave at the Jyväskylä Summer School. The purpose of the talk is to give a quick non-technical overview of concepts and methodologies in data science. Topics include a wide overview of both pattern mining and machine learning.
See also Part 2 of the lecture: Industrial Data Science. You can find it in my profile (click the face)
Creating a Data Science Ecosystem for Scientific, Societal and Educational Im...Ilkay Altintas, Ph.D.
The new era of data science is here. Our lives and society are continuously transformed by our ability to collect data in a systematic fashion and turn that into value. The opportunities created by this change also comes with challenges that push for new and innovative data management and analytical methods as well as translating these new methods to applications in many areas that impact science, society, and education. Collaboration and ability of multi-disciplinary teams to work together and communicate to bring together the best of their knowledge in business, data and computing is vital for impactful solutions. This talk will discusses a reference ecosystem and question-driven methodology, called PPODS, to make impactful data science applications in many fields with specific examples in hazards, smart cities and biomedical research.
What Is Data Science? Data Science Course - Data Science Tutorial For Beginne...Edureka!
This Edureka Data Science course slides will take you through the basics of Data Science - why Data Science, what is Data Science, use cases, BI vs Data Science, Data Science tools and Data Science lifecycle process. This is ideal for beginners to get started with learning data science.
You can read the blog here: https://goo.gl/OoDCxz
You can also take a complete structured training, check out the details here: https://goo.gl/AfxwBc
Introduction to various data science. From the very beginning of data science idea, to latest designs, changing trends, technologies what make then to the application that are already in real world use as we of now.
Una breve introduzione alla data science e al machine learning con un'enfasi sugli scenari applicativi, da quelli tradizionali a quelli più innovativi. La overview copre la definizione di base di data science, una overview del machine learning e esempi su scenari tradizionali, Recommender systems e Social Network Analysis, IoT e Deep Learning
Keynote - An overview on Big Data & Data Science - Dr Gregory Piatetsky-ShapiroData ScienceTech Institute
Data Science Tech Institute - Big Data and Data Science Conference around Dr Gregory Piatetsky-Shapiro.
Keynote - An overview on Big Data & Data Science Dr Gregory Piatetsky-Shapiro - KDnuggets.com Founder & Editor.
Paris May 23rd & Nice May 26th 2016 @ Data ScienceTech Institute (https://www.datasciencetech.institute/)
Introduction to Data Science and AnalyticsSrinath Perera
This webinar serves as an introduction to WSO2 Summer School. It will discuss how to build a pipeline for your organization and for each use case, and the technology and tooling choices that need to be made for the same.
This session will explore analytics under four themes:
Hindsight (what happened)
Oversight (what is happening)
Insight (why is it happening)
Foresight (what will happen)
Recording http://t.co/WcMFEAJHok
From the webinar presentation "Data Science: Not Just for Big Data", hosted by Kalido and presented by:
David Smith, Data Scientist at Revolution Analytics, and
Gregory Piatetsky, Editor, KDnuggets
These are the slides for David Smith's portion of the presentation.
Watch the full webinar at:
http://www.kalido.com/data-science.htm
Key Performance Indicators (KPIs) are typically used to measure the performance of a firm both at the strategic and operational level. KPIs often form the basis of a firm's goal management system: Each KPI is assigned and owner in the firm's top management, who is then responsible for reaching a particular target.
This case study shows that in order to find realistic targets for KPIs the firm's management needs a clear understanding of how the KPIs really measure the firm's performance, how they depend on each other and how KPI targets need to change over time in order to ensure the firm's ultimate goals are reached.
Our client in this case is a successful IT professional service firm that approached us because they wanted to find ways of increasing their growth rates organically. We took a holistic approach to analysing the firm's business model and KPIs using System Dynamics. The case study illustrates this using concrete examples, in particular regarding KPIs such as the firm’s leverage, the average fee, the utilisation, and the profit margin.
Intense competition and slow growth in mature markets have magnified uncertainty and put pressure on costs, just as regulators are escalating their demands. Research shows that CFOs and other senior finance executives believe that their function can play a key role but the ability to impact these challenges depends on levels of maturity and preparedness, which vary widely across companies and industries, as well by sub-functions. Here are the key findings from our research on how enterprises are driving transformation to achieve business impact.
Leveraging Open Source Automated Data Science ToolsDomino Data Lab
The data science process seeks to transform and empower organizations by finding and exploiting market inefficiencies and potentially hidden opportunities, but this is often an expensive, tedious process. However, many steps can be automated to provide a streamlined experience for data scientists. Eduardo Arino de la Rubia explores the tools being created by the open source community to free data scientists from tedium, enabling them to work on the high-value aspects of insight creation and impact validation.
The promise of the automated statistician is almost as old as statistics itself. From the creations of vast tables, which saved the labor of calculation, to modern tools which automatically mine datasets for correlations, there has been a considerable amount of advancement in this field. Eduardo compares and contrasts a number of open source tools, including TPOT and auto-sklearn for automated model generation and scikit-feature for feature generation and other aspects of the data science workflow, evaluates their results, and discusses their place in the modern data science workflow.
Along the way, Eduardo outlines the pitfalls of automated data science and applications of the “no free lunch” theorem and dives into alternate approaches, such as end-to-end deep learning, which seek to leverage massive-scale computing and architectures to handle automatic generation of features and advanced models.
A Practical-ish Introduction to Data ScienceMark West
In this talk I will share insights and knowledge that I have gained from building up a Data Science department from scratch. This talk will be split into three sections:
1. I'll begin by defining what Data Science is, how it is related to Machine Learning and share some tips for introducing Data Science to your organisation.
2. Next up well run through some commonly used Machine Learning algorithms used by Data Scientists, along with examples for use cases where these algorithms can be applied.
3. The final third of the talk will be a demonstration of how you can quickly get started with Data Science and Machine Learning using Python and the Open Source scikit-learn Library.
Presentation at Data ScienceTech Institute campuses, Paris and Nice, May 2016 , including Intro, Data Science History and Terms; 10 Real-World Data Science Lessons; Data Science Now: Polls & Trends; Data Science Roles; Data Science Job Trends; and Data Science Future
Intro to Data Science for Non-Data ScientistsSri Ambati
Erin LeDell and Chen Huang's presentations from the Intro to Data Science for Non-Data Scientists Meetup at H2O HQ on 08.20.15
- Powered by the open source machine learning software H2O.ai. Contributors welcome at: https://github.com/h2oai
- To view videos on H2O open source machine learning software, go to: https://www.youtube.com/user/0xdata
A presentation delivered by Mohammed Barakat on the 2nd Jordanian Continuous Improvement Open Day in Amman. The presentation is about Data Science and was delivered on 3rd October 2015.
Big Data [sorry] & Data Science: What Does a Data Scientist Do?Data Science London
What 'kind of things' does a data scientist do? What are the foundations and principles of data science? What is a Data Product? What does the data science process looks like? Learning from data: Data Modeling or Algorithmic Modeling? - talk by Carlos Somohano @ds_ldn at The Cloud and Big Data: HDInsight on Azure London 25/01/13
Two hour lecture I gave at the Jyväskylä Summer School. The purpose of the talk is to give a quick non-technical overview of concepts and methodologies in data science. Topics include a wide overview of both pattern mining and machine learning.
See also Part 2 of the lecture: Industrial Data Science. You can find it in my profile (click the face)
Creating a Data Science Ecosystem for Scientific, Societal and Educational Im...Ilkay Altintas, Ph.D.
The new era of data science is here. Our lives and society are continuously transformed by our ability to collect data in a systematic fashion and turn that into value. The opportunities created by this change also comes with challenges that push for new and innovative data management and analytical methods as well as translating these new methods to applications in many areas that impact science, society, and education. Collaboration and ability of multi-disciplinary teams to work together and communicate to bring together the best of their knowledge in business, data and computing is vital for impactful solutions. This talk will discusses a reference ecosystem and question-driven methodology, called PPODS, to make impactful data science applications in many fields with specific examples in hazards, smart cities and biomedical research.
What Is Data Science? Data Science Course - Data Science Tutorial For Beginne...Edureka!
This Edureka Data Science course slides will take you through the basics of Data Science - why Data Science, what is Data Science, use cases, BI vs Data Science, Data Science tools and Data Science lifecycle process. This is ideal for beginners to get started with learning data science.
You can read the blog here: https://goo.gl/OoDCxz
You can also take a complete structured training, check out the details here: https://goo.gl/AfxwBc
Introduction to various data science. From the very beginning of data science idea, to latest designs, changing trends, technologies what make then to the application that are already in real world use as we of now.
Una breve introduzione alla data science e al machine learning con un'enfasi sugli scenari applicativi, da quelli tradizionali a quelli più innovativi. La overview copre la definizione di base di data science, una overview del machine learning e esempi su scenari tradizionali, Recommender systems e Social Network Analysis, IoT e Deep Learning
Keynote - An overview on Big Data & Data Science - Dr Gregory Piatetsky-ShapiroData ScienceTech Institute
Data Science Tech Institute - Big Data and Data Science Conference around Dr Gregory Piatetsky-Shapiro.
Keynote - An overview on Big Data & Data Science Dr Gregory Piatetsky-Shapiro - KDnuggets.com Founder & Editor.
Paris May 23rd & Nice May 26th 2016 @ Data ScienceTech Institute (https://www.datasciencetech.institute/)
Introduction to Data Science and AnalyticsSrinath Perera
This webinar serves as an introduction to WSO2 Summer School. It will discuss how to build a pipeline for your organization and for each use case, and the technology and tooling choices that need to be made for the same.
This session will explore analytics under four themes:
Hindsight (what happened)
Oversight (what is happening)
Insight (why is it happening)
Foresight (what will happen)
Recording http://t.co/WcMFEAJHok
From the webinar presentation "Data Science: Not Just for Big Data", hosted by Kalido and presented by:
David Smith, Data Scientist at Revolution Analytics, and
Gregory Piatetsky, Editor, KDnuggets
These are the slides for David Smith's portion of the presentation.
Watch the full webinar at:
http://www.kalido.com/data-science.htm
Key Performance Indicators (KPIs) are typically used to measure the performance of a firm both at the strategic and operational level. KPIs often form the basis of a firm's goal management system: Each KPI is assigned and owner in the firm's top management, who is then responsible for reaching a particular target.
This case study shows that in order to find realistic targets for KPIs the firm's management needs a clear understanding of how the KPIs really measure the firm's performance, how they depend on each other and how KPI targets need to change over time in order to ensure the firm's ultimate goals are reached.
Our client in this case is a successful IT professional service firm that approached us because they wanted to find ways of increasing their growth rates organically. We took a holistic approach to analysing the firm's business model and KPIs using System Dynamics. The case study illustrates this using concrete examples, in particular regarding KPIs such as the firm’s leverage, the average fee, the utilisation, and the profit margin.
Intense competition and slow growth in mature markets have magnified uncertainty and put pressure on costs, just as regulators are escalating their demands. Research shows that CFOs and other senior finance executives believe that their function can play a key role but the ability to impact these challenges depends on levels of maturity and preparedness, which vary widely across companies and industries, as well by sub-functions. Here are the key findings from our research on how enterprises are driving transformation to achieve business impact.
Agile Data Science 2.0 (O'Reilly 2017) defines a methodology and a software stack with which to apply the methods. *The methodology* seeks to deliver data products in short sprints by going meta and putting the focus on the applied research process itself. *The stack* is but an example of one meeting the requirements that it be utterly scalable and utterly efficient in use by application developers as well as data engineers. It includes everything needed to build a full-blown predictive system: Apache Spark, Apache Kafka, Apache Incubating Airflow, MongoDB, ElasticSearch, Apache Parquet, Python/Flask, JQuery. This talk will cover the full lifecycle of large data application development and will show how to use lessons from agile software engineering to apply data science using this full-stack to build better analytics applications. The entire lifecycle of big data application development is discussed. The system starts with plumbing, moving on to data tables, charts and search, through interactive reports, and building towards predictions in both batch and realtime (and defining the role for both), the deployment of predictive systems and how to iteratively improve predictions that prove valuable.
Practical Data Science the WPC Healthcare Strategy for Delivering Meaningful ...Damian R. Mingle, MBA
Learning to make use of Jupyter to document your Data Science process - real time - and in whatever programming language you want! Using this methodology will allow you to provide insights that help your organization make better decisions to solve their business problems.
Creating a clearly articulated data strategy—a roadmap of technology-driven capability investments prioritized to deliver value—helps ensure from the get-go that you are focusing on the right things, so that your work with data has a business impact. In this presentation, the experts at Silicon Valley Data Science share their approach for crafting an actionable and flexible data strategy to maximize business value.
How Celtra Optimizes its Advertising Platformwith DatabricksGrega Kespret
Leading brands such as Pepsi and Macy’s use Celtra’s technology platform for brand advertising. To inform better product design and resolve issues faster, Celtra relies on Databricks to gather insights from large-scale, diverse, and complex raw event data. Learn how Celtra uses Databricks to simplify their Spark deployment, achieve faster project turnaround time, and empower people to make data-driven decisions.
In this webinar, you will learn how Databricks helps Celtra to:
- Utilize Apache Spark to power their production analytics pipeline.
- Build a “Just-in-Time” data warehouse to analyze diverse data sources such as Elastic Load Balancer access logs, raw tracking events, operational data, and reportable metrics.
- Go beyond simple counting and group events into sequences (i.e., sessionization) and perform more complex analysis such as funnel analytics.
Understanding What’s Possible: Getting Business Value from Big Data QuicklyInside Analysis
The Briefing Room with David Loshin and OpenText
Live Webcast April 14, 2015
https://bloorgroup.webex.com/bloorgroup/onstage/g.php?MTID=e079dc562543a394c5c5d0588e7cd9152
To be successful and practical in delivering meaningful insights, companies must embrace the three pillars of enterprise analytics: scalability, open standards, and speed to value. In doing so, organizations enable a range of options that can satisfy both data scientists and self-service business users alike. But getting there requires a thoughtful approach -- and some enterprise knowledge of statistical modeling. How can your company stay ahead of the game?
Register for this episode of The Briefing Room to learn from veteran Analyst David Loshin, as he explains why the fundamentals will always apply to the high-stakes game of analytics. He’ll be briefed by Allen Bonde of Actuate, now part of OpenText, who will showcase his company’s intelligence platform, which was designed from the ground up to embrace open standards and was purpose-built to serve large enterprises with a wide range of data needs. He'll demonstrate recent success stories using a number of Big Data sources, including device and machine data.
Visit InsideAnalysis.com for more information.
Building a healthy data ecosystem around Kafka and Hadoop: Lessons learned at...Yael Garten
2017 StrataHadoop SJC conference talk. https://conferences.oreilly.com/strata/strata-ca/public/schedule/detail/56047
Description:
So, you finally have a data ecosystem with Kafka and Hadoop both deployed and operating correctly at scale. Congratulations. Are you done? Far from it.
As the birthplace of Kafka and an early adopter of Hadoop, LinkedIn has 13 years of combined experience using Kafka and Hadoop at scale to run a data-driven company. Both Kafka and Hadoop are flexible, scalable infrastructure pieces, but using these technologies without a clear idea of what the higher-level data ecosystem should be is perilous. Shirshanka Das and Yael Garten share best practices around data models and formats, choosing the right level of granularity of Kafka topics and Hadoop tables, and moving data efficiently and correctly between Kafka and Hadoop and explore a data abstraction layer, Dali, that can help you to process data seamlessly across Kafka and Hadoop.
Beyond pure technology, Shirshanka and Yael outline the three components of a great data culture and ecosystem and explain how to create maintainable data contracts between data producers and data consumers (like data scientists and data analysts) and how to standardize data effectively in a growing organization to enable (and not slow down) innovation and agility. They then look to the future, envisioning a world where you can successfully deploy a data abstraction of views on Hadoop data, like a data API as a protective and enabling shield. Along the way, Shirshanka and Yael discuss observations on how to enable teams to be good data citizens in producing, consuming, and owning datasets and offer an overview of LinkedIn’s governance model: the tools, process and teams that ensure that its data ecosystem can handle change and sustain #DataScienceHappiness.
Strata 2017 (San Jose): Building a healthy data ecosystem around Kafka and Ha...Shirshanka Das
So, you finally have a data ecosystem with Kafka and Hadoop both deployed and operating correctly at scale. Congratulations. Are you done? Far from it.
As the birthplace of Kafka and an early adopter of Hadoop, LinkedIn has 13 years of combined experience using Kafka and Hadoop at scale to run a data-driven company. Both Kafka and Hadoop are flexible, scalable infrastructure pieces, but using these technologies without a clear idea of what the higher-level data ecosystem should be is perilous. Shirshanka Das and Yael Garten share best practices around data models and formats, choosing the right level of granularity of Kafka topics and Hadoop tables, and moving data efficiently and correctly between Kafka and Hadoop and explore a data abstraction layer, Dali, that can help you to process data seamlessly across Kafka and Hadoop.
Beyond pure technology, Shirshanka and Yael outline the three components of a great data culture and ecosystem and explain how to create maintainable data contracts between data producers and data consumers (like data scientists and data analysts) and how to standardize data effectively in a growing organization to enable (and not slow down) innovation and agility. They then look to the future, envisioning a world where you can successfully deploy a data abstraction of views on Hadoop data, like a data API as a protective and enabling shield. Along the way, Shirshanka and Yael discuss observations on how to enable teams to be good data citizens in producing, consuming, and owning datasets and offer an overview of LinkedIn’s governance model: the tools, process and teams that ensure that its data ecosystem can handle change and sustain #datasciencehappiness.
Intro to Artificial Intelligence w/ Target's Director of PMProduct School
Given that Machine Learning (ML) is on every product enthusiast’s mind, this talk gave a broad view of the investment landscape for future innovation. Director of Product Management at Target, Aarthi Srinivasan, talked about macro AI themes & trends, how you can build your AI team and how to create a ML backed product vision.
Additionally, this talk armed the attendees with enough information to create your Point of View (POV) on how to incorporate AI into your business.
There are more and more organisations thinking about Open Data. We do think that Open Data needs tools. The Business Model Canvas could be one, but it's not totally appropriate. Here is the Open Data Canvas. Let's try it and improve it !
The Right Data Warehouse: Automation Now, Business Value ThereafterInside Analysis
The Briefing Room with Dr. Robin Bloor and WhereScape
Live Webcast on April 1, 2014
Watch the archive: https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=7b23b14b532bd7be60a70f6bd5209f03
In the Big Data shuffle, everyone is looking at Hadoop as “the answer” to collect interesting data from a new set of sources. While Hadoop has given organizations the power to gather more information assets than ever before, the question still looms: which data, regardless of source, structure, volume and all the rest, are significant for affecting business value – and how do we harness it? One effective approach is to bolster the data warehouse environment with a solution capable of integrating all the data sources, including Hadoop, and automating delivery of key information into the rights hands.
Register for this episode of The Briefing Room to hear veteran Analyst Robin Bloor as he explains how a rapidly changing information landscape impacts data management. He will be briefed by Mark Budzinski of WhereScape, who will tout his company’s data warehouse automation solutions. Budzinski will discuss how automation can be the cornerstone for closing the gap between those responsible for data management and the people driving business decisions.
Visit InsideAnlaysis.com for more information.
The Analytic Platform: Empowering the Business NowInside Analysis
The Briefing Room with Dr. Robin Bloor and Actuate
Live Webcast on October 7, 2014
Watch the archive:
https://bloorgroup.webex.com/bloorgroup/lsr.php?RCID=475312d15f46d095797f5842de84925f
As businesses grapple with more and more data, analysts and data consumers have a growing expectation to get at those assets fast. All too often, business users are stymied by governance and performance roadblocks, making time-to-insight a relatively slow process. One solution is to leverage the power of an analytic platform, one that keeps data management in IT’s hands, and lets business analysts jump right in without the need for modeling and provisioning.
Register for this episode of The Briefing Room to hear veteran Analyst Dr. Robin Bloor as he explains the principles behind a meaningful analytic platform. He’ll be briefed by Peter Hoopes and Allen Bonde of Actuate, who will tout their company’s BIRT Analytics, a solution that combines columnar database technology with pre-built algorithms and puts analytics in the hands of the business user in minutes, not days. They will show how their platform makes it easy to perform complex analytics on enterprise data and visualize results, without slowing down other systems or interfering with governance needs.
Visit InsideAnlaysis.com for more information.
https://www.youtube.com/watch?v=nvlHJgRE3pU
Won ITAC Graduation Projects Competition, ITAC ID: GP2015.R10.75
A web application that analyze big volumes of product reviews, social networks posts and tweets related to a given product. Then, present these results of this big data analytical job in a user friendly, understandable, and easily interpreted manner that can be used by different customers for different purposes.
Technologies used:
1- Hadoop
2- Hadoop Streaming
3- R Statistical
4- PHP
5- Google Charts API
Keynote by Seth Grimes, presented at the Knowledge Extraction from Social Media workshop, November 12, 2012, preceding the International Semantic Web Conference
Data Science. Business Analytics is the statistical study of business data to gain insights. Data science is the study of data using statistics, algorithms and technology. Uses mostly structured data. Uses both structured and unstructured data.
Data Centers - Striving Within A Narrow Range - Research Report - MCG - May 2...pchutichetpong
M Capital Group (“MCG”) expects to see demand and the changing evolution of supply, facilitated through institutional investment rotation out of offices and into work from home (“WFH”), while the ever-expanding need for data storage as global internet usage expands, with experts predicting 5.3 billion users by 2023. These market factors will be underpinned by technological changes, such as progressing cloud services and edge sites, allowing the industry to see strong expected annual growth of 13% over the next 4 years.
Whilst competitive headwinds remain, represented through the recent second bankruptcy filing of Sungard, which blames “COVID-19 and other macroeconomic trends including delayed customer spending decisions, insourcing and reductions in IT spending, energy inflation and reduction in demand for certain services”, the industry has seen key adjustments, where MCG believes that engineering cost management and technological innovation will be paramount to success.
MCG reports that the more favorable market conditions expected over the next few years, helped by the winding down of pandemic restrictions and a hybrid working environment will be driving market momentum forward. The continuous injection of capital by alternative investment firms, as well as the growing infrastructural investment from cloud service providers and social media companies, whose revenues are expected to grow over 3.6x larger by value in 2026, will likely help propel center provision and innovation. These factors paint a promising picture for the industry players that offset rising input costs and adapt to new technologies.
According to M Capital Group: “Specifically, the long-term cost-saving opportunities available from the rise of remote managing will likely aid value growth for the industry. Through margin optimization and further availability of capital for reinvestment, strong players will maintain their competitive foothold, while weaker players exit the market to balance supply and demand.”
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Algorithmic optimizations for Dynamic Levelwise PageRank (from STICD) : SHORT...
Agile data science
1. Agile Data Science
INFORMS BIG DATA CONFERENCE
6/23/2013
!
Presented by Joel S Horwitz
Follow me @JSHorwitz
Alpine Data Labs
2. Agile Manifesto
We are uncovering better ways of developing software by doing it and
helping others do it. Through this work we have come to value:
!
Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
!
That is, while there is value in the items on the right, we value the items on
the left more.
3. We are uncovering better ways of developing software models by doing it
and helping others do it. Through this work we have come to value:
!
Individuals and interactions over processes and tools
Working software models over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan
!
That is, while there is value in the items on the right, we value the items on
the left more.
Agile Manifesto
4. Business Analytics Technologist
Linear workflows in non-agile culture
????
Business Owner
Business Analyst
Data Owner
Data Scientist
Business Stakeholders
< / >
001011
= 5
= 10
5. Agile is about continuous interactions
Analytics
• Feature Creation
• Model / Scoring
• Evaluation
Technologist
• Wrangling
• Interpretation
• Pipelines
Business
• Presentation
• Deployment
• Productize
6. Minimally Viable Data Products (MVDP)
crossfilter.js or D3.JS
Tableau
Dashboards
Product
Recommendations
Also, Product
Recommendations
People
Recommendations
One model, many use cases.
7. Agile Data Science Feedback Loop
Instrumentation
/ Data Collection
Analysis &
Design
Results &
Interpretation
Cultivating
Data Intuition
8. What do you need?
1. Business Champion
2.Integrated Environment
3. Analytics Ninjas
9. 1. Business Champion(s): Defining the Problem
Executive Sponsor(s) who have a vested interest… ready to take action from results, has an impact to their business goals.
Chief Technology Officer
SVP of Product
Chief Marketing Officer
VP of Sales
Problem statement… Monthly active user count
growth was stagnating. Evidence of where to look…
acquisition funnel, user engagement, and customer loyalty.
Question to answer… How
do I connect each part of
the acquisition funnel?
We started with the acquisition funnel… web visits, downloads, installs, and activations.
10. 2. Environment: Before
• Many data silos and technology limitations
• Data definitions not well defined.
• Multiple data formats.
• No place to build MVDPs at scale.
11. 2. Environment: After
Open Source Technologies
Flume, Sqoop, Oozie, and others
Analytics Sandbox
Relational
Database
Minimally Viable Data Products
Selectively include data
that is Relevant
12. How was our analytics platform deployed and
connected to the network of systems.
First there was FTP… dump raw log files from web analytics, content delivery network, and
application log files.
!
Second there was MS SQL… Most of the data was in Microsoft SQL databases.
!
Third there was Hadoop (with Hive)… Engineering and Development backup to Hadoop.
!
Now there is web based analytics…
13. What is Hadoop?
• Overview: Apache Hadoop is a framework for running applications on large cluster built of
commodity hardware.
• Storage: Hadoop Distributed File System (HDFS™) is the primary storage system used by
Hadoop applications. HDFS creates multiple replicas of data blocks and distributes them on
compute nodes throughout a cluster to enable reliable, extremely rapid computations.
• Applications: Apache Hive is a large scale Data Warehouse system and Apache Mahout is a
machine learning system.
14. 3. Analytics Ninjas
Build a TIGER Team… subject matter expert, modeler, technologist, and storyteller
Curiosity
Willingness to learn
Resourceful
Risk Takers
!
Schedule Standups… daily is best depending on the project it can vary
Take notes!
Open dialogue (peer review)
Share knowledge
!
Centralize and Version Control Your work… files, code, knowledge, and data lineage are key to success
Create a Wiki
Work backwards from the end goal to the analysis to the data
Share, share, and share! Stand on the shoulder of Giants! Why start new, there is plenty of boilerplate to go around.
15. What is Analytics?
Analytics is the application of computer technology, operational research, and statistics to solve problems in business
and industry.
!
Historically, Analytics was heavily used in banking for portfolio assessment using social status, geographical location, net
value, and many other factors.
!
Today, Analytics is applied to a vast number of industries and is re-emerging due to the phenomenal explosion of data
from our connected world.
!
Big Data consists of data sets that grow so large and complex that they become awkward to work with using on-hand
database management tools
!
McKinsey Global Institute estimates that big data analysis could save the American health care system $300 billion per
year and the European public sector €250 billion.
16. Analytics Example: Platform Monetization
Search Lifetime Value
• 3rd Party Distribution
• SEM
• Organic
Traffic Quality Analysis
• PPC
• PPD
• PPI
• PPA
1. Business Champion
• Sales & Product
2.Integrated Environment
• Web analytics / app data and SQL
Database.
3. Analytics Ninjas
• Search engine marketers, business
analysts, and statisticians.
17. Examples of Analytics: Campaign Management
1. Business Champion
• Sales, Product, Marketing, and Project
Managers.
2.Integrated Environment
• Data Warehouse, SQL DB, Hadoop, and
Tableau
3. Analytics Ninjas
• Web analytics, product managers,
business analysts, and business
intelligence.
18. Examples of Analytics: Customer Sentiment
Keywords by Ratings
1. Business Champion
• Product and Marketing
2.Integrated Environment
• Mobile analytics, app logfiles,
and Hadoop.
3. Analytics Ninjas
• Web analytics, product
managers, business analysts,
mobile developers.
19. What to avoid
Its all about plugins for web analytics… Google Analytics, App Stores (iOS, Android, others), Social (Twitter, Facebook
FQL, others?).
Analysis lifecycle…
1. Give me data… import data and ETL it into submission.
2. What does this data mean… Crunch, Blend, Join, Pivot, Predict, Count, Map, or whatever
3. Show and Tell… Static (powerpoint, excel, etc.) or dynamic (trends, filtering, drilldown). (No more dashboards)...
known knowns = data puking
(dashboards)
Interestingness rocks! Tell me where to look (I’m
feeling lucky…)
get people to "make love to data" to make
actual good use of it
Viva la revolucion! Data democracy!
20. Thank you! Any Questions?
Want to jump start your Agile Data Science project? Head over to
http://start.alpinenow.com
Follow me on @JSHorwitz