strata_2015_kpcb_unboxing_jawbone_eugene_mandel
•
0 likes•422 views
Jawbone collects billions of data events per day from users tracking their movement, sleep, workouts, meals and mood through sensors. They are moving to a unified "data river" infrastructure using services like Kinesis, Redshift and DynamoDB to support analytics, data stories and products from the large volume of user data. Their goal is to build robust systems to handle and analyze user data at scale.
Report
Share
Report
Share
Download to read offline
Recommended
Using R for Climate Raster Data Extraction
Simultaneously sample hundreds of PRISM climate rasters at several point locations to build a multi-year spatial temporal dataset.
The Other 99% of a Data Science Project
Slides from my talk at Open Data Science Conference 2016.
Algorithms and models are an important (and cool) part of data science. This talk is about all the other steps that it takes to deploy a data science project that makes a product slightly smarter. Stuff that you hear from practitioners, but is not covered well enough in books.
Putting Hadoop To Work In The Enterprise
The document discusses how Hadoop can be used in enterprises by providing untethered access to large amounts of raw data for data scientists. It outlines different user needs that Hadoop can address, including data artists, business analysts, and business consumers. The document also presents Teradata's approach to helping Hadoop succeed in enterprises by providing products, services, and support around big data analytics, data integration, and simplifying Hadoop use.
EDA Meets Data Engineering – What's the Big Deal?
Presenter: Guru Sattanathan, Systems Engineer, Confluent
Event-driven architectures have been around for many years, much like Apache Kafka®, which first open sourced in 2011. The reality is that the true potential of Kafka is only being realised now. Kafka is becoming the central nervous system of many of today’s enterprises. It is bringing a profound paradigm shift to the way we think about enterprise IT. What has changed in Kafka to enable this paradigm shift? Is it not just a message broker, and how are enterprises using it today? This session will explore these key questions.
Sydney: https://content.deloitte.com.au/20200221-tel-event-tech-community-syd-registration
Melbourne: https://content.deloitte.com.au/20200221-tel-event-tech-community-mel-registration
Big Data on the Cloud
This document discusses big data processing on the cloud. It describes the 3V model of big data, which refers to volume, velocity and variety. It discusses operational challenges of big data and how distributed processing tools like Hadoop and Spark can help address these challenges. It provides an example use case of using Apache Spark on Amazon Web Services along with Apache Zeppelin and Amazon S3 for distributed data analytics.
It takes a village (to raise a ML model)
Connecting ML and online services takes effort, can't be successfully done without cross-functional work between Data Science (pushing the innovation envelope) and Software Engineering practices.
Introduction to big data
This document provides an introduction to big data, including definitions and key concepts. It discusses the evolution of computing systems and data storage. Big data is defined as large and complex data sets that are difficult to process using traditional methods due to the volume, variety, velocity, and veracity of the data. Examples of big data sources and applications are provided. Finally, different approaches for analyzing big data are described, including MapReduce, Hadoop, real-time analytics using databases, and complex event processing.
Turning an idea into a Data-Driven Production System: An Energy Load Forecas...
https://www.bigdataspain.org/2016/program/thu-turning-idea-into-data-driven-production-system-energy-load-forecasting-case-study.html
https://www.youtube.com/watch?v=Df59GmPcVag&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&t=26s&index=58
Recommended
Using R for Climate Raster Data Extraction
Simultaneously sample hundreds of PRISM climate rasters at several point locations to build a multi-year spatial temporal dataset.
The Other 99% of a Data Science Project
Slides from my talk at Open Data Science Conference 2016.
Algorithms and models are an important (and cool) part of data science. This talk is about all the other steps that it takes to deploy a data science project that makes a product slightly smarter. Stuff that you hear from practitioners, but is not covered well enough in books.
Putting Hadoop To Work In The Enterprise
The document discusses how Hadoop can be used in enterprises by providing untethered access to large amounts of raw data for data scientists. It outlines different user needs that Hadoop can address, including data artists, business analysts, and business consumers. The document also presents Teradata's approach to helping Hadoop succeed in enterprises by providing products, services, and support around big data analytics, data integration, and simplifying Hadoop use.
EDA Meets Data Engineering – What's the Big Deal?
Presenter: Guru Sattanathan, Systems Engineer, Confluent
Event-driven architectures have been around for many years, much like Apache Kafka®, which first open sourced in 2011. The reality is that the true potential of Kafka is only being realised now. Kafka is becoming the central nervous system of many of today’s enterprises. It is bringing a profound paradigm shift to the way we think about enterprise IT. What has changed in Kafka to enable this paradigm shift? Is it not just a message broker, and how are enterprises using it today? This session will explore these key questions.
Sydney: https://content.deloitte.com.au/20200221-tel-event-tech-community-syd-registration
Melbourne: https://content.deloitte.com.au/20200221-tel-event-tech-community-mel-registration
Big Data on the Cloud
This document discusses big data processing on the cloud. It describes the 3V model of big data, which refers to volume, velocity and variety. It discusses operational challenges of big data and how distributed processing tools like Hadoop and Spark can help address these challenges. It provides an example use case of using Apache Spark on Amazon Web Services along with Apache Zeppelin and Amazon S3 for distributed data analytics.
It takes a village (to raise a ML model)
Connecting ML and online services takes effort, can't be successfully done without cross-functional work between Data Science (pushing the innovation envelope) and Software Engineering practices.
Introduction to big data
This document provides an introduction to big data, including definitions and key concepts. It discusses the evolution of computing systems and data storage. Big data is defined as large and complex data sets that are difficult to process using traditional methods due to the volume, variety, velocity, and veracity of the data. Examples of big data sources and applications are provided. Finally, different approaches for analyzing big data are described, including MapReduce, Hadoop, real-time analytics using databases, and complex event processing.
Turning an idea into a Data-Driven Production System: An Energy Load Forecas...
https://www.bigdataspain.org/2016/program/thu-turning-idea-into-data-driven-production-system-energy-load-forecasting-case-study.html
https://www.youtube.com/watch?v=Df59GmPcVag&list=PL6O3g23-p8Tr5eqnIIPdBD_8eE5JBDBik&t=26s&index=58
GateSense at CBS
GateSense alpha release - GateSense is building the Internet of Things step by step with you the community. Join us!
The alpha release focuses on a basic platform (the data plane of the IoT), a middleware to get some data and the GateSense alphaApp.
AI for Earth: Analyzing Global Data with Azure
Microsoft has publicly committed $50 million over 5 years for artificial intelligence projects that support clean water, agriculture, climate, and biodiversity. Join us to learn about APIs that could literally change the way society monitors, models, and ultimately manages Earth’s life support systems.
Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto
Big data technology can collect endpoint security logs from various data sources like clickstream data, shopping cart data, and sensor data. These logs record events in operating systems and software and have characteristics of big data. Open source tools like Logstash and Elasticsearch can be used to ingest, process, and analyze these logs. Logstash is a server-side data processing pipeline that takes in data from multiple sources, while Elasticsearch is a distributed search and analytics engine. These tools can be incorporated into a big data event log architecture using other Hadoop components like Apache Flume, YARN, HDFS, and analytics frameworks.
UTAD - Jornadas de Informática - Potential of Big Data
Short Presentation given at the Universidade de Tras dos Montes (UTAD) IT event for students and faculty members. The talk is meant to be an overview of Big Data and how microsoft technologies tackle that subject and how students could leverage these tools on their projects and future.
Big Data and OSS at IBM
Presentation by Ivan Portilla and Ryan Dejana at Boulder Java User's Group, June 11, 2013. See http://boulderjug.org for more details.
Power of the Run Graph
We explain how we use Grakn as part of a wider solution to deliver next generation Data Operations (Data Ops) tooling, enabling us to deliver sophisticated "Run Graph Analytics".
The Run Graph is a component to passively track and trace our data assets as they move across the organisation, and is used to quickly reverse engineer our global flows of data to better plan change and understand hidden dependencies. When operational failures do arise, we demonstrate how Grakn quickly allows us to assess the inferred impacts downstream, and to prioritise and communicate the impacts of outages to stakeholders.
Airline traffic management analysis
• The objective is airline traffic management analysis is to maintain and analyze it based on various factors
• Tools used: Horton Works Sandbox, Tableau
Developing a successful big data business strategy
The document discusses developing a successful big data business strategy. It describes organizations moving towards more data, real-time operations, analytics insights, and personalization. It also outlines a maturity model for how organizations integrate data and analytics into their business models. Finally, it discusses challenges such as selecting the wrong use cases, management resistance, and lack of skills that can cause big data projects to fail.
SAP HORTONWORKS
The strategic relationship between Hortonworks and SAP enables SAP to resell Hortonworks Data Platform (HDP) and provide enterprise support for their global customer base. This means SAP customers can incorporate enterprise Hadoop as a complement within a data architecture that includes SAP HANA and SAP BusinessObjects enabling a broad range of new analytic applications.
Predicting Hospital Bills at Pre-admission
Healthcare expenditure is set to rise over the coming years. Cost will undoubtedly influence patients’ decision-making when it comes to diagnosis and treatment.
For healthcare providers, providing up-front cost estimates improves patient experience, making patients more willing to return (if required) in the future. For patients, having accurate pre-admission estimates allow for informed decisions and adequate preparation, reducing payment challenges after treatment. Ultimately, this case is a first step towards (i) standardization of healthcare cost estimation and (ii) price transparency to build trust between healthcare providers, payers, and patients.
Indexing the Real World Sensor Networks (at RE.WORK Internet of Things Summit...
This talk focuses on how harnessing sensor data intelligently (proprietary, commercial and public) enables to build better applications, what are the operational challenges of oil spill responses, and what kind of sensor networks are being utilized in weather forecasting, environmental monitoring and beyond.
Planet OS is a software platform for real-world sensor data integration, designed for ocean, land, air and space-based applications. Planet OS has developed a powerful suite that combines data mining, integration, search, visualization, analytics and secure data exchange between parties. It offers a single interface to work with all your proprietary (local and remote), commercial or open data.
#EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sec...
Making earth observation data available in the cloud is accelerating scientific discovery and enabling the creation of new products. Attend and learn how the cloud lets earth scientists, researchers, startups, and GIS professionals gather and analyze earth observation data without worrying about limitations of bandwidth, storage, memory, or processing power. Join us and learn how earth science data projects are becoming more scalable, agile, and efficient with AWS on-demand IT infrastructure.
Large Scale Data Analysis with AWS
This presentation from the AWS Lab at Cloud Expo Europe 2014 explores large scale data analysis on AWS. The cost of data generation is falling. Storing, analyzing and sharing data using the tools that AWS offers a low cost and easy to use solution for creating value from your data assets.
Building Data Lakes & Analytics on AWS
The document discusses mapping biomass in the Amazon forest using Amazon analytics services. It describes how biomass maps are traditionally created using satellite data and LiDAR flights, but the process is computationally intensive. The author aims to generate thousands of maps to capture uncertainty using random forest algorithms, but notes their traditional methods cannot scale to that level. They are exploring using Amazon analytics services to generate the large number of maps needed.
Streaming Analytics for IoT-Oriented Applications
This document discusses streaming analytics for IoT applications. It begins by outlining how IoT is generating new sources of data and value. It then discusses streaming analytics as an approach to analyze data in motion from IoT sensors. The document reviews open source streaming analytics platforms and commercial solutions, highlighting vendors in the emerging streaming analytics as a service space. It concludes by offering tips for organizations to evaluate how streaming analytics can create value from their IoT data.
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
To support the successful production, consumption and governance of data needed to establish a data-driven product team, Scout24 (Europe’s largest online marketplace for cars and real estate) and ThoughtWorks created a manifesto of seven principles for DataDevOps.
Sound cloud - User & Partner Conference - AT Internet
This document discusses SoundCloud's use of Amazon Redshift for big data analytics. It summarizes how SoundCloud uses Hadoop to store large amounts of listener data from multiple sources, but faced challenges with data silos and slow access. It migrated to using Amazon Redshift for its data warehouse which provides fast query performance on petabytes of data. It developed ETL processes to load data from source systems into Redshift, build a data model, and create aggregated fact tables and reporting data cubes for exploration and reporting.
Patching Mr Robot: Mitigating IoT-Related Cyber-social Disasters by getting F...
This document discusses various topics related to the Internet of Things (IoT), including:
1. Issues with current IoT architectures such as lack of interoperability, high costs, and data silos.
2. The concept of fog computing which utilizes decentralized computing between devices and the cloud to address issues like unreliable internet connections, data processing requirements, and privacy/security.
3. The author's work on developing semantic models and query techniques like SPARQL-to-SQL to improve interoperability and processing of IoT data in fog architectures.
Serverless to author, schedule, execute and monitor data workflows.
Expedia Affiliates Network (EAN), is the B2B partnership brand of Expedia, Inc. Our technology powers the hotel offering of thousands of partners around the world. Everyday EAN produces, processes and consumes more than 10TB of data to drive applications, services and business through granular insights. EAN new data cloud platform is an opportunity to be nimble, simplify and consolidate the authoring, schedule, execution and monitoring of complex data workflows. The talk explores EAN’s strategy on data processing in the cloud, moving from traditional workflow tools to a serverless approach, leveraging step-function, lambda, API gateway, EMR and Cloud Watch.
Real-time data integration to the cloud
A presentation pertaining to the integration of real-time data to the cloud with significant potential in the areas of Industrial IT,Real-time sensor information processing and Smart grids applied to various vertical industries. This is related to my blog post at www.cloudshoring.in
More Related Content
Similar to strata_2015_kpcb_unboxing_jawbone_eugene_mandel
GateSense at CBS
GateSense alpha release - GateSense is building the Internet of Things step by step with you the community. Join us!
The alpha release focuses on a basic platform (the data plane of the IoT), a middleware to get some data and the GateSense alphaApp.
AI for Earth: Analyzing Global Data with Azure
Microsoft has publicly committed $50 million over 5 years for artificial intelligence projects that support clean water, agriculture, climate, and biodiversity. Join us to learn about APIs that could literally change the way society monitors, models, and ultimately manages Earth’s life support systems.
Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto
Big data technology can collect endpoint security logs from various data sources like clickstream data, shopping cart data, and sensor data. These logs record events in operating systems and software and have characteristics of big data. Open source tools like Logstash and Elasticsearch can be used to ingest, process, and analyze these logs. Logstash is a server-side data processing pipeline that takes in data from multiple sources, while Elasticsearch is a distributed search and analytics engine. These tools can be incorporated into a big data event log architecture using other Hadoop components like Apache Flume, YARN, HDFS, and analytics frameworks.
UTAD - Jornadas de Informática - Potential of Big Data
Short Presentation given at the Universidade de Tras dos Montes (UTAD) IT event for students and faculty members. The talk is meant to be an overview of Big Data and how microsoft technologies tackle that subject and how students could leverage these tools on their projects and future.
Big Data and OSS at IBM
Presentation by Ivan Portilla and Ryan Dejana at Boulder Java User's Group, June 11, 2013. See http://boulderjug.org for more details.
Power of the Run Graph
We explain how we use Grakn as part of a wider solution to deliver next generation Data Operations (Data Ops) tooling, enabling us to deliver sophisticated "Run Graph Analytics".
The Run Graph is a component to passively track and trace our data assets as they move across the organisation, and is used to quickly reverse engineer our global flows of data to better plan change and understand hidden dependencies. When operational failures do arise, we demonstrate how Grakn quickly allows us to assess the inferred impacts downstream, and to prioritise and communicate the impacts of outages to stakeholders.
Airline traffic management analysis
• The objective is airline traffic management analysis is to maintain and analyze it based on various factors
• Tools used: Horton Works Sandbox, Tableau
Developing a successful big data business strategy
The document discusses developing a successful big data business strategy. It describes organizations moving towards more data, real-time operations, analytics insights, and personalization. It also outlines a maturity model for how organizations integrate data and analytics into their business models. Finally, it discusses challenges such as selecting the wrong use cases, management resistance, and lack of skills that can cause big data projects to fail.
SAP HORTONWORKS
The strategic relationship between Hortonworks and SAP enables SAP to resell Hortonworks Data Platform (HDP) and provide enterprise support for their global customer base. This means SAP customers can incorporate enterprise Hadoop as a complement within a data architecture that includes SAP HANA and SAP BusinessObjects enabling a broad range of new analytic applications.
Predicting Hospital Bills at Pre-admission
Healthcare expenditure is set to rise over the coming years. Cost will undoubtedly influence patients’ decision-making when it comes to diagnosis and treatment.
For healthcare providers, providing up-front cost estimates improves patient experience, making patients more willing to return (if required) in the future. For patients, having accurate pre-admission estimates allow for informed decisions and adequate preparation, reducing payment challenges after treatment. Ultimately, this case is a first step towards (i) standardization of healthcare cost estimation and (ii) price transparency to build trust between healthcare providers, payers, and patients.
Indexing the Real World Sensor Networks (at RE.WORK Internet of Things Summit...
This talk focuses on how harnessing sensor data intelligently (proprietary, commercial and public) enables to build better applications, what are the operational challenges of oil spill responses, and what kind of sensor networks are being utilized in weather forecasting, environmental monitoring and beyond.
Planet OS is a software platform for real-world sensor data integration, designed for ocean, land, air and space-based applications. Planet OS has developed a powerful suite that combines data mining, integration, search, visualization, analytics and secure data exchange between parties. It offers a single interface to work with all your proprietary (local and remote), commercial or open data.
#EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sec...
Making earth observation data available in the cloud is accelerating scientific discovery and enabling the creation of new products. Attend and learn how the cloud lets earth scientists, researchers, startups, and GIS professionals gather and analyze earth observation data without worrying about limitations of bandwidth, storage, memory, or processing power. Join us and learn how earth science data projects are becoming more scalable, agile, and efficient with AWS on-demand IT infrastructure.
Large Scale Data Analysis with AWS
This presentation from the AWS Lab at Cloud Expo Europe 2014 explores large scale data analysis on AWS. The cost of data generation is falling. Storing, analyzing and sharing data using the tools that AWS offers a low cost and easy to use solution for creating value from your data assets.
Building Data Lakes & Analytics on AWS
The document discusses mapping biomass in the Amazon forest using Amazon analytics services. It describes how biomass maps are traditionally created using satellite data and LiDAR flights, but the process is computationally intensive. The author aims to generate thousands of maps to capture uncertainty using random forest algorithms, but notes their traditional methods cannot scale to that level. They are exploring using Amazon analytics services to generate the large number of maps needed.
Streaming Analytics for IoT-Oriented Applications
This document discusses streaming analytics for IoT applications. It begins by outlining how IoT is generating new sources of data and value. It then discusses streaming analytics as an approach to analyze data in motion from IoT sensors. The document reviews open source streaming analytics platforms and commercial solutions, highlighting vendors in the emerging streaming analytics as a service space. It concludes by offering tips for organizations to evaluate how streaming analytics can create value from their IoT data.
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
To support the successful production, consumption and governance of data needed to establish a data-driven product team, Scout24 (Europe’s largest online marketplace for cars and real estate) and ThoughtWorks created a manifesto of seven principles for DataDevOps.
Sound cloud - User & Partner Conference - AT Internet
This document discusses SoundCloud's use of Amazon Redshift for big data analytics. It summarizes how SoundCloud uses Hadoop to store large amounts of listener data from multiple sources, but faced challenges with data silos and slow access. It migrated to using Amazon Redshift for its data warehouse which provides fast query performance on petabytes of data. It developed ETL processes to load data from source systems into Redshift, build a data model, and create aggregated fact tables and reporting data cubes for exploration and reporting.
Patching Mr Robot: Mitigating IoT-Related Cyber-social Disasters by getting F...
This document discusses various topics related to the Internet of Things (IoT), including:
1. Issues with current IoT architectures such as lack of interoperability, high costs, and data silos.
2. The concept of fog computing which utilizes decentralized computing between devices and the cloud to address issues like unreliable internet connections, data processing requirements, and privacy/security.
3. The author's work on developing semantic models and query techniques like SPARQL-to-SQL to improve interoperability and processing of IoT data in fog architectures.
Serverless to author, schedule, execute and monitor data workflows.
Expedia Affiliates Network (EAN), is the B2B partnership brand of Expedia, Inc. Our technology powers the hotel offering of thousands of partners around the world. Everyday EAN produces, processes and consumes more than 10TB of data to drive applications, services and business through granular insights. EAN new data cloud platform is an opportunity to be nimble, simplify and consolidate the authoring, schedule, execution and monitoring of complex data workflows. The talk explores EAN’s strategy on data processing in the cloud, moving from traditional workflow tools to a serverless approach, leveraging step-function, lambda, API gateway, EMR and Cloud Watch.
Real-time data integration to the cloud
A presentation pertaining to the integration of real-time data to the cloud with significant potential in the areas of Industrial IT,Real-time sensor information processing and Smart grids applied to various vertical industries. This is related to my blog post at www.cloudshoring.in
Similar to strata_2015_kpcb_unboxing_jawbone_eugene_mandel (20)
Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto
Collecting Endpoint Security Logs Through Big Data Technology - Dedi Dwianto
UTAD - Jornadas de Informática - Potential of Big Data
UTAD - Jornadas de Informática - Potential of Big Data
Developing a successful big data business strategy
Developing a successful big data business strategy
Indexing the Real World Sensor Networks (at RE.WORK Internet of Things Summit...
Indexing the Real World Sensor Networks (at RE.WORK Internet of Things Summit...
#EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sec...
#EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sec...
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
Sound cloud - User & Partner Conference - AT Internet
Sound cloud - User & Partner Conference - AT Internet
Patching Mr Robot: Mitigating IoT-Related Cyber-social Disasters by getting F...
Patching Mr Robot: Mitigating IoT-Related Cyber-social Disasters by getting F...
Serverless to author, schedule, execute and monitor data workflows.
Serverless to author, schedule, execute and monitor data workflows.
strata_2015_kpcb_unboxing_jawbone_eugene_mandel
- 1. DATA @JAWBONE UNBOXING DATA STARTUPS EUGENE MANDEL (@EUGMANDEL) JAWBONE STRATA + HADOOP WORLD, 2015, SAN JOSE
- 2. THE PATH TO BETTER STARTS HERE UP® is a revolutionary system that guides you every step of the way to a better, healthier you.
- 3. Tracking hardware, sensors and algorithms Engaging application and service Science and data insight API partners and signals CO2 UP IS A SYSTEM
- 4. JAWBONE DATA BILLIONS OF EVENTS/DAY MOVEMENT SLEEP WORKOUTS MEALS MOOD API PARTNER SIGNALS
- 5. WAKING UP TO NAPA EARTHQUAKE
- 7. GOALS ROBUST DATA INFRASTRUCTURE DATA PRODUCTS DATA STORIES
- 8. PAST MULTIPLE DATABASES (MYSQL, MONGO, CASSANDRA) DISTRIBUTED EVENT LOGGING -> SCRIBE -> S3 ANALYSIS BY EMR JOBS ON EVENT FILES OR QUERY PRODUCTION DATABASES
- 9. DATA PIPELINES MYSQL -> ETL (EMR) -> REDSHIFT DISTRIBUTED EVENT LOGGING -> KINESIS -> S3 DISTRIBUTED EVENT LOGGING -> KINESIS -> STREAMING ETL -> REDSHIFT S3 EVENT LOGS -> BATCH ETL (EMR, SPARK) -> REDSHIFT REDSHIFT -> ANALYSIS (SQL, PYTHON, SPARK) -> REDSHIFT REDSHIFT -> DATA PRODUCT LOGIC -> DYNAMO DB
- 10. FUTURE ONE “DATA RIVER” INFRASTRUCTURE TO SUPPORT ANALYTICS, DATA STORIES AND DATA PRODUCTS