FIS: Accelerating Digital Intelligence in FinTech: Spark Summit East talk by Aaron Colcord

Dr. Christian Kurze from Denodo, "Data Virtualization: Fulfilling the Promise...

Dataconomy Media

Building Real-Time Data Pipeline for Diabetes Medication Recommender System U...

Databricks

American Diabetic association states that 29.1 million Americans and 300+ million people all over the world have diabetes. Diabetic medication management is always challenging. Based on doctor’s prescription, patients take insulin dosage one hour before breakfast, lunch or dinner. But the real world scenario insulin intake can be changed based on the blood glucose level, calorie intake on a specific day, etc. This talk explains how a real-time Big Data pipeline recommendation engine can be used to suggest insulin intake for diabetic patients in near real time. Based on calorie intake and blood glucose level from patients as well as generated dataset, insulin dosage can be recommended which will help patients to avoid over/under dosage. Designing medication recommender system is a need for the Healthcare industry. There is a growing trend for the applications to help doctors by recommending medication based on patient’s historic data. This also helps facilitate a doctor friendly and hospital free atmosphere for all users all over the world. This talk would delve into a Diabetes medication recommender system using Databricks and Spark. Databricks supports HIPAA compliant deployment for processing PHI data. This talk would cover building a secure pipeline with encrypted data and the end-to-end recommendation system using Structured streaming and IoT data flowing from sensor.

We are already in the era of ‘connected things’ -- our mobile phones, computers, tablets and wearables sync and talk via ‘cloud’. IoT presents a lot of challenges, especially data coming in at high volume and high velocity - imagine billions of devices each sending readings every few seconds or minutes. All this data need to be stored and analyzed. Plus we need to think about security, privacy and compliance. In this talk we will present a reference data architecture for IoT. We draw on the industry’s best practices on Big Data including Lambda architecture patterns. Lambda architecture outlines generic, scalable and fault-tolerant data processing. We will illustrate a potential data pipeline and discuss how each stage might be implemented and technology choices at each stage.

The Power of Data

This presentation starts off by discussing powerful examples of The Power of Data and the benefits of Data Driven architectures. A Data Governance program is important for the success of Data Driven architectures. We then discuss the challenges of implementing a Data Governance framework on a Big Data Data Lake with open source software including DataPlane, Apache Atlas and Apache Ranger. And finally, we discuss the importance of the democratization of data and the switching to a speed of thought framework with Hive LLAP.

Necessity of Data Lakes in the Financial Services Sector

With the emergence of regulations such as the General Data Protection Regulation from the European Union (effective May 2018), with fines up to 20m Euro, Data Lakes are emerging as the data architecture of choice amongst financial institutions. Banks are embarking on a journey to enable data scientists to unlock the value of the data silo'ed in many disparate data systems. By enabling self service data access and merging multiple streams of data by using data clustering, entity extraction, identity resolution and other techniques - we will show how banks have used Analytics to uncover business value without falling into the abyss of data swamps. The build out of the data lake requires the ingestion of data from multiple operational systems . By leveraging an automated Data Cataloging service, organizations are able to search, profile, discover, tag, track lineage and capture tribal knowledge delivered on the FICO Analytics Cloud enabling the data scientists to build innovative models, make automated decisions, track fraudulent usage, make intelligent marketing campaigns and improve the top line and bottom line for the financial institution. Speaker: Rohit Valia, Product Management and Strategy, Fico

Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype

Ali Hodroj

Big Data for Managers: From hadoop to streaming and beyond

Transforming GE Healthcare with Data Platform Strategy

Databricks

Data and Analytics is foundational to the success of GE Healthcare’s digital transformation and market competitiveness. This use case focuses on a heavy platform transformation that GE Healthcare drove in the last year to move from an On prem legacy data platforming strategy to a cloud native and completely services oriented strategy. This was a huge effort for an 18Bn company and executed in the middle of the pandemic. It enables GE Healthcare to leap frog in the enterprise data analytics strategy.

Finding the needle in the haystack: how Nestle is leveraging big data to defe...

Big Data Telecom

Trick Consulting

How Yellowbrick Data Integrates to Existing Environments Webcast

Hosseinieh Ershad Public Library

Real timedata

How to build a successful Data Lake

Introduction: Architecting for Scale

Architecting a Modern Data Warehouse: Enterprise Must-Haves

The goal of modern data warehousing is to not only deliver insights faster to more users, but provide a richer picture of your operations afforded by a greater volume and variety of data for analysis. This presentation from a Database Trends and Applications webcast will educate IT decision makers and data warehousing professionals about the must-have capabilities for modern data warehousing today – how they work and how best to use them.

The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...

Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...

Unlocking Value in Device Data Using Spark: Spark Summit East talk by John La...

HP ships millions of PCs, Printers, and other devices every year to customers in all market segments. More customers are seeking services provided with our products enabling new opportunities for HP to create services from the data we can collect from our devices. Every device we ship is an IoT endpoint with powerful CPU to capture rich data. Insights from this data are used internally to improve our products and focus on customer needs. In this presentation, John will focus on HP’s journey to enabling Big Data analytics from within a large enterprise environment. He will review the challenges and how HP decided on AWS, Apache Spark and Databricks as the foundation for their entry into Big Data Analytics. John will also review how HP uses Spark to build analytic services from the data they generate from their devices.

Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...

Spark 2.0 provided strong performance enhancements to the Spark core while advancing Spark ML usability to use data frames. But what happens when you run Spark 2.0 machine learning algorithms on a large cluster with a very large data set? Do you even get any benefit from using a very large data set? It depends.  How do new hardware advances affect the topology of high performance Spark clusters. In this talk we will explore Spark 2.0 Machine Learning at scale and share our findings with the community. As our test platform we will be using a new cluster design, different from typical Hadoop clusters, with more cores, more RAM and latest generation NVMe SSD’s and a 100GbE network with a goal of more performance, in a more space and energy efficient footprint.

What's hot

The Big Data Ecosystem for Financial Services

Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...

Webinar: Fighting Fraud with Graph Databases

In-Memory Computing Summit

Real-time Microservices and In-Memory Data Grids

Ali Hodroj

The European Conference on Software Architecture (ECSA) 14 - IBM BigData Refe...Romeo Kienzler

IMCSummit 2015 - Day 2 Developer Track - A Reference Architecture for the Int...

The Power of Data

Necessity of Data Lakes in the Financial Services Sector

Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype

Ali Hodroj

Big Data for Managers: From hadoop to streaming and beyond

Transforming GE Healthcare with Data Platform Strategy

Databricks

Finding the needle in the haystack: how Nestle is leveraging big data to defe...

Big Data Telecom

Trick Consulting

How Yellowbrick Data Integrates to Existing Environments Webcast

Hosseinieh Ershad Public Library

Real timedata

How to build a successful Data Lake

Introduction: Architecting for Scale

Architecting a Modern Data Warehouse: Enterprise Must-Haves

The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...

Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...

What's hot (20)

The Big Data Ecosystem for Financial Services

Stream Processing as Game Changer for Big Data and Internet of Things by Kai ...

Webinar: Fighting Fraud with Graph Databases

Real-time Microservices and In-Memory Data Grids

The European Conference on Software Architecture (ECSA) 14 - IBM BigData Refe...

IMCSummit 2015 - Day 2 Developer Track - A Reference Architecture for the Int...

The Power of Data

Necessity of Data Lakes in the Financial Services Sector

Hybrid Transactional/Analytics Processing: Beyond the Big Database Hype

Big Data for Managers: From hadoop to streaming and beyond

Transforming GE Healthcare with Data Platform Strategy

Finding the needle in the haystack: how Nestle is leveraging big data to defe...

Big Data Telecom

How Yellowbrick Data Integrates to Existing Environments Webcast

Real timedata

How to build a successful Data Lake

Introduction: Architecting for Scale

Architecting a Modern Data Warehouse: Enterprise Must-Haves

The Enterprise and Connected Data, Trends in the Apache Hadoop Ecosystem by A...

Solving the Industry 4.0 challenges on the logistics domain using Apache Meso...

Viewers also liked

Unlocking Value in Device Data Using Spark: Spark Summit East talk by John La...

Practical Large Scale Experiences with Spark 2.0 Machine Learning: Spark Summ...

Modeling Catastrophic Events in Spark: Spark Summit East Talk by Georg Hofman...

Reinsurance company’s core competencies include the quantification of risk associated with catastrophes, such as hurricanes and earthquakes. Various so-called catastrophe models are available publicly, some commercial and some open-source. The volume of data processed by such “cat models” requires Big Data and High Performance capabilities. This is clearly reflected in the landscape of public models. And the observed trend is towards more and more detailed inputs, as well as outputs. This makes scalability an important concern. Companies that deal with catastrophe risk commonly use one or several public cat models. If they wish to differentiate themselves from the market, they may build internal proprietary models, in particular in areas that are not covered by existing models. The result is a deeper understanding and an independent quantification of risk, both of which can lead to a competitive edge.

Scaling Apache Spark MLlib to Billions of Parameters: Spark Summit East talk ...

Apache Spark MLlib provides scalable implementation of popular machine learning algorithms, which lets users train models from big dataset and iterate fast. The existing implementations assume that the number of parameters is small enough to fit in the memory of a single machine. However, many applications require solving problems with billions of parameters on a huge amount of data such as Ads CTR prediction and deep neural network. This requirement far exceeds the capacity of exisiting MLlib algorithms many of who use L-BFGS as the underlying solver. In order to fill this gap, we developed Vector-free L-BFGS for MLlib. It can solve optimization problems with billions of parameters in the Spark SQL framework where the training data are often generated. The algorithm scales very well and enables a variety of MLlib algorithms to handle a massive number of parameters over large datasets. In this talk, we will illustrate the power of Vector-free L-BFGS via logistic regression with real-world dataset and requirement. We will also discuss how this approach could be applied to other ML algorithms.

R&D to Product Pipeline Using Apache Spark in AdTech: Spark Summit East talk ...

The central premise of DataXu is to apply data science to better marketing. At its core, is the Real Time Bidding Platform that processes 2 Petabytes of data per day and responds to ad auctions at a rate of 2.1 million requests per second across 5 different continents. Serving on top of this platform is Dataxu’s analytics engine that gives their clients insightful analytics reports addressed towards client marketing business questions. Some common requirements for both these platforms are the ability to do real-time processing, scalable machine learning, and ad-hoc analytics. This talk will showcase DataXu’s successful use-cases of using the Apache Spark framework and Databricks to address all of the above challenges while maintaining its agility and rapid prototyping strengths to take a product from initial R&D phase to full production. The team will share their best practices and highlight the steps of large scale Spark ETL processing, model testing, all the way through to interactive analytics.

Distributed Real-Time Stream Processing: Why and How: Spark Summit East talk ...

The demand for stream processing is increasing a lot these days. Immense amounts of data have to be processed fast from a rapidly growing set of disparate data sources. This pushes the limits of traditional data processing infrastructures. These stream-based applications include trading, social networks, Internet of things, system monitoring, and many other examples. A number of powerful, easy-to-use open source platforms have emerged to address this. But the same problem can be solved differently, various but sometimes overlapping use-cases can be targeted or different vocabularies for similar concepts can be used. This may lead to confusion, longer development time or costly wrong decisions.

IoT and the Autonomous Vehicle in the Clouds: Simultaneous Localization and M...

Processing real-time analytics of big data streams from sensor data will continue to be an important task as embedded technology increases and we continue to generate new types and ways of data analysis, particularly in regard to the Internet of Things (IoT). Robotics models many of these key challenges well and incorporates the possibility of high- throughput streams as well as complex online machine learning and analytics algorithms. These challenges make it an almost ideal candidate for in depth analysis of real-time streaming analytics. We look at a simultaneous localization and mapping (SLAM) problem, an ongoing research area in robotics for autonomous vehicles, and well recognized as a non-trivial problem space in both industry and research. We will use a new integrated framework on Kafka and Spark Streaming to explore a constrained SLAM problem using online algorithms to navigate and map a space in real time. We present benchmarks of our open-source robot’s integration with Kafka and Spark Streaming for performance against other SLAM algorithms currently in use, explore some of the challenges we faced in our implementation, and make recommendations for improvement of performance and optimization on our framework. Finally, new to this talk, we demo real-time usage of our implementation with the Turtlebot II and explore relevant benchmarks and their implications on the future of autonomous vehicles in the IoT and cloud analytics space.

Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...

This talk will cover the tools we used, the hurdles we faced and the work arounds we developed with the help from Databricks support in our attempt to build a custom machine learning model and use it to predict the TV ratings for different networks and demographics. The Apache Spark machine learning and dataframe APIs make it incredibly easy to produce a machine learning pipeline to solve an archetypal supervised learning problem. In our applications at Cadent, we face a challenge with high dimensional labels and relatively low dimensional features; at first pass such a problem is all but intractable but thanks to a large number of historical records and the tools available in Apache Spark, we were able to construct a multi-stage model capable of forecasting with sufficient accuracy to drive the business application. Over the course of our work we have come across many tools that made our lives easier, and others that forced work around. In this talk we will review our custom multi-stage methodology, review the challenges we faced and walk through the key steps that made our project successful.

Scalable Data Science with SparkR: Spark Summit East talk by Felix Cheung

R is a very popular platform for Data Science. Apache Spark is a highly scalable data platform. How could we have the best of both worlds? How could a Data Scientist leverage the rich 9000+ packages on CRAN, and integrate Spark into their existing Data Science toolset? In this talk we will walkthrough many examples how several new features in Apache Spark 2.x will enable this. We will also look at exciting changes in and coming next in Apache Spark 2.x releases.

Building Real-Time BI Systems with Kafka, Spark, and Kudu: Spark Summit East ...

One of the key challenges in working with real-time and streaming data is that the data format for capturing data is not necessarily the optimal format for ad hoc analytic queries. For example, Avro is a convenient and popular serialization service that is great for initially bringing data into HDFS. Avro has native integration with Flume and other tools that make it a good choice for landing data in Hadoop. But columnar file formats, such as Parquet and ORC, are much better optimized for ad hoc queries that aggregate over large number of similar rows.

Realtime Analytical Query Processing and Predictive Model Building on High Di...

Spark SQL and Mllib are optimized for running feature extraction and machine learning algorithms on row based columnar datasets through full scan but does not provide constructs for column indexing and time series analysis. For dealing with document datasets with timestamps where the features are represented as variable number of columns in each document and use-cases demand searching over columns and time to retrieve documents to generate learning models in realtime, a close integration within Spark and Lucene was needed. We introduced LuceneDAO in Spark Summit Europe 2016 to build distributed lucene shards from data frame but the time series attributes were not part of the data model. In this talk we present our extension to LuceneDAO to maintain time stamps with document-term view for search and allow time filters. Lucene shards maintain the time aware document-term view for search and vector space representation for machine learning pipelines. We used Spark as our distributed query processing engine where each query is represented as boolean combination over terms with filters on time. LuceneDAO is used to load the shards to Spark executors and power sub-second distributed document retrieval for the queries. Our synchronous API uses Spark-as-a-Service to power analytical queries while our asynchronous API uses kafka, spark streaming and HBase to power time series prediction algorithms. In this talk we will demonstrate LuceneDAO write and read performance on millions of documents with 1M+ terms and configurable time stamp aggregate columns. We will demonstrate the latency of APIs on a suite of queries generated from terms. Key takeaways from the talk will be a thorough understanding of how to make Lucene powered time aware search a first class citizen in Spark to build interactive analytical query processing and time series prediction algorithms.

Sketching Data with T-Digest In Apache Spark: Spark Summit East talk by Erik ...

Algorithms for sketching probability distributions from large data sets are a fundamental building block of modern data science. Sketching plays a role in diverse applications ranging from visualization, optimizing data encodings, estimating quantiles, data synthesis and imputation. The T-Digest is a versatile sketching data structure. It operates on any numeric data, models tricky distribution tails with high fidelity, and most crucially it works smoothly with aggregators and map-reduce. T-Digest is a perfect fit for Apache Spark; it is single-pass and intermediate results can be aggregated across partitions in batch jobs or aggregated across windows in streaming jobs. In this talk I will describe a native Scala implementation of the T-Digest sketching algorithm and demonstrate its use in Spark applications for visualization, quantile estimations and data synthesis. Attendees of this talk will leave with an understanding of data sketching with T-Digest sketches, and insights about how to apply T-Digest to their own data analysis applications.

Spark Autotuning: Spark Summit East talk by Lawrence Spracklen

While the performance delivered by Spark has enabled data scientists to undertake sophisticated analyses on big and complex data in actionable timeframes, too often, the process of manually configuring the underlying Spark jobs (including the number and size of the executors) can be a significant and time consuming undertaking. Not only it does this configuration process typically rely heavily on repeated trial-and-error, it necessitates that data scientists have a low-level understanding of Spark and detailed cluster sizing information. At Alpine Data we have been working to eliminate this requirement, and develop algorithms that can be used to automatically tune Spark jobs with minimal user involvement, In this presentation, we discuss the algorithms we have developed and illustrate how they leverage information about the size of the data being analyzed, the analytical operations being used in the flow, the cluster size, configuration and real-time utilization, to automatically determine the optimal Spark job configuration for peak performance.

Algorithms and Tools for Genomic Analysis on Spark: Spark Summit East talk by...

Hammer Lab has built several tools for analyzing genomic data on Spark, as well as libraries for more general computations using RDDs; I’ll discuss some of the most interesting applications and algorithms therein: Guacamole (https://github.com/hammerlab/guacamole) is a somatic variant caller built on Spark; it identifies mutations in cancer genomes in a fraction of the time that comparable tools take. Pageant (https://github.com/hammerlab/pageant) contains miscellaneous other genomic analyses and a few interesting and novel algorithms for massively-parallel Burrows-Wheeler-Transform and FM-Index construction. Magic RDDs (https://github.com/hammerlab/magic-rdds) contains some yet more interesting general-purpose algorithms implemented on RDDs.

Finding Needles in Genomic Haystacks with “Wide” Random Forest: Spark Summit ...

Recent advances in genome sequencing technologies and bioinformatics have enabled whole-genomes to be studied at population-level rather then for small number of individuals. This provides new power to whole genome association studies (WGAS ), which now seek to identify the multi-gene causes of common complex diseases like diabetes or cancer. As WGAS involve studying thousands of genomes, they pose both technological and methodological challenges. The volume of data is significant, for example the dataset from 1000 Genomes project with genomes of 2504 individuals includes nearly 85M genomic variants with raw data size of 0.8 TB. The number of features is enormous and greatly exceeds the number of samples, which makes it challenging to apply traditional statistical approaches. Random forest is one of the methods that was found to be useful in this context, both because of its potential for parallelization and its robustness. Although there is a number of big data implementations available (including Spark ML) they are tuned for typical dataset with large number of samples and relatively small number of variables, and either fail or are inefficient in the GWAS context especially, that a costly data preprocessing is usually required. To address these problems, we have developed the RandomForestHD – a Spark based implementation optimized for highly dimensional data sets. We have successfully RandomForestHD applied it to datasets beyond the reach of other tools and for smaller datasets found its performance superior. We are currently applying RandomForestHD, released as part of the VariantSpark toolkit, to a number of WGAS studies. In the presentation we will introduce the domain of WGAS and related challenges, present RandomForestHD with its design principles and implementation details with regards to Spark, compare its performance with other tools, and finally showcase the results of a few WGAS applications.

Spark and Online Analytics: Spark Summit East talky by Shubham Chopra

Apache Spark was designed as a batch analytics system. By caching RDDs, Spark speeds up jobs that iteratively process the same data. This pattern is also applicable to online analytics. We use Bloomberg’s Spark Server as a server runtime for online analytics. Our framework implements certain useful patterns applicable to online query processing and is centered on the idea of “Managed” DataFrames that can be refreshed and updated as per user requirements, without violating the immutability of RDDs/DataFrames. However, Spark presents significant challenges with respect to availability and resilience in an online setting where Spark is required to respond to queries with high SLAs. In this talk, we try to identify specific areas where slow-down or failures can result in the largest hits on online-query performance and potential solutions to address these.

Spark + Parquet In Depth: Spark Summit East Talk by Emily Curtin and Robbie S...

What if you could get the simplicity, convenience, interoperability, and storage niceties of an old-fashioned CSV with the speed of a NoSQL database and the storage requirements of a gzipped file? Enter Parquet. At The Weather Company, Parquet files are a quietly awesome and deeply integral part of our Spark-driven analytics workflow. Using Spark + Parquet, we’ve built a blazing fast, storage-efficient, query-efficient data lake and a suite of tools to accompany it. We will give a technical overview of how Parquet works and how recent improvements from Tungsten enable SparkSQL to take advantage of this design to provide fast queries by overcoming two major bottlenecks of distributed analytics: communication costs (IO bound) and data decoding (CPU bound).

Virtualizing Analytics with Apache Spark: Keynote by Arsalan Tavakoli

In the race to invent multi-million dollar business opportunities with exclusive insights, data scientists and engineers are hampered by a multitude of challenges just to make one use case a reality – the need to ingest data from multiple sources, apply real-time analytics, build machine learning algorithms, and intermix different data processing models, all while navigating around their legacy data infrastructure that is just not up to the task. This need has created the demand for Virtual Analytics, where the complexities of disparate data and technology silos have been abstracted away, coupled with a powerful range of analytics and processing horsepower, all in one unified data platform. This talk describes how Databricks is powering this revolutionary new trend with Apache Spark.

Predictive Analytics for IoT Network Capacity Planning: Spark Summit East tal...

The Internet of Things (IoT) is a growing network, supporting a wide variety of service types with specific network requirements that differ from traditional human type communications. This has led to emergence of dedicated IoT network standards. To optimize investments for dedicated network infrastructures, we’re investigating a dynamic approach in network capacity planning to accommodate multiple IoT traffic types over a cellular network, while maintaining their specific requirements. We studied models of IoT traffic and used machine learning in prediction and scheduling of future workload under heterogeneous and variable traffic conditions when human-type and machine-type communications are mixed. An integrated analytics framework including Hadoop and Spark were deployed for experimentation and a number of capacity planning use cases were implemented to verify the accuracy of the method.

Using SparkR to Scale Data Science Applications in Production. Lessons from t...