Planning a data solution - &quot;By Failing to prepare, you are preparing to fail&quot;

Architecting for analytics

Wizard Driven AI Anomaly Detection with Databricks in Azure

Fraud is prevalent in every industry, and growing at an increasing rate, as the volume of transactions increases with automation. The National Healthcare Anti-Fraud Association estimates $350B of fraudulent spending. Forbes estimates $25B spending by US banks on anti-money laundering compliance. At the same time as fraud and anomaly detection use cases are booming, the skills gap of expert data scientists available to perform fraud detection is widening. The Kavi Global team will present a cloud native, wizard-driven AI anomaly detection solution, enabling Citizen Data Scientists to easily create anomaly detection models to automatically flag Collective, Contextual, and Point anomalies, at the transaction level, as well as collusion between actors. Unsupervised methods (Distribution, Clustering, Association, Sequencing, Historical Occurrence, Custom Rules) and supervised (Random Forest, Neural Network) models are executed in Apache Spark on Databricks. An innovative aggregation framework converts probabilistic fraud scores and their probabilities into a meaningful and actionable prioritized list of suspicious (a statistical outlier) and potentially fraudulent transaction to be investigated from a business point of view. The AI Anomaly Detection models improve over time using Human-in-the-Loop feedback methods to label data for supervised modeling. Finally, The Kavi team overviews the Anomaly Lifecycle: from statistical outlier to validated business fraud for reclaim and business process changes to long term prevention strategies using proactive audits upstream at the time of estimate to prevent revenue leakage. Two client success stories will be presented acros Pharmaceutical Rx and Transportation industries.

Creating an Enterprise AI Strategy

AtScale

When it comes to creating an enterprise AI strategy: if your company isn’t good at analytics, it’s not ready for AI. Succeeding in AI requires being good at data engineering AND analytics. Unfortunately, management teams often assume they can leapfrog best practices for basic data analytics by directly adopting advanced technologies such as ML/AI – setting themselves up for failure from the get-go. This presentation explains how to get basic data engineering and the right technology in place to create and maintain data pipelines so that you can solve problems with AI successfully.

Merchant Churn Prediction Using SparkML at PayPal with Chetan Nadgire and Ani...

In this session, PayPal will present the techniques used to retain merchants using some of the Machine Learning models using SparkML platform. Retaining merchants directly equates to Dollar value. So, it was very critical for us to identify the right model that trains on our data and predicts merchant behavior giving us insights that help us prevent merchant churn. We will also deep dive on how we captured the right signals filtering the noise that could skew the predictions and some of the challenges we faced in scaling this solution. Lastly, we will see how SparkML orchestrated various events in the pipeline we built thereby enabling us to perform feature engineering, train it, validate and cross-validate it at scale across the different data samples we had.

Big data bi-mature-oanyc summit

Open Analytics

The document discusses the challenges data scientists face in operationalizing big data projects and making the results accessible for broader organizational use. It argues that within the next 18 months, big data will become integrated into standard reporting and analysis used by all employees, not just data scientists. However, current tools like Hadoop are too slow for interactive work. New technologies are needed that provide massively parallel processing and tightly integrate with Hadoop, but also allow for use of existing reporting tools. This will require analytical platforms with in-memory processing capabilities and low latency.

Dsc 2021 presentation_radovan_bacovic

Radovan Baćović

The document discusses how GitLab.com builds its data services and products. It describes how GitLab.com uses its own DevOps platform to build an Enterprise Data Platform that analyzes data from GitLab.com. The data team faces challenges around scaling, visibility, and speed. To address these, the team takes actions like open sourcing tools, adopting DevOps practices, and establishing roles, processes, and technologies to build a trusted data model and framework. The key takeaways emphasize continuous iteration, discipline, automation, and living the company values.

To analyse why operationalizing AI is so challenging, it’s important to understand the full lifecycle of an AI project, and identify the stakeholders involved. Through 2023, Gartner estimates that 50% of IT leaders will struggle to move their AI projects past proof of concept (POC) to a production level of maturity. To reduce this high failure rate, organisations need to build the right roles for AI success. In many organisations, data scientists are still wearing too many hats due to a dearth of talent across other roles. This session will highlight how, in order to successfully operationalise and scale AI POCs, organisations must to build diverse AI roles and skills with a collaborative structure that is paramount.

Data summit connect fall 2020 - rise of data ops

Ryan Gross

Data governance teams attempt to apply manual control at various points for consistency and quality of the data. By thinking of our machine learning data pipelines as compilers that convert data into executable functions and leveraging data version control, data governance and engineering teams can engineer the data together, filing bugs against data versions, applying quality control checks to the data compilers, and other activities. This talk illustrates how innovations are poised to drive process and cultural changes to data governance, leading to order-of-magnitude improvements.

Data Mining - The Big Picture!

Khalid Salama

Recently, in the fields Business Intelligence and Data Management, everybody is talking about data science, machine learning, predictive analytics and many other “clever” terms with promises to turn your data into gold. In this slides, we present the big picture of data science and machine learning. First, we define the context for data mining from BI perspective, and try to clarify various buzzwords in this field. Then we give an overview of the machine learning paradigms. After that, we are going to discuss - at a high level - the various data mining tasks, techniques and applications. Next, we will have a quick tour through the Knowledge Discovery Process. Screenshots from demos will be shown, and finally we conclude with some takeaway points.

Scaling AutoML-Driven Anomaly Detection With Luminaire

Luminaire is an open-source Python library developed by Zillow to provide scalable and automated time series anomaly detection. It uses AutoML techniques to optimize model selection and configuration with minimal input. Luminaire profiles and preprocesses time series data, trains both batch and streaming models, scores anomalies, and supports distributed training and scoring on large datasets using Spark. Zillow uses Luminaire to monitor data quality across many metrics and data sources that power its products and services.

Big data-science-oanyc

Open Analytics

This document provides an overview of the skills, tools, and techniques needed for big data science. It discusses infrastructure requirements like Hadoop and NoSQL, as well as necessary talent and analytic capabilities. A case study is presented using data from Stack Overflow to demonstrate the end-to-end process of exploring data, building features, creating structured and unstructured models, and ensembling models to solve a business problem. The document emphasizes that achieving early success in big data science requires a blend of analysis and scripting skills along with an understanding of relevant techniques, but large teams of PhDs or major investments are not necessarily needed.

"Building Data Foundations and Analytics Tools Across The Product" by Crystal...

Tech in Asia ID

Crystal is a data nerd, self-taught programmer, and avid non-fiction reader. Having joined GO-JEK over two years ago, she has first-hand experience of the many challenges involved with scaling data-driven teams at Indonesia’s first unicorn startup. She currently leads the strategy and vision of the Business Intelligence team’s internal products and data culture across the company. Her team aims to produce actionable insights for all of the different verticals on the GO-JEK platform. This slide was shared at Tech in Asia Product Development Conference 2017 (PDC'17) on 9-10 August 2017. Get more insightful updates from TIA by subscribing techin.asia/updateselalu

Big data webinar may23 nrit by sunil

Sujit Ghosh

This document provides an overview of NRIT, a business intelligence and data warehousing company. It discusses NRIT's services including data warehousing, reporting, predictive analytics, and data science. It also covers topics like big data, NoSQL databases, Hadoop, and cloud-based BI solutions. NRIT offers consulting, implementation, and training services to help clients leverage these new technologies and techniques for business intelligence and analytics.

Applied Data Science Course Part 1: Concepts & your first ML model

Dataiku

CI/DC in MLOps by J.B. Hunt

Overcoming DataOps hurdles for ML in Production

Sandeep Uttamchandani

Data Engineering Challenges - DSE Day at Bandung Institute of Technology

Rendy Bambang Junior

Data engineers build and maintain robust data pipelines to ingest, process, store and retrieve large amounts of data. They face challenges around throughput and scalability during ingestion, dependencies and performance during processing, efficiency and queries during storage, and ease of use during retrieval. Data engineering involves taking on roles in database administration, architecture, programming and analytics simultaneously.

Anne-Sophie Roessler, International Business Developer, Dataiku - "3 ways to ...

Dataconomy Media

Understanding DataOps and Its Impact on Application Quality

DevOps.com

Modern day applications are data driven and data rich. The infrastructure your backends run on are a critical aspect of your environment, and require unique monitoring tools and techniques. In this webinar learn about what DataOps is, and how critical good data ops is to the integrity of your application. Intelligent APM for your data is critical to the success of modern applications. In this webinar you will learn: The power of APM tailored for Data Operations The importance of visibility into your data infrastructure How AIOps makes data ops actionable

Data Ops at TripActions

Commercializing Alternative Data

Graph Hardware Architecture - Enterprise graphs deserve great hardware!

TigerGraph

Agile Big Data Analytics Development: An Architecture-Centric Approach

SoftServe

Data driven decision making through analytics and IoT

Aachen Data & AI Meetup

The document discusses using data analytics and IoT to improve transport logistics operations. It outlines defining processes, collecting data from various sources, building a data pipeline to ingest the data, implementing analytics solutions, and realizing advantages. Key steps include streamlining existing practices, digitalizing processes, setting up integrations, building connectors, and implementing cognitive computing to increase efficiency and reduce costs. The overall goal is to provide fool-proof and cost-effective solutions using data and analytics to accelerate growth.

Testing the Data Warehouse—Big Data, Big Problems

TechWell

Data warehouses are critical systems for collecting, organizing, and making information readily available for strategic decision making. The ability to review historical trends and monitor near real-time operational data is a key competitive advantage for many organizations. Yet the methods for assuring the quality of these valuable assets are quite different from those of transactional systems. Ensuring that appropriate testing is performed is a major challenge for many enterprises. Geoff Horne has led numerous data warehouse testing projects in both the telecommunications and ERP sectors. Join Geoff as he shares his approaches and experiences, focusing on the key “uniques” of data warehouse testing: methods for assuring data completeness, monitoring data transformations, measuring quality, and more. Geoff explores the opportunities for test automation as part of the data warehouse process, describing how you can harness automation tools to streamline the work and minimize overhead.

Using H2O for Mobile Transaction Forecasting & Anomaly Detection - Capital One

Sri Ambati

Presented at #H2OWorld 2017 in Mountain View, CA. Learn more about H2O.ai: https://www.h2o.ai/. Follow @h2oai: https://twitter.com/h2oai. - - - Effective volume anomaly detection presents unique challenges when monitoring customer transaction volumes across thousands of platforms and systems. We overcome this by using H2O, building on open source tools, and delivering machine learning anomaly detection for enterprise scale. Hear how we model, visualize then automatically alert on anomalous Mobile app volumes in real-time. Donald Gennetten has over 15 years experience supporting digital channels in the Financial Services industry. In his current role as a Data Engineer for Capital One’s Monitoring Intelligence team, he leads a cross-functional group of Data, Business, and Engineering subject matter experts to deliver Advanced Analytics solutions for real-time customer transaction monitoring and issue detection. Rahul Gupta is a Data Engineer in Capital One's Center for Machine Learning, focusing heavily on back-end development and model creation. His primary efforts include building an Algorithmic IT Operations (AIOps) platform that utilizes a combination of batch and streaming data with Machine Learning capabilities to improve the stability of Capital One services and overall customer experience.

ONE-SIZE DOESN'T FIT ALL - EFFECTIVELY (RE)EVALUATE A DATA SOLUTION FOR YOUR ...

DevOpsDays Tel Aviv

The document discusses effectively evaluating data solutions for systems. It recommends identifying the business value and needs, such as scale and future costs. It also suggests analyzing expected performance by considering the solution's purpose, resource requirements, and resilience. Additionally, it advises overseeing future plans like the system's scale support, R&D growth plan, and learning curve. The overall document provides guidance on comprehensively evaluating data solutions by examining business value, performance, and future oversight.

Behind The Scenes Data Science Coolblue 2018-03-22

Matthias Schuurmans

What's hot

AI is a Team Sport

Harvinder Atwal

Data summit connect fall 2020 - rise of data ops

Ryan Gross

Data Mining - The Big Picture!

Khalid Salama

Scaling AutoML-Driven Anomaly Detection With Luminaire

Big data-science-oanyc

Open Analytics

"Building Data Foundations and Analytics Tools Across The Product" by Crystal...

Tech in Asia ID

Big data webinar may23 nrit by sunil

Sujit Ghosh

Applied Data Science Course Part 1: Concepts & your first ML model

Dataiku

CI/DC in MLOps by J.B. Hunt

Overcoming DataOps hurdles for ML in Production

Sandeep Uttamchandani

Data Engineering Challenges - DSE Day at Bandung Institute of Technology

Rendy Bambang Junior

Anne-Sophie Roessler, International Business Developer, Dataiku - "3 ways to ...

Dataconomy Media

Understanding DataOps and Its Impact on Application Quality

DevOps.com

Data Ops at TripActions

Commercializing Alternative Data

Graph Hardware Architecture - Enterprise graphs deserve great hardware!

TigerGraph

Agile Big Data Analytics Development: An Architecture-Centric Approach

SoftServe

Data driven decision making through analytics and IoT

Aachen Data & AI Meetup

Testing the Data Warehouse—Big Data, Big Problems

TechWell

Using H2O for Mobile Transaction Forecasting & Anomaly Detection - Capital One

Sri Ambati

What's hot (20)

AI is a Team Sport

Data summit connect fall 2020 - rise of data ops

Data Mining - The Big Picture!

Scaling AutoML-Driven Anomaly Detection With Luminaire

Big data-science-oanyc

"Building Data Foundations and Analytics Tools Across The Product" by Crystal...

Big data webinar may23 nrit by sunil

Applied Data Science Course Part 1: Concepts & your first ML model

CI/DC in MLOps by J.B. Hunt

Overcoming DataOps hurdles for ML in Production

Data Engineering Challenges - DSE Day at Bandung Institute of Technology

Anne-Sophie Roessler, International Business Developer, Dataiku - "3 ways to ...

Understanding DataOps and Its Impact on Application Quality

Data Ops at TripActions

Commercializing Alternative Data

Graph Hardware Architecture - Enterprise graphs deserve great hardware!

Agile Big Data Analytics Development: An Architecture-Centric Approach

Data driven decision making through analytics and IoT

Testing the Data Warehouse—Big Data, Big Problems

Using H2O for Mobile Transaction Forecasting & Anomaly Detection - Capital One

A Day in the Life of a Druid Implementor and Druid's Roadmap

This document summarizes a typical day for a Druid architect. It describes common tasks like evaluating production clusters, analyzing data and queries, and recommending optimizations. The architect asks stakeholders questions to understand usage and helps evaluate if Druid is a good fit. When advising on Druid, the architect considers factors like data sources, query types, and technology stacks. The document also provides tips on configuring clusters for performance and controlling segment size.

Scalable Incremental Index for Druid

Dr. Edward (Eddie) Bortnikov (Senior Director of Research) @ Verizon Media: Ingestion and queries of real-time data in Druid are performed by a core software component named Incremental Index (I^2). I^2’s scalability is paramount to the speed of the ingested data becoming queryable as well as to the operational efficiency of the Druid cluster. The current I^2 Implementation is based on the traditional ordered JDK key-value (KV-)map. We present an experimental I^2 implementation that is based on a novel data structure named OakMap - a scalable thread-safe off-heap KV-map for Big Data applications in Java. With OakMap, I^2 can ingest data at almost 2x speed while using 30% less RAM. The project is expected to become GA in 2020.

Funnel Analysis with Spark and Druid

Itai Yaffe (Tech Lead, Big Data group) @ Nielsen: Every day, millions of advertising campaigns are happening around the world. As campaign owners, measuring the ongoing campaign effectiveness (e.g “how many distinct users saw my online ad VS how many distinct users saw my online ad, clicked it and purchased my product?”) is super important. However, this task (often referred to as “funnel analysis”) is not an easy task, especially if the chronological order of events matters. So, while the combination of Druid and ThetaSketch aggregators can answer some of these questions, it still can’t answer the question "how many distinct users viewed the brand’s homepage FIRST and THEN viewed product X page?" In this talk, we will discuss how we combine Spark, Druid and ThetaSketch aggregators to answer such questions at scale.

The benefits of running Spark on your own Docker

Shir Bromberg (Big Data team leader) @ Yotpo: Nowadays, many of an organization’s main applications rely on Spark pipelines. As these applications become more significant to businesses, so does the need to quickly deploy, test and monitor them. The standard way of running spark jobs is to deploy it on a dedicated managed cluster. However, this solution is relatively expensive with potentially high setup time. Therefore, we developed a way to run Spark on any container orchestration platform. This allows us to run Spark in a simple, custom and testable way. In this talk, we will present our open-source dockers for running Spark on Nomad servers. We will cover: * The issues we had running spark on managed clusters and the solution we developed. * How to build a spark docker. * And finally, what you may achieve by using Spark on Nomad.

Optimizing Spark-based data pipelines - are you up for it?

Etti Gur (Senior Big Data developer) and Itai Yaffe (Tech Lead, Big Data group) @ Nielsen: At Nielsen Marketing Cloud, we provide our customers (marketers and publishers) real-time analytics tools to measure their ongoing campaigns' efficiency. To achieve that, we need to ingest billions of events per day into our big data stores and we need to do it in a scalable yet cost-efficient manner. In this talk, we will discuss how we significantly optimized our Spark-based in-flight analytics daily pipeline, reducing its total execution time from over 20 hours down to 2 hours, resulting in a huge cost reduction. Topics include: * Ways to identify optimization opportunities * Optimizing Spark resource allocation * Parallelizing Spark output phase with dynamic partition inserts * Running multiple Spark "jobs" in parallel within a single Spark application

Scheduling big data workloads on serverless infrastructure

Ilai Malka from Nielsen at AWS Community Day TLV, December 2019 (https://awscommunitydaytelaviv2019.splashthat.com/): Scheduling big data workloads is challenging. It's extra challenging when running on Serverless infrastructure. At Nielsen Marketing Cloud, we've built a system that uploads 250 billion events per day to partner ad platforms, running on Serverless infrastructure (AWS Lambda and OpenFaaS). Creating a 'scheduler' for this system required: 1. Rate-limiting to prevent flooding partner platforms. 2. High utilization to keep costs low 3. Careful bottleneck management to keep the system humming https://www.linkedin.com/in/ilai-malka-93b06172/ https://twitter.com/IlaiMalka #Nielsen #NielsenMarketingCloud #AWSCommunityDay #Serverless

GraphQL API on a Serverless Environment