CICD Pipeline and delivery of Apache Spark Applications on the cloud using AWS

Moving at the speed of a startup often means rapid iterative development, which can lead to a patchwork of systems and processes. In the early days at Kik (one of the most popular chat apps among U.S. teens), the data team was able to move extremely quickly but often at the expense of scalable data engineering. In this session, Kik’s head of data will share the eight things they did to save time and money. The team took their data stack from a complex combination of systems and processes to a scalable, simple, and robust platform leveraging Apache Spark and Databricks to make data super easy for everyone in the company to use.

Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...

Alluxio Webinar April 6, 2021 For more Alluxio events: https://www.alluxio.io/events/ Speakers: Alex Ma, Alluxio Peter Behrakis, Alluxio Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows. In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see. In this tech talk, we'll go over: - What is Alluxio Data Orchestration? - How does it work? - Alluxio customer results

Google Cloud Platform for Data Science teams

Barton Rhodes

Developing high frequency indicators using real time tick data on apache supe...

Zekeriya Besiroglu

Accelerate Analytics and ML in the Hybrid Cloud Era

Alluxio, Inc.

Spark and Hadoop at Production Scale-(Anil Gadre, MapR)

Data Science Across Data Sources with Apache Arrow

Spark Summit EU talk by Christos Erotocritou

We have deployed a hybrid cloud storage solution that leverages compute in the public cloud along with our specialized hardware storage. We will discuss the tradeoffs of hybrid cloud storage, which workloads are best suited for this model, the pipeline we have deployed, and the challenges and best practices we have learned. Spark provides a flexible compute environment that can be used alongside todays cloud compute providers. However in read-heavy workloads that dominate much of analysis and machine learning today, storage costs scale poorly on these same cloud storage models. Hybrid cloud offers an alternative approach to get amortized storage costs over a dedicated link while using elastic compute in the cloud. We are currently running an end to end data science stack with multiple production workloads with this setup – A Spark-based ETL for transforming the real time log data that we ingest from our devices in the field into databases, a scale-out general regular expression search over log files that provides our support engineers real time access to searching for pathologies across our customer base, and a Spark based machine learning system for time series analysis to predict various customer metrics.

An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...

Learn how to deploy a managed Presto environment to interactively query log data on AWS Organizations often need to quickly analyze large amounts of data, such as logs, generated from a wide variety of sources and formats. However, traditional approaches require a lot of time and effort designing complex data transformation and loading processes; and configuring data warehouses. Using AWS, you can start querying your datasets within minutes In this webinar you will learn how you can deploy a managed Presto environment in minutes to interactively query log data using plain ANSI SQL. Presto is a popular open source SQL engine for running interactive analytic queries against data sources of all sizes. We will talk about common use cases and best practices for running Presto on Amazon EMR. Learning Objectives: • Learn how to deploy a managed Presto environment running on Amazon EMR • Understand best practices for running Presto on Amazon EMR, including use of Amazon EC2 Spot instances • Learn how other customers are using Presto to analyze large data sets

Running Fast, Interactive Queries on Petabyte Datasets using Presto - AWS Jul...

Metrics play an important role in data-driven companies like LinkedIn, where we leverage them extensively for reporting, experimentation, and in-product applications. We built an offline platform to help people define and produce metrics driven through their transformation code, mostly in Pig or Hive, and metadata-rich configurations. Many of our users would like to look at these metrics in a real-time fashion. To support this, we recently built an extension to the platform that auto-generates Samza real-time flow from existing offline transformation code with just a single command. Combining with the existing offline platform, we delivered Lambda architecture without maintaining multiple code bases. In this talk, we will describe how we use Apache Calcite to translate our offline logic, served as the single source of truth, into both Samza code and configuration for real-time execution.

Serverless Data Architecture at scale on Google Cloud Platform

MeetupDataScienceRoma

High Performance Data Lake with Apache Hudi and Alluxio at T3Go

Alluxio, Inc.

Conquering the Lambda architecture in LinkedIn metrics platform with Apache C...

Khai Tran

Spark and Ignite are two of the most popular open source projects in the area of high-performance Big Data and Fast Data. But did you know that one of the best ways to boost performance for your next generation real-time applications is to use them together? In this session, Christos Erotocritou – Lead GridGain solutions architect, will explain in detail how IgniteRDD – an implementation of native Spark RDD and DataFrame APIs – shares the state of the RDD across other Spark jobs, applications and workers. Christos will also demonstrate how IgniteRDD, with its advanced in-memory indexing capabilities, allows execution of SQL queries many times faster than native Spark RDDs or Data Frames. Furthermore we will be discussing the newest feature additions and what the future holds for this integration.

Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou

BigDL is a distributed deep Learning framework built for Big Data platform using Apache Spark. It combines the benefits of “high performance computing” and “Big Data” architecture, providing native support for deep learning functionalities in Spark, orders of magnitude speedup than out-of-box open source DL frameworks (e.g., Caffe/Torch) wrt single node performance (by leveraging Intel MKL), and the scale-out of deep learning workloads based on the Spark architecture. We’ll also share how our users adopt BigDL for their deep learning applications (such as image recognition, object detection, NLP, etc.), which allows them to use their Big Data (e.g., Apache Hadoop and Spark) platform as the unified data analytics platform for data storage, data processing and mining, feature engineering, traditional (non-deep) machine learning, and deep learning workloads.

BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...

Databricks is going to Strata San Jose! This presentation introduces our flagship product, Databricks Cloud. More details: Databricks Cloud combines the power of Spark with a zero-management hosted platform and an initial set of applications built around common workflows to simplify the pain of provisioning a Spark cluster, exploring data, and building data products. Spark is a unified processing engine that eliminates the need to stitch together a disjointed set of tools, and provides support for interactive queries (Spark SQL), streaming data (Spark Streaming), machine learning (MLlib) and graph computation (GraphX) in a single common API across the entire pipeline. Additionally, Databricks Cloud reaps the benefit of the rapid pace of innovation in Spark, the fastest growing Apache project with over 400 contributors

Databricks @ Strata SJ

Big data remains a rapidly evolving field with new applications and infrastructure appearing every year. In this talk, I’ll cover new trends in 2016 / 2017 and how Apache Spark is moving to meet them. In particular, I’ll talk about work Databricks is doing to make Apache Spark interact better with native code (e.g. deep learning libraries), support heterogeneous hardware, and simplify production data pipelines in both streaming and batch settings through Structured Streaming.

Trends for Big Data and Apache Spark in 2017 by Matei Zaharia

Data production continues to scale up and the techniques for managing it need to scale too. Building pipelines that can process petabytes per day in turn create data lakes with exabytes of historical data. At Databricks, we help our customers turn these data lakes into gold mines of valuable information using Apache Spark. This talk will cover techniques to optimize access to these data lakes using Delta Lakes, including range partitioning, file-based data skipping, multi-dimensional clustering, and read-optimized files. We'll cover sample implementations and see examples of querying petabytes of data in seconds, not hours. We'll also discuss tradeoffs that data engineers deal with everyday like read speed vs. write throughput, managing storage costs, and duplicating data to support multiple query profiles. We'll also discuss combining batch with streaming to achieve desired query performance. After this session, you will have new ideas for managing truly massive Delta Lakes.

Petabytes, Exabytes, and Beyond: Managing Delta Lakes for Interactive Queries...

In this session you will learn about how H&M have created a reference architecture for deploying their machine learning models on azure utilizing databricks following devOps principles. The architecture is currently used in production and has been iterated over multiple times to solve some of the discovered pain points. The team that are presenting is currently responsible for ensuring that best practices are implemented on all H&M use cases covering 100''s of models across the entire H&M group. <br> This architecture will not only give benefits to data scientist to use notebooks for exploration and modeling but also give the engineers a way to build robust production grade code for deployment. The session will in addition cover topics like lifecycle management, traceability, automation, scalability and version control.

Automated Production Ready ML at Scale

Staying competitive in a turbulent market requires more than traditional practices of manual tests and siloed development – it requires maintaining accurate, repeatable builds and predictable deployment times. Chef Automate gives you a single comprehensive workflow across your entire organization, allowing you to treat infrastructure as code and facilitates DevOps automation. In this webinar, we’ll cover some of the latest Chef integrations with AWS. Gannet, a leading media company and publisher of USA Today, will also join us to talk about how they build, test, and deliver over 400 cookbooks on AWS. They'll talk tools and process for building AMI's and managing 1,000 jobs to continuously deliver their Chef environment. Join us to learn: • How to develop at high velocity with Chef on AWS • How to create a culture of treating your AWS infrastructure as code • How Gannet uses Chef "cookbooks" on AWS to manage their USA Today infrastructure Who should attend: • CTOs, CIOs, CISOs, Directors and Managers of Security, IT Administrators, IT Architects and IT Security Engineers.

What's hot (20)

How to teach your data scientist to leverage an analytics cluster with Presto...

Make your PySpark Data Fly with Arrow!

Scaling Through Simplicity—How a 300 million User Chat App Reduced Data Engin...

Google Cloud Platform for Data Science teams

Developing high frequency indicators using real time tick data on apache supe...

Accelerate Analytics and ML in the Hybrid Cloud Era

Spark and Hadoop at Production Scale-(Anil Gadre, MapR)

Data Science Across Data Sources with Apache Arrow

Spark Summit EU talk by Christos Erotocritou

An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...

Running Fast, Interactive Queries on Petabyte Datasets using Presto - AWS Jul...

Serverless Data Architecture at scale on Google Cloud Platform

High Performance Data Lake with Apache Hudi and Alluxio at T3Go

Conquering the Lambda architecture in LinkedIn metrics platform with Apache C...

Fast Data with Apache Ignite and Apache Spark with Christos Erotocritou

BigDL: A Distributed Deep Learning Library on Spark: Spark Summit East talk b...

Databricks @ Strata SJ

Trends for Big Data and Apache Spark in 2017 by Matei Zaharia

Petabytes, Exabytes, and Beyond: Managing Delta Lakes for Interactive Queries...

Automated Production Ready ML at Scale

Similar to CICD Pipeline and delivery of Apache Spark Applications on the cloud using AWS

Automated DevOps Workflows with Chef on AWS

Continuous delivery makes teams more agile and quickens the pace of innovation. Too often, though, teams adopt continuous delivery without defining the meta components or putting the right safety mechanisms in place. In this session, we'll start from the meta components and transform a typical software release process into one that will scale and is safe. We'll use DevOps techniques like continuous integration, a variety of non-production testing stages, rollbacks, redundancy, canary deployments and synthetic tests. We'll use AWS services such as Lambda, CloudFormation, CodePipeline, CodeBuild, CodeDeploy and CloudWatch alarms and dashboards. Speakers: Brent Maxwell, Partner Solutions Architect, Amazon Web Services Daniel Zoltak, Solutions Architect, Amazon Web Services

Advanced Continuous Delivery on AWS

by Nick Brandaleone, Solutions Architect AWS Join us to learn about continuous integration, continuous delivery, and DevOps. The AWS Developer Tools have been designed based on the tools used by Amazon engineers to rapidly and reliably deliver products and features to customers. We’ll provide overviews of the services and best practices followed by a hands-on workshop to help you learn how to automate your software release processes, deploy application code, and monitor your application and infrastructure performance.

DevOps on AWS - Building Systems to Deliver Faster

Martin will show how Accenture uses Docker to build a continuous-delivery enabled development. Working against the clock, in 15 minutes: Martin will deploy a web application and development tools onto AWS. He will them make an enhancement to the deployed application completing a suite of QA gates: unit-test, functional-test, OWASP security scan, performance test and finally verify the change using A-B testing. If you aren’t familiar with DevOps / Continuous Delivery this practical example should help bring it to life.

Continuous Delivery Live

Docker, Inc.

Software release cycles are now measured in days instead of months. Cutting edge companies are continuously delivering high-quality software at a fast pace. In this session, we will cover how you begin your DevOps journey by sharing best practices and tools by the "two pizza" engineering teams at Amazon. We will showcase how you can accelerate developer productivity by implementing continuous integration and delivery workflows. We will also cover an introduction to AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and AWS CodeDeploy, the services inspired by Amazon's internal devloper tools and DevOps practices.

Intro to AWS Developer Tools feat. AWS Codestar, and AWS SDKs & Developer Res...

As software teams transition to cloud-based architectures and adopt more agile processes, the tools they need to support their development cycles will change. In this session, we'll take you through the transition that Amazon made to a service-oriented architecture over a decade ago. We will share the lessons we learned, the processes we adopted, and the tools we built to increase both our agility and reliability. We will also introduce you to AWS CodeCommit, AWS CodePipeline, and AWS CodeDeploy, three new services born out of Amazon's internal DevOps experience.

Global Azure 2024 - On-Premises to Azure Cloud: .NET Web App Journey

Callon Campbell

Build an app on aws for your first 10 million users (2)

AWS Vietnam Community

DevOps at Amazon: A Look at Our Tools and Processes

Enterprise DevOps is different then DevOps in startups and smaller companies. This session how AWS/CSC address this. How AWS IaaS level automation via CloudFormation, UserData, Console, APIS and some PaaS OpsWorks/Beanstalk is complimented by CSC Agility Platform. CSC Agility adds application compliance and security to the AWS infrastructure compliance and security. CSC Agility allows for the creation of architecture blueprints for predefined application offerings.

CSC AWS re:Invent Enterprise DevOps session

Tom Laszewski

Track 4 Session 4_ MAD02 MAD 04 如何藉由 CICD 流程管理容器化和無伺服器應用

Accelerating Software Delivery with AWS Developer Tools & AWS Mobile services...

Microsoft Azure For Solutions Architects

Roy Kim

Azure DevOps Best Practices Webinar

Cambay Digital

Intro to AWS Developer Tools, featuring AWS CodeStar

Software release cycles are now measured in days instead of months. Cutting edge companies are continuously delivering high-quality software at a fast pace. In this session, we will cover how you can begin your DevOps journey by sharing best practices and tools used by the engineering teams at Amazon. We will showcase how you can accelerate developer productivity by implementing continuous Integration and delivery workflows. We will also cover an introduction to AWS CodeStar, AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, AWS CodeDeploy, AWS Cloud9, and AWS X-Ray the services inspired by Amazon's internal developer tools and DevOps practice. Level: 200 Speaker: Nick Brandaleone - Solutions Architect, AWS

DevOps on AWS

In this session, we explore different approaches to migrating Microsoft applications to AWS. We walk through important considerations when planning a migration, and we show how to develop and implement a migration plan to move applications from on-premises (or traditional hosting) to AWS. We use a case study format to dive deep into successfully planning an application migration. Participants work in teams to plan a SharePoint migration that integrates with an existing Active Directory.

Kubernetes vs App Service

Lorenzo Barbieri

WIN401_Migrating Microsoft Applications to AWS

Today, companies seeking to innovate faster are looking for new ways to make it easier and faster for their development teams to build and deploy software. Today, many companies and organizations face some challenges in implementing a more agile, dynamic software development process. They seek ways to make it easier and faster for their development teams to build and deploy software while delivering technical innovations to the marketplace. Implementing continuous integration, continuous delivery, and continuous deployment processes can aid you in building and releasing well-tested software safely and quickly. In this session, we will discuss how you can quickly implement a CI/CD workflow using the newly released cloud service, AWS CodeStar. We will take a deep dive walkthrough into the AWS CodeStar service and how it can aid you in the accelerating application delivery by making it easier to set up your entire development and continuous delivery toolchain for coding, building, testing, and deploying your application code with many popular programming languages.

Introducing AWS CodeStar and the AWS CI:CD workflow - AWS Summit Tel Aviv 2017

The technical advantages of a microservices architecture pattern are understood by many AWS customers. In this session, the innovation advantages of microservices are explored from a business perspective together with business agility lessons learned during an evolution from a single monolithic application to cloud based microservices. Speaker: Craig Dickson, Solutions Architect, Amazon Web Services

AWS CodeStar - AWS TelAviv Summit 2017

Tara Walker

Microservices: Architecting for Innovation - Level 300

Similar to CICD Pipeline and delivery of Apache Spark Applications on the cloud using AWS (20)

Automated DevOps Workflows with Chef on AWS

Advanced Continuous Delivery on AWS

DevOps on AWS - Building Systems to Deliver Faster

Continuous Delivery Live

Intro to AWS Developer Tools feat. AWS Codestar, and AWS SDKs & Developer Res...

Global Azure 2024 - On-Premises to Azure Cloud: .NET Web App Journey

Build an app on aws for your first 10 million users (2)

DevOps at Amazon: A Look at Our Tools and Processes

CSC AWS re:Invent Enterprise DevOps session

Track 4 Session 4_ MAD02 MAD 04 如何藉由 CICD 流程管理容器化和無伺服器應用

Accelerating Software Delivery with AWS Developer Tools & AWS Mobile services...

Microsoft Azure For Solutions Architects

Azure DevOps Best Practices Webinar

Intro to AWS Developer Tools, featuring AWS CodeStar

DevOps on AWS

Kubernetes vs App Service

WIN401_Migrating Microsoft Applications to AWS

Introducing AWS CodeStar and the AWS CI:CD workflow - AWS Summit Tel Aviv 2017

AWS CodeStar - AWS TelAviv Summit 2017

Microservices: Architecting for Innovation - Level 300

More from Data Con LA

Data Con LA 2022 Keynotes

Data Con LA 2022 Keynotes

Data Con LA 2022 Keynote

Data Con LA 2022 - Startup Showcase

Data Con LA 2022 Keynote

Mike Limcaco, Analytics Specialist / Customer Engineer at Google Measure trends in a particular topic or search term across Google Search across the US down to the city-level. Integrate these data signals into analytic pipelines to drive product, retail, media (video, audio, digital content) recommendations tailored to your audience segment. We'll discuss how Google unique datasets can be used with Google Cloud smart analytic services to process, enrich and surface the most relevant product or content that matches the ever-changing interests of your local customer segment.

Data Con LA 2022 - Using Google trends data to build product recommendations

Melinda Thielbar, Data Science Practice Lead and Director of Data Science at Fidelity Investments From corporations to governments to private individuals, most of the AI community has recognized the growing need to incorporate ethics into the development and maintenance of AI models. Much of the current discussion, though, is meant for leaders and managers. This talk is directed to data scientists, data engineers, ML Ops specialists, and anyone else who is responsible for the hands-on, day-to-day of work building, productionalizing, and maintaining AI models. We'll give a short overview of the business case for why technical AI expertise is critical to developing an AI Ethics strategy. Then we'll discuss the technical problems that cause AI models to behave unethically, how to detect problems at all phases of model development, and the tools and techniques that are available to support technical teams in Ethical AI development.

Data Con LA 2022 - AI Ethics

Antje Barth, Principal Developer Advocate, AI/ML at AWS & Chris Fregly, Principal Engineer, AI & ML at AWS The frequency and severity of natural disasters are increasing. In response, governments, businesses, nonprofits, and international organizations are placing more emphasis on disaster preparedness and response. Many organizations are accelerating their efforts to make their data publicly available for others to use. Repositories such as the Registry of Open Data on AWS and Humanitarian Data Exchange contain troves of data available for use by developers, data scientists, and machine learning practitioners. In this session, see how a community of developers came together though the AWS Disaster Response hackathon to build models to support natural disaster preparedness and response.

Data Con LA 2022 - Improving disaster response with machine learning

Sig Narvaez, Executive Solution Architect at MongoDB MongoDB is now a Developer Data Platform. Come learn whatï¿½s new in the 6.0 release and Atlas following all the recent announcements made at MongoDB World 2022. Topics will include - Atlas Search which combines 3 systems into one (database, search engine, and sync mechanisms) letting you focus on your product's differentiation. - Atlas Data Federation to seamlessly query, transform, and aggregate data from one or more MongoDB Atlas databases, Atlas Data Lake and AWS S3 buckets - Queryable Encryption lets you run expressive queries on fully randomized encrypted data to meet the most stringent security requirements - Relational Migrator which analyzes your existing relational schemas and helps you design a new MongoDB schema. - And more!

Data Con LA 2022 - What's new with MongoDB 6.0 and Atlas

Jaysen Gillespie, Head of Analytics and Data Science at RTB House 1. Shopkick has over 30M downloads, but the userbase is very heterogeneous. Anecdotal evidence indicated a wide variety of users for whom the app holds long-term appeal. 2. Marketing and other teams challenged Analytics to get beyond basic summary statistics and develop a holistic segmentation of the userbase. 3. Shopkick's data science team used SQL and python to gather data, clean data, and then perform a data-driven segmentation using a k-means algorithm. 4. Interpreting the results is more work -- and more fun -- than running the algo itself. We'll discuss how we transform from ""segment 1"", ""segment 2"", etc. to something that non-analytics users (Marketing, Operations, etc.) could actually benefit from. 5. So what? How did team across Shopkick change their approach given what Analytics had discovered.

Data Con LA 2022 - Real world consumer segmentation

Ravi Pillala, Chief Data Architect & Distinguished Engineer at Intuit TurboTax is one of the well known consumer software brand which at its peak serves 385K+ concurrent users. In this session, We start with looking at how user behavioral data & tax domain events are captured in real time using the event bus and analyzed to drive real time personalization with various TurboTax data pipelines. We will also look at solutions performing analytics which make use of these events, with the help of Kafka, Apache Flink, Apache Beam, Spark, Amazon S3, Amazon EMR, Redshift, Athena and Amazon lambda functions. Finally, we look at how SageMaker is used to create the TurboTax model to predict if a customer is at risk or needs help.

Data Con LA 2022 - Modernizing Analytics & AI for today's needs: Intuit Turbo...

Data Con LA 2022 - Moving Data at Scale to AWS

Anand Ranganathan, Chief AI Officer at Unscrambl Conversational AI is getting more and more widely used for customer support and employee support use-cases. In this session, I'm going to talk about how it can be extended for data analysis and data science use-cases ... i.e., how users can interact with a bot to ask analytical questions on data in relational databases. This allows users to explore complex datasets using a combination of text and voice questions, in natural language, and then get back results in a combination of natural language and visualizations. Furthermore, it allows collaborative exploration of data by a group of users in a channel in platforms like Microsoft Teams, Slack or Google Chat. For example, a group of users in a channel can ask questions to a bot in plain English like ""How many cases of Covid were there in the last 2 months by state and gender"" or ""Why did the number of deaths from Covid increase in May 2022"", and jointly look at the results that come back. This facilitates data awareness, data-driven collaboration and joint decision making among teams in enterprises and outside. In this talk, I'll describe how we can bring together various features including natural-language understanding, NL-to-SQL translation, dialog management, data story-telling, semantic modeling of data and augmented analytics to facilitate collaborate exploration of data using conversational AI.

Data Con LA 2022 - Collaborative Data Exploration using Conversational AI

Anil Inamdar, VP & Head of Data Solutions at Instaclustr The most modernized enterprises utilize polyglot architecture, applying the best-suited database technologies to each of their organization's particular use cases. To successfully implement such an architecture, though, you need a thorough knowledge of the expansive NoSQL data technologies now available. Attendees of this Data Con LA presentation will come away with: -- A solid understanding of the decision-making process that should go into vetting NoSQL technologies and how to plan out their data modernization initiatives and migrations. -- They will learn the types of functionality that best match the strengths of NoSQL key-value stores, graph databases, columnar databases, document-type databases, time-series databases, and more. -- Attendees will also understand how to navigate database technology licensing concerns, and to recognize the types of vendors they'll encounter across the NoSQL ecosystem. This includes sniffing out open-core vendors that may advertise as â€œopen source,"" but are driven by a business model that hinges on achieving proprietary lock-in. -- Attendees will also learn to determine if vendors offer open-code solutions that apply restrictive licensing, or if they support true open source technologies like Hadoop, Cassandra, Kafka, OpenSearch, Redis, Spark, and many more that offer total portability and true freedom of use.

Data Con LA 2022 - Why Database Modernization Makes Your Data Decisions More ...

Zia Khan, Computer Systems Analyst and Data Scientist at LearningFuze Data Science tutorial is designed for people who are new to Data Science. This is a beginner level session so no prior coding or technical knowledge is required. Just bring your laptop with WiFi capability. The session starts with a review of what is data science, the amount of data we generate and how companies are using that data to get insight. We will pick a business use case, define the data science process, followed by hands-on lab using python and Jupyter notebook. During the hands-on portion we will work with pandas, numpy, matplotlib and sklearn modules and use a machine learning algorithm to approach the business use case.

Data Con LA 2022 - Intro to Data Science

Data Con LA 2022 - How are NFTs and DeFi Changing Entertainment

Curtis ODell, Global Director Data Integrity at Tricentis Join me to learn about a new end-to-end data testing approach designed for modern data pipelines that fills dangerous gaps left by traditional data management toolsâ€”one designed to handle structured and unstructured data from any source. You'll hear how you can use unique automation technology to reach up to 90 percent test coverage rates and deliver trustworthy analytical and operational data at scale. Several real world use cases from major banks/finance, insurance, health analytics, and Snowflake examples will be presented. Key Learning Objective 1. Data journeys are complex and you have to ensure integrity of the data end to end across this journey from source to end reporting for compliance 2. Data Management tools do not test data, they profile and monitor at best, and leave serious gaps in your data testing coverage 3. Automation with integration to DevOps and DataOps' CI/CD processes are key to solving this. 4. How this approach has impact in your vertical

Data Con LA 2022 - Why Data Quality vigilance requires an End-to-End, Automat...

Arif Ansari, Professor at University of Southern California Super Bowl Ad cost $7 million and each year a few Super Bowl ads go viral. The traditional A/B testing does not predict virality. Some highly shared ones reach over 60 million organic views, which can be more valuable than views on TV. Not only are these voluntary, but they are typically without distraction, and win viewer engagement in the form of likes, comments, or shares. A Super Bowl ad that wins 69 million views on YouTube (e.g., Alexa Mind Reader) costs less than 10 cents per quality view! However, the challenge is triggering virality. We developed a method to predict virality and engineer virality into Ads. 1. Prof. Gerard J. Tellis and co-authors recommended that advertisers use YouTube to tease, test, and tweak (TTT) their ads to maximize sharing and viewing. 2022 saw that maxim put into practice. 2. We developed viral Ads prediction using two scientific models: a. Prof. Gerard Tellis et al.'s model for viral prediction b. Deep Learning viral prediction using social media effect 3. The model was able to identify all the top 15 Viral Ads it performed better than the traditional agencies. 4. New proposed method is Tease, Test, Tweak, Target and Spots Ad.

Data Con LA 2022-Perfect Viral Ad prediction of Superbowl 2022 using Tease, T...

Jai Bansal, Senior Manager, Data Science at Aetna This talk describes an internal data product called Member Embeddings that facilitates modeling of member medical journeys with machine learning. Medical claims are the key data source we use to understand health journeys at Aetna. Claims are the data artifacts that result from our members' interactions with the healthcare system. Claims contain data like the amount the provider billed, the place of service, and provider specialty. The primary medical information in a claim is represented in codes that indicate the diagnoses, procedures, or drugs for which a member was billed. These codes give us a semi-structured view into the medical reason for each claim and so contain rich information about members' health journeys. However, since the codes themselves are categorical and high-dimensional (10K cardinality), it's challenging to extract insight or predictive power directly from the raw codes on a claim. To transform claim codes into a more useful format for machine learning, we turned to the concept of embeddings. Word embeddings are widely used in natural language processing to provide numeric vector representations of individual words. We use a similar approach with our claims data. We treat each claim code as a word or token and use embedding algorithms to learn lower-dimensional vector representations that preserve the original high-dimensional semantic meaning. This process converts the categorical features into dense numeric representations. In our case, we use sequences of anonymized member claim diagnosis, procedure, and drug codes as training data. We tested a variety of algorithms to learn embeddings for each type of claim code. We found that the trained embeddings showed relationships between codes that were reasonable from the point of view of subject matter experts. In addition, using the embeddings to predict future healthcare-related events outperformed other basic features, making this tool an easy way to improve predictive model performance and save data scientist time.

Data Con LA 2022- Embedding medical journeys with machine learning to improve...

Jie Chen, Manager Advisory, KPMG Data is the new oil. However, many organizations have fragmented data in siloed line of businesses. In this topic, we will focus on identifying the legacy patterns and their limitations and introducing the new patterns packed by Kafka's core design ideas. The goal is to tirelessly pursue better solutions for organizations to overcome the bottleneck in data pipelines and modernize the digital assets for ready to scale their businesses. In summary, we will walk through three uses cases, recommend Dos and Donts, Take aways for Data Engineers, Data Scientist, Data architect in developing forefront data oriented skills.

Data Con LA 2022 - Data Streaming with Kafka