Enriching data by_cooking_recipes_in_cloud_dataprep

•

1 like•162 views

This document provides an overview and agenda for a presentation on Google Cloud Dataprep. It introduces Dataprep as an intelligent data preparation tool that allows for easy and powerful data preparation in a serverless environment. The presentation will cover Dataprep features, objects, pricing, permissions, importing/exporting, creating recipes and flows, running and scheduling jobs, and include a demo. It is meant to explain what Dataprep is, where it fits in the Google Cloud Platform architecture, and how to use its capabilities.

Enriching data
by cooking
recipes in Cloud
Dataprep
with SUPRIYA BADGUJAR
July 11, 2020

Agenda
► Dataprep Overview
► Why Dataprep
► Where does it fit in GCP Architecture
► Features
► Understanding different Dataprep Objects
► Pricing, Permissions and backend
► Import/ Export
► Creating flow and recipes
► Running and scheduling Dataprep jobs
► Demo

Dataprep Overview
Intelligent data
preparation
Easy & powerful
data preparation
Serverless
simplicity
Fast exploration &
anomaly detection

Features
► Automator
► Cluster Clean
► Job Monitoring
► Macros
► Operationalization
► Parameterization
► Pattern Matching
► Predictive Transformation
► RapidTarget
► Sampling
► Sharing
► Standardization
► TBE
► Visual Profiling

Pricing, Permissions and Backend
► Pricing:
► 1.16 * cost of Dataflow job
► IAM:
► Dataprep User - Run Dataprep in a project
► Dataprep Service Agent - Gives Trifecta necessary access to project resources:
► Access GCS buckets, Dataflow Developer, BigQuery user/data editor
► Necessary for cross-project access + GCE service account
► Backend:
► Dataflow
► Supported file types:
► Input: CSV, JSON (including nested), Plain text, Excel, LOG, TSV, and Avro
► Output: CSV, JSON, Avro, BigQuery table:
► CSV/JSON can be compressed or uncompressed

Demo
Questions?
Connect with me on
https://www.linkedin.com/in/badgujarsupriya6/

(Gunnar Morling, RedHat) Kafka Summit SF 2018 Debezium (noun | de·be·zi·um | /dɪ:ˈbɪ:ziːəm/): secret sauce for change data capture (CDC) streaming changes from your datastore that enables you to solve multiple challenges: synchronizing data between microservices, gradually extracting microservices from existing monoliths, maintaining different read models in CQRS-style architectures, updating caches and full-text indexes and feeding operational data to your analytics tools Join this session to learn what CDC is about, how it can be implemented using Debezium, an open source CDC solution based on Apache Kafka and how it can be utilized for your microservices. Find out how Debezium captures all the changes from datastores such as MySQL, PostgreSQL and MongoDB, how to react to the change events in near real time and how Debezium is designed to not compromise on data correctness and completeness also if things go wrong. In a live demo we’ll show how to set up a change data stream out of your application’s database without any code changes needed. You’ll see how to sink the change events into other databases and how to push data changes to your clients using WebSockets.

#SlimScalding - Less Memory is More Capacity

Gera Shegalov

Building Pinterest Real-Time Ads Platform Using Kafka Streams

confluent

Building Pinterest Real-Time Ads Platform Using Kafka Streams (Liquan Pei + Boyang Chen, Pinterest) Kafka Summit SF 2018 In this talk, we are sharing the experience of building Pinterest’s real-time Ads Platform utilizing Kafka Streams. The real-time budgeting system is the most mission-critical component of the Ads Platform as it controls how each ad is delivered to maximize user, advertiser and Pinterest value. The system needs to handle over 50,000 queries per section (QPS) impressions, requires less than five seconds of end-to-end latency and recovers within five minutes during outages. It also needs to be scalable to handle the fast growth of Pinterest’s ads business. The real-time budgeting system is composed of real-time stream-stream joiner, real-time spend aggregator and a spend predictor. At Pinterest’s scale, we need to overcome quite a few challenges to make each component work. For example, the stream-stream joiner needs to maintain terabyte size state while supporting fast recovery, and the real-time spend aggregator needs to publish to thousands of ads servers while supporting over one million read QPS. We choose Kafka Streams as it provides milliseconds latency guarantee, scalable event-based processing and easy-to-use APIs. In the process of building the system, we performed tons of tuning to RocksDB, Kafka Producer and Consumer, and pushed several open source contributions to Apache Kafka. We are also working on adding a remote checkpoint for Kafka Streams state to reduce the time of code start when adding more machines to the application. We believe that our experience can be beneficial to people who want to build real-time streaming solutions at large scale and deeply understand Kafka Streams.

Distributing big astronomical catalogues with Greenplum - Greenplum Summit 2019

VMware Tanzu

Presto Summit 2018 - 04 - Netflix Containers

kbajda

Add Historical Analysis of Operational Data with Easy Configurations in Fivet...

Databricks

Fivetran makes it easy to automate data ingestion particularly for operational data sources such as Salesforce, Zendesk, and Oracle Eloqua, no matter how source schemas and APIs change. Achieving historical analysis is cumbersome, time-consuming, and costly to build and maintain manually. A common approach is to include snapshots, which only take into account changes at a given time. Plus, the additional storage requirements can become unwieldy to manage. Type 2 Slowly Changing Dimension (SCD) allows you to track any change at any point in time. This session shows how Fivetran History Mode, which uses Type 2 SCD, can be easily configured and then switched on with 1-click and synchronized for a desired time period. This accelerates time to insights, making it easy to both automate data ingestion and historical analysis.

Fluvius is the network operator for electricity and gas in Flanders, Belgium. Their goal is to modernize the way people look at energy consumption using a digital meter that captures consumption and injection data from any electrical installation in Flanders ranging from households to large companies. After full roll-out there will be roughly 7 million digital meters active in Flanders collecting up to terabytes of data per day. Combine this with regulation that Fluvius has to maintain a record of these reading for at least 3 years, we are talking petabyte scale. delaware BeLux was assigned by Fluvius to setup a modern data platform and did so on Azure using Databricks as the core component to collect, store, process and serve these volumes of data to every single consumer and beyond in Flanders. This enables the Belgian energy market to innovate and move forward. Maarten took up the role as project manager and solution architect.

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

Why Open Source Works for DevOps Monitoring

DevOps.com

Learn how to use open source tools for your performance monitoring of your Infrastructure, Application, & Cloud in a way that is faster, easier, and to scale. In this webinar we will provide you with step by step instruction on how to use InfluxDB, a complete Open Source Platform built from the ground up for metrics, events, and other time-based data. We will cover how to download & configure, how to collect metrics, build dashboards and alerts.

Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019

VMware Tanzu

The Fast Path to Building Operational Applications with Spark

SingleStore

Google Cloud Dataflow meets TensorFlow

Hayato Yoshikawa

Data / Streaming / Microservices Platform with Devops

Kidong Lee

Architecture Blue Print

Bogdan Nedelcu

Automating Federal Aviation Administration’s (FAA) System Wide Information Ma...

Databricks

GCP CloudRun Overview

Oliver Fierro

The Future of Data Engineering - 2019 InfoQ QConSF

Chris Riccomini

KliqMap for Esri: Actionable Location Analytics

KT-Labs

At KT Labs we strongly believe that there is much more to “Location Intelligence” that simply linking a map to a report. KliqMap leverages QlikView, Esri ArcGIS (the leading GIS platform) and KT Labs’ Planning Suite to create a fully integrated and interactive environment where analytics and simulation can be performed in the context of business processes. The native access to “ArcGIS Online” makes this solution accessible to businesses of any size with an amazing simplicity and speed of deployment. With KliqMap the GIS becomes a key element in empowering decision making through analysis, simulation, prediction and optimization. This is what we call “Actionable Location Analytics”.

empirical analysis modeling of power dissipation control in internet data ce...

saadjamil31

KliqObjects Overview

KT-Labs

Leveraging ArcGIS Online for Public Utility Data

True North Geographic Technologies

KliqPlan Overview

KT-Labs

If you love QlikView and you ever thought "Wouldn't it be great to do data-entry inside QlikView and to be able to develop budgeting and planning applications?" ... now you can, with KliqPlan and the other members of the “Planning Suite” by KT Labs. KliqPlan is the first of a new line of Extension Objects by KT Labs (QlikTech Technology Partner). It supports data-entry within QlikView with a rich set of features including: data spread&break-back, blocked totals, cells locking, workflow, smart Paste, audit trails and a table editor. Find out more on www.ktlabs.com

Graphite

David Lutz

Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016

Coburn Watson

Funnel Analysis with Apache Spark and Druid

Databricks

Every day, millions of advertising campaigns are happening around the world. As campaign owners, measuring the ongoing campaign effectiveness (e.g “how many distinct users saw my online ad VS how many distinct users saw my online ad, clicked it and purchased my product?”) is super important. However, this task (often referred to as “funnel analysis”) is not an easy task, especially if the chronological order of events matters. One way to mitigate this challenge is combining Apache Druid and Apache DataSketches, to provide fast analytics on large volumes of data. However, while that combination can answer some of these questions, it still can’t answer the question “how many distinct users viewed the brand’s homepage FIRST and THEN viewed product X page?” In this talk, we will discuss how we combine Spark, Druid and DataSketches to answer such questions at scale.

Big Data Best Practices on GCP

AllCloud

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

HostedbyConfluent

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. In the TigerGraph database, Kafka Connect framework was used to build the native S3 data loader. In TigerGraph Cloud, we will be building native integration with many data sources such as Azure Blob Storage and Google Cloud Storage using Kafka as an integrated component for the Cloud Portal. In this session, we will be discussing both architectures: 1. built-in Kafka Connect framework within TigerGraph database; 2. using Kafka cluster for cloud native integration with other popular data sources. Demo will be provided for both data streaming processes.

Dsdt meetup 2017 11-21

JDA Labs MTL

What's hot

Building the Next-gen Digital Meter Platform for Fluvius

Databricks

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

HostedbyConfluent

Why Open Source Works for DevOps Monitoring

DevOps.com

Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019

VMware Tanzu

The Fast Path to Building Operational Applications with Spark

SingleStore

Google Cloud Dataflow meets TensorFlow

Hayato Yoshikawa

Data / Streaming / Microservices Platform with Devops

Kidong Lee

Architecture Blue Print

Bogdan Nedelcu

Automating Federal Aviation Administration’s (FAA) System Wide Information Ma...

Databricks

GCP CloudRun Overview

Oliver Fierro

The Future of Data Engineering - 2019 InfoQ QConSF

Chris Riccomini

KliqMap for Esri: Actionable Location Analytics

KT-Labs

empirical analysis modeling of power dissipation control in internet data ce...

saadjamil31

KliqObjects Overview

KT-Labs

Leveraging ArcGIS Online for Public Utility Data

True North Geographic Technologies

KliqPlan Overview

KT-Labs

Graphite

David Lutz

Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016

Coburn Watson

Funnel Analysis with Apache Spark and Druid

Databricks

Big Data Best Practices on GCP

AllCloud

What's hot (20)

Building the Next-gen Digital Meter Platform for Fluvius

Building Streaming Data Pipelines with Google Cloud Dataflow and Confluent Cl...

Why Open Source Works for DevOps Monitoring

Pivotal Greenplum Cloud Marketplaces - Greenplum Summit 2019

The Fast Path to Building Operational Applications with Spark

Google Cloud Dataflow meets TensorFlow

Data / Streaming / Microservices Platform with Devops

Architecture Blue Print

Automating Federal Aviation Administration’s (FAA) System Wide Information Ma...

GCP CloudRun Overview

The Future of Data Engineering - 2019 InfoQ QConSF

KliqMap for Esri: Actionable Location Analytics

empirical analysis modeling of power dissipation control in internet data ce...

KliqObjects Overview

Leveraging ArcGIS Online for Public Utility Data

KliqPlan Overview

Graphite

Cloud Capacity Planning Tooling - South Bay SRE Meetup Aug-09-2016

Funnel Analysis with Apache Spark and Druid

Big Data Best Practices on GCP

Similar to Enriching data by_cooking_recipes_in_cloud_dataprep

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

HostedbyConfluent

Dsdt meetup 2017 11-21

JDA Labs MTL

DSDT Meetup Nov 2017

DSDT_MTL

The journey of Moving from AWS ELK to GCP Data Pipeline

Randy Huang

CodeCamp Iasi - Creating serverless data analytics system on GCP using BigQuery

Márton Kodok

Teaser: provide developers a new way of understanding advanced analytics and choosing the right cloud architecture The new buzzword is #serverless, as there are many great services that helps us abstract away the complexity associated with managing servers. In this session we will see how serverless helps on large data analytics backends. We will see how to architect for Cloud and implement into an existing project components that will take us into the #serverless architecture that will ingest our streaming data, run advanced analytics on petabytes of data using BigQuery on Google Cloud Platform - all this next to an existing stack, without being forced to reengineer our app. BigQuery enables super-fast, SQL/Javascript queries against petabytes of data using the processing power of Google’s infrastructure. We will cover its core features, SQL 2011 standard, working with streaming inserts, User Defined Functions written in Javascript, reference external JS libraries, and several use cases for everyday backend developer: funnel analytics, email heatmap, custom data processing, building dashboards, extracting data using JS functions, emitting rows based on business logic.

RAPIDS cuGraph – Accelerating all your Graph needs

Connected Data World

The relationships between data sets matter. Discovering, analyzing, and learning those relationships is a central part to expanding our understand, and is a critical step to being able to predict and act upon the data. Unfortunately, these are not always simple or quick tasks. To help the analyst we introduce RAPIDS, a collection of open-source libraries, incubated by NVIDIA and focused on accelerating the complete end-to-end data science ecosystem. Graph analytics is a critical piece of the data science ecosystem for processing linked data, and RAPIDS is pleased to offer cuGraph as our accelerated graph library. Simply accelerating algorithms only addressed a portion of the problem. To address the full problem space, RAPIDS cuGraph strives to be feature-rich, easy to use, and intuitive. Rather than limiting the solution to a single graph technology, cuGraph supports Property Graphs, Knowledge Graphs, Hyper-Graphs, Bipartite graphs, and the basic directed and undirected graph. A Python API allows the data to be manipulated as a DataFrame, similar and compatible with Pandas, with inputs and outputs being shared across the full RAPIDS suite, for example with the RAPIDS machine learning package, cuML. This talk will present an overview of RAPIDS and cuGraph. Discuss and show examples of how to manipulate and analyze bipartite and property graph, plus show how data can be shared with machine learning algorithms. The talk will include some performance and scalability metrics. Then conclude with a preview of upcoming features, like graph query language support, and the general RAPIDS roadmap.

About The Event-Driven Data Layer & Adobe Analytics

Kevin Haag

[Study Guide] Google Professional Cloud Architect (GCP-PCA) Certification

Amaaira Johns

Start Here---> https://bit.ly/3bGEd9l <---Get complete detail on GCP-PCA exam guide to crack Professional Cloud Architect. You can collect all information on GCP-PCA tutorial, practice test, books, study material, exam questions, and syllabus. Firm your knowledge on Professional Cloud Architect and get ready to crack GCP-PCA certification. Explore all information on GCP-PCA exam with the number of questions, passing percentage, and time duration to complete the test.

Better Together: How Graph database enables easy data integration with Spark ...

TigerGraph

Supercharge your data analytics with BigQuery

Márton Kodok

Powering interactive data analysis require massive architecture, and Know-How to build a fast real-time computing system. BigQuery solves this problem by enabling super-fast, SQL-like queries against petabytes of data using the processing power of Google’s infrastructure. We will cover its core features, creating tables, columns, views, working with partitions, clustering for cost optimizations, streaming inserts, User Defined Functions, and several use cases for everydaay developer: funnel analytics, behavioral analytics, exploring unstructured data. The other part will be about BigQuery ML, which enables users to create and execute machine learning models in BigQuery using standard SQL queries. BigQuery ML democratizes machine learning by enabling SQL practitioners to build models using existing SQL tools and skills. BigQuery ML increases development speed by eliminating the need to move data.

Comparing three data ingestion approaches where Apache Kafka integrates with ...

HostedbyConfluent

Using Kafka to stream data into TigerGraph, a distributed graph database, is a common pattern in our customers’ data architecture. We have seen the integration in three different layers around TigerGraph’s data flow architecture, and many key use case areas such as customer 360, entity resolution, fraud detection, machine learning, and recommendation engine. Firstly, TigerGraph’s internal data ingestion architecture relies on Kafka as an internal component. Secondly, TigerGraph has a builtin Kafka Loader, which can connect directly with an external Kafka cluster for data streaming. Thirdly, users can use an external Kafka cluster to connect other cloud data sources to TigerGraph cloud database solutions through the built-in Kafka Loader feature. In this session, we will present the high-level architecture in three different approaches and demo the data streaming process.

SEC302 Twitter's GCP Architecture for its petabyte scale data storage in gcs...

Vrushali Channapattan

Twitter collects petabytes of data every day and empowers its engineers and data scientists for large data processing with an hybrid on-premises and cloud model. In this talk, we will look at its GCP architecture and the resource hierarchy. We will deep dive into the storage design that uses Google Cloud Storage to organize petabytes of data that are replicated from on-premises HDFS clusters. We will take a look at how the user-management tooling has been designed to create and manage access for thousands of accounts (human and service accounts) at Twitter. We will talk about how the design deals with the security measures for accounts and tooling systems running in GCP and the complexities of dataset permissions. We will share the challenges we faced as we tried to design our system at scale and our learnings and solutions.

Architecting Analytic Pipelines on GCP - Chicago Cloud Conference 2020

Mariano Gonzalez

GPPB2020 - Milan - Power BI dataflows deep dive

Riccardo Perico

Rapids: Data Science on GPUs

inside-BigData.com

In this deck from FOSDEM'19, Christoph Angerer from NVIDIA presents: Rapids - Data Science on GPUs. "The next big step in data science will combine the ease of use of common Python APIs, but with the power and scalability of GPU compute. The RAPIDS project is the first step in giving data scientists the ability to use familiar APIs and abstractions while taking advantage of the same technology that enables dramatic increases in speed in deep learning. This session highlights the progress that has been made on RAPIDS, discusses how you can get up and running doing data science on the GPU, and provides some use cases involving graph analytics as motivation. GPUs and GPU platforms have been responsible for the dramatic advancement of deep learning and other neural net methods in the past several years. At the same time, traditional machine learning workloads, which comprise the majority of business use cases, continue to be written in Python with heavy reliance on a combination of single-threaded tools (e.g., Pandas and Scikit-Learn) or large, multi-CPU distributed solutions (e.g., Spark and PySpark). RAPIDS, developed by a consortium of companies and available as open source code, allows for moving the vast majority of machine learning workloads from a CPU environment to GPUs. This allows for a substantial speed up, particularly on large data sets, and affords rapid, interactive work that previously was cumbersome to code or very slow to execute. Many data science problems can be approached using a graph/network view, and much like traditional machine learning workloads, this has been either local (e.g., Gephi, Cytoscape, NetworkX) or distributed on CPU platforms (e.g., GraphX). We will present GPU-accelerated graph capabilities that, with minimal conceptual code changes, allows both graph representations and graph-based analytics to achieve similar speed ups on a GPU platform. By keeping all of these tasks on the GPU and minimizing redundant I/O, data scientists are enabled to model their data quickly and frequently, affording a higher degree of experimentation and more effective model generation. Further, keeping all of this in compatible formats allows quick movement from feature extraction, graph representation, graph analytic, enrichment back to the original data, and visualization of results. RAPIDS has a mission to build a platform that allows data scientist to explore data, train machine learning algorithms, and build applications while primarily staying on the GPU and GPU platforms." Learn more: https://rapids.ai/ and https://fosdem.org/2019/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

NVIDIA Rapids presentation

testSri1

GDG London Workshop: Build GCP infrastructure with Terraform

Pradeep Bhadani

Data Provision API with BigQuery - Google Cloud Summit Jakarta 18

Imre Nagi

BlazingSQL & Graphistry - Netflow Demo

Rodrigo Aramburu

b04-DataflowArchitecture.pdf

RAJA RAY

Similar to Enriching data by_cooking_recipes_in_cloud_dataprep (20)

How a distributed graph analytics platform uses Apache Kafka for data ingesti...

Dsdt meetup 2017 11-21

DSDT Meetup Nov 2017

The journey of Moving from AWS ELK to GCP Data Pipeline

CodeCamp Iasi - Creating serverless data analytics system on GCP using BigQuery

RAPIDS cuGraph – Accelerating all your Graph needs

About The Event-Driven Data Layer & Adobe Analytics

[Study Guide] Google Professional Cloud Architect (GCP-PCA) Certification

Better Together: How Graph database enables easy data integration with Spark ...

Supercharge your data analytics with BigQuery

Comparing three data ingestion approaches where Apache Kafka integrates with ...

SEC302 Twitter's GCP Architecture for its petabyte scale data storage in gcs...

Architecting Analytic Pipelines on GCP - Chicago Cloud Conference 2020

GPPB2020 - Milan - Power BI dataflows deep dive

Rapids: Data Science on GPUs

NVIDIA Rapids presentation

GDG London Workshop: Build GCP infrastructure with Terraform

Data Provision API with BigQuery - Google Cloud Summit Jakarta 18

BlazingSQL & Graphistry - Netflow Demo

b04-DataflowArchitecture.pdf

Recently uploaded

UiPath Test Automation using UiPath Test Suite series, part 5

DianaGray10

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Uni Systems S.M.S.A.

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

How to Get CNIC Information System with Paksim Ga.pptx

danishmna97

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI

Vladimir Iglovikov, Ph.D.

Presented by Vladimir Iglovikov: - https://www.linkedin.com/in/iglovikov/ - https://x.com/viglovikov - https://www.instagram.com/ternaus/ This presentation delves into the journey of Albumentations.ai, a highly successful open-source library for data augmentation. Created out of a necessity for superior performance in Kaggle competitions, Albumentations has grown to become a widely used tool among data scientists and machine learning practitioners. This case study covers various aspects, including: People: The contributors and community that have supported Albumentations. Metrics: The success indicators such as downloads, daily active users, GitHub stars, and financial contributions. Challenges: The hurdles in monetizing open-source projects and measuring user engagement. Development Practices: Best practices for creating, maintaining, and scaling open-source libraries, including code hygiene, CI/CD, and fast iteration. Community Building: Strategies for making adoption easy, iterating quickly, and fostering a vibrant, engaged community. Marketing: Both online and offline marketing tactics, focusing on real, impactful interactions and collaborations. Mental Health: Maintaining balance and not feeling pressured by user demands. Key insights include the importance of automation, making the adoption process seamless, and leveraging offline interactions for marketing. The presentation also emphasizes the need for continuous small improvements and building a friendly, inclusive community that contributes to the project's growth. Vladimir Iglovikov brings his extensive experience as a Kaggle Grandmaster, ex-Staff ML Engineer at Lyft, sharing valuable lessons and practical advice for anyone looking to enhance the adoption of their open-source projects. Explore more about Albumentations and join the community at: GitHub: https://github.com/albumentations-team/albumentations Website: https://albumentations.ai/ LinkedIn: https://www.linkedin.com/company/100504475 Twitter: https://x.com/albumentations

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

GraphRAG is All You need? LLM & Knowledge Graph

Guy Korland

Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs. 1. Unifying Large Language Models and Knowledge Graphs: A Roadmap. https://arxiv.org/abs/2306.08302 2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs: https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Securing your Kubernetes cluster_ a step-by-step guide to success !

KatiaHIMEUR1

Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster. However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks. In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.

GridMate - End to end testing is a critical piece to ensure quality and avoid...

ThomasParaiso2

RESUME BUILDER APPLICATION Project for students

KAMESHS29

PCI PIN Basics Webinar from the Controlcase Team

ControlCase

By Design, not by Accident - Agile Venture Bolzano 2024

Pierluigi Pugliese

Large Language Model (LLM) and it’s Geospatial Applications

Rohit Gautam

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Nexer Digital

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Recently uploaded (20)

UiPath Test Automation using UiPath Test Suite series, part 5

A tale of scale & speed: How the US Navy is enabling software delivery from l...

Uni Systems Copilot event_05062024_C.Vlachos.pdf

Video Streaming: Then, Now, and in the Future

20240605 QFM017 Machine Intelligence Reading List May 2024

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

How to Get CNIC Information System with Paksim Ga.pptx

Climate Impact of Software Testing at Nordic Testing Days

Enchancing adoption of Open Source Libraries. A case study on Albumentations.AI

Introduction to CHERI technology - Cybersecurity

GraphRAG is All You need? LLM & Knowledge Graph

The Art of the Pitch: WordPress Relationships and Sales

Securing your Kubernetes cluster_ a step-by-step guide to success !

GridMate - End to end testing is a critical piece to ensure quality and avoid...

RESUME BUILDER APPLICATION Project for students

PCI PIN Basics Webinar from the Controlcase Team

By Design, not by Accident - Agile Venture Bolzano 2024

Large Language Model (LLM) and it’s Geospatial Applications

Elizabeth Buie - Older adults: Are we really designing for our future selves?

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

Enriching data by_cooking_recipes_in_cloud_dataprep

1. Enriching data by cooking recipes in Cloud Dataprep with SUPRIYA BADGUJAR July 11, 2020

2. Agenda ► Dataprep Overview ► Why Dataprep ► Where does it fit in GCP Architecture ► Features ► Understanding different Dataprep Objects ► Pricing, Permissions and backend ► Import/ Export ► Creating flow and recipes ► Running and scheduling Dataprep jobs ► Demo

3. Dataprep Overview Intelligent data preparation Easy & powerful data preparation Serverless simplicity Fast exploration & anomaly detection

4. Why Dataprep?

5. Where does it fit in GCP Architecture?

6. Features ► Automator ► Cluster Clean ► Job Monitoring ► Macros ► Operationalization ► Parameterization ► Pattern Matching ► Predictive Transformation ► RapidTarget ► Sampling ► Sharing ► Standardization ► TBE ► Visual Profiling

7. Dataprep Objects

8. Pricing, Permissions and Backend ► Pricing: ► 1.16 * cost of Dataflow job ► IAM: ► Dataprep User - Run Dataprep in a project ► Dataprep Service Agent - Gives Trifecta necessary access to project resources: ► Access GCS buckets, Dataflow Developer, BigQuery user/data editor ► Necessary for cross-project access + GCE service account ► Backend: ► Dataflow ► Supported file types: ► Input: CSV, JSON (including nested), Plain text, Excel, LOG, TSV, and Avro ► Output: CSV, JSON, Avro, BigQuery table: ► CSV/JSON can be compressed or uncompressed

9. Dataprep UI

10. Demo Questions? Connect with me on https://www.linkedin.com/in/badgujarsupriya6/

Enriching data by_cooking_recipes_in_cloud_dataprep

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Enriching data by_cooking_recipes_in_cloud_dataprep

Similar to Enriching data by_cooking_recipes_in_cloud_dataprep (20)

Recently uploaded

Recently uploaded (20)

Enriching data by_cooking_recipes_in_cloud_dataprep