Apache Beam: Lote portátil y procesamiento de transmisión

•

0 likes•50 views

Speaker: Albert Ramírez Cerquera Video: https://youtu.be/18Tbr1LZmtM En esta charla, vemos cómo utilizar el modelo de programación Apache Beam para procesar datos por batch y en streaming. Además, se enseñará cómo se puede ejecutar Beam en ejecutores como Flink y Google Cloud Dataflow. --------------------------------------------------------------------------------------------------------------------------------------------------------------- Hay trabajos y hay carreras. Las oportunidades vienen a golpear la puerta cuando menos lo esperas. La decisión es tuya. Desde tener la oportunidad de hacer algo significativo día tras día, hasta estar rodeado de gente supremamente inteligente y motivada. ¿Estás listo? Descúbre todas nuestras oportunidades acá: https://bit.ly/2PWKky9 --------------------------------------------------------------------------------------------------------------------------------------------------------------- Síguenos en: Facebook: https://www.facebook.com/Globant/ Twitter: https://twitter.com/Globant Instagram: https://www.instagram.com/globantpics/ Linkedin: https://www.linkedin.com/company/globant Visita nuestra página web: https://bit.ly/2XLVYQD

Technology

Portable Batch and Streaming Processing
APACHE BEAM

ABOUT ME
● Data Architect at Globant
● Certified Google Data
Engineer
● Applying software engineering
practices to Big Data
development

AGENDA
1. Introduction
2. What is Apache Beam?
3. Why Apache Beam?
4. Crash course
5. Beam on Flink
6. Beam on DataFlow
7. Summary

INTRODUCTION
Computation
Result
Computation Computation Computation Computation
Result

INTRODUCTION
Computation Computation Computation Computation
Result
Computation Computation Computation Computation
Result
Batch Streaming

WHAT IS APACHE BEAM?
● Original code from Google (Cloud Dataflow SDK).
● Apache open-source project since 2016.
● Parallel/distributed data processing.
● Unified programming model for batch and streaming.
● Portable, execution engine and programming language
of your choice.

CRASH COURSE
Input Transform PCollection Transform PCollection Sink
Pipeline
Output

PIPELINE
A Pipeline encapsulates your entire data processing task, from start to finish. This
includes reading input data, transforming that data, and writing output data.

PCollection
The PCollection abstraction represents a potentially distributed, multi-element data
set. You can think of a PCollection as “pipeline” data; Beam transforms use
PCollection objects as inputs and outputs. As such, if you want to work with data in
your pipeline, it must be in the form of a PCollection.

PCollection
● The elements of a PCollection may be of any type, but
must all be of the same type.
● A PCollection is immutable.
● A PCollection can be either bounded or unbounded in
size.
● Each element in a PCollection has an associated intrinsic
timestamp.

PCollection
PCollection PCollection
Pipeline

Transforms
● Read - parallel connectors to external systems
● ParDo - per element processing
● GroupByKey - aggregating elements per key and window
● Flatten - union of PCollections
● Window - set the windowing strategy for a PCollection

Transforms
Transform PCollection Transform PCollection Sink
Pipeline

Pipeline I/O
Input Transform PCollection Transform PCollection Sink
Pipeline
Output

BE AWARE OF
Portable protocol
JAVA Objects

Summary
● Apache Beam is not an execution engine, is a programming model.
● Apache Flink and Google Cloud Dataflow are the execution engines
that supports all the features of Beam.
● The JAVA SDK is the the most up to date.
● PCollections are the main Data Structure in Beam.
● Transformations receive PCollections as inputs and produces the
same as outputs.
● Check the compatibility table before jump into the development.

What's hot

Spring, Spring Boot and Spring Cloud are tools that allow developers to speed up the creation of new business features. But a new feature is only useful if it's in production. Companies spend a lot of time and resources on building their own deployment pipelines using a plethora of technologies. Spring Cloud Pipelines provides an opinionated way for getting your features to production in a fast, reliable, reproducible and fully automated way.

Cloud Native CI/CD with Spring Cloud Pipelines

Lars Rosenquist

Go programming language

GoWitek Consulting Pvt.Ltd

A Functional Approach to Architecture - Kafka & Kafka Streams - Kevin Mas Rui...

Thoughtworks

Five Real-World Strategies for Perforce Streams

Perforce

Building your first aplication using Apache Apex

Yogi Devendra Vyavahare

Bodywork - GitOps for Machine Learning

Alex Ioannides

Go lambda-presentation

Steven White

Php Inspections (EA Extended): realtime static code analysis

Vladimir Reznichenko

Multitenant SaaS Apps In Rails By Iqbal Hasnan

iqbal hasnan

Open source tools for logic synthesis and soc design an overview

Vaibhav R

Traditional DevOps wisdom is to utilize automation tools for configuring all the systems, so all pieces of configuration should be described with code stored in version control. Then the people responsible for these systems can use standard Dev practices to collaborate on this code (reviews, tests, quality gates, etc.). Kubernetes provides a declarative API, so you can describe the desired state of the system. And the control plane ensures that the actual state converges to the desired state. But we still need config mgmt for API objects to the point when they are applied to the cluster. Helm helps organizing these configs into charts, template them, and manage releases. And GitOps lets you use a git repo as a single source of truth for the desired state of the whole system. Then all changes to this state are delivered as git commits instead of using kubectl apply or helm upgrade. In this talk I introduce the GitOps model for operating cloud native environments and give a short demo of the following tools: Flux CD, Helm, Flux Helm Operator.

Config management for kubernetes: GitOps + Helm

Tomasz Tarczyński

Trailblazer Rails Architecture

iqbal hasnan

RxSwift

Kristaps Grinbergs

Layer by Layer: Printing Your Own External Input Plugin for Telegraf The goal of this talk is to give listeners the context they need to create their own external plugin today. Sebastian provides the how-to knowledge by going over an external input plugin he created for monitoring 3D printing, starting with the minimum requirements to get started and then showing how he extended it for his own personal use case. The talk highlights how choosing to create your plugin idea externally lets you own your plugin’s full lifecycle and is a great way to immediately begin using it with Telegraf.

Sebastian Spaink [InfluxData] | Layer by Layer: Printing Your Own External In...

InfluxData

https://www.youtube.com/watch?v=YmIAatr3Who Presented at Cloud and AI DevFest GDG Montreal on September 27, 2019. Are you looking to get more flexibility out of your CICD platform? Interested how GitOps fits into the mix? Learn how Argo CD, Workflows, and Events can be combined to craft custom CICD flows. All while staying Kubernetes native, enabling you to leverage existing observability tooling.

Designing a complete ci cd pipeline using argo events, workflow and cd products

Julian Mazzitelli

Berlin AWS meetup: here.com on AWS

Cristian Măgherușan-Stanciu

helm, the real world

Oleg Suharevich

44CON 2014 - Binary Protocol Analysis with CANAPE, James Forshaw CANAPE is an open source network proxy written in .NET. It has been developed to aid in the analysis and exploitation of unknown application network protocols using a similar use case to common HTTP proxies such as Burp or CAT. This workshop will go through the basics of analysing an unknown application protocol with hands on training examples. By the end of the workshop candidates should be able to better understand CANAPE’s functionality and be able to apply that to other protocols they come across.

44CON 2014 - Binary Protocol Analysis with CANAPE, James Forshaw

44CON

Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors. In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others. I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.

THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io

DevOpsDays Tel Aviv

.Net Introduction

Muzzammil Wani

What's hot (20)

Cloud Native CI/CD with Spring Cloud Pipelines

Go programming language

A Functional Approach to Architecture - Kafka & Kafka Streams - Kevin Mas Rui...

Five Real-World Strategies for Perforce Streams

Building your first aplication using Apache Apex

Bodywork - GitOps for Machine Learning

Go lambda-presentation

Php Inspections (EA Extended): realtime static code analysis

Multitenant SaaS Apps In Rails By Iqbal Hasnan

Open source tools for logic synthesis and soc design an overview

Config management for kubernetes: GitOps + Helm

Trailblazer Rails Architecture

RxSwift

Sebastian Spaink [InfluxData] | Layer by Layer: Printing Your Own External In...

Designing a complete ci cd pipeline using argo events, workflow and cd products

Berlin AWS meetup: here.com on AWS

helm, the real world

44CON 2014 - Binary Protocol Analysis with CANAPE, James Forshaw

THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io

.Net Introduction

Similar to Apache Beam: Lote portátil y procesamiento de transmisión

Introduction to Apache Beam

Knoldus Inc.

The world of big data involves an ever-changing field of players. Much as SQL stands as a lingua franca for declarative data analysis, Apache Beam aims to provide a portable standard for expressing robust, out-of-order data processing pipelines in a variety of languages across a variety of platforms. In a way, Apache Beam is a glue that can connect the big data ecosystem together; it enables users to "run any data processing pipeline anywhere." This talk will briefly cover the capabilities of the Beam model for data processing and discuss its architecture, including the portability model. We’ll focus on the present state of the community and the current status of the Beam ecosystem. We’ll cover the state of the art in data processing and discuss where Beam is going next, including completion of the portability framework and the Streaming SQL. Finally, we’ll discuss areas of improvement and how anybody can join us on the path of creating the glue that interconnects the big data ecosystem. Speaker Davor Bonaci, Apache Software Foundation; Simbly, V.P. of Apache Beam; Founder/CEO at Operiant

Present and future of unified, portable, and efficient data processing with A...

DataWorks Summit

Brisbane MuleSoft Meetup 2023-03-22 - Anypoint Code Builder and Splunk Loggin...

BrianFraser29

Introduction to DevOps and the Practical Use Cases at Credit OK

Kriangkrai Chaonithi

Apache Beam was open sourced by the big data team at Google in 2016, and has become an active community with participants from all over. Beam is a framework to define data processing workflows and run them on various runners (Flink included). In this talk, I will talk about some cool things you can do with Beam + Flink such as running pipelines written in Go and Python; then I’ll mention some cool tools in the Beam ecosystem. Finally, we’ll wrap up with some cool things we expect to be able to do soon - and how you can get involved.

Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...

Flink Forward

Apache Beam is a top-level Apache project which aims at providing a unified API for efficient and portable data processing pipeline. Beam handles both batch and streaming use cases and neatly separates properties of the data from runtime characteristics, allowing pipelines to be portable across multiple runtimes, both open-source (e.g., Apache Flink, Apache Spark, Apache Apex, ...) and proprietary (e.g., Google Cloud Dataflow). This talk will cover the basics of Apache Beam, describe the main concepts of the programming model and talk about the current state of the project (new python support, first stable version). We'll illustrate the concepts with a use case running on several runners.

Portable batch and streaming pipelines with Apache Beam (Big Data Application...

Malo Denielou

Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...

Kaxil Naik

LCU14 310- Cisco ODP v2

Linaro

Prefect Paris Airflow Meetup Jeff Hale April 2023.pdf

Jeff Hale

Apache Beam @ GCPUG.TW Flink.TW 20161006

Randy Huang

SpringOne 2016 in a nutshell

Jeroen Resoort

Unbounded, unordered, global scale datasets are increasingly common in day-to-day business, and consumers of these datasets have detailed requirements for latency, cost, and completeness. Apache Beam defines a new data processing programming model that evolved from more than a decade of experience building Big Data infrastructure within Google, including MapReduce, FlumeJava, Millwheel, and Cloud Dataflow. Apache Beam handles both batch and streaming use cases, offering a powerful, unified model. It neatly separates properties of the data from run-time characteristics, allowing pipelines to be portable across multiple run-time environments, both open source, including Apache Apex, Apache Flink, Apache Gearpump, Apache Spark, and proprietary. Finally, Beam's model enables newer optimizations, like dynamic work rebalancing and autoscaling, resulting in an efficient execution. This talk will cover the basics of Apache Beam, touch on its evolution, and describe main concepts in its powerful programming model. We'll show how Beam unifies batch and streaming use cases, and show efficient execution in real-world scenarios. Finally, we'll demonstrate pipeline portability across Apache Apex, Apache Flink, Apache Spark and Google Cloud Dataflow in a live setting.

Unified, Efficient, and Portable Data Processing with Apache Beam

DataWorks Summit/Hadoop Summit

Portable Streaming Pipelines with Apache Beam

confluent

Apache Spark has been gaining steam, with rapidity, both in the headlines and in real-world adoption. Spark was developed in 2009, and open sourced in 2010. Since then, it has grown to become one of the largest open source communities in big data with over 200 contributors from more than 50 organizations. This open source analytics engine stands out for its ability to process large volumes of data significantly faster than contemporaries such as MapReduce, primarily owing to in-memory storage of data on its own processing framework. That being said, one of the top real-world industry use cases for Apache Spark is its ability to process ‘streaming data‘.

Structured Streaming in Spark

Digital Vidya

Apache Beam and Google Cloud Dataflow - IDG - final

Sub Szabolcs Feczak

"OpenLineage is an open platform for the collection and analysis of data lineage, which includes an open standard for lineage data collection, integration libraries for the most common tools, and a metadata repository/reference implementation (Marquez). In recent months, stream processing, which is an important use case for Apache Kafka, has gained the particular focus of the OpenLineage community with many useful features completed or begun, including: * A seamless OpenLineage & Apache Flink integration, * Support for streaming jobs in Marquez, * Progress on a built-in lineage API within the Flink codebase. Cross-platform lineage allows for a holistic overview of data flow and its dependencies within organizations, including stream processing. This talk will provide an overview of the most recent developments in the OpenLineage Flink integration and share what’s in store for this important collaboration. This talk is a must-attend for those wishing to stay up-to-date on lineage developments in the stream processing world."

OpenLineage for Stream Processing | Kafka Summit London

HostedbyConfluent

DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...

Haggai Philip Zagury

Google Cloud Dataflow

Alex Van Boxel

CCT is a Web App developed for an IT Project Management exam. This Web App hasbeen developed using various tools including Python, MySQL, Flask, Anaconda, and VisualStudio Code. An app aimed at companies able to manage the transfers of a worker withinthe company, taking into account all the costs that he faces and allowing him to updatethem through this webApp. Finally, to make understanding easier, there was a graphicalinterface dedicated to understanding costs, transfers through plots.

CCT (Check and Calculate Transfer)

Francesca Pappalardo

Presentation CCT

Francesca Pappalardo

Similar to Apache Beam: Lote portátil y procesamiento de transmisión (20)

Introduction to Apache Beam

Present and future of unified, portable, and efficient data processing with A...

Brisbane MuleSoft Meetup 2023-03-22 - Anypoint Code Builder and Splunk Loggin...

Introduction to DevOps and the Practical Use Cases at Credit OK

Flink Forward San Francisco 2019: Apache Beam portability in the times of rea...

Portable batch and streaming pipelines with Apache Beam (Big Data Application...

Apache Airflow in the Cloud: Programmatically orchestrating workloads with Py...

LCU14 310- Cisco ODP v2

Prefect Paris Airflow Meetup Jeff Hale April 2023.pdf

Apache Beam @ GCPUG.TW Flink.TW 20161006

SpringOne 2016 in a nutshell

Unified, Efficient, and Portable Data Processing with Apache Beam

Portable Streaming Pipelines with Apache Beam

Structured Streaming in Spark

Apache Beam and Google Cloud Dataflow - IDG - final

OpenLineage for Stream Processing | Kafka Summit London

DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...

Google Cloud Dataflow

CCT (Check and Calculate Transfer)

Presentation CCT

More from Globant

Webinar MLOps: When AA gets serious.

Globant

Google Cloud Spanner y NewSQL

Globant

Eventos Asíncronos como estrategia virtual

Globant

Cultura y valores 4.0 para líderes 4.0

Globant

Tech Insiders Salesforce: SFDX e Integración Continua

Globant

Como impulsar tu carrera Salesforce

Globant

3D Programming Basics: WebGL

Globant

Converge augmented report

Globant

Sistema de recomendación entiempo real usando Delta Lake

Globant

Kubeflow: Machine Learning en Cloud para todos

Globant

Orquestando Pipelines de Datosen AWS con Step Function y AWS Glue

Globant

Cómo las empresas de servicios financieros de Latam pueden utilizar las tecnologías digitales para acelerar su evolución y éxito en la "nueva normalidad" Ante una época de cambios constantes, expectativas crecientes de los clientes, y del surgimiento disruptivo de nuevas tecnologías y nuevos competidores, los límites de las industrias se desdibujan y los modelos de negocios actuales comienzan a ser insuficientes para la generación sostenible de valor. La industria de Servicios Financieros está viviendo una época de profunda transformación. Aquellas compañías con modelos de negocio y de servicio tradicionales, están viendo cada vez más afectadas su capacidad de sumar nuevos clientes, crecer y generar nuevos negocios, como así también su capacidad de atraer talento y desarrollar capacidades competitivas diferenciales. Las respuestas tradicionales a estos desafíos, aunque pueden ayudar muchas veces a reducir las brechas, no logran solucionar los desafíos más profundos relativos a la agilidad organizacional y cultural, la flexibilidad y escalabilidad tecnológica, y la evolución de paradigmas pasados que deben actualizarse a los tiempos presentes. El contexto actual generado por la Pandemia ha acelerado nuestras vidas digitales y de las empresas de manera drástica, dejando en claro la urgencia y criticidad de la transformación y evolución necesaria en cada aspecto de nuestras Organizaciones y la manera en que creamos y entregamos valor a los usuarios. Frente a un futuro incierto, cada vez más volátil y dinámico, la “Nueva Normalidad” obliga a los Bancos a repensar sus modelos de negocio, cartera de servicios, procesos internos, modelos de relacionamiento con sus clientes, su visión del ecosistema y de cómo ser relevantes en la vida de sus clientes. En un mundo en donde la agilidad y capacidad de adaptación es una de las ventajas competitivas más necesarias, la Industria de Servicios Financieros se encuentra ante el mayor desafío de transformación de las últimas décadas.

Navegando el desafío de transformación digital de los servicios financieros

Globant

Converge 2020

Globant

Converge 2020

Globant

Video acá: https://youtu.be/awL0c7qUyu8 En esta charla, Edgar Pérez -Tech Manager en Globant, nos brinda un análisis general a las últimas tendencias y desafíos en la industria TI para la nueva fuerza laboral. ¿Qué temáticas se abordarán? - Lo que la industria está solicitando hoy en el mercado laboral internacional. - Herramientas para enfocar intereses profesionales, conectando lo que se aprende en la academia con el mundo real en el trabajo. ___________________________________________________ Hay trabajos y hay carreras. Las oportunidades vienen a golpear la puerta cuando menos lo esperas. La decisión es tuya. Desde tener la oportunidad de hacer algo significativo día tras día, hasta estar rodeado de gente supremamente inteligente y motivada. ¿Estás listo? Descubre todas nuestras oportunidades acá: https://mycareer.globant.com/ ------------------------------------------------------------------------------------- Síguenos en: Facebook: https://www.facebook.com/Globant/ Twitter: https://twitter.com/Globant Instagram: https://www.instagram.com/globantpics/ Linkedin: https://www.linkedin.com/company/globant

Tendencias de tecnología para el recién egresado

Globant

Video: https://youtu.be/_ENsu84pH2U Hay trabajos y hay carreras. Las oportunidades vienen a golpear la puerta cuando menos lo esperas. La decisión es tuya. Desde tener la oportunidad de hacer algo significativo día tras día, hasta estar rodeado de gente supremamente inteligente y motivada. ¿Estás listo? Descúbre todas nuestras oportunidades acá:https://mycareer.globant.com/ --------------------------------------------------------------------------------------------------------------------------------------------------------------- Siguenos en: Facebook: https://www.facebook.com/Globant/ Twitter: https://twitter.com/Globant Instagram: https://www.instagram.com/globantpics/ Linkedin: https://www.linkedin.com/company/globant

SRE: ¿Qué es y cómo gestionar el Toil?

Globant

Monitoreo en tiempo real para la mejora continua de una aplicación

Globant

Video : https://youtu.be/pU5bt1M27X4 Hay trabajos y hay carreras. Las oportunidades vienen a golpear la puerta cuando menos lo esperas. La decisión es tuya. Desde tener la oportunidad de hacer algo significativo día tras día, hasta estar rodeado de gente supremamente inteligente y motivada. ¿Estás listo? Descúbre todas nuestras oportunidades acá:https://mycareer.globant.com/ --------------------------------------------------------------------------------------------------------------------------------------------------------------- Siguenos en: Facebook: https://www.facebook.com/Globant/ Twitter: https://twitter.com/Globant Instagram: https://www.instagram.com/globantpics/ Linkedin: https://www.linkedin.com/company/globant

¿Cómo automatizar pruebas de infraestructura y no morir en el intento?

Globant

Video: https://youtu.be/64DUg97We84 Hay trabajos y hay carreras. Las oportunidades vienen a golpear la puerta cuando menos lo esperas. La decisión es tuya. Desde tener la oportunidad de hacer algo significativo día tras día, hasta estar rodeado de gente supremamente inteligente y motivada. ¿Estás listo? Descúbre todas nuestras oportunidades acá:https://mycareer.globant.com/ --------------------------------------------------------------------------------------------------------------------------------------------------------------- Siguenos en: Facebook: https://www.facebook.com/Globant/ Twitter: https://twitter.com/Globant Instagram: https://www.instagram.com/globantpics/ Linkedin: https://www.linkedin.com/company/globant

Automatización en AWS con Chatbot Serverless (Amazon Lex)

Globant

¿Cómo importar funcionalidades de C++ a NodeJS?

Globant

More from Globant (20)

Webinar MLOps: When AA gets serious.

Google Cloud Spanner y NewSQL

Eventos Asíncronos como estrategia virtual

Cultura y valores 4.0 para líderes 4.0

Tech Insiders Salesforce: SFDX e Integración Continua

Como impulsar tu carrera Salesforce

3D Programming Basics: WebGL

Converge augmented report

Sistema de recomendación entiempo real usando Delta Lake

Kubeflow: Machine Learning en Cloud para todos

Orquestando Pipelines de Datosen AWS con Step Function y AWS Glue

Navegando el desafío de transformación digital de los servicios financieros

Converge 2020

Tendencias de tecnología para el recién egresado

SRE: ¿Qué es y cómo gestionar el Toil?

Monitoreo en tiempo real para la mejora continua de una aplicación

¿Cómo automatizar pruebas de infraestructura y no morir en el intento?

Automatización en AWS con Chatbot Serverless (Amazon Lex)

¿Cómo importar funcionalidades de C++ a NodeJS?

Recently uploaded

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

Tech Trends Report 2024 Future Today Institute.pdf

hans926745

GenCyber Cyber Security Day Presentation

Michael W. Hawkins

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Advantages of Hiring UIUX Design Service Providers for Your Business

Pixlogix Infotech

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Data Cloud, More than a CDP by Matt Robison

Anna Loughnan Colquhoun

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

What is a good lead in your organisation? Which leads are priority? What happens to leads? When sales and marketing give different answers to these questions, or perhaps aren't sure of the answers at all, frustrations build and opportunities are left on the table. Join us for an illuminating session with Cian McLoughlin, HubSpot Principal Customer Success Manager, as we look at that crucial piece of the customer journey in which leads are transferred from marketing to sales.

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

HampshireHUG

presentation ICT roal in 21st century education

jfdjdjcjdnsjd

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Recently uploaded (20)

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Tech Trends Report 2024 Future Today Institute.pdf

GenCyber Cyber Security Day Presentation

GenAI Risks & Security Meetup 01052024.pdf

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Advantages of Hiring UIUX Design Service Providers for Your Business

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Data Cloud, More than a CDP by Matt Robison

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Driving Behavioral Change for Information Management through Data-Driven Gree...

Partners Life - Insurer Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Handwritten Text Recognition for manuscripts and early printed texts

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

04-2024-HHUG-Sales-and-Marketing-Alignment.pptx

presentation ICT roal in 21st century education

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Apache Beam: Lote portátil y procesamiento de transmisión

1. Portable Batch and Streaming Processing APACHE BEAM

2. ABOUT ME ● Data Architect at Globant ● Certified Google Data Engineer ● Applying software engineering practices to Big Data development

3. AGENDA 1. Introduction 2. What is Apache Beam? 3. Why Apache Beam? 4. Crash course 5. Beam on Flink 6. Beam on DataFlow 7. Summary

4. INTRODUCTION Computation Result Computation Computation Computation Computation Result

5. INTRODUCTION Computation Computation Computation Computation Result Computation Computation Computation Computation Result Batch Streaming

6. WHAT IS APACHE BEAM? ● Original code from Google (Cloud Dataflow SDK). ● Apache open-source project since 2016. ● Parallel/distributed data processing. ● Unified programming model for batch and streaming. ● Portable, execution engine and programming language of your choice.

7. WHY APACHE BEAM? Write Translate

8. SDK INSTALLATION

9. CRASH COURSE Input Transform PCollection Transform PCollection Sink Pipeline Output

10. PIPELINE A Pipeline encapsulates your entire data processing task, from start to finish. This includes reading input data, transforming that data, and writing output data.

11. PIPELINE Pipeline

12. PCollection The PCollection abstraction represents a potentially distributed, multi-element data set. You can think of a PCollection as “pipeline” data; Beam transforms use PCollection objects as inputs and outputs. As such, if you want to work with data in your pipeline, it must be in the form of a PCollection.

13. PCollection

14. PCollection ● The elements of a PCollection may be of any type, but must all be of the same type. ● A PCollection is immutable. ● A PCollection can be either bounded or unbounded in size. ● Each element in a PCollection has an associated intrinsic timestamp.

15. PCollection PCollection PCollection Pipeline

16. Transforms

17. Transforms

18. Transforms

19. Transforms ● Read - parallel connectors to external systems ● ParDo - per element processing ● GroupByKey - aggregating elements per key and window ● Flatten - union of PCollections ● Window - set the windowing strategy for a PCollection

20. Transforms Transform PCollection Transform PCollection Sink Pipeline

21. Pipeline I/O

22. Pipeline I/O

23. Pipeline I/O Input Transform PCollection Transform PCollection Sink Pipeline Output

24. Beam on Flink

25. Beam on Google Cloud DataFlow

26. BE AWARE OF Write Translate

27. BE AWARE OF Portable protocol JAVA Objects

28. What is being computed?

29. Where in event time?

30. Summary ● Apache Beam is not an execution engine, is a programming model. ● Apache Flink and Google Cloud Dataflow are the execution engines that supports all the features of Beam. ● The JAVA SDK is the the most up to date. ● PCollections are the main Data Structure in Beam. ● Transformations receive PCollections as inputs and produces the same as outputs. ● Check the compatibility table before jump into the development.

31. Q&A

32. Thank you for listening!

Apache Beam: Lote portátil y procesamiento de transmisión

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Apache Beam: Lote portátil y procesamiento de transmisión

Similar to Apache Beam: Lote portátil y procesamiento de transmisión (20)

More from Globant

More from Globant (20)

Recently uploaded

Recently uploaded (20)

Apache Beam: Lote portátil y procesamiento de transmisión