The journey of Moving from AWS ELK to GCP Data Pipeline

•

7 likes•1,338 views

This is a real case from VMfive to shifting ELK architecture from AWS. Currently GCP Data Pipeline provide us more efficiency and stable environment for running our service.

Build DMP on top of
GCP
VMFive - Randy Huang

Agenda
• Migrated Pipeline to GCP
• Cost Comparison
• Business Use Case
• Fluentd Demo

Pros & Cons
• Pros :
• Well Support.
• Well docs.
• Easy to ﬁnd Reference.
• Cons :
• High Cost.
• Not open source.
• Have to set the scale at ﬁrst.

Pipeline on GCP
Dataflow
BigQuery
Machine Learning
Data Visualization
Compute Engine
Global Load Balancing

The Products and Services logos may be used to accurately reference Google's technology and tools, for instance in architecture diagrams. 7
Batch
BI Analysis
Storage 
Cloud Storage
Processing 
Cloud DataflowStreaming
Time Series Streaming 
Cloud Pub/Sub
Storage 
BigQuery

The Products and Services logos may be used to accurately reference Google's technology and tools, for instance in architecture diagrams. 8
Targeting Engines
Data Sources
Machine Learning
Applications
API Backend 
Compute Engine
Spark MLlib 
Cloud Dataproc
App Engine
Transform Data
Hosted Models 
Cloud Machine Learning
Real-Time 
Prediction API
Device Related 
Cloud Pub/Sub
Behavior Related 
Cloud Pub/Sub
3rd Party Data 
Cloud Pub/Sub
Redis 
Compute Engine

Pros & Cons
• Pros :
• Cost-effective.
• Operation-effective.
• Google got your back.
• Cons :
• API/SDK changes everyday.
• Some still in beta mode.
• Docs everywhere.

Workﬂow Monitoring
• Digdag <Airﬂow/Oozie/Luigi>
• Native support Python & Ruby
• Multi-Cloud
• Modular
• Workﬂow as code
• Docker Support
• Altering to Slack

Cost Comparison
• $2000 on AWS per month
• about $200 on GCP production
• about another $200 for dev
• 50M events per month

Business Use Case
• Digital Ads Targeting
• User Behavior Tagging
• BI
• GEO Reporting
• KPI Reporting
• User Demographic

Some Tips
• BigQuery
• https://status.cloud.google.com/incident/bigquery/
18022
• Solved by Fluentd’s Retry and HA
• Dataﬂow’s SDK & docs is not sync
• Dataﬂow Sideinput has a bug with Streaming mode
• Compute Engine SLB - TCP/UDP setup for forwarding

Flunetd Update
• Release note for v0.14
• sub second event ﬂush
• New Plugin APIS
support formatting configurations dynamically
(e.g., path /my/dest/${tag}/mydata.%Y-%m-%d.log)
• Secure Forward

Demo
• Nginx -> Fluentd -> BigQuery -> DataStudio
• MySQL -> Fluentd -> BigQuery

Kafka Connectors are used extensively in data migration solutions, serving as a middle tier when migrating data across databases. In addition, microservice architectures also use Kafka Connectors heavily when communicating with one another while still operating independently on their own data stores. In this talk, we cover these use cases in more detail along with a deep dive into the architecture of the source and sink Kafka Connectors for Cosmos DB.

Streaming data in the cloud with Confluent and MongoDB Atlas | Robert Waters,...

HostedbyConfluent

Are you looking for a cloud-based architecture that includes the best of breed streaming and database technologies? In this session you will learn how to setup and configure the Confluent Cloud with MongoDB Atlas. We'll start the journey learning about the basic connectivity between the two cloud services and end with a brief discovery of what you can do with data once it is in MongoDB Atlas. By the end of this session you will know how to securely setup and configure the MongoDB Atlas connectors in the Confluent Cloud in both a source and sink configuration.

Serverless Reality

Lynn Langit

Building scalable data with kafka and spark

babatunde ekemode

Implementing tools, let alone an entire Unified Data Platform, like Databricks, can be quite the undertaking. Implementing a tool which you have not yet learned all the ins and outs of can be even more frustrating. Have you ever wished that you could take some of that uncertainty away? Four years ago, Western Governors University (WGU) took on the task of rewriting all of our ETL pipelines in Scala/Python, as well as migrating our Enterprise Data Warehouse into Delta, all on the Databricks platform. Starting with 4 users and rapidly growing to over 120 users across 8 business units, our Databricks environment turned into an entire unified platform, being used by individuals of all skill levels, data requirements, and internal security requirements. Through this process, our team has had the chance and opportunity to learn while making a lot of mistakes. Taking a look back at those mistakes, there are a lot of things we wish we had known before opening the platform to our enterprise. We would like to share with you 10 things we wish we had known before WGU started operating in our Databricks environment. Covering topics surrounding user management from both an AWS and Databricks perspective, understanding and managing costs, creating custom pipelines for efficient code management, learning about new Apache Spark snippets that helped save us a fortune, and more. We would like to provide our recommendations on how one can overcome these pitfalls to help new, current and prospective users to make their environments easier, safer, and more reliable to work in.

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

HostedbyConfluent

At Wells-Fargo, we move 150 TB of logs data from our syslogs to Splunk forwarders that get indexed and organized for analytic queries. As we modernize and migrate our applications to our hybrid cloud the performance expectations for this infrastructure will proportionately increase. Those improvements include the resilience of the end to end infrastructure. First, we decoupled the applications from their logging interface through a loglibrary which split the streams of logs from their sources to KAFKA which routed them to two separate destinations Splunk and ELK respectively. We also used prometheus and grafana for monitoring the metrics. We also deployed KAFKA, Splunk, ELK, Prometheus and Grafana on the Kubernetes clusters. Confluent had released a version of KAFKA without Zookeeper and replaced its functionality with Quorum Controller. The Quorum-Controller version exhibited better disposability one of the 12factors that's important for Cloud-Nativeness. We packaged this version into a Kubernetes operator called Keda and deployed this for auto-scaling. We tested this to simulate the amount of logdata that we typically generate in production. Based on the above we have also implemented distributed tracing and help make it just as resilient. We will share our lessons learnt, the patterns and practices to modernize both our underlying runtime platforms and our applications with highly performing and resilient event-driven architectures.

How Docker Accelerates Continuous Development at ironSource: Containers #101 ...

Brittany Ingram

Containers 101 meetup talk recording posted here- https://codefresh.io/blog/containers-101-meetup-docker-accelerates-continuous-development/ Shimon Tolts, General Manager/ CTO of Data Solutions at ironSouce, joined us to talk about how they leverage Docker to simplify their workflow and deliver Big Data solutions to their customers faster. He shared their experience running Docker containers in production and how they took one of their base systems, considered "the backbone of the company," and transformed it using containers.

Introduction to knime

Bernardo Najlis

Google Cloud Platform

Balvinder Hira

Introduction to Modern DevOps Technologies

Kriangkrai Chaonithi

Apache Airflow Architecture

Gerard Toonstra

Presentation given at Coolblue B.V. demonstrating Apache Airflow (incubating), what we learned from the underlying design principles and how an implementation of these principles reduce the amount of ETL effort. Why choose Airflow? Because it makes your engineering life easier, more people can contribute to how data flows through the organization, so that you can spend more time applying your brain to more difficult problems like Machine Learning, Deep Learning and higher level analysis.

Presto Summit 2018 - 03 - Starburst CBO

kbajda

Aengus Rooney [Grafana] | What's New with Grafana and InfluxDB | InfluxDays E...

InfluxData

Elastic Stack Basic - All The Capabilities in 6.3!

brad_quarry

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

As Kafka deployments grow within your organization, so do the challenges around lifecycle management. For instance, do you really know what streams exist, who is producing and consuming them? What is the effect of upstream changes? How is this information kept up to date, so it is relevant and consistent to others looking to reuse these streams? Ever wish you had a way to view and visualize graphically the relationships between schemas, topics and applications? In this talk we will show you how to do that and get more value from your Kafka Streaming infrastructure using an event portal. It’s like an API portal but specialized for event streams and publish/subscribe patterns. Join us to see how you can automatically discover event streams from your Kafka clusters, import them to a catalog and then leverage code gen capabilities to ease development of new applications.

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Confluent Cloud makes Devops engineers lives a lot more easier. Yet moving 1500 microservices, 10K topics and 100K partitions to a multi-cluster Confluent cloud can be a challenge. In this talk you will hear about 5 lessons that Wix has learned in order to successfully meet this challenge. These lessons include: 1. Automation, Automation, Automation - all the process has to be completely automated at such scale 2. Prefer a gradual approach - E.g. migrate topics in small chunks and not all at once. Reduces risks if things go bad 3. Cleanup first - avoid migrating unused topics or topics with too many unnecessary partitions

Logging in The World of DevOps

DevOps Indonesia

Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...

HostedbyConfluent

Being a pioneer in the interactive gaming industry, SONY PlayStation has played a vital role in implementing technological advancements thus help bringing global video gaming community together. With the recent launch of next generation console PS-5 into the market by partnering with thousands of game developers and millions of video gamers across the globe, humongous volumes of data generation in playstation servers is quite inevitable. This presentation talks about how we leveraged big data technologies along with Apache Kafka to solve some of the realtime data analytical problems. Two important case studies we carryout recently are: ""Competitive pricing analysis of game titles across online video game marketplaces"" & ""understand the gamers sentiment by streaming data from social feeds and perform NLP"" Along with Apache Kafka, the technologies that we have used to architect the solution are: REST API, ZooKeeper, D3.js visualization, DoMo, Python, SQL, NLP, AWS Cloud & JSON.

Real-Time Vote Platform Benchmark

Lahav Savir

An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...

Databricks

We have deployed a hybrid cloud storage solution that leverages compute in the public cloud along with our specialized hardware storage. We will discuss the tradeoffs of hybrid cloud storage, which workloads are best suited for this model, the pipeline we have deployed, and the challenges and best practices we have learned. Spark provides a flexible compute environment that can be used alongside todays cloud compute providers. However in read-heavy workloads that dominate much of analysis and machine learning today, storage costs scale poorly on these same cloud storage models. Hybrid cloud offers an alternative approach to get amortized storage costs over a dedicated link while using elastic compute in the cloud. We are currently running an end to end data science stack with multiple production workloads with this setup – A Spark-based ETL for transforming the real time log data that we ingest from our devices in the field into databases, a scale-out general regular expression search over log files that provides our support engineers real time access to searching for pathologies across our customer base, and a Spark based machine learning system for time series analysis to predict various customer metrics.

Presto Summit 2018 - 04 - Netflix Containers

kbajda

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...

HostedbyConfluent

Real-time connectivity of databases and systems is critical in enterprises adopting digital transformation to support super-fast decisioning to drive applications like fraud detection, digital payments, recommendation engines. This talk will focus on the many functions that database streaming serves with Kafka, Spark and Aerospike. We will explore how to eliminate the wall between transaction processing and analytics by synthesizing streaming data with system of record data, to gain key insights in real-time.

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

InfluxData

Red Hat is the provider of enterprise open source solutions. Its portfolio of products includes hybrid cloud infrastructure, middleware, cloud-native apps and automation solutions. Its internal network supports all lines of business — including 60+ sites. Discover how Red Hat uses InfluxDB and Flux for better real-time monitoring of their networks to improve performance and to understand utilization better.

Elastically Scaling Kafka Using Confluent

confluent

A Microservice Architecture for Big Data Pipelines

Daniel Mescheder

Building data pipelines

Jonathan Holloway

What's hot

10 Things Learned Releasing Databricks Enterprise Wide

Databricks

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

HostedbyConfluent

How Docker Accelerates Continuous Development at ironSource: Containers #101 ...

Brittany Ingram

Introduction to knime

Bernardo Najlis

Google Cloud Platform

Balvinder Hira

Introduction to Modern DevOps Technologies

Kriangkrai Chaonithi

Apache Airflow Architecture

Gerard Toonstra

Presto Summit 2018 - 03 - Starburst CBO

kbajda

Aengus Rooney [Grafana] | What's New with Grafana and InfluxDB | InfluxDays E...

InfluxData

Elastic Stack Basic - All The Capabilities in 6.3!

brad_quarry

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

HostedbyConfluent

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

HostedbyConfluent

Logging in The World of DevOps

DevOps Indonesia

Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...

HostedbyConfluent

Real-Time Vote Platform Benchmark

Lahav Savir

An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...

Databricks

Presto Summit 2018 - 04 - Netflix Containers

kbajda

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...

HostedbyConfluent

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

InfluxData

Elastically Scaling Kafka Using Confluent

confluent

What's hot (20)

10 Things Learned Releasing Databricks Enterprise Wide

Moving 150 TB of data resiliently on Kafka With Quorum Controller on Kubernet...

How Docker Accelerates Continuous Development at ironSource: Containers #101 ...

Introduction to knime

Google Cloud Platform

Introduction to Modern DevOps Technologies

Apache Airflow Architecture

Presto Summit 2018 - 03 - Starburst CBO

Aengus Rooney [Grafana] | What's New with Grafana and InfluxDB | InfluxDays E...

Elastic Stack Basic - All The Capabilities in 6.3!

How to Discover, Visualize, Catalog, Share and Reuse your Kafka Streams (Jona...

5 lessons learned for successful migration to Confluent cloud | Natan Silinit...

Logging in The World of DevOps

Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...

Real-Time Vote Platform Benchmark

An End-to-End Spark-Based Machine Learning Stack in the Hybrid Cloud with Far...

Presto Summit 2018 - 04 - Netflix Containers

Distributed Data Storage & Streaming for Real-time Decisioning Using Kafka, S...

Martin Moucka [Red Hat] | How Red Hat Uses gNMI, Telegraf and InfluxDB to Gai...

Elastically Scaling Kafka Using Confluent

Viewers also liked

A Microservice Architecture for Big Data Pipelines

Daniel Mescheder

Building data pipelines

Jonathan Holloway

Building a Big Data Pipeline

Jesus Rodriguez

Data pipelines from zero to solid

Lars Albertsson

This presentation is an attempt do demystify the practice of building reliable data processing pipelines. We go through the necessary pieces needed to build a stable processing platform: data ingestion, processing engines, workflow management, schemas, and pipeline development processes. The presentation also includes component choice considerations and recommendations, as well as best practices and pitfalls to avoid, most learnt through expensive mistakes.

Apache Beam: A unified model for batch and stream processing data

DataWorks Summit/Hadoop Summit

Airflow - An Open Source Platform to Author and Monitor Data PipelinesDataWorks Summit

(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct

Amazon Web Services

"As data volumes grow, managing and scaling data pipelines for ETL and batch processing can be daunting. With more than 13.5 million learners worldwide, hundreds of courses, and thousands of instructors, Coursera manages over a hundred data pipelines for ETL, batch processing, and new product development. In this session, we dive deep into AWS Data Pipeline and Dataduct, an open source framework built at Coursera to manage pipelines and create reusable patterns to expedite developer productivity. We share the lessons learned during our journey: from basic ETL processes, such as loading data from Amazon RDS to Amazon Redshift, to more sophisticated pipelines to power recommendation engines and search services. Attendees learn: Do's and don’ts of Data Pipeline Using Dataduct to streamline your data pipelines How to use Data Pipeline to power other data products, such as recommendation systems What’s next for Dataduct"

Deploying deep learning models with Docker and Kubernetes

PetteriTeikariPhD

Building a Data Pipeline from Scratch - Joe Crobak

Hakka Labs

Viewers also liked (9)

A Microservice Architecture for Big Data Pipelines

Building data pipelines

Building a Big Data Pipeline

Data pipelines from zero to solid

Apache Beam: A unified model for batch and stream processing data

Airflow - An Open Source Platform to Author and Monitor Data Pipelines

(BDT404) Large-Scale ETL Data Flows w/AWS Data Pipeline & Dataduct

Deploying deep learning models with Docker and Kubernetes

Building a Data Pipeline from Scratch - Joe Crobak

Similar to The journey of Moving from AWS ELK to GCP Data Pipeline

GCP Meetup #3 - Approaches to Cloud Native Architectures

nine

Openbar Kontich // Google Cloud: past, present and the (oh so sweet) future b...

Openbar

Although a giant player in anything software related Google Cloud still feels a bit under appreciated. How did it get where it is now? What are its core strengths? Most of all, we want to provide a glimpse of the future in determining major shifts in Cloud computing. Every company is a data company but their data still remains under-utilised due to a lack of execution power, let’s find this power. Due to Cloud pricing models efficient software engineering is gaining in importance, let’s unlock this efficiency. Hybrid and multi-Cloud is easily one of the largest investment domains in the Cloud world. Let’s find out why and see how we can stay vendor neutral as much as possible.

CodeCamp Iasi - Creating serverless data analytics system on GCP using BigQuery

Márton Kodok

Teaser: provide developers a new way of understanding advanced analytics and choosing the right cloud architecture The new buzzword is #serverless, as there are many great services that helps us abstract away the complexity associated with managing servers. In this session we will see how serverless helps on large data analytics backends. We will see how to architect for Cloud and implement into an existing project components that will take us into the #serverless architecture that will ingest our streaming data, run advanced analytics on petabytes of data using BigQuery on Google Cloud Platform - all this next to an existing stack, without being forced to reengineer our app. BigQuery enables super-fast, SQL/Javascript queries against petabytes of data using the processing power of Google’s infrastructure. We will cover its core features, SQL 2011 standard, working with streaming inserts, User Defined Functions written in Javascript, reference external JS libraries, and several use cases for everyday backend developer: funnel analytics, email heatmap, custom data processing, building dashboards, extracting data using JS functions, emitting rows based on business logic.

Introduction to Google Cloud Services / Platforms

Nilanchal

DevOps KPIs as a Service: Daimler’s Solution

VMware Tanzu

Serverless Comparison: AWS vs Azure vs Google vs IBM

RightScale

Code first in the cloud: going serverless with Azure

Jeremy Likness

The popularity of microservices combined with the emergence of serverless based solutions has transformed how modern developers tackle cloud native apps. Microsoft's Azure cloud provides a feature known as serverless functions (including Azure Functions and Logic Apps) that enable developers to stand up integrated end points leveraging the programming language of their choice without having to worry about the supporting infrastructure. Learn how to develop serverless .NET apps and connect them with queues, web requests, and databases or seamlessly integrate with third-party APIs like Twitter and Slack.

Building real-time data analytics on Google Cloud

Jonny Daenen

A Framework to Measure and Maximize Cloud ROI

RightScale

Best practices for developing your Magento Commerce on Cloud

Oleg Posyniak

Gcp intro-20160721

Haeseung Lee

Accessing Google Cloud APIs

wesley chun

You may know Google for search, YouTube, Android, Chrome, and Gmail, but that's only as an end-user of OUR apps. Did you know you can also integrate Google technologies into YOUR apps? We have many APIs and open source libraries that help you do that! If you have tried and found it challenging, didn't find not enough examples, run into roadblocks, got confused, or just curious about what Google APIs can offer, join us to resolve any blockers. Code samples will be in Python and/or Node.js/JavaScript. This session focuses on showing you how to access Google Cloud APIs from one of Google Cloud's compute platforms, whether serverless or otherwise.

Exploring Google APIs with Python

wesley chun

30-45-min tech talk given at user groups or technical conferences to introducing developers to integrating with Google APIs from Python . ABSTRACT Want to integrate Google technologies into the web+mobile apps that you build? Google has various open source libraries & developer tools that help you do exactly that. Users who have run into roadblocks like authentication or found our APIs confusing/challenging, are welcome to come and make these non-issues moving forward. Learn how to leverage the power of Google technologies in the next apps you build!!

Session 4 GCCP.pptx

DSCIITPatna

Kube journey 2017-04-19

Doug Davis

Azure and web sites hackaton deck

Alexey Bokov

Powerful Google Cloud tools for your hack

wesley chun

This 1-hour presentation is meant to give univeresity hackathoners a deeper yes still high-level overview of Google Cloud and its developer APIs with the purpose of inspiring students to consider these products for their hacks. It follows and dives deeper into the products introduced at the opening ceremony lightning talk. Of particular focus are the serverless and machine learning platforms & APIs... tools that have an immediate impact on projects, alleviating the need to manage VMs, operating systems, etc., as well as dispensing with the need to have expertise with machine learning.

Google Cloud Platform Introduction - 2016Q3

Simon Su

[Cloud OnAir] Talks by DevRel Vol.4 データ管理とデータベース 2020年8月27日放送

Google Cloud Platform - Japan

!GDSC NYUST Infrastructure and Application Modernization with Google Cloud .pptx

GangTingFan

Similar to The journey of Moving from AWS ELK to GCP Data Pipeline (20)

GCP Meetup #3 - Approaches to Cloud Native Architectures

Openbar Kontich // Google Cloud: past, present and the (oh so sweet) future b...

CodeCamp Iasi - Creating serverless data analytics system on GCP using BigQuery

Introduction to Google Cloud Services / Platforms

DevOps KPIs as a Service: Daimler’s Solution

Serverless Comparison: AWS vs Azure vs Google vs IBM

Code first in the cloud: going serverless with Azure

Building real-time data analytics on Google Cloud

A Framework to Measure and Maximize Cloud ROI

Best practices for developing your Magento Commerce on Cloud

Gcp intro-20160721

Accessing Google Cloud APIs

Exploring Google APIs with Python

Session 4 GCCP.pptx

Kube journey 2017-04-19

Azure and web sites hackaton deck

Powerful Google Cloud tools for your hack

Google Cloud Platform Introduction - 2016Q3

[Cloud OnAir] Talks by DevRel Vol.4 データ管理とデータベース 2020年8月27日放送

!GDSC NYUST Infrastructure and Application Modernization with Google Cloud .pptx

Recently uploaded

Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B

Sreedhar Chowdam

AP LAB PPT.pdf ap lab ppt no title specific

BrazilAccount1

Investor-Presentation-Q1FY2024 investor presentation document.pptx

AmarGB2

Hierarchical Digital Twin of a Naval Power System

Kerry Sado

A hierarchical digital twin of a Naval DC power system has been developed and experimentally verified. Similar to other state-of-the-art digital twins, this technology creates a digital replica of the physical system executed in real-time or faster, which can modify hardware controls. However, its advantage stems from distributing computational efforts by utilizing a hierarchical structure composed of lower-level digital twin blocks and a higher-level system digital twin. Each digital twin block is associated with a physical subsystem of the hardware and communicates with a singular system digital twin, which creates a system-level response. By extracting information from each level of the hierarchy, power system controls of the hardware were reconfigured autonomously. This hierarchical digital twin development offers several advantages over other digital twins, particularly in the field of naval power systems. The hierarchical structure allows for greater computational efficiency and scalability while the ability to autonomously reconfigure hardware controls offers increased flexibility and responsiveness. The hierarchical decomposition and models utilized were well aligned with the physical twin, as indicated by the maximum deviations between the developed digital twin hierarchy and the hardware.

CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx

R&R Consult

CFD analysis is incredibly effective at solving mysteries and improving the performance of complex systems! Here's a great example: At a large natural gas-fired power plant, where they use waste heat to generate steam and energy, they were puzzled that their boiler wasn't producing as much steam as expected. R&R and Tetra Engineering Group Inc. were asked to solve the issue with reduced steam production. An inspection had shown that a significant amount of hot flue gas was bypassing the boiler tubes, where the heat was supposed to be transferred. R&R Consult conducted a CFD analysis, which revealed that 6.3% of the flue gas was bypassing the boiler tubes without transferring heat. The analysis also showed that the flue gas was instead being directed along the sides of the boiler and between the modules that were supposed to capture the heat. This was the cause of the reduced performance. Based on our results, Tetra Engineering installed covering plates to reduce the bypass flow. This improved the boiler's performance and increased electricity production. It is always satisfying when we can help solve complex challenges like this. Do your systems also need a check-up or optimization? Give us a call! Work done in cooperation with James Malloy and David Moelling from Tetra Engineering. More examples of our work https://www.r-r-consult.dk/en/cases-en/

Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf

WENKENLI1

CME397 Surface Engineering- Professional Elective

karthi keyan

Student information management system project report ii.pdf

Kamal Acharya

ML for identifying fraud using open blockchain data.pptx

Vijay Dialani, PhD

English lab ppt no titlespecENG PPTt.pdf

BrazilAccount1

Water Industry Process Automation and Control Monthly - May 2024.pdf

Water Industry Process Automation & Control

Welcome to WIPAC Monthly the magazine brought to you by the LinkedIn Group Water Industry Process Automation & Control. In this month's edition, along with this month's industry news to celebrate the 13 years since the group was created we have articles including A case study of the used of Advanced Process Control at the Wastewater Treatment works at Lleida in Spain A look back on an article on smart wastewater networks in order to see how the industry has measured up in the interim around the adoption of Digital Transformation in the Water Industry.

The Benefits and Techniques of Trenchless Pipe Repair.pdf

Pipe Restoration Solutions

Explore the innovative world of trenchless pipe repair with our comprehensive guide, "The Benefits and Techniques of Trenchless Pipe Repair." This document delves into the modern methods of repairing underground pipes without the need for extensive excavation, highlighting the numerous advantages and the latest techniques used in the industry. Learn about the cost savings, reduced environmental impact, and minimal disruption associated with trenchless technology. Discover detailed explanations of popular techniques such as pipe bursting, cured-in-place pipe (CIPP) lining, and directional drilling. Understand how these methods can be applied to various types of infrastructure, from residential plumbing to large-scale municipal systems. Ideal for homeowners, contractors, engineers, and anyone interested in modern plumbing solutions, this guide provides valuable insights into why trenchless pipe repair is becoming the preferred choice for pipe rehabilitation. Stay informed about the latest advancements and best practices in the field.

在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样

obonagu

学校原件一模一样【微信：741003700 】《(ANU毕业证书)澳洲国立大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原。 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 【主营项目】一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Immunizing Image Classifiers Against Localized Adversary Attacks

gerogepatton

This paper addresses the vulnerability of deep learning models, particularly convolutional neural networks (CNN)s, to adversarial attacks and presents a proactive training technique designed to counter them. We introduce a novel volumization algorithm, which transforms 2D images into 3D volumetric representations. When combined with 3D convolution and deep curriculum learning optimization (CLO), itsignificantly improves the immunity of models against localized universal attacks by up to 40%. We evaluate our proposed approach using contemporary CNN architectures and the modified Canadian Institute for Advanced Research (CIFAR-10 and CIFAR-100) and ImageNet Large Scale Visual Recognition Challenge (ILSVRC12) datasets, showcasing accuracy improvements over previous techniques. The results indicate that the combination of the volumetric input and curriculum learning holds significant promise for mitigating adversarial attacks without necessitating adversary training.

Nuclear Power Economics and Structuring 2024

Massimo Talia

ethical hacking in wireless-hacking1.ppt

Jayaprasanna4

RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...

thanhdowork

space technology lecture notes on satellite

ongomchris

block diagram and signal flow graph representation

Divya Somashekar

AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf

SamSarthak3

Recently uploaded (20)

Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B

AP LAB PPT.pdf ap lab ppt no title specific

Investor-Presentation-Q1FY2024 investor presentation document.pptx

Hierarchical Digital Twin of a Naval Power System

CFD Simulation of By-pass Flow in a HRSG module by R&R Consult.pptx

Governing Equations for Fundamental Aerodynamics_Anderson2010.pdf

CME397 Surface Engineering- Professional Elective

Student information management system project report ii.pdf

ML for identifying fraud using open blockchain data.pptx

English lab ppt no titlespecENG PPTt.pdf

Water Industry Process Automation and Control Monthly - May 2024.pdf

The Benefits and Techniques of Trenchless Pipe Repair.pdf

在线办理(ANU毕业证书)澳洲国立大学毕业证录取通知书一模一样

Immunizing Image Classifiers Against Localized Adversary Attacks

Nuclear Power Economics and Structuring 2024

ethical hacking in wireless-hacking1.ppt

RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...

space technology lecture notes on satellite

block diagram and signal flow graph representation

AKS UNIVERSITY Satna Final Year Project By OM Hardaha.pdf

The journey of Moving from AWS ELK to GCP Data Pipeline

1. Build DMP on top of GCP VMFive - Randy Huang

2. Agenda • Migrated Pipeline to GCP • Cost Comparison • Business Use Case • Fluentd Demo

3. ELK + AWS EMR Kinesis Lambda

4. Pros & Cons • Pros : • Well Support. • Well docs. • Easy to ﬁnd Reference. • Cons : • High Cost. • Not open source. • Have to set the scale at ﬁrst.

5. Pipeline on GCP Dataflow BigQuery Machine Learning Data Visualization Compute Engine Global Load Balancing

6. Datastudio

7. The Products and Services logos may be used to accurately reference Google's technology and tools, for instance in architecture diagrams. 7 Batch BI Analysis Storage  Cloud Storage Processing  Cloud DataflowStreaming Time Series Streaming  Cloud Pub/Sub Storage  BigQuery

8. The Products and Services logos may be used to accurately reference Google's technology and tools, for instance in architecture diagrams. 8 Targeting Engines Data Sources Machine Learning Applications API Backend  Compute Engine Spark MLlib  Cloud Dataproc App Engine Transform Data Hosted Models  Cloud Machine Learning Real-Time  Prediction API Device Related  Cloud Pub/Sub Behavior Related  Cloud Pub/Sub 3rd Party Data  Cloud Pub/Sub Redis  Compute Engine

9. Pros & Cons • Pros : • Cost-effective. • Operation-effective. • Google got your back. • Cons : • API/SDK changes everyday. • Some still in beta mode. • Docs everywhere.

10. Workflow Monitoring • Digdag <Airflow/Oozie/Luigi> • Native support Python & Ruby • Multi-Cloud • Modular • Workflow as code • Docker Support • Altering to Slack

11. Digdag Sample

12. Digdag

13.

14. Cost Comparison • $2000 on AWS per month • about $200 on GCP production • about another $200 for dev • 50M events per month

15. Business Use Case • Digital Ads Targeting • User Behavior Tagging • BI • GEO Reporting • KPI Reporting • User Demographic

16. Some Tips • BigQuery • https://status.cloud.google.com/incident/bigquery/ 18022 • Solved by Fluentd’s Retry and HA • Dataﬂow’s SDK & docs is not sync • Dataﬂow Sideinput has a bug with Streaming mode • Compute Engine SLB - TCP/UDP setup for forwarding

17. Flunetd Update • Release note for v0.14 • sub second event ﬂush • New Plugin APIS support formatting configurations dynamically (e.g., path /my/dest/${tag}/mydata.%Y-%m-%d.log) • Secure Forward

18. Demo • Nginx -> Fluentd -> BigQuery -> DataStudio • MySQL -> Fluentd -> BigQuery

The journey of Moving from AWS ELK to GCP Data Pipeline

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to The journey of Moving from AWS ELK to GCP Data Pipeline

Similar to The journey of Moving from AWS ELK to GCP Data Pipeline (20)

Recently uploaded

Recently uploaded (20)

The journey of Moving from AWS ELK to GCP Data Pipeline