This document discusses learnings from Google's experience with microservices and the Stubby framework. It covers why HTTP/JSON is insufficient, the importance of establishing common protocols and data formats, designing for fault tolerance, collecting service analytics and tracing, and load balancing. It then introduces gRPC as an open source framework that addresses these lessons by providing language-independent service definitions, performance via HTTP/2, pluggable features, and usability across platforms and languages.
The document discusses gRPC, an open-source framework for building distributed applications and services. It was motivated by Google's experience with internal RPC systems handling over 1010 RPCs per second. gRPC uses HTTP/2 for transport, allowing it to take advantage of existing network infrastructure while enabling features like bidirectional streaming. It generates client and server code from IDL definitions, supporting multiple languages.
gRPC is an open source high performance RPC framework that can connect services across data centers efficiently using pluggable support for load balancing, tracing, health checking and authentication. It supports languages like Go, Java, Python and C++ and can connect devices, mobile apps, and browsers to backend services. gRPC uses HTTP/2 for communication over TCP, supports Protocol Buffers and JSON, and offers features like request multiplexing and server push. Many large companies use gRPC in production including Google, Square, Dropbox, and Netflix.
- gRPC is an open source RPC framework originally developed by Google in 2015. It uses HTTP/2 for transport, Protocol Buffers as the interface definition language, and provides features like authentication, bidirectional streaming and interface definitions.
- Compared to REST, gRPC is faster, more efficient through binary encoding (Protocol Buffers), supports bidirectional streaming, and generates client and server code. However, it lacks browser support and has fewer available tools.
- gRPC is best suited for internal microservices communication where high performance is required and tight coupling is acceptable. It supports unary, server streaming, client streaming and bidirectional streaming RPC patterns.
Slides for the talk in Ruby SG meetup. (Apr 2018)
https://github.com/sathiyaseelan/grpc-client-ruby
https://github.com/sathiyaseelan/grpc-client-go
https://github.com/sathiyaseelan/grpc-rails-api
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, GoogleAmbassador Labs
Varun Talwar, product manager on Google's gRPC project discusses the fundamentals and specs of gRPC inside of a Google-scale microservices architecture.
Google and Intel speak on NFV and SFC service delivery
The slides are as presented at the meet up "Out of Box Network Developers" sponsored by Intel Networking Developer Zone
Here is the Agenda of the slides:
How DPDK, RDT and gRPC fit into SDI/SDN, NFV and OpenStack
Key Platform Requirements for SDI
SDI Platform Ingredients: DPDK, IntelⓇRDT
gRPC Service Framework
IntelⓇ RDT and gRPC service framework
Przemek Nowakowski: Podejście mikroserwisowe w naszym projekcie, oprócz oczywistych zalet, potrafi wygenerować też szereg problemów – między innymi z komunikacją pomiędzy poszczególnymi serwisami. W prezentacji omówimy od podstaw framework gRPC, dzięki któremu będziemy w stanie w szybki i przyjemny sposób połączyć nasze serwisy. Sprawdzimy też, do czego się nadaje, do czego nie oraz prześledzimy podstawowe problemy, jakie możemy napotkać przy takim podejściu. Nie obejdzie się również bez porównania z obecnymi na rynku alternatywami i sprawdzenia, co z tą obiecywaną wysoką wydajnością.
This document discusses learnings from Google's experience with microservices and the Stubby framework. It covers why HTTP/JSON is insufficient, the importance of establishing common protocols and data formats, designing for fault tolerance, collecting service analytics and tracing, and load balancing. It then introduces gRPC as an open source framework that addresses these lessons by providing language-independent service definitions, performance via HTTP/2, pluggable features, and usability across platforms and languages.
The document discusses gRPC, an open-source framework for building distributed applications and services. It was motivated by Google's experience with internal RPC systems handling over 1010 RPCs per second. gRPC uses HTTP/2 for transport, allowing it to take advantage of existing network infrastructure while enabling features like bidirectional streaming. It generates client and server code from IDL definitions, supporting multiple languages.
gRPC is an open source high performance RPC framework that can connect services across data centers efficiently using pluggable support for load balancing, tracing, health checking and authentication. It supports languages like Go, Java, Python and C++ and can connect devices, mobile apps, and browsers to backend services. gRPC uses HTTP/2 for communication over TCP, supports Protocol Buffers and JSON, and offers features like request multiplexing and server push. Many large companies use gRPC in production including Google, Square, Dropbox, and Netflix.
- gRPC is an open source RPC framework originally developed by Google in 2015. It uses HTTP/2 for transport, Protocol Buffers as the interface definition language, and provides features like authentication, bidirectional streaming and interface definitions.
- Compared to REST, gRPC is faster, more efficient through binary encoding (Protocol Buffers), supports bidirectional streaming, and generates client and server code. However, it lacks browser support and has fewer available tools.
- gRPC is best suited for internal microservices communication where high performance is required and tight coupling is acceptable. It supports unary, server streaming, client streaming and bidirectional streaming RPC patterns.
Slides for the talk in Ruby SG meetup. (Apr 2018)
https://github.com/sathiyaseelan/grpc-client-ruby
https://github.com/sathiyaseelan/grpc-client-go
https://github.com/sathiyaseelan/grpc-rails-api
Bringing Learnings from Googley Microservices with gRPC - Varun Talwar, GoogleAmbassador Labs
Varun Talwar, product manager on Google's gRPC project discusses the fundamentals and specs of gRPC inside of a Google-scale microservices architecture.
Google and Intel speak on NFV and SFC service delivery
The slides are as presented at the meet up "Out of Box Network Developers" sponsored by Intel Networking Developer Zone
Here is the Agenda of the slides:
How DPDK, RDT and gRPC fit into SDI/SDN, NFV and OpenStack
Key Platform Requirements for SDI
SDI Platform Ingredients: DPDK, IntelⓇRDT
gRPC Service Framework
IntelⓇ RDT and gRPC service framework
Przemek Nowakowski: Podejście mikroserwisowe w naszym projekcie, oprócz oczywistych zalet, potrafi wygenerować też szereg problemów – między innymi z komunikacją pomiędzy poszczególnymi serwisami. W prezentacji omówimy od podstaw framework gRPC, dzięki któremu będziemy w stanie w szybki i przyjemny sposób połączyć nasze serwisy. Sprawdzimy też, do czego się nadaje, do czego nie oraz prześledzimy podstawowe problemy, jakie możemy napotkać przy takim podejściu. Nie obejdzie się również bez porównania z obecnymi na rynku alternatywami i sprawdzenia, co z tą obiecywaną wysoką wydajnością.
gRPC is an open source RPC framework that makes it easy to build a distributed system across multiple languages. It uses HTTP/2 for transport, has features like streaming, load balancing and authentication built-in. It is used widely at Google and is now available open source with implementations in 10 languages. gRPC benefits from being layered on HTTP/2 for interoperability and has a pluggable architecture for advanced features like monitoring and proxies.
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...AboutYouGmbH
The document discusses gRPC, an open source framework for building microservices. It was created by Google to be a high performance, open source universal RPC framework. gRPC uses HTTP/2 for transport, Protocol Buffers as the interface definition language, and generated client/server code for many languages to make cross-platform communications simple and efficient. The document provides an overview of gRPC's goals and architecture, how to define a service using .proto files, and examples of common RPC patterns like unary, streaming, and bidirectional calls.
Presentation at NetPonto community: "We’re going to discuss gRPC, Google’s open-source RPC framework. I’ll dive a bit into the history of RPC as a protocol, and what its historical use has been. I’ll also highlight some benefits to adopt gRPC and how its possible to swap out parts of gRPC and still take advantage of gRPC’s benefits. Finally I’ll answer the question that has been on many lips since gRPC was announced — what does this mean for REST?"
Back in 2015, Square and Google collaborated to launch gRPC, an open source RPC framework backed by protocol buffers and HTTP/2, based on real-world experiences operating microservices at scale. If you build microservices, you will be interested in gRPC.
This webcast covers:
- a technical overview of gRPC
- use cases and applicability in your stack
- a deep dive into the practicalities of operationalizing gRPC
Generating Unified APIs with Protocol Buffers and gRPCC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2k1eZ00.
Chris Roche and Christopher Burnett discuss how they extended the Protocol Buffer (PB) IDL to create unified APIs and data models. From validation logic to automatic logging and statistics, PBs allows them to speed up development across Go, Python, and PHP stacks. They also show how they leverage the Envoy proxy to transparently upgrade HTTP/1.1 services to speak gRPC on the wire. Filmed at qconnewyork.com.
Chris Roche is a Core Libraries Engineer at Lyft focused on enriching the tools to create distributed systems. Christopher Burnett is Lead Core Libraries Engineer at Lyft, where he designs software for distributed systems.
This document discusses gRPC, an open-source high performance RPC framework created by Google. It provides benefits over REST such as binary encoding which allows for higher performance, as well as a contract-first approach using protocol buffers to define services. Protocol buffers can generate code for multiple platforms and allow flexible evolution of structures. gRPC uses HTTP/2 for a single connection with header compression. While REST is still necessary for some use cases, gRPC provides higher performance for internal APIs and microservices.
gRPC is an open source framework that allows for communication between services using HTTP/2 and Protocol Buffers. It provides features like low latency and high scalability. Key benefits include focusing on service design, language interoperability, and growing community support. gRPC uses Protocol Buffers for serialization, HTTP/2 for transport, and an IDL for service definitions. It supports various request-response and streaming call types and provides libraries in many languages.
In this talk we will discuss the use of gRPC to build a microservices-based application on Kubernetes. We’ll firstly focus on the basic concepts of gRPC: protocol buffers, code generation and HTTP 2.0. We’ll then discuss some more intermediate-level features, such as native Kubernetes load balancing, gRPC streaming, error propagation, HTTP tunneling and instrumentation. Finally we’ll touch on some of the newer, exciting developments in the space: middleware and context propagation for distributed tracing, gRPC Web for client-side rich web apps and gogoproto for blazing-fast generated code. This talk is based on my experience building a hosted, Prometheus-based monitoring service, which itself used gRPC for all intra-server communication and runs on Kubernetes. There’ll be live coding, real world examples and discussion of the benefits (and drawbacks) of gRPC.
Presented at GDG Devfest Ukraine 14th Oct 2017.
This document compares REST and gRPC for microservices communication. It discusses how gRPC addresses limitations of REST like inefficient binary encoding and lack of bidirectional streaming. gRPC uses protocol buffers for interface definition and code generation. It supports multiple languages and platforms and uses HTTP/2 for performance. Companies like Google and Netflix use gRPC for its performance advantages. The document provides examples of unary, server streaming, client streaming and bidirectional streaming RPC calls in gRPC. It recommends using gRPC for microservices and client-server applications where performance is important.
gRPC is a remote procedure call framework developed by Google that uses HTTP/2 for transport, Protocol Buffers as the interface definition language, and provides features such as authentication, bidirectional streaming and blocking or nonblocking bindings. It aims to be fast, lightweight, easy to use and supports many languages. Key benefits include low latency using HTTP/2, efficient serialization with Protocol Buffers and multi-language support.
This document provides an introduction and overview of GRPC, a remote procedure call framework developed by Google. It discusses the history and limitations of previous RPC technologies like CORBA, DCOM, RMI, SOAP, and REST. GRPC aims to provide a simple, high performance, language-agnostic, and extensible RPC framework. It uses protocol buffers for defining service interfaces and HTTP/2 for transport, allowing for features like bi-directional streaming. The document also covers how GRPC supports versioning of services and error handling.
Introduction to gRPC: A general RPC framework that puts mobile and HTTP/2 fir...Codemotion
gRPC is a high performance, language-neutral, general RPC framework developed and open sourced by Google. Built on the HTTP/2 standard, gRPC brings many benefits such as bidirectional streaming, flow control, header compression, multiplexing and more. In this session, you will learn about gRPC and how you can use it in your applications.
gRPC in Golang presentation
In this talk, I introduced gRPC, Protocol buffer, and how to use them with golang.
Source code used in the presentation: http://github.com/AlmogBaku/grpc-in-go
The document compares REST and gRPC approaches to building APIs. It notes that while REST uses JSON over HTTP, gRPC uses protocol buffers over HTTP/2, allowing for benefits like binary encoding, multiplexing, and simpler parsing. An example shows a character request is 205 characters in JSON but only 44 bytes in protocol buffers. gRPC is also said to allow lower latency, better CPU utilization, and server push capabilities compared to REST for real-time use cases.
A short introductory talk given as part of the April 2018 Kong meetup "Introducing Kubernetes Ingress Controller for Kong".
This talk covers the new features and improvements made to Kong from 2017 to 2018, including the groundwork conducted by Kong Inc. and open source contributors that allowed for the development of the Kong Ingress Controller for Kubernetes.
The Kong Ingress Controller for Kubernetes was then announced during the meetup:
https://github.com/Kong/kubernetes-ingress-controller
gRPC is a modern high performance RPC (Remote Procedure Call) framework that can run in any environment. gRPC is based on HTTP/2, Protocol Buffers and other modern standard-based technologies. Together with excellent tooling, it helps you create high throughput, low latency, real-time services.
Apache Deep Learning 201 - Philly Open SourceTimothy Spann
#phillyopensource
Introduction talk for data engineers for deep learning on apache with apache mxnet, apache nifi, apache hive, apache hadoop, apache spark, python and other tools.
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments.
This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.
gRPC is designed to be fast and high performance. It's basically can overcome performance issues caused by REST communications between different dependent microservices.
gRPC supports Binary data transfer in the form of Protocol Buffers which makes it extremely efficient in the Microservice cloud environment.
The use of containers to simplify and speed the deployment and development of applications is taking off. Most container usage is around stateless micro-services, but data and transactions are key components of most applications.
This presentation reviews:
- The purpose of containers and their usage
- How to containerize your EDB Postgres deployment
- How to deal with issues of managing your database and storage
- How to set up a cluster for high availability
- How to build a container with the EDB Postgres Enterprise Manager Agent in the container
Target Audience:
This technical presentation is for DBAs, Data Architects, Developers, DevOps, IT Operations and anyone responsible for supporting a Postgres interested in learning about Containers. It is equally suitable for organizations using community PostgreSQL as well as EDB’s Postgres Plus product family.
To listen to the recording which includes a demonstration, visit EnterpriseDB > Resources > Webcasts
OpenTSDB is used at Criteo for monitoring their large Hadoop infrastructure which includes over 2500 servers running many different services like HDFS, YARN, HBase, Kafka, and Storm. OpenTSDB was chosen because it can handle the scale of metrics collected, store metrics for long periods of time with fine-grained resolution, and is easily extensible to add new metrics. It uses HBase for storage which is optimized for the time series data stored in OpenTSDB and can scale to meet Criteo's needs of storing billions of data points and handling high query loads.
gRPC is an open source RPC framework that makes it easy to build a distributed system across multiple languages. It uses HTTP/2 for transport, has features like streaming, load balancing and authentication built-in. It is used widely at Google and is now available open source with implementations in 10 languages. gRPC benefits from being layered on HTTP/2 for interoperability and has a pluggable architecture for advanced features like monitoring and proxies.
Robert Kubis - gRPC - boilerplate to high-performance scalable APIs - code.t...AboutYouGmbH
The document discusses gRPC, an open source framework for building microservices. It was created by Google to be a high performance, open source universal RPC framework. gRPC uses HTTP/2 for transport, Protocol Buffers as the interface definition language, and generated client/server code for many languages to make cross-platform communications simple and efficient. The document provides an overview of gRPC's goals and architecture, how to define a service using .proto files, and examples of common RPC patterns like unary, streaming, and bidirectional calls.
Presentation at NetPonto community: "We’re going to discuss gRPC, Google’s open-source RPC framework. I’ll dive a bit into the history of RPC as a protocol, and what its historical use has been. I’ll also highlight some benefits to adopt gRPC and how its possible to swap out parts of gRPC and still take advantage of gRPC’s benefits. Finally I’ll answer the question that has been on many lips since gRPC was announced — what does this mean for REST?"
Back in 2015, Square and Google collaborated to launch gRPC, an open source RPC framework backed by protocol buffers and HTTP/2, based on real-world experiences operating microservices at scale. If you build microservices, you will be interested in gRPC.
This webcast covers:
- a technical overview of gRPC
- use cases and applicability in your stack
- a deep dive into the practicalities of operationalizing gRPC
Generating Unified APIs with Protocol Buffers and gRPCC4Media
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2k1eZ00.
Chris Roche and Christopher Burnett discuss how they extended the Protocol Buffer (PB) IDL to create unified APIs and data models. From validation logic to automatic logging and statistics, PBs allows them to speed up development across Go, Python, and PHP stacks. They also show how they leverage the Envoy proxy to transparently upgrade HTTP/1.1 services to speak gRPC on the wire. Filmed at qconnewyork.com.
Chris Roche is a Core Libraries Engineer at Lyft focused on enriching the tools to create distributed systems. Christopher Burnett is Lead Core Libraries Engineer at Lyft, where he designs software for distributed systems.
This document discusses gRPC, an open-source high performance RPC framework created by Google. It provides benefits over REST such as binary encoding which allows for higher performance, as well as a contract-first approach using protocol buffers to define services. Protocol buffers can generate code for multiple platforms and allow flexible evolution of structures. gRPC uses HTTP/2 for a single connection with header compression. While REST is still necessary for some use cases, gRPC provides higher performance for internal APIs and microservices.
gRPC is an open source framework that allows for communication between services using HTTP/2 and Protocol Buffers. It provides features like low latency and high scalability. Key benefits include focusing on service design, language interoperability, and growing community support. gRPC uses Protocol Buffers for serialization, HTTP/2 for transport, and an IDL for service definitions. It supports various request-response and streaming call types and provides libraries in many languages.
In this talk we will discuss the use of gRPC to build a microservices-based application on Kubernetes. We’ll firstly focus on the basic concepts of gRPC: protocol buffers, code generation and HTTP 2.0. We’ll then discuss some more intermediate-level features, such as native Kubernetes load balancing, gRPC streaming, error propagation, HTTP tunneling and instrumentation. Finally we’ll touch on some of the newer, exciting developments in the space: middleware and context propagation for distributed tracing, gRPC Web for client-side rich web apps and gogoproto for blazing-fast generated code. This talk is based on my experience building a hosted, Prometheus-based monitoring service, which itself used gRPC for all intra-server communication and runs on Kubernetes. There’ll be live coding, real world examples and discussion of the benefits (and drawbacks) of gRPC.
Presented at GDG Devfest Ukraine 14th Oct 2017.
This document compares REST and gRPC for microservices communication. It discusses how gRPC addresses limitations of REST like inefficient binary encoding and lack of bidirectional streaming. gRPC uses protocol buffers for interface definition and code generation. It supports multiple languages and platforms and uses HTTP/2 for performance. Companies like Google and Netflix use gRPC for its performance advantages. The document provides examples of unary, server streaming, client streaming and bidirectional streaming RPC calls in gRPC. It recommends using gRPC for microservices and client-server applications where performance is important.
gRPC is a remote procedure call framework developed by Google that uses HTTP/2 for transport, Protocol Buffers as the interface definition language, and provides features such as authentication, bidirectional streaming and blocking or nonblocking bindings. It aims to be fast, lightweight, easy to use and supports many languages. Key benefits include low latency using HTTP/2, efficient serialization with Protocol Buffers and multi-language support.
This document provides an introduction and overview of GRPC, a remote procedure call framework developed by Google. It discusses the history and limitations of previous RPC technologies like CORBA, DCOM, RMI, SOAP, and REST. GRPC aims to provide a simple, high performance, language-agnostic, and extensible RPC framework. It uses protocol buffers for defining service interfaces and HTTP/2 for transport, allowing for features like bi-directional streaming. The document also covers how GRPC supports versioning of services and error handling.
Introduction to gRPC: A general RPC framework that puts mobile and HTTP/2 fir...Codemotion
gRPC is a high performance, language-neutral, general RPC framework developed and open sourced by Google. Built on the HTTP/2 standard, gRPC brings many benefits such as bidirectional streaming, flow control, header compression, multiplexing and more. In this session, you will learn about gRPC and how you can use it in your applications.
gRPC in Golang presentation
In this talk, I introduced gRPC, Protocol buffer, and how to use them with golang.
Source code used in the presentation: http://github.com/AlmogBaku/grpc-in-go
The document compares REST and gRPC approaches to building APIs. It notes that while REST uses JSON over HTTP, gRPC uses protocol buffers over HTTP/2, allowing for benefits like binary encoding, multiplexing, and simpler parsing. An example shows a character request is 205 characters in JSON but only 44 bytes in protocol buffers. gRPC is also said to allow lower latency, better CPU utilization, and server push capabilities compared to REST for real-time use cases.
A short introductory talk given as part of the April 2018 Kong meetup "Introducing Kubernetes Ingress Controller for Kong".
This talk covers the new features and improvements made to Kong from 2017 to 2018, including the groundwork conducted by Kong Inc. and open source contributors that allowed for the development of the Kong Ingress Controller for Kubernetes.
The Kong Ingress Controller for Kubernetes was then announced during the meetup:
https://github.com/Kong/kubernetes-ingress-controller
gRPC is a modern high performance RPC (Remote Procedure Call) framework that can run in any environment. gRPC is based on HTTP/2, Protocol Buffers and other modern standard-based technologies. Together with excellent tooling, it helps you create high throughput, low latency, real-time services.
Apache Deep Learning 201 - Philly Open SourceTimothy Spann
#phillyopensource
Introduction talk for data engineers for deep learning on apache with apache mxnet, apache nifi, apache hive, apache hadoop, apache spark, python and other tools.
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments.
This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.
gRPC is designed to be fast and high performance. It's basically can overcome performance issues caused by REST communications between different dependent microservices.
gRPC supports Binary data transfer in the form of Protocol Buffers which makes it extremely efficient in the Microservice cloud environment.
The use of containers to simplify and speed the deployment and development of applications is taking off. Most container usage is around stateless micro-services, but data and transactions are key components of most applications.
This presentation reviews:
- The purpose of containers and their usage
- How to containerize your EDB Postgres deployment
- How to deal with issues of managing your database and storage
- How to set up a cluster for high availability
- How to build a container with the EDB Postgres Enterprise Manager Agent in the container
Target Audience:
This technical presentation is for DBAs, Data Architects, Developers, DevOps, IT Operations and anyone responsible for supporting a Postgres interested in learning about Containers. It is equally suitable for organizations using community PostgreSQL as well as EDB’s Postgres Plus product family.
To listen to the recording which includes a demonstration, visit EnterpriseDB > Resources > Webcasts
OpenTSDB is used at Criteo for monitoring their large Hadoop infrastructure which includes over 2500 servers running many different services like HDFS, YARN, HBase, Kafka, and Storm. OpenTSDB was chosen because it can handle the scale of metrics collected, store metrics for long periods of time with fine-grained resolution, and is easily extensible to add new metrics. It uses HBase for storage which is optimized for the time series data stored in OpenTSDB and can scale to meet Criteo's needs of storing billions of data points and handling high query loads.
BlackStor - World's fastest & most reliable Cloud Native Software Defined Sto...Michal Němec
BlackStor is a new cloud native software defined storage solution built using DRBD (Distributed Replicated Block Device) technology. It aims to provide high performance, reliability and scalability with predictable costs. BlackStor addresses limitations in existing open source solutions like Ceph by leveraging the proven DRBD technology. It includes features like policy-driven storage, multi-tenancy, deep metrics and analytics, backup capabilities, and support options. The goal is to deliver enterprise-grade storage capabilities with the simplicity of open source at a lower total cost of ownership.
We describe an application of CEP using a microservice-based streaming architecture. We use Drools business rule engine to apply rules in real time to an event stream from IoT traffic sensor data.
This document discusses Oracle's SPARC systems and their ability to modernize legacy Unix applications and provide a path to the cloud. It describes how SPARC systems offer a modern, cloud-ready infrastructure that can leverage existing investments while improving security, capacity, and flexibility. It provides examples of SPARC solutions that delivered benefits like reduced costs, increased throughput, and scalability for customers in various industries.
Dyn delivers exceptional Internet Performance. Enabling high quality services requires data centers around the globe. In order to manage services, customers need timely insight collected from all over the world. Dyn uses DataStax Enterprise (DSE) to deploy complex clusters across multiple datacenters to enable sub 50 ms query responses for hundreds of billions of data points. From granular DNS traffic data, to aggregated counts for a variety of report dimensions, DSE at Dyn has been up since 2013 and has shined through upgrades, data center migrations, DDoS attacks and hardware failures. In this webinar, Principal Engineers Tim Chadwick and Rick Bross cover the requirements which led them to choose DSE as their go-to Big Data solution, the path which led to SPARK, and the lessons that we’ve learned in the process.
July 2018 talk to SW Data Meetup by Rob Vesse, Software Engineer, Cray Inc, discussing open source technologies for data science on high performance systems (Spark, Hadoop, PyData ecosystem, containers, etc), focusing on some of the implementation and scaling challenges they face.
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...Databricks
As Apache Spark applications move to a containerized environment, there are many questions about how to best configure server systems in the container world. In this talk we will demonstrate a set of tools to better monitor performance and identify optimal configuration settings. We will demonstrate how Prometheus, a project that is now part of the Cloud Native Computing Foundation (CNCF: https://www.cncf.io/projects/), can be applied to monitor and archive system performance data in a containerized spark environment.
In our examples, we will gather spark metric output through Prometheus and present the data with Grafana dashboards. We will use our examples to demonstrate how performance can be enhanced through different tuned configuration settings. Our demo will show how to configure settings across the cluster as well as within each node.
DRBD + OpenStack (Openstack Live Prague 2016)Jaroslav Jacjuk
Michal Němec, Cooperation Director PROZETA
PROZETA společně s rakouským LINBITem přináší na trh nejrychlejší open-source Software Defined Storage (SDS) řešení postavené na DRBD s nemalým cílem - vytlačit ze své pozice v současnosti nejvyužívanější a často nevyhovující SDS pro OpenStack Ceph. Co stálo za rozhodnutím pustit se do tohoto ambiciózního projektu, který využívá technologii nasazovanou a rozvíjenou po více než 10 let, instalovanou ve více než 250 000 produkčních instancí po celém světě? Jaká úskalí bylo třeba překonat, jaké nástroje jsou přesně k dispozici pro produkční provoz? Jak si na tom nové SDS stojí výkonnostně a jaká je roadmapa do budoucna?
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...Docker, Inc.
The SDACK architecture stands for Spark, Docker, Akka, Cassandra, and Kafka. At TrendMicro, we adopted the SDACK architecture to implement a security event inspection platform for APT attack analysis. In this talk, we will introduce SDACK stack with Spark lambda architecture, Akka and Kafka for streaming data pipeline, Cassandra for time series data, and Docker for microservices. Specifically, we will show you how we Dockerize each SDACK component to facilitate the RD team of algorithms development, help the QA team test the product easily, and use the Docker as a Service strategy to ship our products to customers. Next, we will show you how we monitor each Docker container and adjust the resource usage based on monitoring metrics. And then, we will share our Docker security policy which ensures our products are safety before shipping to customers. After that, we'll show you how we develop an all-in-one Docker based data product and scale it out to multi-host Docker cluster to solve the big data problem. Finally, we will share some challenges we faced during the product development and some lesson learned.
Fast Cars, Big Data - How Streaming Can Help Formula 1Tugdual Grall
Modern cars produce data. Lots of data. And Formula 1 cars produce more than their share. I will present a working demonstration of how modern data streaming can be applied to the data acquisition and analysis problem posed by modern motorsports.
Instead of bringing multiple Formula 1 cars to the talk, I will show how we instrumented a high fidelity physics-based automotive simulator to produce realistic data from simulated cars running on the Spa-Francorchamps track. We move data from the cars, to the pits, to the engineers back at HQ.
The result is near real-time visualization and comparison of performance and a great exposition of how to move data using messaging systems like Kafka, and process data in real time with Apache Spark, then analyse data using SQL with Apache Drill.
Code available here: https://github.com/mapr-demos/racing-time-series
Moving your Oracle Databases to the Oracle CloudAlex Zaballa
This document discusses moving Oracle databases to Oracle Cloud. It begins with introductions and background on the presenter. It then provides an overview of various Oracle Database Cloud services and options for migrating Oracle databases to the cloud. Specific migration methods like Data Pump export/import, transportable tablespaces, and full transportable are described. Considerations for choosing a migration method based on database version and architecture are also covered.
This is a reupload of the talk I delivered at the Spark London Meetup group, November 2016. Original link to the event: https://www.meetup.com/Spark-London/events/235626954/
I share observations and best practices.
Elastic data services on Apache Mesos via Mesosphere’s DCOSharrythewiz
Adam Bordelon and Mohit Soni demonstrate how projects like Apache Myriad (incubating) can install Hadoop on Mesosphere DC/OS alongside other data center-scale applications, enabling efficient resource sharing and isolation across a variety of distributed applications while sharing the same cluster resources and hence breaking silos.
Top 10 present and future innovations in the NoSQL Cassandra ecosystem (2022)Cédrick Lunven
Are you new to Apache Cassandra® and wondering what all the excitement is about? Or a veteran Cassandra user interested in understanding what’s new in the project?
Attend our live webinar on October 18 to learn about the latest Cassandra release and why it represents a big step forward but also all the initiative and new projects rising in the ecosystem, DataStax Director of Developer Relations Cedrick Lunven will walk you through new features in version 4.1.
Get the inside scoop on how version 4.1 adds exciting new features for operators and improves the security posture, without compromising the stability achieved in Cassandra 4.0. Get some insights about projects actually in progress to make Cassandra more easy to use (Stargate) but also to deploy (K8ssandra).
You will learn:
System-wide Guardrails
Denylisting Partition Keys
Diagnostic events via CQL, not just JMX
CQLSH Auth support for LDAP, Kerberos and more
Lots of new, pluggable extension points
Also, celebrate our open source community with highlights from the 2022 Apache Cassandra World Party and a look ahead to Cassandra 5.0!
MySQL day Dublin - OCI & Application DevelopmentHenry J. Kröger
Slide deck from the MySQL day on the 23rd of October 2018 in the Oracle Dublin office. Presents Oracle's Cloud Infrastructure and Application Development Platform using Docker and Kubernetes.
Similar to Performance is not an Option - gRPC and Cassandra (20)
Takashi Kobayashi and Hironori Washizaki, "SWEBOK Guide and Future of SE Education," First International Symposium on the Future of Software Engineering (FUSE), June 3-6, 2024, Okinawa, Japan
Need for Speed: Removing speed bumps from your Symfony projects ⚡️Łukasz Chruściel
No one wants their application to drag like a car stuck in the slow lane! Yet it’s all too common to encounter bumpy, pothole-filled solutions that slow the speed of any application. Symfony apps are not an exception.
In this talk, I will take you for a spin around the performance racetrack. We’ll explore common pitfalls - those hidden potholes on your application that can cause unexpected slowdowns. Learn how to spot these performance bumps early, and more importantly, how to navigate around them to keep your application running at top speed.
We will focus in particular on tuning your engine at the application level, making the right adjustments to ensure that your system responds like a well-oiled, high-performance race car.
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Mobile App Development Company In Noida | Drona InfotechDrona Infotech
Looking for a reliable mobile app development company in Noida? Look no further than Drona Infotech. We specialize in creating customized apps for your business needs.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
Revolutionizing Visual Effects Mastering AI Face Swaps.pdfUndress Baby
The quest for the best AI face swap solution is marked by an amalgamation of technological prowess and artistic finesse, where cutting-edge algorithms seamlessly replace faces in images or videos with striking realism. Leveraging advanced deep learning techniques, the best AI face swap tools meticulously analyze facial features, lighting conditions, and expressions to execute flawless transformations, ensuring natural-looking results that blur the line between reality and illusion, captivating users with their ingenuity and sophistication.
Web:- https://undressbaby.com/
Hand Rolled Applicative User ValidationCode KataPhilip Schwarz
Could you use a simple piece of Scala validation code (granted, a very simplistic one too!) that you can rewrite, now and again, to refresh your basic understanding of Applicative operators <*>, <*, *>?
The goal is not to write perfect code showcasing validation, but rather, to provide a small, rough-and ready exercise to reinforce your muscle-memory.
Despite its grandiose-sounding title, this deck consists of just three slides showing the Scala 3 code to be rewritten whenever the details of the operators begin to fade away.
The code is my rough and ready translation of a Haskell user-validation program found in a book called Finding Success (and Failure) in Haskell - Fall in love with applicative functors.
DDS Security Version 1.2 was adopted in 2024. This revision strengthens support for long runnings systems adding new cryptographic algorithms, certificate revocation, and hardness against DoS attacks.
Transform Your Communication with Cloud-Based IVR SolutionsTheSMSPoint
Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony
Software Engineering, Software Consulting, Tech Lead, Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Transaction, Spring MVC, OpenShift Cloud Platform, Kafka, REST, SOAP, LLD & HLD.
E-commerce Development Services- Hornet DynamicsHornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
Neo4j - Product Vision and Knowledge Graphs - GraphSummit ParisNeo4j
Dr. Jesús Barrasa, Head of Solutions Architecture for EMEA, Neo4j
Découvrez les dernières innovations de Neo4j, et notamment les dernières intégrations cloud et les améliorations produits qui font de Neo4j un choix essentiel pour les développeurs qui créent des applications avec des données interconnectées et de l’IA générative.
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
What is Augmented Reality Image Trackingpavan998932
Augmented Reality (AR) Image Tracking is a technology that enables AR applications to recognize and track images in the real world, overlaying digital content onto them. This enhances the user's interaction with their environment by providing additional information and interactive elements directly tied to physical images.
Odoo ERP software
Odoo ERP software, a leading open-source software for Enterprise Resource Planning (ERP) and business management, has recently launched its latest version, Odoo 17 Community Edition. This update introduces a range of new features and enhancements designed to streamline business operations and support growth.
The Odoo Community serves as a cost-free edition within the Odoo suite of ERP systems. Tailored to accommodate the standard needs of business operations, it provides a robust platform suitable for organisations of different sizes and business sectors. Within the Odoo Community Edition, users can access a variety of essential features and services essential for managing day-to-day tasks efficiently.
This blog presents a detailed overview of the features available within the Odoo 17 Community edition, and the differences between Odoo 17 community and enterprise editions, aiming to equip you with the necessary information to make an informed decision about its suitability for your business.
Atelier - Innover avec l’IA Générative et les graphes de connaissancesNeo4j
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsPeter Muessig
The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.
Utilocate offers a comprehensive solution for locate ticket management by automating and streamlining the entire process. By integrating with Geospatial Information Systems (GIS), it provides accurate mapping and visualization of utility locations, enhancing decision-making and reducing the risk of errors. The system's advanced data analytics tools help identify trends, predict potential issues, and optimize resource allocation, making the locate ticket management process smarter and more efficient. Additionally, automated ticket management ensures consistency and reduces human error, while real-time notifications keep all relevant personnel informed and ready to respond promptly.
The system's ability to streamline workflows and automate ticket routing significantly reduces the time taken to process each ticket, making the process faster and more efficient. Mobile access allows field technicians to update ticket information on the go, ensuring that the latest information is always available and accelerating the locate process. Overall, Utilocate not only enhances the efficiency and accuracy of locate ticket management but also improves safety by minimizing the risk of utility damage through precise and timely locates.
1. Copyright 2016 Expero, Inc. All Rights Reserved 1
In today's development ecosystem building a service oriented architecture based on a micro services is
common practice. With the rise of Big Data and Internet of Things applications making these services highly
performant services is no longer an option. In order to accomplish the scalability and performance
requirements that customers expect we are required to start thinking differently about how we architect and
build these applications in order to meet those demands.
This session will demonstrate a method for creating a highly performant service based application using
Google’s GRPC and Apache Cassandra in .NET. We will show how you can combine gRPC to minimize
communication overhead while leveraging Cassandra to optimize storage of time series data. We will explore
these concepts by creating an Internet of Things (IoT) application to demonstrate how you can effectively
meet the performance and scalability challenges posed by these new breeds of applications.
Abstract
2. Copyright 2016 Expero, Inc. All Rights Reserved
Performance is not an
Option
Building High performance Web Services
with gRPC and Cassandra
June 9th, 2016
#build4prfrmnc
3. Copyright 2016 Expero, Inc. All Rights Reserved 3
● What is gRPC
● What is Cassandra
● How to build a simple gRPC Microservice
● How to persist time series data in Cassandra
● Why you might want to use gRPC/Cassandra instead of a
more traditional REST/RDBMS for a Time-Series IoT
application
What I hope you take away
4. Copyright 2016 Expero, Inc. All Rights Reserved 4
Dave Bechberger
Senior Architect
dave@experoinc.com
@bechbd
https://www.linkedin.com/in/davebechberger
About me
5. Copyright 2016 Expero, Inc. All Rights Reserved 5
Expero - Bringing challenging product ideas to reality
Architecture &
Development
Product Strategy
User Experience
Domain
Expert
6. Copyright 2016 Expero, Inc. All Rights Reserved 6
Expero - Select Clients
6
Austin(HQ) • Houston • New York City • Founded 2003
7. Copyright 2016 Expero, Inc. All Rights ReservedCopyright 2016 Expero, Inc. All Rights Reserved
What is gRPC?
7
8. Copyright 2016 Expero, Inc. All Rights Reserved
● gRPC is a general purpose RPC
framework
● Built on standards
● Free and Open Source
● Built for distributed systems
8
What is gRPC?
9. Copyright 2016 Expero, Inc. All Rights Reserved
● Allows client to call methods on
the server as if they were local
● Built for low latency highly
scalable microservices
● Payload agnostic
● Bi-Directional Streaming
● Pluggable and Extensible
9
gRPC Architecture
13. Copyright 2016 Expero, Inc. All Rights ReservedCopyright 2016 Expero, Inc. All Rights Reserved
What is Cassandra?
13
14. Copyright 2016 Expero, Inc. All Rights Reserved
What is Cassandra?
● Distributed Datastore
● Open Source Apache Project
● No Single Point of Failure
● Scalable
14
15. Copyright 2016 Expero, Inc. All Rights Reserved
CAP Theorem - Pick 2
● Consistency - all nodes see the same data
at the same time
● Availability - every requests receives a
response
● Partition Tolerant - the system continues
to operate even during network failures
15
16. Copyright 2016 Expero, Inc. All Rights Reserved
ACID vs. BASE
RDBMS World
● Atomic - transactions are “all or nothing”
● Consistency - On completion all data is
the same
● Isolated - transactions do not interfere
with one another
● Durable - results of a transaction are
permanent
16
NoSQL World
● Base Availability - The datastore works
most of the time
● Soft State - Stores are not write
consistent, data can differ between
replicas
● Eventually Consistent - Stores become
consistent over time
18. Copyright 2016 Expero, Inc. All Rights Reserved
Hash Ring Architecture
18
● All nodes own a portion of the token
ring
● All nodes know which token ranges
belong to which nodes
● Partitioner generates a token from the
Partition Key
● Tokens determine where data is
located on the ring
19. Copyright 2016 Expero, Inc. All Rights Reserved
How Tokens Work
19
Partitioner
Token:12
Client Driver
PK: Expero
Data Written
20. Copyright 2016 Expero, Inc. All Rights Reserved
Data Replication
● Data replication is automatic
● Number of replicas is called the Replication Factor or RF
● Data is replicated between Data Centers
● Hinted Handoff
21. Copyright 2016 Expero, Inc. All Rights Reserved
Data Replication in Action
Client
Write A
Data Written
Replica Written
Replica Written
Coordinator
Driver
Partitioner
Token:12
PK: Expero
22. Copyright 2016 Expero, Inc. All Rights Reserved
What does it mean to be Eventually Consistent?
22
● Data will “eventually” match on all replicas, usually in terms of
milliseconds
● Consistency Level or CL (11 for writes, 10 for reads)
● Tuning consistency affects performance and availability
● CL can be tuned for R/W performance on a per query basis using CQL
23. Copyright 2016 Expero, Inc. All Rights Reserved
Why use Cassandra over your RDBMS?
● Performance
● Linearly Scalable
● Natively built as a distributed datastore
● Always-On Architecture
23
24. Copyright 2016 Expero, Inc. All Rights Reserved
What is DataStax vs. Apache Cassandra?
● Certified Cassandra – Delivers highest quality
Cassandra software for confidence and peace of
mind for production environments
● Enterprise Security – Full protection for sensitive
data
● Automatic Management Services – Automates
key maintenance functions to keep the database
running smoothly
● OpsCenter – Advanced management and
monitoring functionality for production applications
● Expert Support – Answers and assistance from the
Cassandra experts for all production needs
24
25. Copyright 2016 Expero, Inc. All Rights ReservedCopyright 2016 Expero, Inc. All Rights Reserved
The Problem - Engine Monitoring
25
26. Copyright 2016 Expero, Inc. All Rights Reserved
● Truck Engine Monitoring
Software
● Currently ~ 1000 trucks taking
readings every 10 seconds
● WebAPI REST on a SQL Server
2014 Database
26
Setup
27. Copyright 2016 Expero, Inc. All Rights Reserved
● You were recently landed a huge
new client Expero Trucking Inc.
● Sensor readings now 1/second
and add geolocation (lat/long)
data
● Adding 10,000 trucks.
● Minimize costs and zero
downtime
27
The Requirements
28. Copyright 2016 Expero, Inc. All Rights Reserved
● 100 measurements/second to
22,000 measurements/second
● Data load from ~35 MB/day to
~2.2 GB/day
● Your architecture needs to
change
28
The Problem
29. Copyright 2016 Expero, Inc. All Rights ReservedCopyright 2016 Expero, Inc. All Rights Reserved
The Solution - gRPC and Cassandra
29
Code Available Here https://github.com/experoinc/NDC-Oslo-2016/tree/master/NDC.Oslo
30. Copyright 2016 Expero, Inc. All Rights Reserved
● Change SQL Server Database to a Cassandra Cluster
● Replace REST based services with gRPC services
30
Proposed Solution
32. Copyright 2016 Expero, Inc. All Rights Reserved 32
Generating Client/Server Stubs
Model Definition Service Definition
33. Copyright 2016 Expero, Inc. All Rights Reserved 33
Creating Cassandra KeySpace and Table
34. Copyright 2016 Expero, Inc. All Rights Reserved 34
Connecting to Apache Cassandra using DataStax Driver
DataStax Open Source C# Driver - https://github.com/datastax/csharp-driver
37. Copyright 2016 Expero, Inc. All Rights Reserved 37
Time to See Some Running Code
Average Ping Time
Running in Oregon
38. Copyright 2016 Expero, Inc. All Rights ReservedCopyright 2016 Expero, Inc. All Rights Reserved
Tradeoffs of gRPC and Cassandra
38
39. Copyright 2016 Expero, Inc. All Rights Reserved 39
● Not for Browsers
● Chunk “Big” (>1MB ) Data
● No Nullable Data Types
● Not Production Yet
Tradeoffs of using gRPC
40. Copyright 2016 Expero, Inc. All Rights Reserved 40
● No joins between tables
● No Ad-Hoc queries
● Minimal Aggregations
● Complexity
● Cassandra is Not Relational
Tradeoffs of using Cassandra
41. Copyright 2016 Expero, Inc. All Rights Reserved 41
● gRPC
○ http://www.grpc.io/
○ https://developers.google.com/protocol-buffers/docs/proto3
● Cassandra
○ http://cassandra.apache.org/
○ http://www.planetcassandra.org/
○ https://academy.datastax.com/
○ http://www.datastax.com/
Learning More
42. Copyright 2016 Expero, Inc. All Rights ReservedCopyright 2016 Expero, Inc. All Rights Reserved
Thank you, any Questions?
42