OSDC 2014: Jordan Sissel - Find Happiness in your Logs
•
1 like•672 views
Got logs? With so much technology powering your business, you need tools to help you identify problems and analyze past behavior. Apache 2.0-licensed Elasticsearch ELK stack is here to help you process, store, and visualize any kind of logging data, in real time, from any source imaginable! Log management seems so boring. Log rotation, retention policy, grep, yuck! What are your servers are doing? Did last night's upgrade break anything? How your users are interacting with your products? Why did the site go down last weekend? Get ready to turn your log pains into awesome visual insights and more! BAM! Elasticsearch ELK! ELK stands for Elasticsearch, Logstash, and Kibana. Each of these three are lovely, open source projects that, together, give you and your business log management superpowers. This talk will primarily be done in three parts: open source and community, technology, and use cases. * The first part will introduce each project and its success as open source software, most notably through supportive and open communities. * The second part will discuss the each project and the problems solved. * The third (and most exciting!) part will highlight a variety of use cases and problem that real humans are using Elasticsearch ELK to solve. Live demos of some use cases will be provided. Attendees will leave the presentation totally full of excitement about this toolset and bursting with fresh ideas about how to tackle their sour logging problems.
Report
Share
Report
Share
Download to read offline
Recommended
Distributed tracing - get a grasp on your production
Slides from my presentation on distributed tracing, explaining what is latency and why it matters. We took a look at openzipkin and its concepts like how the core annotations works, what are tags/logs, etc. Followed by a demo application created using golang and java (spring boot , spring cloud sleuth zipkin) . You can find source code here
https://github.com/nklmish/go-distributed-tracing-demo
https://github.com/nklmish/java-distributed-tracing-demo
24 Weeks at AdroitLogic - My Internship Experience
Explains about the things I have learned during my internship period at AdroitLogic Lanka (Pvt.) Ltd. Created by Yasas Gunarathne.
Jaeger and OpenTracing Cloud Native Computing (CNCF) meetup Zurich
The document summarizes Jaeger and OpenTracing. It discusses distributed tracing concepts and how Jaeger is a distributed tracing system that uses the OpenTracing API. Jaeger uses technologies like Go, Cassandra, and React and can trace services in architectures like Istio and microservices. It also demonstrates traces using Jaeger's UI.
Monitoring to the Nth tier: The state of distributed tracing in 2016
This document discusses distributed tracing and monitoring of distributed systems. It provides examples of common distributed architectures and outlines challenges with instrumentation, trace ID propagation, and extracting valuable monitoring information. Distributed tracing allows understanding request flows, latencies, errors and more across tiers. While initially for performance, tracing provides a rich data set for analyzing individual components and custom metrics. Future directions include context propagation for things beyond performance like authentication and flow control.
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing
Slides from my talk & demo at Go NYC Meeetup 19-Jan-2017.
We present Jaeger, Uber’s open source distributed tracing system, featuring Go backend, React based UI, and OpenTracing API support. We show examples of instrumenting application code for tracing and using distributed context propagation to attribute backend resource usage to top level consumers.
Reactive Programming with RxJS
This document provides an overview of reactive programming with RxJS. It defines reactive programming as programming with asynchronous data streams, where anything can be a stream like variables, user inputs, and data structures. It introduces RxJS as a library that implements reactive programming in JavaScript and other languages. It describes the key components in RxJS - Observables that emit data and Observers that receive data. Finally, it provides examples of Observable operators like map, filter, concat, and buffer that allow transforming and combining streams.
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors.
In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others.
I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
Log management with Graylog2 - FrOSCon 2012
The document discusses log management with Graylog2. It provides an overview of Graylog2 as a free and open source log management system. It also discusses different levels of log management maturity and how Graylog2 allows centralized collection, analysis, and visualization of logs through its server and web interface components. Key components like GELF and Logstash that facilitate log collection and processing are also summarized.
Recommended
Distributed tracing - get a grasp on your production
Slides from my presentation on distributed tracing, explaining what is latency and why it matters. We took a look at openzipkin and its concepts like how the core annotations works, what are tags/logs, etc. Followed by a demo application created using golang and java (spring boot , spring cloud sleuth zipkin) . You can find source code here
https://github.com/nklmish/go-distributed-tracing-demo
https://github.com/nklmish/java-distributed-tracing-demo
24 Weeks at AdroitLogic - My Internship Experience
Explains about the things I have learned during my internship period at AdroitLogic Lanka (Pvt.) Ltd. Created by Yasas Gunarathne.
Jaeger and OpenTracing Cloud Native Computing (CNCF) meetup Zurich
The document summarizes Jaeger and OpenTracing. It discusses distributed tracing concepts and how Jaeger is a distributed tracing system that uses the OpenTracing API. Jaeger uses technologies like Go, Cassandra, and React and can trace services in architectures like Istio and microservices. It also demonstrates traces using Jaeger's UI.
Monitoring to the Nth tier: The state of distributed tracing in 2016
This document discusses distributed tracing and monitoring of distributed systems. It provides examples of common distributed architectures and outlines challenges with instrumentation, trace ID propagation, and extracting valuable monitoring information. Distributed tracing allows understanding request flows, latencies, errors and more across tiers. While initially for performance, tracing provides a rich data set for analyzing individual components and custom metrics. Future directions include context propagation for things beyond performance like authentication and flow control.
Tracing 2000+ polyglot microservices at Uber with Jaeger and OpenTracing
Slides from my talk & demo at Go NYC Meeetup 19-Jan-2017.
We present Jaeger, Uber’s open source distributed tracing system, featuring Go backend, React based UI, and OpenTracing API support. We show examples of instrumenting application code for tracing and using distributed context propagation to attribute backend resource usage to top level consumers.
Reactive Programming with RxJS
This document provides an overview of reactive programming with RxJS. It defines reactive programming as programming with asynchronous data streams, where anything can be a stream like variables, user inputs, and data structures. It introduces RxJS as a library that implements reactive programming in JavaScript and other languages. It describes the key components in RxJS - Observables that emit data and Observers that receive data. Finally, it provides examples of Observable operators like map, filter, concat, and buffer that allow transforming and combining streams.
THE STATE OF OPENTELEMETRY, DOTAN HOROVITS, Logz.io
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors.
In this talk I will present OpenTelemetry, an ambitious open source project with the promise of a unified framework for collecting observability data. With OpenTelemetry you could instrument your application in a vendor-agnostic way, and then analyze the telemetry data in your backend tool of choice, whether Prometheus, Jaeger, Zipkin, or others.
I will cover the current state of the various projects of OpenTelemetry (across programming languages, exporters, receivers, protocols), some of which not even GA yet, and provide useful guidance on how to get started with it.
Log management with Graylog2 - FrOSCon 2012
The document discusses log management with Graylog2. It provides an overview of Graylog2 as a free and open source log management system. It also discusses different levels of log management maturity and how Graylog2 allows centralized collection, analysis, and visualization of logs through its server and web interface components. Key components like GELF and Logstash that facilitate log collection and processing are also summarized.
Quick test professional (qtp) online training
This document provides an overview of topics that will be covered in an online Quick Test Professional (QTP) training course with VB scripting. The course will introduce QTP and differences from other testing tools, cover objects and repositories, recording modes, test results, and VB scripting basics. Additional topics will include working with the registry, files, timers, checkpoints, output values, actions, environment variables, descriptive programming, and exception handling. Upon completion, students will be able to automate tests and implement checkpoints and output values through VB scripting in QTP. The training aims to provide the skills needed for manual and automation testing.
Altitude NY 2018: Observability workshop
This workshop is a hands-on deep-dive into the major pillars of observability. Peter Bourgon will explore the defining characteristics of metrics, tracing, and logging, and use that knowledge to instrument an example service, including interfacing with the Fastly stats and logging pipelines. Attendees will leave with specific, actionable intelligence on how to make their own systems more observable and easier to operate at scale.
Juraci Paixão Kröhling - All you need to know about OpenTelemetry
OpenTelemetry is one of the newest projects in the realm of Observability at the CNCF and is already the second most active project there. In this session, Juraci Paixão Kröhling will talk about the different subprojects and how to get started using them. Even if you heard about OpenTelemetry before, you'll leave this session with a better understanding of what this is all about, the several faces of OpenTelemetry, and what you can do to make your projects more observable.
Opentracing 101
In this slide, we go through the Google Dapper, OpenTracing, Jaeger to OpenTelemetry. By reading and studying the history of Dapper, we could lean the experience and design theory of a large-scale distributed tracing system and then know how it affects other solutions, like OpenTracing and Jaeger.
We also discuss the difference between the OpenTracing and Jaeger and also demonstrate how Jaeger works and looks like.
After, we talked about the future of OpenTracing, the new organization called OpenTelemetry, what's its goal and how to do that.
Tracer
Tracing in Go allows gathering fine-grained information about an HTTP request's lifecycle, including connection creation/reuse, DNS lookup, and reading/writing to the wire. This helps diagnose latency issues by identifying when critical goroutines are blocked for long periods. Tracing provides a more detailed breakdown than monitoring tools to find "time guzzlers". The runtime/trace package captures execution events while tools like httpstat and tracetools process output into profiles. Tracing has overhead and generates large logs but helps optimize performance.
Documenting an API written in Django Rest Framework
A presentation given at the San Francisco Django meetup on DjaoDjin experience setting up a pipeline to generate API docs.
IoT Research Project
Research project for Digital Systems that helped us understand better how to extend the communication aspects of micro-controllers and their connected interfaces to develop an IoT systems that can communicate with an online service without human intervention.
OpenTelemetry For Operators
This document provides an overview of OpenTelemetry for operators. It discusses some of the limitations of current observability platforms and how OpenTelemetry addresses these issues. It introduces the OpenTelemetry project which combines distributed tracing, metrics, and logging APIs. It describes the OpenTelemetry Collector which receives, processes, and exports telemetry data. It provides examples of Collector configuration and running it in production. It also discusses some innovations in the observability space from vendors like Dynatrace, New Relic, Splunk SignalFX, and others.
Protocol testing course content
The protocol testing course covers networking fundamentals like the OSI model and TCP/IP layers, networking devices, data flow, and IP addressing. It examines application layer protocols like HTTP, FTP, and DNS, transport protocols TCP and UDP, and layer 2 technologies like switching, VLANs, and STP. Layer 3 routing protocols such as RIP, OSPF, BGP, and EIGRP are also evaluated. The course teaches protocol testing methods, the testing lifecycle, manual and automated testing, and bug tracking. Automation techniques using TCL and Expect are presented for developing test cases on Cisco routers.
Log Aggregation
Log aggregation involves collecting logs from multiple sources, indexing the logs, and visualizing and analyzing the logs. Popular tools for log aggregation include Logstash, which can input logs from various sources, apply filters, and output to destinations like Elasticsearch or Solr for indexing. The indexed logs can then be visualized and analyzed using tools like Kibana or Banana to monitor systems, detect issues, and gain insights from log data.
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Warren Strange, Principal Systems Engineer, ForgeRock, presents a Breakout Session on the ELK Stack at the 2014 IRM Summit in Phoenix, Arizona.
Jdd2014: High performance logging - Peter Lawrey
In many applications, there is a tension between how much you can log without slowing down your application, and how much information you would like to have.
Chronicle provided a number of solutions which allow you to record millions of events per second, with micro-second latencies in a persisted way without contributing to your garbage.
How does this simplify the design, help you increase the determinism and vertical scalability of your application?
Taras Girnyk "Debugging and Profiling distributed applications using Opentrac...
Whether you're on a quest to find buggy piece of code or search for a bottleneck in distributed application,- usual tools like XDebug, XHProf (or Tideways), strace and gdb won't be nearly as good as they are in monolith. Despite that there is something you can do to ease your journey. We will talk about some common solutions to this problem when working with microservice-oriented applications and then go on with Opentracing. I'll try to walk you through the idea and instruments with real use cases to make you profiling sessions easy and painless.
Real time cloud native open source streaming of any data to apache solr
Real time cloud native open source streaming of any data to apache solr
Utilizing Apache Pulsar and Apache NiFi we can parse any document in real-time at scale. We receive a lot of documents via cloud storage, email, social channels and internal document stores. We want to make all the content and metadata to Apache Solr for categorization, full text search, optimization and combination with other datastores. We will not only stream documents, but all REST feeds, logs and IoT data. Once data is produced to Pulsar topics it can instantly be ingested to Solr through Pulsar Solr Sink.
Utilizing a number of open source tools, we have created a real-time scalable any document parsing data flow. We use Apache Tika for Document Processing with real-time language detection, natural language processing with Apache OpenNLP, Sentiment Analysis with Stanford CoreNLP, Spacy and TextBlob. We will walk everyone through creating an open source flow of documents utilizing Apache NiFi as our integration engine. We can convert PDF, Excel and Word to HTML and/or text. We can also extract the text to apply sentiment analysis and NLP categorization to generate additional metadata about our documents. We also will extract and parse images that if they contain text we can extract with TensorFlow and Tesseract.
Implementing OpenAPI and GraphQL services with gRPC
Behind every API there's code. REST and GraphQL are powerful interface abstractions but are not so great for writing code (we’re still looking for the programming language where every command is a GET, POST, PUT, or DELETE). When programmers work, they are usually making function calls, and an RPC framework like gRPC allows those functions to be written in a mixture of languages and distributed among many servers. This means that gRPC can be a great way to implement REST and GraphQL APIs at scale. We’ll share open source projects from Google that can be used to implement OpenAPI and GraphQL services with gRPC and give you hands-on experience with both.
Presented at the 2019 API Specifications Conference.
https://asc2019.sched.com/event/T6u9/workshop-implementing-openapi-and-graphql-services-with-grpc-tim-burks-google
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Spark
DevNexus 2022 Atlanta
https://devnexus.com/presentations/7150/
This talk is a quick overview of the How, What and WHY of Apache Pulsar, Apache Flink and Apache NiFi. I will show you how to design event-driven applications that scale the cloud native way.
This talk was done live in person at DevNexus across from the booth in room 311
Tim Spann
Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar, Apache Flink, Flink SQL, Apache NiFi, MiniFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
Tuning Elasticsearch Indexing Pipeline for Logs
This document summarizes techniques for optimizing Logstash and Rsyslog for high volume log ingestion into Elasticsearch. It discusses using Logstash and Rsyslog to ingest logs via TCP and JSON parsing, applying filters like grok and mutate, and outputting to Elasticsearch. It also covers Elasticsearch tuning including refresh rate, doc values, indexing performance, and using time-based indices on hot and cold nodes. Benchmark results show Logstash and Rsyslog can handle thousands of events per second with appropriate configuration.
Modern Data Architectures for Real Time Analytics & Engagement
The AWS Workshop Series Online is a series of live webinars designed for IT professionals who are looking to leverage the AWS Cloud to build and transform their business, are new to the AWS Cloud or looking to further expand their skills and expertise. In this series, we will cover: 'Modern Data Architectures for Real-time Analytics and Engagement'.
Flink in action
Artsem Semianenko (Adform) - "Flink in action или как приручить белочку"
Slides for presentation: https://www.youtube.com/watch?v=YSI5_RFlcPE
Source: https://github.com/art4ul/flink-demo
Effective microservices development with ballerina
This is a presentation on the Ballerina programming language and how it can help in implementing a microservices architecture.
Log Management Systems
This document compares message and metric management solutions like Fluentd and Logstash. It discusses how these solutions can collect, store, visualize, and alert on log and metric data from heterogeneous environments. While commercial solutions like Splunk are very expensive, open source solutions like Fluentd, Logstash, Elasticsearch, and Kibana provide similar functionality through various "bricks" or components at no cost. The document analyzes key differences between Fluentd and Logstash, such as their configuration, buffering approaches, high availability features, and plugin ecosystems.
"How about no grep and zabbix?". ELK based alerts and metrics.
This document provides an overview of the ELK (Elasticsearch, Logstash, Kibana) stack for collecting, analyzing, and visualizing log and metrics data. It describes the components of the ELK stack and how they work together, including how Logstash can be used to transform raw log data into structured JSON documents for indexing in Elasticsearch. The document also discusses how Kibana can be used to visualize and explore the data in Elasticsearch, and how the ELK stack can be used for advanced capabilities like custom metrics, alerts, and monitoring through tools like Elastalert and Kibana dashboards.
More Related Content
What's hot
Quick test professional (qtp) online training
This document provides an overview of topics that will be covered in an online Quick Test Professional (QTP) training course with VB scripting. The course will introduce QTP and differences from other testing tools, cover objects and repositories, recording modes, test results, and VB scripting basics. Additional topics will include working with the registry, files, timers, checkpoints, output values, actions, environment variables, descriptive programming, and exception handling. Upon completion, students will be able to automate tests and implement checkpoints and output values through VB scripting in QTP. The training aims to provide the skills needed for manual and automation testing.
Altitude NY 2018: Observability workshop
This workshop is a hands-on deep-dive into the major pillars of observability. Peter Bourgon will explore the defining characteristics of metrics, tracing, and logging, and use that knowledge to instrument an example service, including interfacing with the Fastly stats and logging pipelines. Attendees will leave with specific, actionable intelligence on how to make their own systems more observable and easier to operate at scale.
Juraci Paixão Kröhling - All you need to know about OpenTelemetry
OpenTelemetry is one of the newest projects in the realm of Observability at the CNCF and is already the second most active project there. In this session, Juraci Paixão Kröhling will talk about the different subprojects and how to get started using them. Even if you heard about OpenTelemetry before, you'll leave this session with a better understanding of what this is all about, the several faces of OpenTelemetry, and what you can do to make your projects more observable.
Opentracing 101
In this slide, we go through the Google Dapper, OpenTracing, Jaeger to OpenTelemetry. By reading and studying the history of Dapper, we could lean the experience and design theory of a large-scale distributed tracing system and then know how it affects other solutions, like OpenTracing and Jaeger.
We also discuss the difference between the OpenTracing and Jaeger and also demonstrate how Jaeger works and looks like.
After, we talked about the future of OpenTracing, the new organization called OpenTelemetry, what's its goal and how to do that.
Tracer
Tracing in Go allows gathering fine-grained information about an HTTP request's lifecycle, including connection creation/reuse, DNS lookup, and reading/writing to the wire. This helps diagnose latency issues by identifying when critical goroutines are blocked for long periods. Tracing provides a more detailed breakdown than monitoring tools to find "time guzzlers". The runtime/trace package captures execution events while tools like httpstat and tracetools process output into profiles. Tracing has overhead and generates large logs but helps optimize performance.
Documenting an API written in Django Rest Framework
A presentation given at the San Francisco Django meetup on DjaoDjin experience setting up a pipeline to generate API docs.
IoT Research Project
Research project for Digital Systems that helped us understand better how to extend the communication aspects of micro-controllers and their connected interfaces to develop an IoT systems that can communicate with an online service without human intervention.
OpenTelemetry For Operators
This document provides an overview of OpenTelemetry for operators. It discusses some of the limitations of current observability platforms and how OpenTelemetry addresses these issues. It introduces the OpenTelemetry project which combines distributed tracing, metrics, and logging APIs. It describes the OpenTelemetry Collector which receives, processes, and exports telemetry data. It provides examples of Collector configuration and running it in production. It also discusses some innovations in the observability space from vendors like Dynatrace, New Relic, Splunk SignalFX, and others.
What's hot (8)
Juraci Paixão Kröhling - All you need to know about OpenTelemetry
Juraci Paixão Kröhling - All you need to know about OpenTelemetry
Documenting an API written in Django Rest Framework
Documenting an API written in Django Rest Framework
Similar to OSDC 2014: Jordan Sissel - Find Happiness in your Logs
Protocol testing course content
The protocol testing course covers networking fundamentals like the OSI model and TCP/IP layers, networking devices, data flow, and IP addressing. It examines application layer protocols like HTTP, FTP, and DNS, transport protocols TCP and UDP, and layer 2 technologies like switching, VLANs, and STP. Layer 3 routing protocols such as RIP, OSPF, BGP, and EIGRP are also evaluated. The course teaches protocol testing methods, the testing lifecycle, manual and automated testing, and bug tracking. Automation techniques using TCL and Expect are presented for developing test cases on Cisco routers.
Log Aggregation
Log aggregation involves collecting logs from multiple sources, indexing the logs, and visualizing and analyzing the logs. Popular tools for log aggregation include Logstash, which can input logs from various sources, apply filters, and output to destinations like Elasticsearch or Solr for indexing. The indexed logs can then be visualized and analyzed using tools like Kibana or Banana to monitor systems, detect issues, and gain insights from log data.
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Warren Strange, Principal Systems Engineer, ForgeRock, presents a Breakout Session on the ELK Stack at the 2014 IRM Summit in Phoenix, Arizona.
Jdd2014: High performance logging - Peter Lawrey
In many applications, there is a tension between how much you can log without slowing down your application, and how much information you would like to have.
Chronicle provided a number of solutions which allow you to record millions of events per second, with micro-second latencies in a persisted way without contributing to your garbage.
How does this simplify the design, help you increase the determinism and vertical scalability of your application?
Taras Girnyk "Debugging and Profiling distributed applications using Opentrac...
Whether you're on a quest to find buggy piece of code or search for a bottleneck in distributed application,- usual tools like XDebug, XHProf (or Tideways), strace and gdb won't be nearly as good as they are in monolith. Despite that there is something you can do to ease your journey. We will talk about some common solutions to this problem when working with microservice-oriented applications and then go on with Opentracing. I'll try to walk you through the idea and instruments with real use cases to make you profiling sessions easy and painless.
Real time cloud native open source streaming of any data to apache solr
Real time cloud native open source streaming of any data to apache solr
Utilizing Apache Pulsar and Apache NiFi we can parse any document in real-time at scale. We receive a lot of documents via cloud storage, email, social channels and internal document stores. We want to make all the content and metadata to Apache Solr for categorization, full text search, optimization and combination with other datastores. We will not only stream documents, but all REST feeds, logs and IoT data. Once data is produced to Pulsar topics it can instantly be ingested to Solr through Pulsar Solr Sink.
Utilizing a number of open source tools, we have created a real-time scalable any document parsing data flow. We use Apache Tika for Document Processing with real-time language detection, natural language processing with Apache OpenNLP, Sentiment Analysis with Stanford CoreNLP, Spacy and TextBlob. We will walk everyone through creating an open source flow of documents utilizing Apache NiFi as our integration engine. We can convert PDF, Excel and Word to HTML and/or text. We can also extract the text to apply sentiment analysis and NLP categorization to generate additional metadata about our documents. We also will extract and parse images that if they contain text we can extract with TensorFlow and Tesseract.
Implementing OpenAPI and GraphQL services with gRPC
Behind every API there's code. REST and GraphQL are powerful interface abstractions but are not so great for writing code (we’re still looking for the programming language where every command is a GET, POST, PUT, or DELETE). When programmers work, they are usually making function calls, and an RPC framework like gRPC allows those functions to be written in a mixture of languages and distributed among many servers. This means that gRPC can be a great way to implement REST and GraphQL APIs at scale. We’ll share open source projects from Google that can be used to implement OpenAPI and GraphQL services with gRPC and give you hands-on experience with both.
Presented at the 2019 API Specifications Conference.
https://asc2019.sched.com/event/T6u9/workshop-implementing-openapi-and-graphql-services-with-grpc-tim-burks-google
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Spark
DevNexus 2022 Atlanta
https://devnexus.com/presentations/7150/
This talk is a quick overview of the How, What and WHY of Apache Pulsar, Apache Flink and Apache NiFi. I will show you how to design event-driven applications that scale the cloud native way.
This talk was done live in person at DevNexus across from the booth in room 311
Tim Spann
Tim Spann is a Developer Advocate for StreamNative. He works with StreamNative Cloud, Apache Pulsar, Apache Flink, Flink SQL, Apache NiFi, MiniFi, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal DataFlow Field Engineer at Cloudera, a Senior Solutions Architect at AirisData, a Senior Field Engineer at Pivotal and a Team Leader at HPE. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
Tuning Elasticsearch Indexing Pipeline for Logs
This document summarizes techniques for optimizing Logstash and Rsyslog for high volume log ingestion into Elasticsearch. It discusses using Logstash and Rsyslog to ingest logs via TCP and JSON parsing, applying filters like grok and mutate, and outputting to Elasticsearch. It also covers Elasticsearch tuning including refresh rate, doc values, indexing performance, and using time-based indices on hot and cold nodes. Benchmark results show Logstash and Rsyslog can handle thousands of events per second with appropriate configuration.
Modern Data Architectures for Real Time Analytics & Engagement
The AWS Workshop Series Online is a series of live webinars designed for IT professionals who are looking to leverage the AWS Cloud to build and transform their business, are new to the AWS Cloud or looking to further expand their skills and expertise. In this series, we will cover: 'Modern Data Architectures for Real-time Analytics and Engagement'.
Flink in action
Artsem Semianenko (Adform) - "Flink in action или как приручить белочку"
Slides for presentation: https://www.youtube.com/watch?v=YSI5_RFlcPE
Source: https://github.com/art4ul/flink-demo
Effective microservices development with ballerina
This is a presentation on the Ballerina programming language and how it can help in implementing a microservices architecture.
Log Management Systems
This document compares message and metric management solutions like Fluentd and Logstash. It discusses how these solutions can collect, store, visualize, and alert on log and metric data from heterogeneous environments. While commercial solutions like Splunk are very expensive, open source solutions like Fluentd, Logstash, Elasticsearch, and Kibana provide similar functionality through various "bricks" or components at no cost. The document analyzes key differences between Fluentd and Logstash, such as their configuration, buffering approaches, high availability features, and plugin ecosystems.
"How about no grep and zabbix?". ELK based alerts and metrics.
This document provides an overview of the ELK (Elasticsearch, Logstash, Kibana) stack for collecting, analyzing, and visualizing log and metrics data. It describes the components of the ELK stack and how they work together, including how Logstash can be used to transform raw log data into structured JSON documents for indexing in Elasticsearch. The document also discusses how Kibana can be used to visualize and explore the data in Elasticsearch, and how the ELK stack can be used for advanced capabilities like custom metrics, alerts, and monitoring through tools like Elastalert and Kibana dashboards.
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
by Timothy Spann
Developer Advocate
StreamNative
datainmotion.dev
Iot platform supporting million requests per second
This document discusses architectural patterns for IoT systems. It compares different communication protocols for sensors and edge servers, as well as different databases and monitoring solutions. The key points are:
1. It evaluates protocols like CoAP, AMQP, HTTP, and MQTT for sensor to edge communication based on factors like number of sensors supported, throughput, bandwidth usage, and support for battery-powered devices.
2. It analyzes different database options for edge servers based on requirements like time-to-live, read/write patterns, SQL support, and number of writers. Solutions discussed include RocksDB, Cassandra, VoltDB, and OrientDB.
3. Finally, it discusses centralized monitoring architectures using
Monitoring&Logging - Stanislav Kolenkin
Microservices add complexity to monitoring that was not present with monolithic architectures. While microservices provide benefits, they also introduce significant monitoring challenges around communication between services. Prometheus has emerged as a powerful open source solution for monitoring microservices as it was designed to address issues of scale and flexibility that monitoring microservices requires.
A vision for ejabberd - ejabberd SF Meetup
This is Mickaël Rémond presentation from ejabberd San Francisco Bay Area Meetup in march 2015. He presents latest achievements and progress made on ejabberd during the last few months. He then share his vision about the next steps to bring ejabberd XMPP platform to the next level.
elk_stack_alexander_szalonnas
This document provides an overview of the ELK stack, including Logstash for collecting and parsing logs, Elasticsearch for indexing logs, and Kibana for visualizing logs. It discusses using the open source ELK stack as an alternative to Splunk and provides instructions for getting started with a basic ELK implementation.
Kiwipycon command line
Linaro aims to improve Linux support for ARM processors. They created LAVA (Linaro Automated Validation) to automate testing of new kernels on ARM hardware. LAVA needs a way to trigger test runs when kernel builds finish. Linaro implemented an XML-RPC API with HTTPS and token-based authentication to securely allow remote triggering of test runs. They open sourced the server and client code to make it easy for others to add authenticated XML-RPC to projects.
Similar to OSDC 2014: Jordan Sissel - Find Happiness in your Logs (20)
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Customer Intelligence: Using the ELK Stack to Analyze ForgeRock OpenAM Audit ...
Taras Girnyk "Debugging and Profiling distributed applications using Opentrac...
Taras Girnyk "Debugging and Profiling distributed applications using Opentrac...
Real time cloud native open source streaming of any data to apache solr
Real time cloud native open source streaming of any data to apache solr
Implementing OpenAPI and GraphQL services with gRPC
Implementing OpenAPI and GraphQL services with gRPC
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Designing Event-Driven Applications with Apache NiFi, Apache Flink, Apache Sp...
Modern Data Architectures for Real Time Analytics & Engagement
Modern Data Architectures for Real Time Analytics & Engagement
Effective microservices development with ballerina
Effective microservices development with ballerina
"How about no grep and zabbix?". ELK based alerts and metrics.
"How about no grep and zabbix?". ELK based alerts and metrics.
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Pulsar summit asia 2021 apache pulsar with mqtt for edge computing
Iot platform supporting million requests per second
Iot platform supporting million requests per second
Recently uploaded
Unveiling the Advantages of Agile Software Development.pdf
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
Charla impartida en el evento de "KuberTENes Birthday Bash Guadalajara" para celebrar el 10mo. aniversario de Kubernetes #kuberTENes #celebr8k8s #k8s
Oracle 23c New Features For DBAs and Developers.pptx
Most important New features of Oracle 23c for DBAs and Developers. You can get more idea from my youtube channel video from https://youtu.be/XvL5WtaC20A
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...The Third Creative Media
"Navigating Invideo: A Comprehensive Guide" is an essential resource for anyone looking to master Invideo, an AI-powered video creation tool. This guide provides step-by-step instructions, helpful tips, and comparisons with other AI video creators. Whether you're a beginner or an experienced video editor, you'll find valuable insights to enhance your video projects and bring your creative ideas to life.E-commerce Development Services- Hornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
Webinar On-Demand: Using Flutter for Embedded
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
Measures in SQL (SIGMOD 2024, Santiago, Chile)
SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. Composable calculations are what is missing from SQL, and we propose a new kind of column, called a measure, that attaches a calculation to a table. Like regular tables, tables with measures are composable and closed when used in queries.
SQL-with-measures has the power, conciseness and reusability of multidimensional languages but retains SQL semantics. Measure invocations can be expanded in place to simple, clear SQL.
To define the evaluation semantics for measures, we introduce context-sensitive expressions (a way to evaluate multidimensional expressions that is consistent with existing SQL semantics), a concept called evaluation context, and several operations for setting and modifying the evaluation context.
A talk at SIGMOD, June 9–15, 2024, Santiago, Chile
Authors: Julian Hyde (Google) and John Fremlin (Google)
https://doi.org/10.1145/3626246.3653374
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Consistent toolbox talks are critical for maintaining workplace safety, as they provide regular opportunities to address specific hazards and reinforce safe practices.
These brief, focused sessions ensure that safety is a continual conversation rather than a one-time event, which helps keep safety protocols fresh in employees' minds. Studies have shown that shorter, more frequent training sessions are more effective for retention and behavior change compared to longer, infrequent sessions.
Engaging workers regularly, toolbox talks promote a culture of safety, empower employees to voice concerns, and ultimately reduce the likelihood of accidents and injuries on site.
The traditional method of conducting safety talks with paper documents and lengthy meetings is not only time-consuming but also less effective. Manual tracking of attendance and compliance is prone to errors and inconsistencies, leading to gaps in safety communication and potential non-compliance with OSHA regulations. Switching to a digital solution like Safelyio offers significant advantages.
Safelyio automates the delivery and documentation of safety talks, ensuring consistency and accessibility. The microlearning approach breaks down complex safety protocols into manageable, bite-sized pieces, making it easier for employees to absorb and retain information.
This method minimizes disruptions to work schedules, eliminates the hassle of paperwork, and ensures that all safety communications are tracked and recorded accurately. Ultimately, using a digital platform like Safelyio enhances engagement, compliance, and overall safety performance on site. https://safelyio.com/
Project Management: The Role of Project Dashboards.pdf
Project management is a crucial aspect of any organization, ensuring that projects are completed efficiently and effectively. One of the key tools used in project management is the project dashboard, which provides a comprehensive view of project progress and performance. In this article, we will explore the role of project dashboards in project management, highlighting their key features and benefits.
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Salesforce Healthcare CRM, implemented by VALiNTRY360, revolutionizes patient management by enhancing patient engagement, streamlining administrative processes, and improving care coordination. Its advanced analytics, robust security, and seamless integration with telehealth services ensure that healthcare providers can deliver personalized, efficient, and secure patient care. By automating routine tasks and providing actionable insights, Salesforce Healthcare CRM enables healthcare providers to focus on delivering high-quality care, leading to better patient outcomes and higher satisfaction. VALiNTRY360's expertise ensures a tailored solution that meets the unique needs of any healthcare practice, from small clinics to large hospital systems.
For more info visit us https://valintry360.com/solutions/health-life-sciences
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
In today's fiercely competitive mobile app market, the role of the QA team is pivotal for continuous improvement and sustained success. Effective testing strategies are essential to navigate the challenges confidently and precisely. Ensuring the perfection of mobile apps before they reach end-users requires thoughtful decisions in the testing plan.
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Wondering how X-Sign gained popularity in a quick time span? This eSign functionality of XfilesPro DocuPrime has many advancements to offer for Salesforce users. Explore them now!
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech is a premier mobile app development company in Noida, providing cutting-edge solutions for businesses.
Visit Us For : https://www.dronainfotech.com/mobile-application-development/
Energy consumption of Database Management - Florina Jonuzi
Presentation from Florina Jonuzi at the GSD Community Stage Meetup on June 06, 2024
WWDC 2024 Keynote Review: For CocoaCoders Austin
Overview of WWDC 2024 Keynote Address.
Covers: Apple Intelligence, iOS18, macOS Sequoia, iPadOS, watchOS, visionOS, and Apple TV+.
Understandable dialogue on Apple TV+
On-device app controlling AI.
Access to ChatGPT with a guest appearance by Chief Data Thief Sam Altman!
App Locking! iPhone Mirroring! And a Calculator!!
ACE - Team 24 Wrapup event at ahmedabad.
Atlassian Team 24 event wrap-up Ahmedabad. Recap of Atlassian team24 event.
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers. This video you can watch from my youtube channel at https://youtu.be/m-F-mZA3MkU
How Can Hiring A Mobile App Development Company Help Your Business Grow?
ToXSL Technologies is an award-winning Mobile App Development Company in Dubai that helps businesses reshape their digital possibilities with custom app services. As a top app development company in Dubai, we offer highly engaging iOS & Android app solutions. https://rb.gy/necdnt
UI5con 2024 - Bring Your Own Design System
How do you combine the OpenUI5/SAPUI5 programming model with a design system that makes its controls available as Web Components? Since OpenUI5/SAPUI5 1.120, the framework supports the integration of any Web Components. This makes it possible, for example, to natively embed own Web Components of your design system which are created with Stencil. The integration embeds the Web Components in a way that they can be used naturally in XMLViews, like with standard UI5 controls, and can be bound with data binding. Learn how you can also make use of the Web Components base class in OpenUI5/SAPUI5 to also integrate your Web Components and get inspired by the solution to generate a custom UI5 library providing the Web Components control wrappers for the native ones.
Recently uploaded (20)
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
Oracle 23c New Features For DBAs and Developers.pptx
Oracle 23c New Features For DBAs and Developers.pptx
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
8 Best Automated Android App Testing Tool and Framework in 2024.pdf
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Unlock the Secrets to Effortless Video Creation with Invideo: Your Ultimate G...
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Safelyio Toolbox Talk Softwate & App (How To Digitize Safety Meetings)
Project Management: The Role of Project Dashboards.pdf
Project Management: The Role of Project Dashboards.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
Top Benefits of Using Salesforce Healthcare CRM for Patient Management.pdf
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
A Comprehensive Guide on Implementing Real-World Mobile Testing Strategies fo...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Everything You Need to Know About X-Sign: The eSign Functionality of XfilesPr...
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Energy consumption of Database Management - Florina Jonuzi
Energy consumption of Database Management - Florina Jonuzi
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
How Can Hiring A Mobile App Development Company Help Your Business Grow?
How Can Hiring A Mobile App Development Company Help Your Business Grow?
OSDC 2014: Jordan Sissel - Find Happiness in your Logs
- 1. Logstash find happiness in your logs with Elasticsearch ELK
- 2. Open Source
- 3. Apache 2.0 License Open Source
- 4. Open and Friendly Community Open Source
- 5. If a new user has a bad time, it’s a bug. Open Source
- 6. Technology
- 7. Logstash
- 9. Logstash FiltersInputs Outputs 42 51 54 Processing & Transport
- 10. Logstash Processing & Transport LAMP Example Apache Mysql PHP Syslog Nagios Elasticsearch Graphite IRC Parse: Latency, Bandwidth, and Errors Load Balancer
- 12. Logstash Example Filters Grok Date FingerprintGeoIP Multiline User Agent Key-Value
- 13. Logstash Example Outputs Elasticsearch Graphite XMPP Nagios Email Pagerduty S3
- 14. Elasticsearch
- 15. Near Real-Time Search & Analysis Elasticsearch
- 17. REST + JSON API Elasticsearch
- 18. Kibana
- 21. Use Cases
- 22. Share Logs with Your Tech Support Team Use Cases
- 23. Graph all things! Use Cases
- 24. Exploration by Non- Technical Users Use Cases
- 26. Movie Releases Demo Use Cases
- 27. Apache Logs Demo Use Cases