DevOpsDays Amsterdam 2016 workshop
•Download as PPTX, PDF•
0 likes•308 views
This document provides information about a DevOpsDays 2016 workshop on creating an IT operations data lake using the ELK stack and Graphite. The workshop will be led by Arnold van Wijnbergen, who has 16 years of experience in IT automation. Attendees will learn how to use Logstash to ingest and process logs and metrics from various sources into Elasticsearch for analysis and visualization with Kibana and Grafana. The main components that will be covered are Logstash, Elasticsearch, Kibana, Graphite, and Kafka.
Report
Share
Report
Share
Recommended
Introduction to GraphQL & Serverless
This document summarizes a presentation about GraphQL and serverless architectures. It introduces GraphQL as a new philosophy for building APIs that gives clients control and reduces bandwidth. It then discusses how GraphQL compares favorably to REST by allowing a single data request and being self-documenting. The document outlines how serverless GraphQL works using AWS AppSync as an example and discusses some companies that use GraphQL and AppSync.
Take control of your DevOps Dumping Ground; Melissa Sussmann
Relay is a system that pulls together various DevOps tools and technologies to effectively manage environments. It listens to events from cloud services and orchestrates automated actions across tools using reusable workflow steps. Workflows are defined as code using YAML for parameters, secrets, outputs, and integration with tools like Kubernetes, serverless functions, and CI/CD pipelines. The system provides a library of integrations and makes it easy to author and share new steps.
SnapLogic Live: ServiceNow Integration
This document provides an overview of a SnapLogic Live presentation about integrating ServiceNow with SnapLogic. The presentation will demonstrate how SnapLogic can improve implementation quality, increase agility, reduce implementation times, lower costs and improve data accuracy when integrating with ServiceNow. Specifically, it will show a Salesforce to ServiceNow connection and compare point-to-point integration to using SnapLogic.
SnapLogic Live: Powering Cloud Analytics
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
2016 Tableau in the Cloud - A Netflix Original (AWS Re:invent)
Building a data platform doesn’t have to be like entering a portal to Stranger Things.
Join us in one hour for Tableau in the Cloud: A Netflix Original where Albert Wong, Netflix’s analytics expert, will show you how to simplify your data stack to deliver self-service analytics at scale.
Albert will discuss the details of connecting to big data, finding datasets, and discovering critical insights from visualizations. He will also share how Netflix is developing and growing their analytics ecosystem with Tableau, and how they prioritize sustaining their data culture of freedom and responsibility.
Why Dashboards Are Useless and Observability Is the New Buzzword
This document discusses the shift from traditional monitoring to observability as the new buzzword in software. It outlines how the modern paradigm of cloud, microservices, and ephemeral systems makes dashboards less useful for gaining insights. Observability requires processing large volumes of metrics, logs, and traces in real-time to understand complex interactions and identify anomalies. Streaming approaches are needed to operate on infinite streams of high-cardinality data at scale.
SnapLogic Live: IoT Integration
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
Massive Streaming Analytics with Spark Streaming
Streaming data allows for faster reactions and more profits by enabling real-time processing compared to typical batch analysis workflows. Prediction is more valuable than rule-based categorization and clustering because it allows for pattern discovery, adaptation to changing data, and "smart thinking". Spark Streaming is presented as a tool for building streaming architectures with ingestion, processing, and serving layers. A demo of card transaction analysis with real-time classification, fraud detection, and alert generation is shown.
Recommended
Introduction to GraphQL & Serverless
This document summarizes a presentation about GraphQL and serverless architectures. It introduces GraphQL as a new philosophy for building APIs that gives clients control and reduces bandwidth. It then discusses how GraphQL compares favorably to REST by allowing a single data request and being self-documenting. The document outlines how serverless GraphQL works using AWS AppSync as an example and discusses some companies that use GraphQL and AppSync.
Take control of your DevOps Dumping Ground; Melissa Sussmann
Relay is a system that pulls together various DevOps tools and technologies to effectively manage environments. It listens to events from cloud services and orchestrates automated actions across tools using reusable workflow steps. Workflows are defined as code using YAML for parameters, secrets, outputs, and integration with tools like Kubernetes, serverless functions, and CI/CD pipelines. The system provides a library of integrations and makes it easy to author and share new steps.
SnapLogic Live: ServiceNow Integration
This document provides an overview of a SnapLogic Live presentation about integrating ServiceNow with SnapLogic. The presentation will demonstrate how SnapLogic can improve implementation quality, increase agility, reduce implementation times, lower costs and improve data accuracy when integrating with ServiceNow. Specifically, it will show a Salesforce to ServiceNow connection and compare point-to-point integration to using SnapLogic.
SnapLogic Live: Powering Cloud Analytics
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
2016 Tableau in the Cloud - A Netflix Original (AWS Re:invent)
Building a data platform doesn’t have to be like entering a portal to Stranger Things.
Join us in one hour for Tableau in the Cloud: A Netflix Original where Albert Wong, Netflix’s analytics expert, will show you how to simplify your data stack to deliver self-service analytics at scale.
Albert will discuss the details of connecting to big data, finding datasets, and discovering critical insights from visualizations. He will also share how Netflix is developing and growing their analytics ecosystem with Tableau, and how they prioritize sustaining their data culture of freedom and responsibility.
Why Dashboards Are Useless and Observability Is the New Buzzword
This document discusses the shift from traditional monitoring to observability as the new buzzword in software. It outlines how the modern paradigm of cloud, microservices, and ephemeral systems makes dashboards less useful for gaining insights. Observability requires processing large volumes of metrics, logs, and traces in real-time to understand complex interactions and identify anomalies. Streaming approaches are needed to operate on infinite streams of high-cardinality data at scale.
SnapLogic Live: IoT Integration
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
Massive Streaming Analytics with Spark Streaming
Streaming data allows for faster reactions and more profits by enabling real-time processing compared to typical batch analysis workflows. Prediction is more valuable than rule-based categorization and clustering because it allows for pattern discovery, adaptation to changing data, and "smart thinking". Spark Streaming is presented as a tool for building streaming architectures with ingestion, processing, and serving layers. A demo of card transaction analysis with real-time classification, fraud detection, and alert generation is shown.
Librecon 2016 bilbao: kappa architecture IoT of the cars
This document discusses Kappa Architecture, which is a data processing architecture using Apache Kafka as the centralized data backbone. It summarizes Jay Kreps' original definition of Kappa Architecture and describes how it provides a more flexible way to do real-time and batch processing compared to the traditional Lambda Architecture. The document also provides an example of how a company implemented Kappa Architecture using Kafka and Spark Streaming to monitor car telemetry data from many vehicles and support flexible real-time queries over that data.
2016 09 measurecamp - event data modeling
Presentation by Christophe Bogaert to Measurecamp London September 2016. Christophe discussed what makes consuming and analysing event-streams difficult, and outlined a number of techniques for overcoming those obstacles.
SnapLogic Live: Anaplan Integration
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
From Data Analytics to Fast Data Intelligence
1) How to understand users with Data Analytics ?
2) How to build Real-time Music Recommender System from Data Stream ?
3) How to boost profit with Cross Sale in Real-time ?
Key Ideas to build Fast Data Intelligence Platform from Open Source Tools:
+ Apache Kafka
+ Apache Spark
+ RFX framework
Open Source Operations Analytics With Elastic
This document discusses using the Elastic Stack, which is an open source analytics stack for log analytics, system metrics analytics, and application metrics analytics. It includes Elasticsearch for indexing and searching data, Kibana for visualizing data, Beats for collecting and shipping data to Elasticsearch, and Logstash for processing pipelines. The document provides an overview of these tools and includes a demo.
Serverless Days Milano - Developing Serverless applications with GraphQL
This is the presentation that I gave at Serverless Days Milano 2019. Its a 10-minute presentation with lots of videos.
If you want to learn more about AppSync check my playlist on how to get started with this. https://www.youtube.com/playlist?list=PLGyRwGktEFqdX2cjO5xQVKb96q2DpwASR
Webinar: Evolution of Data Management for the IoT
The Internet of Things is here - slowly, unevenly and in vertical markets, creating new data silos. Evolving data management will be required to realize the full potential of the IoT. There are specific needs for properly addressing IoT data, which legacy ETL tools and database management systems simply don't handle well.
This presentation is from a webinar with Joseph A. di Paolantonio, industry expert working at the convergence of IoT with data management and analytics, where he discusses five recommendations on gaining advantage through the latest IoT data management technologies and business processes.
Xtending nintex workflow cloud w azure functions - xchange conference
This document summarizes two presentations given at the Nintex xchange 2018 conference about extending Nintex Workflow Cloud capabilities using Azure Functions. The first presentation described using an Azure Function to log compliance activities from Nintex workflows to Azure storage. The second presentation described building a Function to retrieve US Government per diem travel rates by making a call to the GSA API and returning the rates in JSON for use in Nintex workflows. Both presentations included demos of creating the Functions with Visual Studio and configuring them as custom actions in Nintex Workflow Cloud.
Creando tu primera aplicación con Angular 2, el nuevo súper framework de Google
Sesión presentada en SG Virtual 11a. edición.
Presentada por: Alex Arriaga.
Durante esta sesión aprenderemos cómo crear una aplicación con la nueva versión de Angular (que ya es estable), asimismo revisaremos los archivos de configuración más comunes y cómo utilizar TypeScript para poder generar componentes.
ALM Works Structure - Boston Atlassian User Group
Structure allows users to manage Jira projects at scale by providing portfolio-level overviews, cross-project tracking and management, and workload balancing capabilities. It creates a unified interface across multiple Jira applications and provides flexibility in customizing hierarchies of any depth and visualizing dependencies between issues from different projects. Structure also enables powerful data presentation and reporting, extensive customization options, and integration with other tools like Confluence.
Add ons for stash
This document summarizes several add-ons for Atlassian Stash, including plugins that provide graphs of commit activity, allow Subversion to Git migration, add gamification through achievements for coding activities, enable categorizing projects, and allow archiving source code downloads. The plugins discussed add analytics, migration tools, gamification, improved organization and archiving capabilities to the Stash platform.
Fast Data processing with RFX
This document discusses using RFX (Reactive Function X), a design pattern and collection of open source tools, to solve fast data problems. It presents an example of using RFX for web analytics to count pageviews and unique users and detect DDOS attacks. The RFX approach applies the BEAM methodology for agile data warehousing. It demonstrates RFX concepts like event data actors, agents, collectors, routers, processors, storage and reactors using a pageview analytics demo with source code on GitHub.
Aws slide
The document discusses AWS Lambda including how it allows serverless applications to be built without provisioning or managing servers, how functions can be triggered by events and executed on demand, and some common use cases for AWS Lambda including powering chatbots, voice apps, data processing, web and mobile backends, and IoT applications.
Serverless microservices in the wild
Managing Serverless Microservices in the Wild
Handing off responsibility for your microservice's infrastructure with AWS API Gateway and Lambda is great, but it also means losing visibility and influence over your app in production.
In this talk, we'll discuss different strategies to gain insights into what actually is going on inside your serverless app. We will present a tool we've built at atomData to get high quality, actionable logs and metrics about our services.
Misusing MLflow To Help Deduplicate Data At Scale
At Intuit, we have a lot of data – and a lot of duplicate data collected over decades. So we built a rule-based, self-serve tool to identify and merge duplicate records. It takes experimentation and iteration to get deduplication just right for 100s of millions of records, and spreadsheet-based tracking just wasn’t enough. We now use MLflow to automatically capture execution notes, rule settings, weights, key validation metrics, etc., all without requiring end-user action. In this talk, we’ll talk about our use case and why MLflow is useful outside its traditional ML Ops use cases.
AWS Dev Day 2018
Attended Amazon Web Services Dev Day 2018. Shared experience with all office colleagues by taking a presentation. Presentation covers Chaos Engineering, Serverless Architecture, IoT, Serverless Data Lake.
Microservices: Architecture and Practice
Microservices are a service-oriented architecture composed of loosely coupled elements that have bounded contexts. Each microservice can be updated independently without risking other services. Considerations for microservices include the complexity of building distributed systems, increased number of components to manage, steps needed to maintain data consistency across services, and increased network latency and API traffic. Iron.io advocates for empowering developers by providing abstraction, self-service tools, and focusing on code.
AllThingsOpen 2018 - Deployment Design Patterns (Dan Zaratsian)
While the adoption of machine learning and deep learning techniques continue to grow, many organizations find it difficult to actually deploy these sophisticated models into production. It is common to see data scientists build powerful models, yet these models are not deployed because of the complexity of the technology used or lack of understanding related to the process of pushing these models into production.
As part of this talk, I will review several deployment design patterns for both real-time and batch use cases. I’ll show how these models can be deployed as scalable, distributed deployments within the cloud, scaled across hadoop clusters, as APIs, and deployed within streaming analytics pipelines. I will also touch on topics related to security, end-to-end governance, pitfalls, challenges, and useful tools across a variety of platforms. This presentation will involve demos and sample code for the the deployment design patterns.
Effortless HVAC simulation using ApacheHVAC
This document provides an agenda and overview of an introduction webinar on ApacheHVAC. The agenda includes welcome remarks, an introduction to ApacheHVAC by Niall Gibson, and a demonstration of the ApacheHVAC Wizard and autosizing features by Hans Dhargalkar, followed by a question and answer session. Niall Gibson's introduction discusses the need for ApacheHVAC due to the climate emergency and the technology's ability to close the performance gap between predicted and actual building performance to 5-10% by using dynamic simulation models and operational data from actual buildings. The ApacheHVAC Wizard is highlighted as simplifying the process and generating load and system sizing reports.
A Look at TensorFlow.js
TensforFlow.js is a WebGL accelerated, browser based JavaScript library for training and deploying ML models. We will take a look at this library and how it can be used to train and run deep learning models in the browser.
During this talk we will go through the basics of Deep Learning and TensforFlow.js and how we can use it to create artificial neural networks, train and run existing models in the browser.
Speaker: Jamal O'Garro
I'm a former finance guy on the business side turned finance guy on the tech side that also loves entrepreneurship and NYC startup culture. I'm a huge fan of full-stack web development with JavaScript and enjoy building software with AngularJS and Node.js as well as building native iOS apps. In my spare time I like to write and backtest simple algo trading strategies in Python, attend hackathons, give tech talks and teach others how to code. In my spare spare time (whenever that is) I like to read, blog, write songs,shoot street photography and make documentary films. I love to create and my motto is simply, "THINK IT, BELIEVE IT, CREATE IT!"
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Once you start working with distributed Big Data systems, you start discovering a whole bunch of problems you won’t find in monolithic systems.
All of a sudden to monitor all of the components becomes a big data problem itself.
In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system once you’re using tools like:
Web Services, Apache Spark, Cassandra, MongoDB, Amazon Web Services.
Not only the tools, what should you monitor about the actual data that flows in the system?
And we’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system using tools like: Web Services,Spark,Cassandra,MongoDB,AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.
More Related Content
What's hot
Librecon 2016 bilbao: kappa architecture IoT of the cars
This document discusses Kappa Architecture, which is a data processing architecture using Apache Kafka as the centralized data backbone. It summarizes Jay Kreps' original definition of Kappa Architecture and describes how it provides a more flexible way to do real-time and batch processing compared to the traditional Lambda Architecture. The document also provides an example of how a company implemented Kappa Architecture using Kafka and Spark Streaming to monitor car telemetry data from many vehicles and support flexible real-time queries over that data.
2016 09 measurecamp - event data modeling
Presentation by Christophe Bogaert to Measurecamp London September 2016. Christophe discussed what makes consuming and analysing event-streams difficult, and outlined a number of techniques for overcoming those obstacles.
SnapLogic Live: Anaplan Integration
Watch this recorded demonstration of SnapLogic from our team of experts who answer your hybrid cloud and big data integration questions.
From Data Analytics to Fast Data Intelligence
1) How to understand users with Data Analytics ?
2) How to build Real-time Music Recommender System from Data Stream ?
3) How to boost profit with Cross Sale in Real-time ?
Key Ideas to build Fast Data Intelligence Platform from Open Source Tools:
+ Apache Kafka
+ Apache Spark
+ RFX framework
Open Source Operations Analytics With Elastic
This document discusses using the Elastic Stack, which is an open source analytics stack for log analytics, system metrics analytics, and application metrics analytics. It includes Elasticsearch for indexing and searching data, Kibana for visualizing data, Beats for collecting and shipping data to Elasticsearch, and Logstash for processing pipelines. The document provides an overview of these tools and includes a demo.
Serverless Days Milano - Developing Serverless applications with GraphQL
This is the presentation that I gave at Serverless Days Milano 2019. Its a 10-minute presentation with lots of videos.
If you want to learn more about AppSync check my playlist on how to get started with this. https://www.youtube.com/playlist?list=PLGyRwGktEFqdX2cjO5xQVKb96q2DpwASR
Webinar: Evolution of Data Management for the IoT
The Internet of Things is here - slowly, unevenly and in vertical markets, creating new data silos. Evolving data management will be required to realize the full potential of the IoT. There are specific needs for properly addressing IoT data, which legacy ETL tools and database management systems simply don't handle well.
This presentation is from a webinar with Joseph A. di Paolantonio, industry expert working at the convergence of IoT with data management and analytics, where he discusses five recommendations on gaining advantage through the latest IoT data management technologies and business processes.
Xtending nintex workflow cloud w azure functions - xchange conference
This document summarizes two presentations given at the Nintex xchange 2018 conference about extending Nintex Workflow Cloud capabilities using Azure Functions. The first presentation described using an Azure Function to log compliance activities from Nintex workflows to Azure storage. The second presentation described building a Function to retrieve US Government per diem travel rates by making a call to the GSA API and returning the rates in JSON for use in Nintex workflows. Both presentations included demos of creating the Functions with Visual Studio and configuring them as custom actions in Nintex Workflow Cloud.
Creando tu primera aplicación con Angular 2, el nuevo súper framework de Google
Sesión presentada en SG Virtual 11a. edición.
Presentada por: Alex Arriaga.
Durante esta sesión aprenderemos cómo crear una aplicación con la nueva versión de Angular (que ya es estable), asimismo revisaremos los archivos de configuración más comunes y cómo utilizar TypeScript para poder generar componentes.
ALM Works Structure - Boston Atlassian User Group
Structure allows users to manage Jira projects at scale by providing portfolio-level overviews, cross-project tracking and management, and workload balancing capabilities. It creates a unified interface across multiple Jira applications and provides flexibility in customizing hierarchies of any depth and visualizing dependencies between issues from different projects. Structure also enables powerful data presentation and reporting, extensive customization options, and integration with other tools like Confluence.
Add ons for stash
This document summarizes several add-ons for Atlassian Stash, including plugins that provide graphs of commit activity, allow Subversion to Git migration, add gamification through achievements for coding activities, enable categorizing projects, and allow archiving source code downloads. The plugins discussed add analytics, migration tools, gamification, improved organization and archiving capabilities to the Stash platform.
Fast Data processing with RFX
This document discusses using RFX (Reactive Function X), a design pattern and collection of open source tools, to solve fast data problems. It presents an example of using RFX for web analytics to count pageviews and unique users and detect DDOS attacks. The RFX approach applies the BEAM methodology for agile data warehousing. It demonstrates RFX concepts like event data actors, agents, collectors, routers, processors, storage and reactors using a pageview analytics demo with source code on GitHub.
Aws slide
The document discusses AWS Lambda including how it allows serverless applications to be built without provisioning or managing servers, how functions can be triggered by events and executed on demand, and some common use cases for AWS Lambda including powering chatbots, voice apps, data processing, web and mobile backends, and IoT applications.
Serverless microservices in the wild
Managing Serverless Microservices in the Wild
Handing off responsibility for your microservice's infrastructure with AWS API Gateway and Lambda is great, but it also means losing visibility and influence over your app in production.
In this talk, we'll discuss different strategies to gain insights into what actually is going on inside your serverless app. We will present a tool we've built at atomData to get high quality, actionable logs and metrics about our services.
Misusing MLflow To Help Deduplicate Data At Scale
At Intuit, we have a lot of data – and a lot of duplicate data collected over decades. So we built a rule-based, self-serve tool to identify and merge duplicate records. It takes experimentation and iteration to get deduplication just right for 100s of millions of records, and spreadsheet-based tracking just wasn’t enough. We now use MLflow to automatically capture execution notes, rule settings, weights, key validation metrics, etc., all without requiring end-user action. In this talk, we’ll talk about our use case and why MLflow is useful outside its traditional ML Ops use cases.
AWS Dev Day 2018
Attended Amazon Web Services Dev Day 2018. Shared experience with all office colleagues by taking a presentation. Presentation covers Chaos Engineering, Serverless Architecture, IoT, Serverless Data Lake.
Microservices: Architecture and Practice
Microservices are a service-oriented architecture composed of loosely coupled elements that have bounded contexts. Each microservice can be updated independently without risking other services. Considerations for microservices include the complexity of building distributed systems, increased number of components to manage, steps needed to maintain data consistency across services, and increased network latency and API traffic. Iron.io advocates for empowering developers by providing abstraction, self-service tools, and focusing on code.
AllThingsOpen 2018 - Deployment Design Patterns (Dan Zaratsian)
While the adoption of machine learning and deep learning techniques continue to grow, many organizations find it difficult to actually deploy these sophisticated models into production. It is common to see data scientists build powerful models, yet these models are not deployed because of the complexity of the technology used or lack of understanding related to the process of pushing these models into production.
As part of this talk, I will review several deployment design patterns for both real-time and batch use cases. I’ll show how these models can be deployed as scalable, distributed deployments within the cloud, scaled across hadoop clusters, as APIs, and deployed within streaming analytics pipelines. I will also touch on topics related to security, end-to-end governance, pitfalls, challenges, and useful tools across a variety of platforms. This presentation will involve demos and sample code for the the deployment design patterns.
Effortless HVAC simulation using ApacheHVAC
This document provides an agenda and overview of an introduction webinar on ApacheHVAC. The agenda includes welcome remarks, an introduction to ApacheHVAC by Niall Gibson, and a demonstration of the ApacheHVAC Wizard and autosizing features by Hans Dhargalkar, followed by a question and answer session. Niall Gibson's introduction discusses the need for ApacheHVAC due to the climate emergency and the technology's ability to close the performance gap between predicted and actual building performance to 5-10% by using dynamic simulation models and operational data from actual buildings. The ApacheHVAC Wizard is highlighted as simplifying the process and generating load and system sizing reports.
A Look at TensorFlow.js
TensforFlow.js is a WebGL accelerated, browser based JavaScript library for training and deploying ML models. We will take a look at this library and how it can be used to train and run deep learning models in the browser.
During this talk we will go through the basics of Deep Learning and TensforFlow.js and how we can use it to create artificial neural networks, train and run existing models in the browser.
Speaker: Jamal O'Garro
I'm a former finance guy on the business side turned finance guy on the tech side that also loves entrepreneurship and NYC startup culture. I'm a huge fan of full-stack web development with JavaScript and enjoy building software with AngularJS and Node.js as well as building native iOS apps. In my spare time I like to write and backtest simple algo trading strategies in Python, attend hackathons, give tech talks and teach others how to code. In my spare spare time (whenever that is) I like to read, blog, write songs,shoot street photography and make documentary films. I love to create and my motto is simply, "THINK IT, BELIEVE IT, CREATE IT!"
What's hot (20)
Librecon 2016 bilbao: kappa architecture IoT of the cars
Librecon 2016 bilbao: kappa architecture IoT of the cars
Serverless Days Milano - Developing Serverless applications with GraphQL
Serverless Days Milano - Developing Serverless applications with GraphQL
Xtending nintex workflow cloud w azure functions - xchange conference
Xtending nintex workflow cloud w azure functions - xchange conference
Creando tu primera aplicación con Angular 2, el nuevo súper framework de Google
Creando tu primera aplicación con Angular 2, el nuevo súper framework de Google
AllThingsOpen 2018 - Deployment Design Patterns (Dan Zaratsian)
AllThingsOpen 2018 - Deployment Design Patterns (Dan Zaratsian)
Similar to DevOpsDays Amsterdam 2016 workshop
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Once you start working with distributed Big Data systems, you start discovering a whole bunch of problems you won’t find in monolithic systems.
All of a sudden to monitor all of the components becomes a big data problem itself.
In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system once you’re using tools like:
Web Services, Apache Spark, Cassandra, MongoDB, Amazon Web Services.
Not only the tools, what should you monitor about the actual data that flows in the system?
And we’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system using tools like: Web Services,Spark,Cassandra,MongoDB,AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
This document discusses monitoring big data systems in a simple way. It begins with introducing the speaker and their background. The rest of the document outlines monitoring concepts, common big data architectures involving Spark and Cassandra, and potential problems that can arise. It then provides recommendations for setting up a monitoring stack involving metrics collection, logging, dashboards, and alerting. Specifically, it recommends using Graphite, Grafana, Coralogix, and Redash. The document emphasizes the importance of correlating application and system metrics and asking the right monitoring questions.
Maintainable Machine Learning Products
Machine learning applications are typically stitched together from hopes and dreams, shell scripts, cron jobs, home-grown schedulers, snippets of configuration clipped from multiple blog posts, thousands of hard-coded business rules, a.k.a. "our SQL corpus," and a few lines of training and testing code. Organizing all the moving parts into something maintainable and supportive of ongoing development is a challenge most teams have on their TODO list, roadmap, or tech debt pile. Getting ahead of the day-to-day demands and settling into a sane architecture often seems like an unattainable goal. The past several years have seen an explosion of tool-building in the data engineering and analytics area, including in Apache projects spanning the areas of search and information retrieval, job orchestration, file and stream formats, and machine learning libraries. In this talk we will cover our product and development teams' choices of architecture and tools, from data ingestion and storage, through transformations and processing, to presentation of results and publishing to web services, reports, and applications.
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
Presentation at ODTUG KScope'18 on the data engineering and advanced analytics capabilities in Oracle Analytics Cloud Data Lake Edition, Oracle Big Data Cloud and Oracle Event Hub Cloud Service
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Once you start working with Big Data systems, you discover a whole bunch of problems you won’t find in monolithic systems. Monitoring all of the components becomes a big data problem itself. In the talk, we’ll mention all of the aspects that you should take into consideration when monitoring a distributed system using tools like Web Services, Spark, Cassandra, MongoDB, AWS. Not only the tools, what should you monitor about the actual data that flows in the system? We’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Once you start working with distributed Big Data systems, you start discovering a whole bunch of problems you won’t find in monolithic systems.
All of a sudden to monitor all of the components becomes a big data problem itself.
In the talk we’ll mention all of the aspects that you should take in consideration when monitoring a distributed system once you’re using tools like:
Web Services, Apache Spark, Cassandra, MongoDB, Amazon Web Services.
Not only the tools, what should you monitor about the actual data that flows in the system?
And we’ll cover the simplest solution with your day to day open source tools, the surprising thing, that it comes not from an Ops Guy.
Time's Up! Getting Value from Big Data Now
The Briefing Room with Dr. Robin Bloor and CASK
We all know the promise of big data, but who gets the value? There are plenty of success stories already, and most of them involve one key ingredient: facilitated access to important data sets. Most research studies suggest that the Pareto principle applies: 80 percent goes to data integration, and only 20 to analysis. Inverting that balance is the Holy Grail.
Register for this episode of The Briefing Room to hear veteran Analyst Dr. Robin Bloor explain why the time has finally come for turning the tables on the status quo in analytics. He'll be briefed by CASK CEO Jonathan Gray, who will showcase his company's big data integration platform, CDAP, which was specifically designed to expedite time-to-value for big data.
Big Data Beers - Introducing Snowplow
This document introduces Snowplow, an open-source web and event analytics platform. Snowplow was created in 2012 by Alex Dean and Yali Sassoon to address limitations in traditional analytics programs. It allows users to capture, transform, store and analyze granular event-level data in their own data warehouses. Snowplow uses a loosely coupled architecture and supports collecting data from any source and storing it in various databases. It aims to provide a platform for real-time and offline analytics across an organization.
SharePoint and the User Interface with JavaScript
How JavaScript can be used as the magic glue to bring the user interface and interactive pages to life in SharePoint.
Anaconda and PyData Solutions
Continuum Analytics provides the Anaconda platform for data science. It includes popular Python data science packages like NumPy, SciPy, Pandas, Scikit-learn, and the Jupyter notebook. Continuum was founded by Travis Oliphant, creator of NumPy and Numba, to support the open source Python data science community and make it easier to do data analytics and visualization using Python. The Anaconda platform has over 2 million users and makes it simple to install and work with Python and related packages for data science and machine learning.
Cycling for noobs
The document provides an overview of CycleJS and reactive programming (RP). It discusses why the author chose CycleJS and RP due to their component model, declarative code, and explicit data flow. It addresses some initial concerns with RP like requiring a "brain rewire" and barriers to reuse. It demonstrates a simple app called Brian designed for people with cognitive disabilities. It describes the learning process of getting used to RP concepts like state management and higher order operators. Resources for learning more about CycleJS, RxJS, and debugging reactive apps are provided. The author notes improvements from the XStream library and looks forward to better visualization and debugging tools.
Dev Ops Training
The document provides an agenda for a DevOps advanced class on Spark being held in June 2015. The class will cover topics such as RDD fundamentals, Spark runtime architecture, memory and persistence, Spark SQL, PySpark, and Spark Streaming. It will include labs on DevOps 101 and 102. The instructor has over 5 years of experience providing Big Data consulting and training, including over 100 classes taught.
.NET per la Data Science e oltre
Come può .NET contribuire alla Data Science? Cosa è .NET Interactive? Cosa c'entrano i notebook? E Apache Spark? E il pythonismo? E Azure? Vediamo in questa sessione di mettere in ordine le idee.
Lambda architecture with Spark
Presentation on the struggles with traditional architectures and an overview of the Lambda Architecture utilizing Spark to drive massive amounts of both batch and streaming data for processing and analytics
How bol.com makes sense of its logs, using the Elastic technology stack.
Bol.com uses the Elastic (ELK) stack to make sense of logs from over 1,600 servers and 500-600 million events per day. Key aspects of their system include:
1. Shipping JSON-formatted log events from sources like Apache, databases, and applications to Redis queues to allow multiple Logstash instances to process events in real-time without data loss.
2. Enriching log events with information like request IDs to correlate requests across services, and IP-to-role mappings to identify client roles.
3. Using Elasticsearch aggregations and transformations to generate a directed graph of service dependencies based on logs, to help understand their distributed architecture.
Continuum Analytics and Python
Talk given to the Philly Python Users Group (PUG) on October 1, 2015: http://www.meetup.com/phillypug/ Thanks SIG (http://www.sig.com) for hosting!
Lyft talks #4 Orchestrating big data and ML pipelines at Lyft
This document discusses how Lyft uses big data and machine learning pipelines at scale. It provides examples of how Lyft uses data to improve maps, calculate optimal pickup locations, detect inaccurate destinations, estimate routes and times, and forecast demand and supply. It describes Lyft's data ecosystem including 50PB of data in S3, 650k ETL pipeline runs processing 24M tasks monthly. It also summarizes the key differences between Airflow and Flyte orchestration tools, when each is best suited, and provides a live demo of Flyte.
Architecting an Open Source AI Platform 2018 edition
How to build a scalable AI platform using open source software. The end-to-end architecture covers data integration, interactive queries & visualization, machine learning & deep learning, deploying models to production, and a full 24x7 operations toolset in a high-compliance environment.
Data streaming
- The document profiles Alberto Paro and his experience including a Master's Degree in Computer Science Engineering from Politecnico di Milano, experience as a Big Data Practise Leader at NTTDATA Italia, authoring 4 books on ElasticSearch, and expertise in technologies like Apache Spark, Playframework, Apache Kafka, and MongoDB. He is also an evangelist for the Scala and Scala.JS languages.
The document then provides an overview of data streaming architectures, popular message brokers like Apache Kafka, RabbitMQ, and Apache Pulsar, streaming frameworks including Apache Spark, Apache Flink, and Apache NiFi, and streaming libraries such as Reactive Streams.
Similar to DevOpsDays Amsterdam 2016 workshop (20)
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Milan 2017 - D...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Demi Ben-Ari - Monitoring Big Data Systems Done "The Simple Way" - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
Monitoring Big Data Systems Done "The Simple Way" - Codemotion Berlin 2017
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
From BI Developer to Data Engineer with Oracle Analytics Cloud Data Lake Edition
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems Done "The Simple Way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
Monitoring Big Data Systems "Done the simple way" - Demi Ben-Ari - Codemotion...
How bol.com makes sense of its logs, using the Elastic technology stack.
How bol.com makes sense of its logs, using the Elastic technology stack.
Lyft talks #4 Orchestrating big data and ML pipelines at Lyft
Lyft talks #4 Orchestrating big data and ML pipelines at Lyft
Architecting an Open Source AI Platform 2018 edition
Architecting an Open Source AI Platform 2018 edition
More from Arnold Van Wijnbergen
Security Analytics with OpenSearch
Securing platforms like Kubernetes can be challenging. Luckily there are tools to create insights into potential security threats. Get an introduction into the world of Security Information Event Monitoring (SIEM) and how to make OpenSearch your favorite solution for Security Analytics. You get familiar with the technology and concepts behind this powerful platform. Talk includes hands-on demo to get a grasp of provided functionality.
Reliability as a Discipline
Reliability as a Discipline (RaaD). Presentation about Chaos Engineering background, practices and approach
Contributing Today: Chaos Engineering mini demo Litmus Chaos
See the actual recording here: https://www.youtube.com/watch?v=m-WZEvrPvAY&t=395s
Also you can see the actual mentioned code here: https://github.com/avwsolutions/litmus-chaos-elmo-demo
Kong Ingress Controller - Fullstaq Show N Tell
These are part of the a live session, which has a recording available at https://www.bigmarker.com/fullstaq/Show-n-Tell-Kubernetes-Ingress-with-Kong . I hope you will enjoy the live session.
What can you expect when using Kong ? Kong can help you in various use cases like API gateway for authentication or broader API management uses cases. In this specific scenario Kong is part of the Kong Ingress Controller for Kubernetes, which in essence provides HTTP(S) or HTTP/2 intelligent traffic routing, that operates at layer 7 and additionally can provide load balancing capabilities to your kubernetes cluster. To automatically obtain certificates we setup cert-manager, which provides access to ACME providers like Venafi cloud or Let's Encrypt.
Why Tooling (Only) Isn’t The Answer
The increasing importance of our tooling, the rely on their ability to automate and potential damage when they don’t work properly should make us think about repositioning your tooling strategy. Nowadays tools are an essential part and getting more important every day for delivering high quality productivity, but why are we still handle them as non critical parts?
We can all agree that it’s seems such simple thing selecting your favourite tools, but that it’s actually a complex undertaking, by avoiding pitfalls and select your Swiss knife. Most companies are still struggling to find their best fitted tools for their organisation, processes and technology stack, caused by various legitimate reasons, but end up with no beneficial use or even not used at all. Think that we all recognize this situation?
During this presentation I will take you on the journey of defining and refining your end goal, effectively learn from the things that matter and define your Enterprise Tooling strategy. We will explore some typical pitfalls and practices that help you to identify their strengths and opportunities for your organisation to leverage from. After all we want to get productive as early as possible.
Life of an event - A never ending tool chain
A story about Continuous Monitoring, which takes you as an engineer through the various aspects of monitoring to survive high competitive business and demanding customers by applying the DevOps approach.
More from Arnold Van Wijnbergen (6)
Contributing Today: Chaos Engineering mini demo Litmus Chaos
Contributing Today: Chaos Engineering mini demo Litmus Chaos
Recently uploaded
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
Charla impartida en el evento de "KuberTENes Birthday Bash Guadalajara" para celebrar el 10mo. aniversario de Kubernetes #kuberTENes #celebr8k8s #k8s
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Dr. Jesús Barrasa, Head of Solutions Architecture for EMEA, Neo4j
Découvrez les dernières innovations de Neo4j, et notamment les dernières intégrations cloud et les améliorations produits qui font de Neo4j un choix essentiel pour les développeurs qui créent des applications avec des données interconnectées et de l’IA générative.
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Explore the seamless transition to e-invoicing with this comprehensive guide tailored for Saudi Arabian businesses. Navigate the process effortlessly with step-by-step instructions designed to streamline implementation and enhance efficiency.
原版定制美国纽约州立大学奥尔巴尼分校毕业证学位证书原版一模一样
原版一模一样【微信：741003700 】【美国纽约州立大学奥尔巴尼分校毕业证学位证书】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。
本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma
1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700
留信网认证的作用:
1:该专业认证可证明留学生真实身份
2:同时对留学生所学专业登记给予评定
3:国家专业人才认证中心颁发入库证书
4:这个认证书并且可以归档倒地方
5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息
6:个人职称评审加20分
7:个人信誉贷款加10分
8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Webinar On-Demand: Using Flutter for Embedded
Flutter is a popular open source, cross-platform framework developed by Google. In this webinar we'll explore Flutter and its architecture, delve into the Flutter Embedder and Flutter’s Dart language, discover how to leverage Flutter for embedded device development, learn about Automotive Grade Linux (AGL) and its consortium and understand the rationale behind AGL's choice of Flutter for next-gen IVI systems. Don’t miss this opportunity to discover whether Flutter is right for your project.
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Presentation from Ghazal Aakel and Eric Jochum at the GSD Community Stage Meetup on June 6th, 2024
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers. This video you can watch from my youtube channel at https://youtu.be/m-F-mZA3MkU
SWEBOK and Education at FUSE Okinawa 2024
Takashi Kobayashi and Hironori Washizaki, "SWEBOK Guide and Future of SE Education," First International Symposium on the Future of Software Engineering (FUSE), June 3-6, 2024, Okinawa, Japan
Hand Rolled Applicative User Validation Code Kata
Could you use a simple piece of Scala validation code (granted, a very simplistic one too!) that you can rewrite, now and again, to refresh your basic understanding of Applicative operators <*>, <*, *>?
The goal is not to write perfect code showcasing validation, but rather, to provide a small, rough-and ready exercise to reinforce your muscle-memory.
Despite its grandiose-sounding title, this deck consists of just three slides showing the Scala 3 code to be rewritten whenever the details of the operators begin to fade away.
The code is my rough and ready translation of a Haskell user-validation program found in a book called Finding Success (and Failure) in Haskell - Fall in love with applicative functors.
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
https://2024.springio.net/sessions/automated-software-refactoring-with-openrewrite-and-generative-ai/
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.
Unveiling the Advantages of Agile Software Development.pdf
Learn about Agile Software Development's advantages. Simplify your workflow to spur quicker innovation. Jump right in! We have also discussed the advantages.
What is Master Data Management by PiLog Group
PiLog Group's Master Data Record Manager (MDRM) is a sophisticated enterprise solution designed to ensure data accuracy, consistency, and governance across various business functions. MDRM integrates advanced data management technologies to cleanse, classify, and standardize master data, thereby enhancing data quality and operational efficiency.
LORRAINE ANDREI_LEQUIGAN_HOW TO USE WHATSAPP.pptx
WhatsApp offers simple, reliable, and private messaging and calling services for free worldwide. With end-to-end encryption, your personal messages and calls are secure, ensuring only you and the recipient can access them. Enjoy voice and video calls to stay connected with loved ones or colleagues. Express yourself using stickers, GIFs, or by sharing moments on Status. WhatsApp Business enables global customer outreach, facilitating sales growth and relationship building through showcasing products and services. Stay connected effortlessly with group chats for planning outings with friends or staying updated on family conversations.
socradar-q1-2024-aviation-industry-report.pdf
SOCRadar's Aviation Industry Q1 Incident Report is out now!
The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers.
SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.
OpenMetadata Community Meeting - 5th June 2024
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
Graspan: A Big Data System for Big Code Analysis
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
E-commerce Development Services- Hornet Dynamics
For any business hoping to succeed in the digital age, having a strong online presence is crucial. We offer Ecommerce Development Services that are customized according to your business requirements and client preferences, enabling you to create a dynamic, safe, and user-friendly online store.
Recently uploaded (20)
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
KuberTENes Birthday Bash Guadalajara - Introducción a Argo CD
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
Neo4j - Product Vision and Knowledge Graphs - GraphSummit Paris
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian Companies
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
ALGIT - Assembly Line for Green IT - Numbers, Data, Facts
Oracle Database 19c New Features for DBAs and Developers.pptx
Oracle Database 19c New Features for DBAs and Developers.pptx
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
Automated software refactoring with OpenRewrite and Generative AI.pptx.pdf
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
UI5con 2024 - Boost Your Development Experience with UI5 Tooling Extensions
Unveiling the Advantages of Agile Software Development.pdf
Unveiling the Advantages of Agile Software Development.pdf
Fundamentals of Programming and Language Processors
Fundamentals of Programming and Language Processors
DevOpsDays Amsterdam 2016 workshop
- 1. DevOpsDays 2016 Amsterdam First steps to create your IT Operations data lake with ELK Stack and Graphite
- 2. 2 Arnold van Wijnbergen @BSMConsultant / Arnold.van.Wijnbergen@gmail.com •16 years working experience with automating IT •DevOps Evangelist with great passion about Architecture and Automation Tooling •Full time Tooling Geek Automation Monitoring, tasted them all Predictive Analytics Infrastructure-as-Code #IlovIT ☁
- 3. 3 INTRODUCE OURSELVES Who we are and what do we expect
- 4. 4 CHOOSE YOUR WAY Dev, Ops or just DevOps DOD-AMS-WORKSHOP-BANKIT TOURIST ROUTE DOD-AMS-WORKSHOP DANGER ROUTE
- 5. JUST ANOTHER DATA LAKE CONCEPT Metrics, logs, health-states, etc
- 6. “ELK Stack + Graphite + Kafka makes a great combination”
- 8. Main component for managing your element is LogStash Reference : https://www.elastic.co/guide/en/logstash/current/introduction.html
- 9. Logstash Structured way of Working – Processing Pipeline Reference : https://www.elastic.co/guide/en/logstash/current/pipeline.html
- 10. Pipeline processing of an event is mainly existing of three important stages Reference : https://www.elastic.co/guide/en/logstash/current/filter-plugins.html (Filter) Processing Parse fields out Enrich fields values Correlate On field value matching
- 11. Parse unstructured data and make IT readable structured
- 12. Enrich & correlate our structured data set
- 13. Success with the workshop https://goo.gl/97xOX2 13
- 14. Backup slides 14
- 15. Meet Mr LogStash What can he do for you ? I Love your data … :-)
- 16. Many test options are available to validate your code and expected functional behaviour Code Validation with CONFIGTEST Unit Testing with RSPEC Reference : https://www.elastic.co/blog/logstash-functionality-through-testing
- 17. Now combine the forces and create a structured configuration Configuration folder Input.conf Input.conf Input.conf Filter.conf Input.conf Output.conf file stdi n mutat egro k stdou telasticsear chredi s Reference : https://www.elastic.co/guide/en/logstash/current/configuration-file- structure.html
Editor's Notes
- Goal of this presentation is that with infra monitoring your are not there yet
- Syntax validation logstash –configtest Functional testinglogstash.rb logstash –rspec # encoding: utf-8 require "test_utils" file = "/etc/logstash/conf.d/test.conf" @@configuration = String.new @@configuration << File.read(file) describe "Test event" do extend LogStash::RSpec config(@@configuration) message = %(2015-21-01 12:01:02.003 UTC Hello 42 3.14159) sample("message" => message, "type" => "test") do insist { subject["type"] } == "test" insist { subject["timestamp"] } == "2015-21-01 12:01:02.003 UTC" insist { subject["word1"] } == "Hello" insist { subject["int1"] } == 42 insist { subject["inner"]["float1"] } == 3.14159 end end