Presentation at the 2017 LAST (Lean, Agile, Systems Thinking) Conference.
A presentation about the challenges involved in building a production Big Data system used directly by customers.
Capgemini: Observability within the Dutch governmentElasticsearch
The Dutch government relies on a complex mix of technologies to deliver digital services. This makes it difficult to monitor performance and identify issues when they arise. Capgemini implemented Elastic solutions to provide observability across the heterogeneous infrastructure. This allowed problems to be traced and resolved in minutes rather than days. The improved visibility has enhanced operational stability and reduced breakdowns. Capgemini sees continued growth in demand for Elastic technologies from both government and commercial customers.
Fineo Technical Overview - NextSQL for IoTJesse Yates
Fineo is a turn-key data management platform for enterprise IoT that provides a NoSQL time-series database integrated with an analytics warehouse. It offers insights with 10x lower cost and the ability to scale to 100x more data. Fineo provides a "simple" big data deployment through its web scale architecture, security/compliance features, and one-click ETL tools to enable faster adoption and lower complexity.
Security Events Logging at Bell with the Elastic StackElasticsearch
One of Canada’s largest telecommunications company is using Elastic to drive improved security analysis in their SOC. With a need to ingest all security logs, build threat detection models, and normalize many new types of logs, the Bell security team turned to Elastic. Learn how they’ve streamlined alerts, deepened log analysis, and addressed challenges unique to being an ISP.
Improving search at Wellcome CollectionElasticsearch
Wellcome Collection is a free museum and library challenging how we think and feel about health. See how the Elasticsearch Service is used to aggregate descriptive data and provide unified search and discovery.
See the video: https://www.elastic.co/elasticon/tour/2019/london/improving-search-at-wellcome-collection
Massive scale with no fear and no secrets for Apigee and our customers. Both for our cloud or their data centers. At I Love APIs 2014 Nicola Cardace, a global performance practice lead at Apigee along with Aaron Strey, senior engineer at Target walked us through a customer success story. A journey into zen-like discipline, obsessive customer focus where the secrets of the mastery will be disclosed. A report about the delivery of architectures at scale and how Apigee applies thought processes during a customer engagement.
We have a software solution that enables enterprise customers to transfer their data around the world 17 times faster that they can not using current standards.
Unlike other companies that have data acceleration technologies our solution does not require additional hardware or bandwidth as a result, we are half the cost of any other solution.
Our product will help you cope with the explosive growth of traffic reducing per-unit investment and energy consumption.
Capgemini: Observability within the Dutch governmentElasticsearch
The Dutch government relies on a complex mix of technologies to deliver digital services. This makes it difficult to monitor performance and identify issues when they arise. Capgemini implemented Elastic solutions to provide observability across the heterogeneous infrastructure. This allowed problems to be traced and resolved in minutes rather than days. The improved visibility has enhanced operational stability and reduced breakdowns. Capgemini sees continued growth in demand for Elastic technologies from both government and commercial customers.
Fineo Technical Overview - NextSQL for IoTJesse Yates
Fineo is a turn-key data management platform for enterprise IoT that provides a NoSQL time-series database integrated with an analytics warehouse. It offers insights with 10x lower cost and the ability to scale to 100x more data. Fineo provides a "simple" big data deployment through its web scale architecture, security/compliance features, and one-click ETL tools to enable faster adoption and lower complexity.
Security Events Logging at Bell with the Elastic StackElasticsearch
One of Canada’s largest telecommunications company is using Elastic to drive improved security analysis in their SOC. With a need to ingest all security logs, build threat detection models, and normalize many new types of logs, the Bell security team turned to Elastic. Learn how they’ve streamlined alerts, deepened log analysis, and addressed challenges unique to being an ISP.
Improving search at Wellcome CollectionElasticsearch
Wellcome Collection is a free museum and library challenging how we think and feel about health. See how the Elasticsearch Service is used to aggregate descriptive data and provide unified search and discovery.
See the video: https://www.elastic.co/elasticon/tour/2019/london/improving-search-at-wellcome-collection
Massive scale with no fear and no secrets for Apigee and our customers. Both for our cloud or their data centers. At I Love APIs 2014 Nicola Cardace, a global performance practice lead at Apigee along with Aaron Strey, senior engineer at Target walked us through a customer success story. A journey into zen-like discipline, obsessive customer focus where the secrets of the mastery will be disclosed. A report about the delivery of architectures at scale and how Apigee applies thought processes during a customer engagement.
We have a software solution that enables enterprise customers to transfer their data around the world 17 times faster that they can not using current standards.
Unlike other companies that have data acceleration technologies our solution does not require additional hardware or bandwidth as a result, we are half the cost of any other solution.
Our product will help you cope with the explosive growth of traffic reducing per-unit investment and energy consumption.
Reinventing enterprise defense with the Elastic StackElasticsearch
Tune in to hear the most impactful lessons learned from Uber's security journey, and how security practitioners everywhere can tackle pervasive enterprise security challenges using the Elastic Stack.
Mike Metral from Rackspace will discuss how the open cloud model with OpenStack and Rackspace can help organizations build cloud-aware applications and leverage unified management and automation across hybrid cloud environments. The agenda includes an overview of OpenStack, the benefits of cloud-aware applications, hybrid IT models, and a demonstration. Rackspace and RightScale provide a unified solution to manage applications and infrastructure across public and private clouds.
Sync17 is a new file sync app that can help with slow data traffic, long sync processes, and expensive bandwidth. It is most efficient, reliable, easy to scale and integrate. Sync17 solves data transfer challenges by reducing costs and time while increasing quality, security, reliability and efficiency. It provides 17x faster traffic, 17x increased data volume, 8x reduced delivery time, and 5x reduced power consumption compared to other solutions. A business case shows Sync17 reduces total annual costs by 18-42% and increases gross profits for a company compared to alternative solutions over 12 months.
Monitoring involves analyzing infrastructure issues and failures in both virtual and physical systems. As virtual systems increased, the number of systems needing monitoring also increased significantly. Monitoring is a key part of approaches like DevOps and SRE that focus on system reliability. It involves collecting metrics in real-time, logs about events and activities, and curated alerts based on metrics and logs within their proper context. Tools like Cloudwatch, Riemann, DataDog, Loggly, Splunk, ELK, and Bosun can help bring these facets of monitoring together.
O monitoramento da infraestrutura facilitado, da ingestão ao insightElasticsearch
Descubra como a integração simplificada de dados com integrações predefinidas, insights automatizados com alertas e aprendizado de máquina e novas ferramentas visuais estão otimizando o caso de uso de monitoramento de infraestrutura.
Telegraf is an open-source server agent designed to collect metrics from stacks, sensors, and systems — with nearly 300 inputs and outputs. Telegraf Operator makes it easy to use Telegraf for monitoring your Kubernetes workloads. It enables developers to define a common output destination for all metrics, and configure Sidecar monitoring on your application pods using annotations. With the Telegraf sidecar container added, it will collect data and start pushing the metrics to a time series database, like InfluxDB. Discover how to use the Telegraf Operator as a control center for managing individual Telegraf instances which are deployed throughout Kubernetes clusters. Find out how to use the InfluxDB and Telegraf Operator to monitor and get metrics from your Kubernetes workloads.
Join this webinar as InfluxData's Pat Gaughen and Wojciech Kocjan provide:
InfluxDB & Telegraf overview
Telegraf Operator deep-dive
Live demos of sample deployments!
Automatize a detecção de ameaças e evite falsos positivosElasticsearch
Detecte ameaças e evite falsos positivos com o mecanismo de detecção no Elastic Security. Automatize a detecção de ameaças por meio de correlações e machine learning com exemplos do reais.
Sumit Goel - Monitoring Cloud Applications Using Zabbix | ZabConf2016Zabbix
With global shift towards flexibility of cloud there are different demands on monitoring availability and performance of applications provided in the cloud. There are obvious limitations in accessing components of app hosted by third party run outside of internal environment. Same time there are opportunities of using vendor API and status page. In Salesforce, one of the most innovative company in the world by Forbes and one of the biggest cloud service provider, we understand the need of customer to be able to see in real time availability and performance of cloud application. In the following presentation we're going to list and describe multiple ways of monitoring cloud apps. Some of the methods are: building in web monitoring using Curl, web browser automation tools like Selenium, external scripts (reading vendor status dashboard) and API calls to the app.
Detection, Response and the Azazel RootkitThreat Stack
This document discusses continuous monitoring in the cloud and introduces Cloud Sight as a solution. It provides context on the cloud threat landscape and shared security responsibilities in the cloud. It then details how Cloud Sight uses continuous monitoring and behavioral profiling to detect the Azazel rootkit on cloud assets, reconstructing terminal sessions and gathering full context on asset behavior. The presentation concludes with a Q&A.
This document describes a field device diagnostic solution developed by MapleLabs. The solution involves collecting large amounts of diagnostic data from thousands of field devices, ingesting and parsing the data using microservices running on a Kubernetes cluster in AWS, and storing the structured data in Elasticsearch for analysis and detection of faults. Some key challenges addressed are scalability to handle bursty data loads, security of the data as it moves through the pipeline, debuggability of the microservices, and reducing storage costs through tiering to S3.
It’s one thing to support many data sources with megabytes of data. It’s a completely different problem supporting thousands of data sources with terabytes of data every day. How do you create systems that scale infinitely?
The answer is; you don’t . You can not design for infinite scalability. Rather, consider a pod approach where each pod supports a defined capacity. Scalability results from deployment of multiple cooperating pods.
Systems handling extremely large data sources with significant processing requirements are difficult at best to validate. Attempting to deploy such a system without well understood capacity limits is destined for failure.
This was first presented at Cloud Expo NYC.
Elastic APM: amplificação dos seus logs e métricas para proporcionar um panor...Elasticsearch
Não importa onde você esteja em sua jornada rumo à nuvem, o Elastic APM ajuda a oferecer melhores experiências ao cliente, identificando gargalos de desempenho e identificando regressões de novas implantações com mais rapidez.
OSMC 2017 | Icinga2 in a 24/7 Broadcast Environment by Dave KempeNETWAYS
I will present some war stories and implementation details from our Icinga2 deployments into television broadcast environments. From plugins we needed to develop, to challenges in effecting change in staff practices I will walk through the projects and share my experiences on the way.
This will be a useful talk for anyone looking to run a monitoring project and the approach used to get management and general staff on board.
Then we will cover the implementation of distributed monitoring in Icinga2 with strict firewalls, building dashboards using Nagvis and integration of Opsgenie for alerting.
In addition, the process of training staff and using the Windows Agent installer to deploy Icinga to various windows servers will also be covered.
This webinar will instruct data scientists and machine learning engineers how to build manage and deploy auto-adaptive machine learning models in production. Data is ever changing, leaving your models outdated and built on old data. This can lead to underperforming models and a lot of manual work to fix it. By allowing your models to continually learn you’ll ensure that they run at peak performance. Using state of the art Kubernetes infrastructure, we’ll show you how to automatically track and manage your auto-adaptive machine learning models while in production. By building auto-adaptive machine learning models, data engineers can bridge the gap between research and production. After this webinar you’ll be able to build and deploy machine learning pipelines that automatically adapt and retrain based on any validation trigger you choose.
Key webinar takeaways:
How to build auto-adaptive machine learning pipelines
How to use Kubernetes to manage and scale models in production
How to automatically monitor for peak performance
How to set up continuous deployment of ML pipeline
Watch all our webinars at https://cnvrg.io/webinars-and-workshops/
Using OPC-UA to Extract IIoT Time Series Data from PLC and SCADA SystemsInfluxData
Algist Bruggeman NV produces yeast for large-scale bakeries and home bakers. The company lacked insight into its fermentation process as its sensor data collection process was manual. Production data was committed to paper, making it difficult to compare batches, aggregate production parameters or detect anomalies.
Factry.IO’s data historian, built on InfluxDB, has helped the company collect process data, enabling it to gain more insight into its production process and provide predictive maintenance.
In this webinar, learn about Algist Bruggeman NV’s business outcomes and the technical setup of linking time series data with ERP, planning and quality data for operational improvement.
Webinar slides: DevOps Tutorial: how to automate your database infrastructureSeveralnines
Join our guest speaker Riaan Nolan of mukuru.com, the First Puppet Labs Certified Professional in South Africa, as he walks us through the facets of DevOps integrations and the mission-critical advantages that database automation can bring to your database infrastructure.
Infrastructure automation isn’t easy, but it’s not rocket science either. Done right, it is a worthwhile investment, but deciding on which tools to invest in can be a confusing and overwhelming process. Riaan will share some of his secrets on how to proceed with this and he knows what he’s talking about: he saves the companies he works for substantial amounts on their monthly IT bills, typically around 50%.
Don’t miss out on this opportunity to understand how you can find efficiencies for your database infrastructure and do watch this webinar to understand the key pain points, which indicate that it’s time to invest in database automation.
AGENDA
DevOps and databases - what are the challenges
Managing databases in a DevOps environment
- Requirements from microservice environments
- Automated deployments
- Performance monitoring
- Backups
- Schema changes
- Version upgrades
- Automated failover
- Integration with ChatOps and other tools
Data distribution
- Database hosting in cloud environments
- Managing data flows
Cloud Automation on AWS
SPEAKERS
Riaan Nolan was the First Puppet Labs Certified Professional in South Africa. Riaan uses Amazon EC2, VPC and Autoscale with Cloudformation to spin up complete stacks with Autoscaling Fleets. He saves companies substantial amounts on their monthly IT bills, typically around 50% - yes, at one company that meant $500k+ per year. And he’s participated in a number of community tech related forums. He uses next generation technologies such as AWS, Cloudformation, Autoscale, Puppet, GlusterFS, NGINX, Magento and PHP to power huge eCommerce stores. His specialties are Puppet Automation, Cloud Deployments, eCommerce, eMarketing, Specialized Linux Services, Windows, Process making, Budgets, Asset Tracking, Procurement.
- Devops Lead, Mukuru
- Expert Live Systems Administrator, foodpanda | Hellofood
- Senior Systems Administrator / Infrastructure Lead, Rocket Internet GmbH
- Senior Technology Manager, Africa Internet Accelerator
Art van Scheppingen is a Senior Support Engineer at Severalnines. He’s a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.
Automate threat detections and avoid false positivesElasticsearch
Detect threats and avoid the noise of false positives with the detection engine in Elastic Security. Automate threat detection via correlations and machine learning through real-world examples.
This document proposes a solution to streamline the database monitoring workflow by removing manual steps and integrating tools. Currently, alerts are sent over email requiring manual lookups of host details and comparisons to ignore lists. The proposed solution is to configure the monitoring tool to push alerts to a script that processes them along with inventory data to generate a web dashboard. The dashboard would group alerts and allow one-click access to production databases eliminating manual SSH sessions and menu navigation. Benefits include a task-focused interface, no data copying/pasting, and potential to integrate with configuration tools.
[WSO2Con USA 2018] Microservices, Containers, and BeyondWSO2
This slide deck discusses what's next in this highly agile, massively distributed environment. It will focus on fine-tuned DevOps processes, governance, and observability in a massively distributed container native microservices platform.
Historic Opportunities: Discover the Power of Ignition's HistorianInductive Automation
This document provides an overview of Ignition's tag historian capabilities. It discusses Ignition's technology stack for serving as an historian, including storing tag data in SQL databases. The document addresses common historian questions around scaling, data access, visualization and benchmarks. It highlights that Ignition can handle millions of tags across various SQL databases and supports features like dashboards, reports, streaming data to other systems and remote data collection.
Data Science in Production: Technologies That Drive Adoption of Data Science ...Nir Yungster
Critical to a data science team’s ability to drive impact is its effectiveness in incorporating its solutions into new or existing products. When collaborating with other engineering teams, and especially when solutions must operate at scale, technological choices can be critical factors in determining what type of outcome you'll have. We walk through strategies and specific technologies - Airflow, Docker, Kubernetes - that can help promote successful collaboration between data science and engineering.
Data Con LA 2018 - Enabling real-time exploration and analytics at scale at H...Data Con LA
Enabling real-time exploration and analytics at scale to drive operational intelligence at Hulu by Indrasis Mondal, Director, Data Engineering and Data Products, Hulu
Data is one of most powerful assets for companies today and a key driver for innovation, product development and business efficiency. Operational intelligence allows modern organization to use that data asset in real-time to enable immediate insights to their business operations and allow rapid decision making for strategic advantage. In this presentation we will walk through the operational intelligence capabilities Hulu has built to process tens of millions of events per minute to enable fast exploration of data and real-time decision making .
Reinventing enterprise defense with the Elastic StackElasticsearch
Tune in to hear the most impactful lessons learned from Uber's security journey, and how security practitioners everywhere can tackle pervasive enterprise security challenges using the Elastic Stack.
Mike Metral from Rackspace will discuss how the open cloud model with OpenStack and Rackspace can help organizations build cloud-aware applications and leverage unified management and automation across hybrid cloud environments. The agenda includes an overview of OpenStack, the benefits of cloud-aware applications, hybrid IT models, and a demonstration. Rackspace and RightScale provide a unified solution to manage applications and infrastructure across public and private clouds.
Sync17 is a new file sync app that can help with slow data traffic, long sync processes, and expensive bandwidth. It is most efficient, reliable, easy to scale and integrate. Sync17 solves data transfer challenges by reducing costs and time while increasing quality, security, reliability and efficiency. It provides 17x faster traffic, 17x increased data volume, 8x reduced delivery time, and 5x reduced power consumption compared to other solutions. A business case shows Sync17 reduces total annual costs by 18-42% and increases gross profits for a company compared to alternative solutions over 12 months.
Monitoring involves analyzing infrastructure issues and failures in both virtual and physical systems. As virtual systems increased, the number of systems needing monitoring also increased significantly. Monitoring is a key part of approaches like DevOps and SRE that focus on system reliability. It involves collecting metrics in real-time, logs about events and activities, and curated alerts based on metrics and logs within their proper context. Tools like Cloudwatch, Riemann, DataDog, Loggly, Splunk, ELK, and Bosun can help bring these facets of monitoring together.
O monitoramento da infraestrutura facilitado, da ingestão ao insightElasticsearch
Descubra como a integração simplificada de dados com integrações predefinidas, insights automatizados com alertas e aprendizado de máquina e novas ferramentas visuais estão otimizando o caso de uso de monitoramento de infraestrutura.
Telegraf is an open-source server agent designed to collect metrics from stacks, sensors, and systems — with nearly 300 inputs and outputs. Telegraf Operator makes it easy to use Telegraf for monitoring your Kubernetes workloads. It enables developers to define a common output destination for all metrics, and configure Sidecar monitoring on your application pods using annotations. With the Telegraf sidecar container added, it will collect data and start pushing the metrics to a time series database, like InfluxDB. Discover how to use the Telegraf Operator as a control center for managing individual Telegraf instances which are deployed throughout Kubernetes clusters. Find out how to use the InfluxDB and Telegraf Operator to monitor and get metrics from your Kubernetes workloads.
Join this webinar as InfluxData's Pat Gaughen and Wojciech Kocjan provide:
InfluxDB & Telegraf overview
Telegraf Operator deep-dive
Live demos of sample deployments!
Automatize a detecção de ameaças e evite falsos positivosElasticsearch
Detecte ameaças e evite falsos positivos com o mecanismo de detecção no Elastic Security. Automatize a detecção de ameaças por meio de correlações e machine learning com exemplos do reais.
Sumit Goel - Monitoring Cloud Applications Using Zabbix | ZabConf2016Zabbix
With global shift towards flexibility of cloud there are different demands on monitoring availability and performance of applications provided in the cloud. There are obvious limitations in accessing components of app hosted by third party run outside of internal environment. Same time there are opportunities of using vendor API and status page. In Salesforce, one of the most innovative company in the world by Forbes and one of the biggest cloud service provider, we understand the need of customer to be able to see in real time availability and performance of cloud application. In the following presentation we're going to list and describe multiple ways of monitoring cloud apps. Some of the methods are: building in web monitoring using Curl, web browser automation tools like Selenium, external scripts (reading vendor status dashboard) and API calls to the app.
Detection, Response and the Azazel RootkitThreat Stack
This document discusses continuous monitoring in the cloud and introduces Cloud Sight as a solution. It provides context on the cloud threat landscape and shared security responsibilities in the cloud. It then details how Cloud Sight uses continuous monitoring and behavioral profiling to detect the Azazel rootkit on cloud assets, reconstructing terminal sessions and gathering full context on asset behavior. The presentation concludes with a Q&A.
This document describes a field device diagnostic solution developed by MapleLabs. The solution involves collecting large amounts of diagnostic data from thousands of field devices, ingesting and parsing the data using microservices running on a Kubernetes cluster in AWS, and storing the structured data in Elasticsearch for analysis and detection of faults. Some key challenges addressed are scalability to handle bursty data loads, security of the data as it moves through the pipeline, debuggability of the microservices, and reducing storage costs through tiering to S3.
It’s one thing to support many data sources with megabytes of data. It’s a completely different problem supporting thousands of data sources with terabytes of data every day. How do you create systems that scale infinitely?
The answer is; you don’t . You can not design for infinite scalability. Rather, consider a pod approach where each pod supports a defined capacity. Scalability results from deployment of multiple cooperating pods.
Systems handling extremely large data sources with significant processing requirements are difficult at best to validate. Attempting to deploy such a system without well understood capacity limits is destined for failure.
This was first presented at Cloud Expo NYC.
Elastic APM: amplificação dos seus logs e métricas para proporcionar um panor...Elasticsearch
Não importa onde você esteja em sua jornada rumo à nuvem, o Elastic APM ajuda a oferecer melhores experiências ao cliente, identificando gargalos de desempenho e identificando regressões de novas implantações com mais rapidez.
OSMC 2017 | Icinga2 in a 24/7 Broadcast Environment by Dave KempeNETWAYS
I will present some war stories and implementation details from our Icinga2 deployments into television broadcast environments. From plugins we needed to develop, to challenges in effecting change in staff practices I will walk through the projects and share my experiences on the way.
This will be a useful talk for anyone looking to run a monitoring project and the approach used to get management and general staff on board.
Then we will cover the implementation of distributed monitoring in Icinga2 with strict firewalls, building dashboards using Nagvis and integration of Opsgenie for alerting.
In addition, the process of training staff and using the Windows Agent installer to deploy Icinga to various windows servers will also be covered.
This webinar will instruct data scientists and machine learning engineers how to build manage and deploy auto-adaptive machine learning models in production. Data is ever changing, leaving your models outdated and built on old data. This can lead to underperforming models and a lot of manual work to fix it. By allowing your models to continually learn you’ll ensure that they run at peak performance. Using state of the art Kubernetes infrastructure, we’ll show you how to automatically track and manage your auto-adaptive machine learning models while in production. By building auto-adaptive machine learning models, data engineers can bridge the gap between research and production. After this webinar you’ll be able to build and deploy machine learning pipelines that automatically adapt and retrain based on any validation trigger you choose.
Key webinar takeaways:
How to build auto-adaptive machine learning pipelines
How to use Kubernetes to manage and scale models in production
How to automatically monitor for peak performance
How to set up continuous deployment of ML pipeline
Watch all our webinars at https://cnvrg.io/webinars-and-workshops/
Using OPC-UA to Extract IIoT Time Series Data from PLC and SCADA SystemsInfluxData
Algist Bruggeman NV produces yeast for large-scale bakeries and home bakers. The company lacked insight into its fermentation process as its sensor data collection process was manual. Production data was committed to paper, making it difficult to compare batches, aggregate production parameters or detect anomalies.
Factry.IO’s data historian, built on InfluxDB, has helped the company collect process data, enabling it to gain more insight into its production process and provide predictive maintenance.
In this webinar, learn about Algist Bruggeman NV’s business outcomes and the technical setup of linking time series data with ERP, planning and quality data for operational improvement.
Webinar slides: DevOps Tutorial: how to automate your database infrastructureSeveralnines
Join our guest speaker Riaan Nolan of mukuru.com, the First Puppet Labs Certified Professional in South Africa, as he walks us through the facets of DevOps integrations and the mission-critical advantages that database automation can bring to your database infrastructure.
Infrastructure automation isn’t easy, but it’s not rocket science either. Done right, it is a worthwhile investment, but deciding on which tools to invest in can be a confusing and overwhelming process. Riaan will share some of his secrets on how to proceed with this and he knows what he’s talking about: he saves the companies he works for substantial amounts on their monthly IT bills, typically around 50%.
Don’t miss out on this opportunity to understand how you can find efficiencies for your database infrastructure and do watch this webinar to understand the key pain points, which indicate that it’s time to invest in database automation.
AGENDA
DevOps and databases - what are the challenges
Managing databases in a DevOps environment
- Requirements from microservice environments
- Automated deployments
- Performance monitoring
- Backups
- Schema changes
- Version upgrades
- Automated failover
- Integration with ChatOps and other tools
Data distribution
- Database hosting in cloud environments
- Managing data flows
Cloud Automation on AWS
SPEAKERS
Riaan Nolan was the First Puppet Labs Certified Professional in South Africa. Riaan uses Amazon EC2, VPC and Autoscale with Cloudformation to spin up complete stacks with Autoscaling Fleets. He saves companies substantial amounts on their monthly IT bills, typically around 50% - yes, at one company that meant $500k+ per year. And he’s participated in a number of community tech related forums. He uses next generation technologies such as AWS, Cloudformation, Autoscale, Puppet, GlusterFS, NGINX, Magento and PHP to power huge eCommerce stores. His specialties are Puppet Automation, Cloud Deployments, eCommerce, eMarketing, Specialized Linux Services, Windows, Process making, Budgets, Asset Tracking, Procurement.
- Devops Lead, Mukuru
- Expert Live Systems Administrator, foodpanda | Hellofood
- Senior Systems Administrator / Infrastructure Lead, Rocket Internet GmbH
- Senior Technology Manager, Africa Internet Accelerator
Art van Scheppingen is a Senior Support Engineer at Severalnines. He’s a pragmatic MySQL and Database expert with over 15 years experience in web development. He previously worked at Spil Games as Head of Database Engineering, where he kept a broad vision upon the whole database environment: from MySQL to Couchbase, Vertica to Hadoop and from Sphinx Search to SOLR. He regularly presents his work and projects at various conferences (Percona Live, FOSDEM) and related meetups.
Automate threat detections and avoid false positivesElasticsearch
Detect threats and avoid the noise of false positives with the detection engine in Elastic Security. Automate threat detection via correlations and machine learning through real-world examples.
This document proposes a solution to streamline the database monitoring workflow by removing manual steps and integrating tools. Currently, alerts are sent over email requiring manual lookups of host details and comparisons to ignore lists. The proposed solution is to configure the monitoring tool to push alerts to a script that processes them along with inventory data to generate a web dashboard. The dashboard would group alerts and allow one-click access to production databases eliminating manual SSH sessions and menu navigation. Benefits include a task-focused interface, no data copying/pasting, and potential to integrate with configuration tools.
[WSO2Con USA 2018] Microservices, Containers, and BeyondWSO2
This slide deck discusses what's next in this highly agile, massively distributed environment. It will focus on fine-tuned DevOps processes, governance, and observability in a massively distributed container native microservices platform.
Historic Opportunities: Discover the Power of Ignition's HistorianInductive Automation
This document provides an overview of Ignition's tag historian capabilities. It discusses Ignition's technology stack for serving as an historian, including storing tag data in SQL databases. The document addresses common historian questions around scaling, data access, visualization and benchmarks. It highlights that Ignition can handle millions of tags across various SQL databases and supports features like dashboards, reports, streaming data to other systems and remote data collection.
Data Science in Production: Technologies That Drive Adoption of Data Science ...Nir Yungster
Critical to a data science team’s ability to drive impact is its effectiveness in incorporating its solutions into new or existing products. When collaborating with other engineering teams, and especially when solutions must operate at scale, technological choices can be critical factors in determining what type of outcome you'll have. We walk through strategies and specific technologies - Airflow, Docker, Kubernetes - that can help promote successful collaboration between data science and engineering.
Data Con LA 2018 - Enabling real-time exploration and analytics at scale at H...Data Con LA
Enabling real-time exploration and analytics at scale to drive operational intelligence at Hulu by Indrasis Mondal, Director, Data Engineering and Data Products, Hulu
Data is one of most powerful assets for companies today and a key driver for innovation, product development and business efficiency. Operational intelligence allows modern organization to use that data asset in real-time to enable immediate insights to their business operations and allow rapid decision making for strategic advantage. In this presentation we will walk through the operational intelligence capabilities Hulu has built to process tens of millions of events per minute to enable fast exploration of data and real-time decision making .
The journey to Native Cloud Architecture & Microservices, tracing the footste...Mek Srunyu Stittri
The document discusses Netflix's adoption of microservices and continuous delivery to improve speed and agility. Key points include:
1) Netflix moved to microservices and continuous delivery on the cloud to dramatically speed up product development and deployment.
2) This allowed independent teams to deploy code frequently without coordination, with automated testing and deployment replacing handoffs and long release cycles.
3) Netflix's approach involved building stateless, independently deployable microservices; continuous monitoring; and other techniques to enable developers to deploy code safely and rapidly.
This document provides a summary of Netflix's architecture and use of open source software. It discusses:
- Why Netflix open sources software, including gathering feedback, collaboration, and improving retention and recruiting
- Popular Netflix open source projects like Eureka, Ribbon, and Hystrix that are widely used in cloud architectures
- Netflix's microservices architecture and emphasis on automation, high availability, and continuous delivery
- How Netflix ensures operational visibility and security at scale through open source tools like Turbine, Atlas, and Security Monkey
- Getting started resources for understanding and running Netflix's technologies like ZeroToCloud and ZeroToDocker workshops
Presentazione dello speech tenuto da Carmine Spagnuolo (Postdoctoral Research Fellow - Università degli Studi di Salerno/ ACT OR) dal titolo "Technology insights: Decision Science Platform", durante il Decision Science Forum 2019, il più importante evento italiano sulla Scienza delle Decisioni.
Maturing IoT solutions with Microsoft Azure (Sam Vanhoutte & Glenn Colpaert a...Codit
“Internet of Things” is changing our world and today the Internet of Things knows almost as many applications as there are types of devices connected. In this session, Sam and Glenn will give an overview of the latest IoT solutions, the different learnings from the field and explain which key components are instrumental to integrating your solutions to the Azure IoT platform to ensure they are robust, future-proof and secure.
We will show what aspects of hawkBit need to be completed or implemented to use it in a production environment.
We will look at hawkBit's extension points and how they can be used and show some practices for deploying and managing a hawkBit-based product.
The document discusses DevOps practices for TYPO3 projects. It defines DevOps as the confluence of development and operations. It highlights the importance of communication between different roles like developers, system administrators, and integrators. It also provides examples of tools and techniques that can be used at different stages of a TYPO3 project to facilitate DevOps practices, such as automated testing, deployment automation, and content synchronization.
Triangle Devops Meetup covering Netflix open source, cloud architecture, and what Andrew did in his first year working as a senior software engineer in the cloud platform group.
Grant Delp has over 21 years of experience in IT management, network design and engineering, network administration, storage infrastructure, and network security. As a senior network engineer, he has led teams and taken on responsibilities such as managing IT field operations, designing new data centers and networks, administering firewalls, storage systems, and servers, and completing projects like office moves and equipment replacements.
These are the slides of the second talk of the first Tech Talk@TransferWise Singapore, which happened on the 23rd of November 2017.
These slides share how TransferWise codebase is moving from a monolith architecture to a microservices architecture.
The document discusses the benefits and challenges of running big data workloads on cloud native platforms. Some key points discussed include:
- Big data workloads are migrating to the cloud to take advantage of scalability, flexibility and cost effectiveness compared to on-premises solutions.
- Enterprise cloud platforms need to provide centralized management and monitoring of multiple clusters, secure data access, and replication capabilities.
- Running big data on cloud introduces challenges around storage, networking, compute resources, and security that systems need to address, such as consistency issues with object storage, network throughput reductions, and hardware variations across cloud vendors.
- The open source community is helping users address these challenges to build cloud native data architectures
DATA @ NFLX (Tableau Conference 2014 Presentation)Blake Irvine
I presented this at a 2014 Tableau Conference session with Albert Wong.
Netflix relies on data to make decisions ranging from buying and recommending content, to improving the streaming experience on devices.
This presentation shares our Big Data analytics architecture and the tools used to make data accessible throughout our business, focusing on how Tableau fits into our organization and why it aligns well with our culture.
Workshop: Delivering chnages for applications and databasesEduardo Piairo
This document provides an overview of DevOps concepts and practices for delivering changes for applications and databases. It discusses topics like continuous integration, continuous delivery, infrastructure as code, database automation, and managing changes. The document is presented as a training material divided into multiple chapters that cover application development, infrastructure, databases, managing changes, and establishing a DevOps engineering practice. It emphasizes the importance of collaboration and communication between development and operations teams to enable continuous software delivery.
PLNOG19 - Piotr Marecki - Espresso: Scalable and Programmable Peering EdgePROIDEA
Prezentacja rozwiązania SDN ( projekt espresso - https://blog.google/topics/google-cloud/making-google-cloud-faster-more-available-and-cost-effective-extending-sdn-public-internet-espresso/ ) dla sieci brzegowej Google. Opisuje rozproszoną architekture warstwy kontrolnej i warstwy przesyłania pakietów, system mapowania oraz omawia doświadczenie operatorskie zebrane przy wspieraniu systemu w warunkach produkcyjnych.
DevOpsDays Tel Aviv DEC 2022 | Building A Cloud-Native Platform Brick by Bric...Haggai Philip Zagury
The overwhelming growth of technologies in the Cloud Native foundation overtook our toolbox and completely changed (well, really enhanced) the Developer Experience.
In this talk, I will try to provide my personal journey from the "Operator to Developer's chair" and the practices which helped me along my journey as a Cloud-Native Dev ;)
The document provides an overview of cloud computing, including its history, models, architecture, security concerns, and importance. Cloud computing allows users to access software and store data on remote servers rather than local hardware. It has evolved from early concepts in the 1950s-60s to platforms like Amazon Web Services today. There are three main service models: Infrastructure as a Service, Platform as a Service, and Software as a Service. Cloud computing also raises issues around data protection, identity management, and compliance with varying legal requirements across jurisdictions. Overall it allows for improved data storage, maintenance, and security compared to traditional computing.
Similar to Last Conference 2017: Big Data in a Production Environment: Lessons Learnt (20)
8 Best Automated Android App Testing Tool and Framework in 2024.pdfkalichargn70th171
Regarding mobile operating systems, two major players dominate our thoughts: Android and iPhone. With Android leading the market, software development companies are focused on delivering apps compatible with this OS. Ensuring an app's functionality across various Android devices, OS versions, and hardware specifications is critical, making Android app testing essential.
Do you want Software for your Business? Visit Deuglo
Deuglo has top Software Developers in India. They are experts in software development and help design and create custom Software solutions.
Deuglo follows seven steps methods for delivering their services to their customers. They called it the Software development life cycle process (SDLC).
Requirement — Collecting the Requirements is the first Phase in the SSLC process.
Feasibility Study — after completing the requirement process they move to the design phase.
Design — in this phase, they start designing the software.
Coding — when designing is completed, the developers start coding for the software.
Testing — in this phase when the coding of the software is done the testing team will start testing.
Installation — after completion of testing, the application opens to the live server and launches!
Maintenance — after completing the software development, customers start using the software.
SOCRadar's Aviation Industry Q1 Incident Report is out now!
The aviation industry has always been a prime target for cybercriminals due to its critical infrastructure and high stakes. In the first quarter of 2024, the sector faced an alarming surge in cybersecurity threats, revealing its vulnerabilities and the relentless sophistication of cyber attackers.
SOCRadar’s Aviation Industry, Quarterly Incident Report, provides an in-depth analysis of these threats, detected and examined through our extensive monitoring of hacker forums, Telegram channels, and dark web platforms.
What is Master Data Management by PiLog Groupaymanquadri279
PiLog Group's Master Data Record Manager (MDRM) is a sophisticated enterprise solution designed to ensure data accuracy, consistency, and governance across various business functions. MDRM integrates advanced data management technologies to cleanse, classify, and standardize master data, thereby enhancing data quality and operational efficiency.
Zoom is a comprehensive platform designed to connect individuals and teams efficiently. With its user-friendly interface and powerful features, Zoom has become a go-to solution for virtual communication and collaboration. It offers a range of tools, including virtual meetings, team chat, VoIP phone systems, online whiteboards, and AI companions, to streamline workflows and enhance productivity.
Most important New features of Oracle 23c for DBAs and Developers. You can get more idea from my youtube channel video from https://youtu.be/XvL5WtaC20A
Graspan: A Big Data System for Big Code AnalysisAftab Hussain
We built a disk-based parallel graph system, Graspan, that uses a novel edge-pair centric computation model to compute dynamic transitive closures on very large program graphs.
We implement context-sensitive pointer/alias and dataflow analyses on Graspan. An evaluation of these analyses on large codebases such as Linux shows that their Graspan implementations scale to millions of lines of code and are much simpler than their original implementations.
These analyses were used to augment the existing checkers; these augmented checkers found 132 new NULL pointer bugs and 1308 unnecessary NULL tests in Linux 4.4.0-rc5, PostgreSQL 8.3.9, and Apache httpd 2.2.18.
- Accepted in ASPLOS ‘17, Xi’an, China.
- Featured in the tutorial, Systemized Program Analyses: A Big Data Perspective on Static Analysis Scalability, ASPLOS ‘17.
- Invited for presentation at SoCal PLS ‘16.
- Invited for poster presentation at PLDI SRC ‘16.
E-Invoicing Implementation: A Step-by-Step Guide for Saudi Arabian CompaniesQuickdice ERP
Explore the seamless transition to e-invoicing with this comprehensive guide tailored for Saudi Arabian businesses. Navigate the process effortlessly with step-by-step instructions designed to streamline implementation and enhance efficiency.
UI5con 2024 - Boost Your Development Experience with UI5 Tooling ExtensionsPeter Muessig
The UI5 tooling is the development and build tooling of UI5. It is built in a modular and extensible way so that it can be easily extended by your needs. This session will showcase various tooling extensions which can boost your development experience by far so that you can really work offline, transpile your code in your project to use even newer versions of EcmaScript (than 2022 which is supported right now by the UI5 tooling), consume any npm package of your choice in your project, using different kind of proxies, and even stitching UI5 projects during development together to mimic your target environment.
Atelier - Innover avec l’IA Générative et les graphes de connaissancesNeo4j
Atelier - Innover avec l’IA Générative et les graphes de connaissances
Allez au-delà du battage médiatique autour de l’IA et découvrez des techniques pratiques pour utiliser l’IA de manière responsable à travers les données de votre organisation. Explorez comment utiliser les graphes de connaissances pour augmenter la précision, la transparence et la capacité d’explication dans les systèmes d’IA générative. Vous partirez avec une expérience pratique combinant les relations entre les données et les LLM pour apporter du contexte spécifique à votre domaine et améliorer votre raisonnement.
Amenez votre ordinateur portable et nous vous guiderons sur la mise en place de votre propre pile d’IA générative, en vous fournissant des exemples pratiques et codés pour démarrer en quelques minutes.
Using Query Store in Azure PostgreSQL to Understand Query PerformanceGrant Fritchey
Microsoft has added an excellent new extension in PostgreSQL on their Azure Platform. This session, presented at Posette 2024, covers what Query Store is and the types of information you can get out of it.
Introducing Crescat - Event Management Software for Venues, Festivals and Eve...Crescat
Crescat is industry-trusted event management software, built by event professionals for event professionals. Founded in 2017, we have three key products tailored for the live event industry.
Crescat Event for concert promoters and event agencies. Crescat Venue for music venues, conference centers, wedding venues, concert halls and more. And Crescat Festival for festivals, conferences and complex events.
With a wide range of popular features such as event scheduling, shift management, volunteer and crew coordination, artist booking and much more, Crescat is designed for customisation and ease-of-use.
Over 125,000 events have been planned in Crescat and with hundreds of customers of all shapes and sizes, from boutique event agencies through to international concert promoters, Crescat is rigged for success. What's more, we highly value feedback from our users and we are constantly improving our software with updates, new features and improvements.
If you plan events, run a venue or produce festivals and you're looking for ways to make your life easier, then we have a solution for you. Try our software for free or schedule a no-obligation demo with one of our product specialists today at crescat.io
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
UI5con 2024 - Keynote: Latest News about UI5 and it’s EcosystemPeter Muessig
Learn about the latest innovations in and around OpenUI5/SAPUI5: UI5 Tooling, UI5 linter, UI5 Web Components, Web Components Integration, UI5 2.x, UI5 GenAI.
Recording:
https://www.youtube.com/live/MSdGLG2zLy8?si=INxBHTqkwHhxV5Ta&t=0
OpenMetadata Community Meeting - 5th June 2024OpenMetadata
The OpenMetadata Community Meeting was held on June 5th, 2024. In this meeting, we discussed about the data quality capabilities that are integrated with the Incident Manager, providing a complete solution to handle your data observability needs. Watch the end-to-end demo of the data quality features.
* How to run your own data quality framework
* What is the performance impact of running data quality frameworks
* How to run the test cases in your own ETL pipelines
* How the Incident Manager is integrated
* Get notified with alerts when test cases fail
Watch the meeting recording here - https://www.youtube.com/watch?v=UbNOje0kf6E
Transform Your Communication with Cloud-Based IVR SolutionsTheSMSPoint
Discover the power of Cloud-Based IVR Solutions to streamline communication processes. Embrace scalability and cost-efficiency while enhancing customer experiences with features like automated call routing and voice recognition. Accessible from anywhere, these solutions integrate seamlessly with existing systems, providing real-time analytics for continuous improvement. Revolutionize your communication strategy today with Cloud-Based IVR Solutions. Learn more at: https://thesmspoint.com/channel/cloud-telephony
Odoo ERP software
Odoo ERP software, a leading open-source software for Enterprise Resource Planning (ERP) and business management, has recently launched its latest version, Odoo 17 Community Edition. This update introduces a range of new features and enhancements designed to streamline business operations and support growth.
The Odoo Community serves as a cost-free edition within the Odoo suite of ERP systems. Tailored to accommodate the standard needs of business operations, it provides a robust platform suitable for organisations of different sizes and business sectors. Within the Odoo Community Edition, users can access a variety of essential features and services essential for managing day-to-day tasks efficiently.
This blog presents a detailed overview of the features available within the Odoo 17 Community edition, and the differences between Odoo 17 community and enterprise editions, aiming to equip you with the necessary information to make an informed decision about its suitability for your business.
Why Choose Odoo 17 Community & How it differs from Odoo 17 Enterprise Edition
Last Conference 2017: Big Data in a Production Environment: Lessons Learnt
1. Featured Project:
Marina Bay Sands Casino Resort, Singapore
Connecting teams project-wide
Big Data in a production environment:
Lessons Learnt
LAST Conference 2017
Mark Grebler - Aconex
2. CONFIDENTIAL | 2
Featured Project:
Marina Bay Sands Casino Resort, Singapore
Connecting teams project-wide
4. Summary
• What is the Insights project
• Big Data for Data Science
• Big Data in a production, user-facing environment
• Lessons Learnt
• Problems still to solve
6. Highly flexible and customisable data model with low level concepts
=
useful for many types of projects
Aconex has Flexible data
The Insights Project
7. Highly flexible and customisable data model with low level concepts
=
Difficult to produce meaningful customer reports
Flexible Data Needs transformation
The Insights Project
11. But the differences exist
We use the AWS console
to deploy new
infrastructure
I add new hardware by
buying a new box and
connecting it to the
network
Quotes from Data Engineers interviewed
We deploy by copying the
jar file to the cluster
We don’t have any CI, I
just build it on my box
We test by running it over
some data and ensuring it
doesn’t crash
We have some
rudimentary tests
12. What are the differences
Other Big Data Projects
● Internal client
● Simple authentication
● For Data Scientists
● Single environment
○ Sometimes 2 or 3
● Manual infrastructure management
● Sanity testing
● Manual integration
● Manual deployment
● Unrestricted data access
Insights Project
● External client
● Integrated authentication
● For end users
● Multiple environments
○ Due to data sovereignty (10)
● Infrastructure as code
● Unit → end-to-end testing
● Continuous integration
● Single-step deployment
● Data access restrictions
It’s not always so black and white, but the left side represents quite a lot of other projects I’ve seen.
13. Lessons Learnt
● VPN to control data access
● Autoscaling application server
● Network independence
● Zero downtime-deployments with
automatic rollback
○ ElasticBeanstalk provides this
14. Lessons learned: Infrastructure-as-code
● Must be easily reproducible because we need to do it 10+ times
● Automation of infrastructure management
○ Infrastructure is a core part of the Big Data project, so it must be treated as important as our
application code
○ Terraform is used to manage the infrastructure, including:
■ Networking and VPN management
■ Security
■ Provisioning VMs and other infrastructure
■ Replication and ingestion of data from Data Centres
■ Database Administration and Automation
15. Lessons learned: Access segregation
● Different accounts for testing and
production
● Separate VPCs for each environment
● Multiple user roles allows fine-grained
control of access
● VPN used as a further level to restrict
data access
16. Lessons learned: Integration and deployment
Continuous Integration
Once built, versioned artifacts are pushed to s3 buckets
Deployments
Ansible is used to roll out new versions of the
application and transformations
Infrastructure
Terraform controls the base infrastructure
● Deployments run in parallel across environments
● Docker image used for deployments to control
dependencies
17. Lessons Learnt: Automate Testing
● Big Data testing is hard
● Automated unit tests to ensure transformations are correct
○ We pair with our QA to generate the data, and validate the expected output for the unit tests
○ TDD-ish, but often testing done after development
● Automated Integration tests using a large data set
○ To ensure regressions haven’t occurred
● Manual end-to-end sanity tests
○ This should be automated in the future
● Manual exploratory testing
18. Problems to resolve
● Testing
○ Big Data testing is time consuming
■ Particularly around data generation
○ How to effectively automate testing of the infrastructure
○ How to automate end-to-end sanity testing.
● Infrastructure
○ CI/CD with Terraform
○ So many moving parts makes management difficult
● Ingestion and transformations
○ How to move from batch processing to incremental or streaming
○ Removing the database clones
● Effectively communicating to the business what/why we’re doing what we are
○ Why are things so slow?
Editor's Notes
Who's had a house built for them, built their own house, or organised a significant renovation?
How many documents were needed? How many conversations were had?
Think of the number of documents to build a skyscraper, or a refinery, etc.
Looks similar. From the outside, no real differences.