This document proposes using machine learning techniques to analyze logs and surface the most relevant ones. It discusses using both unsupervised and supervised learning. Unsupervised techniques like clustering could analyze large amounts of unlabeled data to group similar logs. Supervised learning would involve acquiring labels to train classifiers on what is relevant versus irrelevant. The proposed solution involves normalizing logs, acquiring labels, training models, and then classifying and enhancing new logs. It suggests this could be done at scale using tools like Spark.
Machine Learning Concepts for Software Monitoring - Lior Redlus, Coralogix - ...DevOpsDays Tel Aviv
"Cloud environments and Open Source software have lowered the bar for anyone to implement software solutions.
Complex relationships between system components are frequently missed by the human eye, and small but important changes are neglected. This, along with the sheer amount of monitoring data, call for a new approach.
"
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start ‘hunting’ for signs of compromises and anomalies in our own environments.
In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company.
Visualization. Data science. No machine learning. But pretty pictures.What is internal threat intelligence? Check out http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225
The Heatmap - Why is Security Visualization so Hard?Raffael Marty
This presentation explores why it is so hard to come up with a security monitoring (or shall we call it security intelligence) approach that helps find sophisticated attackers in all the data collected. It explores the question of how to visualize a billion events. To do so, the presentation dives deeply into heatmaps - matrices - as an example of a simple type of visualization. While these heatmaps are very simple, they are incredibly versatile and help us think about the problem of security visualization. They help illustrate how data mining and user experience design help get a handle of the security visualization challenges - enabling us to gain deep insight for a number of security use-cases.
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start 'hunting' for signs of compromises and anomalies in our own environments.
In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company.
Visualization. Data science. No machine learning. But pretty pictures.
Here is a blog post I wrote a bit ago about the general theme of internal threat intelligence:
http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225?
Join us to see how JReport 12 can help you visualize your Big Data. Get a glimpse of Visual Analysis, an ad hoc tool that enables self-service interactive data analysis powered by JReport in-memory cubes to gain deeper insights into your Big Data. Seamlessly integrate the dashboards you create into your host application -- all through a customized interface, all with JReport 12.
Ensuring security of a company’s data and infrastructure has largely become a data analytics challenge. It is about finding and understanding patterns and behaviors that are indicative of malicious activities or deviations from the norm. Data, Analytics, and Visualization are used to gain insights and discover those malicious activities. These three components play off of each other, but also have their inherent challenges. A few examples will be given to explore and illustrate some of these challenges,
Machine Learning Concepts for Software Monitoring - Lior Redlus, Coralogix - ...DevOpsDays Tel Aviv
"Cloud environments and Open Source software have lowered the bar for anyone to implement software solutions.
Complex relationships between system components are frequently missed by the human eye, and small but important changes are neglected. This, along with the sheer amount of monitoring data, call for a new approach.
"
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start ‘hunting’ for signs of compromises and anomalies in our own environments.
In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company.
Visualization. Data science. No machine learning. But pretty pictures.What is internal threat intelligence? Check out http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225
The Heatmap - Why is Security Visualization so Hard?Raffael Marty
This presentation explores why it is so hard to come up with a security monitoring (or shall we call it security intelligence) approach that helps find sophisticated attackers in all the data collected. It explores the question of how to visualize a billion events. To do so, the presentation dives deeply into heatmaps - matrices - as an example of a simple type of visualization. While these heatmaps are very simple, they are incredibly versatile and help us think about the problem of security visualization. They help illustrate how data mining and user experience design help get a handle of the security visualization challenges - enabling us to gain deep insight for a number of security use-cases.
Creating Your Own Threat Intel Through Hunting & VisualizationRaffael Marty
The security industry is talking a lot about threat intelligence; external information that a company can leverage to understand where potential threats are knocking on the door and might have already perpetrated the network boundaries. Conversations with many CERTs have shown that we have to stop relying on knowledge about how attacks have been conducted in the past and start 'hunting' for signs of compromises and anomalies in our own environments.
In this presentation we explore how the decade old field of security visualization has emerged. We show how we have applied advanced analytics and visualization to create our own threat intelligence and investigated lateral movement in a Fortune 50 company.
Visualization. Data science. No machine learning. But pretty pictures.
Here is a blog post I wrote a bit ago about the general theme of internal threat intelligence:
http://www.darkreading.com/analytics/creating-your-own-threat-intel-through-hunting-and-visualization/a/d-id/1321225?
Join us to see how JReport 12 can help you visualize your Big Data. Get a glimpse of Visual Analysis, an ad hoc tool that enables self-service interactive data analysis powered by JReport in-memory cubes to gain deeper insights into your Big Data. Seamlessly integrate the dashboards you create into your host application -- all through a customized interface, all with JReport 12.
Ensuring security of a company’s data and infrastructure has largely become a data analytics challenge. It is about finding and understanding patterns and behaviors that are indicative of malicious activities or deviations from the norm. Data, Analytics, and Visualization are used to gain insights and discover those malicious activities. These three components play off of each other, but also have their inherent challenges. A few examples will be given to explore and illustrate some of these challenges,
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...Data Con LA
While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes per day of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. This session is especially recommended for data infrastructure engineers and architects planning, building, or maintaining similar systems.
Cloud computing has changed the way businesses operate, the way businesses make money, and the way business have to protect their assets and information. More and more software applications are moving into the cloud. People are running their proxies in the cloud and soon you will be collecting your logs in the cloud. You shouldn't have to deal with log collection and log management. You should be able to focus your time on getting value out of the logs; to do log analysis and visualization.
In this presentation we will explore how we can leverage the cloud to build security visualization tools. We will discuss some common visualization libraries and have a look at how they can be deployed to solve security problems. We will see how easy it is to quickly stand up such an application. To close the presentation, we will look at a number of security visualization examples that show how security data benefits from visual representations. For example, how can network traffic, firewall data, or IDS data be visualized effectively?
The presentation will describe methods for discovering interesting and actionable patterns in log files for security management without specifically knowing what you are looking for. This approach is different from "classic" log analysis and it allows gaining an insight into insider attacks and other advanced intrusions, which are extremely hard to discover with other methods. Specifically, I will demonstrate how data mining can be used as a source of ideas for designing future log analysis techniques, that will help uncover the coming threats. The important part of the presentation will be the demonstration how the above methods worked in a real-life environment.
Large-Scale Machine Learning with Apache SparkDB Tsai
Spark is a new cluster computing engine that is rapidly gaining popularity — with over 150 contributors in the past year, it is one of the most active open source projects in big data, surpassing even Hadoop MapReduce. Spark was designed to both make traditional MapReduce programming easier and to support new types of applications, with one of the earliest focus areas being machine learning. In this talk, we’ll introduce Spark and show how to use it to build fast, end-to-end machine learning workflows. Using Spark’s high-level API, we can process raw data with familiar libraries in Java, Scala or Python (e.g. NumPy) to extract the features for machine learning. Then, using MLlib, its built-in machine learning library, we can run scalable versions of popular algorithms. We’ll also cover upcoming development work including new built-in algorithms and R bindings.
Bio:
Xiangrui Meng is a software engineer at Databricks. He has been actively involved in the development of Spark MLlib since he joined. Before Databricks, he worked as an applied research engineer at LinkedIn, where he was the main developer of an offline machine learning framework in Hadoop MapReduce. His thesis work at Stanford is on randomized algorithms for large-scale linear regression.
A presentation that I gave at the Query Log Analysis: From Research to Best Practice Workshop 27 - 28 May 20098 in London, UK http://ir.shef.ac.uk/cloughie/qlaw2009/index.html
MongoDB: How We Did It – Reanimating Identity at AOLMongoDB
AOL experienced explosive growth and needed a new database that was both flexible and easy to deploy with little effort. They chose MongoDB. Due to the complexity of internal systems and the data, most of the migration process was spent building a new identity platform and adapters for legacy apps to talk to MongoDB. Systems were migrated in 4 phases to ensure that users were not impacted during the switch. Turning on dual reads/writes to both legacy databases and MongoDB also helped get production traffic into MongoDB during the process. Ultimately, the project was successful with the help of MongoDB support. Today, the team has 15 shards, with 60-70 GB per shard.
Delivered at the FISL13 conference in Brazil: http://www.youtube.com/watch?v=K9w2cipqfvc
This talk introduces the USE Method: a simple strategy for performing a complete check of system performance health, identifying common bottlenecks and errors. This methodology can be used early in a performance investigation to quickly identify the most severe system performance issues, and is a methodology the speaker has used successfully for years in both enterprise and cloud computing environments. Checklists have been developed to show how the USE Method can be applied to Solaris/illumos-based and Linux-based systems.
Many hardware and software resource types have been commonly overlooked, including memory and I/O busses, CPU interconnects, and kernel locks. Any of these can become a system bottleneck. The USE Method provides a way to find and identify these.
This approach focuses on the questions to ask of the system, before reaching for the tools. Tools that are ultimately used include all the standard performance tools (vmstat, iostat, top), and more advanced tools, including dynamic tracing (DTrace), and hardware performance counters.
Other performance methodologies are included for comparison: the Problem Statement Method, Workload Characterization Method, and Drill-Down Analysis Method.
Big Data Day LA 2016/ Hadoop/ Spark/ Kafka track - Building an Event-oriented...Data Con LA
While we frequently talk about how to build interesting products on top of machine and event data, the reality is that collecting, organizing, providing access to, and managing this data is where most people get stuck. In this session, we’ll follow the flow of data through an end to end system built to handle tens of terabytes per day of event-oriented data, providing real time streaming, in-memory, SQL, and batch access to this data. We’ll go into detail on how open source systems such as Hadoop, Kafka, Solr, and Impala/Hive are actually stitched together; describe how and where to perform data transformation and aggregation; provide a simple and pragmatic way of managing event metadata; and talk about how applications built on top of this platform get access to data and extend its functionality. This session is especially recommended for data infrastructure engineers and architects planning, building, or maintaining similar systems.
Cloud computing has changed the way businesses operate, the way businesses make money, and the way business have to protect their assets and information. More and more software applications are moving into the cloud. People are running their proxies in the cloud and soon you will be collecting your logs in the cloud. You shouldn't have to deal with log collection and log management. You should be able to focus your time on getting value out of the logs; to do log analysis and visualization.
In this presentation we will explore how we can leverage the cloud to build security visualization tools. We will discuss some common visualization libraries and have a look at how they can be deployed to solve security problems. We will see how easy it is to quickly stand up such an application. To close the presentation, we will look at a number of security visualization examples that show how security data benefits from visual representations. For example, how can network traffic, firewall data, or IDS data be visualized effectively?
The presentation will describe methods for discovering interesting and actionable patterns in log files for security management without specifically knowing what you are looking for. This approach is different from "classic" log analysis and it allows gaining an insight into insider attacks and other advanced intrusions, which are extremely hard to discover with other methods. Specifically, I will demonstrate how data mining can be used as a source of ideas for designing future log analysis techniques, that will help uncover the coming threats. The important part of the presentation will be the demonstration how the above methods worked in a real-life environment.
Large-Scale Machine Learning with Apache SparkDB Tsai
Spark is a new cluster computing engine that is rapidly gaining popularity — with over 150 contributors in the past year, it is one of the most active open source projects in big data, surpassing even Hadoop MapReduce. Spark was designed to both make traditional MapReduce programming easier and to support new types of applications, with one of the earliest focus areas being machine learning. In this talk, we’ll introduce Spark and show how to use it to build fast, end-to-end machine learning workflows. Using Spark’s high-level API, we can process raw data with familiar libraries in Java, Scala or Python (e.g. NumPy) to extract the features for machine learning. Then, using MLlib, its built-in machine learning library, we can run scalable versions of popular algorithms. We’ll also cover upcoming development work including new built-in algorithms and R bindings.
Bio:
Xiangrui Meng is a software engineer at Databricks. He has been actively involved in the development of Spark MLlib since he joined. Before Databricks, he worked as an applied research engineer at LinkedIn, where he was the main developer of an offline machine learning framework in Hadoop MapReduce. His thesis work at Stanford is on randomized algorithms for large-scale linear regression.
A presentation that I gave at the Query Log Analysis: From Research to Best Practice Workshop 27 - 28 May 20098 in London, UK http://ir.shef.ac.uk/cloughie/qlaw2009/index.html
MongoDB: How We Did It – Reanimating Identity at AOLMongoDB
AOL experienced explosive growth and needed a new database that was both flexible and easy to deploy with little effort. They chose MongoDB. Due to the complexity of internal systems and the data, most of the migration process was spent building a new identity platform and adapters for legacy apps to talk to MongoDB. Systems were migrated in 4 phases to ensure that users were not impacted during the switch. Turning on dual reads/writes to both legacy databases and MongoDB also helped get production traffic into MongoDB during the process. Ultimately, the project was successful with the help of MongoDB support. Today, the team has 15 shards, with 60-70 GB per shard.
Delivered at the FISL13 conference in Brazil: http://www.youtube.com/watch?v=K9w2cipqfvc
This talk introduces the USE Method: a simple strategy for performing a complete check of system performance health, identifying common bottlenecks and errors. This methodology can be used early in a performance investigation to quickly identify the most severe system performance issues, and is a methodology the speaker has used successfully for years in both enterprise and cloud computing environments. Checklists have been developed to show how the USE Method can be applied to Solaris/illumos-based and Linux-based systems.
Many hardware and software resource types have been commonly overlooked, including memory and I/O busses, CPU interconnects, and kernel locks. Any of these can become a system bottleneck. The USE Method provides a way to find and identify these.
This approach focuses on the questions to ask of the system, before reaching for the tools. Tools that are ultimately used include all the standard performance tools (vmstat, iostat, top), and more advanced tools, including dynamic tracing (DTrace), and hardware performance counters.
Other performance methodologies are included for comparison: the Problem Statement Method, Workload Characterization Method, and Drill-Down Analysis Method.
A talk about Open Source logging and monitoring tools, using the ELK stack (ElasticSearch, Logstash, Kibana) to aggregate logs, how to track metrics from systems and logs, and how Drupal.org uses the ELK stack to aggregate and process billions of logs a month.
Logging and Exception handling is one of the easiest tools to use when debugging; but how can you take those massive logs, thousands of errors and effortlessly use them to build a better product? This presentation share our developers team's lesson-learned to expedite releases and fix app issues faster. It discuss best practices that will help your dev team build a culture of logging such as: what to log, how to log it, and how to proactively put it to use.
Docker Logging and analysing with Elastic StackJakub Hajek
Collecting logs from the entire stateless environment is challenging parts of the application lifecycle. Correlating business logs with operating system metrics to provide insights is a crucial part of the entire organization. What aspects should be considered while you design your logging solutions?
Docker Logging and analysing with Elastic Stack - Jakub Hajek PROIDEA
Collecting logs from the entire stateless environment is challenging parts of the application lifecycle. Correlating business logs with operating system metrics to provide insights is a crucial part of the entire organization. We will see the technical presentation on how to manage a large amount of the data in a typical environment with microservices.
What's inside the black box? Using ML to tune and manage Kafka. (Matthew Stum...confluent
We use machine learning to delve deep into the internals of how systems like Kafka work. In this talk I'll dive into what variables affect performance and reliability, including previously unknown leading indicators of major performance problems, failure conditions and how to tune for specific use cases. I'll cover some of the specific methodology we use, including Bayesian optimization, and reinforcement learning. I'll also talk about our own internal infrastructure that makes heavy use of Kafka and Kubernetes to deliver real-time predictions to our customers.
This presentation was given to the Dublin Node (JS) Community on May 29th 2014.
Presented by: Chris Lawless, Kevin Yu Wei Xia, Fergal Carroll @phergalkarl, Ciarán Ó hUallacháin, and Aman Kohli @akohli
Speeding Up Atlas Deep Learning Platform with Alluxio + FluidAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Speeding Up Atlas Deep Learning Platform with Alluxio + Fluid
Yuandong Xie, Platform Researcher (Unisound)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Similar to Debugging Skynet: A Machine Learning Approach to Log Analysis - Ianir Ideses, logz.io - DevOpsDays Tel Aviv 2016 (20)
YOUR OPEN SOURCE PROJECT IS LIKE A STARTUP, TREAT IT LIKE ONE, EYAR ZILBERMAN...DevOpsDays Tel Aviv
From idea to execution, the challenges of publishing an open source project are very similar to initializing a startup when it comes to creating a successful product that people will love and use.
Most open source projects are not “taking-off”, although they are really good! This is because developers (which are usually the creators of open source projects) think that writing the code is the hard part and “neglect” the other parts of publishing a good open source project.
In this talk, I will use my experience as a contributor to open source and product head of a startup, to go beyond writing the code itself and cover the other central aspects of creating an open source project, like MVP, product/market fit, marketing and more.
If you have never used GraphQL before, you probably think that it is just another buzzword that will be forgotten in a few years. You might think: “Why do I need to learn a new way to write APIs when REST already answers all my needs?”. Or, you are excited to learn something new but don’t believe GraphQL is mature enough for production.
In this talk, I will remind you of some of the pain points you have probably experienced when using REST. I will then explain what GraphQL is and demonstrate how it solves these pain points. Next, I will discuss the disadvantages of GraphQL. Finally, I will provide some guidelines for choosing between REST and GraphQL. By the end of this talk, you will understand what GraphQL is and when to use it.
MICROSERVICES ABOVE THE CLOUD - DESIGNING THE INTERNATIONAL SPACE STATION FOR...DevOpsDays Tel Aviv
“The International Space Station has been orbiting the Earth for over 20 years. It was not launched fully formed, as a monolith in space. Instead, it is built out of dozens of individual modules, each with a dedicated role - life support, engineering, science, commercial applications and more. Each module (or container) functions as a microservice, adding additional capabilities to the whole. Not only do the modules need to function together, delivering both functional and non-functional capabilities, they were designed, developed and built by different countries on Earth and once launched into space (deployed in multiple different ways), had to work together - perfectly.
Despite the many (minor) reliability issues which have occurred over the decades, the ISS remains a highly reliable platform for cutting edge scientific and engineering research.
In this session I will describe the way the space station was developed and the lessons Site Reliability and DevOps Engineers can learn from it.
THE (IR)RATIONAL INCIDENT RESPONSE: HOW PSYCHOLOGICAL BIASES AFFECT INCIDENT ...DevOpsDays Tel Aviv
Have you ever felt you took every wrong turn possible in the process of mitigating a production incident? Did you go through a 3-hour hell during incident response and felt the incident wasn’t complex enough to justify the horrors you’ve experienced? Did it cause you to question your engineering or problem-solving skills?
Well, it’s only partially you. Our brain is wired to make decision-making simpler. In doing so, it exposes itself to biases, heuristics, and other quirks that may seem like “bad decisions” in hindsight.
In this talk, through real-life outages, we’ll project those psychological principles onto the world of production monitor, and incident management. As a responder, you’ll learn why those behavioral patterns emerge during production incidents and what can be done to limit their effect, and as a manager, you’ll learn how to enable and encourage a healthy environment to better support those patterns.
The word observable entered the English language roughly 400 years ago, but the concepts of what it means to see, comprehend, and understand something have been debated since time immemorial. Starting in the 19th century, a series of postulates and criteria coalesced into control theory, and it is from this body of knowledge that we gained the word “observability”. Today, with the advent of complex, interconnected computer systems, that word has taken on new meanings and connotations—some useful, some detrimental, and some just plain confusing.
In this talk, we’ll mix a little history, a touch of philosophy, and a healthy dose of reality, to demystify what observability means to us as professional computer people. We’ll tear through the marketing material and unearth foundational principles that will help us to build better infrastructure, write better software, and promote healthier business practices. Finally, we’ll explore some potential new avenues for discussion and understanding.
NUDGE AND SLUDGE: DRIVING SECURITY WITH DESIGN // J. WOLFGANG GOERLICH, Duo S...DevOpsDays Tel Aviv
Security people say users are the weakest link. But are they? When complying with security becomes too burdensome, users take shortcuts, find workarounds, and end up jeopardizing security. Blaming users is lazy and easy. Making security usable is time consuming and challenging. How does design research help us understand our customers? What patterns and principles drive secure behavior? How can we build empathy with customers and make the right thing to do the easiest thing to do? This session explores these questions, and provides examples of how design thinking and research can help us be more secure. We will walk through our creation of core user personas, design principles, and how these inform and direct our design choices and intent. Don’t blame your users anymore. Come learn how to be part of a future where usability leads security.
(Ignite) TAKE A HIKE: PREVENTING BATTERY CORROSION - LEAH VOGEL, CHEGGDevOpsDays Tel Aviv
This is for you, you rockstar, ninja coffee drinking workaholic who doesn’t know what a vacation day looks like. Even though you love your job and are dedicated and are super important, you need a break too.
We tend to think that working all the time is an effective practice while the truth is that finding the time for self care and recharging your batteries is beneficial for both you and your company. Additionally, if you’re a leader, you’re responsible for the wellbeing of your team. In this talk I’ll discuss the importance of taking time off of work and creating a positive culture surrounding vacation time.
BUILDING A DR PLAN FOR YOUR CLOUD INFRASTRUCTURE FROM THE GROUND UP, MOSHE BE...DevOpsDays Tel Aviv
This is a story about taking the cloud infrastructure of a successful company, that is still managed as infrastructure of a startup company, and rebuilding it to support the growing business requirements, especially around disaster recovery and business continuity. In the session I will share Next Insurance’s journey - where we started, where we are now and what we learned on the way so far. I will talk about how we managed to build our proven DR plans, and actually execute them in our DR drills. I will also talk about why we decided that the only way to prove your DR plan works is to continue running your business in the DR account and make it your production account, and go on to build your next DR account. If you are a part of a company that is about to embark on a similar journey, this session might equip you with some very useful insights on how to think about such a challenge, and some very useful and practical tips on how to execute it.
THE THREE DISCIPLINES OF CI/CD SECURITY, DANIEL KRIVELEVICH, Cider SecurityDevOpsDays Tel Aviv
CI/CD pipelines are quickly becoming the path of least resistance for would-be attackers into sensitive internal systems, gaining access to critical data, with minimal effort.
In the InfoSec world when we talk about CI/CD security often times this focuses on specific aspects of securing your pipeline - scanning the code, protecting secrets, securely managing code deployments, or even authentication and authorization mechanisms, but we rarely talk about all of these together.
After years of being in the trenches and realizing that the attack surface is growing and the threat landscape becoming more and more complex, it has become increasingly apparent that security teams need to adapt and modify strategies to keep up with the new reality of CI/CD protection, without compromising developer velocity.
In this talk I would like to propose a new way of thinking about CI/CD security - that encompasses the three disciplines that comprise CI/CD security - security in the pipeline, of the pipeline, and around the pipeline. Partial coverage of any or all of these disciplines simply will not cut it with the continuously evolving risk landscape. Security engineers need to address each of these aspects in their entirety to provide the full scope of coverage that modern organizations need, and I will take a deep dive on the challenges each introduce, and the approaches and techniques for mitigating them based on adversarial sec research.
The last two decades have been all about SaaS, with advantages that cannot be overstated. Except SaaS isn’t always an option, nor is it always the right choice: businesses in tightly regulated industries, or where information security is paramount, for example, will not - often can not - consider any software that isn’t under their control. For many software enterprises, this leads to the dreaded inevitability of on-premise deployment.
Fortunately, the situation today is dramatically different to a scant few years ago, let alone a decade or two: the same technologies that enable SaaS have also radically transformed on-prem deployment. Modern tools like Docker, Consul, ELK and Kubernetes - to name a few - can be leveraged to completely transform the experience for both customers and vendors. In this talk we’ll contrast the challenges and advantages of SaaS and on-prem, see how things have evolved in recent history, and see how modern on-prem deployment can be, if not pleasurable, at least relatively painless.
CONFIGURATION MANAGEMENT IN THE CLOUD NATIVE ERA, SHAHAR MINTZ, EggPackDevOpsDays Tel Aviv
Configuration Management is at the core of Ops. It’s the biggest enabler of any compute operation, small and big. In the past decade, we have switched from thinking about the machines we are configuring, to think about the software and services we are controlling. With that change of mindset, so did the tools we are using. Traditional tools like Puppet, chef, salt and Ansible are slowly declining while new tools such as Terraform, Pulumi, Helm and Kustomize are on the rise. In this talk I will try to describe the pain-points and the opportunities of this transformation as well as suggesting a future direction based on tools developed at the big-tech companies (Mainly facebook and google).
SOLVING THE DEVOPS CRISIS, ONE PERSON AT A TIME, CHRISTINA BABITSKI, DeveleapDevOpsDays Tel Aviv
We all know how hard it is to find DevOps engineers, and creating a diverse team despite gender and ethnicity bias? Nearly impossible. At this talk we will show our tools and methods implemented in the Develeap hiring process that overcome this inherited bias.
About 2 years ago we faced a crisis in our DevOps consulting company - the market demand was higher than we could supply. The traditional recruiting process depending on CV and artificial credentials was not working. So we came up with an alternative solution, and since then - we are growing exponentially and diversely. In this talk we will show the practical tools we deployed in order to increase our capacity, and we will show how these tools overcome the inherited bias in the process.
OPTIMIZING PERFORMANCE USING CONTINUOUS PRODUCTION PROFILING ,YONATAN GOLDSCH...DevOpsDays Tel Aviv
Everyone wants observability into their system, but find themselves with too many vendors and tools, each with its own API, SDK, agent and collectors.
With the increasing complexity of modern applications, continuous profiling methods and tools are gaining popularity among the Developer and Engineering communities. In this session, we cover what continuous profiling entails and why you should implement a profiler into your tech stack (if you haven’t done so already). We’ll then bring theory to practice and demonstrate a real-life scenario using gProfiler, a free open-source continuous profiling tool, covering Linux servers on multiple architectures (such as Graviton).
HOW TO SCALE YOUR ONCALL OPERATION, AND SURVIVE TO TELL, ANTON DRUKHDevOpsDays Tel Aviv
“Being oncall sucks. But it doesn’t have to!” We all heard this one before. Why is it though, that oncall still remains the biggest scar for many? What can a modern Engineering org do to rein the oncall dragons, and actually help people grow as professionals as they go oncall?
In this talk, I will present the main reasons why oncall is difficult in modern orgs, and describe ways to mitigate these hardships. The idea is that oncall is often the ‘backroom’ of an org, where all the technical and organizational debt take their toll. Be it unwieldy systems or broken processes between teams, oncall checks all the ‘weak boxes’. Therefore, the only way to win at oncall is to sort out your debts, starting with the organizational ones.
I will dive into the detail of the oncall rotation at Snyk as the org scaled from 1 to 220 people, what worked well about it, and what was less than perfect. I will discuss the decisions made to turn oncall into a building block of the org, and show a path to rein oncall in your organization as well.
Github Copilot and tools that help us code better are cool. But I’m lucky if I spend 90 minutes a day writing code. We really need to optimize the hours we spend reviewing code, updating tickets and tracing where our code is deployed. Learn how I save an hour a day streamlining non-coding tasks.
This talk is unique because 99% of developer productivity tools and hacks are about coding faster, better, smarter. And yet the vast majority of our time is spent doing all of this other stuff. After I started focusing on optimizing the 10 hours I spend every day on non-coding tasks, I found I my productivity went up and my frustration at annoying stuff went way down. I cover how to save time by reducing cognitive load and by cutting menial, non-coding tasks that we have to perform 10-50 times every day. For example:
Bug or hotfix comes through and you want to start working on it right away so you create a branch and start fixing. What you don’t do is create a Jira ticket but then later your boss/PM/CSM yells at your due to lack of visibility. I share how I automated ticket creation in Slack by correlating Github to Jira.
You have 20 minutes until your next meeting and you open a pull request and start a review. But you get pulled away half way through and when you come back the next day you forgot everything and have to start over. Huge waste of time. I share an ML job I wrote that tells me how long the review will take so I can pick PRs that fit the amount of time I have.
You build. You ship it. You own it. Great. But after I merge my code I never know where it actually is. Did the CI job fail? Is it release under feature flag? Did it just go GA to everyone? I share a bot I wrote that personally tells me where my code is in the pipeline after it leaves my hands so I can actually take full ownership without spending tons of time figuring out what code is in what release.
FLYING BLIND - ACCESSIBILITY IN MONITORING, FEU MOUREK, IcingaDevOpsDays Tel Aviv
Do you know what it feels like to navigate as someone who can’t distinguish between green and red - looking at those badges that tell you whether something is broken or a-okay? I’ll give you a quick look into what it feels like with some examples from the monitoring tool Icinga Web 2.
We all tend to forget, that not everyone sees the world like we do. In this talk I’ll be walking you through different views in Icinga Web 2 with side-by-side comparisons for the default views and how different kinds of vision impairments affect those. The talks also features a few suggestions on how to improve colour schemes and making websites and webapps better to navigate with screen readers!
(Ignite) WHAT'S BURNING THROUGH YOUR CLOUD BILL - GIL BAHAT, CIDER SECURITYDevOpsDays Tel Aviv
Recent years have exposed startups to a major plague - cloud overspend. No vaccine appears to exist, plethora of tools and consultants fail to stop the bleeding. And yet, some companies manage to stay safe. What makes them different? Is it the tools? Is it the mindset? Is it developer training?
In this session we will examine the cultural factors involved in sound and responsible financial management in the cloud. We will also look at relevant system design elements and product design elements which enable us to spend wisely while our business runs smoothly.
Following this session, you should be better versed in cost-aware system design and some of the cultural and structural requirements to keeping your cloud bill low.
In every development process there is the question, do we invest enough on quality? Do we need to invest more? Every team knows about the dilemma of how many tests is the right amount of tests we should write. Is 80% test coverage is good enough? Maybe 90%? 100%? Should we invest more time in unit testing? Are we wasting too much time on unit-testing? Should we invest time on a faster rollback mechanism?
WIIFM
“Without data, you’re just another person with an opinion” - W. Edwards Deming
SLO Driven Development is a framework that helps the developers focus on impact and balance of every aspect of the dev process. When working currently with SLI, SLA, SLO and error budget you can learn where to invest in the development process.
Let’s talk about the importance of good SLOs and how they can help us improve our day2day
In this talk, I will share do's and don'ts on how to onboard successfully in a remote or hybrid setup including moving to a leadership role, speaking from my own journey onboarding remotely in the midst of a global pandemic.
I will share the tips that worked for me for successful onboarding, how I was able to be productive, impactful, and make a good impression on others. The key issues as an “onbordee” that I will talk about are how to create relationships, make yourself visible in the company, time management, and more.
Since I started working in Augury over 100 new employees have joined the company. Each month I give a session that is part of their general onboarding process. This became a crucial step due to the fact that we are now a hybrid company and a lot of people are onboarding remotely or in a hybrid setup for the first time in their lives.
I joined the company as a backend developer and a few months into my role, the squad leader position in my squad was up for grabs and I was fortunate enough to grab it :) This is my first official leadership role, which I also needed to onboard into in a hybrid setup. I will share the process that I built for myself on “How to lead”. Also, a word or two on the process we built as a squad on how we work in a hybrid setup, what are we optimizing for when we do meet and how to include new members of the team.
DON'T PANIC: GETTING YOUR INFRASTRUCTURE DRIFT UNDER CONTROL, ERAN BIBI, FireflyDevOpsDays Tel Aviv
In your ever-changing Infrastructure, some changes are intentional while others are not.
Drift is what happens whenever the real-world state of your infrastructure differs from the state defined in your configuration. This can happen for many reasons, sometimes it happens when adding or removing resources, other times when changing resource definitions upon resource termination or failure, and even when changes have been made manually or via other automation tools.
While Terraform itself can detect drifts, in most cases, you will be informed about it too late: just before you are about to deploy new changes to your infrastructure. What’s interesting about Terraform though, is that you can apply changes in two separate and distinct steps of “Planning” and “Applying”. This means that you have full visibility of what Terraform is planning on doing beforehand, and if you are satisfied with the changes, you can choose to apply them.
So how does this work? When something is changed intentionally, it will appear in the source code, and the Terraform plan will not do anything. However, if any part of the infrastructure has been changed manually, Terraform’s plan will identify this, and alert you to the change. In other words, if your IaC drifted from its expected state, then Terraform’s plan will, in fact, detect it.
Applying this simple solution can empower DevOps and developer velocity, with the reassurance and context for unexpected changes in your IaC, in near real-time. This talk will showcase real-world examples, and practical ways to apply this in your production environments while doing so safely and at the pace of your engineering cycles.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
2. The Problem - Overlogging
• Millions of logs per week
• Important logs get lost in the clutter
• Need to surface the relevant logs, deemphasize irrelevant logs
3. Proposed Solution
• A Machine Learning approach
• Can sift through large amounts of data
• Can evolve and react to changes in data
• Requires large amounts of data to be effective
5. Unsupervised Machine Learning
• No labels are needed, just lots of data
• Useful when reducing a large amount of data points to a smaller
cluster subset
6. Unsupervised Machine Learning
"GET /twiki/bin/edit/Main/Double_bounce_sender?topicparent=Main.Confi
"GET /twiki/bin/rdiff/TWiki/NewUserTemplate?rev1=1.3&rev2=1.2 HTTP/1.
"GET /mailman/listinfo/hsdivision HTTP/1.1" 200 6291
"GET /twiki/bin/view/TWiki/WikiSyntax HTTP/1.1" 200 7352
"GET /twiki/bin/view/Main/DCCAndPostFix HTTP/1.1" 200 5253
"GET /twiki/bin/oops/TWiki/AppendixFileSystem?template=oopsmore¶m1=1.
"GET /twiki/bin/view/Main/PeterThoeny HTTP/1.1" 200 4924
"GET /twiki/bin/edit/Main/Header_checks?topicparent=Main.Configuratio
"GET /twiki/bin/attach/Main/OfficeLocations HTTP/1.1" 401 12851
"GET /twiki/bin/view/TWiki/WebTopicEditTemplate HTTP/1.1" 200 3732
"GET /app_dev.php/ HTTP/1.1" 200 6715 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X
10_10_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36"
"GET /bundles/framework/css/body.css HTTP/1.1" 200 6657 "http://my.log-
sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.231
"GET /bundles/framework/css/structure.css HTTP/1.1" 200 1191 "http://my.log-
sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.
"GET /bundles/acmedemo/css/demo.css HTTP/1.1" 200 2204 "http://my.log-
sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311
"GET /bundles/acmedemo/images/welcome-quick-tour.gif HTTP/1.1" 200 4770
"http://my.log-sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3)
AppleWebKit/537.36 (KHTML, like Gecko)
"GET /bundles/acmedemo/images/welcome-demo.gif HTTP/1.1" 200 4053 "http://my.log-
sandbox/app_dev.php/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_3)
AppleWebKit/537.36 (KHTML, like Gecko) Chrom
Nov 20 17:27:55 HANNIBAL MyProgram[13163]: Program started by User 1000
Nov 21 17:27:53 HANNIBAL MyProgram[13163]: Program terminated by User 1000
Nov 21 17:27:58 JANE MyProgram[13163]: Program started by User 555
Nov 23 18:27:53 ARILOU MyProgram[13163]: Program stopped by User 777
7. Supervised Machine Learning
• Learning from labeled examples
• Requires a well defined question:
• Is this email spam?
• Is this object a car?
• Is this log interesting?
• Deployed successfully in many domains, most notable classifiers are
NN, SVM, Bayesian Classifiers
8. Supervised Machine Learning - SVM
• Data elements are arranged in vectors
• Each vector index is assigned a weight in the training phase
• A score is computed by summing up the relevant weights
0.1
0.5
-0.9
0.3
Xconnection error success failure
“Connection failure”: 0.1 + 0.3 = 0.4
“Connection success”: 0.1 - 0.9 = -0.8
9. Log Relevancy
• An ill posed problem
• Relevancy is user specific
• People tend to search for
known issues
• There are also unknown
unknowns
• Labels are potentially
very tedious to acquire
10. Proposed Solution - Labels
• Acquiring labels:
• Implicit/explicit user behavior
• Inter-user similarities
• Public knowledge bases
11. Machine Learning in Practice
• Data is textual, numerical and alphanumerical
• Classifiers that have shown good results:
• Random Forests, resemble flow chart decision making
• Linear SVM
• Both classifiers are easy to interpret in the feature space
12. Machine Learning in Practice
connected: -0.157199772246
to provider: -0.15319903564
connected successfully: -0.15319903564
unable: 0.671539714688
topic: 0.678756599452
error: 0.788508324168
13. Machine Learning in Practice - Modules
• Log normalization
• Label acquisition
• Model training
• Log classification and enhancement
14. Log Normalization
• Lower case, stem, stop words
• Identify common fields (timestamp, severity, etc’)
• Identify variable, functions, class names
• Identify known reserved words
• Cluster logs that share the same prototype
15. Labeler
• Different sources for labels
• CQA sites
• Explicit user interaction
• Implicit user interaction
• Heuristics
16. Log Enhancer
• Use knowledge about log events to add prior data
• Suggest solutions to known problems
• Tag relevant logs for display to the user