Ledion Bitincka from Splunk spoke at the AWS Big Data Meetup in Palo Alto and give an overview of Splunk’s processing pipeline topology and explained their approach to indexing data at scale.
SplunkLive Sydney Scaling and best practice for Splunk on premise and in the ...Splunk
Leverage the Splunk architecture to provide the best possible performance. Whether deploying on premise, in the cloud or on Splunk Cloud, this session will guide you through scenarios that will assist in getting the best from all these options. The agenda also covers how you can plan your searches and reporting to provide the best results for your end users.
What is in All of Those SSTable Files Not Just the Data One but All the Rest ...DataStax
Have you ever wondered what is in all of those SSTable files and how it helps Cassandra find and manage your data? If you go to the Datastax website they will give you a high level explanation of what is in each file. In this talk we will go much deeper explaining each file and walking through a dump of its contents. We will also explore the differences between Cassandra 2.1 and 3.4.
About the Speaker
John Schulz Prinicipal Consultant, The Pythian Group
John has 40 of years experience working with data. Data in files and in Databases from flat files through ISAM to relational databases and most recently NoSQL. For the last 15 he's worked on a variety of Open source technologies including MySQL, PostgreSQL, Cassandra, Riak, Hadoop and Hbase. He has been working with Cassandra since 2010. For the last eighteen months he has been working for The Pythian Group to help their customers improve their existing databases and select new ones.
In this video from the DDN User Group Meeting at SC14, Steve Simms from Indiana University presents: Indiana University's Data Capacitor II.
"The High Performance File Systems unit of UITSResearch Technologies operates two separate high-speed file systems for temporary storage of research data. Both use the open sourceLustre parallel distributed file system running on a version of theLinux operating system: Data Capacitor II (DC2) is a larger, faster replacement for the former Data Capacitor, which was decommissioned January 7, 2014. Like its predecessor, DC2 is a large-capacity, high-throughput, high-bandwidth Lustre-based file system serving all IU campuses. It is mounted on the Big Red II, Karst,Quarry, and Mason research computing systems."
SplunkLive Sydney Scaling and best practice for Splunk on premise and in the ...Splunk
Leverage the Splunk architecture to provide the best possible performance. Whether deploying on premise, in the cloud or on Splunk Cloud, this session will guide you through scenarios that will assist in getting the best from all these options. The agenda also covers how you can plan your searches and reporting to provide the best results for your end users.
What is in All of Those SSTable Files Not Just the Data One but All the Rest ...DataStax
Have you ever wondered what is in all of those SSTable files and how it helps Cassandra find and manage your data? If you go to the Datastax website they will give you a high level explanation of what is in each file. In this talk we will go much deeper explaining each file and walking through a dump of its contents. We will also explore the differences between Cassandra 2.1 and 3.4.
About the Speaker
John Schulz Prinicipal Consultant, The Pythian Group
John has 40 of years experience working with data. Data in files and in Databases from flat files through ISAM to relational databases and most recently NoSQL. For the last 15 he's worked on a variety of Open source technologies including MySQL, PostgreSQL, Cassandra, Riak, Hadoop and Hbase. He has been working with Cassandra since 2010. For the last eighteen months he has been working for The Pythian Group to help their customers improve their existing databases and select new ones.
In this video from the DDN User Group Meeting at SC14, Steve Simms from Indiana University presents: Indiana University's Data Capacitor II.
"The High Performance File Systems unit of UITSResearch Technologies operates two separate high-speed file systems for temporary storage of research data. Both use the open sourceLustre parallel distributed file system running on a version of theLinux operating system: Data Capacitor II (DC2) is a larger, faster replacement for the former Data Capacitor, which was decommissioned January 7, 2014. Like its predecessor, DC2 is a large-capacity, high-throughput, high-bandwidth Lustre-based file system serving all IU campuses. It is mounted on the Big Red II, Karst,Quarry, and Mason research computing systems."
What You Need To Know About The Top Database TrendsDell World
The last 5 years have seen transformative changes in both personal and enterprise technologies. Many of these changes have been driven by or are driving paradigm shifts in database technologies and information systems. These include trends such as engineered systems including Exadata, "Big Data" technologies such as Hadoop ,"NoSQL" databases, SSDs, in-memory and columnar technologies. In this presentation we’ll review these big trends and describe how they are changing the database landscape and influencing the career prospects for database professionals.
Project collaboration between Stanford University and NIST (National Institute of Standards and Technology) to preserve the Stephen M. Cabrinety Collection in the History of Microcomputing, ca. 1975-1995
AWR Difference Reports are very helpful when overall performance information about two different periods needs to be compared. However, if the requirement is to review the trends of performance of a specific query, average length of a particular wait event, or different of a specific statistic over time with a purpose of identification of peaks, the AWR Difference Reports are of little help. This presentation will concentrate on techniques of extracting information from the Automatic Workload Repository to analyze how things change over time, which is useful for both - forecasting and identification of specific time periods when issues affect specific areas of the database.
Managing your Black Friday Logs NDC OsloDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
Managing your black friday logs - Code EuropeDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
One of the great challenges of of monitoring any large cluster is how much data to collect and how often to collect it. Those responsible for managing the cloud infrastructure want to see everything collected centrally which places limits on how much and how often. Developers on the other hand want to see as much detail as they can at as high a frequency as reasonable without impacting the overall cloud performance.
To address what seems to be conflicting requirements, we've chosen a hybrid model at HP. Like many others, we have a centralized monitoring system that records a set of key system metrics for all servers at the granularity of 1 minute, but at the same time we do fine-grained local monitoring on each server of hundreds of metrics every second so when there are problems that need more details than are available centrally, one can go to the servers in question to see exactly what was going on at any specific time.
The tool of choice for this fine-grained monitoring is the open source tool collectl, which additionally has an extensible api. It is through this api that we've developed a swift monitoring capability to not only capture the number of gets, put, etc every second, but using collectl's colmux utility, we can also display these in a top-like formact to see exactly what all the object and/or proxy servers are doing in real-time.
We've also developer a second cability that allows one to see what the Virtual Machines are doing on each compute node in terms of CPU, disk and network traffic. This data can also be displayed in real-time with colmux.
This talk will briefly introduce the audience to collectl's capabilities but more importantly show how it's used to augment any existing centralized monitoring infrastructure.
Speakers
Mark Seger
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018Codemotion
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Strata London 16: sightseeing, venues, and friendsNatalino Busa
Which venues have similar visiting patterns? How can we detect when a user is on vacation? Can we predict which venues will be favorited by users by examining their friends' preferences? Natalino Busa explains how these predictive analytics tasks can be accomplished by using Spark SQL, Spark ML, and just a few lines of Scala code.
Flink Forward San Francisco 2018: Stefan Richter - "How to build a modern str...Flink Forward
Stream Processing has evolved quickly in a short time: a few years ago, stream processing was mostly simple real-time aggregations with limited throughput and consistency. Today, many stream processing applications have complex logic, strict correctness guarantees, high performance, low latency, and maintain large state without databases. Since then, Stream processing has become much more sophisticated because the stream processors – the systems that run the application code, coordinate the distributed execution, route the data streams, and ensure correctness in the face of failures and crashes – have become much more technologically advanced. In this talk, we walk through some of the techniques and innovations behind Apache Flink, one of the most powerful open source stream processors. In particular, we plan to discuss: The evolution of fault tolerance in stream processing, Flink’s approach of distributed asynchronous snapshots, and how that approach looks today after multiple years of collaborative work with users running large scale stream processing deployments. How Flink supports applications with terabytes of state and offers efficient snapshots, fast recovery, rescaling, and high throughput. How to build end-to-end consistency (exactly-once semantics) and transactional integration with other systems. How batch and streaming can both run on the same execution model with best-in-class performance.
Application-engaged Dynamic Orchestration of Optical Network ResourcesTal Lavian Ph.D.
Grids & C. freed us from the cuffs of bit-blasting races
Apps such as Grids call for a complex mix of:
Bit-blasting
Finesse (granularity of control)
Virtualization (access to diverse knobs)
Resource bundling (network AND …)
Security (AAA to start)
Free from GUIs, any human intervention
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
More Related Content
Similar to Splunk talk at the AWS Big Data Meetup in Palo Alto on Nov 17 2015
What You Need To Know About The Top Database TrendsDell World
The last 5 years have seen transformative changes in both personal and enterprise technologies. Many of these changes have been driven by or are driving paradigm shifts in database technologies and information systems. These include trends such as engineered systems including Exadata, "Big Data" technologies such as Hadoop ,"NoSQL" databases, SSDs, in-memory and columnar technologies. In this presentation we’ll review these big trends and describe how they are changing the database landscape and influencing the career prospects for database professionals.
Project collaboration between Stanford University and NIST (National Institute of Standards and Technology) to preserve the Stephen M. Cabrinety Collection in the History of Microcomputing, ca. 1975-1995
AWR Difference Reports are very helpful when overall performance information about two different periods needs to be compared. However, if the requirement is to review the trends of performance of a specific query, average length of a particular wait event, or different of a specific statistic over time with a purpose of identification of peaks, the AWR Difference Reports are of little help. This presentation will concentrate on techniques of extracting information from the Automatic Workload Repository to analyze how things change over time, which is useful for both - forecasting and identification of specific time periods when issues affect specific areas of the database.
Managing your Black Friday Logs NDC OsloDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
Managing your black friday logs - Code EuropeDavid Pilato
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Topics include:
* monitoring architectures
* optimal bulk size
* distributing the load
* index and shard size
* optimizing disk IO
Takeaway: best practices when building a monitoring system with the Elastic Stack, advanced tuning to optimize and increase event ingestion performance.
One of the great challenges of of monitoring any large cluster is how much data to collect and how often to collect it. Those responsible for managing the cloud infrastructure want to see everything collected centrally which places limits on how much and how often. Developers on the other hand want to see as much detail as they can at as high a frequency as reasonable without impacting the overall cloud performance.
To address what seems to be conflicting requirements, we've chosen a hybrid model at HP. Like many others, we have a centralized monitoring system that records a set of key system metrics for all servers at the granularity of 1 minute, but at the same time we do fine-grained local monitoring on each server of hundreds of metrics every second so when there are problems that need more details than are available centrally, one can go to the servers in question to see exactly what was going on at any specific time.
The tool of choice for this fine-grained monitoring is the open source tool collectl, which additionally has an extensible api. It is through this api that we've developed a swift monitoring capability to not only capture the number of gets, put, etc every second, but using collectl's colmux utility, we can also display these in a top-like formact to see exactly what all the object and/or proxy servers are doing in real-time.
We've also developer a second cability that allows one to see what the Virtual Machines are doing on each compute node in terms of CPU, disk and network traffic. This data can also be displayed in real-time with colmux.
This talk will briefly introduce the audience to collectl's capabilities but more importantly show how it's used to augment any existing centralized monitoring infrastructure.
Speakers
Mark Seger
Managing your Black Friday Logs - Antonio Bonuccelli - Codemotion Rome 2018Codemotion
Monitoring an entire application is not a simple task, but with the right tools it is not a hard task either. However, events like Black Friday can push your application to the limit, and even cause crashes. As the system is stressed, it generates a lot more logs, which may crash the monitoring system as well. In this talk I will walk through the best practices when using the Elastic Stack to centralize and monitor your logs. I will also share some tricks to help you with the huge increase of traffic typical in Black Fridays.
Strata London 16: sightseeing, venues, and friendsNatalino Busa
Which venues have similar visiting patterns? How can we detect when a user is on vacation? Can we predict which venues will be favorited by users by examining their friends' preferences? Natalino Busa explains how these predictive analytics tasks can be accomplished by using Spark SQL, Spark ML, and just a few lines of Scala code.
Flink Forward San Francisco 2018: Stefan Richter - "How to build a modern str...Flink Forward
Stream Processing has evolved quickly in a short time: a few years ago, stream processing was mostly simple real-time aggregations with limited throughput and consistency. Today, many stream processing applications have complex logic, strict correctness guarantees, high performance, low latency, and maintain large state without databases. Since then, Stream processing has become much more sophisticated because the stream processors – the systems that run the application code, coordinate the distributed execution, route the data streams, and ensure correctness in the face of failures and crashes – have become much more technologically advanced. In this talk, we walk through some of the techniques and innovations behind Apache Flink, one of the most powerful open source stream processors. In particular, we plan to discuss: The evolution of fault tolerance in stream processing, Flink’s approach of distributed asynchronous snapshots, and how that approach looks today after multiple years of collaborative work with users running large scale stream processing deployments. How Flink supports applications with terabytes of state and offers efficient snapshots, fast recovery, rescaling, and high throughput. How to build end-to-end consistency (exactly-once semantics) and transactional integration with other systems. How batch and streaming can both run on the same execution model with best-in-class performance.
Application-engaged Dynamic Orchestration of Optical Network ResourcesTal Lavian Ph.D.
Grids & C. freed us from the cuffs of bit-blasting races
Apps such as Grids call for a complex mix of:
Bit-blasting
Finesse (granularity of control)
Virtualization (access to diverse knobs)
Resource bundling (network AND …)
Security (AAA to start)
Free from GUIs, any human intervention
Similar to Splunk talk at the AWS Big Data Meetup in Palo Alto on Nov 17 2015 (20)
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
How world-class product teams are winning in the AI era by CEO and Founder, P...
Splunk talk at the AWS Big Data Meetup in Palo Alto on Nov 17 2015
1. Data
Through
Splunk
1
Ledion
Bi6ncka
(ledion@splunk.com)
Alex
Batsakis
(abatsakis@splunk.com)
Architects
2. Spelunking:
Splunking:
to
explore
underground
caves
to
explore
machine
data
Splunk
Make
machine
data
accessible,
usable
and
valuable
to
everyone.
3. What
Does
Machine
Data
Look
Like?
3
Sources
Twi2er
Care
IVR
Middleware
Error
Order
Processing
4. Machine
Data
Contains
Cri6cal
Insights
4
Customer
ID
Order
ID
Customer’s
Tweet
Time
Wai6ng
On
Hold
TwiMer
ID
Product
ID
Company’s
TwiMer
ID
Sources
Twi2er
Care
IVR
Middleware
Error
Order
Processing
Customer
ID
Order
ID
Customer
ID
5. Machine
Data
Contains
Cri6cal
Insights
5
Order
ID
Customer’s
Tweet
Time
Wai6ng
On
Hold
Product
ID
Company’s
TwiMer
ID
Sources
Twi2er
Care
IVR
Middleware
Error
Order
Processing
Order
ID
Customer
ID
TwiMer
ID
Customer
ID
Customer
ID
6. Web
Services
Search,
Inves6gate
and
Explore
Your
Data
6
Find
and
fix
issues
and
incidents
drama6cally
faster
across
your
organiza6on
Energy
Manufacturing
Shipping
RFID
Web
Services
Developers
App
Support
Telecoms
Networking
Desktops
Servers
Security
Databases/
DWH
Storage
Messaging
Online
Shopping
Carts
Clickstream
GPS/Cellular
Social
Media
7. Search
and
Inves6gate
Proac6ve
Monitoring
and
Aler6ng
Opera6onal
Visibility
Real-‐6me
Business
Insight
Turning
Machine
Data
into
Opera6onal
Intelligence
7
Proac6ve
Reac6ve
9. Massive
Linear
Scalability
to
100s
of
TBs/Day
9
Auto
load-‐balanced
forwarding
to
as
many
Splunk
Indexers
as
you
need
to
index
TB/day
Offload
search
load
to
Splunk
Search
Heads
11. Consider
this
chunk
of
data
from
a
log
file:
/var/log/secure.log
...
2013/07/01T14:30:24.234-‐0400
Brian
pretends
to
be
from
South
Africa
2013/07/01T14:31:24.234-‐0400
Sean
is
originally
Canadian
2013/07/01T14:30:50.234-‐0400
Brian
spends
his
time
in:
-‐
Kentucky
with
phone
number
345.567.3456
-‐
New
Jersey
2013/07/01T14:32:24.234-‐0400
Matty
has
lived
in
the
following
cities:
-‐
Tijuana:
345
Main
St.
-‐
Saskatchewan:
3
One
Lane
-‐
Colombia:
567
White
line
Dr.
Bogota
2013/07/01T14:33:24.234-‐0400
Cesar
prefers
Burbon
Manhattans
over
beer
2013/07/01T14:33:24.234-‐0400
Matty
loves
GiGi
Mellow
Burgers
2013/07/01T14:33:24.234-‐0400
Sean
is
not
the
only
one
to
not
like
them
...
11
12. Host
my_host
Index
my_index
_raw
2013/07/01T14:30:24.234-‐0400
Brian
pretends
to
be
from
South
Africa
2013/07/01T14:31:24.234-‐0400
Sean
is
originally
Canadian
2013/07/01T14:30:50.234-‐0400
Brian
spends
his
time
in:
...
UTF-‐8
Line
Broken
_conf
<key
here>
Pipeline
Data
13. Pipelines/Processors
Parsing
Queue
Agg
Queue
Typing
Queue
Index
Queue
uk8
header
aggregator
regex
replacement
annotator
tcp
out
syslog
out
indexer
Parsing
Pipeline
Merging
Pipeline
Typing
Pipeline
Index
Pipeline
linebreaker
TCP/UDP
pipeline
Tailing
FIFO
pipeline
FSChange
Exec
pipeline
14. Queue
pData
pData
pData
pData
Queue
Thread
Thread
Process
Process
Remove
Insert
ü Queue
size
bounded
by
memory
ü Variable
size
Pipeline
Data
15. Persistent
Queue
Splunk
Host
Internal
Queues
Full
pData
pData
Tcpout
Q
Input
Q
Persistent
Q
A
Full
Network
Much
Bigger
Queue
Network
16. Indexing
Parsing
Queue
Agg
Queue
Typing
Queue
Index
Queue
uk8
header
aggregator
regex
replacement
annotator
tcp
out
syslog
out
indexer
Parsing
Pipeline
Merging
Pipeline
Typing
Pipeline
Index
Pipeline
linebreaker
TCP/UDP
pipeline
Tailing
FIFO
pipeline
FSChange
Exec
pipeline
17. What’s
an
index
Collec6ve
term
used
to
describe
rawdata
and
associated
tsidx
&
metadata
files.
17
18. Inside
an
index
18
[09:31:39]
[1065]::
lbi6ncka@lbi6ncka:
/opt/splunk/var/lib/splunk/_internaldb/
$
ls
-‐l
total
0
drwx-‐-‐-‐-‐-‐-‐
2
lbi6ncka
admin
68
Feb
6
12:57
colddb
drwx-‐-‐-‐-‐-‐-‐
17
lbi6ncka
admin
578
Jul
1
09:31
db
drwx-‐-‐-‐-‐-‐-‐
13
lbi6ncka
admin
442
Jun
27
16:36
summary
drwx-‐-‐-‐-‐-‐-‐
2
lbi6ncka
admin
68
Aug
24
2012
thaweddb
Index
name
Bucket
loca6ons
20. Inside
a
bucket
20
[10:31:32]
[1092]::
lbi6ncka@lbi6ncka:
/opt/splunk/var/lib/splunk/_internaldb/db/db_1371998025_1371214200_158/
$
ll
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
27M
Jun
21
16:49
1371847782-‐1371214200-‐1941140693112088843.tsidx
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
7.1M
Jun
26
12:43
1371998025-‐1371847783-‐907852835360656754.tsidx
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
2.5M
Jun
26
12:43
merged_lexicon.lex
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
459K
Jun
26
12:43
bloomfilter
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
1.3K
Jun
23
10:33
Sources.data
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
615B
Jun
23
10:33
SourceTypes.data
drwx-‐-‐-‐-‐-‐-‐
17
lbi6ncka
admin
578B
Jul
1
10:31
..
drwx-‐-‐x-‐-‐x
16
lbi6ncka
admin
544B
Jun
26
12:50
.
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
451B
Jun
23
10:31
Strings.data
drwx-‐-‐-‐-‐-‐-‐
4
lbi6ncka
admin
136B
Jun
26
12:42
rawdata
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
116B
Jun
23
10:33
Hosts.data
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
76B
Jun
23
10:33
splunk-‐autogen-‐params.dat
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
52B
Jun
26
12:50
bucket_info.csv
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
49B
Jun
26
12:43
op6mize.result
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
10B
Jun
26
12:43
.rawSize
-‐rw-‐-‐-‐-‐-‐-‐-‐
1
lbi6ncka
admin
8B
Jun
26
12:43
.sizeManifest4.1
21. Metadata
&
Bloomfilters
*.data
– metadata
about
sources,
sourcetypes
and
hosts
of
the
events
contained
in
each
bucket
Bloomfilters
– Efficient
data
structure
that
authorita6vely
rules
out
buckets
ê i.e.
tells
you
with
100%
certainty
that
a
querying
term
is
NOT
in
present
in
a
bucket
– By
default
consulted
by
every
search
21
22. Rawdata
(not
raw
data)
Collec6on
of
compressed
(gzipped)
blocks,
called
slices,
– Concatenated
together
in
a
rawdata/journal.gz
– Think
”cat
chunkA.gz
chunkB.gz
...chunkN.gz
>
journal.gz”).
Slices
contain
the
actual
raw
events.
Pool
of
concatenated
slices
allows
be
seeked
into
– Loca6ons
offsets
are
pointed
to
by
the
values
array
pointers
in
tsidx.
Such
organiza6on
allows
us
to
zoom
in
to
the
right
slice
– reduces
the
amount
of
decompression
6me
&
volume
compared
to
having
a
single,
massive
rawdata
file.
22
23. TSIDX
Time
series
index
(Inverted
index
op6mized
for
6me)
Lexicon:
– Keywords
within
the
specified
6me
range
– Pos6ngs
list
array
Values
array:
– Structure
that
contains
pos6ng
values,
seek
address,
_6me
etc.
– Seek
address
points
to
offsets
in
rawdata
Time
is
of
transcendent
importance
in
Splunk,
– tsidx
filenames
expose
et
and
lt
– Values
arrays
arranged
in
6me
order
as
well
23
24. Lexicon
24
2013/07/01T14:30:24.234-‐0400
Brian
pretends
to
be
from
South
Africa
2013/07/01T14:31:24.234-‐0400
Sean
is
originally
Canadian
2013/07/01T14:30:50.234-‐0400
Brian
spends
his
time
in:
-‐
Kentucky
with
phone
number
345.567.3456
-‐
New
Jersey
2013/07/01T14:32:24.234-‐0400
Matty
has
lived
in
the
following
cities:
-‐
Tijuana:
345
Main
St.
-‐
Saskatchewan:
3
One
Lane
-‐
Colombia:
567
White
line
Dr.
Bogota
2013/07/01T14:33:24.234-‐0400
Cesar
prefers
Burbon
Manhattans
over
beer
2013/07/01T14:33:24.234-‐0400
Matty
loves
GiGi
Mellow
Burgers
2013/07/01T14:33:24.234-‐0400
Sean
is
not
the
only
one
to
not
like
them
Term
Posbng
List
3
4
345
3,4
…
…
Africa
0
Brian
0,2
Bogota
4
…
…
MaMy
5,6
Tijuana
4
25. Values
Array
25
2013/07/01T14:30:24.234-‐0400
Brian
pretends
to
be
from
South
Africa
2013/07/01T14:31:24.234-‐0400
Sean
is
originally
Canadian
2013/07/01T14:30:50.234-‐0400
Brian
spends
his
time
in:
-‐
Kentucky
with
phone
number
345.567.3456
-‐
New
Jersey
2013/07/01T14:32:24.234-‐0400
Matty
has
lived
in
the
following
cities:
-‐
Tijuana:
345
Main
St.
-‐
Saskatchewan:
3
One
Lane
-‐
Colombia:
567
White
line
Dr.
Bogota
2013/07/01T14:33:24.234-‐0400
Cesar
prefers
Burbon
Manhattans
over
beer
2013/07/01T14:33:24.234-‐0400
Matty
loves
GiGi
Mellow
Burgers
2013/07/01T14:33:24.234-‐0400
Sean
is
not
the
only
one
to
not
like
them
Posbng
Seek
addr
_bme
host
…
0
130
1372689024
my_host
…
1
150
1372689084
my_host
…
2
190
1372689050
my_host
…
3
389
1372689050
my_host
…
4
589
1372689050
my_host
…
5
800
1372689050
my_host
…
6
1399
1372689050
my_host
…
…
…
…
…
*all
values
for
illustra6on
purposes.
Not
necessarily
accurate
26. Tsidx
merging
Many
small
tsidx
files
due
to
data
streaming
Searching
is
inefficient
when
going
against
many
tsidx
files
splunk-‐op6mize
– Merging
of
small
tsidx
files
into
a
larger
ones
– Consolida6on
of
lexicons
and
pos6ng
list
26
27. Puzng
it
together
27
IDX
1
IDX
2
IDX
3
Cold
Path
Thawed
Path
Rawdata
TSIDX
hot_v1_100
hot_v1_101
db_lt_et_80
db_lt_et_101
*.data
*.tsidx
rawdata
db_lt_et_70
apple
beer
LEXICON
POSTING
“apple
pie
and
ice
cream
is
delicious”
“an
apple
a
day
keeps
doctor
away”
150
100
et
et
lt
lt
it
it
apple
beer
coke
ice
java
…
Home
Path
Source/Sourcetype/Host
Metadata
1
source
:
:
/my/log
2
source:
:
/blah
cream
28. Bucket
Lifecycle
28
Events
[Too
Many
Warms]
[Hot
Bucket
is
Full]
[Out
of
Space
or
Bucket
is
Old]
[Explicit
User
Ac6on]
$
Thawed
Path
$
Home
Path
$
Cold
Path
[Cheaper
Storage]
$
Frozen
Path
or
Deleted
29. How
do
we
search?
Consult
the
lexicon
and
combine
the
pos6ng
lists
– brian
OR
tijuana
=>
(0,
2)
OR
(4)
=
(0,
2,
4)
Use
values
array
to
get
seek
address,
_6me,
source
and
sourcetype
for
(0,
2,
4)
Use
the
seek
addresses
to
read
rawdata
in
offset
(130,
150,
190)
Send
“results”
to
the
search
29
30. Search
Model
Example
sourcetype=syslog ERROR | top user | fields - percent
Fetch
events
from
disk,
apply
schema
Summarize
into
table
of
top
10
users
Remove
column
showing
percentage
Intermediate
results table
Intermediate
results table
Final results
table
Disk
31. What
can
we
do
with
events?
It’s
not
just
search
…
SPL
=
Search
Processing
Language
– Inspired
by
*nix
pipes
– Schema
on
read
– 130+
search
commands
for
slicing
thru
data
Versa6le
visualiza6on
library
Scheduling
and
aler6ng
…
31
32. LOB
Owners/
Execu6ves
System
Administrator
Opera6ons
Teams
Security
Analysts
IT
Execu6ves
Applica6on
Developers
Auditors
Website/Business
Analysts
Customer
Support
32
IT
Opera6ons
Management
Web
Intelligence
Business
Analy6cs
Applica6on
Management
Security
and
Compliance
33. Take
it
for
a
spin
…
hMp://www.splunk.com/download/
-‐ Download
-‐ Try
Splunk
Cloud
–
AWS
WE’RE
HIRING
!!
(in
SF
&
valley)