LSM Trees

•Download as PPTX, PDF•

19 likes•6,874 views

The document discusses LSM-trees, which are data structures used in Cassandra and other databases. An LSM-tree improves write performance by storing data in an in-memory tree and writing large batches to disk. It supports fast reads by merging data from memory and disk. The document provides examples of how Cassandra uses LSM-trees to handle writes, reads, and compaction of SSTables from memory to disk.

Technology

LSM-Trees
Chris Lohfink
Email: clohfink@blackbirdit.com
Twitter: cnlohfink

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
● Senior Engineer at Pythian (recently acquired Blackbird)
o Lead of Cassandra practice
● Experience
o Worked with Cassandra since 0.8 (2011)
o Java/Python developer
About Me
Chris Lohfink
1

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
• Fault tolerance
• Multiple DCs
• Large data sets
• Data model gotchas
• Unstable releases
• Bootstrapping
• Repairs
• Unit tests
- My Opinions -
2
Cassandra

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
● Can be good, bad, or ugly
● Same as with everything else
Where is Performance?
3

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
Lies, Dirty Lies, & Benchmarks
4

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra 5

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra 6

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
● An LSM-tree is composed of two or more tree-like components, each optimized
for their type of storage
o A small in-memory tree
o One or more on disk trees
● Used in Cassandra, HBase, LevelDB, Google Big Table, SQLite4 & more
Log Structured Merge Tree
LSM-Tree
7

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
Log Structured Merge Tree
One tree in memory, one or more on disk
Memory Disk
ReadsWrites
8

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
Log Structured Merge Tree
Match Cassandra nomenclature
Memtable SSTable
ReadsWrites
9

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
PAGE© Pythian Cassandra
• Very Fast Writes
– Cost of IO amortized across large batches
– Append only
• Comparable Reads
– Comparable best-case
– Potential for very slow
Log Structured Merge Tree
10
Characteristics

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Write Path
Represent numbers of shovels in stock
Memtable SSTables
Temp -20
Shovels 0
State MN
11

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Write Path
Decide we need to include humidity
Memtable SSTables
Temp -20
Shovels 0
State MN
Humidity 10
12

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Write Path
Its Minnesota so the Temperature changed
Memtable SSTables
Temp -20
Shovels 0
State MN
Humidity 10
Temp 80
13

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Write Path
Shovel demand drops so its time to put some in stock again
Memtable SSTables
Temp -20
Shovels 0
State MN
Humidity 10
Temp 80
Shovels 25
14

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Write Path
Reached memtable limit, flush to disk
Memtable SSTables
Temp -20
Shovels 0
State MN
Humidity 10
Temp 80
Shovels 25
Humidity 10
Temp 80
Shovels 25
15

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Write Path
We still in Minnesota and its been a few days so adjust Temp
Memtable SSTables
Temp -20
Shovels 0
State MN
Humidity 10
Temp 80
Shovels 25
Temp 10
16

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Read Path
Check the current temp, take most recent from all SSTables
and the Memtable
Memtable SSTables
Temp -20
Shovels 0
State MN
Humidity 10
Temp 80
Shovels 25
Temp 10
17

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Compaction
Create new SSTable thats the merged result
Memtable SSTables
Temp -20
Shovels 0
State MN
Humidity 10
Temp 80
Shovels 25
Temp 10 Humidity 10
Shovels 25
Temp 80
State MN
18

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Compaction
Delete old SSTables
Memtable SSTables
Temp 10 Humidity 10
Shovels 25
Temp 80
State MN
19

www.pythian.com
twitter: cnlohfink | e-mail: clohfink@blackbirdit.com
Pythian PAGE© Pythian© Pythian Cassandra
Questions?
20

Netflix’s playback data records every user interaction with video on the service, from trailers on the home page to full-length movies. This is a critical dataset with high volume that is used broadly across Netflix, powering product experiences, AB test metrics, and offline insights. In processing playback data, we depend heavily on event-time partitioning to handle a long tail of late arriving events. In this talk, I’ll provide an overview of our recent implementation of generic event-time partitioning on high volume streams using Apache Flink and Apache Iceberg (Incubating). Built as configurable Flink components that leverage Iceberg as a new output table format, we are now able to write playback data and other large scale datasets directly from a stream into a table partitioned on event time, replacing the common pattern of relying on a post-processing batch job that “puts the data in the right place”. We’ll talk through what it took to apply this to our playback data in practice, as well as challenges we hit along the way and tradeoffs with a streaming approach to event-time partitioning.

Using Kafka and Kudu for fast, low-latency SQL analytics on streaming data

Mike Percy

The Future of Data Science and Machine Learning at Scale: A Look at MLflow, D...

Databricks

Many had dubbed 2020 as the decade of data. This is indeed an era of data zeitgeist. From code-centric software development 1.0, we are entering software development 2.0, a data-centric and data-driven approach, where data plays a central theme in our everyday lives. As the volume and variety of data garnered from myriad data sources continue to grow at an astronomical scale and as cloud computing offers cheap computing and data storage resources at scale, the data platforms have to match in their abilities to process, analyze, and visualize at scale and speed and with ease — this involves data paradigm shifts in processing and storing and in providing programming frameworks to developers to access and work with these data platforms. In this talk, we will survey some emerging technologies that address the challenges of data at scale, how these tools help data scientists and machine learning developers with their data tasks, why they scale, and how they facilitate the future data scientists to start quickly. In particular, we will examine in detail two open-source tools MLflow (for machine learning life cycle development) and Delta Lake (for reliable storage for structured and unstructured data). Other emerging tools such as Koalas help data scientists to do exploratory data analysis at scale in a language and framework they are familiar with as well as emerging data + AI trends in 2021. You will understand the challenges of machine learning model development at scale, why you need reliable and scalable storage, and what other open source tools are at your disposal to do data science and machine learning at scale.

Efficient Data Storage for Analytics with Apache Parquet 2.0Cloudera, Inc.

Trino: A Ludicrously Fast Query Engine - Pulsar Summit NA 2021

StreamNative

You may be familiar with the Presto plugin used to run fast interactive queries over Pulsar using ANSI SQL and can be joined with other data sources. This plugin will soon get a rename to align with the rename of the PrestoSQL project to Trino. What is the purpose of this rename and what does it mean for those using the Presto plugin? We cover the history of the community shift from PrestoDB to PrestoSQL, as well as, the future plans for the Pulsar community to donate this plugin to the Trino project. One of the connector maintainers will then demo the connector and show what is possible when using Trino and Pulsar!

Introduction to memcached

Jurriaan Persyn

Simplifying Real-Time Architectures for IoT with Apache Kudu

Cloudera, Inc.

Enterprise data is moving into Hadoop, but some data has to stay in operational systems. Apache Calcite (the technology behind Hive’s new cost-based optimizer, formerly known as Optiq) is a query-optimization and data federation technology that allows you to combine data in Hadoop with data in NoSQL systems such as MongoDB and Splunk, and access it all via SQL. Hyde shows how to quickly build a SQL interface to a NoSQL system using Calcite. He shows how to add rules and operators to Calcite to push down processing to the source system, and how to automatically build materialized data sets in memory for blazing-fast interactive analysis.

ksqlDB: A Stream-Relational Database System

confluent

Speaker: Matthias J. Sax, Software Engineer, Confluent ksqlDB is a distributed event streaming database system that allows users to express SQL queries over relational tables and event streams. The project was released by Confluent in 2017 and is hosted on Github and developed with an open-source spirit. ksqlDB is built on top of Apache Kafka®, a distributed event streaming platform. In this talk, we discuss ksqlDB’s architecture that is influenced by Apache Kafka and its stream processing library, Kafka Streams. We explain how ksqlDB executes continuous queries while achieving fault tolerance and high vailability. Furthermore, we explore ksqlDB’s streaming SQL dialect and the different types of supported queries. Matthias J. Sax is a software engineer at Confluent working on ksqlDB. He mainly contributes to Kafka Streams, Apache Kafka's stream processing library, which serves as ksqlDB's execution engine. Furthermore, he helps evolve ksqlDB's "streaming SQL" language. In the past, Matthias also contributed to Apache Flink and Apache Storm and he is an Apache committer and PMC member. Matthias holds a Ph.D. from Humboldt University of Berlin, where he studied distributed data stream processing systems. https://db.cs.cmu.edu/events/quarantine-db-talk-2020-confluent-ksqldb-a-stream-relational-database-system/

Presto Summit 2018 - 09 - Netflix Iceberg

kbajda

Druid deep dive

Kashif Khan

Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...

GetInData

Did you like it? Check out our E-book: Apache NiFi - A Complete Guide https://ebook.getindata.com/apache-nifi-complete-guide Apache NiFi is one of the most popular services for running ETL pipelines otherwise it’s not the youngest technology. During the talk, there are described all details about migrating pipelines from the old Hadoop platform to the Kubernetes, managing everything as the code, monitoring all corner cases of NiFi and making it a robust solution that is user-friendly even for non-programmers. Author: Albert Lewandowski Linkedin: https://www.linkedin.com/in/albert-lewandowski/ ___ Getindata is a company founded in 2014 by ex-Spotify data engineers. From day one our focus has been on Big Data projects. We bring together a group of best and most experienced experts in Poland, working with cloud and open-source Big Data technologies to help companies build scalable data architectures and implement advanced analytics over large data sets. Our experts have vast production experience in implementing Big Data projects for Polish as well as foreign companies including i.a. Spotify, Play, Truecaller, Kcell, Acast, Allegro, ING, Agora, Synerise, StepStone, iZettle and many others from the pharmaceutical, media, finance and FMCG industries. https://getindata.com

Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...

Databricks

Structured Streaming has proven to be the best platform for building distributed stream processing applications. Its unified SQL/Dataset/DataFrame APIs and Spark’s built-in functions make it easy for developers to express complex computations. Delta Lake, on the other hand, is the best way to store structured data because it is a open-source storage layer that brings ACID transactions to Apache Spark and big data workloads Together, these can make it very easy to build pipelines in many common scenarios. However, expressing the business logic is only part of the larger problem of building end-to-end streaming pipelines that interact with a complex ecosystem of storage systems and workloads. It is important for the developer to truly understand the business problem that needs to be solved. Apache Spark, being a unified analytics engine doing both batch and stream processing, often provides multiples ways to solve the same problem. So understanding the requirements carefully helps you to architect your pipeline that solves your business needs in the most resource efficient manner. In this talk, I am going examine a number common streaming design patterns in the context of the following questions. WHAT are you trying to consume? What are you trying to produce? What is the final output that the business wants? What are your throughput and latency requirements? WHY do you really have those requirements? Would solving the requirements of the individual pipeline actually solve your end-to-end business requirements? HOW are going to architect the solution? And how much are you willing to pay for it? Clarity in understanding the ‘what and why’ of any problem can automatically much clarity on the ‘how’ to architect it using Structured Streaming and, in many cases, Delta Lake.

Cassandra at eBay - Cassandra Summit 2012

Jay Patel

Top 5 Mistakes to Avoid When Writing Apache Spark Applications

Cloudera, Inc.

Top 10 Best Practices for Apache Cassandra and DataStax Enterprise

DataStax

No matter how diligent your organization is at driving toward efficiency, databases are complex and it’s easy to make mistakes on your way to production. The good news is, these mistakes are completely avoidable. In this webinar, Jeff Carpenter shares with you exactly how to get started in the right direction — and stay on the path to a successful database launch. View recording: https://youtu.be/K9Zj3bhjdQg Explore all DataStax webinars: https://www.datastax.com/resources/webinars

Apache Spark Architecture

Alexey Grishchenko

Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...

confluent

Delta lake and the delta architecture

Adam Doyle

Kudu Deep-Dive

Supriya Sahay

Kudu is popularly referred to as "Fast Analytics on Fast Data" capable of performing both OLAP & OLTP operations. Understand right from essentials to deep-dive into Kudu internals and architecture for building applications based on Kudu and integrating with Hadoop ecosystem. Read about Kudu clusters, architecture, operations, primary key design and column optimizations, partitioning and other performance considerations.

Understanding Query Plans and Spark UIs

Databricks

"The common use cases of Spark SQL include ad hoc analysis, logical warehouse, query federation, and ETL processing. Spark SQL also powers the other Spark libraries, including structured streaming for stream processing, MLlib for machine learning, and GraphFrame for graph-parallel computation. For boosting the speed of your Spark applications, you can perform the optimization efforts on the queries prior employing to the production systems. Spark query plans and Spark UIs provide you insight on the performance of your queries. This talk discloses how to read and tune the query plans for enhanced performance. It will also cover the major related features in the recent and upcoming releases of Apache Spark. "

Productizing Structured Streaming Jobs

Databricks

"Structured Streaming was a new streaming API introduced to Spark over 2 years ago in Spark 2.0, and was announced GA as of Spark 2.2. Databricks customers have processed over a hundred trillion rows in production using Structured Streaming. We received dozens of questions on how to best develop, monitor, test, deploy and upgrade these jobs. In this talk, we aim to share best practices around what has worked and what hasn't across our customer base. We will tackle questions around how to plan ahead, what kind of code changes are safe for structured streaming jobs, how to architect streaming pipelines which can give you the most flexibility without sacrificing performance by using tools like Databricks Delta, how to best monitor your streaming jobs and alert if your streams are falling behind or are actually failing, as well as how to best test your code."

Extending Druid Index File

Navis Ryu

Hive User Meeting August 2009 Facebook

ragho

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Anant Corporation

LSMの壁

guest6cf6c1

LSM

Graciela Remis

What's hot

A Closer Look at Apache Kudu

Andriy Zabavskyy

Introduction to Kafka Cruise Control

Jiangjie Qin

SQL on everything, in memory

Julian Hyde

ksqlDB: A Stream-Relational Database System

confluent

Presto Summit 2018 - 09 - Netflix Iceberg

kbajda

Druid deep dive

Kashif Khan

Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...

GetInData

Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...

Databricks

Cassandra at eBay - Cassandra Summit 2012

Jay Patel

Top 5 Mistakes to Avoid When Writing Apache Spark Applications

Cloudera, Inc.

Top 10 Best Practices for Apache Cassandra and DataStax Enterprise

DataStax

Apache Spark Architecture

Alexey Grishchenko

Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...

confluent

Delta lake and the delta architecture

Adam Doyle

Kudu Deep-Dive

Supriya Sahay

Understanding Query Plans and Spark UIs

Databricks

Productizing Structured Streaming Jobs

Databricks

Extending Druid Index File

Navis Ryu

Hive User Meeting August 2009 Facebook

ragho

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Anant Corporation

What's hot (20)

A Closer Look at Apache Kudu

Introduction to Kafka Cruise Control

SQL on everything, in memory

ksqlDB: A Stream-Relational Database System

Presto Summit 2018 - 09 - Netflix Iceberg

Druid deep dive

Best Practices for ETL with Apache NiFi on Kubernetes - Albert Lewandowski, G...

Designing ETL Pipelines with Structured Streaming and Delta Lake—How to Archi...

Cassandra at eBay - Cassandra Summit 2012

Top 5 Mistakes to Avoid When Writing Apache Spark Applications

Top 10 Best Practices for Apache Cassandra and DataStax Enterprise

Apache Spark Architecture

Apache Kafka vs. Traditional Middleware (Kai Waehner, Confluent) Frankfurt 20...

Delta lake and the delta architecture

Kudu Deep-Dive

Understanding Query Plans and Spark UIs

Productizing Structured Streaming Jobs

Extending Druid Index File

Hive User Meeting August 2009 Facebook

Data Engineer's Lunch #83: Strategies for Migration to Apache Iceberg

Viewers also liked

LSMの壁

LSM

Lsm february 2010

RocksDB detail

Some key value stores using log-structure

Zhichao Liang

Análisis fonológico

Oscar E

La lengua de señas mexicana es la lengua natural de las comunidades de Sordos, por ello el estudio de esta lengua contribuye a su reconocimiento social y a su re valoración para la enseñanza en la escuela. Una de las características de las lenguas de señas en general es su forma de transmisión visual, lo cual representa el primer obstáculo para su especialización, si consideramos que la LSM es la lengua materna de sólo un 5% de la población de niños sordos (Agosta, 1997). No obstante las limitaciones que han existido históricamente para la enseñanza formal de las lenguas de señas, éstas han persistido y son el medio de comunicación esencial de sesenta millones de personas en el mundo (Rodrigue, 1992). Estas lenguas han surgido en las comunidades de sordos de manera espontánea, son parte de su cultura y reflejo de su visión del mundo; además se encuentran en un proceso productivo constante, como sucede con cualquier otra lengua natural.

Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook

The Hive

This presentation describes the reasons why Facebook decided to build yet another key-value store, the vision and architecture of RocksDB and how it differs from other open source key-value stores. Dhruba describes some of the salient features in RocksDB that are needed for supporting embedded-storage deployments. He explains typical workloads that could be the primary use-cases for RocksDB. He also lays out the roadmap to make RocksDB the key-value store of choice for highly-multi-core processors and RAM-speed storage devices.

RocksDB compaction

MIJIN AN

MyRocks Deep Dive

Yoshinori Matsunobu

The Power of the Log

Ben Stopford

See the full talk here: https://www.infoq.com/presentations/lsm-append-data-structures This talk is about the beauty of sequential access and append only data structures. We'll do this in the context of a little known paper entitled “Log Structured Merge Trees”. LSM describes a surprisingly counterintuitive approach to storing and accessing data in a sequential fashion. It came to prominence in Google's Big Table paper and today, the use of Logs, LSM and append only data structures drive many of the world's most influential storage systems: Cassandra, HBase, RocksDB, Kafka and more. Finally we'll look at how the beauty of sequential access goes beyond database internals, right through to how applications communicate, share data and scale.

Lsm treesRed Over

Ucl ppt

Chetan Acharya

数据库系统设计漫谈

james tong

The right read optimization is actually write optimization

james tong

Cassandra SF 2012 - Technical Deep Dive: query performance

aaronmorton

Using Time Window Compaction Strategy For Time Series Workloads

Jeff Jirsa

Cassandra is a great fit for high write use cases, which makes it a popular choice for storing time series and sensor-collection workloads. At Crowdstrike, we've been using Cassandra for just that purpose, collecting petabytes of expiring time series data. In this talk, I'll discuss compaction in time series workloads, and the TimeWindowCompactionStrategy we developed specifically for this purpose. I'll detail TWCS specific configuration properties, some lesser known compaction sub-properties that apply to all compaction strategies, and also cover other general tricks and tuning that are useful for very large time-series workloads.

X internet framework

Neha Malik

Augmented Reality Smartglasses 2016 trends and challenges

Alexandre BOUCHET

[243]kaleido 노현걸

NAVER D2

[@IndeedEng] Imhotep Workshop

indeedeng

Link to video: http://youtu.be/LBDZFtqL-ck?list=UURVEh0SlyrZNTeIbEDwj3wQ We are excited to announce the open source availability of Imhotep, the interactive data analytics platform that powers data-driven decision making at Indeed. In a previous talk, we explained how we developed Imhotep, a distributed system for building decision trees for machine learning. We went on to describe how we build large scale interactive analytics tools using the same platform. Next we showed how our engineering and product organizations use Imhotep to focus on key metrics at scale. During this session, Product Manager Tom Bergman provided examples of valuable insights that can be gained by using Imhotep. After the presentation, attendees explored their own data in Imhotep. Product engineers were on hand to answer questions.

Viewers also liked (20)

LSMの壁

LSM

Lsm february 2010

RocksDB detail

Some key value stores using log-structure

Análisis fonológico

Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook

RocksDB compaction

MyRocks Deep Dive

The Power of the Log

Lsm trees

Ucl ppt

数据库系统设计漫谈

The right read optimization is actually write optimization

Cassandra SF 2012 - Technical Deep Dive: query performance

Using Time Window Compaction Strategy For Time Series Workloads

X internet framework

Augmented Reality Smartglasses 2016 trends and challenges

[243]kaleido 노현걸

[@IndeedEng] Imhotep Workshop

Recently uploaded

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

James Anderson

Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management. The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM). Speakers: Bob Boule Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle. Gopinath Rebala Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.

Introduction to CHERI technology - Cybersecurity

mikeeftimakis1

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs

Alex Pruden

This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second). Paper: https://eprint.iacr.org/2023/1886

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

SOFTTECHHUB

The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing. One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Sri Ambati

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

BookNet Canada

The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more. Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/ Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.

Leading Change strategies and insights for effective change management pdf 1.pdf

OnBoard

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

Albert Hoitingh

The Art of the Pitch: WordPress Relationships and Sales

Laura Byrne

Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes? All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.

Essentials of Automations: The Art of Triggers and Actions in FME

Safe Software

In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation. We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios. Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!

Enhancing Performance with Globus and the Science DMZ

Globus

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

UiPathCommunity

In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni. 📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath: Autopilot per Studio Web Autopilot per Studio Autopilot per Apps Clipboard AI GenAI applicata alla Document Understanding 👨‍🏫👨‍💻 Speakers: Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath Andrei Tasca, RPA Solutions Team Lead @NTT Data

DevOps and Testing slides at DASA Connect

Kari Kakkonen

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

DanBrown980551

Do you want to learn how to model and simulate an electrical network from scratch in under an hour? Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)! During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook. PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides: - A fully editable and extendable library for grid component modelling; - Visualization tools to display your network; - Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses; The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well. What you will learn during the webinar: - For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills; - For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.

By Design, not by Accident - Agile Venture Bolzano 2024

Pierluigi Pugliese

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Peter Spielvogel

Building better applications for business users with SAP Fiori. • What is SAP Fiori and why it matters to you • How a better user experience drives measurable business benefits • How to get started with SAP Fiori today • How SAP Fiori elements accelerates application development • How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities • How SAP Fiori paves the way for using AI in SAP apps

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

A tale of scale & speed: How the US Navy is enabling software delivery from l...

sonjaschweigert1

Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved: - Reduction in onboarding time from 5 weeks to 1 day - Improved developer experience and productivity through actionable findings and reduction of false positives - Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO) Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production. We will cover: - How to remove silos in DevSecOps - How to build efficient development pipeline roles and component templates - How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence) - How to streamline operations with automated policy checks on container images

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

UiPathCommunity

💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™: See how to accelerate model training and optimize model performance with active learning Learn about the latest enhancements to out-of-the-box document processing – with little to no training required Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath. Speakers: 👨‍🏫 Andras Palfi, Senior Product Manager, UiPath 👩‍🏫 Lenka Dulovicova, Product Program Manager, UiPath

Recently uploaded (20)

Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...

Introduction to CHERI technology - Cybersecurity

Removing Uninteresting Bytes in Software Fuzzing

zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs

Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...

GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...

Transcript: Selling digital books in 2024: Insights from industry leaders - T...

Leading Change strategies and insights for effective change management pdf 1.pdf

Encryption in Microsoft 365 - ExpertsLive Netherlands 2024

The Art of the Pitch: WordPress Relationships and Sales

Essentials of Automations: The Art of Triggers and Actions in FME

Enhancing Performance with Globus and the Science DMZ

Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™

DevOps and Testing slides at DASA Connect

LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...

By Design, not by Accident - Agile Venture Bolzano 2024

SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdf

Video Streaming: Then, Now, and in the Future

A tale of scale & speed: How the US Navy is enabling software delivery from l...

Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...

LSM Trees

1. LSM-Trees Chris Lohfink Email: clohfink@blackbirdit.com Twitter: cnlohfink

2. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra ● Senior Engineer at Pythian (recently acquired Blackbird) o Lead of Cassandra practice ● Experience o Worked with Cassandra since 0.8 (2011) o Java/Python developer About Me Chris Lohfink 1

3. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra • Fault tolerance • Multiple DCs • Large data sets • Data model gotchas • Unstable releases • Bootstrapping • Repairs • Unit tests - My Opinions - 2 Cassandra

4. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra ● Can be good, bad, or ugly ● Same as with everything else Where is Performance? 3

5. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra Lies, Dirty Lies, & Benchmarks 4

8. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra ● An LSM-tree is composed of two or more tree-like components, each optimized for their type of storage o A small in-memory tree o One or more on disk trees ● Used in Cassandra, HBase, LevelDB, Google Big Table, SQLite4 & more Log Structured Merge Tree LSM-Tree 7

9. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra Log Structured Merge Tree One tree in memory, one or more on disk Memory Disk ReadsWrites 8

10. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra Log Structured Merge Tree Match Cassandra nomenclature Memtable SSTable ReadsWrites 9

11. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com PAGE© Pythian Cassandra • Very Fast Writes – Cost of IO amortized across large batches – Append only • Comparable Reads – Comparable best-case – Potential for very slow Log Structured Merge Tree 10 Characteristics

12. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Represent numbers of shovels in stock Memtable SSTables Temp -20 Shovels 0 State MN 11

13. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Decide we need to include humidity Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 12

14. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Its Minnesota so the Temperature changed Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 13

15. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Shovel demand drops so its time to put some in stock again Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 14

16. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path Reached memtable limit, flush to disk Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Humidity 10 Temp 80 Shovels 25 15

17. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Write Path We still in Minnesota and its been a few days so adjust Temp Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Temp 10 16

18. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Read Path Check the current temp, take most recent from all SSTables and the Memtable Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Temp 10 17

19. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Compaction Create new SSTable thats the merged result Memtable SSTables Temp -20 Shovels 0 State MN Humidity 10 Temp 80 Shovels 25 Temp 10 Humidity 10 Shovels 25 Temp 80 State MN 18

20. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Compaction Delete old SSTables Memtable SSTables Temp 10 Humidity 10 Shovels 25 Temp 80 State MN 19

21. www.pythian.com twitter: cnlohfink | e-mail: clohfink@blackbirdit.com Pythian PAGE© Pythian© Pythian Cassandra Questions? 20

Editor's Notes

Attention! Before you open this template be sure what you have the following fonts installed: Novecento Sans wide font family (6 free styles) http://typography.synthview.com Abattis Cantarell http://www.fontsquirrel.com/fonts/cantarell Icon Sets Fonts: raphaelicons-webfont.ttf from this page: http://icons.marekventur.de iconic_stroke.ttf from this page: http://somerandomdude.com/work/open-iconic modernpics.otf from this page: http://www.fontsquirrel.com/fonts/modern-pictograms general_foundicons.ttf, social_foundicons.ttf from this page: http://www.zurb.com/playground/foundation-icons accessibility_foundicons_new.ttf fixed for PowerPoint Mac from here: http://docs.google.com/open?id=0Byf3seS60YFLR3VJN3dMbGp0UDA fontawesome-webfont.ttf from this page: http://fortawesome.github.io/Font-Awesome Entypo.otf from this page: http://www.fontsquirrel.com/fonts/entypo sosa-regular-webfont.ttf from this page: http://tenbytwenty.com/?xxxx_posts=sosa All fonts are permitted free use in commercial projects. If you have difficulties to install those fonts or have no time to find all of them, please follow the FAQs: http://graphicriver.net/item/six-template/3626243/support

LSM Trees

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Recently uploaded

Recently uploaded (20)

LSM Trees

Editor's Notes