The document outlines the schedule and content for a university computer science class. It includes an introduction to scheduling, a kernel timer winner, scheduling demos from student teams, a discussion of scheduling strategies like first-come first-served and round-robin, and a review of priority scheduling. It also announces details about an upcoming midterm exam and shows the results of a Rust program performance test on different laptops.
The document discusses the process for context switching between tasks in Rust. It explains that the current task is grabbed from thread-local storage and its ability to sleep is checked. The next task and cleanup function are prepared. Unsafe transmutes are used to get mutable references to tasks. The task contexts are swapped using a raw operation, placing the scheduler and next task in the proper locations. On the return swap, the cleanup function is immediately run.
Putting a Fork in Fork (Linux Process and Memory Management)David Evans
The document discusses several topics related to computer science class cs4414 at University of Virginia:
- Updates were due Sunday at 11:59pm including progress updates and scheduling design reviews.
- Tuesday's class will feature a guest lecture on authentication using single sign-on.
- The last class covered translation lookaside buffers and paging/segmentation concepts.
- A code sample is shown and analyzed that causes a segmentation fault due to accessing memory outside the allocated space.
- Details are provided on limiting resources and viewing process limits.
This document summarizes lecture slides from a university computer science class on operating system processes and virtual memory. The slides covered:
- Last week's discussion of process creation in Rust and the fork system call.
- The plan for this week's lectures, including how the kernel makes processes and diving into the fork.c source code.
- An overview of virtual memory and how it provides memory isolation between processes using paging and segmentation.
- Details of how x86 processors implement virtual memory using segmentation and paging tables, address translation, and handling of page faults.
- The history and evolution of virtual memory from early mainframe systems to modern desktop processors.
The document discusses the Rust runtime and how it interfaces with the Linux kernel to spawn new processes. It covers how fork() is implemented at the libc level and how it ultimately triggers a system call by generating an interrupt. It also discusses entering supervisor mode in the kernel and recaps some PS3 benchmarking results.
Multi-Tasking Map (MapReduce, Tasks in Rust)David Evans
The document discusses parallelizing the map function in Rust. It begins with an explanation of the original sequential map implementation and a first attempt at parallelization using spawn. This raises issues around shared memory that are solved through the use of channels. The presentation then discusses tasks in Rust and how they enable safely sharing immutable data. It concludes with questions about how much faster a parallel map implementation would be on different machines with varying numbers of cores.
University of Virginia
cs4414: Operating Systems
http://rust-class.org
Explicit Memory Management
4.3BSD
Morris Worm
fingerd code
NX bit
For embedded notes, see: http://rust-class.org/class-8-managing-memory.html
The document discusses the process for context switching between tasks in Rust. It explains that the current task is grabbed from thread-local storage and its ability to sleep is checked. The next task and cleanup function are prepared. Unsafe transmutes are used to get mutable references to tasks. The task contexts are swapped using a raw operation, placing the scheduler and next task in the proper locations. On the return swap, the cleanup function is immediately run.
Putting a Fork in Fork (Linux Process and Memory Management)David Evans
The document discusses several topics related to computer science class cs4414 at University of Virginia:
- Updates were due Sunday at 11:59pm including progress updates and scheduling design reviews.
- Tuesday's class will feature a guest lecture on authentication using single sign-on.
- The last class covered translation lookaside buffers and paging/segmentation concepts.
- A code sample is shown and analyzed that causes a segmentation fault due to accessing memory outside the allocated space.
- Details are provided on limiting resources and viewing process limits.
This document summarizes lecture slides from a university computer science class on operating system processes and virtual memory. The slides covered:
- Last week's discussion of process creation in Rust and the fork system call.
- The plan for this week's lectures, including how the kernel makes processes and diving into the fork.c source code.
- An overview of virtual memory and how it provides memory isolation between processes using paging and segmentation.
- Details of how x86 processors implement virtual memory using segmentation and paging tables, address translation, and handling of page faults.
- The history and evolution of virtual memory from early mainframe systems to modern desktop processors.
The document discusses the Rust runtime and how it interfaces with the Linux kernel to spawn new processes. It covers how fork() is implemented at the libc level and how it ultimately triggers a system call by generating an interrupt. It also discusses entering supervisor mode in the kernel and recaps some PS3 benchmarking results.
Multi-Tasking Map (MapReduce, Tasks in Rust)David Evans
The document discusses parallelizing the map function in Rust. It begins with an explanation of the original sequential map implementation and a first attempt at parallelization using spawn. This raises issues around shared memory that are solved through the use of channels. The presentation then discusses tasks in Rust and how they enable safely sharing immutable data. It concludes with questions about how much faster a parallel map implementation would be on different machines with varying numbers of cores.
University of Virginia
cs4414: Operating Systems
http://rust-class.org
Explicit Memory Management
4.3BSD
Morris Worm
fingerd code
NX bit
For embedded notes, see: http://rust-class.org/class-8-managing-memory.html
University of Virginia
cs4414: Operating Systems
http://rust-class.org
Scheduling in Linux, 2002-2014
Energy and Scheduling
OSX Mavericks Timer Coalescing
Scheduling Web Servers
Healthcare.gov
For embedded notes, see: http://rust-class.org/class-12-scheduling-in-linux-and-web-servers.html
University of Virginia
cs4414: Operating Systems
http://rust-class.org
The Internet
Benchmarking: Customer vs. Developer
Cheating on Benchmarks
Networking
Latency and Bandwidth
Tracing Routes
Network Layers
For embedded notes and videos, see:
http://rust-class.org/class-13-the-internet.html
University of Virginia
cs4414: Operating Systems
http://rust-class.org
What happened with Apple's SSL implementation
How to make sure this doesn't happen to you!
Sharing data
ARCs in Rust
Scheduling
For embedded notes, see:
Performance Wins with eBPF: Getting Started (2021)Brendan Gregg
This document provides an overview of using eBPF (extended Berkeley Packet Filter) to quickly get performance wins as a sysadmin. It recommends installing BCC and bpftrace tools to easily find issues like periodic processes, misconfigurations, unexpected TCP sessions, or slow file system I/O. A case study examines using biosnoop to identify which processes were causing disk latency issues. The document suggests thinking like a sysadmin first by running tools, then like a programmer if a problem requires new tools. It also outlines recommended frontends depending on use cases and provides references to learn more about BPF.
Kernel Recipes 2019 - Faster IO through io_uringAnne Nicolas
io_uring provides a new asynchronous I/O interface in Linux that aims to address limitations with existing interfaces like aio and libaio. It uses a ring-based model for submission and completion queues to efficiently support asynchronous I/O operations with low latency and high throughput. Though initially skeptical, Linus Torvalds ultimately merged io_uring into the Linux kernel due to improvements in missing features, ease of use, and efficiency over alternatives.
The document outlines the plan for the day which includes administrative tasks, communication methods, grading, and recording classes. It discusses IRC, a class website, and email for communication. It notes grading will not be stressful and includes a grading form. It describes future problem sets will include an auto-grader and demos. Recording classes will be somewhat edited and to press a button if you don't want to be recorded. It ends with asking if there are any questions.
USENIX LISA2021 talk by Brendan Gregg (https://www.youtube.com/watch?v=_5Z2AU7QTH4). This talk is a deep dive that describes how BPF (eBPF) works internally on Linux, and dissects some modern performance observability tools. Details covered include the kernel BPF implementation: the verifier, JIT compilation, and the BPF execution environment; the BPF instruction set; different event sources; and how BPF is used by user space, using bpftrace programs as an example. This includes showing how bpftrace is compiled to LLVM IR and then BPF bytecode, and how per-event data and aggregated map data are fetched from the kernel.
Linux kernel tracing superpowers in the cloudAndrea Righi
The Linux 4.x series introduced a new powerful engine of programmable tracing (BPF) that allows to actually look inside the kernel at runtime. This talk will show you how to exploit this engine in order to debug problems or identify performance bottlenecks in a complex environment like a cloud. This talk will cover the latest Linux superpowers that allow to see what is happening “under the hood” of the Linux kernel at runtime. I will explain how to exploit these “superpowers” to measure and trace complex events at runtime in a cloud environment. For example, we will see how we can measure latency distribution of filesystem I/O, details of storage device operations, like individual block I/O request timeouts, or TCP buffer allocations, investigating stack traces of certain events, identify memory leaks, performance bottlenecks and a whole lot more.
This document provides an overview of performance analysis tools for Linux systems. It describes Brendan Gregg's background and work analyzing performance at Netflix. It then discusses different types of tools, including observability tools to monitor systems, benchmarking tools to test performance, and tuning tools to optimize systems. A number of command line monitoring tools are outlined, such as vmstat, iostat, mpstat, and netstat, as well as more advanced tools like strace and tcpdump.
Spying on the Linux kernel for fun and profitAndrea Righi
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
Delivered as plenary at USENIX LISA 2013. video here: https://www.youtube.com/watch?v=nZfNehCzGdw and https://www.usenix.org/conference/lisa13/technical-sessions/plenary/gregg . "How did we ever analyze performance before Flame Graphs?" This new visualization invented by Brendan can help you quickly understand application and kernel performance, especially CPU usage, where stacks (call graphs) can be sampled and then visualized as an interactive flame graph. Flame Graphs are now used for a growing variety of targets: for applications and kernels on Linux, SmartOS, Mac OS X, and Windows; for languages including C, C++, node.js, ruby, and Lua; and in WebKit Web Inspector. This talk will explain them and provide use cases and new visualizations for other event types, including I/O, memory usage, and latency.
Profiling your Applications using the Linux Perf ToolsemBO_Conference
This document provides an overview of using the Linux perf tools to profile applications. It discusses setting up perf, benchmarking applications, profiling both CPU usage and sleep times, and analyzing profiling data. The document covers perf commands like perf record to collect profiling data, perf report to analyze the data, and perf script to convert it to other formats. It also discusses profiling options like call graphs and collecting kernel vs. user mode events.
This document discusses the history and importance of system calls, beginning with:
1. An introduction to system calls as functions called by users into the kernel to perform work, typically implemented with traps.
2. An overview of early syscall tracing tools from the 1980s like truss and strace, which used structured /proc to provide insight into what a system was doing.
3. A discussion of how syscalls remain highly significant events that provide the right level of granularity for understanding system performance and behavior, even decades later.
LinuxCon Europe, 2014. Video: https://www.youtube.com/watch?v=SN7Z0eCn0VY . There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This talk summarizes the three types of performance tools: observability, benchmarking, and tuning, providing a tour of what exists and why they exist. Advanced tools including those based on tracepoints, kprobes, and uprobes are also included: perf_events, ktap, SystemTap, LTTng, and sysdig. You'll gain a good understanding of the performance tools landscape, knowing what to reach for to get the most out of your systems.
This document describes OpenCL buffer management and provides examples of its use. It discusses declaring buffers, copying data between the host and device, and provides simple examples of image rotation and matrix multiplication. The goal is to demonstrate the basic OpenCL host code needed for buffer handling and to serve as templates for more complex kernels.
Patching: answers to questions you probably were afraid to ask about oracle s...DATA SECURITY SOLUTIONS
This document discusses Oracle database security updates and patching. It begins by addressing common questions and concerns about patching, such as whether patching is needed if behind a firewall, how often to apply proactive bugfixes, and potential issues like downtime and breakage. It then covers the different types of proactive patches, how to apply them to reduce downtime, and resources for further information. The conclusion emphasizes the importance of reading the patch README for installation instructions and details.
This document discusses optimizations for implementing an N-body simulation algorithm on GPUs using OpenCL. It begins with an overview of the basic N-body algorithm and its parallel implementation. Two key optimizations are explored: using local memory to enable data reuse across work items, and unrolling the computation loop. Performance results on AMD and Nvidia GPUs show that data reuse provides significant speedup, and loop unrolling further improves performance on the AMD GPU. An example N-body application is provided to experiment with these optimization techniques.
This document appears to be a series of slides from a class on the history and future of technology. It discusses many important innovations from the past few decades like the invention of the internet, development of the World Wide Web, creation of search engines and e-commerce sites, growth of social media, and advances in areas like GPS, mobile devices, and digital assistants. Each slide focuses on a different technology, company, or concept, and provides brief details and dates to give context to the technological progression being outlined.
The document discusses processes and threads in operating systems. It describes how processes contain multiple threads that can run concurrently on multicore systems. Each thread has its own execution state and context stored in a thread control block. When a thread is not actively running, its context is saved so it can resume execution later. The document provides examples of how threads are implemented in Windows and Linux operating systems.
University of Virginia
cs4414: Operating Systems
http://rust-class.org
Scheduling in Linux, 2002-2014
Energy and Scheduling
OSX Mavericks Timer Coalescing
Scheduling Web Servers
Healthcare.gov
For embedded notes, see: http://rust-class.org/class-12-scheduling-in-linux-and-web-servers.html
University of Virginia
cs4414: Operating Systems
http://rust-class.org
The Internet
Benchmarking: Customer vs. Developer
Cheating on Benchmarks
Networking
Latency and Bandwidth
Tracing Routes
Network Layers
For embedded notes and videos, see:
http://rust-class.org/class-13-the-internet.html
University of Virginia
cs4414: Operating Systems
http://rust-class.org
What happened with Apple's SSL implementation
How to make sure this doesn't happen to you!
Sharing data
ARCs in Rust
Scheduling
For embedded notes, see:
Performance Wins with eBPF: Getting Started (2021)Brendan Gregg
This document provides an overview of using eBPF (extended Berkeley Packet Filter) to quickly get performance wins as a sysadmin. It recommends installing BCC and bpftrace tools to easily find issues like periodic processes, misconfigurations, unexpected TCP sessions, or slow file system I/O. A case study examines using biosnoop to identify which processes were causing disk latency issues. The document suggests thinking like a sysadmin first by running tools, then like a programmer if a problem requires new tools. It also outlines recommended frontends depending on use cases and provides references to learn more about BPF.
Kernel Recipes 2019 - Faster IO through io_uringAnne Nicolas
io_uring provides a new asynchronous I/O interface in Linux that aims to address limitations with existing interfaces like aio and libaio. It uses a ring-based model for submission and completion queues to efficiently support asynchronous I/O operations with low latency and high throughput. Though initially skeptical, Linus Torvalds ultimately merged io_uring into the Linux kernel due to improvements in missing features, ease of use, and efficiency over alternatives.
The document outlines the plan for the day which includes administrative tasks, communication methods, grading, and recording classes. It discusses IRC, a class website, and email for communication. It notes grading will not be stressful and includes a grading form. It describes future problem sets will include an auto-grader and demos. Recording classes will be somewhat edited and to press a button if you don't want to be recorded. It ends with asking if there are any questions.
USENIX LISA2021 talk by Brendan Gregg (https://www.youtube.com/watch?v=_5Z2AU7QTH4). This talk is a deep dive that describes how BPF (eBPF) works internally on Linux, and dissects some modern performance observability tools. Details covered include the kernel BPF implementation: the verifier, JIT compilation, and the BPF execution environment; the BPF instruction set; different event sources; and how BPF is used by user space, using bpftrace programs as an example. This includes showing how bpftrace is compiled to LLVM IR and then BPF bytecode, and how per-event data and aggregated map data are fetched from the kernel.
Linux kernel tracing superpowers in the cloudAndrea Righi
The Linux 4.x series introduced a new powerful engine of programmable tracing (BPF) that allows to actually look inside the kernel at runtime. This talk will show you how to exploit this engine in order to debug problems or identify performance bottlenecks in a complex environment like a cloud. This talk will cover the latest Linux superpowers that allow to see what is happening “under the hood” of the Linux kernel at runtime. I will explain how to exploit these “superpowers” to measure and trace complex events at runtime in a cloud environment. For example, we will see how we can measure latency distribution of filesystem I/O, details of storage device operations, like individual block I/O request timeouts, or TCP buffer allocations, investigating stack traces of certain events, identify memory leaks, performance bottlenecks and a whole lot more.
This document provides an overview of performance analysis tools for Linux systems. It describes Brendan Gregg's background and work analyzing performance at Netflix. It then discusses different types of tools, including observability tools to monitor systems, benchmarking tools to test performance, and tuning tools to optimize systems. A number of command line monitoring tools are outlined, such as vmstat, iostat, mpstat, and netstat, as well as more advanced tools like strace and tcpdump.
Spying on the Linux kernel for fun and profitAndrea Righi
Do you ever wonder what the kernel is doing while your code is running? This talk will explore some methodologies and techniques (eBPF, ftrace, etc.) to look under the hood of the Linux kernel and understand what it’s actually doing behind the scenes.
Delivered as plenary at USENIX LISA 2013. video here: https://www.youtube.com/watch?v=nZfNehCzGdw and https://www.usenix.org/conference/lisa13/technical-sessions/plenary/gregg . "How did we ever analyze performance before Flame Graphs?" This new visualization invented by Brendan can help you quickly understand application and kernel performance, especially CPU usage, where stacks (call graphs) can be sampled and then visualized as an interactive flame graph. Flame Graphs are now used for a growing variety of targets: for applications and kernels on Linux, SmartOS, Mac OS X, and Windows; for languages including C, C++, node.js, ruby, and Lua; and in WebKit Web Inspector. This talk will explain them and provide use cases and new visualizations for other event types, including I/O, memory usage, and latency.
Profiling your Applications using the Linux Perf ToolsemBO_Conference
This document provides an overview of using the Linux perf tools to profile applications. It discusses setting up perf, benchmarking applications, profiling both CPU usage and sleep times, and analyzing profiling data. The document covers perf commands like perf record to collect profiling data, perf report to analyze the data, and perf script to convert it to other formats. It also discusses profiling options like call graphs and collecting kernel vs. user mode events.
This document discusses the history and importance of system calls, beginning with:
1. An introduction to system calls as functions called by users into the kernel to perform work, typically implemented with traps.
2. An overview of early syscall tracing tools from the 1980s like truss and strace, which used structured /proc to provide insight into what a system was doing.
3. A discussion of how syscalls remain highly significant events that provide the right level of granularity for understanding system performance and behavior, even decades later.
LinuxCon Europe, 2014. Video: https://www.youtube.com/watch?v=SN7Z0eCn0VY . There are many performance tools nowadays for Linux, but how do they all fit together, and when do we use them? This talk summarizes the three types of performance tools: observability, benchmarking, and tuning, providing a tour of what exists and why they exist. Advanced tools including those based on tracepoints, kprobes, and uprobes are also included: perf_events, ktap, SystemTap, LTTng, and sysdig. You'll gain a good understanding of the performance tools landscape, knowing what to reach for to get the most out of your systems.
This document describes OpenCL buffer management and provides examples of its use. It discusses declaring buffers, copying data between the host and device, and provides simple examples of image rotation and matrix multiplication. The goal is to demonstrate the basic OpenCL host code needed for buffer handling and to serve as templates for more complex kernels.
Patching: answers to questions you probably were afraid to ask about oracle s...DATA SECURITY SOLUTIONS
This document discusses Oracle database security updates and patching. It begins by addressing common questions and concerns about patching, such as whether patching is needed if behind a firewall, how often to apply proactive bugfixes, and potential issues like downtime and breakage. It then covers the different types of proactive patches, how to apply them to reduce downtime, and resources for further information. The conclusion emphasizes the importance of reading the patch README for installation instructions and details.
This document discusses optimizations for implementing an N-body simulation algorithm on GPUs using OpenCL. It begins with an overview of the basic N-body algorithm and its parallel implementation. Two key optimizations are explored: using local memory to enable data reuse across work items, and unrolling the computation loop. Performance results on AMD and Nvidia GPUs show that data reuse provides significant speedup, and loop unrolling further improves performance on the AMD GPU. An example N-body application is provided to experiment with these optimization techniques.
This document appears to be a series of slides from a class on the history and future of technology. It discusses many important innovations from the past few decades like the invention of the internet, development of the World Wide Web, creation of search engines and e-commerce sites, growth of social media, and advances in areas like GPS, mobile devices, and digital assistants. Each slide focuses on a different technology, company, or concept, and provides brief details and dates to give context to the technological progression being outlined.
The document discusses processes and threads in operating systems. It describes how processes contain multiple threads that can run concurrently on multicore systems. Each thread has its own execution state and context stored in a thread control block. When a thread is not actively running, its context is saved so it can resume execution later. The document provides examples of how threads are implemented in Windows and Linux operating systems.
Invent the Future (Operating Systems in 2029)David Evans
University of Virginia
cs4414: Operating Systems
http://rust-class.org
For embedded notes, see:
http://rust-class.org/class-23-invent-the-future.html
"Even so, mankind will suffer badly from the disease of boredom, a disease spreading more widely each year and growing in intensity. This will have serious mental, emotional and sociological consequences, and I dare say that psychiatry will be far and away the most important medical specialty in 2014. The lucky few who can be involved in creative work of any sort will be the true elite of mankind, for they alone will do more than serve a machine.
Indeed, the most somber speculation I can make about A.D. 2014 is that in a society of enforced leisure, the most glorious single word in the vocabulary will have become work!"
Isaac Asimov, visit to the 2014 World's Fair, 1964
This document summarizes the key points from a class on version control and linked lists in Rust. The class covered how to use Git for version control, including resolving conflicts when changes are merged. It then discussed implementing linked lists in Rust by defining a Node struct containing an integer head and Option tail, and a List type alias. Methods like to_str and increment were demonstrated. The class concluded by discussing how to map over lists by defining a Map trait.
Smarter Scheduling (Priorities, Preemptive Priority Scheduling, Lottery and S...David Evans
University of Virginia
cs4414: Operating Systems
http://rust-class.org
Scheduling Recap
Real-Time Scheduling
On-Demand vs. Planned Scheduling
First Come, First Served
Round-Robin
Priorities
Priority Preemptive
Priority Inversion
Lottery Scheduling
Stride Scheduling
For embedded notes, see: http://rust-class.org/class-11-smarter-scheduling.html
Threads allow a process to divide work into multiple simultaneous tasks. On a single processor system, multithreading uses fast context switching to give the appearance of simultaneity, while on multi-processor systems the threads can truly run simultaneously. There are benefits to multithreading like improved responsiveness and resource sharing.
Embedded Recipes 2018 - Finding sources of Latency In your system - Steven Ro...Anne Nicolas
Having just an RTOS is not enough for a real-time system. The hardware must be deterministic as well as the applications that run on the system. When you are missing deadlines, the first thing that must be done is to find what is the source of the latency that caused the issue. It could be the hardware, the operating system or the application, or even a combination of the above. This talk will discuss how to determine where the latency is using tools that come with the Linux Kernel, and will explain a few cases that caused issues.
The document discusses a plan for the day which includes topics on backdoors in Linux and Rust, a talk by Nishant, and the midterm exam. It then goes into details on the talk topics - discussing a past attempted Linux backdoor, whether a backdoor could be introduced in Rust, and Thompson's "Trusting Trust" speech. It also provides details on the midterm format and answers questions from students. Finally, it discusses the dining philosophers problem and shows an example implementation using a binary semaphore in Rust.
You have a system with an advanced programmatic tracer: do you know what to do with it? Brendan has used numerous tracers in production environments, and has published hundreds of tracing-based tools. In this talk he will share tips and know-how for creating CLI tracing tools and GUI visualizations, to solve real problems effectively. Programmatic tracing is an amazing superpower, and this talk will show you how to wield it!
How to find what is making your Oracle database slowSolarWinds
The document discusses time-based analysis for improving Oracle performance. It introduces the OraPub 3-Circle Analysis method which examines the operating system, Oracle database, and application simultaneously. It emphasizes the importance of using quantitative, trustworthy data and understanding how work is processed over time. The document provides guidance on gathering diagnostic data, identifying performance bottlenecks, and focusing on the right types of SQL to optimize Oracle performance.
The document discusses analyzing database systems using a 3D method for performance analysis. It introduces the 3D method, which looks at performance from the perspectives of the operating system (OS), Oracle database, and applications. The 3D method provides a holistic view of the system that can help identify issues and direct solutions. It also covers topics like time-based analysis in Oracle, how wait events are classified, and having a diagnostic framework for quick troubleshooting using tools like the Automatic Workload Repository report.
The bubble sort algorithm repeatedly steps through a list of items, compares adjacent pairs of items, and swaps them if they are in the wrong order. This process is repeated in passes through the list until it is fully sorted from lowest to highest value. The example demonstrates sorting the array [5, 1, 4, 2, 8] using bubble sort in three passes, with swaps occurring on the first two passes until the list is sorted after the third pass.
UKOUG version of a presentation trying to establish the sensible limits of parallelism on a couple of hardware configurations. Detailed white paper is at http://oracledoug.com/px_slaves.pdf
UKOUG, Lies, Damn Lies and I/O StatisticsKyle Hailey
1. Many factors can cause storage performance anomalies that make benchmarking difficult. Caching, shared infrastructure, I/O consolidation and fragmentation, and tiered storage are some of the top issues.
2. It is important to use real workloads, capture latency histograms rather than just averages, ensure results are reproducible, and run tests long enough to reach steady state.
3. Proper testing methodology is required to accurately characterize storage performance and avoid anomalies. Tools like FIO can help simulate real workloads.
Operationalizing Clojure in mature enterprises can be difficult. I'm presenting a case study from my experience deploying and maintaining a clojure application for delivering ad-free videos to ISS for NASA. The goal is to tease out the core principles that makes an application "operational".
This document discusses ways to diagnose performance issues in PostgreSQL. It begins with an introduction to common system resources like CPU, memory, disks, and network that can cause bottlenecks. It then covers specific PostgreSQL internal processes like locks that can lead to performance problems. The document provides examples of using tools like pg_stat_statements, gdb, perf, SystemTap, and trace files to analyze issues further. It emphasizes that performance problems can have complex causes and provides recommendations for improving monitoring and diagnostics.
An online semantic enhanced dirichlet model for short textJay Kumarr
The document presents a new online semantic-enhanced Dirichlet model (OSDM) for clustering short text streams. OSDM addresses challenges with existing approaches like semantic ambiguity, concept drift over time, and batch vs online processing. It maintains active topics online using a non-parametric probabilistic graphical model that incorporates semantic information through term co-occurrence and performs automatic topic detection. Experimental results on news, tweet and Reuters datasets show OSDM outperforms other models in clustering performance over data streams and is robust to parameter changes.
Video: https://www.youtube.com/watch?v=uibLwoVKjec . Talk by Brendan Gregg for Sysdig CCWFS 2016. Abstract:
"You have a system with an advanced programmatic tracer: do you know what to do with it? Brendan has used numerous tracers in production environments, and has published hundreds of tracing-based tools. In this talk he will share tips and know-how for creating CLI tracing tools and GUI visualizations, to solve real problems effectively. Programmatic tracing is an amazing superpower, and this talk will show you how to wield it!"
44CON London 2015 - 15-Minute Linux Incident Response Live Analysis44CON
This document summarizes a talk on conducting a 15-minute Linux live analysis to determine if a system has been hacked with minimal disturbance. The talk discusses opening a case, collecting key system data like processes and users through scripted network listeners, and analyzing the data to look for signs of compromise. It also covers next steps like dead analysis if evidence of hacking is found. The goal is to quickly identify breaches while preserving evidence through an automated and mostly non-invasive process.
This presentation was provided by Mark Llauferseiler of the University of Oklahoma, during part one of the NISO two-part webinar "Labor and Capacity for Research Data Management," which was held on March 11, 2020.
This document provides an overview and introduction to the Rust programming language. It begins with a brief history of computing systems and an explanation of why Rust is being used in the course. It then covers some basic Rust constructs like if expressions and higher-order functions. It also discusses challenges like lack of documentation and provides strategies for solving programming mysteries in Rust. The document ends with an example of Rust code for a simple HTTP server.
This document provides an overview of community profiling using QIIME. It discusses how next-generation sequencing is generating massive amounts of microbial sequence data. QIIME is introduced as a widely used open-source bioinformatics pipeline for analyzing microbiome census data from high-throughput sequencing experiments. The document outlines the typical QIIME workflow, which involves preprocessing raw sequencing data, picking operational taxonomic units (OTUs), assigning taxonomy, computing diversity metrics, and building phylogenetic trees.
En esta charla miraremos al futuro introduciendo Spark como alternativa al clásico motor de Hadoop MapReduce. Describiremos las diferencias más importantes frente al mismo, se detallarán los componentes principales que componen el ecosistema Spark, e introduciremos conceptos básicos que permitan empezar con el desarrollo de aplicaciones básicas sobre el mismo.
This document contains the notes from a class about cryptocurrency. It discusses the final exam, which will involve explaining bitcoin to different audiences and answering substantive questions. It then lists the names of students in the class divided into teams based on their answers to a registration question. The rest of the document outlines a jeopardy game about cryptocurrency topics played between the student teams, including questions about Satoshi Nakamoto, hashing, scripts, cryptography, randomness, and altcoins.
Trick or Treat?: Bitcoin for Non-Believers, Cryptocurrencies for CypherpunksDavid Evans
David Evans
DC Area Crypto Day
Johns Hopkins University
30 October 2015
This (non-research) talk will start with a tutorial introduction to cryptocurrencies and how bitcoin works (and doesn’t work) today. We’ll touch on some of the legal, policy, and business aspects of bitcoin and discuss some potential research opportunities in cryptocurrencies.
This document summarizes a class about hidden services using Tor and zero knowledge proofs. It discusses the rise of Bitcoin prices in August 2015, provides an overview of how Tor hidden services work through a network of nodes, and how the FBI was able to locate the Silk Road server. It also mentions that Problem Set 3 is due and lists upcoming office hours for students to attend.
This document summarizes anonymity and unlinkability in bitcoin transactions. It discusses how using different bitcoin addresses, or pseudonyms, makes it difficult to link transactions. Techniques like coinjoin and mixers are described that further confound tracing transactions by combining inputs from multiple users. The document mentions Silk Road, an illegal darknet market, and how its founder Ross Ulbricht now aims to create an economic simulation without coercion. It covers some threats to validity in analyzing anonymity and ends discussing communication privacy techniques like onion routing.
1) The midterm discussion covered confirmations in cryptocurrency transactions and the average wait time for the first confirmation.
2) It was noted that the threshold for being considered a "bitcoin expert" based on answering questions well on the midterm was around 85% of questions answered correctly.
3) Students were given updates on assignment due dates and opportunities to improve their midterm score by identifying and correcting incorrect statements in a referenced blockchain report.
The document summarizes a class on scripting and transactions in cryptocurrency. It discusses how Bitcoin core code has evolved over time to interpret scripts for locking and unlocking transactions. Examples are provided of common script patterns used prior to 2010, including pay-to-pubkey-hash and an important bug discovered that could allow stealing outputs. More advanced scripting options are also mentioned, such as checkmultisig.
The document summarizes a class on cryptocurrency and Bitcoin script. It discusses generating Bitcoin addresses through hashing public keys, describes the Bitcoin script language as a stack-based language similar to JVML used to write programs in transactions. It also notes that while Bitcoin script has limitations, altcoins are taking different approaches to scripting languages. Finally, it reminds students that project 2 is due Friday and the next class will feature a guest lecture from Tom Dukes on cyberlaw.
- Cryptocurrency mining requires a massive amount of energy. A single large bitcoin mining facility in China uses $60,000 worth of electricity per month.
- The total hashing power of the bitcoin network is estimated to be around 4.2 x 10^17 hashes per second, equivalent to around 212 megawatts of power continuously. This is around 9 times the power output of Dominion Power's Lake Anna Power Station.
- It is estimated that it takes around 35,395 kWh of electricity to mine a single bitcoin, costing $2,831 at a rate of $0.08 per kWh. However, the reward for mining a block is currently around 25 BTC, worth $5
This document summarizes a class lecture on cryptocurrency mining. It discusses the mining process, which involves finding a nonce value that satisfies the mining difficulty target for a block. Miners include transactions and solve cryptographic puzzles to validate blocks and earn rewards. The document explains Merkle trees, which improve transaction verification scalability. It also discusses the high computational costs and energy requirements of mining, noting specialized mining hardware can solve puzzles thousands of times faster than CPUs. The goal of mining is to process and validate transactions in a decentralized manner to maintain blockchain integrity.
- The document provides an overview of the schedule and topics for a cryptography class, including an introduction to cryptography today, Elliptic Curve Cryptography and signatures on Wednesday, and a checkup on the first three classes next Monday.
- It also lists the assigned readings for chapters 1-4 of the textbook and provides information about the backgrounds of students in the class.
- The remainder of the document discusses setting up a Bitcoin wallet, downloading the blockchain, hierarchical deterministic wallets, and provides a recap of the concepts from the previous class around what makes something a currency and how ownership of digital goods can be established.
This document provides an overview of a class on cryptocurrency and bitcoin. It discusses what makes a good currency, the history of currencies like salt and fiat currency, and challenges with decentralized digital currencies. It introduces bitcoin's approach of using a public ledger recorded through mining to record all transactions in a decentralized way without requiring trust in a central authority. The class will cover cryptography, computer science, economics and other topics through studying bitcoin as a concrete system. Students are assigned to set up a bitcoin wallet and complete readings before the next class.
This document contains the agenda for a cryptocurrency class. It lists several student presentations on topics related to cryptocurrency that will take place, including analyses of SHA hashing in Bitcoin, financial markets and game theory related to cryptocurrencies, and studying coinbase reserves to predict market price. It also references materials on the history of banking and reserve requirements. The document provides details on cryptocurrency student projects and presentations for an upcoming class.
This document summarizes a class on cryptocurrency and Silk Road. It discusses sidechains and how they allow bitcoin to evolve. It covers the legality of bitcoin in different jurisdictions, with some considering it legal, others contentious, and some viewing it as hostile. It then discusses Silk Road, the illegal online marketplace that was shut down, and how it used Tor and bitcoin. It summarizes how the FBI claims to have found the Silk Road server despite its use of Tor anonymity technology.
This document discusses Bloom filters and their use in Bitcoin simplified payment verification (SPV) nodes. It also covers merged mining, which allows mining of multiple cryptocurrencies like Bitcoin and Namecoin using the same hashing power. Sidechains are also mentioned. The document provides details on Bloom filter design and analysis, including the probability of false matches. It notes examples of merged mining blocks and addresses potential issues like those found in the Namecoin code. Project presentation dates are provided at the end.
This document summarizes a class about proofs-of-work for cryptocurrencies like Bitcoin. It discusses how Bitcoin and other cryptocurrencies use computationally intensive but useless proofs-of-work like SHA-256 to motivate investment in specialized hardware. It also explores the possibility of proofs-of-work that have useful outputs, like protein folding, and challenges in designing proofs-of-work that produce useful work while maintaining security properties. Finally, it announces an upcoming class about project proposals.
🔥🔥🔥🔥🔥🔥🔥🔥🔥
إضغ بين إيديكم من أقوى الملازم التي صممتها
ملزمة تشريح الجهاز الهيكلي (نظري 3)
💀💀💀💀💀💀💀💀💀💀
تتميز هذهِ الملزمة بعِدة مُميزات :
1- مُترجمة ترجمة تُناسب جميع المستويات
2- تحتوي على 78 رسم توضيحي لكل كلمة موجودة بالملزمة (لكل كلمة !!!!)
#فهم_ماكو_درخ
3- دقة الكتابة والصور عالية جداً جداً جداً
4- هُنالك بعض المعلومات تم توضيحها بشكل تفصيلي جداً (تُعتبر لدى الطالب أو الطالبة بإنها معلومات مُبهمة ومع ذلك تم توضيح هذهِ المعلومات المُبهمة بشكل تفصيلي جداً
5- الملزمة تشرح نفسها ب نفسها بس تكلك تعال اقراني
6- تحتوي الملزمة في اول سلايد على خارطة تتضمن جميع تفرُعات معلومات الجهاز الهيكلي المذكورة في هذهِ الملزمة
واخيراً هذهِ الملزمة حلالٌ عليكم وإتمنى منكم إن تدعولي بالخير والصحة والعافية فقط
كل التوفيق زملائي وزميلاتي ، زميلكم محمد الذهبي 💊💊
🔥🔥🔥🔥🔥🔥🔥🔥🔥
How to Download & Install Module From the Odoo App Store in Odoo 17Celine George
Custom modules offer the flexibility to extend Odoo's capabilities, address unique requirements, and optimize workflows to align seamlessly with your organization's processes. By leveraging custom modules, businesses can unlock greater efficiency, productivity, and innovation, empowering them to stay competitive in today's dynamic market landscape. In this tutorial, we'll guide you step by step on how to easily download and install modules from the Odoo App Store.
Level 3 NCEA - NZ: A Nation In the Making 1872 - 1900 SML.pptHenry Hollis
The History of NZ 1870-1900.
Making of a Nation.
From the NZ Wars to Liberals,
Richard Seddon, George Grey,
Social Laboratory, New Zealand,
Confiscations, Kotahitanga, Kingitanga, Parliament, Suffrage, Repudiation, Economic Change, Agriculture, Gold Mining, Timber, Flax, Sheep, Dairying,
A Visual Guide to 1 Samuel | A Tale of Two HeartsSteve Thomason
These slides walk through the story of 1 Samuel. Samuel is the last judge of Israel. The people reject God and want a king. Saul is anointed as the first king, but he is not a good king. David, the shepherd boy is anointed and Saul is envious of him. David shows honor while Saul continues to self destruct.
Gender and Mental Health - Counselling and Family Therapy Applications and In...PsychoTech Services
A proprietary approach developed by bringing together the best of learning theories from Psychology, design principles from the world of visualization, and pedagogical methods from over a decade of training experience, that enables you to: Learn better, faster!
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...TechSoup
Whether you're new to SEO or looking to refine your existing strategies, this webinar will provide you with actionable insights and practical tips to elevate your nonprofit's online presence.
Elevate Your Nonprofit's Online Presence_ A Guide to Effective SEO Strategies...
Scheduling
1.
2. Plan for Today
• Scheduling Intro
• Kernel Timer Winner!
• Scheduling
• Results of Performance Test
1 October 2013 University of Virginia cs4414 1
Shell Demos! (Stride Scheduled)
Evan Teague, Josh Lisko
Chris McFarland, Clarisse Abalos
Nishant Shukla, Jasdev Singh
Daniel Nizri, Alex Fabian
If you requested a demo, but
weren’t selected the
(undocumented) secondary tie-
breaker was submission time
(earlier is better) when multiple
teams had similar extra features.
3. 1 October 2013 University of Virginia cs4414 2
RememberfromClass4:
1. How should the supervisor decide which program to run?
2. How long should the alarm clock be set for?
4. Scheduler Desiderata
1 October 2013 University of Virginia cs4414 3
Go placidly amid the noise and haste, and
remember what peace there may be in silence.
As far as possible without surrender be on good
terms with all persons. Speak your truth quietly
and clearly; and listen to others, even the dull
and the ignorant; they too have their story.
Avoid loud and aggressive persons, they are
vexations to the spirit. … Exercise caution in
your business affairs; for the world is full of
trickery. …And whether or not it is clear to you,
no doubt the universe is unfolding as it
should…whatever your labors and aspirations,
in the noisy confusion of life keep peace with
your soul. With all its sham, drudgery, and
broken dreams, it is still a beautiful world. Be
cheerful. Strive to be happy.
Max Ehrmann, “Desiderata” (1927)
5. 1 October 2013 University of Virginia cs4414 4
How well do traffic lights do?
6. 1 October 2013 University of Virginia cs4414 5
How well do traffic lights do?
7. Kernel Timer: Wil Thomason
1 October 2013 University of Virginia cs4414 6
8. 1 October 2013 University of Virginia cs4414 7
Midterm
Exemption
vs.
“Dare and Do”
“To Dare is To Do” (Tottenham Hotspur FC)
9. About the Midterm
• It is not meant to be painful or tedious!
• A web form you will do outside of class
(posted Thursday Oct 10, due Monday Oct 14)
• Open resources except other humans
• You should not be surprised if questions from
the notes appear on the midterm
– It is fine (and encouraged!) to discuss those
questions in the course forum before the midterm
is posted
1 October 2013 University of Virginia cs4414 8
10. Scheduling Strategies
1 October 2013 University of Virginia cs4414 9
First Come, First Served (FIFO)
P1 P2 P3
(effectively: non-preemptive multi-processing)
11. Scheduling Strategies
1 October 2013 University of Virginia cs4414 10
First Come, First Served (FIFO)
P1 P2 P3
(effectively: non-preemptive multi-processing)
Round-Robin
P1 P1 P1P1P2 P2P3 P3P3P3
Time Slice Blocked
Each process gets to run for a set time slice, or until it finished or gets blocked.
13. Which one is my laptop using?
1 October 2013 University of Virginia cs4414 12
14. Priorities
• More important processes: “higher priority”
(except Linux inverts priority values!)
– Highest priority = 0 (gets most preferential
treatment)
– Lowest priority = 99 (highest number varies by
Linux flavor)
1 October 2013 University of Virginia cs4414 13
19. 1 October 2013 University of Virginia cs4414 18
“mdworker is short for ‘metadata server
worker’ and if you’re still confused don’t feel
bad. mdworker is basically the core technology
behind Mac OS X’s awesome search engine
Spotlight, it spiders meta data from your Mac
and its files and creates a readable index so
that you can find things practically
instantaneously via Spotlight (command-
spacebar).”
20. Pre-emptive Priority Scheduling
• Always run the highest priority process that is
ready to run
• Round-robin schedule among equally high,
ready to run, highest-priority processes
1 October 2013 University of Virginia cs4414 19
22. Shell Demos!
1 October 2013 University of Virginia cs4414 21
Nishant Shukla and Jasdev Singh
Daniel Nizri and Alex Fabian
23. 1 October 2013 University of Virginia cs4414 22
2013 MacBook Pro
Intel i7-3740QM
2.7 GHz, 4 cores (8 threads)
6MB shared L3 cache
2011 MacBook Air
Intel i5-2557M
1.7 GHz, 2 cores (4 threads)
3 MB shared L3 cache
both support “hyperthreading” (two threads per core)
60 seconds
(normalized time,
running on 16-
element list)
?
From Class 9: How much faster will my Rust
mapping program be on my new machine?
24. 1 October 2013 University of Virginia cs4414 23
impl Map for List {
fn mapr(&self, f: extern fn(int) -> int) -> List {
match(*self) {
None => None,
Some(ref node) => {
let (port, chan) : (Port<int>, Chan<int>) = stream();
let val = node.head;
do spawn {
chan.send(f(val));
}
let newtail = node.tail.mapr(f);
Some(~Node{ head: port.recv(), tail: newtail })
}
}
}
}
25. 1 October 2013 University of Virginia cs4414 24
fn collatz_steps(n: int) -> int {
if n == 1 { 0 } else { 1 + collatz_steps(if n % 2 == 0 { n / 2 } else { 3*n + 1 }) }
}
fn find_collatz(k: int) -> int { // Minimum value, n, with Collatz stopping time >= k.
let mut n = 1;
while collatz_steps(n) < k { n += 1; }
n
}
fn test_list(n: int, x: int) -> List {
if n == 0 { None } else { Some(~Node{head: x, tail: test_list(n - 1, x + 1)}) }
}
fn main() {
let lst0 = test_list(16, 565);
println(lst0.to_str());
let lst1 = lst0.mapr(find_collatz);
println(lst1.to_str());
}
Formatted to fit on slide. Your code should
have many more newlines than this!
26. 1 October 2013 University of Virginia cs4414 25
x 105
Stopping Time distribution for 1-100 Million
http://en.wikipedia.org/wiki/File:CollatzStatistic100million.png
27. Results
Old MacBook Air
$ rustc listspawn.rs
$ time ./listspawn
[565, 566, 567, 568, 569, 570, 571, 572,
573, 574, 575, 576, 577, 578, 579, 580]
[3542887, 3542887, 3542887, 3542887,
3542887, 3542887, 3542887, 3542887,
3542887, 3542887, 3542887, 3542887,
3542887, 3542887, 3542887, 3542887]
real 0m58.843s
user 3m39.459s
sys 0m0.266s
New MacBook Pro
$ time ./listspawn
[565, 566, 567, 568, 569, 570, 571, 572,
573, 574, 575, 576, 577, 578, 579, 580]
[3542887, 3542887, 3542887, 3542887,
3542887, 3542887, 3542887, 3542887,
3542887, 3542887, 3542887, 3542887,
3542887, 3542887, 3542887, 3542887]
real 0m18.593s
user 2m17.990s
sys 0m0.120s
1 October 2013 University of Virginia cs4414 26
28. Checking Results
Old MacBook Air
$ time ./listspawn
real 0m58.843s
user 3m39.459s
sys 0m0.266s
$ !!
real 1m0.931s
user 3m47.609s
sys 0m0.291s
$ !!
real 0m57.967s
user 3m36.343s
sys 0m0.266s
New MacBook Pro
$ time ./listspawn
real 0m18.593s
user 2m17.990s
sys 0m0.120s
$ !!
real 0m18.757s
user 2m20.419s
sys 0m0.116s
$ !!
real 0m18.505s
user 2m18.653s
sys 0m0.130s
1 October 2013 University of Virginia cs4414 27
Median over three trials:
real = 58.8s (user 219.5s)
Median over three trials:
real = 18.6s (user 138.0s)
29. 1 October 2013 University of Virginia cs4414 28
Median over three trials:
real = 18.6s (user 138.0s)
30. Does this make sense?
Old MacBook Air
1.7 GHz Processor
2 cores (4 hyperthreads)
New MacBook Pro
2.7 GHz Processor
4 cores (8 hyperthreads)
1 October 2013 University of Virginia cs4414 29
Median over three trials:
real = 58.8s (user 219.5s)
Median over three trials:
real = 18.6s (user 138.0s)
Cycle speed increase:
(2.7/1.7) = 1.59
Thread increase: 2
>>> 219.5 / 138.0
1.5905797101449275
>>> 58.8 / 18.6
3.1612903225806446
>>> (2.7 / 1.7) * 2
3.1764705882352944
31. What sh-Ould I have done
to make it faster?
1 October 2013 University of Virginia cs4414 30
32. -O >> $$Old MacBook Air
$ rustc -O listspawn.rs
$ time ./listspawn
real 0m10.573s
user 0m36.498s
sys 0m0.127s
bash-3.2$ time ./listspawn
real 0m10.299s
user 0m36.545s
sys 0m0.119s
bash-3.2$ time ./listspawn
real 0m10.330s
user 0m36.555s
sys 0m0.112s
New MacBook Pro
$ time ./listspawn
real 0m2.975s
user 0m22.913s
sys 0m0.022s
$ time ./listspawn
real 0m2.922s
user 0m22.950s
sys 0m0.016s
$ time ./listspawn
real 0m2.982s
user 0m22.875s
sys 0m0.019s
1 October 2013 University of Virginia cs4414 31
Median over three trials:
-- real = 58.8s (user 219.5s)
-O real = 10.3s (user 36.5s)
Median over three trials:
-- real = 18.6s (user 138.0s)
-O real = 2.97s (user 22.91s)
predicted (2.7/1.7 * 2): 3.18 times faster
without: 3.16 times faster
with -O: 3.47 times faster
33. Open Midterm
Exemption/HopperPatch
Opportunities
• Complete implementation of map_reduce
function from Class 9 (must use all cores
effectively and be able to do examples from
MapReduce paper)
• “Good Auld Shell” at UVa Football Game
• [New!] Write a program that runs at least 20x
faster on my new Mac than it does on my old
Mac
1 October 2013 University of Virginia cs4414 32
34. What algorithm did you use to
schedule your PS2 demos?
1 October 2013 University of Virginia cs4414 33