600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014)

•

20 likes•10,196 views

The slides for Rick Reed's presentation, “WhatsApp: Half a billion unsuspecting FreeBSD users”, given at MeetBSD California 2014 in San Jose. A recording of the talk can be viewed at: http://bit.ly/1HFc53n.

Technology

Half a Billion 600M+
Unsuspecting FreeBSD Users
Rick Reed
WhatsApp

topics
history
statistics
architecture
wins
challenges

rr
Joined WhatsApp server team in 2011
Performance, multimedia, security, ops
Previously: Yahoo!, SGI, Metaphor, Aion
First FreeBSD: 2.2.8 at Yahoo!

Why FreeBSD?
Previous Yahoo! experience
High-quality, high-performance TCP stack
Scale
Stability

Numbers
Over 600M monthly users
140M concurrent connections
440k connections per second
1.1M messages sent per second
1B pics sent per day

Numbers
~800 servers (incl. 200 chat + 350 mms)

Performance
SMP scaling is great
kernel relatively free of contention
Erlang (actor model)
BEAM (Erlang VM) very SMP-aware

Hardware (typical)
SoftLayer bare-metal
1U/2U/4U Supermicro
dual E5-2690v2 (40 threads)
(public, private) x 2 lagg x 1g igb (some 10g ix)
64-768GB RAM
1-12 800GB SSD (12x4TB HDD for video)
JBOD w/ mobo, Adaptec, and LSI controllers

Software Platform
FreeBSD 9.1 - 9.3
ufs
gmirror for HDD video servers
starting to evaluate 10.1

Software Platform
Erlang R16B01 (w/ our patchset)
rtprio
patches address specific contention issues
schedulers bound to cpus
allocators tuned for superpage promotion
yaws for web-serving (but disable sendfile)

Software Platform
stud for SSL termination (proxy via loopback)

Data storage
Erlang mnesia
account data, group membership, etc.
in-memory + dump/log persistence
Flat-file on hashed directory tree
offline mailboxes, media
fs layout tuned per workload

System metrics
6 x 800GB SSD per offline storage server

Versions
FreeBSD 7: 2009-2010
good
bugs: some

Versions
FreeBSD 8: 2011-2012
better
bugs: zero-copy sockets, igb, tsc tc
tuning got us to ~2.8M conns/server

Versions
FreeBSD 9: 2013-
yes!
bugs: very, very rare panic
enhancements:
>2G dirhash size
max tcp rexmit interval

Tuning
kernconf
remove unused modules
enable debugging/tracing

$Tuning loader vars igb (numq=8, rx_process_limit=-1, r/txd=4k) kern.nbuf=128k kern.max{files,filesperproc,proc} net.inet.tcp.tcbhashsize=512k net.inet.tcp.hostcache.bucketlimit=1$

Tuning
Network
capacity (mbufs, sockets, etc.)

Tuning
vfs
ufs.dirhash_maxmem (patched for >2G)
dirty buffer limits & sync delays

Issues
ufs panics (fs inconsistencies after crash/pfail)
bursty syncer behavior
filedesc table contention
pcb table contention (1.5m igb + 3m loopback)

Now vs. then
machdep.panic_on_nmi
VM pressure

Wishlist
Multipath TCP
Faster large RAM boot

This document provides an overview of CUDA (Compute Unified Device Architecture) and GPU programming. It begins with definitions of CUDA and GPU hardware architecture. The history of GPU development from basic graphics cards to modern programmable GPUs is discussed. The document then covers the CUDA programming model including the device model with multiprocessors and threads, and the execution model with grids, blocks and threads. It includes a code example to calculate squares on the GPU. Performance results are shown for different GPUs on a radix sort algorithm. The document concludes that GPU computing is powerful and will continue growing in importance for applications.

Introduction to Kubernetes

rajdeep

Kubernetes is an open-source system for managing containerized applications across multiple hosts. It includes key components like Pods, Services, ReplicationControllers, and a master node for managing the cluster. The master maintains state using etcd and schedules containers on worker nodes, while nodes run the kubelet daemon to manage Pods and their containers. Kubernetes handles tasks like replication, rollouts, and health checking through its API objects.

Accelerating Hyper-Converged Enterprise Virtualization using Proxmox and Ceph

Bangladesh Network Operators Group

This document discusses accelerating hyper-converged enterprise virtualization using Proxmox and Ceph. It defines hyper-converged infrastructure and describes the key features of Proxmox VE and Ceph storage. Proxmox VE is an open-source virtualization platform that can integrate computing, storage and networking. Ceph provides scalable and fault-tolerant open source storage. The document also provides steps to install Proxmox and configure Ceph storage, and summarizes a case study of deploying this solution at the University of Dhaka.

Building and deploying to an AKS cluster using Terraform and Azure DevOps wit...

Kenneth Nnadikwe

Cuda

Amy Devadas

The document discusses Compute Unified Device Architecture (CUDA), which is a parallel computing platform and programming model created by Nvidia that allows software developers to use GPUs for general-purpose processing. It provides an overview of CUDA, including its execution model, implementation details, applications, and advantages/drawbacks. The document also covers CUDA programming, compiling CUDA code, CUDA architectures, and concludes that CUDA has brought significant innovations to high performance computing.

Introducing Xen Server

StephenRice86

An AI accelerator ASIC architecture

Khanh Le

The document proposes a scalable AI accelerator ASIC platform for edge AI processing. It describes a high-level architecture based on a scalable AI compute fabric that allows for fast learning and inference. The architecture is flexible and can scale from single-chip solutions to multi-chip solutions connected via high-speed interfaces. It also provides details on the AI compute fabric, processing elements, and how the platform could enable high-performance edge AI processing.

LISA17 Container Performance Analysis

Brendan Gregg

Talk for USENIX LISA17: "Containers pose interesting challenges for performance monitoring and analysis, requiring new analysis methodologies and tooling. Resource-oriented analysis, as is common with systems performance tools and GUIs, must now account for both hardware limits and soft limits, as implemented using cgroups. A reverse diagnosis methodology can be applied to identify whether a container is resource constrained, and by which hard or soft resource. The interaction between the host and containers can also be examined, and noisy neighbors identified or exonerated. Performance tooling can need special usage or workarounds to function properly from within a container or on the host, to deal with different privilege levels and name spaces. At Netflix, we're using containers for some microservices, and care very much about analyzing and tuning our containers to be as fast and efficient as possible. This talk will show you how to identify bottlenecks in the host or container configuration, in the applications by profiling in a container environment, and how to dig deeper into kernel and container internals."

The document summarizes new features and the development roadmap for ClickHouse. Some key things that have been completed include custom partition keys, geospatial functions, integration with CatBoost machine learning models, improved date/time functions, and proper support for time zones. Future plans include supporting foreign databases through table functions, improving SQL compatibility, an ODBC driver, and adding support for updates, deletes, and joins. The development is very active with contributions from Yandex and a growing community.

Part 02 Linux Kernel Module Programming

Tushar B Kute

Embedded Android : System Development - Part III

Emertxe Information Technologies Pvt Ltd

For new age touch-based embedded devices, Android is becoming a popular OS going beyond mobile phones. With its roots from Embedded Linux, Android framework offers benefits in terms of rich libraries, open-source and multi-device support. Emertxe’s hands-on Embedded Android Training Course is designed to customize, build and deploy custom Embedded OS on ARM target. Rich set of projects will make your learning complete.

Installing Postgres on Linux

EDB

Time Sensitive Networking in the Linux Kernel

henrikau

Introduction to GPU Programming

Chakkrit (Kla) Tantithamthavorn

GPU computing provides a way to access the power of massively parallel graphics processing units (GPUs) for general purpose computing. GPUs contain over 100 processing cores and can achieve over 500 gigaflops of performance. The CUDA programming model allows programmers to leverage this parallelism by executing compute kernels on the GPU from their existing C/C++ applications. This approach democratizes parallel computing by making highly parallel systems accessible through inexpensive GPUs in personal computers and workstations. Researchers can now explore manycore architectures and parallel algorithms using GPUs as a platform.

Jenkins presentation

Valentin Buryakov

Jenkins is a continuous integration server that detects code changes, runs automated builds and tests, and can deploy code. It supports defining build pipelines as code to make them version controlled and scalable. Popular plugins allow Jenkins pipelines to integrate with tools for testing, reporting, notifications, and deployments. Pipelines can define stages, run steps in parallel, and leverage existing Jenkins functionality.

C++ GUI 라이브러리 소개: Qt & Nana

Lazy Ahasil

Qemu

Koganti Ravikumar

QEMU is a free and open-source hypervisor that performs hardware virtualization by emulating CPUs through dynamic binary translation and providing device models. This allows it to run unmodified guest operating systems. It can be used to create virtual machines similarly to VMWare, VirtualBox, KVM, and Xen. QEMU also supports emulating different CPU architectures and can save and restore the state of a virtual machine.

PCCC23：日本AMD株式会社テーマ１「AMD Instinct™ アクセラレーターの概要」

PC Cluster Consortium

Booting Android: bootloaders, fastboot and boot images

Chris Simmonds

This document discusses booting Android devices. It covers Android boot images, bootloaders, fastboot protocol, and file systems used for different types of flash memory in Android devices. The key topics covered include bootloaders loading the boot and recovery images, the fastboot protocol for flashing and debugging, and file systems like ext4, f2fs, yaffs2 used on different flash chips like eMMC, SD cards, and raw NAND flash.

MongoDB Management & Ansible

MongoDB

Presented by Michael Lynn, Senior Solutions Architect, MongoDB Deploying databases, applications and infrastructure can be a difficult task. Once the applications and databases have been deployed, the tasks associated with managing, monitoring, backing up can be even more complex. Ansible provides developers the ability to deploy, provision and configure your application and database infrastructure for swift delivery to any hosting platform: physical, virtual, cloud or on-premise. Ops Manager, simply put, is the best way to run MongoDB in your environment. It provides the ability to deploy, monitor, manage, and backup your MongoDB databases. In this presentation, you will learn how to automate deployment of a MongoDB Ops Manager environment from the ground up, and deploy it to datacenters around the world with a few simple commands using Ansible. Learning Objectives: - Attendees will learn about Ansible, and how playbooks and tasks work - Attendees will learn how to create simple playbooks to deploy MongoDB servers for management via MongoDB Ops Manager - Attendees will learn how to monitor, manage and backup their MongoDB infrastructure using Ops Manager from MongoDB

Drone CI/CD Platform

Bo-Yi Wu

Power Management from Linux Kernel to Android

National Cheng Kung University

The document provides an overview of power management from Linux kernel to Android, covering introduction to Linux power management concepts, concepts behind Android power management, design and implementation details, and areas for potential improvements. It describes goals and approaches of Advanced Power Management (APM) and Advanced Configuration and Power Interface (ACPI) in Linux, and how Android power management is built as a slim wrapper on top of Linux power management, utilizing components like wake locks and different types of wake locks to control power state transitions.

Docker introduction for the beginners

Juneyoung Oh

1. Docker allows creating lightweight virtual environments called containers that package code and dependencies together. Containers are more portable than virtual machines. 2. Docker uses images to build containers. Images are immutable templates and containers are instances of images that can be run. The Dockerfile defines how to build images. 3. Common Docker commands include docker pull to download images, docker run to create and start containers, docker exec to run commands in running containers, and docker commit to save container changes as new images.

Hardware Acceleration for Machine Learning

CastLabKAIST

This document provides an overview of a lecture on hardware acceleration for machine learning. The lecture will cover deep neural network models like convolutional neural networks and recurrent neural networks. It will also discuss various hardware accelerators developed for machine learning, including those designed for mobile/edge and cloud computing environments. The instructor's background and the agenda topics are also outlined.

Kubernetes Introduction

Martin Danielsson

History of Linux

Gray-chan Desu

Using Rook to Manage Kubernetes Storage with Ceph

CloudOps2005

Interrupts on xv6

Takuya ASADA

The Cell Processor

Heiko Joerg Schick

The document summarizes the Cell processor architecture, which was developed as a collaboration between IBM, Sony, and Toshiba to address limitations in processor performance. The Cell consists of 9 cores - 1 PowerPC core called the PPE and 8 synergistic processor elements (SPEs) optimized for SIMD operations. It has a peak performance of over 200 GFLOPS and was used in the PlayStation 3 game console to enable graphics-intensive applications. The document outlines the Cell architecture and how it aims to overcome performance walls related to power, memory, and frequency limitations.

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Tommy Lee

This document provides an overview of performance analysis and tuning techniques in Red Hat Enterprise Linux (RHEL). It discusses the tuned profile packages and how they optimize systems for different workloads. Specific topics covered include disk I/O tuning, memory tuning, network performance tuning, and power management techniques. A variety of Linux performance analysis tools are also introduced, including tuned, turbostat, netsniff-ng, and Performance Co-Pilot.

What's hot

ClickHouse new features and development roadmap, by Aleksei Milovidov

Altinity Ltd

Part 02 Linux Kernel Module Programming

Tushar B Kute

Embedded Android : System Development - Part III

Emertxe Information Technologies Pvt Ltd

Installing Postgres on Linux

EDB

Time Sensitive Networking in the Linux Kernel

henrikau

Introduction to GPU Programming

Chakkrit (Kla) Tantithamthavorn

Jenkins presentation

Valentin Buryakov

C++ GUI 라이브러리 소개: Qt & Nana

Lazy Ahasil

Qemu

Koganti Ravikumar

PCCC23：日本AMD株式会社テーマ１「AMD Instinct™ アクセラレーターの概要」

PC Cluster Consortium

Booting Android: bootloaders, fastboot and boot images

Chris Simmonds

MongoDB Management & Ansible

MongoDB

Drone CI/CD Platform

Bo-Yi Wu

Power Management from Linux Kernel to Android

National Cheng Kung University

Docker introduction for the beginners

Juneyoung Oh

Hardware Acceleration for Machine Learning

CastLabKAIST

Kubernetes Introduction

Martin Danielsson

History of Linux

Gray-chan Desu

Using Rook to Manage Kubernetes Storage with Ceph

CloudOps2005

Interrupts on xv6

Takuya ASADA

What's hot (20)

ClickHouse new features and development roadmap, by Aleksei Milovidov

Part 02 Linux Kernel Module Programming

Embedded Android : System Development - Part III

Installing Postgres on Linux

Time Sensitive Networking in the Linux Kernel

Introduction to GPU Programming

Jenkins presentation

C++ GUI 라이브러리 소개: Qt & Nana

Qemu

PCCC23：日本AMD株式会社テーマ１「AMD Instinct™ アクセラレーターの概要」

Booting Android: bootloaders, fastboot and boot images

MongoDB Management & Ansible

Drone CI/CD Platform

Power Management from Linux Kernel to Android

Docker introduction for the beginners

Hardware Acceleration for Machine Learning

Kubernetes Introduction

History of Linux

Using Rook to Manage Kubernetes Storage with Ceph

Interrupts on xv6

Similar to 600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014)

The Cell Processor

Heiko Joerg Schick

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Tommy Lee

Experience In Building Scalable Web Sites Through Infrastructure's View

Phuwadon D

The document discusses strategies for building scalable web sites, including using caching technologies like Memcached, database replication and sharding, and load balancing. It provides recommendations for hardware, software architectures, and technologies to use at different stages of a site's growth to scale efficiently. Tips are also given for optimizing performance through caching, splitting content delivery, and other best practices.

Ceph Day New York 2014: Ceph, a physical perspective

Ceph Community

The document summarizes the results of testing a Ceph storage cluster configuration using Supermicro hardware. Key findings include: - Using SSDs for journals improved sequential write bandwidth significantly. - Erasure coded pools provided reasonable performance at a lower cost compared to replicated pools. - A single client could saturate the network connection with two 36-bay OSD nodes. - Network performance was critical as the cluster scaled to support more clients and objects. - Further testing was needed on erasure coded performance under failure conditions and using newer Ceph and Linux versions.

Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...

Виталий Стародубцев

##Что такое Storage Replica ##Архитектура и сценарии ##Синхронная и асинхронная репликация ##Междисковая, межсерверная, внутрикластерная и межкластерная репликация ##Дизайн и проектирование Storage Replica ##Нововведения в Windows Server 2016 TP5 ##Графический интерфейс управления, и другие возможности - демонстрация и планы развития ##Интеграция Storage Replica с Storage Spaces Direct

SUN主机产品介绍.ppt

PencilData

Sun produces a range of server products including x64 servers, SPARC Enterprise servers, and CMT products. The document discusses Sun's T-series of CMT servers which use concurrent multi-threading (CMT) technology to improve processor performance through thread-level parallelism. The highest-end model is the T5440 server which supports up to 256 threads and 512GB of memory in a redundant 4RU chassis.

02 computer components

Chalard Jaipan

The document discusses various computer components including hardware like processors, memory, storage and input/output devices, as well as software components like operating systems, utility programs, middleware, and application software. It also covers emerging technologies, database systems, and trends in hardware components over time as technologies advanced and prices declined according to Moore's Law.

Extending the life of your device (firmware updates over LoRa) - LoRa AMM

Jan Jongboom

This document discusses extending the lifespan of IoT devices through firmware updates and outlines some challenges and solutions. It proposes a standardized approach using multicast transmissions, forward error correction, and an update server to efficiently deliver firmware over constrained low-power wide area networks. An open-source reference implementation is available to demonstrate feasibility on current hardware within radio regulations.

Overview of HPC Interconnects

inside-BigData.com

In this deck from ATPESC 2019, Ken Raffenetti from Argonne presents an overview of HPC interconnects. "The Argonne Training Program on Extreme-Scale Computing (ATPESC) provides intensive, two-week training on the key skills, approaches, and tools to design, implement, and execute computational science and engineering applications on current high-end computing systems and the leadership-class computing systems of the future." Watch the video: https://wp.me/p3RLHQ-luc Learn more: https://extremecomputingtraining.anl.gov/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter

Threading Successes 06 Allegorithmic

guest40fc7cd

Allegorithmic developed Substance, a middleware for procedurally generating textures on CPUs to reduce texture memory and streaming bottlenecks. Substance uses a node-based graph to procedurally generate textures. It is designed to take advantage of multi-core CPUs through techniques like task parallelism, data parallelism, and lockless synchronization to efficiently generate textures across CPU cores in parallel. Testing showed Substance could utilize 4 CPU cores to generate textures 3.8 times faster than a single core, helping to maintain high framerates during texture streaming.

Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)

Ontico

HighLoad++ 2017 Зал «Москва», 7 ноября, 13:00 Тезисы: http://www.highload.ru/2017/abstracts/2909.html OpenDataPlane (ODP, https://www.opendataplane.org) является open-source-разработкой API для сетевых data plane-приложений, представляющий абстракцию между сетевым чипом и приложением. Сейчас вендоры, такие как TI, Freescale, Cavium, выпускают SDK с поддержкой ODP на своих микросхемах SoC. Если проводить аналогию с графическим стеком, то ODP можно сравнить с OpenGL API, но только в области сетевого программирования. ...

JetStor NAS series 2016

Gene Leyzarovich

AC&NC provides full product line up of Network Attached Storage (NAS) systems that are all built for reliability and ease of use. AC&NC also offers combined NAS and Storage Area Networks (SAN) into a single system, allowing for a consolidated storage and network environment. Focused intently on storage without distractions of tape backup or bundled servers, AC&NC manufacturers in-house and delivers complete solutions in 24-48 hours from in-stock JetStor RAID, iSCSI, FC, NAS / Unified, All Flash and JBOD SAS systems that set the bar for performance.

Syllabus for interview

Swapnil Kapate

The document provides information on several topics related to IT and networking: 1. It discusses the components of a motherboard including the north bridge and south bridge, and their functions. 2. It provides steps for performing a zero-level format of a hard drive using Windows 98. 3. It describes different types of RAID configurations (RAID 0-10) and their characteristics in terms of performance and fault tolerance.

Bare Metal Club ATX: Networking Discussion

Carl Perry

robust-storage-solution

Tecsun Yeep

The document discusses Robust Data Solution's enterprise storage solutions including their Twelve+ high density mid-tower NAS, RackStor scale-out storage servers, and Maxx JBOD expansion units. It provides details on the specifications, features, and use cases of each product line including processor, memory, bays, controllers, networking, management and more. The document also outlines Robust Data Solution's services, support plans, customers and mission to deliver high performance and scalable storage solutions.

Sector Sphere 2009

lilyco

Sector is a distributed file system that stores files on local disks of nodes without splitting files. Sphere is a parallel data processing engine that processes data locally using user-defined functions like MapReduce. Sector/Sphere is open source, supports fault tolerance through replication, and provides security through user accounts and encryption. Performance tests show Sector/Sphere outperforms Hadoop for sorting and malware analysis benchmarks by processing data locally.

sector-sphere

xlight

Sector is a distributed file system that stores files on local disks of nodes without splitting files. Sphere is a parallel data processing engine that processes data locally using user-defined functions like MapReduce. Sector/Sphere is open source, written in C++, and provides high performance distributed storage and processing for large datasets across wide areas using techniques like UDT for fast data transfer. Experimental results show it outperforms Hadoop for certain applications by exploiting data locality.

Multiple Cores, Multiple Pipes, Multiple Threads – Do we have more Parallelis...

Slide_N

This document summarizes a presentation given at the 2005 IEEE Hot Chips conference about parallelism in modern processors and how it relates to programming models. It discusses different types of parallelism available at the processor, system, and application levels. It then examines approaches to parallelism used by general-purpose CPUs, special-purpose CPUs like the Cell processor, and GPUs. While parallelism is increasing in these devices, programming them effectively remains challenging due to the difficulty of parallel programming and lack of appropriate language and tooling support. The document calls for more research in parallel programming models and languages to make better use of emerging multi-core architectures.

Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...

Databricks

This document summarizes a presentation about using the Crail distributed storage system to improve Spark performance on high-performance computing clusters with RDMA networking and NVMe flash storage. The key points are: 1) Traditional Spark storage and networking APIs do not bypass the operating system kernel, limiting performance on modern hardware. 2) The Crail system provides user-level APIs for RDMA networking and NVMe flash to improve Spark shuffle, join, and sorting workloads by 2-10x on a 128-node cluster. 3) Crail allows Spark workloads to fully utilize high-speed networks and disaggregate memory and flash storage across nodes without performance penalties.

How to Get a Game Changing Performance Advantage with Intel SSDs and Aerospike

Aerospike, Inc.

Frank Ober of Intel’s Solutions Group will review how he achieved 1+ million transactions per second on a single dual socket Xeon Server with SSDs using the open source tools of Aerospike for benchmarking. The presentation will include a live demo showing the performance of a sample system. We will cover: The state of Key-value Stores on modern SSDs. What choices you make in your selection process of hardware that will most benefit a consistent deployment of Aerospike. How to run an Aerospike mesh on a single machine. How to work replication of that mesh, and what values allow for maximum threading and scale. We will also focus on some key learnings and the Total Cost of Ownership choices that will make your deployment more effective long term.

Similar to 600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014) (20)

The Cell Processor

Shak larry-jeder-perf-and-tuning-summit14-part2-final

Experience In Building Scalable Web Sites Through Infrastructure's View

Ceph Day New York 2014: Ceph, a physical perspective

Технологии работы с дисковыми хранилищами и файловыми системами Windows Serve...

SUN主机产品介绍.ppt

02 computer components

Extending the life of your device (firmware updates over LoRa) - LoRa AMM

Overview of HPC Interconnects

Threading Successes 06 Allegorithmic

Dataplane networking acceleration with OpenDataplane / Максим Уваров (Linaro)

JetStor NAS series 2016

Syllabus for interview

Bare Metal Club ATX: Networking Discussion

robust-storage-solution

Sector Sphere 2009

sector-sphere

Multiple Cores, Multiple Pipes, Multiple Threads – Do we have more Parallelis...

Running Apache Spark on a High-Performance Cluster Using RDMA and NVMe Flash ...

How to Get a Game Changing Performance Advantage with Intel SSDs and Aerospike

More from iXsystems

The True Story of FreeNAS

iXsystems

BayLISA - FreeNAS 10 by Jordan Hubbard

iXsystems

FreeNAS 10 is an open source storage and application hosting software based on FreeBSD 11.0. It uses ZFS as its default filesystem and includes over 400 third party packages. The goals of FreeNAS 10 are to make complex storage systems simple for users while retaining powerful features for administrators. It aims to provide flexibility for application hosting and storage, as well as situational awareness through its GUI and CLI interfaces. FreeNAS 10 simplifies many storage and networking tasks including ZFS storage setup, file sharing, directory services, services management, virtualization, and containerized applications.

History of ZFS (MeetBSD California 2016)

iXsystems

PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)

iXsystems

FreeNAS 10: Challenges of Building a Modern Storage Appliance based on FreeBS...

iXsystems

NeXTBSD aka FreeBSD X

iXsystems

FreeBSD: The Next 10 Years (MeetBSD 2014)

iXsystems

UCL All of the Things (MeetBSD California 2014 Lightning Talk)

iXsystems

Netflix Update (MeetBSD California 2014 Lightning Talk)

iXsystems

PBI v10: Application Management Made Easy by Ken Moore

iXsystems

These slides accompanied a presentation given at Southeast Linux Fest. The PBI system provides an easy-to-use package management framework for installing, removing, and running applications while also providing simple methods for keeping your system up to date and in working order. In this session we will discuss the recent developments in the 10.0 PBI format, describe the background systems that make everything run, and demonstrate how an end-user interacts with this system via the AppCafe on PC-BSD. The slides also show how this system can be utilized to manage and use jails on PC-BSD/FreeBSD, enabling the user to isolate applications away from the general system for improved safety and security.

Introduction to FreeNAS development by John Hixson

iXsystems

At SCALE 12x, John Hixson, Senior Software Developer at iXsystems, gave a his talk, "Introduction to FreeNAS development". FreeNAS has been around for several years now but development on it has been by very few people. Even with corporate sponsorshipt and a team of full time developers, outside interest has been minimal. Not a week goes by when a bug report or feature request is not filed. Documentation on how to develop on FreeNAS simply does not exist. Currently, the only way to come up to speed on FreeNAS development is to obtain the source code, read through it, modify it and verify it works. The goal of this paper is to create a simple FreeNAS application to demonstrate some of the common methods used when dealing with FreeNAS development, as well as showcase some of the API.

Jenkins, Bhyve, and Webdriver: Continuous Integration testing on FreeNAS by C...

iXsystems

On November 7, 2013, the FreeBSD Vendor Summit was held at the Yahoo! campus in Sunnyvale, California. Craig Rodrigues, iXsystems software engineer, gave a presentation, "Jenkins, BHyve, and WebDriver: Continuous Integration Testing on FreeNAS". Craig's presentation described how iXsystems is using modern best practices for building and testing FreeNAS code. Jenkins is a framework for doing continuous builds and integration, and is used by hundreds of companies. BHyve (BSD Hypvervisor) is the new virtual machine system which will be part of FreeBSD 10. Webdriver is a Python toolkit for testing web applications. By combining these technologies, iXsystems is developing a modern and sophisticated workflow for testing and improving the quality of FreeNAS.

More from iXsystems (12)

The True Story of FreeNAS

BayLISA - FreeNAS 10 by Jordan Hubbard

History of ZFS (MeetBSD California 2016)

PCI Pass-through - FreeBSD VM on Hyper-V (MeetBSD California 2016)

FreeNAS 10: Challenges of Building a Modern Storage Appliance based on FreeBS...

NeXTBSD aka FreeBSD X

FreeBSD: The Next 10 Years (MeetBSD 2014)

UCL All of the Things (MeetBSD California 2014 Lightning Talk)

Netflix Update (MeetBSD California 2014 Lightning Talk)

PBI v10: Application Management Made Easy by Ken Moore

Introduction to FreeNAS development by John Hixson

Jenkins, Bhyve, and Webdriver: Continuous Integration testing on FreeNAS by C...

Recently uploaded

Mind map of terminologies used in context of Generative AI

Kumud Singh

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

Paige Cruz

Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack. While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack. I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:

20240605 QFM017 Machine Intelligence Reading List May 2024

Matthew Sinclair

“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...

Edge AI and Vision Alliance

For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/ Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit. In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing. van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.

みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...

名前です男

Building Production Ready Search Pipelines with Spark and Milvus

Zilliz

UiPath Test Automation using UiPath Test Suite series, part 5

DianaGray10

TrustArc Webinar - 2024 Global Privacy Survey

TrustArc

How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024? In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores. See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe. This webinar will review: - The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey - The top challenges for privacy leaders, practitioners, and organizations in 2024 - Key themes to consider in developing and maintaining your privacy program

Video Streaming: Then, Now, and in the Future

Alpen-Adria-Universität

In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.

Climate Impact of Software Testing at Nordic Testing Days

Kari Kakkonen

My slides at Nordic Testing Days 6.6.2024 Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.

UiPath Test Automation using UiPath Test Suite series, part 6

DianaGray10

Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI. UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities. Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes. What will you get from this session? 1. Insights into integrating generative AI. 2. Understanding how this integration enhances test automation within the UiPath platform 3. Practical demonstrations 4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath Topics covered: What is generative AI Test Automation with generative AI and Open AI. UiPath integration with generative AI Speaker: Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP

RESUME BUILDER APPLICATION Project for students

KAMESHS29

“I’m still / I’m still / Chaining from the Block”

Claudio Di Ciccio

GenAI Pilot Implementation in the organizations

kumardaparthi1024

Removing Uninteresting Bytes in Software Fuzzing

Aftab Hussain

Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process. In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds. - These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.

Full-RAG: A modern architecture for hyper-personalization

Zilliz

Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

Malak Abu Hammad

Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers: * What is Vector Search? * Importance and benefits of vector search * Practical use cases across various industries * Step-by-step implementation guide * Live demos with code snippets * Enhancing LLM capabilities with vector search * Best practices and optimization strategies Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications. #MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

Neo4j

Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.

HCL Notes and Domino License Cost Reduction in the World of DLAU

panagenda

Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/ The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this! We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model. Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward. These topics will be covered - Reducing license cost by finding and fixing misconfigurations and superfluous accounts - How do CCB and CCX licenses really work? - Understanding the DLAU tool and how to best utilize it - Tips for common problem areas, like team mailboxes, functional/test users, etc - Practical examples and best practices to implement right away

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

Neo4j

Recently uploaded (20)

Mind map of terminologies used in context of Generative AI

Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdf

20240605 QFM017 Machine Intelligence Reading List May 2024

“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...

Building Production Ready Search Pipelines with Spark and Milvus

UiPath Test Automation using UiPath Test Suite series, part 5

TrustArc Webinar - 2024 Global Privacy Survey

Video Streaming: Then, Now, and in the Future

Climate Impact of Software Testing at Nordic Testing Days

UiPath Test Automation using UiPath Test Suite series, part 6

RESUME BUILDER APPLICATION Project for students

“I’m still / I’m still / Chaining from the Block”

GenAI Pilot Implementation in the organizations

Removing Uninteresting Bytes in Software Fuzzing

Full-RAG: A modern architecture for hyper-personalization

Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf

GraphSummit Singapore | Graphing Success: Revolutionising Organisational Stru...

HCL Notes and Domino License Cost Reduction in the World of DLAU

GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024

600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014)

1. Half a Billion 600M+ Unsuspecting FreeBSD Users Rick Reed WhatsApp

2. topics history statistics architecture wins challenges

3. rr Joined WhatsApp server team in 2011 Performance, multimedia, security, ops Previously: Yahoo!, SGI, Metaphor, Aion First FreeBSD: 2.2.8 at Yahoo!

4. Why FreeBSD? Previous Yahoo! experience High-quality, high-performance TCP stack Scale Stability

6. Numbers Over 600M monthly users 140M concurrent connections 440k connections per second 1.1M messages sent per second 1B pics sent per day

7. Numbers ~800 servers (incl. 200 chat + 350 mms)

10. Performance SMP scaling is great kernel relatively free of contention Erlang (actor model) BEAM (Erlang VM) very SMP-aware

11.

12.

13.

14.

15. System metrics

16. Hardware (typical) SoftLayer bare-metal 1U/2U/4U Supermicro dual E5-2690v2 (40 threads) (public, private) x 2 lagg x 1g igb (some 10g ix) 64-768GB RAM 1-12 800GB SSD (12x4TB HDD for video) JBOD w/ mobo, Adaptec, and LSI controllers

17. Software Platform FreeBSD 9.1 - 9.3 ufs gmirror for HDD video servers starting to evaluate 10.1

18. Software Platform Erlang R16B01 (w/ our patchset) rtprio patches address specific contention issues schedulers bound to cpus allocators tuned for superpage promotion yaws for web-serving (but disable sendfile)

19. Software Platform stud for SSL termination (proxy via loopback)

20. Data storage Erlang mnesia account data, group membership, etc. in-memory + dump/log persistence Flat-file on hashed directory tree offline mailboxes, media fs layout tuned per workload

21. System metrics 6 x 800GB SSD per offline storage server

22. Versions FreeBSD 7: 2009-2010 good bugs: some

23. Versions FreeBSD 8: 2011-2012 better bugs: zero-copy sockets, igb, tsc tc tuning got us to ~2.8M conns/server

24. Versions FreeBSD 9: 2013- yes! bugs: very, very rare panic enhancements: >2G dirhash size max tcp rexmit interval

25. Tuning kernconf remove unused modules enable debugging/tracing

26. Tuning loader vars igb (numq=8, rx_process_limit=-1, r/txd=4k) kern.nbuf=128k kern.max{files,filesperproc,proc} net.inet.tcp.tcbhashsize=512k net.inet.tcp.hostcache.bucketlimit=1

27. Tuning Network capacity (mbufs, sockets, etc.)

28. Tuning vfs ufs.dirhash_maxmem (patched for >2G) dirty buffer limits & sync delays

29. Issues ufs panics (fs inconsistencies after crash/pfail) bursty syncer behavior filedesc table contention pcb table contention (1.5m igb + 3m loopback)

30. Now vs. then machdep.panic_on_nmi VM pressure

31. Wishlist Multipath TCP Faster large RAM boot

32. Questions?

600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to 600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014)

Similar to 600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014) (20)

More from iXsystems

More from iXsystems (12)

Recently uploaded

Recently uploaded (20)

600M+ Unsuspecting FreeBSD Users (MeetBSD California 2014)