By Andy Wingo.
SDN and Network Programmability Meetup in Barcelona (VI)
21 June 2017
https://www.meetup.com/es-ES/SDN-and-Network-Programmability-Meetup-in-Barcelona
/events/239667457/?eventId=239667457
Возможности интерпретатора Python в NX-OSCisco Russia
The document discusses a webinar presented by Cisco TAC Engineer Anton Tugai about the capabilities of the Python interpreter in NX-OS. Some key points:
- Tugai gave a presentation on trends in Cisco SDN and current solutions.
- The webinar covered an introduction to Python, how Python is integrated into NX-OS, examples, and a demonstration.
- Native Python interpreter is available on Nexus switches starting from certain software versions, allowing Python scripts to run directly on the switch and execute CLI commands.
Monitorama 2015 talk by Brendan Gregg, Netflix. With our large and ever-changing cloud environment, it can be vital to debug instance-level performance quickly. There are many instance monitoring solutions, but few come close to meeting our requirements, so we've been building our own and open sourcing them. In this talk, I will discuss our real-world requirements for instance-level analysis and monitoring: not just the metrics and features we desire, but the methodologies we'd like to apply. I will also cover the new and novel solutions we have been developing ourselves to meet these needs and desires, which include use of advanced Linux performance technologies (eg, ftrace, perf_events), and on-demand self-service analysis (Vector).
Nanog75, Network Device Property as CodeDamien Garros
Device configuration templates have simplified a lot of things for the network industry but many networks are still managing their device properties (aka variables) manually which is very tedious and error prone. This talk will present a new approach to generate and manage network device properties easily using infrastructure as code principles.
About the author: Priya Autee is software engineer at Intel working on various leading edge IA features and Intel(R) RDT expert. She is focused on prototyping and researching open source APIs like DPDK, Intel(R) RDT etc. to support NFV/compute sensitive requirements on Intel Architecture. She holds Masters in Computer Science from Arizona State University, Arizona.
Going FaaSter, Functions as a Service at NetflixYunong Xiao
The document discusses Netflix's use of serverless computing via its own Function as a Service (FaaS) platform. Some key points:
- Netflix built its own FaaS platform called Titus that runs functions at scale using containers for portability and efficiency.
- The platform handles operations concerns so developers can focus on business logic. It provides a full runtime API and handles updates, metrics, and management automatically.
- Netflix developed tools like NEWT to improve the developer experience with one-click setup, local development and debugging, testing, and CI/CD integration for fast and reliable software development.
zebra is an open source implementation as a successor of GNU Zebra and Quagga project. Together with openconfigd, it will work as data plane agnostic Network Operation Stack working with variable protocol / functional modules.
Возможности интерпретатора Python в NX-OSCisco Russia
The document discusses a webinar presented by Cisco TAC Engineer Anton Tugai about the capabilities of the Python interpreter in NX-OS. Some key points:
- Tugai gave a presentation on trends in Cisco SDN and current solutions.
- The webinar covered an introduction to Python, how Python is integrated into NX-OS, examples, and a demonstration.
- Native Python interpreter is available on Nexus switches starting from certain software versions, allowing Python scripts to run directly on the switch and execute CLI commands.
Monitorama 2015 talk by Brendan Gregg, Netflix. With our large and ever-changing cloud environment, it can be vital to debug instance-level performance quickly. There are many instance monitoring solutions, but few come close to meeting our requirements, so we've been building our own and open sourcing them. In this talk, I will discuss our real-world requirements for instance-level analysis and monitoring: not just the metrics and features we desire, but the methodologies we'd like to apply. I will also cover the new and novel solutions we have been developing ourselves to meet these needs and desires, which include use of advanced Linux performance technologies (eg, ftrace, perf_events), and on-demand self-service analysis (Vector).
Nanog75, Network Device Property as CodeDamien Garros
Device configuration templates have simplified a lot of things for the network industry but many networks are still managing their device properties (aka variables) manually which is very tedious and error prone. This talk will present a new approach to generate and manage network device properties easily using infrastructure as code principles.
About the author: Priya Autee is software engineer at Intel working on various leading edge IA features and Intel(R) RDT expert. She is focused on prototyping and researching open source APIs like DPDK, Intel(R) RDT etc. to support NFV/compute sensitive requirements on Intel Architecture. She holds Masters in Computer Science from Arizona State University, Arizona.
Going FaaSter, Functions as a Service at NetflixYunong Xiao
The document discusses Netflix's use of serverless computing via its own Function as a Service (FaaS) platform. Some key points:
- Netflix built its own FaaS platform called Titus that runs functions at scale using containers for portability and efficiency.
- The platform handles operations concerns so developers can focus on business logic. It provides a full runtime API and handles updates, metrics, and management automatically.
- Netflix developed tools like NEWT to improve the developer experience with one-click setup, local development and debugging, testing, and CI/CD integration for fast and reliable software development.
zebra is an open source implementation as a successor of GNU Zebra and Quagga project. Together with openconfigd, it will work as data plane agnostic Network Operation Stack working with variable protocol / functional modules.
Network Automation (Bay Area Juniper Networks Meetup)Alejandro Salinas
Network Automation provides three examples of network automation projects and their learnings:
1. A script to find a host and change its VLAN using Python showed that small, focused scripts are good starting points and don't require extensive systems.
2. Automating a new datacenter configuration using Python templates and YAML files helped manage crises by standardizing cabling and configurations. Permanently improving requires focusing on delivery over systems.
3. Exposing network data through a REST API allowed querying operational status, configurations, and security policies. Sharing information benefits teams and moves beyond just automating the network team's work.
In this session, we’ll review how previous efforts, including Netfilter, Berkley Packet Filter (BPF), Open vSwitch (OVS), and TC, approached the problem of extensibility. We’ll show you an open source solution available within the Red Hat Enterprise Linux kernel, where extending and merging some of the existing concepts leads to an extensible framework that satisfies the networking needs of datacenter and cloud virtualization.
This document discusses Neutron networking status in OpenStack, including features like Distributed Virtual Router (DVR) support. DVR allows distributed routing to remove bottlenecks and enable one-hop east-west traffic between VMs on different hypervisors. The document provides configuration options for enabling DVR and an example multi-node Devstack configuration for testing DVR on compute and network nodes. It also includes diagrams illustrating how DVR works to deliver traffic between VMs on different networks and hypervisors.
Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPFCynthia Thomas
We have introduced Cilium at DockerCon US 2017 this year. Cilium provides application-aware network connectivity, security, and load-balancing for containers. This talk will follow up on the introduction and deep dive into recent kernel developments that address two fundamental questions: How can I provide application-aware security and routing efficiently without overhead embedded into every service? How can container hosts protect themselves from internal and external DDoS attacks? The solutions include:
kproxy: a kernel-based socket proxy which allows for application-aware routing and security enforcement with minimal overhead.
XDP: A lightning-fast packet processing datapath using BPF. The technology is intended for DDoS mitigation, load-balancing, and forwarding.
This talk will deep dive into these exciting technologies and show how Cilium makes BPF and these kernel features available on Linux for your Docker containers.
Using IO Visor to Secure Microservices Running on CloudFoundry [OpenStack Sum...IO Visor Project
As microservices grow, traditional firewall rules based on network ACLs are no longer scalable and fall short of providing fine-grained enforcement. Group Based Policy (GBP) is a flexible policy language that allows users to specify policy enforcement based on intent, independent of network infrastructure and IP addressing. Using micro-segmented virtual domains, administrators can define policies at a centralized location and use IO Visor technology for distributed enforcement. This provides infrastructure independent rules, template-based policy definitions, and scale-out policy enforcement for a solution that secures and scales with microservices. This session will be presented by members of the IO Visor community and will cover how IO Visor technology can be used to define and enforce GBP. The discussion will also cover using GBP for cloud foundry application spaces where microservices are deployed and need scalable, efficient security policies.
Reproducible Computational Pipelines with Docker and Nextflowinside-BigData.com
This document summarizes a presentation about using Docker and Nextflow to create reproducible computational pipelines. It discusses two major challenges in computational biology being reproducibility and complexity. Containers like Docker help address these challenges by creating portable and standardized environments. Nextflow is introduced as a workflow framework that allows pipelines to run across platforms and isolates dependencies using containers, enabling fast prototyping. Examples are given of using Nextflow with Docker to run pipelines on different systems like HPC clusters in a scalable and reproducible way.
FortranCon2020: Highly Parallel Fortran and OpenACC DirectivesJeff Larkin
Fortran has long been the language of computational math and science and it has outlived many of the computer architectures on which it has been used. Modern Fortran must be able to run on modern, highly parallel, heterogeneous computer architectures. A significant number of Fortran programmers have had success programming for heterogeneous machines by pairing Fortran with the OpenACC language for directives-based parallel programming. This includes some of the most widely-used Fortran applications in the world, such as VASP and Gaussian. This presentation will discuss what makes OpenACC a good fit for Fortran programmers and what the OpenACC language is doing to promote the use of native language parallelism in Fortran, such as do concurrent and Co-arrays.
Video Recording: https://www.youtube.com/watch?v=OXZ_Wkae63Y
Kernel advantages for Istio realized with CiliumCynthia Thomas
Istio brings a myriad of options to provide routing rules, encryption, and monitoring for microservices, typically in container environments. Cilium provides accelerated network security using a modern kernel technology called BPF. Put the two together and what do you get? A distributed security solution enabling microservices traffic management, security, and monitoring while enforcing policy as close to the microservices as possible.
Cynthia Thomas and Romain Lenglet discuss the architectural and performance benefits of using Cilium with Istio and provide a demo of this BPF-based, Linux kernel technology. Cilium provides an API-aware security solution that can make a decision on every single microservice flow, with the ability to enforce protocols such as HTTP, Kafka, and gRPC. By addressing security policy at the API layer, you can enforce policy efficiently with kernel capabilities while reducing the attack surface in a microservices deployment.
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...Shirshanka Das
Gobblin is a data integration framework that can handle both batch and streaming data. It provides a logical pipeline specification that is independent of the underlying execution model. Gobblin pipelines can run in both batch and streaming modes using the same system. This allows for cost-efficient batch processing as well as low-latency streaming. The document discusses Gobblin's pipeline specification, deployment options, and roadmap including adding more streaming capabilities and improving security.
Using FLiP with influxdb for edgeai iot at scale 2022Timothy Spann
https://adtmag.com/webcasts/2021/12/influxdata-february-10.aspx?tc=page0
FLiP Stack (Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark) with Influx DB for Edge AI and IoT workloads at scale
Tim Spann
Developer Advocate
StreamNative
datainmotion.dev
Integrating Applications: the Reactive WayNicola Ferraro
This document discusses integrating applications in a reactive way. It begins by defining reactive programming and reactive systems, emphasizing asynchronous, non-blocking architectures. It then discusses application integration patterns like those in Apache Camel, which supports over 200 components and many integration patterns. The document ends by discussing how to build larger reactive systems through microservice integration and communication across reactive and non-reactive applications.
How to Introduce Telemetry Streaming (gNMI) in Your Network with SNMP with Te...InfluxData
This document provides an overview of introducing network telemetry using streaming protocols like gNMI with Telegraf. It discusses gNMI as a streaming telemetry protocol, using Telegraf to collect metrics from network devices via gNMI and SNMP, and how to normalize and enrich the collected data through Telegraf processors before outputting to a time-series database. It also includes a demo of collecting interface counters from devices supporting gNMI and SNMP, and processing the data in Telegraf.
10 ways to shoot yourself in the foot with kubernetes, #9 will surprise you! ...Laurent Bernaille
Kubernetes is a very powerful and complicated system, and many users don’t understand the underlying systems. Come learn how your users can abuse container runtimes, overwhelm your control plane, and cause outages - it’s actually quite easy!
In the last year, we have containerized hundreds of applications and deployed them in large scale clusters (more than 1000 nodes). The journey was eventful and we learned a lot along the way. We’ll share stories of our ten favorite Kubernetes foot guns, including the dangers of cargo culting, rolling updates gone wrong, the pitfalls of initContainers, and nightmarish daemonset upgrades. The talk will present solutions we adopted to avoid or work around some these problems and will finally show several improvements we plan deploy in the future.
Similar to the Kubecon talk with the same title with a few new incidents.
DNS is one of the Kubernetes core systems and can quickly become a source of issues when you’re running clusters at scale. For over a year at Datadog, we’ve run Kubernetes clusters with thousands of nodes that host workloads generating tens of thousands of DNS queries per second. It wasn’t easy to build an architecture able to handle this load, and we’ve had our share of problems along the way.
This talk starts with a presentation of how Kubernetes DNS works. It then dives into the challenges we’ve faced, which span a variety of topics related to load, connection tracking, upstream servers, rolling updates, resolver implementations, and performance. We then show how our DNS architecture evolved over time to address or mitigate these problems. Finally, we share our solutions for detecting these problems before they happen—and identifying misbehaving clients.
DevSecCon London 2019: A Kernel of Truth: Intrusion Detection and Attestation...DevSecCon
Matt Carroll
Infrastructure Security Engineer at Yelp
"Attestation is hard" is something you might hear from security researchers tracking nation states and APTs, but it's actually pretty true for most network-connected systems!
Modern deployment methodologies mean that disparate teams create workloads for shared worker-hosts (ranging from Jenkins to Kubernetes and all the other orchestrators and CI tools in-between), meaning that at any given moment your hosts could be running any one of a number of services, connecting to who-knows-what on the internet.
So when your network-based intrusion detection system (IDS) opaquely declares that one of these machines has made an "anomalous" network connection, how do you even determine if it's business as usual? Sure you can log on to the host to try and figure it out, but (in case you hadn't noticed) computers are pretty fast these days, and once the connection is closed it might as well not have happened... Assuming it wasn't actually a reverse shell...
At Yelp we turned to the Linux kernel to tell us whodunit! Utilizing the Linux kernel's eBPF subsystem - an in-kernel VM with syscall hooking capabilities - we're able to aggregate metadata about the calling process tree for any internet-bound TCP connection by filtering IPs and ports in-kernel and enriching with process tree information in userland. The result is "pidtree-bcc": a supplementary IDS. Now whenever there's an alert for a suspicious connection, we just search for it in our SIEM (spoiler alert: it's nearly always an engineer doing something "innovative")! And the cherry on top? It's stupid fast with negligible overhead, creating a much higher signal-to-noise ratio than the kernels firehose-like audit subsystems.
This talk will look at how you can tune the signal-to-noise ratio of your IDS by making it reflect your business logic and common usage patterns, get more work done by reducing MTTR for false positives, use eBPF and the kernel to do all the hard work for you, accidentally load test your new IDS by not filtering all RFC-1918 addresses, and abuse Docker to get to production ASAP!
As well as looking at some of the technologies that the kernel puts at your disposal, this talk will also tell pidtree-bcc's road from hackathon project to production system and how focus on demonstrating business value early on allowed the organization to give us buy-in to build and deploy a brand new project from scratch.
Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.
ApacheCon 2021 - Apache NiFi Deep Dive 300Timothy Spann
21-September-2021 - ApacheCon - Tuesday 17:10 UTC Apache NIFi Deep Dive 300
* https://github.com/tspannhw/EverythingApacheNiFi
* https://github.com/tspannhw/FLiP-ApacheCon2021
* https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html
* https://github.com/tspannhw/FLiP-IoT
* https://github.com/tspannhw/FLiP-Energy
* https://github.com/tspannhw/FLiP-SOLR
* https://github.com/tspannhw/FLiP-EdgeAI
* https://github.com/tspannhw/FLiP-CloudQueries
* https://github.com/tspannhw/FLiP-Jetson
* https://www.linkedin.com/pulse/2021-schedule-tim-spann/
Tuesday 17:10 UTC
Apache NIFi Deep Dive 300
Timothy Spann
For Data Engineers who have flows already in production, I will dive deep into best practices, advanced use cases, performance optimizations, tips, tricks, edge cases, and interesting examples. This is a master class for those looking to learn quickly things I have picked up after years in the field with Apache NiFi in production.
This will be interactive and I encourage questions and discussions.
You will take away examples and tips in slides, github, and articles.
This talk will cover:
Load Balancing
Parameters and Parameter Contexts
Stateless vs Stateful NiFi
Reporting Tasks
NiFi CLI
NiFi REST Interface
DevOps
Advanced Record Processing
Schemas
RetryFlowFile
Lookup Services
RecordPath
Expression Language
Advanced Error Handling Techniques
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
The document discusses accelerating OpenCV applications using OpenCL on an ARM processor versus using an FPGA. It provides information on OpenCV implementations for OpenCL and FPGA, including SDKs for compiling OpenCL for FPGA boards. A sample OpenCV application for the Zynq FPGA is described that performs image filtering functions like dilation and feature detection in a streaming pipeline. Implementing OpenCV functions optimized for both the ARM CPU and FPGA is suggested to maximize performance. The speaker aims to compare performance of OpenCV accelerated by OpenCL on an ARM processor versus an FPGA implementation.
The document discusses the evolution of Netflix's API architecture from a monolithic Java web server to a microservices architecture using Node.js and containers. It describes how the monolith led to scalability and developer productivity issues. The new architecture uses Node.js scripts in containers with process isolation for improved scalability, availability, and developer experience through rapid local development and debugging. Key aspects of the new architecture include service routing, versioning, operational insights, and container management.
Practical virtual network functions with Snabb (8th SDN Workshop)Igalia
By Andy Wingo.
The Snabb network function toolkit has grown over the last year to include a
number of practical network functions that can be of direct use to network
engineers, from test rigs to packet filtering and capturing tools to core IPv6
transition technology border router implementations. After a brief summary
of what Snabb is, this talk updates the audience on the network functions
included in Snabb, and how they can be easily put to use and even extended to
new uses.
8th SDN Workshop
ETH Zurich, CAB G51
https://www.meetup.com/es-ES/SDN-Switzerland/events/237925842/?eventId=237925842
D. Fast, Simple User-Space Network Functions with Snabb (RIPE 77)Igalia
By Andy Wingo.
Snabb is an open-source toolkit for building fast, flexible network functions. Since its beginnings in 2012, Snabb has seen some modest deployment success ranging from simple one-off diagnosis tools to border routers that process all IPv4 traffic for entire countries. This talk will give an introduction to Snabb. After going over Snabb's fundamental components and how they combine, the talk will move on to examples of how network engineers are taking advantage of Snabb in practice, mentioning a few of the many open-source network functions built on Snabb.
(c) RIPE 77
15 - 19 October 2018
Amsterdam, Netherlands
https://ripe77.ripe.net
Network Automation (Bay Area Juniper Networks Meetup)Alejandro Salinas
Network Automation provides three examples of network automation projects and their learnings:
1. A script to find a host and change its VLAN using Python showed that small, focused scripts are good starting points and don't require extensive systems.
2. Automating a new datacenter configuration using Python templates and YAML files helped manage crises by standardizing cabling and configurations. Permanently improving requires focusing on delivery over systems.
3. Exposing network data through a REST API allowed querying operational status, configurations, and security policies. Sharing information benefits teams and moves beyond just automating the network team's work.
In this session, we’ll review how previous efforts, including Netfilter, Berkley Packet Filter (BPF), Open vSwitch (OVS), and TC, approached the problem of extensibility. We’ll show you an open source solution available within the Red Hat Enterprise Linux kernel, where extending and merging some of the existing concepts leads to an extensible framework that satisfies the networking needs of datacenter and cloud virtualization.
This document discusses Neutron networking status in OpenStack, including features like Distributed Virtual Router (DVR) support. DVR allows distributed routing to remove bottlenecks and enable one-hop east-west traffic between VMs on different hypervisors. The document provides configuration options for enabling DVR and an example multi-node Devstack configuration for testing DVR on compute and network nodes. It also includes diagrams illustrating how DVR works to deliver traffic between VMs on different networks and hypervisors.
Cilium – Kernel Native Security & DDOS Mitigation for Microservices with BPFCynthia Thomas
We have introduced Cilium at DockerCon US 2017 this year. Cilium provides application-aware network connectivity, security, and load-balancing for containers. This talk will follow up on the introduction and deep dive into recent kernel developments that address two fundamental questions: How can I provide application-aware security and routing efficiently without overhead embedded into every service? How can container hosts protect themselves from internal and external DDoS attacks? The solutions include:
kproxy: a kernel-based socket proxy which allows for application-aware routing and security enforcement with minimal overhead.
XDP: A lightning-fast packet processing datapath using BPF. The technology is intended for DDoS mitigation, load-balancing, and forwarding.
This talk will deep dive into these exciting technologies and show how Cilium makes BPF and these kernel features available on Linux for your Docker containers.
Using IO Visor to Secure Microservices Running on CloudFoundry [OpenStack Sum...IO Visor Project
As microservices grow, traditional firewall rules based on network ACLs are no longer scalable and fall short of providing fine-grained enforcement. Group Based Policy (GBP) is a flexible policy language that allows users to specify policy enforcement based on intent, independent of network infrastructure and IP addressing. Using micro-segmented virtual domains, administrators can define policies at a centralized location and use IO Visor technology for distributed enforcement. This provides infrastructure independent rules, template-based policy definitions, and scale-out policy enforcement for a solution that secures and scales with microservices. This session will be presented by members of the IO Visor community and will cover how IO Visor technology can be used to define and enforce GBP. The discussion will also cover using GBP for cloud foundry application spaces where microservices are deployed and need scalable, efficient security policies.
Reproducible Computational Pipelines with Docker and Nextflowinside-BigData.com
This document summarizes a presentation about using Docker and Nextflow to create reproducible computational pipelines. It discusses two major challenges in computational biology being reproducibility and complexity. Containers like Docker help address these challenges by creating portable and standardized environments. Nextflow is introduced as a workflow framework that allows pipelines to run across platforms and isolates dependencies using containers, enabling fast prototyping. Examples are given of using Nextflow with Docker to run pipelines on different systems like HPC clusters in a scalable and reproducible way.
FortranCon2020: Highly Parallel Fortran and OpenACC DirectivesJeff Larkin
Fortran has long been the language of computational math and science and it has outlived many of the computer architectures on which it has been used. Modern Fortran must be able to run on modern, highly parallel, heterogeneous computer architectures. A significant number of Fortran programmers have had success programming for heterogeneous machines by pairing Fortran with the OpenACC language for directives-based parallel programming. This includes some of the most widely-used Fortran applications in the world, such as VASP and Gaussian. This presentation will discuss what makes OpenACC a good fit for Fortran programmers and what the OpenACC language is doing to promote the use of native language parallelism in Fortran, such as do concurrent and Co-arrays.
Video Recording: https://www.youtube.com/watch?v=OXZ_Wkae63Y
Kernel advantages for Istio realized with CiliumCynthia Thomas
Istio brings a myriad of options to provide routing rules, encryption, and monitoring for microservices, typically in container environments. Cilium provides accelerated network security using a modern kernel technology called BPF. Put the two together and what do you get? A distributed security solution enabling microservices traffic management, security, and monitoring while enforcing policy as close to the microservices as possible.
Cynthia Thomas and Romain Lenglet discuss the architectural and performance benefits of using Cilium with Istio and provide a demo of this BPF-based, Linux kernel technology. Cilium provides an API-aware security solution that can make a decision on every single microservice flow, with the ability to enforce protocols such as HTTP, Kafka, and gRPC. By addressing security policy at the API layer, you can enforce policy efficiently with kernel capabilities while reducing the attack surface in a microservices deployment.
Apache Gobblin: Bridging Batch and Streaming Data Integration. Big Data Meetu...Shirshanka Das
Gobblin is a data integration framework that can handle both batch and streaming data. It provides a logical pipeline specification that is independent of the underlying execution model. Gobblin pipelines can run in both batch and streaming modes using the same system. This allows for cost-efficient batch processing as well as low-latency streaming. The document discusses Gobblin's pipeline specification, deployment options, and roadmap including adding more streaming capabilities and improving security.
Using FLiP with influxdb for edgeai iot at scale 2022Timothy Spann
https://adtmag.com/webcasts/2021/12/influxdata-february-10.aspx?tc=page0
FLiP Stack (Apache Flink, Apache Pulsar, Apache NiFi, Apache Spark) with Influx DB for Edge AI and IoT workloads at scale
Tim Spann
Developer Advocate
StreamNative
datainmotion.dev
Integrating Applications: the Reactive WayNicola Ferraro
This document discusses integrating applications in a reactive way. It begins by defining reactive programming and reactive systems, emphasizing asynchronous, non-blocking architectures. It then discusses application integration patterns like those in Apache Camel, which supports over 200 components and many integration patterns. The document ends by discussing how to build larger reactive systems through microservice integration and communication across reactive and non-reactive applications.
How to Introduce Telemetry Streaming (gNMI) in Your Network with SNMP with Te...InfluxData
This document provides an overview of introducing network telemetry using streaming protocols like gNMI with Telegraf. It discusses gNMI as a streaming telemetry protocol, using Telegraf to collect metrics from network devices via gNMI and SNMP, and how to normalize and enrich the collected data through Telegraf processors before outputting to a time-series database. It also includes a demo of collecting interface counters from devices supporting gNMI and SNMP, and processing the data in Telegraf.
10 ways to shoot yourself in the foot with kubernetes, #9 will surprise you! ...Laurent Bernaille
Kubernetes is a very powerful and complicated system, and many users don’t understand the underlying systems. Come learn how your users can abuse container runtimes, overwhelm your control plane, and cause outages - it’s actually quite easy!
In the last year, we have containerized hundreds of applications and deployed them in large scale clusters (more than 1000 nodes). The journey was eventful and we learned a lot along the way. We’ll share stories of our ten favorite Kubernetes foot guns, including the dangers of cargo culting, rolling updates gone wrong, the pitfalls of initContainers, and nightmarish daemonset upgrades. The talk will present solutions we adopted to avoid or work around some these problems and will finally show several improvements we plan deploy in the future.
Similar to the Kubecon talk with the same title with a few new incidents.
DNS is one of the Kubernetes core systems and can quickly become a source of issues when you’re running clusters at scale. For over a year at Datadog, we’ve run Kubernetes clusters with thousands of nodes that host workloads generating tens of thousands of DNS queries per second. It wasn’t easy to build an architecture able to handle this load, and we’ve had our share of problems along the way.
This talk starts with a presentation of how Kubernetes DNS works. It then dives into the challenges we’ve faced, which span a variety of topics related to load, connection tracking, upstream servers, rolling updates, resolver implementations, and performance. We then show how our DNS architecture evolved over time to address or mitigate these problems. Finally, we share our solutions for detecting these problems before they happen—and identifying misbehaving clients.
DevSecCon London 2019: A Kernel of Truth: Intrusion Detection and Attestation...DevSecCon
Matt Carroll
Infrastructure Security Engineer at Yelp
"Attestation is hard" is something you might hear from security researchers tracking nation states and APTs, but it's actually pretty true for most network-connected systems!
Modern deployment methodologies mean that disparate teams create workloads for shared worker-hosts (ranging from Jenkins to Kubernetes and all the other orchestrators and CI tools in-between), meaning that at any given moment your hosts could be running any one of a number of services, connecting to who-knows-what on the internet.
So when your network-based intrusion detection system (IDS) opaquely declares that one of these machines has made an "anomalous" network connection, how do you even determine if it's business as usual? Sure you can log on to the host to try and figure it out, but (in case you hadn't noticed) computers are pretty fast these days, and once the connection is closed it might as well not have happened... Assuming it wasn't actually a reverse shell...
At Yelp we turned to the Linux kernel to tell us whodunit! Utilizing the Linux kernel's eBPF subsystem - an in-kernel VM with syscall hooking capabilities - we're able to aggregate metadata about the calling process tree for any internet-bound TCP connection by filtering IPs and ports in-kernel and enriching with process tree information in userland. The result is "pidtree-bcc": a supplementary IDS. Now whenever there's an alert for a suspicious connection, we just search for it in our SIEM (spoiler alert: it's nearly always an engineer doing something "innovative")! And the cherry on top? It's stupid fast with negligible overhead, creating a much higher signal-to-noise ratio than the kernels firehose-like audit subsystems.
This talk will look at how you can tune the signal-to-noise ratio of your IDS by making it reflect your business logic and common usage patterns, get more work done by reducing MTTR for false positives, use eBPF and the kernel to do all the hard work for you, accidentally load test your new IDS by not filtering all RFC-1918 addresses, and abuse Docker to get to production ASAP!
As well as looking at some of the technologies that the kernel puts at your disposal, this talk will also tell pidtree-bcc's road from hackathon project to production system and how focus on demonstrating business value early on allowed the organization to give us buy-in to build and deploy a brand new project from scratch.
Apache Kafka sits at the core of the modern scalable event driven architecture. It’s no longer used only as logging infrastructure, but as a core component in thousands of companies around the world. It has the unique capability to provide low-latency, fault-tolerant pipeline at scale that is very important for today’s world of big data. During this talk we’ll see what makes Apache Kafka perfect for the job. We’ll explore how to optimize it for throughput or for durability. And we’ll also go over the messaging semantics it provides. Last but not least, we’ll see how Apache Kafka can help us solve some everyday problems that we face when we build large scale systems in an elegant way.
ApacheCon 2021 - Apache NiFi Deep Dive 300Timothy Spann
21-September-2021 - ApacheCon - Tuesday 17:10 UTC Apache NIFi Deep Dive 300
* https://github.com/tspannhw/EverythingApacheNiFi
* https://github.com/tspannhw/FLiP-ApacheCon2021
* https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html
* https://github.com/tspannhw/FLiP-IoT
* https://github.com/tspannhw/FLiP-Energy
* https://github.com/tspannhw/FLiP-SOLR
* https://github.com/tspannhw/FLiP-EdgeAI
* https://github.com/tspannhw/FLiP-CloudQueries
* https://github.com/tspannhw/FLiP-Jetson
* https://www.linkedin.com/pulse/2021-schedule-tim-spann/
Tuesday 17:10 UTC
Apache NIFi Deep Dive 300
Timothy Spann
For Data Engineers who have flows already in production, I will dive deep into best practices, advanced use cases, performance optimizations, tips, tricks, edge cases, and interesting examples. This is a master class for those looking to learn quickly things I have picked up after years in the field with Apache NiFi in production.
This will be interactive and I encourage questions and discussions.
You will take away examples and tips in slides, github, and articles.
This talk will cover:
Load Balancing
Parameters and Parameter Contexts
Stateless vs Stateful NiFi
Reporting Tasks
NiFi CLI
NiFi REST Interface
DevOps
Advanced Record Processing
Schemas
RetryFlowFile
Lookup Services
RecordPath
Expression Language
Advanced Error Handling Techniques
Tim Spann is a Developer Advocate @ StreamNative where he works with Apache NiFi, Apache Pulsar, Apache Flink, Apache MXNet, TensorFlow, Apache Spark, big data, the IoT, machine learning, and deep learning. Tim has over a decade of experience with the IoT, big data, distributed computing, streaming technologies, and Java programming. Previously, he was a Principal Field Engineer at Cloudera, a senior solutions architect at AirisData and a senior field engineer at Pivotal. He blogs for DZone, where he is the Big Data Zone leader, and runs a popular meetup in Princeton on big data, the IoT, deep learning, streaming, NiFi, the blockchain, and Spark. Tim is a frequent speaker at conferences such as IoT Fusion, Strata, ApacheCon, Data Works Summit Berlin, DataWorks Summit Sydney, and Oracle Code NYC. He holds a BS and MS in computer science.
The document discusses accelerating OpenCV applications using OpenCL on an ARM processor versus using an FPGA. It provides information on OpenCV implementations for OpenCL and FPGA, including SDKs for compiling OpenCL for FPGA boards. A sample OpenCV application for the Zynq FPGA is described that performs image filtering functions like dilation and feature detection in a streaming pipeline. Implementing OpenCV functions optimized for both the ARM CPU and FPGA is suggested to maximize performance. The speaker aims to compare performance of OpenCV accelerated by OpenCL on an ARM processor versus an FPGA implementation.
The document discusses the evolution of Netflix's API architecture from a monolithic Java web server to a microservices architecture using Node.js and containers. It describes how the monolith led to scalability and developer productivity issues. The new architecture uses Node.js scripts in containers with process isolation for improved scalability, availability, and developer experience through rapid local development and debugging. Key aspects of the new architecture include service routing, versioning, operational insights, and container management.
Practical virtual network functions with Snabb (8th SDN Workshop)Igalia
By Andy Wingo.
The Snabb network function toolkit has grown over the last year to include a
number of practical network functions that can be of direct use to network
engineers, from test rigs to packet filtering and capturing tools to core IPv6
transition technology border router implementations. After a brief summary
of what Snabb is, this talk updates the audience on the network functions
included in Snabb, and how they can be easily put to use and even extended to
new uses.
8th SDN Workshop
ETH Zurich, CAB G51
https://www.meetup.com/es-ES/SDN-Switzerland/events/237925842/?eventId=237925842
D. Fast, Simple User-Space Network Functions with Snabb (RIPE 77)Igalia
By Andy Wingo.
Snabb is an open-source toolkit for building fast, flexible network functions. Since its beginnings in 2012, Snabb has seen some modest deployment success ranging from simple one-off diagnosis tools to border routers that process all IPv4 traffic for entire countries. This talk will give an introduction to Snabb. After going over Snabb's fundamental components and how they combine, the talk will move on to examples of how network engineers are taking advantage of Snabb in practice, mentioning a few of the many open-source network functions built on Snabb.
(c) RIPE 77
15 - 19 October 2018
Amsterdam, Netherlands
https://ripe77.ripe.net
Snabb - A toolkit for user-space networking (FOSDEM 2018)Igalia
By Diego Pino.
Snabb is a toolkit for developing user-space network functions. A network function (filtering, NAT, encapsulation) is any program that manipulates network traffic. Snabb eases the effort to write such programs. Snabb fits in the category of user-space networking. Snabb by-passes the Linux kernel talking directly to the hardware. This makes Snabb a very convenient tool for high-performance networking. Unlike other user-space toolkits such as DPDK or VPP, Snabb is entirely developed in Lua which significantly lowers the adoption barrier.
In this talk I introduce the Snabb toolkit. Through real-world examples you will learn how Snabb works and even how to start prototyping your own network functions.
(c) FOSDEM 2018
Brussels, 3 & 4 February 2018
https://fosdem.org/2018/schedule/event/snabb/
Stacks and Layers: Integrating P4, C, OVS and OpenStackOpen-NFP
This document discusses integrating programmable packet processing (P4), traditional software (C), and hardware acceleration using Agilio SmartNICs with OpenStack networking. It reviews traditional OpenStack networking options and their performance issues. It then discusses how P4, C extensions, and SmartNICs can provide flexible, high-performance networking by offloading or extending the OpenStack networking datapaths like OVS and Contrail vRouter. Examples are provided of running P4/C firmware on the SmartNIC to implement a virtual switch or extending existing software. Integration with OpenStack and implications are discussed throughout.
The Swiss ISP SWITCH has developed a scalable IPFIX exporter built using Snabb.
In 2022 the application gained many new features, and was upstreamed into the
main Snabb repository. We will showcase a production-grade Snabb application,
and discuss implementation challenges and how Snabb helps you deal with them.
(c) FOSDEM 2023
4 & 5 February 2023
https://fosdem.org/2023/schedule/event/network_snabbflow_ipfix/
Production high-performance networking with Snabb and LuaJIT (Linux.conf.au 2...Igalia
By Andy Wingo.
It used to be that to set up a serious network, you needed to stock racks and racks with specialized proprietary single-purpose boxes. This was because only specialized hardware could handle the hundreds of gigabits per second that might flow through any given box.
Things have changed. With the rise of cheap commodity Xeon-based servers and widespread availability of 10 gigabit network cards, an off-the-shelf server with a few NICs can now handle the workload. The age of open source software-driven routers is fully here -- but it doesn't look like what we thought it would, 10 years ago.
We thought it would be Linux everywhere, but it turns out that Linux's networking stack is just too slow. To get around this problem, modern high-speed software switches bypass the kernel entirely, instead booting network cards and handling traffic entirely from user-space. The up-side of this is that now we have the possibility of using pleasant, hackable, open source, standalone software stacks to deliver network applications that are tailored to specific needs.
This talk presents Snabb, a toolkit for building user-space network functions. Snabb is entirely written in the expressive Lua language, minimizing the amount of code that you have to write to get stuff done. Snabb specifically uses the LuaJIT implementation of Lua, giving us excellent code generation as
well as efficient access to low-level binary data and AVX2 assembly generation.
Snabb's goal is to be "rewritable software": software that's so simple that you could explain it to someone and they could write their own. By the end of the presentation, you too should have this feeling.
We will also describe how Snabb is used in practice in major telecoms and ISPs to provide IPv6 transition technologies to entire countries. Using Snabb allowed a small team of open-source hackers to ship a product that competed favorably
against offerings from traditional network vendors.
(c) linux.conf.au 2017, CC-BY-SA
Hobart, 16-20 January 2017
https://linux.conf.au
This project report describes the development of an application on the DaVinci platform under the guidance of Prof. TK Dan. Akash Sahoo and Abhijit Tripathy, 7th semester B.Tech students, developed an application to take advantage of the DaVinci's integrated ARM and TMS320C64x+ DSP cores. They ported MontaVista Linux and DSP/BIOS to the DaVinci evaluation module board to enable the application and provide OS support across the hybrid processor system.
Exploring the Final Frontier of Data Center Orchestration: Network Elements -...Puppet
The document discusses network element automation using Puppet. It provides context on the challenges of manual network configuration including lack of agility, reliability issues from errors, and time spent on basic tasks. Puppet can automate network elements similar to how it automates servers, reducing errors and improving speed/productivity. The Cisco Nexus platform and NXAPI enable programmatic access for automation using Puppet through technologies like onePK and LXC containers running on the switch.
Kubernetes is exploding in popularity right now and has all the buzz and cargo-culting that Docker enjoyed just a few years ago. But what even is Kubernetes? How do I run my PHP apps in it? Should I run my PHP apps in it ?
Engineering software is widely employed for its powerful abstraction of scientific and technical knowledge. It enables productive applications, e.g., analysis, prototyping, and manufacturing. Making engineering software requires a profound understanding in the problem domain, as well as the art of engineering it.
Software engineering differs substantially from conventional engineering. To professionally build software, mathematicians, scientists, and engineers need skills including system administration, automatic build, automatic testing, version control, to name but a few. Computer science knowledge like algorithms and data structures is also indispensable. It is a joyful, interdisciplinary, and world-changing enterprise worth sharing with all future engineering practitioners.
BKK16-409 VOSY Switch Port to ARMv8 Platforms and ODP IntegrationLinaro
Virtual Open Systems has developed VOSYSwitch, a high-performance user space networking virtual switch solution enabling NFV, based on the open source packet processing framework SnabbSwitch. In this talk, the experience of porting VOSYSwitch from x86 to ARMv8 will be shared, along with the integration of ODP as a driver layer for the available hardware resources. In addition to this presentation, a live demonstration will showcase chained VNFs connected through VOSYSwitch, where an OpenFastPath web server is implemented behind an ODP enabled packet filtering firewall. The targeted platforms are Freescale (NXP) LS2085A and Cavium's ThunderX.
LibOS as a regression test framework for Linux networking #netdev1.1Hajime Tazaki
This document describes using the LibOS framework to build a regression testing system for Linux networking code. LibOS allows running the Linux network stack in a library, enabling deterministic network simulation. Tests can configure virtual networks and run network applications and utilities to identify bugs in networking code by detecting changes in behavior across kernel versions. Example tests check encapsulation protocols like IP-in-IP and detect past kernel bugs. Results are recorded in JUnit format for integration with continuous integration systems.
XPDDS18: CPUFreq in Xen on ARM - Oleksandr Tyshchenko, EPAM SystemsThe Linux Foundation
The motivation of hypervisor based CPUFreq is to enable the one of the main PM use-cases (Dynamic voltage and frequency scaling) in virtualized system powered by Xen hypervisor. Rationale behind this activity is that CPU virtualization is done by hypervisor and the guest OS doesn't actually know anything about physical CPUs because it is running on virtual CPUs.
In this talk Oleksandr will briefly describe the possible approach of generic CPUFreq in Xen on ARM, the advantages and disadvantages of having DVFS support on ARM boards powered by Xen hypervisor and share results of his CPUFreq PoC which implies power consumption measurements with and without CPUFreq enabled on R-Car Gen3 based board as an example.
This document provides an overview and introduction to Node.js. It covers the basics of Node.js including setting up the environment, creating a first application, using the Node Package Manager (NPM), and an introduction to key concepts like asynchronous programming with callbacks and events. The course appears to be targeted at web developers and teaches additional frameworks that can be used with Node.js like Express.js, MongoDB, and Angular.js.
Intro to open source telemetry linux con 2016Matthew Broberg
Abstract
As part of the team delivering Snap, an open telemetry framework, I've run through dozens of use cases where gathering disparate metrics from services can roll up into meaningful diagrams for operations engineers and developers alike. We will use Snap's plugin model to collect, process and publish these measurements into meaningful graphs using open source tools. By joining this session, you can follow along and install industry-standard open source projects, deploy them and then use Snap to collect, process and visualize these metrics.
Audience
Anyone with an operations-background (or future ahead of them) that wants to see the breadth of available open source tooling around telemetry. This proposal is designed for the hands-on user, who is comfortable running containers or virtual machines locally.
Experience Level
Intermediate
Benefits to the Ecosystem
By joining this session, you can follow along and install industry-standard open source projects, deploy them and then use Snap to collect, process and visualize these metrics. This empowers users within the Linux ecosystem to see their knowledge as powerful when visualized next to other layers of the datacenter.
This document provides an update on PGI compilers and tools for heterogeneous supercomputing. It discusses PGI's support for OpenACC directives to accelerate applications on multicore CPUs and NVIDIA GPUs from a single source. It highlights new compiler features including support for Intel Skylake, AMD EPYC and IBM POWER9 CPUs as well as NVIDIA Volta GPUs. Benchmark results show strong performance of OpenACC applications on these platforms. The document also discusses the growing adoption of OpenACC in HPC applications and resources available to support OpenACC development.
The document discusses how to remotely update IoT devices using Eclipse hawkBit and SWUpdate. It provides an overview of the Android approach to OTA updates, which uses a recovery OS to install updates atomically. It then describes how SWUpdate can be used as an agent on embedded Linux devices to manage updates similarly to Android. Key points covered include SWUpdate's architecture, features like local/remote interfaces and update file format/security, and how it can be integrated with hawkBit for remote management of software updates.
Tutorial to setup OpenStreetMap tileserver with customized boundaries of IndiaArun Ganesh
Step by step instructions on setting up a custom OpenStreetMap tileserver and customizing the borders of India to display it as per the official map of India.
Source: Shatrughan Saxena https://github.com/azaadshatru/OSM
Similar to Practical virtual network functions with Snabb (SDN Barcelona VI) (20)
A Year of the Servo Reboot: Where Are We Now?Igalia
Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry.
(c) Embedded Open Source Summit 2024
April 16-18, 2024
Seattle, Washington (US)
https://events.linuxfoundation.org/embedded-open-source-summit/
https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia
Building End-user Applications on Embedded Devices with WPEIgalia
The Web engine is the most important component of a Web Browser, enabling
developers to harness the power of the Web Platform to build their
applications. However, Web Browsers are not the only type of applications that
can be built with Web Engines, which can also be used to develop other types of
applications using the same Web-based technologies, but for a different type of
use cases other than "browsing the Web".
These use cases can cover a wide range of situations outside of the traditional
desktop or mobile environments, such as the ones embedded systems are usually
used for (e.g. set-top-boxes, smart home appliances, GPS navigation devices, or
in-car/in-flight infotainment systems, to name a few). And in those situations
it is very common to be running on boards with SoCs and a particular set of HW
capabilities that make it crucial for the Web Engine to be able to tightly
integrate with them.
In this session we will focus on how WPE, a fully Open Source port of the
WebKit Web engine for Linux-based embedded devices, can be used to adapt to the
different challenges that embedded devices pose to develop end-user
applications, using the power of the Web Platform underneath.
(c) Embedded Open Source Summit 2024
April 16-18, 2024
Seattle, Washington (US)
https://events.linuxfoundation.org/embedded-open-source-summit/
https://eoss24.sched.com/event/1aNTr/building-end-user-applications-on-embedded-devices-with-wpe-mario-sanchez-prada-igalia
Raspberry Pi 5: Challenges and Solutions in Bringing up an OpenGL/Vulkan Driv...Igalia
The Raspberry Pi 5 was announced on October 2023. This new version of the
popular embedded device comes with a new iteration of Broadcom’s VideoCore GPU
platform, and was released with a fully open source driver stack, developed by
Igalia. The presentation will discuss some of the major changes required to
support this new Video Core iteration, the challenges we faced in the process
and the solutions we provided in order to deliver conformant OpenGL ES and
Vulkan drivers. The talk will also cover the next steps for the open source
Raspberry Pi 5 graphics stack.
(c) Embedded Open Source Summit 2024
April 16-18, 2024
Seattle, Washington (US)
https://events.linuxfoundation.org/embedded-open-source-summit/
https://eoss24.sched.com/event/1aBEx
Automated Testing for Web-based Systems on Embedded DevicesIgalia
Every day, embedded devices are becoming more powerful and capable of running
more elaborate applications. Among these applications are Web-based ones,
enabling to leverage features from the Web APIs to the embedded context, either
through a generic browser running a traditional Web application or through a
customized Web engine tightly integrated within the system.
But such capabilities usually bring new challenges, like testing user
interactions with the application using the embedded device's specific I/O
methods, such as gestures, or inspecting Web application internals with
JavaScript. In this context, using a browser automation framework such as
WebDriver, which is a W3C standard supported by WebKit Web engine, allows
testing Web-based applications on such devices as if the user were actually
using it, alongside running custom JS code.
In this session, we will cover why we need browser automation for testing on
certain types of embedded devices, with a focus on WebDriver as the proposed
tool to achieve that goal. We will also discuss WebDriver's main features and
limitations, as well as other possible approaches and frameworks that could be
considered for this kind of task.
(c) Embedded Open Source Summit 2024
April 16-18, 2024
Seattle, Washington (US)
https://events.linuxfoundation.org/embedded-open-source-summit/
https://eoss24.sched.com/event/1aeSx/automated-testing-for-web-based-systems-on-embedded-devices-lauro-moura-igalia
Embedding WPE WebKit - from Bring-up to MaintenanceIgalia
Embedded devices have become powerful enough to run Web content a decade ago,
and any modern SoC that can run Linux and includes a GPU is a potential
candidate to hide a Web engine under the surface. How did it made it there?
Does it only show Web content? What else can it do? The talk will cover
bring-up tips to build and get WPE WebKit working on your custom embedded
device and make your own simple Web browser, as well as the best practices for
keeping the system up to date. No less important is integration with the rest
of the system: this session will detail the possibilities that WebKit brings to
the table, including how to add new JavaScript APIs which call into native code
to provide tight, performant access to platform functionality.
(c) Embedded Open Source Summit 2024
April 16-18, 2024
Seattle, Washington (US)
https://events.linuxfoundation.org/embedded-open-source-summit/
https://eoss24.sched.com/event/1aBFQ/embedding-wpe-webkit-from-bring-up-to-maintenance-adrian-perez-de-castro-igalia
This talk dives into how the scheduler impacts your gameplay on Linux and
unveils our journey to smoother gameplay. How does task scheduling impact Linux
gaming? Suboptimal task scheduling can cause stuttering while playing games on
the Steam Deck game console. First, we nail down the enemy. What exactly is
"stuttering," and how can we measure its impact on your gameplay? Next, we
extensively analyzed the characteristics of game tasks from the scheduler’s
point of view. Characterizing task behavior in Linux gaming helps to understand
why some schedulers create much stuttering and others create less and to unveil
the secrets behind smooth vs. choppy performance. Lastly, we will share our
progress on the optimized scheduler for reducing the stuttering problems in
Linux gaming, especially Steam Deck. We implemented the scheduling policy based
on sched_ext, a BPF-based extensible scheduling framework.
(c) Open Source Summit North America 2024
April 16-18, 2024
Seatle, Washington (US)
https://events.linuxfoundation.org/open-source-summit-north-america/
https://ossna2024.sched.com/event/1aBOT/optimizing-scheduler-for-linux-gaming-changwoo-min-igalia
So, we are adding a backend for the SpiderMonkey’s codegen to enable JIT
support for JavaScript running through Wasm. Sounds a bit cryptic so let’s
divide it into parts.
SpiderMonkey is a JavaScript engine which is used for running JavaScript inside
the Firefox browser. SpiderMonkey is written in C++ and supports compilation
into the Wasm module, see live demo -
https://mozilla-spidermonkey.github.io/sm-wasi-demo/. However, SpiderMonkey
compiled into the Wasm module supports execution of JavaScript only in the
interpreter-only mode and it doesn’t support just-in-time compilation because
there is no Wasm backend for that. There are backends for Arm, X86, X64 etc but
there is none for Wasm.
Why do we want to add support for JIT? Well, because we want speed. Right now
there is no solution to run JS scripts via Wasm fast, there are only
interpreters.
Why does JIT improve performance?
The reasons are the same for why an interpreter is slower than a compiler -
because it eliminates the interpreter loop, uses a more efficient ABI and, more
importantly, it can specialize polymorphic operations in JavaScript. So, we not
only enable the JIT tier in SpiderMonkey for Wasm but we also provide support
for inline caches.
Inline caches is a mechanism for specializing the behavior of particular
operations like plus or a call to specific arguments provided at runtime.
With all that we can generate Wasm modules on the fly, instantiate them, and
link them to provide from ~2x to ~11x speedup over the interpreter. In the
talks we will cover how the whole scheme works with SpiderMonkey: 1. How to
link modules on the fly into SpiderMonkey.wasm 2. How to add an exotic Wasm
backend into SpiderMonkey’s supported backend line - X64, X86, Arm, Wasm 3. How
to use the whole solution in the cloud instead of QuickJS 4. How to get a
speedup of your JS over wasm with test data.
Wasm I/O 2024
14 - 15 Mar, 2024
Barcelona
https://2024.wasmio.tech/
To crash or not to crash: if you do, at least recover fast!Igalia
What could be possibly worse that an almost unbeatable boss in
a game or a tough maze that consume hours of gameplay with not
much progress? How about a Linux kernel crash that makes you
lose all the game progress with no apparent reason or feedback?
Though rare, it is a real possibility that would make gamers
quite annoyed, given that Linux is used more and more as a
platform for playing games.
Some technologies are available to collect logs and feedback
the user in case such disastrous events happen, mostly related
with kernel crashes handling mechanisms. The main ones available
are kdump and pstore, but still there are work to be done in
this area...
In this talk we're going to present the basics about kernel
crash handling, like how a kernel panic might happen, how to
deal with that (with an overall discussion about kdump and
pstore techs) and the kdumpst tool, developed specially to
deal with this situation on Steam Deck (and generically on
Arch Linux); also we're gonna discuss some missing
pieces / ideas to make it even less likely gamers need to
complain that their device just got hang for no reason!
FOSForums 2023
Aug 26 - Aug 27, 2023
Institute of Computing, State University of Campinas (Unicamp)
Campinas, São Paulo, Brazil
https://www.fosforums.org/
Introducción a Mesa. Caso específico dos dispositivos Raspberry Pi por IgaliaIgalia
Nesta charla impartida por Alejandro Piñeiro de Igalia, darase unha introdución
a Mesa, librería open-source para o desenvolvemento de drivers gráficos.
Explicarase a súa historia, os seus compoñentes máis importantes, que
utilidades proporcionan aos desenvolvedores e unha lista de hardware ás que dan
soporte. Finalmente explicarase o caso concreto do soporte proporcionado para
as GPUs dos dispositivos da serie Raspberry Pi, centrándonos nas Raspberry Pi 4
e Raspberry Pi 5
Igalia é unha empresa galega, con sede na Coruña, especializada en servizos de
consultoría, e que desenvolve solucións innovadoras de código aberto para un
gran conxunto de plataformas de software e hardware. En Igalia traballan nas
áreas máis interesantes do software de código aberto, incluídos navegadores,
gráficos e multimedia.
Igalia desenvolveu os controladores OpenGL ES 3.1 e Vulkan 1.2 conformes para a
GPU VideoCore VII Broadcom que se fornece coa nova Raspberry Pi 5.
Alejandro Piñeiro é enxeñeiro de Software e socio en Igalia, é desenvolvedor de
Software Libre desde 2004. A súa experiencia inclúe unha variedade de proxectos
de GNOME e freedesktop.org, enfocándose desde 2015 en Mesa, especificamente os
drivers Intel e Broadcom. É un dos responsables do desenvolvemento do
controlador Broadcom Vulkan para Raspberry Pi 4 & 5.
Máis información en https://aindustriosa.org/Mesa/
Esta actividade está patrocinada pola Xunta de Galicia e pola Axencia Para a
Modernización Tecnolóxica (AMTEGA).
(c) A Industriosa
https://aindustriosa.org
28 de Outubro (Vigo)
Chimera Linux is a novel Linux distribution built around FreeBSD core tools and
the LLVM toolchain. Since its initial launch in 2021, it has made a lot of
progress and is now in alpha stage. The system can be deployed on a wide array
of hardware and many people are using it as their desktop system; it works on
x86_64, AArch64, POWER (little and big endian) as well as RISC-V and by now
comes with thousands of packages.
While trying to be practical, Chimera is also highly hardened, partly thanks to
the LLVM toolchain, rendering it immune to various security issues other
distros are vulnerable to. It has transparent and robust infrastructure,
ensuring smooth deployment of packages. We are also developing various new
tooling that the whole ecosystem can benefit from, including the Turnstile
session tracker. Service management is based around Dinit, a modern,
supervising system; we maintain and create a variety of tooling around it,
trying to break the existing status quo with systemd, while abandoning legacy
approaches.
2023 has seen several major milestones, so I will focus on these, while also
giving a short overview so that people unfamiliar with the system don't feel
lost. I will also explain how our work benefits the entire Linux ecosystem, as
well as beyond.
(c) FOSDEM 2024
3 & 4 February 2024
https://fosdem.org/2024/schedule/event/fosdem-2024-2524-2023-in-chimera-linux/
For the last 3 years, I've been building a complete Linux distribution, Chimera
Linux (https://chimera-linux.org) using solely LLVM as its system toolchain -
that means Clang, compiler-rt, and libc++, alongside its other tooling. Right
now, it is a complete desktop system that is already used by many, with a
familiar GNOME interface and thousands of packages, targeting 5 CPU
architectures. In this talk I would like to focus on my experiences using the
toolchain, what obstacles got in the way, how I dealt with them, the issues
that are still left and I would like to see addressed, the many benefits using
LLVM gave the project, and overall give the audience an insight into practical
deployment of LLVM in a project where it isn't simply a drop-in alternative to
GCC.
(c) FOSDEM 2024
3 & 4 February 2024
https://fosdem.org/2024/schedule/event/fosdem-2024-2555-building-a-linux-distro-with-llvm/
turnip: Update on Open Source Vulkan Driver for Adreno GPUsIgalia
Turnip changed a lot since the last status update. You could now run AAA
desktop games via FEX + Turnip, Adreno 7xx is now supported, Turnip is used by
emulators on Android, and more!
(c) FOSDEM 2024
3 & 4 February 2024
https://fosdem.org/2024/schedule/event/fosdem-2024-2033-turnip-update-on-open-source-vulkan-driver-for-adreno-gpus/
Graphics stack updates for Raspberry Pi devicesIgalia
This talk will show the efforts done in the Open-Source graphics stack for
supporting Raspberry Pi devices. Although the talk will focus on the recently
launched new Raspberry Pi 5, we will show the improvements done for previous
generations of the Raspberry Pi hardware.
Raspberry Pi 5 has available FLOSS GPU drivers on product launch, exposing
OpenGL-ES 3.1 and Vulkan 1.2. We'll go through the changes needed to enable
desktop OpenGL 3.1 on RPi4/5.
We will also review the changes done to the kernel driver to expose the RPi5
capabilities and the new GPU stats support for RPi4/5.
Finally, we will show the work done to use Wayfire as the default Wayland
compositor on the Raspberry Pi OS.
- https://www.mesa3d.org/
- https://www.raspberrypi.com/
- https://wayfire.org/
(c) FOSDEM 2024
3 & 4 February 2024
https://fosdem.org/2024/schedule/event/fosdem-2024-2841-graphics-stack-updates-for-raspberry-pi-devices/
Delegated Compositing - Utilizing Wayland Protocols for Chromium on ChromeOSIgalia
This talk will cover our experience in utilizing Wayland subsurfaces and
implementing delegated compositing for Chromium on ChromeOS. Several concepts
will be covered - from overlay making decision in Chromium/Viz to design and
implementation of custom Wayland protocols, which were required to pass frame
data as overlays via Wayland and reconstruct that frame on the Wayland server
side.
(c) FOSDEM 2024
3 & 4 February 2024
https://fosdem.org/2024/schedule/event/fosdem-2024-3177-delegated-compositing-utilizing-wayland-protocols-for-chromium-on-chromeos/
MessageFormat: The future of i18n on the webIgalia
Internationalization in JavaScript and on the web platform is very complicated,
but also vastly important for us developers in order to build accessible and
intelligible interfaces. Thankfully, Unicode Consortium's MessageFormat working
group and TC39 have been hard at work standardizing the next generation of i18n
tooling that aims to unify analogous non-standard tools in use today while
approaching this problem from a fresh perspective.
Join me along this tour of i18n in JavaScript, discover some of the newest
additions to the toolkit and learn about the ongoing MessageFormat proposal and
how it aims to radically improve the developer experience.
(c) FOSDEM 2024
3 & 4 February 2024
https://fosdem.org/2024/schedule/event/fosdem-2024-2832-messageformat-the-future-of-i18n-on-the-web/
Replacing the geometry pipeline with mesh shadersIgalia
This talk will discuss the problems with the traditional vertex processing
pipeline and present how mesh shading solves these problems. Instead of
processing a fixed set of input vertices, mesh shaders can create an arbitrary
topology of vertices and primitives. Mesh shading also includes a new solution
for geometry amplification: task shaders.
The talk should be scheduled before Timur's talk about implementing mesh
shaders in the RADV Mesa driver.
(c) X.Org Developer Conference (XDC) 2022
October 4-6, 2022
Minneapolis, Minnesota, USA
https://indico.freedesktop.org/event/2/
Let's talk about developing AMD display drivers in the DRM subsystem as an
external contributor. Part of this work is a trail of breadcrumbs to build
documentation. What are those breadcrumbs? How do they help to review, fix,
improve and enable features of AMD drivers? How would both sides benefit if
those pieces of information were already documented? We are gathering
information from anywhere and also bothering experts for input. Ultimately,
this presentation focuses on AMD driver development but may fit DRM drivers of
any GPU vendors.
(c) X.Org Developer Conference (XDC) 2022
October 4-6, 2022
Minneapolis, Minnesota, USA
https://indico.freedesktop.org/event/2/
There has been a lot of activity in V3DV, the Vulkan driver for Raspberry Pi 4,
over the last year: we have significantly reworked our synchronization code,
obtained Vulkan 1.1 conformance, implemented Vulkan 1.2 support, continued to
work on compiler optimizations and more.
In this talk I would like to go through the main development milestones and
changes we implemented in the driver as well as discussing some limitations of
the underlying hardware platform that have discouraged us from implementing
features such as scalar block layout or fp16.
(c) X.Org Developer Conference (XDC) 2022
October 4-6, 2022
Minneapolis, Minnesota, USA
https://indico.freedesktop.org/event/2/
Skybuffer AI: Advanced Conversational and Generative AI Solution on SAP Busin...Tatiana Kojar
Skybuffer AI, built on the robust SAP Business Technology Platform (SAP BTP), is the latest and most advanced version of our AI development, reaffirming our commitment to delivering top-tier AI solutions. Skybuffer AI harnesses all the innovative capabilities of the SAP BTP in the AI domain, from Conversational AI to cutting-edge Generative AI and Retrieval-Augmented Generation (RAG). It also helps SAP customers safeguard their investments into SAP Conversational AI and ensure a seamless, one-click transition to SAP Business AI.
With Skybuffer AI, various AI models can be integrated into a single communication channel such as Microsoft Teams. This integration empowers business users with insights drawn from SAP backend systems, enterprise documents, and the expansive knowledge of Generative AI. And the best part of it is that it is all managed through our intuitive no-code Action Server interface, requiring no extensive coding knowledge and making the advanced AI accessible to more users.
Digital Marketing Trends in 2024 | Guide for Staying AheadWask
https://www.wask.co/ebooks/digital-marketing-trends-in-2024
Feeling lost in the digital marketing whirlwind of 2024? Technology is changing, consumer habits are evolving, and staying ahead of the curve feels like a never-ending pursuit. This e-book is your compass. Dive into actionable insights to handle the complexities of modern marketing. From hyper-personalization to the power of user-generated content, learn how to build long-term relationships with your audience and unlock the secrets to success in the ever-shifting digital landscape.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Nunit vs XUnit vs MSTest Differences Between These Unit Testing Frameworks.pdfflufftailshop
When it comes to unit testing in the .NET ecosystem, developers have a wide range of options available. Among the most popular choices are NUnit, XUnit, and MSTest. These unit testing frameworks provide essential tools and features to help ensure the quality and reliability of code. However, understanding the differences between these frameworks is crucial for selecting the most suitable one for your projects.
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Taking AI to the Next Level in Manufacturing.pdfssuserfac0301
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfMalak Abu Hammad
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
3. the
domain
Commodity hardware is capable of
high-performance networking
1 core at 15MPPS: 65ns/packet❧
What software to put on the
hardware?
4. alternate
(hi)story
The teleology of open source: “one
day this will all run Linux”
Conventional wisdom: if I walk the
racks of a big ISP, it’s probably all
Linux
5. linux? The teleology of open source: “one
day this will all run Linux”
Conventional wisdom: if I walk the
racks of a big ISP, it’s probably all
Linux
Q: The hardware is ready for 10 Gbps
on a core. Is Linux?
6. not
linux
The teleology of open source: “one
day this will all run Linux”
Conventional wisdom: if I walk the
racks of a big ISP, it’s probably all
Linux
Q: The hardware is ready for 10 Gbps
on a core. Is Linux?
A: Nope
8. user-
space
networking
Cut Linux-the-kernel out of the
picture; bring up card from user
space
tell Linux to forget about this PCI
device
❧
mmap device’s PCI registers into
address space
❧
poke registers as needed❧
set up a ring buffer for receive/
transmit
❧
profit!❧
9. user-
space
networking
Multiple open source user-space
networking projects having success
Prominent examples:
Snabb (2012)❧
DPDK (2012)❧
VPP/fd.io (2016)❧
(Is this SDN? :))
How do software network functions
work?
10. aside Snabb aims to be rewritable software
The hard part: searching program-
space for elegant hacks
“Is that all? I could rewrite that in a
weekend.”
11. nutshell A snabb program consists of a graph
of apps
Apps are connected by directional
links
A snabb program processes packets
in units of breaths
12. local Intel82599 =
require("apps.intel.intel_app").Intel82599
local PcapFilter =
require("apps.packet_filter.pcap_filter").PcapFilter
local c = config.new()
config.app(c, "nic", Intel82599, {pciaddr="82:00.0"})
config.app(c, "filter", PcapFilter, {filter="tcp port 80"})
config.link(c, "nic.tx -> filter.input")
config.link(c, "filter.output -> nic.rx")
engine.configure(c)
while true do engine.breathe() end
13. breaths Each breath has two phases:
inhale a batch of packets into the
network
❧
process those packets❧
To inhale, run pull functions on
apps that have them
To process, run push functions on
apps that have them
14. # Pull function of included Intel 82599 driver
function Intel82599:pull ()
for i = 1, engine.pull_npackets do
if not self.dev:can_receive() then
break
end
local pkt = self.dev:receive()
link.transmit(self.output.tx, pkt)
end
end
15. # Push function of included PcapFilter
function PcapFilter:push ()
while not link.empty(self.input.rx) do
local p = link.receive(self.input.rx)
if self.accept_fn(p.data, p.length) then
link.transmit(self.output.tx, p)
else
packet.free(p)
end
end
end
17. links struct link {
struct packet *packets[1024];
// the next element to be read
int read;
// the next element to be written
int write;
};
// (Some statistics counters elided)
18. voilà At this point, you can rewrite Snabb
(Please do!)
But you might want to use it as-is...
19. inventory apps: software components that
developers compose into network
functions
programs: complete network
functions
bold: new in 2016/2017
italics: not yet merged to mainline
26. programs $ git clone
https://github.com/SnabbCo/snabb
$ cd snabb
$ make
27. $ src/snabb
Usage: src/snabb <program> ...
This snabb executable has the following programs built in:
lisper
lwaftr
packetblaster
pci_bind
snabbmark
snabbnfv
snabbvmx
snsh
top
wall
For detailed usage of any program run:
snabb <program> --help
28. program:
packet
blaster
Generally useful tool: fill TX buffer
of NIC with packets and transmit
them over and over again
snabb packetblaster replay
packets.pcap 82:00.1
Measures received (return) traffic
too
Easily saturates 10G links
29. program:
lwaftr
“Lightweight 4-over-6”: RFC 7596
Snabb-implemented border router
for lw4o6
IPv4 for entire countries!
Remarkable deployment report from
OTE engineer Kostas Zordabelos,
April 2017:
https://www.youtube.com/
watch?v=EEpUWieTr40&t=1h46m
31. program:
nfv
Host switch providing network
connectivity to QEMU instances
“Original” Snabb app
Like Open vSwitch with DPDK data-
path, or OpenContrail
OpenStack integration never
landed... but the market has moved
on
(Has the market moved on from
classic NFV?)
32. program:
vmx
Idea: Snabb data plane, external
control and management planes
Contributed by Juniper engineer
Marcel Wiget
Possibility to delegate to Juniper
vMX to determine next hops; or to
an image with Linux
Juniper Tech Club, March 2017:
https://www.youtube.com/
watch?v=N_CjXgyrUcY
snabb snabbvmx lwaftr --help
33. program:
snabbwall
L7 firewall that optionally uses nDPI
http://snabbwall.org/
Collaboration between Igalia and
NLnet foundation
Landed upstream in 2017
34. program:
ipfix
Prototype NETFLOW collector and
exporter (v9 and IPFIX)
Currently only 5MPPS, working on
single-core improvements then
moving to RSS
Pending to land upstream
35. program:
l2vpn
Alexander Gall’s L2 VPN over IPv6
Pending to land upstream; used in
production AFAIU
Ideal Snabb use case: programmer-
operator builds bespoke tool
37. deploy From prototype to production: what
do you need?
(Re)configurability
State monitoring
38. snabb
config
YANG is great!!!
Native YANG support in Snabb
Load and serialize textual
configurations
❧
Compiled compilations (useful
for big routing tables)
❧
Incremental update❧
State query❧
39. snabb
config
App & link graph a function of config
Update config? Diff graphs, apply
incremental changes
Carefully built to scale
Fast-paths for some incremental
updates, e.g. add lwAFTR
softwire
❧
Config/state query avoids
touching data plane process
❧
Updates cause minimal change❧
Subquery built-in❧
41. near
future
100G in production Snabb
Multiple coordinated data-plane
processes
Horizontal scaling via BGP/ECMP:
terabit lw4o6 deployments
Performance x-ray: where to focus
effort to improve speed?
[Your cool hack here!]
Work in progress!
42. thanks! Make a thing with Snabb!
git clone https://github.com/SnabbCo/snabb
cd snabb
make
wingo@igalia.com
@andywingo
44. Storytime! Modern x86: who’s winning?
Clock speed same since years ago
Main memory just as far away
45. HPC
people
are
winning
“We need to do work on data... but
there’s just so much of it and it’s
really far away.”
Three primary improvements:
CPU can work on more data per
cycle, once data in registers
❧
CPU can load more data per
cycle, once it’s in cache
❧
CPU can make more parallel
fetches to L3 and RAM at once
❧
46. Networking
folks
can
win
too
Instead of chasing zero-copy, tying
yourself to ever-more-proprietary
features of your NIC, just take the hit
once: DDIO into L3.
Copy if you need to – copies with L3
not expensive.
Software will eat the world!
47. Networking
folks
can
win
too
Once in L3, you have:
wide loads and stores via AVX2
and soon AVX-512 (64 bytes!)
❧
pretty good instruction-level
parallelism: up to 16 concurrent
L2 misses per core on haswell
❧
wide SIMD: checksum in
software!
❧
software, not firmware❧