Jafka is a fast and lightweight message queue system that is implemented as a single 271KB JAR file. It uses Zookeeper for coordination and has dependencies on common Java libraries like Log4j and Jackson. Jafka aims to eventually become a full implementation of Apache Kafka with features like persistence, high throughput processing of millions of messages per second, load balancing and a simple message format. It currently focuses on providing basic queue functionality through a simple producer/consumer model.
Slides for presentation on ZooKeeper I gave at Near Infinity (www.nearinfinity.com) 2012 spring conference.
The associated sample code is on GitHub at https://github.com/sleberknight/zookeeper-samples
Basically everything you need to get started on your Zookeeper training, and setup apache Hadoop high availability with QJM setup with automatic failover.
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...Docker, Inc.
The Prometheus monitoring system collects and stores time series data to give valuable insights over hosts, containers, and applications. Its storage engine was designed to be multiple orders of magnitude faster and more space efficient than, say, RRD or SQL storage. However, with the rise of orchestration systems such as Docker Swarm and Kubernetes, and their extensive use of techniques like rolling updates and auto-scaling, environments are becoming increasingly dynamic. This increases the strain on metrics collection systems. To deal with the challenges, a new storage engine has been developed from scratch, bringing a sharp increase in performance and enabling new features.
This talk will describe this new storage engine, its architecture, its data structures, and explain why and how it is well suited to gracefully handle high turnover rates of monitoring targets and provide consistent query performance.
Slides for presentation on ZooKeeper I gave at Near Infinity (www.nearinfinity.com) 2012 spring conference.
The associated sample code is on GitHub at https://github.com/sleberknight/zookeeper-samples
Basically everything you need to get started on your Zookeeper training, and setup apache Hadoop high availability with QJM setup with automatic failover.
How and Why Prometheus' New Storage Engine Pushes the Limits of Time Series D...Docker, Inc.
The Prometheus monitoring system collects and stores time series data to give valuable insights over hosts, containers, and applications. Its storage engine was designed to be multiple orders of magnitude faster and more space efficient than, say, RRD or SQL storage. However, with the rise of orchestration systems such as Docker Swarm and Kubernetes, and their extensive use of techniques like rolling updates and auto-scaling, environments are becoming increasingly dynamic. This increases the strain on metrics collection systems. To deal with the challenges, a new storage engine has been developed from scratch, bringing a sharp increase in performance and enabling new features.
This talk will describe this new storage engine, its architecture, its data structures, and explain why and how it is well suited to gracefully handle high turnover rates of monitoring targets and provide consistent query performance.
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016Susan Potter
This talk will introduce the audience to the Nix packaging, NixOS, and related ecosystem tools for Erlang/Elixir developers.
By reviewing common development, testing, and deployment problems we will look at what Nix has to offer to aid Erlang/Elixir developers in these areas.
From seamless developer environment bootstrapping to consistent CI environments and beyond.
You probably know the mantra that allocation is cheap. It usually is true, but devil is in the details. In your use case object allocation may impact processor caches evicting important data; burn CPU on executing constructor code; impact rates of object promotion to old generation and most importantly increase frequency of stop the word young gen pauses.
This presentation is for you if you are working on a Java based services that need to handle more and more traffic. As number of transactions per second rises you might hit performance wall that are young generation gc stopping whole application for precious milliseconds.
This presentation focuses on optimising object creation rate when dealing with seemingly mundane tasks. I will show few examples of surprising places in JDK and other libraries where garbage is created. I will explain how New Gen GC collection works and what costs are related to it. We will se escape analysis in action. Finally we will conclude that controlling allocation is the concern of library writers so that we can easily implement performant code without doing premature optimisations.
You may all know that JSON is a subset of JavaScript, but… Did you know that HTML5 implements NoSQL databases? Did you know that JavaScript was recommended for REST by HTTP co-creator Roy T. Fielding himself? Did you know that map & reduce are part of the native JavaScript API? Did you know that most NoSQL solutions integrate a JavaScript engine? CouchDB, MongoDB, WakandaDB, ArangoDB, OrientDB, Riak…. And when they don’t, they have a shell client which does. The story of NoSQL and JavaScript goes beyond your expectations and opens more opportunities than you might imagine… What better match could you find than a flexible and dynamic language for schemaless databases? Isn’t an event-driven language what you’ve been waiting for to manage consistency? When NoSQL doesn’t come to JavaScript, JavaScript comes to NoSQL. And does it very well.
In-depth caching in Varnish - GOG Varnish Meetup, march 2019GOG.com dev team
You think Varnish can cache responses by URL only? Not even close. Learn all different caching strategies available in Varnish, their benefits and consequences of usage. Learn how to and when to queue requests for the same endpoint, how to handle requests with conditional caching headers and how to have two levels of cache by tagging the responses.
Custom deployments with sbt-native-packagerGaryCoady
sbt-native-packager offers a comprehensive approach to packaging artifacts with SBT. The user describes a generic layout, which can then be extended for different types of software and deployments. For example, it is flexible enough to describe both a Zip-based archive format, and an RPM package with appropriate Systemd configuration for a service.
This talk will cover the essentials needed to understand the design of sbt-native-packager, and how to extend its structure to create custom layouts and deployments.
Swift Install Workshop - OpenStack Conference Spring 2012Joe Arnold
OpenStack Swift is a highly-available distributed object storage
system which supports highly concurrent workloads. Swift is the
backbone behind Cloud Files, Rackspace's storage-as-a-service
offering.
In this workshop, which will be hosted by members of SwiftStack, Inc.,
we'll walk you through deployment and use of OpenStack Swift. We'll
begin by showing you how to install Swift from the ground up.
You'll learn:
- what you should know about Swift's architecture
- how to bootstrap a basic Swift installation
After that, we'll cover how to use Swift, including information on:
- creating accounts and users
- adding, removing, and managing data
- building applications on top of Swift
Bring your laptop (with virutalization extensions enabled in the BIOS)
and we will walk through setting up Swift in a virtual machine. We'll
also build an entire application on top of Swift to illustrate how to
use Swift as a storage service. This is a workshop you won't want to
miss!
Blocks is a cool concept and is very much needed for performance improvements and responsiveness. GCD helps run blocks effortlessly by scheduling on a desired queue, priority and lots more.
From Zero To Production (NixOS, Erlang) @ Erlang Factory SF 2016Susan Potter
This talk will introduce the audience to the Nix packaging, NixOS, and related ecosystem tools for Erlang/Elixir developers.
By reviewing common development, testing, and deployment problems we will look at what Nix has to offer to aid Erlang/Elixir developers in these areas.
From seamless developer environment bootstrapping to consistent CI environments and beyond.
You probably know the mantra that allocation is cheap. It usually is true, but devil is in the details. In your use case object allocation may impact processor caches evicting important data; burn CPU on executing constructor code; impact rates of object promotion to old generation and most importantly increase frequency of stop the word young gen pauses.
This presentation is for you if you are working on a Java based services that need to handle more and more traffic. As number of transactions per second rises you might hit performance wall that are young generation gc stopping whole application for precious milliseconds.
This presentation focuses on optimising object creation rate when dealing with seemingly mundane tasks. I will show few examples of surprising places in JDK and other libraries where garbage is created. I will explain how New Gen GC collection works and what costs are related to it. We will se escape analysis in action. Finally we will conclude that controlling allocation is the concern of library writers so that we can easily implement performant code without doing premature optimisations.
You may all know that JSON is a subset of JavaScript, but… Did you know that HTML5 implements NoSQL databases? Did you know that JavaScript was recommended for REST by HTTP co-creator Roy T. Fielding himself? Did you know that map & reduce are part of the native JavaScript API? Did you know that most NoSQL solutions integrate a JavaScript engine? CouchDB, MongoDB, WakandaDB, ArangoDB, OrientDB, Riak…. And when they don’t, they have a shell client which does. The story of NoSQL and JavaScript goes beyond your expectations and opens more opportunities than you might imagine… What better match could you find than a flexible and dynamic language for schemaless databases? Isn’t an event-driven language what you’ve been waiting for to manage consistency? When NoSQL doesn’t come to JavaScript, JavaScript comes to NoSQL. And does it very well.
In-depth caching in Varnish - GOG Varnish Meetup, march 2019GOG.com dev team
You think Varnish can cache responses by URL only? Not even close. Learn all different caching strategies available in Varnish, their benefits and consequences of usage. Learn how to and when to queue requests for the same endpoint, how to handle requests with conditional caching headers and how to have two levels of cache by tagging the responses.
Custom deployments with sbt-native-packagerGaryCoady
sbt-native-packager offers a comprehensive approach to packaging artifacts with SBT. The user describes a generic layout, which can then be extended for different types of software and deployments. For example, it is flexible enough to describe both a Zip-based archive format, and an RPM package with appropriate Systemd configuration for a service.
This talk will cover the essentials needed to understand the design of sbt-native-packager, and how to extend its structure to create custom layouts and deployments.
Swift Install Workshop - OpenStack Conference Spring 2012Joe Arnold
OpenStack Swift is a highly-available distributed object storage
system which supports highly concurrent workloads. Swift is the
backbone behind Cloud Files, Rackspace's storage-as-a-service
offering.
In this workshop, which will be hosted by members of SwiftStack, Inc.,
we'll walk you through deployment and use of OpenStack Swift. We'll
begin by showing you how to install Swift from the ground up.
You'll learn:
- what you should know about Swift's architecture
- how to bootstrap a basic Swift installation
After that, we'll cover how to use Swift, including information on:
- creating accounts and users
- adding, removing, and managing data
- building applications on top of Swift
Bring your laptop (with virutalization extensions enabled in the BIOS)
and we will walk through setting up Swift in a virtual machine. We'll
also build an entire application on top of Swift to illustrate how to
use Swift as a storage service. This is a workshop you won't want to
miss!
Blocks is a cool concept and is very much needed for performance improvements and responsiveness. GCD helps run blocks effortlessly by scheduling on a desired queue, priority and lots more.
Open Source Monitoring for Java with JMX and Graphite (GeeCON 2013)Cyrille Le Clerc
Fast feedback from monitoring is a key of Continuous Delivery. JMX is the right Java API to do so but it unfortunately stayed underused and underappreciated as it was difficult to connect to monitoring and graphing systems.
Throw in the sin bin the poor solutions based on log files and weakly secured web interfaces! A new generation of Open Source tooling makes it easy to graph java application metrics and integrate them to traditional monitoring systems like Nagios.
Following the logic of DevOps, we will look together how best to integrate the monitoring dimension in a project: from design to development, to QA and finally to production on both traditional deployment and in the Cloud.
Come and discover how the JmxTrans-Graphite ticket can make your life easier.
Developing Realtime Data Pipelines With Apache KafkaJoe Stein
Developing Realtime Data Pipelines With Apache Kafka. Apache Kafka is publish-subscribe messaging rethought as a distributed commit log. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. Kafka is designed to allow a single cluster to serve as the central data backbone for a large organization. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of co-ordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages without performance impact. Kafka has a modern cluster-centric design that offers strong durability and fault-tolerance guarantees.
This is a presentation given on October 24 by Michael Uzquiano of Cloud CMS (http://www.cloudcms.com) at the MongoDB Boston conference.
In this presentation, we cover Hazelcast - an in-memory data grid that provides distributed object persistence across multiple nodes in a cluster. When backed by MongoDB, objects are naturally written to Mongo by Hazelcast. The integration points are clean and easy to implement.
We cover a few simple cases along with code samples to provide the MongoDB community with some ideas of how to integrate Hazelcast into their own MongoDB Java applications.
Enabling Data Scientists to easily create and own Kafka Consumers | Stefan Kr...HostedbyConfluent
At Stitch Fix, we hire Full Stack Data Scientists (150+) and expect them to perform diverse functions: from conception to modeling to implementation to measurement. Since Kafka is the way we get event data, this inevitably means that a Data Scientist will need to write a Kafka consumer if they’re going to complete their implementation work. E.g. to transform some client data into features, or perform a model prediction, or allocate someone to an A/B test, etc. In this talk I’ll go over how we built an opinionated Kafka client to easily enable Data Scientists to deploy and own production Kafka consumers, by focusing on writing python functions rather than fighting pitfalls with Kafka.
Enabling Data Scientists to easily create and own Kafka ConsumersStefan Krawczyk
At Stitch Fix, we hire Full Stack Data Scientists (145+) and expect them to perform diverse functions: from conception to modeling to implementation to measurement. Since Kafka is the way we get event data, this inevitably means that a Data Scientist will need to write a Kafka consumer if they’re going to complete their implementation work. E.g. to transform some client data into features, or perform a model prediction, or allocate someone to an A/B test, etc. In this talk I’ll go over how we built an opinionated Kafka client to easily enable Data Scientists to deploy and own production Kafka consumers, by focusing on writing python functions rather than fighting pitfalls with Kafka.
Erik Skytthe - Monitoring Mesos, Docker, Containers with Zabbix | ZabConf2016Zabbix
At DBC we are running docker and other container types in a mesos/marathon cluster environment. I will demonstrate how we collect statistics, logs etc. and monitor this environment, showing configuration examples, data flows and templates.
Some of the covered topics:
- Mesos master and agents
- Marathon Framework
- Docker engine
- Containers
- Zookeeper
- Elasticserach/ELK
By Patrick Griffis.
Sandboxing is an important part of modern security and as of WebKitGTK 2.26 it is able to sandbox its web processes. This talk will cover how WebKitGTK using applications can enable it as well as go into the technical details of how the sandbox is implemented.
(c) GUADEC 2019
August 23rd — 28th
Thessaloniki, Greece
https://2019.guadec.org/
This tutorial was presented in KDD 2016 conference in San Francisco, CA. You can find the main presentation at http://www.slideshare.net/NeeraAgarwal2/streaming-analytics
https://telecombcn-dl.github.io/2017-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of large-scale annotated datasets and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which were previously addressed with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or image captioning.
http://imatge-upc.github.io/telecombcn-2016-dlcv/
Deep learning technologies are at the core of the current revolution in artificial intelligence for multimedia data analysis. The convergence of big annotated data and affordable GPU hardware has allowed the training of neural networks for data analysis tasks which had been addressed until now with hand-crafted features. Architectures such as convolutional neural networks, recurrent neural networks and Q-nets for reinforcement learning have shaped a brand new scenario in signal processing. This course will cover the basic principles and applications of deep learning to computer vision problems, such as image classification, object detection or text captioning.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
5. Future
● A full kafka clone (http://incubator.apache.org/kafka)
● Persisten messaging with O(1)
● Constant time performance even with TB messages
● High-throughput (100k/s)
● Distributed messaging system
– producers/brokers/consumers
● Auto load balancing
● Simple message format
jafka mq 5
11. Offset length limit
● Long.MAX_VALUE
– 9223372036854775807 → 9*1018
● If one message size is 1k
– Messages: 9*1015
– If producing 1 billion messages per day
● Continuing days: 9*106
● Continuing years: 24657
jafka mq 11
12. About Filesystem
● 6 7200rpm SATA RAID-5 *
– 300MB/s linear writes (300k/s 1KB per message)
– 50k/s random writes
jafka mq 12
http://incubator.apache.org/kafka/design.html
13. sendfile() (1)
● Common data transfer
1. OS reads data from disk to pagecache
2. Application reads data from pagecache to buffer
3. Application writes data back into socket buffer
4. OS copies data from socket buffer to NIC buffer
● 4 copies + 2 system_call
jafka mq 13