Scaling Streaming - Concepts, Research, Goals
•Download as ODP, PDF•
1 like•765 views
This was a talk, largely on Kamaelia & its original context given at a Free Streaming Workshop in Florence, Italy in Summer 2004. Many of the core concepts still hold valid in Kamaelia today
Report
Share
Report
Share
![Scaling BBC Streaming Concepts, Current Research, Future Goals, Michael Sparks BBC R&D [email_address]](https://image.slidesharecdn.com/scaling-streaming-concepts-research-goals4013/85/Scaling-Streaming-Concepts-Research-Goals-1-320.jpg)
![Streaming at the BBC ,[object Object],[object Object],[object Object],[object Object],[object Object],[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
Caffe - A deep learning framework (Ramin Fahimi)
Caffe is a deep learning framework. It is used for tasks like visual recognition using neural networks and deep learning techniques. Caffe uses plain text configuration files called prototxt to define neural network architectures and hyperparameters. It also supports distributed training on GPUs for large datasets. Caffe provides pre-trained models and tools to load, fine-tune, and publish new models for tasks like image classification and object detection.
Application layer : DNS
The document discusses the application layer in the OSI model and DNS. It explains that the application layer provides services to end users through programs and interacts directly with them. It also describes DNS, including that it translates user-friendly domain names to IP addresses, allowing users to access resources by name instead of numerical address. DNS uses a hierarchical name space with domains, zones, primary/secondary name servers, and root servers to distribute its database around the network.
Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on Hadoop
As the data world undergoes its cambrian explosion phase our data tools need to become more advanced to keep pace. Deep Learning has emerged as a key tool in the non-linear arms race of machine learning. In this session we will take a look at how we parallelize Deep Belief Networks in Deep Learning on Hadoop’s next generation YARN framework with Iterative Reduce. We’ll also look at some real world examples of processing data with Deep Learning such as image classification and natural language processing.
Optimize Performance of I/O-intensive Java applications Using Zero Copy
This session explains how you can improve performance of I/O-intensive Java™ applications through a technique called zero copy. Zero copy lets you to avoid redundant data copies between intermediate buffers and reduces number of context switches between user and kernel space.
Background: Many applications [Web Servers, FTP-like services] serve a significant amount of static content, which amounts to reading data off of a disk and writing the exact data back to the response socket. Each time data traverses the user-kernel boundary; it must be copied, which consumes CPU cycles and memory bandwidth. Fortunately, we can eliminate these copies through a technique called — zero copy.
The Java class libraries support zero copy on Linux and UNIX systems through the transferTo() method in java.nio.channels.FileChannel.
Session Agenda: The session will initially focus on “Zero Copy” concept and its relevance in Data transfer applications. The traditional approach of transferring data b/w processes using File and Socket I/O will be explained in detail. It will demonstrate the overhead incurred when using traditional copy semantics, and will show how transferTo() achieves better performance. transferTo() API brings down the time 65% compared to traditional approach.
Summary
The Session demonstrates performance advantages of using transferTo() compared to the traditional approach. Intermediate buffer copies — even those hidden in the kernel — can have a measurable cost. In applications that do a great deal of copying of data between channels, zero-copy technique can offer a significant performance improvement.
Colorspace: Useful For More Than Just Color? - SF Video Tech Meetup - 27 May ...
This document discusses using colorspaces for more than just representing color, specifically for image and video compression purposes. It provides:
1) A brief history of colorspaces used in compression like YIQ, YUV, and YCbCr and how they were designed more for compression than accurate color representation.
2) Current uses of color transforms in formats like JPEG-XR and JPEG-XL that use colorspaces like YCoCg and XYB specifically designed for compression rather than color accuracy.
3) Potential future uses of reversible color transforms like YCoCg-R, reversible KLT-based transforms, and new proposed spaces like those in the "Alphabet Soup" section that aim to further optimize for compression.
Pmw2 k3ni 1-2a
This document discusses planning, optimizing, and troubleshooting DHCP in a network. It covers creating a DHCP plan by designing DHCP infrastructures, scope reservations, options, and security. To optimize performance, the document recommends monitoring DHCP and adjusting the lease duration if the server is overloaded. Troubleshooting tools like Network Monitor, DHCP Audit Log, and IPConfig can help identify client-side, server-side, or infrastructure problems.
Let's Write a JPEG Decoder (Vimeo Lunch Talks)
Talk given at internal Vimeo lunch talks with an intro to JPEG / image compression. There is a codebase that goes along with this, but it is not public yet, unfortunately.
“A Practical Guide to Implementing ML on Embedded Devices,” a Presentation fr...
“A Practical Guide to Implementing ML on Embedded Devices,” a Presentation fr...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/09/a-practical-guide-to-implementing-ml-on-embedded-devices-a-presentation-from-the-chamberlain-group/
Nathan Kopp, Principal Software Architect for Video Systems at the Chamberlain Group, presents the “Practical Guide to Implementing ML on Embedded Devices” tutorial at the May 2021 Embedded Vision Summit.
Deploying machine learning onto edge devices requires many choices and trade-offs. Fortunately, processor designers are adding inference-enhancing instructions and architectures to even the lowest cost MCUs, tools developers are constantly discovering optimizations that extract a little more performance out of existing hardware, and ML researchers are refactoring the math to achieve better accuracy using faster operations and fewer parameters.
In this presentation, Kopp takes a high-level look at what is involved in running a DNN model on existing edge devices, exploring some of the evolving tools and methods that are finally making this dream a reality. He also takes a quick look at a practical example of running a CNN object detector on low-compute hardware.Recommended
Caffe - A deep learning framework (Ramin Fahimi)
Caffe is a deep learning framework. It is used for tasks like visual recognition using neural networks and deep learning techniques. Caffe uses plain text configuration files called prototxt to define neural network architectures and hyperparameters. It also supports distributed training on GPUs for large datasets. Caffe provides pre-trained models and tools to load, fine-tune, and publish new models for tasks like image classification and object detection.
Application layer : DNS
The document discusses the application layer in the OSI model and DNS. It explains that the application layer provides services to end users through programs and interacts directly with them. It also describes DNS, including that it translates user-friendly domain names to IP addresses, allowing users to access resources by name instead of numerical address. DNS uses a hierarchical name space with domains, zones, primary/secondary name servers, and root servers to distribute its database around the network.
Hadoop Summit 2014 - San Jose - Introduction to Deep Learning on Hadoop
As the data world undergoes its cambrian explosion phase our data tools need to become more advanced to keep pace. Deep Learning has emerged as a key tool in the non-linear arms race of machine learning. In this session we will take a look at how we parallelize Deep Belief Networks in Deep Learning on Hadoop’s next generation YARN framework with Iterative Reduce. We’ll also look at some real world examples of processing data with Deep Learning such as image classification and natural language processing.
Optimize Performance of I/O-intensive Java applications Using Zero Copy
This session explains how you can improve performance of I/O-intensive Java™ applications through a technique called zero copy. Zero copy lets you to avoid redundant data copies between intermediate buffers and reduces number of context switches between user and kernel space.
Background: Many applications [Web Servers, FTP-like services] serve a significant amount of static content, which amounts to reading data off of a disk and writing the exact data back to the response socket. Each time data traverses the user-kernel boundary; it must be copied, which consumes CPU cycles and memory bandwidth. Fortunately, we can eliminate these copies through a technique called — zero copy.
The Java class libraries support zero copy on Linux and UNIX systems through the transferTo() method in java.nio.channels.FileChannel.
Session Agenda: The session will initially focus on “Zero Copy” concept and its relevance in Data transfer applications. The traditional approach of transferring data b/w processes using File and Socket I/O will be explained in detail. It will demonstrate the overhead incurred when using traditional copy semantics, and will show how transferTo() achieves better performance. transferTo() API brings down the time 65% compared to traditional approach.
Summary
The Session demonstrates performance advantages of using transferTo() compared to the traditional approach. Intermediate buffer copies — even those hidden in the kernel — can have a measurable cost. In applications that do a great deal of copying of data between channels, zero-copy technique can offer a significant performance improvement.
Colorspace: Useful For More Than Just Color? - SF Video Tech Meetup - 27 May ...
This document discusses using colorspaces for more than just representing color, specifically for image and video compression purposes. It provides:
1) A brief history of colorspaces used in compression like YIQ, YUV, and YCbCr and how they were designed more for compression than accurate color representation.
2) Current uses of color transforms in formats like JPEG-XR and JPEG-XL that use colorspaces like YCoCg and XYB specifically designed for compression rather than color accuracy.
3) Potential future uses of reversible color transforms like YCoCg-R, reversible KLT-based transforms, and new proposed spaces like those in the "Alphabet Soup" section that aim to further optimize for compression.
Pmw2 k3ni 1-2a
This document discusses planning, optimizing, and troubleshooting DHCP in a network. It covers creating a DHCP plan by designing DHCP infrastructures, scope reservations, options, and security. To optimize performance, the document recommends monitoring DHCP and adjusting the lease duration if the server is overloaded. Troubleshooting tools like Network Monitor, DHCP Audit Log, and IPConfig can help identify client-side, server-side, or infrastructure problems.
Let's Write a JPEG Decoder (Vimeo Lunch Talks)
Talk given at internal Vimeo lunch talks with an intro to JPEG / image compression. There is a codebase that goes along with this, but it is not public yet, unfortunately.
“A Practical Guide to Implementing ML on Embedded Devices,” a Presentation fr...
“A Practical Guide to Implementing ML on Embedded Devices,” a Presentation fr...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/09/a-practical-guide-to-implementing-ml-on-embedded-devices-a-presentation-from-the-chamberlain-group/
Nathan Kopp, Principal Software Architect for Video Systems at the Chamberlain Group, presents the “Practical Guide to Implementing ML on Embedded Devices” tutorial at the May 2021 Embedded Vision Summit.
Deploying machine learning onto edge devices requires many choices and trade-offs. Fortunately, processor designers are adding inference-enhancing instructions and architectures to even the lowest cost MCUs, tools developers are constantly discovering optimizations that extract a little more performance out of existing hardware, and ML researchers are refactoring the math to achieve better accuracy using faster operations and fewer parameters.
In this presentation, Kopp takes a high-level look at what is involved in running a DNN model on existing edge devices, exploring some of the evolving tools and methods that are finally making this dream a reality. He also takes a quick look at a practical example of running a CNN object detector on low-compute hardware.Bcs 052 solved assignment
This document provides information about an assignment for the course "Network Programming and Administration". It includes details like the course code, title, assignment number, maximum marks, weightage, and due dates. The assignment has 4 questions worth 80 marks total. An additional 20 marks are for a viva voce. Question 1 asks about IPv6 and includes a sample solution. Question 2 includes subquestions about sliding window protocols, TCP/IP protocols in the OSI model, and LAN network types. Question 3 asks about HTTP and includes sample request methods and statuses.
Tom Peters, Software Engineer, Ufora at MLconf ATL 2016
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
“TinyML Isn’t Thinking Big Enough,” a Presentation from Perceive
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/08/tinyml-isnt-thinking-big-enough-a-presentation-from-perceive/
Steve Teig, CEO of Perceive, presents the “TinyML Isn’t Thinking Big Enough” tutorial at the May 2021 Embedded Vision Summit.
Today, TinyML focuses primarily on shoehorning neural networks onto microcontrollers or small CPUs but misses the opportunity to transform all of ML because of two unfortunate assumptions: first, that tiny models must make significant performance and accuracy compromises to fit inside edge devices, and second, that tiny models should run on CPUs or microcontrollers.
Regarding the first assumption, information-theoretic considerations would suggest that principled compression (vs., say, just replacing 32-bit weights with 8-bit weights) should make models more accurate, not less. For the second assumption, CPUs are saddled with an intrinsically power-inefficient memory model and mostly serial computation, but the evident parallelism of neural networks naturally leads to high-performance, power-efficient, massively parallel inference hardware. By upending these assumptions, TinyML can revolutionize all of ML–and not just inside microcontrollers.
cas_Knowledge_Network
Knowledge Network is a public broadcaster in British Columbia that relies on an automated, file-based workflow using Telestream products like Vantage and Pipeline to ingest, transcode, and deliver programming. Vantage transcodes content into various formats for broadcast and online distribution. It has replaced their tape-based system, improving efficiency. Vantage allows them to process multiple files simultaneously and handle content in various formats from international distributors. The automated workflow is vital to their operations.
Kaz Sato, Evangelist, Google at MLconf ATL 2016
Machine Intelligence at Google Scale: Tensor Flow and Cloud Machine Learning: The biggest challenge of Deep Learning technology is the scalability. As long as using single GPU server, you have to wait for hours or days to get the result of your work. This doesn’t scale for production service, so you need a Distributed Training on the cloud eventually. Google has been building infrastructure for training the large scale neural network on the cloud for years, and now started to share the technology with external developers. In this session, we will introduce new pre-trained ML services such as Cloud Vision API and Speech API that works without any training. Also, we will look how TensorFlow and Cloud Machine Learning will accelerate custom model training for 10x – 40x with Google’s distributed training infrastructure.
ProcessOne Push Platform: XMPP-based Push Solutions
XMPP can provide a flexible and scalable solution for real-time push notifications across devices and platforms. ProcessOne offers an XMPP-based push platform as a service to enable reliable delivery of notifications to users. Case studies demonstrate how the platform supports use cases like radio program updates, social media feeds, and mobile applications. ProcessOne's expertise in XMPP pubsub helps make these services highly scalable and able to support new features over time.
Let's Be HAV1ng You - London Video Tech October 2019
Talk I have at the October 2019 London Video Tech meetup covering a few of the many AV1 coding tools (old and new), a small rant on some AV1 tests, and some graphs.
Video: <upload pending>
Cartographer, or Building A Next Generation Management Framework
Dr. Bobby Krupczak's slides about the Cartographer management agent and the underlying XMP management framework. Presented at the February 10, 2009 meeting of the Atlanta Network and Systems Management Technical User Group (ANSMTUG).
5 maximazing networkcapacity_v4-jorge_alvarado
This document discusses how to maximize network capacity through bandwidth optimization and data compression techniques. It provides an agenda that covers defining wireless link optimization, maximizing network capacity for internet access, VPN networks, UDP traffic, corporate applications, and cellular backhaul. Specific scenarios and case studies are presented where XipLink's optimization solutions have reduced bandwidth usage by 18-60% for various application types including internet, VPNs, VoIP, video surveillance, and file transfers. The solutions provide a typical return on investment of less than 4 months.
Application Layer
The document outlines a syllabus for a computer networks course taught by Usha Barad. The syllabus covers 5 topics: 1) introduction to computer networks and the Internet, 2) application layer, 3) transport layer, 4) network layer, and 5) link layer and local area networks. It also lists recommended reference books for the course.
Keras Tutorial For Beginners | Creating Deep Learning Models Using Keras In P...
** AI & Deep Learning Training: https://www.edureka.co/ai-deep-learning-with-tensorflow ** )
This Edureka Tutorial on "Keras Tutorial" (Deep Learning Blog Series: https://goo.gl/4zxMfU) provides you a quick and insightful tutorial on the working of Keras along with an interesting use-case! We will be checking out the following topics:
Agenda:
What is Keras?
Who makes Keras?
Who uses Keras?
What Makes Keras special?
Working principle of Keras
Keras Models
Understanding Execution
Implementing a Neural Network
Use-Case with Keras
Coding in Colaboratory
Session in a minute
Check out our Deep Learning blog series: https://bit.ly/2xVIMe1
Check out our complete Youtube playlist here: https://bit.ly/2OhZEpz
Follow us to never miss an update in the future.
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
Amazon Deep Learning
The document provides an overview and agenda for an Amazon Deep Learning presentation. It discusses AI and deep learning at Amazon, gives a primer on deep learning and applications, provides an overview of MXNet and Amazon's investments in it, discusses deep learning tools and usage, and provides two application examples using MXNet on AWS. It concludes by discussing next steps and a call to action.
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...
Title
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTorch + XGBoost + Airflow + MLflow + Spark + Jupyter + TPU
Video
https://youtu.be/vaB4IM6ySD0
Description
In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, and Airflow.
Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google.
KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking.
Airflow is the most-widely used pipeline orchestration framework in machine learning.
Pre-requisites
Modern browser - and that's it!
Every attendee will receive a cloud instance
Nothing will be installed on your local laptop
Everything can be downloaded at the end of the workshop
Location
Online Workshop
Agenda
1. Create a Kubernetes cluster
2. Install KubeFlow, Airflow, TFX, and Jupyter
3. Setup ML Training Pipelines with KubeFlow and Airflow
4. Transform Data with TFX Transform
5. Validate Training Data with TFX Data Validation
6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow
7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow
8. Analyze Models using TFX Model Analysis and Jupyter
9. Perform Hyper-Parameter Tuning with KubeFlow
10. Select the Best Model using KubeFlow Experiment Tracking
11. Reproduce Model Training with TFX Metadata Store and Pachyderm
12. Deploy the Model to Production with TensorFlow Serving and Istio
13. Save and Download your Workspace
Key Takeaways
Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools.
Related Links
1. PipelineAI Home: https://pipeline.ai
2. PipelineAI Community Edition: http://community.pipeline.ai
3. PipelineAI GitHub: https://github.com/PipelineAI/pipeline
4. Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup
5. YouTube Videos: https://youtube.pipeline.ai
6. SlideShare Presentations: https://slideshare.pipeline.ai
7. Slack Support: https://joinslack.pipeline.ai
8. Web Support and Knowledge Base: https://support.pipeline.ai
9. Email Support: support@pipeline.ai
Deep learning with TensorFlow
This document provides an agenda for a presentation on deep learning with TensorFlow. It includes:
1. An introduction to machine learning and deep networks, including definitions of machine learning, neural networks, and deep learning.
2. An overview of TensorFlow, including its architecture, evolution, language features, computational graph, TensorBoard, and use in Google Cloud ML.
3. Details of TensorFlow hands-on examples, including linear models, shallow and deep neural networks for MNIST digit classification, and convolutional neural networks for MNIST.
Convolutional Neural Networks at scale in Spark MLlib
Jeremy Nixon will focus on the engineering and applications of a new algorithm built on top of MLlib. The presentation will focus on the methods the algorithm uses to automatically generate features to capture nonlinear structure in data, as well as the process by which it’s trained. Major aspects of that are the compositional transformations over the data, convolution, and distributed backpropagation via SGD with adaptive gradients and an adaptive learning rate. Applications will look into how to use convolutional neural networks to model data in computer vision, natural language and signal processing. Details around optimal preprocessing, the type of structure that can be learned, and managing its ability to generalize will inform developers looking to apply nonlinear modeling tools to problems that they face.
Probabilistic Approach to Provisioning of ITV - By Amos_Kohn
This white paper discusses a probabilistic approach to provisioning network and computing resources for delivering interactive TV. It develops a proprietary spreadsheet model to estimate the costs and benefits of deploying an interactive TV streaming processor. The model is based on analyzing user behavior, data packaging into MPEG streams, required bit rates, forward and return network paths, processing needs, and financial projections to calculate return on investment.
Dynamic Adaptive Point Cloud Streaming
High-quality point clouds have recently gained interest as an emerg- ing form of representing immersive 3D graphics. Unfortunately, these 3D media are bulky and severely bandwidth intensive, which makes it difficult for streaming to resource-limited and mobile de- vices. This has called researchers to propose efficient and adaptive approaches for streaming of high-quality point clouds.
In this paper, we run a pilot study towards dynamic adaptive point cloud streaming, and extend the concept of dynamic adaptive streaming over HTTP (DASH) towards DASH-PC, a dynamic adaptive bandwidth-efficient and view-aware point cloud streaming system. DASH-PC can tackle the huge bandwidth demands of dense point cloud streaming while at the same time can semantically link to human visual acuity to maintain high visual quality when needed. In order to describe the various quality representations, we pro- pose multiple thinning approaches to spatially sub-sample point clouds in the 3D space, and design a DASH Media Presentation Description manifest specific for point cloud streaming. Our initial evaluations show that we can achieve significant bandwidth and performance improvement on dense point cloud streaming with minor negative quality impacts compared to the baseline scenario when no adaptations is applied.
Huffman ppt
The document discusses Huffman coding, which is a lossless data compression algorithm that uses variable-length codes to encode symbols based on their frequency of occurrence. It begins with definitions of Huffman coding and related terms. It then describes the encoding and decoding processes, which involve constructing a Huffman tree based on symbol frequencies and traversing the tree to encode or decode data. An example is provided that shows the full process of constructing a Huffman tree for a sample frequency table and determining the Huffman codes, average code length, and total encoded length.
IMPROVING IPV6 ADDRESSING TYPES AND SIZE
This document discusses proposed modifications to IPv6 addressing types and address size. It suggests that multicast addressing can mimic anycast and limited broadcast addressing, making those types unnecessary. It also proposes reducing the IPv6 address size from 128-bits to decrease packet overhead, while ensuring the new size supports future internet growth. A formula is presented to predict IP address exhaustion dates for different address sizes based on current usage and population projections.
Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
PeerToPeerComputing (1)
The document discusses the development of a pure peer-to-peer computing system using socket programming. It aims to facilitate parallel computation of complex tasks by distributing work across available peers in a network. This allows heavier calculations to be performed faster by utilizing otherwise idle processing resources. The system is designed to remove scalability and security issues while managing tasks through administrator, query manager, task dispatcher, and processor groups. A literature review found that decentralized peer-to-peer systems like Freenet and GNUtella provide benefits like failure tolerance, efficiency and cost effectiveness.
Serhiy Kalinets "Embracing architectural challenges in the modern .NET world"
For more than decade .NET has been used primarily in enterprise software development. We all remember intranet deployment, IIS, SQL Server, N-tier applications and so on. The toolset (Visual Studio, SQL Management Studio, IIS Management snap-in etc) seemed to be set in stone as well as architecture (controllers, services, repositories). .NET people were isolated from other folks, who were using clusters, containers, clouds, and Linux.
However, adoption of clouds during few past years, the release of .NET Core made much more choices available to developers. It turned out that traditional way of building application is not that efficient from many viewpoints, including costs, time, performance or robustness. It happens because the environment has been changed and many assumptions are not still relevant.
In this talk, we will discuss what and why has been changed and how to deal with that. What are new requirements for our applications? What are new services available, and how to use them wisely? And finally, how should we design our applications to be cost-effective, competitive and have a lot of fun working with .NET Core.
More Related Content
What's hot
Bcs 052 solved assignment
This document provides information about an assignment for the course "Network Programming and Administration". It includes details like the course code, title, assignment number, maximum marks, weightage, and due dates. The assignment has 4 questions worth 80 marks total. An additional 20 marks are for a viva voce. Question 1 asks about IPv6 and includes a sample solution. Question 2 includes subquestions about sliding window protocols, TCP/IP protocols in the OSI model, and LAN network types. Question 3 asks about HTTP and includes sample request methods and statuses.
Tom Peters, Software Engineer, Ufora at MLconf ATL 2016
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
“TinyML Isn’t Thinking Big Enough,” a Presentation from Perceive
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2021/08/tinyml-isnt-thinking-big-enough-a-presentation-from-perceive/
Steve Teig, CEO of Perceive, presents the “TinyML Isn’t Thinking Big Enough” tutorial at the May 2021 Embedded Vision Summit.
Today, TinyML focuses primarily on shoehorning neural networks onto microcontrollers or small CPUs but misses the opportunity to transform all of ML because of two unfortunate assumptions: first, that tiny models must make significant performance and accuracy compromises to fit inside edge devices, and second, that tiny models should run on CPUs or microcontrollers.
Regarding the first assumption, information-theoretic considerations would suggest that principled compression (vs., say, just replacing 32-bit weights with 8-bit weights) should make models more accurate, not less. For the second assumption, CPUs are saddled with an intrinsically power-inefficient memory model and mostly serial computation, but the evident parallelism of neural networks naturally leads to high-performance, power-efficient, massively parallel inference hardware. By upending these assumptions, TinyML can revolutionize all of ML–and not just inside microcontrollers.
cas_Knowledge_Network
Knowledge Network is a public broadcaster in British Columbia that relies on an automated, file-based workflow using Telestream products like Vantage and Pipeline to ingest, transcode, and deliver programming. Vantage transcodes content into various formats for broadcast and online distribution. It has replaced their tape-based system, improving efficiency. Vantage allows them to process multiple files simultaneously and handle content in various formats from international distributors. The automated workflow is vital to their operations.
Kaz Sato, Evangelist, Google at MLconf ATL 2016
Machine Intelligence at Google Scale: Tensor Flow and Cloud Machine Learning: The biggest challenge of Deep Learning technology is the scalability. As long as using single GPU server, you have to wait for hours or days to get the result of your work. This doesn’t scale for production service, so you need a Distributed Training on the cloud eventually. Google has been building infrastructure for training the large scale neural network on the cloud for years, and now started to share the technology with external developers. In this session, we will introduce new pre-trained ML services such as Cloud Vision API and Speech API that works without any training. Also, we will look how TensorFlow and Cloud Machine Learning will accelerate custom model training for 10x – 40x with Google’s distributed training infrastructure.
ProcessOne Push Platform: XMPP-based Push Solutions
XMPP can provide a flexible and scalable solution for real-time push notifications across devices and platforms. ProcessOne offers an XMPP-based push platform as a service to enable reliable delivery of notifications to users. Case studies demonstrate how the platform supports use cases like radio program updates, social media feeds, and mobile applications. ProcessOne's expertise in XMPP pubsub helps make these services highly scalable and able to support new features over time.
Let's Be HAV1ng You - London Video Tech October 2019
Talk I have at the October 2019 London Video Tech meetup covering a few of the many AV1 coding tools (old and new), a small rant on some AV1 tests, and some graphs.
Video: <upload pending>
Cartographer, or Building A Next Generation Management Framework
Dr. Bobby Krupczak's slides about the Cartographer management agent and the underlying XMP management framework. Presented at the February 10, 2009 meeting of the Atlanta Network and Systems Management Technical User Group (ANSMTUG).
5 maximazing networkcapacity_v4-jorge_alvarado
This document discusses how to maximize network capacity through bandwidth optimization and data compression techniques. It provides an agenda that covers defining wireless link optimization, maximizing network capacity for internet access, VPN networks, UDP traffic, corporate applications, and cellular backhaul. Specific scenarios and case studies are presented where XipLink's optimization solutions have reduced bandwidth usage by 18-60% for various application types including internet, VPNs, VoIP, video surveillance, and file transfers. The solutions provide a typical return on investment of less than 4 months.
Application Layer
The document outlines a syllabus for a computer networks course taught by Usha Barad. The syllabus covers 5 topics: 1) introduction to computer networks and the Internet, 2) application layer, 3) transport layer, 4) network layer, and 5) link layer and local area networks. It also lists recommended reference books for the course.
Keras Tutorial For Beginners | Creating Deep Learning Models Using Keras In P...
** AI & Deep Learning Training: https://www.edureka.co/ai-deep-learning-with-tensorflow ** )
This Edureka Tutorial on "Keras Tutorial" (Deep Learning Blog Series: https://goo.gl/4zxMfU) provides you a quick and insightful tutorial on the working of Keras along with an interesting use-case! We will be checking out the following topics:
Agenda:
What is Keras?
Who makes Keras?
Who uses Keras?
What Makes Keras special?
Working principle of Keras
Keras Models
Understanding Execution
Implementing a Neural Network
Use-Case with Keras
Coding in Colaboratory
Session in a minute
Check out our Deep Learning blog series: https://bit.ly/2xVIMe1
Check out our complete Youtube playlist here: https://bit.ly/2OhZEpz
Follow us to never miss an update in the future.
Instagram: https://www.instagram.com/edureka_learning/
Facebook: https://www.facebook.com/edurekaIN/
Twitter: https://twitter.com/edurekain
LinkedIn: https://www.linkedin.com/company/edureka
Amazon Deep Learning
The document provides an overview and agenda for an Amazon Deep Learning presentation. It discusses AI and deep learning at Amazon, gives a primer on deep learning and applications, provides an overview of MXNet and Amazon's investments in it, discusses deep learning tools and usage, and provides two application examples using MXNet on AWS. It concludes by discussing next steps and a call to action.
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...
Title
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTorch + XGBoost + Airflow + MLflow + Spark + Jupyter + TPU
Video
https://youtu.be/vaB4IM6ySD0
Description
In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, and Airflow.
Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google.
KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking.
Airflow is the most-widely used pipeline orchestration framework in machine learning.
Pre-requisites
Modern browser - and that's it!
Every attendee will receive a cloud instance
Nothing will be installed on your local laptop
Everything can be downloaded at the end of the workshop
Location
Online Workshop
Agenda
1. Create a Kubernetes cluster
2. Install KubeFlow, Airflow, TFX, and Jupyter
3. Setup ML Training Pipelines with KubeFlow and Airflow
4. Transform Data with TFX Transform
5. Validate Training Data with TFX Data Validation
6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow
7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow
8. Analyze Models using TFX Model Analysis and Jupyter
9. Perform Hyper-Parameter Tuning with KubeFlow
10. Select the Best Model using KubeFlow Experiment Tracking
11. Reproduce Model Training with TFX Metadata Store and Pachyderm
12. Deploy the Model to Production with TensorFlow Serving and Istio
13. Save and Download your Workspace
Key Takeaways
Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools.
Related Links
1. PipelineAI Home: https://pipeline.ai
2. PipelineAI Community Edition: http://community.pipeline.ai
3. PipelineAI GitHub: https://github.com/PipelineAI/pipeline
4. Advanced Spark and TensorFlow Meetup (SF-based, Global Reach): https://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup
5. YouTube Videos: https://youtube.pipeline.ai
6. SlideShare Presentations: https://slideshare.pipeline.ai
7. Slack Support: https://joinslack.pipeline.ai
8. Web Support and Knowledge Base: https://support.pipeline.ai
9. Email Support: support@pipeline.ai
Deep learning with TensorFlow
This document provides an agenda for a presentation on deep learning with TensorFlow. It includes:
1. An introduction to machine learning and deep networks, including definitions of machine learning, neural networks, and deep learning.
2. An overview of TensorFlow, including its architecture, evolution, language features, computational graph, TensorBoard, and use in Google Cloud ML.
3. Details of TensorFlow hands-on examples, including linear models, shallow and deep neural networks for MNIST digit classification, and convolutional neural networks for MNIST.
Convolutional Neural Networks at scale in Spark MLlib
Jeremy Nixon will focus on the engineering and applications of a new algorithm built on top of MLlib. The presentation will focus on the methods the algorithm uses to automatically generate features to capture nonlinear structure in data, as well as the process by which it’s trained. Major aspects of that are the compositional transformations over the data, convolution, and distributed backpropagation via SGD with adaptive gradients and an adaptive learning rate. Applications will look into how to use convolutional neural networks to model data in computer vision, natural language and signal processing. Details around optimal preprocessing, the type of structure that can be learned, and managing its ability to generalize will inform developers looking to apply nonlinear modeling tools to problems that they face.
Probabilistic Approach to Provisioning of ITV - By Amos_Kohn
This white paper discusses a probabilistic approach to provisioning network and computing resources for delivering interactive TV. It develops a proprietary spreadsheet model to estimate the costs and benefits of deploying an interactive TV streaming processor. The model is based on analyzing user behavior, data packaging into MPEG streams, required bit rates, forward and return network paths, processing needs, and financial projections to calculate return on investment.
Dynamic Adaptive Point Cloud Streaming
High-quality point clouds have recently gained interest as an emerg- ing form of representing immersive 3D graphics. Unfortunately, these 3D media are bulky and severely bandwidth intensive, which makes it difficult for streaming to resource-limited and mobile de- vices. This has called researchers to propose efficient and adaptive approaches for streaming of high-quality point clouds.
In this paper, we run a pilot study towards dynamic adaptive point cloud streaming, and extend the concept of dynamic adaptive streaming over HTTP (DASH) towards DASH-PC, a dynamic adaptive bandwidth-efficient and view-aware point cloud streaming system. DASH-PC can tackle the huge bandwidth demands of dense point cloud streaming while at the same time can semantically link to human visual acuity to maintain high visual quality when needed. In order to describe the various quality representations, we pro- pose multiple thinning approaches to spatially sub-sample point clouds in the 3D space, and design a DASH Media Presentation Description manifest specific for point cloud streaming. Our initial evaluations show that we can achieve significant bandwidth and performance improvement on dense point cloud streaming with minor negative quality impacts compared to the baseline scenario when no adaptations is applied.
Huffman ppt
The document discusses Huffman coding, which is a lossless data compression algorithm that uses variable-length codes to encode symbols based on their frequency of occurrence. It begins with definitions of Huffman coding and related terms. It then describes the encoding and decoding processes, which involve constructing a Huffman tree based on symbol frequencies and traversing the tree to encode or decode data. An example is provided that shows the full process of constructing a Huffman tree for a sample frequency table and determining the Huffman codes, average code length, and total encoded length.
IMPROVING IPV6 ADDRESSING TYPES AND SIZE
This document discusses proposed modifications to IPv6 addressing types and address size. It suggests that multicast addressing can mimic anycast and limited broadcast addressing, making those types unnecessary. It also proposes reducing the IPv6 address size from 128-bits to decrease packet overhead, while ensuring the new size supports future internet growth. A formula is presented to predict IP address exhaustion dates for different address sizes based on current usage and population projections.
Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16
Say What You Mean: Scaling Machine Learning Algorithms Directly from Source Code: Scaling machine learning applications is hard. Even with powerful systems like Spark, Tensor Flow, and Theano, the code you write has more to do with getting these systems to work at all than it does with your algorithm itself. But it doesn’t have to be this way!
In this talk, I’ll discuss an alternate approach we’ve taken with Pyfora, an open-source platform for scalable machine learning and data science in Python. I’ll show how it produces efficient, large scale machine learning implementations directly from the source code of single-threaded Python programs. Instead of programming to a complex API, you can simply say what you mean and move on. I’ll show some classes of problem where this approach truly shines, discuss some practical realities of developing the system, and I’ll talk about some future directions for the project.
What's hot (20)
Tom Peters, Software Engineer, Ufora at MLconf ATL 2016
Tom Peters, Software Engineer, Ufora at MLconf ATL 2016
“TinyML Isn’t Thinking Big Enough,” a Presentation from Perceive
“TinyML Isn’t Thinking Big Enough,” a Presentation from Perceive
ProcessOne Push Platform: XMPP-based Push Solutions
ProcessOne Push Platform: XMPP-based Push Solutions
Let's Be HAV1ng You - London Video Tech October 2019
Let's Be HAV1ng You - London Video Tech October 2019
Cartographer, or Building A Next Generation Management Framework
Cartographer, or Building A Next Generation Management Framework
Keras Tutorial For Beginners | Creating Deep Learning Models Using Keras In P...
Keras Tutorial For Beginners | Creating Deep Learning Models Using Keras In P...
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...
Hands-on Learning with KubeFlow + Keras/TensorFlow 2.0 + TF Extended (TFX) + ...
Convolutional Neural Networks at scale in Spark MLlib
Convolutional Neural Networks at scale in Spark MLlib
Probabilistic Approach to Provisioning of ITV - By Amos_Kohn
Probabilistic Approach to Provisioning of ITV - By Amos_Kohn
Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16
Braxton McKee, CEO & Founder, Ufora at MLconf NYC - 4/15/16
Similar to Scaling Streaming - Concepts, Research, Goals
PeerToPeerComputing (1)
The document discusses the development of a pure peer-to-peer computing system using socket programming. It aims to facilitate parallel computation of complex tasks by distributing work across available peers in a network. This allows heavier calculations to be performed faster by utilizing otherwise idle processing resources. The system is designed to remove scalability and security issues while managing tasks through administrator, query manager, task dispatcher, and processor groups. A literature review found that decentralized peer-to-peer systems like Freenet and GNUtella provide benefits like failure tolerance, efficiency and cost effectiveness.
Serhiy Kalinets "Embracing architectural challenges in the modern .NET world"
For more than decade .NET has been used primarily in enterprise software development. We all remember intranet deployment, IIS, SQL Server, N-tier applications and so on. The toolset (Visual Studio, SQL Management Studio, IIS Management snap-in etc) seemed to be set in stone as well as architecture (controllers, services, repositories). .NET people were isolated from other folks, who were using clusters, containers, clouds, and Linux.
However, adoption of clouds during few past years, the release of .NET Core made much more choices available to developers. It turned out that traditional way of building application is not that efficient from many viewpoints, including costs, time, performance or robustness. It happens because the environment has been changed and many assumptions are not still relevant.
In this talk, we will discuss what and why has been changed and how to deal with that. What are new requirements for our applications? What are new services available, and how to use them wisely? And finally, how should we design our applications to be cost-effective, competitive and have a lot of fun working with .NET Core.
EQR Reporting: Rails + Amazon EC2
The document discusses the architecture of an online survey platform. It covers goals of performance, availability, and scalability. It describes using services like Amazon Web Services for storage, computing, and content delivery. It also discusses optimizing application design, database separation, coding best practices, and automated testing to achieve goals. Video hosting and online analytical reporting features are explained.
Lesson 2
This document discusses changes to Hyper-V virtualization from Windows Server 2008 to 2012. Key changes include the ability to share virtual hard disks between VMs, improved quality of service controls, and more robust resource sharing between host and guest systems. The new features make Hyper-V more reliable and scalable for server virtualization needs over the next 2-3 years.
Lunar Way and the Cloud Native "stack"
This talk was given at the Cloud Native Aarhus meetup at May 30rd 2017. Focus is on how we use Cloud Native projects at Lunar Way.
Nss Labs Dpi Intro V3
What is DPI? How can it be used effectively? What are the different use cases and requirements for such products? We discuss this and the methodologies needed to properly evaluate the DPI functionality of network devices under the demanding network conditions in which they will be deployed.
http://nsslabs.com/DPI
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Twitter is a good example for next generation real-time web applications, but building such an application imposes challenges such as handling an every growing volume of tweets and responses, as well as a large number of concurrent users, who continually *listen* for tweets from users (or topics) they follow. During this session we will review some of the key design principles addressing these challenges, including alternatives *NoSQL* alternatives and blackboard patterns. We will be using Twitter as a use case, while learning how to apply these to any real-time we application
Introduction to requirement of microservices
We are talking about microservices. It is a pattern to resolve the complexity of the system those need to process a high amount of data within a short period.
Financial lose may occur on implementation of this pattern for an application of limited complexity in the initial phase. Initial phases have a learning curve to understand the relation and behavior of domain entities.
Small and medium companies lean this during development. Large companies can allocate additional times for documentation and correction on design phases for a reasonable long period. So, sometimes it is good to start with a monolithic architecture and grow with the achievement of the company then migrate to microservices.
Open Source in the Enterprise
Open Source in the Enterprise: You\'re soaking in it! From Firefox to Apache and Linux, enterprises use Open Source software because it works.
Tuenti teams - Php Conference
The document discusses the technical teams at Tuenti and their work developing various products and services. It covers their frontend, backend, and systems teams and some of the challenges they face in building large-scale, high-performance applications and services to support millions of users. It also provides specifics on their development of Tuenti's instant messaging platform using open-source technologies and Erlang.
Other distributed systems
This document discusses several distributed computing systems:
1) DNS is a distributed system that maps domain names to IP addresses using a hierarchical naming structure and caching DNS servers for efficiency.
2) BOINC is a volunteer computing platform that uses over a million computers worldwide for distributed applications like disease research. It provides incentives and verifies results to prevent cheating.
3) PlanetLab is a research network with over 700 servers globally that allows testing new distributed systems at large scales under realistic conditions. It isolates projects using virtualization and trust relationships.
Handling Data in Mega Scale Systems
This document discusses strategies for handling large amounts of data in web applications. It begins by providing examples of how much data some large websites contain, ranging from terabytes to petabytes. It then covers various techniques for scaling data handling capabilities including vertical and horizontal scaling, replication, partitioning, consistency models, normalization, caching, and using different data engine types beyond relational databases. The key lessons are that data volumes continue growing rapidly, and a variety of techniques are needed to scale across servers, datacenters, and provide high performance and availability.
CH02-Computer Organization and Architecture 10e.pptx
This document discusses computer performance and benchmarking. It covers several topics related to improving computer performance, including designing for performance, microprocessor speed techniques, improvements in chip organization and architecture like multicore processors, and issues that limit further increases in clock speed. It also discusses Amdahl's Law, Little's Law, and ways to measure computer performance, including various types of means to calculate benchmark results. SPEC benchmarks are mentioned as examples of widely used benchmark programs.
DM Radio Webinar: Adopting a Streaming-Enabled Architecture
Architecture matters. That's why today's innovators are taking a hard look at streaming data, an increasingly attractive option that can transform business in several ways: replacing aging data ingestion techniques like ETL; solving long-standing data quality challenges; improving business processes ranging from sales and marketing to logistics and procurement; or any number of activities related to accelerating data warehousing, business intelligence and analytics.
Register for this DM Radio Deep Dive Webinar to learn how streaming data can rejuvenate or supplant traditional data management practices. Host Eric Kavanagh will explain how streaming-first architectures can relieve data engineers from time-consuming, error-prone processes, ideally bidding farewell to those unpleasant batch windows. He'll be joined by Kevin Petrie of Attunity, who will explain why (with real-world story successes) streaming data solutions can keep the business fueled with trusted data in a timely, efficient manner for improved business outcomes.
moveMountainIEEE
This document presents a case study comparing a traditional single-node approach and a cloud-based approach for analyzing a large dataset of over 150 million domain names to determine which are hosted by SoftLayer. The single-node approach ran on a single server and took approximately 300 hours to complete at a cost of $102.67. A cloud-based approach using multiple servers in parallel could complete the task much faster and potentially at a lower overall cost by leveraging elastic computing resources in the cloud.
RFP-Final3
The document provides details of a proposed network solution for ACME Inc. that will allow 70 users to work productively from the company's 3-story office. Key aspects include:
- Implementing Active Directory, file/print services, and a company intranet to centralize management and sharing of files and communications.
- Dividing the network into subnets for different floors/departments and assigning IP addresses and devices.
- Specifying the required hardware, software, and licenses including laptops, desktops, servers, networking equipment, and applications.
- Outlining the conceptual network design with remote and on-site clients connecting through a firewall, VPN server, and other servers.
-
Waters Grid & HPC Course
This is the course that was presented by James Liddle and Adam Vile for Waters in September 2008.
The book of this course can be found at: http://www.lulu.com/content/4334860
UnConference for Georgia Southern Computer Science March 31, 2015
I presented to the Georgia Southern Computer Science ACM group. Rather than one topic for 90 minutes, I decided to do an UnConference. I presented them a list of 8-9 topics, let them vote on what to talk about, then repeated.
Each presentation was ~8 minutes, (Except Career) and was by no means an attempt to explain the full concept or technology. Only to wake up their interest.
Monitoring&Logging - Stanislav Kolenkin
Microservices add complexity to monitoring that was not present with monolithic architectures. While microservices provide benefits, they also introduce significant monitoring challenges around communication between services. Prometheus has emerged as a powerful open source solution for monitoring microservices as it was designed to address issues of scale and flexibility that monitoring microservices requires.
Similar to Scaling Streaming - Concepts, Research, Goals (20)
Serhiy Kalinets "Embracing architectural challenges in the modern .NET world"
Serhiy Kalinets "Embracing architectural challenges in the modern .NET world"
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
Designing a Scalable Twitter - Patterns for Designing Scalable Real-Time Web ...
CH02-Computer Organization and Architecture 10e.pptx
CH02-Computer Organization and Architecture 10e.pptx
DM Radio Webinar: Adopting a Streaming-Enabled Architecture
DM Radio Webinar: Adopting a Streaming-Enabled Architecture
UnConference for Georgia Southern Computer Science March 31, 2015
UnConference for Georgia Southern Computer Science March 31, 2015
More from kamaelian
Kamaelia lightning2010opensource
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming massively concurrent, providing an opportunity. While concurrency is viewed as hard, the fundamental problem may be lack of proper tools. Imperative languages often overlook concurrency as a core concept. A variety of desktop and media applications could benefit from a concurrent approach. The document advocates using concurrent components that communicate via messages while keeping data private. It also discusses software transactional memory and different perspectives in APIs for concurrent systems. Finally, it presents examples of using pipelines and graphlines as part of a concurrency domain specific language.
Kamaelia Europython Tutorial
This was the Kamaelia Tutorial at Europython. It goes from basics - ie building a mini-kamaelia from scratch, through to a file multicaster, through a video recording application all the walk through to a multiuser bulletin board system.
Embracing concurrency for fun utility and simpler code
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming more concurrent, but most programming languages and tools treat concurrency as difficult. The Kamaelia project aims to make concurrency easy and usable for novice and advanced developers alike through fundamental control structures and messaging between components. Examples shown include using pipelines, graphlines, servers, and backplanes to build concurrent applications in a simple way.
Kamaelia Protocol Walkthrough
This document guides the reader through building a system where a user connects to a server over a secure connection and receives a sequence of JSON-encoded objects. It begins by introducing the ServerCore component and shows how to fill its protocol handler factory hole. It then demonstrates creating a stackedjson protocol handler using a pipeline of components like PeriodicWakeup, Chooser, and MarshallJSON. This protocol securely transmits JSON data chunks to clients like a ConsoleEchoer. It discusses how the client-side mirrors the server components to receive and display the messages.
Sharing Data and Services Safely in Concurrent Systems using Kamaelia
Kamaelia is generally a \"shared nothing\" architecture, but occasionally you *really* need to share data explicitly. When you do, you need to constrain how you share data and careful about how you advertise services. This is the first presentation done on the facilities that exist in Kamaelia to support this.
This presentation was given at Pycon UK 2008, Birmingham uk. Lots of good feedback was had during the q&a and an updated & improved version will be posted at some point in the relatively near future.
Practical concurrent systems made simple using Kamaelia
This talk was given at Pycon UK 2008 in Birmingham.
This presentation aims to teach you how to get started with Kamaelia, building a variety of systems, as well as walking through the design and implementation of some systems built in the last year. Systems built over the past year include tools for dealing with spam (greylisting), through database modelling, video & image transcoding for a youtube/flickr type system, paint programs, webserving, XMPP, games, and a bunch of other things.
Sociable Software
This presentation aims to show people that they already know how to deal with concurrency.
It argues that if we have the tools for large scale concurrency (mashups) and small scale (hardware) that midrange (normal apps) can be done in a similar way, using existing tools.
This is done by showing useful systems that have been produced in this manner using existing tools. ie from existing practice, not theory
During the actual presentation I also talked about Kamaelia projects created by novice programmers of varying ability which show high levels of concurrency.
These include: previewing PVR content on mobiles, multicast island joining, as-live streaming using bit torrent, Open GL based user interfaces & integration, seaside style webserving, speex based secure phone, IRC/IM systems, a shakespeare script player, and games tools.
Other systems created include Atom/RSS routing, memcached integration, P2P whiteboarding (with audio + mixing), gesture recognition, presentation tools, a kids development environment, topology visualisation tools, database modelling etc.
Kamaelia Grey
This presentation was given at Python North West. It explains a complete Kamaelia application for greylisting which was written specifically to eliminate my personal spam problem. It walks through the code as well (though that's best looked at with the code side by side!)
Open Source at the BBC: When, Why, Why not & How
This talk was given at Linux World 2006. It covers 3 aspects of open source at the BBC - use, extension & origination through the 4 lenses of when, why, why not & how. It focusses entirely on pragmatics in all cases. The style is Lessig style. A write up on the text can be found here: http://tinyurl.com/yd4j2y
Open Source at the BBC
This was an invited talk at Open Source Forum Russia in April 2005. It covers open source at the BBC from the perspective of "why use open source?" "what sort of stuff gets used?" "what has the BBC released as open source & why?" open source vs open standards
Kamaelia - Fave 2005
This was a talk on Kamaelia, Open source & challenges Kamaelia was originally designed to help explore solutions to.
SWP - A Generic Language Parser
This talk was part tongue in cheek, part serious, but entirely fun and given twice as a lightning talk - once at Europython & once at the ACCU python uk 05. It presents a generic python like language parser which does actually work. Think of it as an alternative to brackets in Lisp!
Kamaelia - Networking Using Generators
This was a talk given at the ACCU Python UK conference in 2005 relatively early in Kamaelia's development.
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
This presentation on Kamaelia at Euro OSCON 2006, and specifically focusses
on a particular system - Kamaelia Macro which is essentially a system for
timeshifting pretty much everything.
Kamaelia Internals
This talk was given at Pycon UK 07. It's actually a thin wrapper around the
Kamaelia Mini Axon tutorial which can be found here:
http://kamaelia.sourceforge.net/MiniAxon/
Managing Creativity
In this talk I talked about how,in the Kamaelia project, we manage the dilemma of encouraging innovation and creativity in a project whilst maintaining an engineered solution. Why? Because we find it allows a high level of creative freedom, whilst also providing a path through to a high level of confidence in the reliabilty of the final code.
Building systems with Kamaelia
This was a talk on how to build systems with Kamaelia given at Pycon UK. It
goes through from basics through to building a swarming P2P live radio
system.
Free software: How does it work?
The document discusses free and open source software. It begins by defining free software as software that users have the freedom to use, study, distribute, and change. It notes that free software is also known as libre software or open source software. The document outlines several advantages of free software such as giving users control, reducing costs, using open standards, sustainability, skills development, and improved security and quality. It also briefly discusses some potential disadvantages like smaller installed bases and issues of compatibility with proprietary software. Overall, the document presents an overview of the key concepts around free and open source software.
The Selfish Programmer
This talk was the keynote talk at the EBU's Seminar on Open Source Oct 1st, 2nd 2007. http://www.ebu.ch/en/technical/opensource/
The video referenced is IBM's "Prodigy" advert, which can be found here: http://youtube.com/watch?v=q5Kp1Q39VwI
More from kamaelian (19)
Embracing concurrency for fun utility and simpler code
Embracing concurrency for fun utility and simpler code
Sharing Data and Services Safely in Concurrent Systems using Kamaelia
Sharing Data and Services Safely in Concurrent Systems using Kamaelia
Practical concurrent systems made simple using Kamaelia
Practical concurrent systems made simple using Kamaelia
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaelia
Recently uploaded
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
ここ3000字までしか入らないけどタイトルの方がたくさん文字入ると思います。
Microsoft - Power Platform_G.Aspiotis.pdf
Revolutionizing Application Development
with AI-powered low-code, presentation by George Aspiotis, Sr. Partner Development Manager, Microsoft
Climate Impact of Software Testing at Nordic Testing Days
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Simplify your search for a reliable Python development partner! This list presents the top 10 trusted US providers offering comprehensive Python development services, ensuring your project's success from conception to completion.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
CAKE: Sharing Slices of Confidential Data on Blockchain
Presented at the CAiSE 2024 Forum, Intelligent Information Systems, June 6th, Limassol, Cyprus.
Synopsis: Cooperative information systems typically involve various entities in a collaborative process within a distributed environment. Blockchain technology offers a mechanism for automating such processes, even when only partial trust exists among participants. The data stored on the blockchain is replicated across all nodes in the network, ensuring accessibility to all participants. While this aspect facilitates traceability, integrity, and persistence, it poses challenges for adopting public blockchains in enterprise settings due to confidentiality issues. In this paper, we present a software tool named Control Access via Key Encryption (CAKE), designed to ensure data confidentiality in scenarios involving public blockchains. After outlining its core components and functionalities, we showcase the application of CAKE in the context of a real-world cyber-security project within the logistics domain.
Paper: https://doi.org/10.1007/978-3-031-61000-4_16
Removing Uninteresting Bytes in Software Fuzzing
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Essentials of Automations: The Art of Triggers and Actions in FME
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
Taking AI to the Next Level in Manufacturing.pdf
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
HCL Notes and Domino License Cost Reduction in the World of DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
How to Get CNIC Information System with Paksim Ga.pptx
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Discover how MongoDB Atlas and vector search technology can revolutionize your application's search capabilities. This comprehensive presentation covers:
* What is Vector Search?
* Importance and benefits of vector search
* Practical use cases across various industries
* Step-by-step implementation guide
* Live demos with code snippets
* Enhancing LLM capabilities with vector search
* Best practices and optimization strategies
Perfect for developers, AI enthusiasts, and tech leaders. Learn how to leverage MongoDB Atlas to deliver highly relevant, context-aware search results, transforming your data retrieval process. Stay ahead in tech innovation and maximize the potential of your applications.
#MongoDB #VectorSearch #AI #SemanticSearch #TechInnovation #DataScience #LLM #MachineLearning #SearchTechnology
OpenID AuthZEN Interop Read Out - Authorization
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-und-domino-lizenzkostenreduzierung-in-der-welt-von-dlau/
DLAU und die Lizenzen nach dem CCB- und CCX-Modell sind für viele in der HCL-Community seit letztem Jahr ein heißes Thema. Als Notes- oder Domino-Kunde haben Sie vielleicht mit unerwartet hohen Benutzerzahlen und Lizenzgebühren zu kämpfen. Sie fragen sich vielleicht, wie diese neue Art der Lizenzierung funktioniert und welchen Nutzen sie Ihnen bringt. Vor allem wollen Sie sicherlich Ihr Budget einhalten und Kosten sparen, wo immer möglich. Das verstehen wir und wir möchten Ihnen dabei helfen!
Wir erklären Ihnen, wie Sie häufige Konfigurationsprobleme lösen können, die dazu führen können, dass mehr Benutzer gezählt werden als nötig, und wie Sie überflüssige oder ungenutzte Konten identifizieren und entfernen können, um Geld zu sparen. Es gibt auch einige Ansätze, die zu unnötigen Ausgaben führen können, z. B. wenn ein Personendokument anstelle eines Mail-Ins für geteilte Mailboxen verwendet wird. Wir zeigen Ihnen solche Fälle und deren Lösungen. Und natürlich erklären wir Ihnen das neue Lizenzmodell.
Nehmen Sie an diesem Webinar teil, bei dem HCL-Ambassador Marc Thomas und Gastredner Franz Walder Ihnen diese neue Welt näherbringen. Es vermittelt Ihnen die Tools und das Know-how, um den Überblick zu bewahren. Sie werden in der Lage sein, Ihre Kosten durch eine optimierte Domino-Konfiguration zu reduzieren und auch in Zukunft gering zu halten.
Diese Themen werden behandelt
- Reduzierung der Lizenzkosten durch Auffinden und Beheben von Fehlkonfigurationen und überflüssigen Konten
- Wie funktionieren CCB- und CCX-Lizenzen wirklich?
- Verstehen des DLAU-Tools und wie man es am besten nutzt
- Tipps für häufige Problembereiche, wie z. B. Team-Postfächer, Funktions-/Testbenutzer usw.
- Praxisbeispiele und Best Practices zum sofortigen Umsetzen
Infrastructure Challenges in Scaling RAG with Custom AI models
Building Retrieval-Augmented Generation (RAG) systems with open-source and custom AI models is a complex task. This talk explores the challenges in productionizing RAG systems, including retrieval performance, response synthesis, and evaluation. We’ll discuss how to leverage open-source models like text embeddings, language models, and custom fine-tuned models to enhance RAG performance. Additionally, we’ll cover how BentoML can help orchestrate and scale these AI components efficiently, ensuring seamless deployment and management of RAG systems in the cloud.
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...Edge AI and Vision Alliance
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.UiPath Test Automation using UiPath Test Suite series, part 6
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays, June 2024 with Maria Copot 20
Artificial Intelligence for XMLDevelopment
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
Recently uploaded (20)
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
みなさんこんにちはこれ何文字まで入るの？40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの？えこ...
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slack
CAKE: Sharing Slices of Confidential Data on Blockchain
CAKE: Sharing Slices of Confidential Data on Blockchain
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
HCL Notes and Domino License Cost Reduction in the World of DLAU
HCL Notes and Domino License Cost Reduction in the World of DLAU
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
Infrastructure Challenges in Scaling RAG with Custom AI models
Infrastructure Challenges in Scaling RAG with Custom AI models
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
Scaling Streaming - Concepts, Research, Goals
- 1. Scaling BBC Streaming Concepts, Current Research, Future Goals, Michael Sparks BBC R&D [email_address]
- 75. What problem does streaming solve?