This presentation was given to the system adminstration team to give them an idea of how GC works and what to look for when there is abottleneck and troubles.
FortranCon2020: Highly Parallel Fortran and OpenACC DirectivesJeff Larkin
Fortran has long been the language of computational math and science and it has outlived many of the computer architectures on which it has been used. Modern Fortran must be able to run on modern, highly parallel, heterogeneous computer architectures. A significant number of Fortran programmers have had success programming for heterogeneous machines by pairing Fortran with the OpenACC language for directives-based parallel programming. This includes some of the most widely-used Fortran applications in the world, such as VASP and Gaussian. This presentation will discuss what makes OpenACC a good fit for Fortran programmers and what the OpenACC language is doing to promote the use of native language parallelism in Fortran, such as do concurrent and Co-arrays.
Video Recording: https://www.youtube.com/watch?v=OXZ_Wkae63Y
Hyper-Parameter Tuning Across the Entire AI Pipeline GPU Tech Conference San ...Chris Fregly
Chris Fregly, Founder @ PipelineAI, will walk you through a real-world, complete end-to-end Pipeline-optimization example. We highlight hyper-parameters - and model pipeline phases - that have never been exposed until now.
While most Hyperparameter Optimizers stop at the training phase (ie. learning rate, tree depth, ec2 instance type, etc), we extend model validation and tuning into a new post-training optimization phase including 8-bit reduced precision weight quantization and neural network layer fusing - among many other framework and hardware-specific optimizations.
Next, we introduce hyperparameters at the prediction phase including request-batch sizing and chipset (CPU v. GPU v. TPU).
Lastly, we determine a PipelineAI Efficiency Score of our overall Pipeline including Cost, Accuracy, and Time. We show techniques to maximize this PipelineAI Efficiency Score using our massive PipelineDB along with the Pipeline-wide hyper-parameter tuning techniques mentioned in this talk.
Bio
Chris Fregly is Founder and Applied AI Engineer at PipelineAI, a Real-Time Machine Learning and Artificial Intelligence Startup based in San Francisco.
He is also an Apache Spark Contributor, a Netflix Open Source Committer, founder of the Global Advanced Spark and TensorFlow Meetup, author of the O’Reilly Training and Video Series titled, "High Performance TensorFlow in Production with Kubernetes and GPUs."
Previously, Chris was a Distributed Systems Engineer at Netflix, a Data Solutions Engineer at Databricks, and a Founding Member and Principal Engineer at the IBM Spark Technology Center in San Francisco.
Bonjour !
Ici je vais vous expliquer un peu qui je suis, ce que je vis, ce que j'aime afin que vous sachiez !
Je m'appelle Matéo LEPETIT (c'est pas une blague !), j'ai 16 ans et je vais bientôt mourir...
Lorsque ça arrivera, je n'aurais pas connu l'amour, je n'aurais pas eu mon permis et pleins d'autre choses encore.
il y a deux ans, on m'a diagnostiqué un cancer...féroce. J'aime la vie, je ne sais pas pourquoi elle m'a fait ça, mais je veux continuer à profiter même si je suis affaibli.
Vous pouvez m'aider par des mots de soutient qui me permettront de garder le moral ou par des dons qui financeront le traitement qui pourrait me guérir et qui coûte 187 000 $.
This presentation was given to the system adminstration team to give them an idea of how GC works and what to look for when there is abottleneck and troubles.
FortranCon2020: Highly Parallel Fortran and OpenACC DirectivesJeff Larkin
Fortran has long been the language of computational math and science and it has outlived many of the computer architectures on which it has been used. Modern Fortran must be able to run on modern, highly parallel, heterogeneous computer architectures. A significant number of Fortran programmers have had success programming for heterogeneous machines by pairing Fortran with the OpenACC language for directives-based parallel programming. This includes some of the most widely-used Fortran applications in the world, such as VASP and Gaussian. This presentation will discuss what makes OpenACC a good fit for Fortran programmers and what the OpenACC language is doing to promote the use of native language parallelism in Fortran, such as do concurrent and Co-arrays.
Video Recording: https://www.youtube.com/watch?v=OXZ_Wkae63Y
Hyper-Parameter Tuning Across the Entire AI Pipeline GPU Tech Conference San ...Chris Fregly
Chris Fregly, Founder @ PipelineAI, will walk you through a real-world, complete end-to-end Pipeline-optimization example. We highlight hyper-parameters - and model pipeline phases - that have never been exposed until now.
While most Hyperparameter Optimizers stop at the training phase (ie. learning rate, tree depth, ec2 instance type, etc), we extend model validation and tuning into a new post-training optimization phase including 8-bit reduced precision weight quantization and neural network layer fusing - among many other framework and hardware-specific optimizations.
Next, we introduce hyperparameters at the prediction phase including request-batch sizing and chipset (CPU v. GPU v. TPU).
Lastly, we determine a PipelineAI Efficiency Score of our overall Pipeline including Cost, Accuracy, and Time. We show techniques to maximize this PipelineAI Efficiency Score using our massive PipelineDB along with the Pipeline-wide hyper-parameter tuning techniques mentioned in this talk.
Bio
Chris Fregly is Founder and Applied AI Engineer at PipelineAI, a Real-Time Machine Learning and Artificial Intelligence Startup based in San Francisco.
He is also an Apache Spark Contributor, a Netflix Open Source Committer, founder of the Global Advanced Spark and TensorFlow Meetup, author of the O’Reilly Training and Video Series titled, "High Performance TensorFlow in Production with Kubernetes and GPUs."
Previously, Chris was a Distributed Systems Engineer at Netflix, a Data Solutions Engineer at Databricks, and a Founding Member and Principal Engineer at the IBM Spark Technology Center in San Francisco.
Bonjour !
Ici je vais vous expliquer un peu qui je suis, ce que je vis, ce que j'aime afin que vous sachiez !
Je m'appelle Matéo LEPETIT (c'est pas une blague !), j'ai 16 ans et je vais bientôt mourir...
Lorsque ça arrivera, je n'aurais pas connu l'amour, je n'aurais pas eu mon permis et pleins d'autre choses encore.
il y a deux ans, on m'a diagnostiqué un cancer...féroce. J'aime la vie, je ne sais pas pourquoi elle m'a fait ça, mais je veux continuer à profiter même si je suis affaibli.
Vous pouvez m'aider par des mots de soutient qui me permettront de garder le moral ou par des dons qui financeront le traitement qui pourrait me guérir et qui coûte 187 000 $.
From the Cocoon GetTogether 2005:
When flowscript came up it was not only the powerful idea of continuations that helped making it a big success. In combination with its scripting nature it provided a much quicker development cycle.Soon people wished to have such a short turnaround in their java based development environments. And that's what we have in the latest Cocoon trunk - today! Auto-compilation of javaflow (the java based alternative to flowscript) and instant reloading of components helps to cut down development times tremendously. Without further need of restarting your servlet container after every little change, java development has finally become as easy as it should be.
This session will demonstrates on how to use features like reloading or auto-compilation of the current Cocoon trunk. It will also try to demystify the magic behind javaflow and will provide an overview about the current status and limitations.
So you've been deploying Java in the cloud and are wondering how to handle the new world of containers, microservices, and memory constraints. Cold starts got you down? Come to this session to learn about how the OpenJ9 and the JVM in general can help you on your Cloud Native journey.
Slides from SREcon_Americas 2019 presentation.
Presents some of the learnings we gained while deploying our containers based infrastructure at Booking.com, migrating from Openshift to vanilla Kubernetes.
We focused on observability (RFOs, SLIs, SLOs), replicability and consistency.
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...IWMW
Web Standards Briefing session at IWMW 2002 event by Lorna Campbell and Neil Sclater.
See http://www.ukoln.ac.uk/web-focus/events/workshops/webmaster-2002/materials/sclater/
Free GitOps Workshop + Intro to Kubernetes & GitOpsWeaveworks
Follow along in this free workshop and experience GitOps!
AGENDA:
Welcome - Tamao Nakahara, Head of DX (Weaveworks)
Introduction to Kubernetes & GitOps - Mark Emeis, Principal Engineer (Weaveworks)
Weave Gitops Overview - Tamao Nakahara
Free Gitops Workshop - David Harris, Product Manager (Weaveworks)
If you're new to Kubernetes and GitOps, we'll give you a brief introduction to both and how GitOps is the natural evolution of Kubernetes.
Weave GitOps Core is a continuous delivery product to run apps in any Kubernetes. It is free and open source, and you can get started today!
https://www.weave.works/product/gitops-core
If you’re stuck, also come talk to us at our Slack channel! #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Tackling non-determinism in Hadoop - Testing and debugging distributed system...Akihiro Suda
[Presented at FOSDEM 2016: https://fosdem.org/2016/schedule/event/nondeterminism_in_hadoop/]
Developing and maintaining distributed systems like Hadoop is difficult. The difficulty comes from many factors, but we believe that one of the most important reasons is lacking of a good debugger for bugs specific to distributed systems. (e.g., non-deterministic hardware faults, message ordering, ..)
In the talk, we will show Earthquake, our open-source debugging framework for distributed systems. Earthquakes permutes Ethernet packets, Filesystem events, Java/C function calls, and injected faults in various orders so as to control non-determinism in the cluster. Basically, Earthquake permutes events in a random order, but the user can write his/her own state exploration policy (in Go language) for finding deep bugs efficiently. Earthquake also controls non-determinism of the thread interleaving by calling sched_setattr(2) with randomized parameters.
We will also share our successful stories about testing some Hadoop components with Earthquake. For ZooKeeper, we found a distributed race condition bug which decreases availability of a ZooKeeper cluster. We also reproduced a known ZooKeeper bug that no one had successfully reproduced for 2 years, and analyzed its cause. For YARN, we found a disk-fault tolerance bug that inappropriately marks faulty node as healthy. We also found bugs of non-Hadoop softwares, such as etcd.
With Earthquake, you can also test your real distibuted systems without any modification.
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...Anne Nicolas
The purpose of this talk is to provide a short overview of the current live patching facility inside the linux kernel (with a brief history excursion), describe the features it currently provides, and most importantly things that still need to be implemented / designed.
This was the Kamaelia Tutorial at Europython. It goes from basics - ie building a mini-kamaelia from scratch, through to a file multicaster, through a video recording application all the walk through to a multiuser bulletin board system.
From the Cocoon GetTogether 2005:
When flowscript came up it was not only the powerful idea of continuations that helped making it a big success. In combination with its scripting nature it provided a much quicker development cycle.Soon people wished to have such a short turnaround in their java based development environments. And that's what we have in the latest Cocoon trunk - today! Auto-compilation of javaflow (the java based alternative to flowscript) and instant reloading of components helps to cut down development times tremendously. Without further need of restarting your servlet container after every little change, java development has finally become as easy as it should be.
This session will demonstrates on how to use features like reloading or auto-compilation of the current Cocoon trunk. It will also try to demystify the magic behind javaflow and will provide an overview about the current status and limitations.
So you've been deploying Java in the cloud and are wondering how to handle the new world of containers, microservices, and memory constraints. Cold starts got you down? Come to this session to learn about how the OpenJ9 and the JVM in general can help you on your Cloud Native journey.
Slides from SREcon_Americas 2019 presentation.
Presents some of the learnings we gained while deploying our containers based infrastructure at Booking.com, migrating from Openshift to vanilla Kubernetes.
We focused on observability (RFOs, SLIs, SLOs), replicability and consistency.
IWMW 2002: Interoperability and Learning Standards briefing: Does Interoperab...IWMW
Web Standards Briefing session at IWMW 2002 event by Lorna Campbell and Neil Sclater.
See http://www.ukoln.ac.uk/web-focus/events/workshops/webmaster-2002/materials/sclater/
Free GitOps Workshop + Intro to Kubernetes & GitOpsWeaveworks
Follow along in this free workshop and experience GitOps!
AGENDA:
Welcome - Tamao Nakahara, Head of DX (Weaveworks)
Introduction to Kubernetes & GitOps - Mark Emeis, Principal Engineer (Weaveworks)
Weave Gitops Overview - Tamao Nakahara
Free Gitops Workshop - David Harris, Product Manager (Weaveworks)
If you're new to Kubernetes and GitOps, we'll give you a brief introduction to both and how GitOps is the natural evolution of Kubernetes.
Weave GitOps Core is a continuous delivery product to run apps in any Kubernetes. It is free and open source, and you can get started today!
https://www.weave.works/product/gitops-core
If you’re stuck, also come talk to us at our Slack channel! #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Tackling non-determinism in Hadoop - Testing and debugging distributed system...Akihiro Suda
[Presented at FOSDEM 2016: https://fosdem.org/2016/schedule/event/nondeterminism_in_hadoop/]
Developing and maintaining distributed systems like Hadoop is difficult. The difficulty comes from many factors, but we believe that one of the most important reasons is lacking of a good debugger for bugs specific to distributed systems. (e.g., non-deterministic hardware faults, message ordering, ..)
In the talk, we will show Earthquake, our open-source debugging framework for distributed systems. Earthquakes permutes Ethernet packets, Filesystem events, Java/C function calls, and injected faults in various orders so as to control non-determinism in the cluster. Basically, Earthquake permutes events in a random order, but the user can write his/her own state exploration policy (in Go language) for finding deep bugs efficiently. Earthquake also controls non-determinism of the thread interleaving by calling sched_setattr(2) with randomized parameters.
We will also share our successful stories about testing some Hadoop components with Earthquake. For ZooKeeper, we found a distributed race condition bug which decreases availability of a ZooKeeper cluster. We also reproduced a known ZooKeeper bug that no one had successfully reproduced for 2 years, and analyzed its cause. For YARN, we found a disk-fault tolerance bug that inappropriately marks faulty node as healthy. We also found bugs of non-Hadoop softwares, such as etcd.
With Earthquake, you can also test your real distibuted systems without any modification.
Kernel Recipes 2018 - Live (Kernel) Patching: status quo and status futurus -...Anne Nicolas
The purpose of this talk is to provide a short overview of the current live patching facility inside the linux kernel (with a brief history excursion), describe the features it currently provides, and most importantly things that still need to be implemented / designed.
Similar to Kamaelia - Networking Using Generators (20)
This was the Kamaelia Tutorial at Europython. It goes from basics - ie building a mini-kamaelia from scratch, through to a file multicaster, through a video recording application all the walk through to a multiuser bulletin board system.
Embracing concurrency for fun utility and simpler codekamaelian
This was a talk at O'Reilly UK's "Ignite" event in leeds. It tries in 20 slides/5 minutes to give an overview of what makes concurrency actually useable and fun in general purpose languages. Whilst Kamaelia is python based, a proof of concept for the core has been done in C++, Ruby & Java, and our "introduction" tutorial teaches anyone (even complete novices) how to build their own proof of concept in python.
Sharing Data and Services Safely in Concurrent Systems using Kamaeliakamaelian
Kamaelia is generally a \"shared nothing\" architecture, but occasionally you *really* need to share data explicitly. When you do, you need to constrain how you share data and careful about how you advertise services. This is the first presentation done on the facilities that exist in Kamaelia to support this.
This presentation was given at Pycon UK 2008, Birmingham uk. Lots of good feedback was had during the q&a and an updated & improved version will be posted at some point in the relatively near future.
Practical concurrent systems made simple using Kamaeliakamaelian
This talk was given at Pycon UK 2008 in Birmingham.
This presentation aims to teach you how to get started with Kamaelia, building a variety of systems, as well as walking through the design and implementation of some systems built in the last year. Systems built over the past year include tools for dealing with spam (greylisting), through database modelling, video & image transcoding for a youtube/flickr type system, paint programs, webserving, XMPP, games, and a bunch of other things.
This presentation aims to show people that they already know how to deal with concurrency.
It argues that if we have the tools for large scale concurrency (mashups) and small scale (hardware) that midrange (normal apps) can be done in a similar way, using existing tools.
This is done by showing useful systems that have been produced in this manner using existing tools. ie from existing practice, not theory
During the actual presentation I also talked about Kamaelia projects created by novice programmers of varying ability which show high levels of concurrency.
These include: previewing PVR content on mobiles, multicast island joining, as-live streaming using bit torrent, Open GL based user interfaces & integration, seaside style webserving, speex based secure phone, IRC/IM systems, a shakespeare script player, and games tools.
Other systems created include Atom/RSS routing, memcached integration, P2P whiteboarding (with audio + mixing), gesture recognition, presentation tools, a kids development environment, topology visualisation tools, database modelling etc.
This presentation was given at Python North West. It explains a complete Kamaelia application for greylisting which was written specifically to eliminate my personal spam problem. It walks through the code as well (though that's best looked at with the code side by side!)
Open Source at the BBC: When, Why, Why not & Howkamaelian
This talk was given at Linux World 2006. It covers 3 aspects of open source at the BBC - use, extension & origination through the 4 lenses of when, why, why not & how. It focusses entirely on pragmatics in all cases. The style is Lessig style. A write up on the text can be found here: http://tinyurl.com/yd4j2y
This was an invited talk at Open Source Forum Russia in April 2005. It covers open source at the BBC from the perspective of "why use open source?" "what sort of stuff gets used?" "what has the BBC released as open source & why?" open source vs open standards
This talk was part tongue in cheek, part serious, but entirely fun and given twice as a lightning talk - once at Europython & once at the ACCU python uk 05. It presents a generic python like language parser which does actually work. Think of it as an alternative to brackets in Lisp!
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaeliakamaelian
This presentation on Kamaelia at Euro OSCON 2006, and specifically focusses
on a particular system - Kamaelia Macro which is essentially a system for
timeshifting pretty much everything.
This was a talk, largely on Kamaelia & its original context given at a Free Streaming Workshop in Florence, Italy in Summer 2004. Many of the core
concepts still hold valid in Kamaelia today
This talk was given at Pycon UK 07. It's actually a thin wrapper around the
Kamaelia Mini Axon tutorial which can be found here:
http://kamaelia.sourceforge.net/MiniAxon/
In this talk I talked about how,in the Kamaelia project, we manage the dilemma of encouraging innovation and creativity in a project whilst maintaining an engineered solution. Why? Because we find it allows a high level of creative freedom, whilst also providing a path through to a high level of confidence in the reliabilty of the final code.
This was a talk on how to build systems with Kamaelia given at Pycon UK. It
goes through from basics through to building a swarming P2P live radio
system.
This talk was the second talk at the EBU's Seminar on Open Source Oct 1st, 2nd 2007. http://www.ebu.ch/en/technical/opensource/
This is an introduction to the concepts behind free software.
This talk was the keynote talk at the EBU's Seminar on Open Source Oct 1st, 2nd 2007. http://www.ebu.ch/en/technical/opensource/
The video referenced is IBM's "Prodigy" advert, which can be found here: http://youtube.com/watch?v=q5Kp1Q39VwI
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
1. Kamaelia - Networking
Using Generators
Michael Sparks
BBC Research & Development
ACCU Python 2005, Oxford
2. Kamaelia
• Project to explore long term systems for large
scale media delivery
• Forms a concurrency toolkit, focussed mainly on experimenting
with network protocols.
• 2 key portions:
• Axon - Core Component infrastructure, based on communicating
generators
• Kamaelia - Collection of components that use Axon.
• Aim: Scalable, easy & safe concurrent systems
(c) 2005 BBC R&D
3. Kamaelia Status
• Released as open source:
• http://kamaelia.sourceforge.net/
• Axon is at version 1.0.3, and considered feature
stable.
• Runs on Linux, Windows (variety), Mac OS X
• Specialised distribution for Nokia Series 60 mobiles
• Kamaelia is at version 0.1.2, and growing
• Ability to write TCP & Multicast clients and servers
• Variety of simple servers, clients and protocols included
(c) 2005 BBC R&D
4. Kamaelia Status
• Kamaelia 0.1.2:
• Tested on Linux, Windows (variety), Mac OS X
• Subset on Nokia Series 60 mobiles
• Ease of use hypothesis has been tested with 1 pre-
university trainee, looks promising
(c) 2005 BBC R&D
5. Kamaelia Motivations
• Large Scale Streaming
• Several million streams per day
• Big events have tens of thousands of concurrent viewers
• Want to scale to handling millions of concurrent viewers
• Since this could happen.
(c) 2005 BBC R&D
6. Kamaelia Motivations
• What If 10 years from now...
• the BBC opened the entire archive?
• Creative Archive is NOT that ambitious! (AFAIK)
• the entire UK got broadband?
• Instantly hit long tail problems
• 20 million homes?
• 20 million different things?
• Not like 20 million people watching BBC1 !
(c) 2005 BBC R&D
7. Kamaelia Motivations
• Key Problems:
• RTP was originally concieved for A/V conferencing/telephony
• Aspects don’t scale well for large scale unidirectional streaming
• Need a platform for designing, implementing and testing new open
standards to scale in this way.
• Scalability and ability to experiment often conflict.
• Large scale means highly parallel
• Scalable concurrency often has a high barrier to entry
• Limits new ideas, collaboration
(c) 2005 BBC R&D
8. Axon
• Kamaelia's Core Concurrency System
• Key aims:
• Scalable appoach
• Reusable
• Simple - easy enough for novice programmer to pick up and
produce useful systems.
• Novices see possibilities, not problems
• Safe - it should be possible to write programs without worrying
about race hazards
• Non locking if possible
(c) 2005 BBC R&D
9. Scaling Concurrency
• quot;Threads are for people who cant program state
machines.quot; -- Alan Cox (http://tinyurl.com/a66es)
• Processes/Threads/Build your own
• Processes and threads are well known to be not scalable cross
platform.
• Build your own:
• Normally means state machines
• What about people who quot;cant program state machinesquot; ?
• (Not a dig at Alan !)
(c) 2005 BBC R&D
10. Scalability : State machines
• Hard to get 100% right - especially for novices
• Debugging someone else’s - twice as hard
• State machine is a piece of sequential processing that
can release control half way and be restarted retaining
state
• Twisted - at it’s heart very state machine based.
• Provides a very good framework for this and provides lots of
high quality assistance
• Still has this barrier to entry (my personal opinion,YMMV)
(c) 2005 BBC R&D
11. Scalability or ease ?
Do we really have to choose?
• Consider:
• A state machine is a piece of sequential processing that can
release control half way and be restarted
• A generator is a piece of sequential processing that can release
control half way and be restarted
• Twisted also recognises this: twisted.flow
• Takes a different approach to composition
• Kamaelia uses generators
• Hypothesised this would be easier for novices
(c) 2005 BBC R&D
12. Kamaelia vs Twisted?
• NO!
• Kamaelia could be integrated into twisted (or vice versa) - we just
haven't looked at that yet
• Twisted is stable, mature and usable for production
systems
• Kamaelia isn't mature or suitable for production systems at
present
• Won’t always be that way, but even when it isn’t we’d rather
collaborate rather than compete.
• Lengthy answer in Kamaelia’s blog
(c) 2005 BBC R&D
13. Concurrency is Easy ?
• Concurrency is hard
• ... so why do we let sys admins do it?
• Think unix pipelines:
• find -type f | egrep -v '/build/|^./MANIFEST' |while read i ;
do cp ../Source/$i $i done
• This has 4 logically concurrent units!
• Do unix sys admins think of themselves concurrent programmers?
• Do you think of it that way?
(c) 2005 BBC R&D
14. Unix Pipelines
• Concurrent sequential processes - linear
• Items don't know what's next in the pipeline
• Simply communicate with local file handles
• Often forgotten “hidden” details:
• How data passes between processes
• The system environment
(c) 2005 BBC R&D
15. Axon - Key classes
• Components - self pausing sequential objects that
send data to local interfaces
• Linkages - a facility for joining interfaces, allowing
system composition
• Scheduler - gives components CPU time
• Postman - The facility for tracking linkages, and
handling data transferral
• Co-ordinating Assistant/Tracker (cat) - Provides
environmental facilities akin to a Linda tuple space
(c) 2005 BBC R&D
16. Axon Components
• Classes with a generator method called quot;mainquot;
• Augmented by:
• List of Inboxes - defaults: inbox, control
• List of Outboxes - defaults: outbox, signal
• class Echo(component):
def main(self):
while 1:
if self.dataReady(quot;inboxquot;):
data = self.recv(quot;inboxquot;)
self.send(data,quot;outboxquot;)
yield 1
(c) 2005 BBC R&D
17. Axon Scheduler
• Operation
• Holds a run queue containing activated components
• Calls the generator for each component sequentially
• Component Activation
• If the return value is a newComponent object the components
contained are activated (essentially their main() method is called,
and the resulting generator stored)
• Component Deactivation
• If the return value is false, the component is removed from the run
queue
(c) 2005 BBC R&D
18. Linkages
• Normally join outboxes to inboxes between
components
• out-out and in-in also allowed between parent and nest
components
• Linkages can only be create inside a component
• Inboxes and outboxes designed for connection to subcomponents
are considered private and have the naming convention of a
leading underscore
• Encourages composition and reuse
(c) 2005 BBC R&D
19. Linkage Example
• class SimpleStreamingClient(component):
def main(self):
client=TCPClient(quot;127.0.0.1quot;,1500)
decoder = VorbisDecode()
player = AOAudioPlaybackAdaptor()
self.link((client,quot;outboxquot;), (decoder,quot;inboxquot;)
self.link((decoder,quot;outboxquot;), (player,quot;inboxquot;))
self.addChildren(decoder, player, client)
yield newComponent(decoder, player, client)
while 1:
self.pause()
yield 1
(c) 2005 BBC R&D
20. Linkage Example 2
def AdHocFileProtocolHandler(filename):
class klass(Kamaelia.ReadFileAdaptor.ReadFileAdaptor):
def __init__(self,*argv,**argd):
self.__super.__init__(filename, readmode=quot;bitratequot;, bitrate=400000)
return klass
class SimpleStreamingServer(component):
def main(self):
server = SimpleServer(protocol=AdHocFileProtocolHandler (quot;foo.oggquot;),
port=clientServerTestPort)
self.addChildren(server)
yield _Axon.Ipc.newComponent(*(self.children))
while 1:
self.pause()
yield 1
(c) 2005 BBC R&D
22. Co-ordinating Assistant Tracker
• Tracking Services
• This allows for the concept of services
• A service is a mapping of name to (component, inbox) tuple
• Only ever quot;needquot; one 'select' statement in a program for example.
(want is a different matter!)
• The Kamaelia.Internet.Selector component offers a quot;selectorquot;
service
• Tracking Values
• Provides look up and modification of values for keys
• Use case: to enable stats collection in servers
(c) 2005 BBC R&D
23. Howto: Example Component
• MIME/RFC2822 type objects are common in
network protocols
• Email, web, usenet, etc..
• Essentially serialised key/value pairs - much like a
dict.
• Create a “MIME Dict” component.
• Accepts dict like objects, but translates them to MIME-like
messages
• Accepts MIME-like messages, and converts them to dicts.
(c) 2005 BBC R&D
24. MimeDictComponent
• How it was written
• First of all a class that could be a quot;MIME dictquot; was written
• Subclasses dict
• Always adds a __BODY__ key
• Replaces __str__ with something that displays the dict as an
RFC2822/MIME style message
• Adds a staticmethod quot;fromStringquot; as a factory method.
• Written entirely test first without a view
to being used as a component
(c) 2005 BBC R&D
25. MimeDictComponent 2
• Wanted a component thus:
• control - on which we may receive a shutdown message
• signal - one which we will send shutdown messages
• demarshall - an inbox to which you send strings for turning into
dicts
• marshall - an inbox to which you send objects for turning into
strings
• demarshalled - an outbox which spits out parsed strings as
dicts
• marshalled = an outbox which spits out translated dicts as
strings
(c) 2005 BBC R&D
26. MimeDictComponent 3
• Turned out to be simpler to write a generic marshalling
component instead, main loop looked like this:
while 1:
self.pause()
if self.dataReady(quot;controlquot;):
data = self.recv(quot;controlquot;)
if isinstance(data, Axon.Ipc.producerFinished)
self.send(Axon.Ipc.producerFinished(), quot;signalquot;)
return
if self.dataReady(quot;marshallquot;):
data = self.recv(quot;marshallquot;)
self.send(str(data),quot;marshalledquot;)
if self.dataReady(quot;demarshallquot;):
data = self.recv(quot;demarshallquot;)
self.send(self.klass.fromString(data),quot;demarshalledquot;)
yield 1
(c) 2005 BBC R&D
27. MimeDictComponent 4
• Subclassing approach:
• class MimeDictMarshaller(MarshallComponent):
def __init__(self,*argv,**argd):
self.__super.__init__(MimeDict, *argv,**argd)
• Class decoration approach:
• def MarshallerFactory(klass):
class newclass(MarshallComponent):
def __init__(self,*argv,**argd):
self.__super.__init__(klass, *argv,**argd)
return newclass
MimeDictMarshaller=MarshallerFactory(MimeDict)
(c) 2005 BBC R&D
28. Summary: New Components
• Longer tutorial based around a multicast
transceiver on the website.
• Same approach:
• Don't worry about concurrency, write single threaded
• When code works, then convert to components
• Change control methods into inboxes/outboxes
(c) 2005 BBC R&D
29. Ease of use?
• Tested on Ciaran Eaton, a pre-university trainee
• Happy to let me call him a novice programmer (triple checked)
• Previous experience: A-Level computer studies - small amount of
Visual Basic programming and Access
• 3 Month placement with our group
• Started off learning python & axon (2 weeks)
• Created a “learning system” based around parsing a Shakespeare
play:
• Performs filtering, character identification, demultiplexing etc
• Used pygame for display, stopped short of using pyTTS...
(c) 2005 BBC R&D
30. Ease of use? 2
• Ciaran’s project:
• Created a simplistic low bandwidth video streamer
• Server has an MPEG video, and takes a frame as JPEG every n
seconds
• This is sent to the client over a framing protocol Ciaran designed
and implemented
• The client then displays the images as they arrive
• On a PC this uses pygame
• On a series 60 mobile this uses the native image display calls
• The idea is this simulates previewing PVR content on a mobile
(c) 2005 BBC R&D
31. Ease of use? 3
• Project was successful, Ciaran achieved the goals
• Ciaran wrote all the components for every part of
the description.
• Relied on a “SimpleServer” and simple “TCPclient”
components - but these only provide reliable data
transfer over the network.
• He’s noted that it was a fun experience
• I find it interesting it was not frustrating given his background.
(c) 2005 BBC R&D
32. CSP vs State Machines
• Is this approach inherently worse or better?
• We would suggest neither.
• State machine systems often have intermediate
buffers (even single variables) for handoff between
state machines
• This is akin to outboxes and inboxes. If they are
collapsed into one, as planned, this is equivalent
• If we do collapse outboxes into inboxes when we create linkages,
then the system should be as efficient as frameworks like twisted.
• This is currently hypothetical.
(c) 2005 BBC R&D
33. Integration with other systems
• Default component provides a default main, which
calls 3 default callbacks.
• Looks like this:
• def main(self):
result = self.initialiseComponent()
if not result:
result = 1
yield result
while(result):
result = self.mainBody()
if result:
yield result
yield self.closeDownComponent()
(c) 2005 BBC R&D
34. Integration: 2
• Purpose of the 3 callback form is for 2 main
reasons
• For those who find callback forms easier to work with
• To allow these methods to be overridden by classes written in:
• Pyrex
• C
• C++
• ie optimisation of components
(c) 2005 BBC R&D
35. Futures
• C++ Version.
• Simple “miniaxon” version including C++ based generators
working. see: cvs:/Code/CPP/Scratch/miniaxon.cpp
• Python Axon will be optimised
• Syntactic Sugar will be added
• Automated component distribution over clusters
• Kamaelia Component Repository
• More protocols, experimental servers:
• RTSP/RTP initially. New protocols to follow!
(c) 2005 BBC R&D
36. Finally: Collaboration
• If you’re interested in working with us, please do
• If you find the code looks vaguely interesting, please use and give
us feedback
• We’re very open to exploring changes to the system and willing to
give people CVS commit access in order to try their ideas.
• Anyone working with twisted is very welcome to come and
criticise and suggest new ideas - integration would be very nice!
• Contacts, project blog:
• michaels@rd.bbc.co.uk, kamaelia-list@lists.sourceforge.net
• http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi
(c) 2005 BBC R&D