This was the Kamaelia Tutorial at Europython. It goes from basics - ie building a mini-kamaelia from scratch, through to a file multicaster, through a video recording application all the walk through to a multiuser bulletin board system.
4: Embracing Concurrency (Michael Sparks)Imran Ali
This document summarizes a presentation about embracing concurrency using the Kamaelia framework. It discusses how Kamaelia represents components as objects with inbox and outbox ports that communicate asynchronously through message passing. This avoids shared state and makes concurrency easier and safer to work with. Kamaelia provides high-level abstractions for common concurrency patterns like pipelines, graphs, servers and backplanes for publish-subscribe. Examples demonstrate how to build games, multimedia and server applications using these abstractions. The document argues this approach can make concurrency more accessible and less error-prone for novice and expert developers alike.
This document summarizes two different approaches to porting mobile apps from the iPhone to netbooks:
1) A "Picard-like" method taken by developer Mike Kasprzak, which involved rewriting the app in C++ using custom libraries and assets to facilitate an easier port.
2) A "Kirk-like" rapid porting approach by Chris Skaggs of exporting code to the Adobe Flex framework and adding a C++ wrapper to meet submission requirements.
Both approaches adjusted the apps to take advantage of the netbook's features and had to address issues like screen orientation differences. The summary advocates planning for cross-platform development and optimizing apps for each specific device.
This is the second presentation of the course "Developing Software that Matters". This course is being given by Franco Gasperoni to the second year students of the ENST in Paris. Distributed under the GFDL
Embracing concurrency for fun utility and simpler codekamaelian
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming more concurrent, but most programming languages and tools treat concurrency as difficult. The Kamaelia project aims to make concurrency easy and usable for novice and advanced developers alike through fundamental control structures and messaging between components. Examples shown include using pipelines, graphlines, servers, and backplanes to build concurrent applications in a simple way.
This document guides the reader through building a system where a user connects to a server over a secure connection and receives a sequence of JSON-encoded objects. It begins by introducing the ServerCore component and shows how to fill its protocol handler factory hole. It then demonstrates creating a stackedjson protocol handler using a pipeline of components like PeriodicWakeup, Chooser, and MarshallJSON. This protocol securely transmits JSON data chunks to clients like a ConsoleEchoer. It discusses how the client-side mirrors the server components to receive and display the messages.
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming massively concurrent, providing an opportunity. While concurrency is viewed as hard, the fundamental problem may be lack of proper tools. Imperative languages often overlook concurrency as a core concept. A variety of desktop and media applications could benefit from a concurrent approach. The document advocates using concurrent components that communicate via messages while keeping data private. It also discusses software transactional memory and different perspectives in APIs for concurrent systems. Finally, it presents examples of using pipelines and graphlines as part of a concurrency domain specific language.
Practical concurrent systems made simple using Kamaeliakamaelian
This talk was given at Pycon UK 2008 in Birmingham.
This presentation aims to teach you how to get started with Kamaelia, building a variety of systems, as well as walking through the design and implementation of some systems built in the last year. Systems built over the past year include tools for dealing with spam (greylisting), through database modelling, video & image transcoding for a youtube/flickr type system, paint programs, webserving, XMPP, games, and a bunch of other things.
The document is a quiz about characters and details from the anime Bleach. It asks questions about Ichigo's zanpakto name, Grimmjow's Espada number, the main character's name, and Zohan's favorite drink. The respondent answers some questions correctly and others incorrectly, receiving feedback that is sometimes encouraging and other times critical depending on whether the answer was right or wrong.
4: Embracing Concurrency (Michael Sparks)Imran Ali
This document summarizes a presentation about embracing concurrency using the Kamaelia framework. It discusses how Kamaelia represents components as objects with inbox and outbox ports that communicate asynchronously through message passing. This avoids shared state and makes concurrency easier and safer to work with. Kamaelia provides high-level abstractions for common concurrency patterns like pipelines, graphs, servers and backplanes for publish-subscribe. Examples demonstrate how to build games, multimedia and server applications using these abstractions. The document argues this approach can make concurrency more accessible and less error-prone for novice and expert developers alike.
This document summarizes two different approaches to porting mobile apps from the iPhone to netbooks:
1) A "Picard-like" method taken by developer Mike Kasprzak, which involved rewriting the app in C++ using custom libraries and assets to facilitate an easier port.
2) A "Kirk-like" rapid porting approach by Chris Skaggs of exporting code to the Adobe Flex framework and adding a C++ wrapper to meet submission requirements.
Both approaches adjusted the apps to take advantage of the netbook's features and had to address issues like screen orientation differences. The summary advocates planning for cross-platform development and optimizing apps for each specific device.
This is the second presentation of the course "Developing Software that Matters". This course is being given by Franco Gasperoni to the second year students of the ENST in Paris. Distributed under the GFDL
Embracing concurrency for fun utility and simpler codekamaelian
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming more concurrent, but most programming languages and tools treat concurrency as difficult. The Kamaelia project aims to make concurrency easy and usable for novice and advanced developers alike through fundamental control structures and messaging between components. Examples shown include using pipelines, graphlines, servers, and backplanes to build concurrent applications in a simple way.
This document guides the reader through building a system where a user connects to a server over a secure connection and receives a sequence of JSON-encoded objects. It begins by introducing the ServerCore component and shows how to fill its protocol handler factory hole. It then demonstrates creating a stackedjson protocol handler using a pipeline of components like PeriodicWakeup, Chooser, and MarshallJSON. This protocol securely transmits JSON data chunks to clients like a ConsoleEchoer. It discusses how the client-side mirrors the server components to receive and display the messages.
The document discusses embracing concurrency for simpler code. It notes that hardware is becoming massively concurrent, providing an opportunity. While concurrency is viewed as hard, the fundamental problem may be lack of proper tools. Imperative languages often overlook concurrency as a core concept. A variety of desktop and media applications could benefit from a concurrent approach. The document advocates using concurrent components that communicate via messages while keeping data private. It also discusses software transactional memory and different perspectives in APIs for concurrent systems. Finally, it presents examples of using pipelines and graphlines as part of a concurrency domain specific language.
Practical concurrent systems made simple using Kamaeliakamaelian
This talk was given at Pycon UK 2008 in Birmingham.
This presentation aims to teach you how to get started with Kamaelia, building a variety of systems, as well as walking through the design and implementation of some systems built in the last year. Systems built over the past year include tools for dealing with spam (greylisting), through database modelling, video & image transcoding for a youtube/flickr type system, paint programs, webserving, XMPP, games, and a bunch of other things.
The document is a quiz about characters and details from the anime Bleach. It asks questions about Ichigo's zanpakto name, Grimmjow's Espada number, the main character's name, and Zohan's favorite drink. The respondent answers some questions correctly and others incorrectly, receiving feedback that is sometimes encouraging and other times critical depending on whether the answer was right or wrong.
Photoshop is a program used for graphic design, anime photos, and real life photos. The Photoshop Club meets on Tuesdays and Thursdays at lunch in Portable 5 to learn Photoshop skills like editing photos. Photoshop allows users to edit and enhance images for both work and fun purposes.
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaeliakamaelian
This presentation on Kamaelia at Euro OSCON 2006, and specifically focusses
on a particular system - Kamaelia Macro which is essentially a system for
timeshifting pretty much everything.
This presentation was given at Python North West. It explains a complete Kamaelia application for greylisting which was written specifically to eliminate my personal spam problem. It walks through the code as well (though that's best looked at with the code side by side!)
This talk was the keynote talk at the EBU's Seminar on Open Source Oct 1st, 2nd 2007. http://www.ebu.ch/en/technical/opensource/
The video referenced is IBM's "Prodigy" advert, which can be found here: http://youtube.com/watch?v=q5Kp1Q39VwI
In this talk I talked about how,in the Kamaelia project, we manage the dilemma of encouraging innovation and creativity in a project whilst maintaining an engineered solution. Why? Because we find it allows a high level of creative freedom, whilst also providing a path through to a high level of confidence in the reliabilty of the final code.
RabbitMQ with python and ruby RuPy 2009Paolo Negri
The document discusses using RabbitMQ, an open-source message broker based on AMQP, for asynchronous messaging between Python and Ruby applications. It provides an overview of AMQP concepts like producers, consumers, exchanges and queues, and how RabbitMQ implements these using Erlang. Code examples are shown for sending and receiving messages asynchronously in Python and Ruby.
This document discusses messaging with AMQP and RabbitMQ. It provides an overview of why messaging is useful for decoupling systems and making them asynchronous and reliable. It then discusses why AMQP and RabbitMQ are good choices for messaging and covers some basic AMQP concepts like exchanges, queues, bindings and routing. It also discusses how Spring supports AMQP and RabbitMQ through its AMQP libraries and templates. Finally, it briefly covers how web messaging can be done through STOMP over websockets.
Data Analysis and Statistics in Python using pandas and statsmodelsWes McKinney
The document summarizes Wes McKinney's talk on statistical computing using Python. The talk introduces the scientific Python stack, including pandas for data structures and data analysis, and statsmodels for statistical modeling. It discusses the "research-production gap" in current statistical tools and how Python aims to bridge that gap. McKinney asserts that Python is the best solution for both research and production use of statistics and data analysis. He then demonstrates pandas and statsmodels functionality.
This document provides an overview of the Django web framework. It discusses Django's mission of encouraging rapid development and clean design. It demonstrates how to create a blog application with Django, including generating models, activating the admin interface, defining URLs and views, and using templates to display data. It also briefly compares Django to other frameworks like Ruby on Rails.
Django is a high-level Python web framework that encourages rapid development and clean design. It makes building web apps faster with less code by providing models, templates, views, and URL patterns. To build a project, you create a project folder using startproject, then add apps using startapp which contain models, views, and other files. You sync the database, configure URLs and views, then run the development server to view your new app.
Python for Financial Data Analysis with pandasWes McKinney
This document discusses using Python and the pandas library for financial data analysis. It provides an overview of pandas, describing it as a tool that offers rich data structures and SQL-like functionality for working with time series and cross-sectional data. The document also outlines some key advantages of Python for financial data analysis tasks, such as its simple syntax, powerful built-in data types, and large standard library.
This document discusses object-oriented programming concepts in Python including multiple inheritance, method resolution order, method overriding, and static and class methods. It provides examples of multiple inheritance where a class inherits from more than one parent class. It also explains method resolution order which determines the search order for methods and attributes in cases of multiple inheritance. The document demonstrates method overriding where a subclass redefines a method from its parent class. It describes static and class methods in Python, noting how static methods work on class data instead of instance data and can be called through both the class and instances, while class methods always receive the class as the first argument.
Python Advanced – Building on the foundationKevlin Henney
This is a two-day course in Python programming aimed at professional programmers. The course material provided here is intended to be used by teachers of the language, but individual learners might find some of this useful as well.
The course assume the students already know Python, to the level taught in the Python Foundation course: http://www.slideshare.net/Kevlin/python-foundation-a-programmers-introduction-to-python-concepts-style)
The course is released under Creative Commons Attribution 4.0. Its primary location (along with the original PowerPoint) is at https://github.com/JonJagger/two-day-courses/tree/master/pa
Python Tricks That You Can't Live WithoutAudrey Roy
Audrey Roy gave a presentation on Python tricks for code readability and reuse at PyCon Philippines 2012. She discussed writing clean, understandable code by following PEP8 style guidelines and using linters. She also explained how to find and install reusable Python libraries from the standard library and PyPI, and how to write packages and modules to create reusable code.
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...Spark Summit
This document summarizes Wes McKinney's talk on improving Python and Spark performance and interoperability. The talk discusses how Spark DataFrames can appear faster than they really are since Python is only a scripting front-end. It also covers issues like inefficient data movement and interpreter overhead when using Python UDFs. Apache Arrow is presented as a technology to help make things faster by enabling zero-copy data transfers and fast messaging between processes. Work done to improve the speed of DataFrame.toPandas using Arrow is discussed. The talk concludes with areas for future work like performing the Spark SQL to Arrow conversion locally on task executors.
Slides from our CodeMash 2013 Precompiler session, "Web Development with Python and Django", including a breezy introduction to the Python programming language and the Django web framework. The example code repository is available at https://github.com/finiteloopsoftware/django-precompiler/
Study: The Future of VR, AR and Self-Driving CarsLinkedIn
We asked LinkedIn members worldwide about their levels of interest in the latest wave of technology: whether they’re using wearables, and whether they intend to buy self-driving cars and VR headsets as they become available. We asked them too about their attitudes to technology and to the growing role of Artificial Intelligence (AI) in the devices that they use. The answers were fascinating – and in many cases, surprising.
This SlideShare explores the full results of this study, including detailed market-by-market breakdowns of intention levels for each technology – and how attitudes change with age, location and seniority level. If you’re marketing a tech brand – or planning to use VR and wearables to reach a professional audience – then these are insights you won’t want to miss.
The document discusses embracing concurrency for simplicity and utility. It argues that current programming tools teach concurrency wrong by emphasizing shared data and threads over message passing. It presents an approach using concurrent components that communicate through message passing. This avoids shared data issues while enabling reusable concurrency patterns through high-level constructs like pipelines, graphs, and backplanes. Examples demonstrate creating servers and streaming applications using this approach.
Building a full-stack app with Golang and Google Cloud Platform in one weekDr. Felix Raab
This document discusses building a full-stack application called MemeMail using Golang and Google Cloud Platform within one week. It describes choosing Google Cloud over other cloud providers for its ease of use. It then discusses the frontend implementation using Vue.js with a simple state mutation approach. The backend is built with Golang on App Engine using Cloud services like Datastore and Cloud Build for CI/CD. It emphasizes keeping the architecture simple rather than over-engineering for an MVP within a tight deadline.
Photoshop is a program used for graphic design, anime photos, and real life photos. The Photoshop Club meets on Tuesdays and Thursdays at lunch in Portable 5 to learn Photoshop skills like editing photos. Photoshop allows users to edit and enhance images for both work and fun purposes.
Timeshift Everything, Miss Nothing - Mashup your PVR with Kamaeliakamaelian
This presentation on Kamaelia at Euro OSCON 2006, and specifically focusses
on a particular system - Kamaelia Macro which is essentially a system for
timeshifting pretty much everything.
This presentation was given at Python North West. It explains a complete Kamaelia application for greylisting which was written specifically to eliminate my personal spam problem. It walks through the code as well (though that's best looked at with the code side by side!)
This talk was the keynote talk at the EBU's Seminar on Open Source Oct 1st, 2nd 2007. http://www.ebu.ch/en/technical/opensource/
The video referenced is IBM's "Prodigy" advert, which can be found here: http://youtube.com/watch?v=q5Kp1Q39VwI
In this talk I talked about how,in the Kamaelia project, we manage the dilemma of encouraging innovation and creativity in a project whilst maintaining an engineered solution. Why? Because we find it allows a high level of creative freedom, whilst also providing a path through to a high level of confidence in the reliabilty of the final code.
RabbitMQ with python and ruby RuPy 2009Paolo Negri
The document discusses using RabbitMQ, an open-source message broker based on AMQP, for asynchronous messaging between Python and Ruby applications. It provides an overview of AMQP concepts like producers, consumers, exchanges and queues, and how RabbitMQ implements these using Erlang. Code examples are shown for sending and receiving messages asynchronously in Python and Ruby.
This document discusses messaging with AMQP and RabbitMQ. It provides an overview of why messaging is useful for decoupling systems and making them asynchronous and reliable. It then discusses why AMQP and RabbitMQ are good choices for messaging and covers some basic AMQP concepts like exchanges, queues, bindings and routing. It also discusses how Spring supports AMQP and RabbitMQ through its AMQP libraries and templates. Finally, it briefly covers how web messaging can be done through STOMP over websockets.
Data Analysis and Statistics in Python using pandas and statsmodelsWes McKinney
The document summarizes Wes McKinney's talk on statistical computing using Python. The talk introduces the scientific Python stack, including pandas for data structures and data analysis, and statsmodels for statistical modeling. It discusses the "research-production gap" in current statistical tools and how Python aims to bridge that gap. McKinney asserts that Python is the best solution for both research and production use of statistics and data analysis. He then demonstrates pandas and statsmodels functionality.
This document provides an overview of the Django web framework. It discusses Django's mission of encouraging rapid development and clean design. It demonstrates how to create a blog application with Django, including generating models, activating the admin interface, defining URLs and views, and using templates to display data. It also briefly compares Django to other frameworks like Ruby on Rails.
Django is a high-level Python web framework that encourages rapid development and clean design. It makes building web apps faster with less code by providing models, templates, views, and URL patterns. To build a project, you create a project folder using startproject, then add apps using startapp which contain models, views, and other files. You sync the database, configure URLs and views, then run the development server to view your new app.
Python for Financial Data Analysis with pandasWes McKinney
This document discusses using Python and the pandas library for financial data analysis. It provides an overview of pandas, describing it as a tool that offers rich data structures and SQL-like functionality for working with time series and cross-sectional data. The document also outlines some key advantages of Python for financial data analysis tasks, such as its simple syntax, powerful built-in data types, and large standard library.
This document discusses object-oriented programming concepts in Python including multiple inheritance, method resolution order, method overriding, and static and class methods. It provides examples of multiple inheritance where a class inherits from more than one parent class. It also explains method resolution order which determines the search order for methods and attributes in cases of multiple inheritance. The document demonstrates method overriding where a subclass redefines a method from its parent class. It describes static and class methods in Python, noting how static methods work on class data instead of instance data and can be called through both the class and instances, while class methods always receive the class as the first argument.
Python Advanced – Building on the foundationKevlin Henney
This is a two-day course in Python programming aimed at professional programmers. The course material provided here is intended to be used by teachers of the language, but individual learners might find some of this useful as well.
The course assume the students already know Python, to the level taught in the Python Foundation course: http://www.slideshare.net/Kevlin/python-foundation-a-programmers-introduction-to-python-concepts-style)
The course is released under Creative Commons Attribution 4.0. Its primary location (along with the original PowerPoint) is at https://github.com/JonJagger/two-day-courses/tree/master/pa
Python Tricks That You Can't Live WithoutAudrey Roy
Audrey Roy gave a presentation on Python tricks for code readability and reuse at PyCon Philippines 2012. She discussed writing clean, understandable code by following PEP8 style guidelines and using linters. She also explained how to find and install reusable Python libraries from the standard library and PyPI, and how to write packages and modules to create reusable code.
Improving Python and Spark Performance and Interoperability: Spark Summit Eas...Spark Summit
This document summarizes Wes McKinney's talk on improving Python and Spark performance and interoperability. The talk discusses how Spark DataFrames can appear faster than they really are since Python is only a scripting front-end. It also covers issues like inefficient data movement and interpreter overhead when using Python UDFs. Apache Arrow is presented as a technology to help make things faster by enabling zero-copy data transfers and fast messaging between processes. Work done to improve the speed of DataFrame.toPandas using Arrow is discussed. The talk concludes with areas for future work like performing the Spark SQL to Arrow conversion locally on task executors.
Slides from our CodeMash 2013 Precompiler session, "Web Development with Python and Django", including a breezy introduction to the Python programming language and the Django web framework. The example code repository is available at https://github.com/finiteloopsoftware/django-precompiler/
Study: The Future of VR, AR and Self-Driving CarsLinkedIn
We asked LinkedIn members worldwide about their levels of interest in the latest wave of technology: whether they’re using wearables, and whether they intend to buy self-driving cars and VR headsets as they become available. We asked them too about their attitudes to technology and to the growing role of Artificial Intelligence (AI) in the devices that they use. The answers were fascinating – and in many cases, surprising.
This SlideShare explores the full results of this study, including detailed market-by-market breakdowns of intention levels for each technology – and how attitudes change with age, location and seniority level. If you’re marketing a tech brand – or planning to use VR and wearables to reach a professional audience – then these are insights you won’t want to miss.
The document discusses embracing concurrency for simplicity and utility. It argues that current programming tools teach concurrency wrong by emphasizing shared data and threads over message passing. It presents an approach using concurrent components that communicate through message passing. This avoids shared data issues while enabling reusable concurrency patterns through high-level constructs like pipelines, graphs, and backplanes. Examples demonstrate creating servers and streaming applications using this approach.
Building a full-stack app with Golang and Google Cloud Platform in one weekDr. Felix Raab
This document discusses building a full-stack application called MemeMail using Golang and Google Cloud Platform within one week. It describes choosing Google Cloud over other cloud providers for its ease of use. It then discusses the frontend implementation using Vue.js with a simple state mutation approach. The backend is built with Golang on App Engine using Cloud services like Datastore and Cloud Build for CI/CD. It emphasizes keeping the architecture simple rather than over-engineering for an MVP within a tight deadline.
Building frameworks: from concept to completionRuben Goncalves
What are considerations when building a framework/library? How does that apply to OutSystems components? In this session, we’ll do a deep dive into the importance of addressing certain concepts like code granularity, and architecture, in order to create useful, future-proof and coherent frameworks that deliver the best possible developer experience.
APIdays Barcelona 2019 - How to build a social network on Serverless with Yan...apidays
This document discusses how to build a social network using serverless architecture. It outlines reasons for using a serverless approach, including delivering better user experiences faster, delivering value faster, and being more cost efficient. It then describes how to build a social network serverlessly using microservices, event-driven architecture, and AWS Lambda. It provides examples of challenges overcome and benefits realized by a company that migrated an existing social network to serverless.
How to build a social network on serverlessYan Cui
Many people are building different workloads using serverless technologies these days, but how would a non-trivial system such as a social network look like on serverless?
In this talk Yan will discuss his journey of migrating a social network startup to serverless, and how his team was able to improve performance, scalability and feature delivery using serverless technologies.
Yan will discuss how serverless technologies such as Lambda are used to implement each part of their system, including search, push notifications, timeline, user recommendations, and business intelligence. If you're wondering how serverless can be used to solve a wide variety of challenges in your business, this is the talk for you.
The document discusses emerging technologies for online classrooms and distance learning. It compares the features of different online classroom platforms like Connect Pro and Webex, sharing the experiences of using these platforms for a blended learning project. Recommendations are provided for effective use of these platforms based on what worked well and challenges encountered, like using chat, polls and icons to maintain student engagement. Emerging technologies that could further enhance online learning are also discussed.
Introduction to Building Wireframes - Part 2lomalogue
This document provides an introduction to building wireframes. It discusses what wireframes are and why they are useful, as well as tools for sketching, creating mockups, and prototypes. Low-fidelity wireframes can be sketched on paper, while high-fidelity ones can be created using mockup software like Keynote, Balsamiq, or by coding prototypes with frameworks like Twitter Bootstrap. The document also covers getting stakeholder buy-in for wireframes, testing them with users, and techniques like A/B testing to improve designs.
Eric Long provides a summary of his work experience and skills. He has over 20 years of experience developing software, primarily using C++. Some of his major projects include animation software Moho and MotionArtist, as well as iOS apps like Sock Puppets. He strengthened security and expanded features for these products. Long is skilled in multiple programming languages and build systems, and enjoys helping users achieve their goals through software.
Internet Programming With Python PresentationAkramWaseem
This document provides an overview and agenda for the "Internet Programming with Python" tutorial being given at OSCON 2009. The tutorial will cover network programming with sockets in Python, including an introduction to client-server architecture and networking concepts. It will also cover internet client programming, CGI programming, and conclude with a question and answer session. The target audience is software engineers and system administrators with a basic Python knowledge seeking an introduction to internet programming topics in Python.
This was a session Brian Verkley and I delivered in Las Vegas for EMC World 2016 called 12 Factor App FTW ! In this presentation we talked to each of the 12 factors and how it can relate to the operations side of the house.
This document discusses whether there is a need for a Bahasa Malaysia version of Firefox. It notes that Firefox usage in Malaysia has grown 34% in the past 6 months. While some say that only English software is used, the author believes there is a need for localized software. Examples like Facebook's localization are discussed. The document invites discussion on whether a Malay version of Firefox is needed and how to spread the word if so. It provides contact information for those interested in joining the open source project.
This is a retrospective of my 30 year career in Software Engineering. Most of the tools I worked on have to do with Java, Eclipse, Python, visualization, profiling, and people.
1) The document discusses how technology has evolved from being controlled by IT departments to being driven by users through open source platforms and social/mobile applications.
2) It describes how users now want consumer-like experiences in the workplace, leading to the rise of systems of engagement that are more open, social, and user-centric.
3) These systems of engagement work alongside traditional systems of record, with both being needed to balance user experience with information security and governance.
The document discusses the evolution of IT from centralized systems controlled by vendors to more open and social systems. It describes how user needs and expectations changed, driving demand for more consumer-like applications in the enterprise. Systems of engagement emerged to meet this demand. Content became more diverse and social features like collaboration and sharing became important. The boundaries between systems of record and systems of engagement blurred as both were needed. Open source, cloud computing, and social aspects became more prominent. Alfresco is positioned as an open source social content management platform to address these trends.
The document provides guidance on designing a complex web application by breaking it into multiple microservices or applications. It recommends asking questions about team size, traffic patterns, priorities for speed vs stability, existing APIs or libraries, and programming languages. Based on the answers, it suggests appropriate frameworks, languages, data storage, testing/deployment processes, and server/container management options. The overall goal is to modularize the application, leverage existing tools when possible, and not overengineer parts of the design.
ITCamp 2013 - Andy Cross - Broadcasting Music from the CloudITCamp
This document summarizes a presentation on using real-time web technologies like SignalR to broadcast music from Spotify to multiple users simultaneously. The presentation discusses challenges like scalability and synchronizing playback across clients. It proposes using Azure Services like Websites and Service Bus to host the real-time web application and help it scale. The goal is to demonstrate how these services can be used to build an interactive music sharing experience where all users' playback remains synchronized.
Sharing Data and Services Safely in Concurrent Systems using Kamaeliakamaelian
Kamaelia is generally a \"shared nothing\" architecture, but occasionally you *really* need to share data explicitly. When you do, you need to constrain how you share data and careful about how you advertise services. This is the first presentation done on the facilities that exist in Kamaelia to support this.
This presentation was given at Pycon UK 2008, Birmingham uk. Lots of good feedback was had during the q&a and an updated & improved version will be posted at some point in the relatively near future.
This presentation aims to show people that they already know how to deal with concurrency.
It argues that if we have the tools for large scale concurrency (mashups) and small scale (hardware) that midrange (normal apps) can be done in a similar way, using existing tools.
This is done by showing useful systems that have been produced in this manner using existing tools. ie from existing practice, not theory
During the actual presentation I also talked about Kamaelia projects created by novice programmers of varying ability which show high levels of concurrency.
These include: previewing PVR content on mobiles, multicast island joining, as-live streaming using bit torrent, Open GL based user interfaces & integration, seaside style webserving, speex based secure phone, IRC/IM systems, a shakespeare script player, and games tools.
Other systems created include Atom/RSS routing, memcached integration, P2P whiteboarding (with audio + mixing), gesture recognition, presentation tools, a kids development environment, topology visualisation tools, database modelling etc.
Open Source at the BBC: When, Why, Why not & Howkamaelian
This talk was given at Linux World 2006. It covers 3 aspects of open source at the BBC - use, extension & origination through the 4 lenses of when, why, why not & how. It focusses entirely on pragmatics in all cases. The style is Lessig style. A write up on the text can be found here: http://tinyurl.com/yd4j2y
This was an invited talk at Open Source Forum Russia in April 2005. It covers open source at the BBC from the perspective of "why use open source?" "what sort of stuff gets used?" "what has the BBC released as open source & why?" open source vs open standards
This talk was part tongue in cheek, part serious, but entirely fun and given twice as a lightning talk - once at Europython & once at the ACCU python uk 05. It presents a generic python like language parser which does actually work. Think of it as an alternative to brackets in Lisp!
This was a talk, largely on Kamaelia & its original context given at a Free Streaming Workshop in Florence, Italy in Summer 2004. Many of the core
concepts still hold valid in Kamaelia today
This talk was given at Pycon UK 07. It's actually a thin wrapper around the
Kamaelia Mini Axon tutorial which can be found here:
http://kamaelia.sourceforge.net/MiniAxon/
This was a talk on how to build systems with Kamaelia given at Pycon UK. It
goes through from basics through to building a swarming P2P live radio
system.
The document discusses free and open source software. It begins by defining free software as software that users have the freedom to use, study, distribute, and change. It notes that free software is also known as libre software or open source software. The document outlines several advantages of free software such as giving users control, reducing costs, using open standards, sustainability, skills development, and improved security and quality. It also briefly discusses some potential disadvantages like smaller installed bases and issues of compatibility with proprietary software. Overall, the document presents an overview of the key concepts around free and open source software.
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Skybuffer SAM4U tool for SAP license adoptionTatiana Kojar
Manage and optimize your license adoption and consumption with SAM4U, an SAP free customer software asset management tool.
SAM4U, an SAP complimentary software asset management tool for customers, delivers a detailed and well-structured overview of license inventory and usage with a user-friendly interface. We offer a hosted, cost-effective, and performance-optimized SAM4U setup in the Skybuffer Cloud environment. You retain ownership of the system and data, while we manage the ABAP 7.58 infrastructure, ensuring fixed Total Cost of Ownership (TCO) and exceptional services through the SAP Fiori interface.
In the realm of cybersecurity, offensive security practices act as a critical shield. By simulating real-world attacks in a controlled environment, these techniques expose vulnerabilities before malicious actors can exploit them. This proactive approach allows manufacturers to identify and fix weaknesses, significantly enhancing system security.
This presentation delves into the development of a system designed to mimic Galileo's Open Service signal using software-defined radio (SDR) technology. We'll begin with a foundational overview of both Global Navigation Satellite Systems (GNSS) and the intricacies of digital signal processing.
The presentation culminates in a live demonstration. We'll showcase the manipulation of Galileo's Open Service pilot signal, simulating an attack on various software and hardware systems. This practical demonstration serves to highlight the potential consequences of unaddressed vulnerabilities, emphasizing the importance of offensive security practices in safeguarding critical infrastructure.
Building Production Ready Search Pipelines with Spark and MilvusZilliz
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Taking AI to the Next Level in Manufacturing.pdfssuserfac0301
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
This presentation provides valuable insights into effective cost-saving techniques on AWS. Learn how to optimize your AWS resources by rightsizing, increasing elasticity, picking the right storage class, and choosing the best pricing model. Additionally, discover essential governance mechanisms to ensure continuous cost efficiency. Whether you are new to AWS or an experienced user, this presentation provides clear and practical tips to help you reduce your cloud costs and get the most out of your budget.
Have you ever been confused by the myriad of choices offered by AWS for hosting a website or an API?
Lambda, Elastic Beanstalk, Lightsail, Amplify, S3 (and more!) can each host websites + APIs. But which one should we choose?
Which one is cheapest? Which one is fastest? Which one will scale to meet our needs?
Join me in this session as we dive into each AWS hosting service to determine which one is best for your scenario and explain why!
Main news related to the CCS TSI 2023 (2023/1695)Jakub Marek
An English 🇬🇧 translation of a presentation to the speech I gave about the main changes brought by CCS TSI 2023 at the biggest Czech conference on Communications and signalling systems on Railways, which was held in Clarion Hotel Olomouc from 7th to 9th November 2023 (konferenceszt.cz). Attended by around 500 participants and 200 on-line followers.
The original Czech 🇨🇿 version of the presentation can be found here: https://www.slideshare.net/slideshow/hlavni-novinky-souvisejici-s-ccs-tsi-2023-2023-1695/269688092 .
The videorecording (in Czech) from the presentation is available here: https://youtu.be/WzjJWm4IyPk?si=SImb06tuXGb30BEH .
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on integration of Salesforce with Bonterra Impact Management.
Interested in deploying an integration with Salesforce for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
leewayhertz.com-AI in predictive maintenance Use cases technologies benefits ...alexjohnson7307
Predictive maintenance is a proactive approach that anticipates equipment failures before they happen. At the forefront of this innovative strategy is Artificial Intelligence (AI), which brings unprecedented precision and efficiency. AI in predictive maintenance is transforming industries by reducing downtime, minimizing costs, and enhancing productivity.
zkStudyClub - LatticeFold: A Lattice-based Folding Scheme and its Application...Alex Pruden
Folding is a recent technique for building efficient recursive SNARKs. Several elegant folding protocols have been proposed, such as Nova, Supernova, Hypernova, Protostar, and others. However, all of them rely on an additively homomorphic commitment scheme based on discrete log, and are therefore not post-quantum secure. In this work we present LatticeFold, the first lattice-based folding protocol based on the Module SIS problem. This folding protocol naturally leads to an efficient recursive lattice-based SNARK and an efficient PCD scheme. LatticeFold supports folding low-degree relations, such as R1CS, as well as high-degree relations, such as CCS. The key challenge is to construct a secure folding protocol that works with the Ajtai commitment scheme. The difficulty, is ensuring that extracted witnesses are low norm through many rounds of folding. We present a novel technique using the sumcheck protocol to ensure that extracted witnesses are always low norm no matter how many rounds of folding are used. Our evaluation of the final proof system suggests that it is as performant as Hypernova, while providing post-quantum security.
Paper Link: https://eprint.iacr.org/2024/257
Best 20 SEO Techniques To Improve Website Visibility In SERPPixlogix Infotech
Boost your website's visibility with proven SEO techniques! Our latest blog dives into essential strategies to enhance your online presence, increase traffic, and rank higher on search engines. From keyword optimization to quality content creation, learn how to make your site stand out in the crowded digital landscape. Discover actionable tips and expert insights to elevate your SEO game.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
1. Kamaelia:
Pragmatic Concurrency
A Tutorial
Michael Sparks
Europython '09, Birmingham UK
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
2. About me
• - Been using python for several years
• - Always been interested in concurrency
• - Kamaelia aims to embody safe practices.
• - Work at BBC R&D's Northern Lab, based in
Manchester, moving to Media City:UK in 2011
• - Kamaelia is born out of a variety of R&D
projects, and shaped by needs, not aesthetics
or purity.
• Kamaelia is not an active R&D project. It is used in R&D projects,
and hence actively maintained.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
3. About me
•
Disclaimer: Like you, I'm doing this on my time,
not the BBC's. This doesn't represent BBC
opinion on anything.
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
4. About Kamaelia
• - Also born from a desire to make concurrency
in programs easier to work with
•
- Because it's nice in the shell – I want software
concurrency that easy :-)
• - Expressiveness is favoured over performance,
but not to preclude optimisation
• - Means we're cautious about adding syntactic
sugar.
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
5. About Kamaelia
• - Adapts Unix Philosophy to make a program
concurrent internally, but with the purpose of
simplifying maintenance.
• Unix Philosophy:
•
Write programs that do one thing and do it well.
Write programs to work together.
Write programs to handle text streams, because that is
a universal interface.
• --Doug McIlroy
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
6. About Kamaelia
• - Adapts Unix Philosophy to make a program
concurrent internally, but with the purpose of
simplifying maintenance.
• Kamaelia Philosophy:
•
Write components that do one thing and do it well.
Write components to work together.
Write components to handle python object streams,
because that is a universal interface.
• --With apologies to Doug McIlroy
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
7. What we're covering
• - An overview of Kamaelia
• & it's view on concurrency
• - How to build a mini Kamaelia
• – to get under the hood.
• - Building components & systems
• - Examining larger systems, and debugging.
• - Building a large(ish) system
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
8. Time Estimates
• First part:
• - An overview of Kamaelia (lightning talkstyle : 5-10 mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
After break:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
9. Format
• I generally welcome questions at any point
• That said...
• 'Except in the overview
• - the rest of this morning should answer them!!
• Copious notes provided - covers more than
today
• Materials available from URL below
• Is a mixture of “explain then do”.
Perhaps hold questions for during “do” :-) ?
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
10. Caveat
•
• * First time I've given this tutorial !
• * If we run short of time on any section, we'll
skip ahead.
•
• * BUT, have copious notes that cover
everything.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
11. Questions?
• ... before we dive in?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
12. Part 1
•
• First part:
• - An overview of Kamaelia (lightning talk style : 5-10
mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
13. Kamaelia, a 20:20 overview
20:20 Presentation
A presentation style based on the “Pecha Kucha”
style 20 slides, 20 seconds each
(Similar to a lightning talk)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
14. Why? Hardware finally going
massively concurrent ...
Opportunity! .... PS3, high end servers, trickling down to desktops, laptops)
“many hands make light
work” but Viewed as Hard
... do we just have crap tools?
“And one language to in
Problems
the darkness bind them”
... can just we REALLY abandon 50 years of code for Erlang, Haskell
and occam?
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
15. Missing Something?
Fundamental Control Structures
... in imperative languages number greater than 3! (despite what you get taught...!)
Control Structure Traditional Abstraction Biggest Pain Points
Sequence Function Global Var
Selection Function Global Var
Iteration Function Global Var
Parallel Thread Shared Data
Usually Skipped Lost or duplicate update
are most common bugs
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
16. Regarding using
concurrency, what sort
of applications are we
talking about here?
Desktop gsoc
Media Novice
APPS trainee
Network 3rd Party
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
17. Think backend Speak 'n Write
P2P Whiteboard Programming
needed for Simple
ER DB Modeller (logo)
youtube/flickr Kids Games P2P Radio
Torrent
type systems 3D Systems
Compose Realtime Music
UGC Desktop gsoc Paint App
Backend P2P Web Server
Secure “phone”
Transcoder Social Net Vis
Media Novice ...
Shot Change
Detection APPS trainee
MiniAxon
ScriptReader
Mobile DVB MediaPreview
Reframing on Mobile
Reliable
Macro
Podcasts
Network 3rd Party Multicast
Sedna
“record
everything” XMPP XMLDB
Email & AWS pubsub
Spam Web (Amazon)
SMTP IRC Qt
Serving Gtk
Greylisting
Pop3Proxy ClientSide AIM microblogging
Spam Tools
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
18. Core Approach:
Concurrent things with comms points
Generally send messages
Keep data private, don't share
outbox inbox outbox
inbox
signal control signal
control
... ... ...
...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
19. But I must share data?
Use Software Transactional Memory
ie version control for variables.
1. Check out the collection
of values you wish to
work on
2. Change them
3. Check them back in
4. If conflict/clash, go
back to 1
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
20. Perspectives in APIs! (1/2)
If you have concurrency it becomes
natural to think in terms of 1st 2nd and 3rd
person. This affects an API's structure, 1st Person - I change my state
and can be vital for understanding it!
This is one we've found that makes sense
2nd Person – YOU 3rd Person –
want to me to do Bla should
something do something
(you send outbox (I send a message)
inbox
me a message)
control signal
... ...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
21. Perspectives in APIs! (2/2)
If you have concurrency it becomes
natural to think in terms of 1st 2nd and 3rd
person. This affects an API's structure, private real methods
and can be vital for understanding it!
This is one we've found that makes sense
Messages Messages sent
from public to public outboxes
inboxes
inbox outbox
control signal
Also, think ... ... Also, think
about stdin about stdout
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
22. Actor Systems
Distinction can be unclear,
potential source of ambiguity* private real methods
Messages
from public No outbox concept
inboxes Possible issues with
inbox rate limiting*
control
Hardcodes recipient
...
in the sender
*system dependent issue
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
23. Advantages of outboxes
No hardcoding of recipient
allows:
- Late Binding
- Dynamic rewiring
inbox outbox
Concurrency Patterns as
control signal Reusable Code
... ... ... a concurrency DSL
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
24. A Core Concurrency DSL
Pipeline(A,B,C)
Graphline(A=A,B=B, C=C, linkages = {})
Tpipe(cond, C)
Seq(A,B,C), PAR(), ALT()
Backplane(“name”), PublishTo(“name”), SubscribeTo(“name”)
Carousel(...)
PureTransformer(...)
StatefulTransformer(...)
PureServer(...)
MessageDemuxer(...)
Source(*messages)
NullSink
Some of these are work in progress
– they've been identified as useful,
but not implemented as chassis, yet
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
26. Graphline Example
Graphline(
NEXT = Button(...),
PREVIOUS = Button(...), PREVIOUS NEXT
FIRST = Button(...), (button) (button)
LAST = Button(...),
CHOOSER = Chooser(...),
IMAGE = Image(...), FIRST LAST
(button) (button)
...
).run()
Chooser
Image
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
27. Server Example
data
from
user data
Main Socket handler
Server Core to
user
Created at runtime
to handle the
Protocol Handler Factory connection Remote
Protocol handler User
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
28. Server Example
You therefore
need to provide
Main this bit.
Server Core
Protocol Handler Factory
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
29. Server Example
from Kamaelia.Chassis.ConnectedServer import ServerCore
from Kamaelia.Util.PureTransformer import PureTransformer
def greeter(*argv, **argd):
return PureTransformer(lambda x: "hello" +x)
class GreeterServer(ServerCore):
protocol=greeter
port=1601
GreeterServer().run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
30. Backplane Example
# Streaming Server for raw DVB of Radio 1
Backplane(“Radio”).activate()
Pipeline(
DVB_Multiplex(850.16, [6210], feparams), # RADIO ONE
PublishTo("RADIO"),
).activate()
def radio(*argv,**argd):
return SubscribeTo(“RADIO”)
ServerCore(protocol=radio, port=1600).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
31. So that's the 5 minute version
Short Q&A before we move on?
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
32. Part 1
•
• First part:
• - An overview of Kamaelia (lightning talk style : 5-10
mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
33. Mini Axon
• Kamaelia is divided into two halves
• * One part handles all the concurrency
stuff, providing you a component model
• * The other is a large collection of
components and some apps using them.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
34. Mini Axon
• Axon is the part that handles
concurrency and provides the
component model, and is the key to
understanding why & how Kamaelia
works.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
35. Mini Axon
• ... is a collection of exercises where
you build just such a beast.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
36. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
37. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
38. Generators
• * Python's smallest unit of concurrency
• * Single function you call, get a
generator back
• * Can then call it's .next() method to
* Get a new value from it
* Give it CPU time
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
39. Fibonacci Generator
•
def fib(a,b):
while 1:
yield a
a, b = b, b + a
•
•
Demo
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
40. Fibonacci Generator
•
>>> def fib(a,b):
... while 1:
... yield a
... a, b = b, b + a
...
>>> g = fib(1,1)
>>> g
<generator object at 0xb7bf9c6c>
>>> [ g.next() for _ in range(10) ]
[1, 1, 2, 3, 5, 8, 13, 21, 34, 55]
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
41. Lots of Generators
•
Using the same fib generator, make 10 of them:
>>> GS = [ fib(x,x) for x in range(10) ]
• And “run” them:
>>> [ G.next() for G in GS ]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [ G.next() for G in GS ]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> [ G.next() for G in GS ]
[0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
>>> [ G.next() for G in GS ]
[0, 3, 6, 9, 12, 15, 18, 21, 24, 27]
>>> [ G.next() for G in GS ]
[0, 5, 10, 15, 20, 25, 30, 35, 40, 45]
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
42. Generators as co-routines
•
def fib(a,b):
while 1:
yield 1 # Just to say “keep running me”
print a
a, b = b, b + a
•
•
Demo
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
43. Generators as co-routines
•
def printer(tag):
while 1:
yield 1 # Makes it a generator
print tag
•
•
Demo
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
44. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
45. Microprocesses
• * Generators with context
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
46. microprocess exercise
•
Write a class microprocess with methods:
• __init__(self)
* Takes no arguments
* Uses super to call superclass __init__ method
• main(self)
* No arguments
* Should yield 1
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
47. microprocess answer
•
• class microprocess(object):
def __init__(self):
super(microprocess, self).__init__()
def main(self):
yield 1
•
Generally, we'll skip answers in this presentation,
they're all in the web version of this tutorial here:
•
http://www.kamaelia.org/MiniAxon/
•
(and in the notes :-)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
48. microprocess usage
•
class printer(microprocess):
def __init__(self, tag):
super(printer, self).__init__()
self.tag = tag
def main(self):
while 1:
yield 1
print self.tag
•
• “Printer” isn't particularly interesting, but allows things like
components, but let's see how this can be used.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
50. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
51. Scheduler
•
* Something to run lots of microprocesses
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
52. scheduler exercise
•
Write a class scheduler with 3 methods:
• __init__(self)
* Uses super to call superclass __init__ method
* subclasses microprocess
* Creates 2 queues – active & newqueue
• main(self)
* Is a generator
* Runs the microprocesses activated
•
activateMicroprocess(self, someprocess):
* Calls somprocess.main()
* Adds generator to newqueue
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
53. scheduler exercise
• Scheduler logic
•
main(self)
* loops 100 times, yields 1 at start of loop
* loop through generators in self.active
* call their .next() method
* If result is not -1 and no StopIteration,
append to newqueue
* at end of loop, newqueue becomes active
* newqueue reset to empty list
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
54. scheduler usage
•
Using same printer class..
•
>>> X = printer("Hello World")
>>> Y = printer("Game Over") # :-)
• >>> myscheduler = scheduler()
>>> myscheduler.activateMicroprocess(X)
>>> myscheduler.activateMicroprocess(Y)
• >>> for _ in myscheduler.main():
... pass
• <prints Hello world/Game over repeatedly>
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
55. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
56. Component
•
• * microprocess with a standard interface.
• boxes as inboxes/outboxes
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
57. component exercise
•
Write a class component subclass of
microprocess with 4 methods:
•
__init__(self)
send(self, value, outboxname)
recv(self, inboxname)
dataReady(self, inboxname)
• Behaviour coming up!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
59. component exercise 2/4
•
send(self,value, outboxname) logic:
Finds the list named outboxname in self.boxes,
and appends value to it.
•
Before: X.send(“hello”, “outbox”)
{ “inbox”: [], “outbox”: [] }
• After:
{ “inbox”: [], “outbox”: [“hello”] }
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
60. component exercise 3/4
•
recv(self, inboxname) logic:
Finds the list named inboxname in self.boxes, and
pops the first value
• Given:
{ “inbox”: [“hello”, “world”], “outbox”: [] }
•
self.recv(“inbox”) returns, “hello” leaving...
{ “inbox”: [“world”], “outbox”: [] }
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
61. component exercise 4/4
•
dataReady(self, inboxname) logic:
Finds the list named inboxname in self.boxes:
returns the length of the list.
• Given:
{ “inbox”: [“hello”, “world”], “outbox”: [] }
•
dataReady(“inbox”) --> 2
(allows if self.dataReady(“inbox”) )
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
62. component usage
•
Until we add a means for data to get
from outboxes to inboxes, using
components is no more interesting than
microprocesses
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
63. Mini Axon
•
•
Generators
* Python's smallest unit of concurrency
•
Microprocesses
* Generators with context
• Scheduler
* Something to run lots of microprocesses
• Components
* A microprocess with boxes (treated as in & outboxes)
•
Postman
* Something to do deliveries
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
64. Postman
• * Something to do deliveries
•
• Note:
This is more conceptual in real Axon,
but was real in v. old Axon. ie real
Axon is more efficient!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
65. postman exercise
•
Write a class postman subclass of microprocess
with 2 methods:
•
__init__(self, source, sourcebox,
sink, sinkbox)
main(self)
• Behaviour coming up!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
66. postman exercise 1/4
•
__init__(self, source, sourcebox,
sink, sinkbox) logic:
* Copy all the arguments as local attributes
•
* Ensure you call the superclass __init__ method
appropriately!
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
67. postman exercise 2/4
•
main(self) logic:
• In a loop:
• yield a non -1 value (eg 1)
Check if source's sourcebox has
dataReady
• If it has, use recv to collect it from there, and sink's
send method to deliver it.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
68. Mini Axon
•
• Using it!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
70. Producer/Consumer 2/3
•
• class Consumer(component):
def main(self):
count = 0
while 1:
yield 1
count += 1
if self.dataReady("inbox"):
data = self.recv("inbox")
print data, count
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
71. Producer/Consumer 3/3
•
• p = Producer("Hello World")
c = Consumer()
postie = postman(p, "outbox", c, "inbox")
• myscheduler = scheduler()
myscheduler.activateMicroprocess(p)
myscheduler.activateMicroprocess(c)
myscheduler.activateMicroprocess(postie)
•
for _ in myscheduler.main():
pass
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
72. Producer/Consumer Output
•
• Hello World 2
Hello World 3
Hello World 4
• ...
•
Hello World 96
Hello World 97
Hello World 98
•
•
Not 100, because of yields before start of scheduler
loop. (scheduler terminates early)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
73. Mini Axon
•
• Using it for more useful stuff
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
74. File Reader
•
• class FileReader(component):
• def __init__(self, filename):
super(FileReader, self).__init__()
self.file = open(filename, "rb",0)
•
def main(self):
yield 1
for line in self.file.xreadlines():
self.send(line, "outbox")
yield 1
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
78. Mini Axon --> Axon
•
• The rest is:
Syntactic Sugar
Ways of using it
Optimisations
(eg direct delivery – no postman)
+ a couple of other ideas we'll come
to (STM, threadedcomponents, services,
processcomponents)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
79. Questions?
•
• ... before we look how to build some
real components, and how to use them?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
80. Part 1
•
• First part:
• - An overview of Kamaelia (lightning talk style : 5-10
mins)
• - How to build a mini Kamaelia (exercise 40-60 mins)
• - Starting building components & systems (remainder)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
81. Building a Video Recorder
•
• Before the break, we'll build a simple video recorder.
• Approach:
• 1 Build a webcam
• 2 Clean up the code for “normal” reuse
• 3 Componentise in least effort manner
• 4 Separate input/transform/display parts
• 5 Hook webcam up to a dirac encoder and filewriter
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
82. Step 1: Build a simple Webcam
• Pygame 1.9.1 alpha has basic Linux webcam support
which works nicely, so we're using that.
• First some initialisation
• import pygame
import pygame.camera
• pygame.init()
pygame.camera.init()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
84. Step 1: Build a simple Webcam
• Initialise the display, allocate a camera, and activate it
• display = pygame.display.set_mode(displaysize)
camera = pygame.camera.Camera(device,
capturesize)
camera.start()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
85. Step 1: Build a simple Webcam
• Loop round capturing images and display them
• while 1:
snapshot = camera.get_image()
snapshot = pygame.transform.scale(snapshot,
imagesize)
display.blit(snapshot, imageorigin)
pygame.display.flip()
•
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
86. Step 2: Clean up for reuse
• Given a collection of config options, wrapping this in a
class makes sense.
• As before these parts are unchanged:
• import pygame
import pygame.camera
• pygame.init()
pygame.camera.init()
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
87. Step 2: Clean up for reuse
• We then define a class, and put the config options as
class attributes:
• class VideoCapturePlayer(object):
• displaysize = (800, 600)
capturesize = (640, 480)
imagesize = (352, 288)
imageorigin = (0,0)
device = “/dev/video0”
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
88. Step 2: Clean up for reuse
• Put our initialisation in the __init__. Allow the user to
override our defaults
def __init__(self, **argd):
self.__dict__.update(**argd)
super(VideoCapturePlayer,self).__init__()
self.display = pygame.display.set_mode(self.displaysize)
self.camera = pygame.camera.Camera(self.device,
self.capturesize)
self.camera.start()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
89. Step 2: Clean up for reuse
• Wrap up body of loop in a method
def get_and_flip(self):
snapshot = self.camera.get_image()
snapshot = pygame.transform.scale(snapshot,
self.imagesize)
self.display.blit(snapshot, self.imageorigin)
pygame.display.flip()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
90. Step 2: Clean up for reuse
• Provide a hook to start it going, wrapping the main loop.
def main(self):
while 1:
self.get_and_flip()
•
Then run it!
VideoCapturePlayer().main()
•
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
91. Step 3: Componentise
•
• In this case, least effort approach is to use a threaded
component. Would could make it a generator component
later if needed.
• Changes:
• * imports
* class's baseclass / extraction of display_flip into a
method. (to simplify later integration with existing
components)
* how we run it.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
93. Step 3: Componentise
•
• Baseclass changes & initialiser changes:
class VideoCapturePlayer(threadedcomponent ):
...
def __init__(self, **argd):
# no longer update __dict__ here
super(VideoCapturePlayer,self).__init__(**argd )
...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
94. Step 3: Componentise
•
• Extract display flipping out to a separate method:
def pygame_display_flip(self):
pygame.display.flip()
def get_and_flip(self):
snapshot = self.camera.get_image()
snapshot = pygame.transform.scale(snapshot,
self.imagesize)
self.display.blit(snapshot, self.imageorigin)
self.pygame_display_flip()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
95. Step 3: Componentise
•
• Change to how we run:
• We had:
•
VideoCapturePlayer().main()
• We now:
• VideoCapturePlayer().run()
•
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
96. Step 4: Split into input & output
•
• Now we can split the component in two:
• * VideoCaptureSource
• * Surface Displayer
• The Video capture source now needs to be self regulating,
so it needs to sleep during the loop, relative to a target
frame rate.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
97. Step 4: Split into input & output
•
• VideoCaptureSource class preamble:
• import time
class VideoCaptureSource(threadedcomponent):
capturesize = (352, 288)
delay = 0
fps = -1
device = “/dev/video0”
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
99. Step 4: Split into input & output
•
• Main loop body now just captures images, capturing a
reference
•
def capture_one(self):
•
self.snapshot = None
self.snapshot = self.camera.get_image()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
100. Step 4: Split into input & output
•
• Main loop still runs it, and sends the image out an
outbox, and then sleeps.
•
def main(self):
•
while 1:
self.capture_one()
self.send(self.snapshot, “outbox”)
time.sleep(self.delay)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
101. Step 4: Split into input & output
•
• The surface displayer takes the other code chunks
• from Axon.Component import component
•
class SurfaceDisplayer(component):
displaysize = (800,600)
imagesize = (352, 288)
imageorigin = (0,0)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
102. Step 4: Split into input & output
•
• Has it's own initialiser...
• def __init__(self, **argd):
super(SurfaceDisplayer, self).__init__(**argd)
self.display = pygame.display.set_mode(self.displaysize)
•
Retains the following method unchanged:
•
def pygame_display_flip(self):
pygame.display.slip()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
103. Step 4: Split into input & output
• Then mainbody waits for surfaces to display, sleeping
when there isn't any:
• def main(self):
•
while 1:
while self.dataReady(“inbox”):
snapshot = self.recv(“inbox”)
snapshot = pygame.transform.scale(snapshot,
self.imagesize)
self.display.blit(snapshot, self.imageorigin)
•
while not self.anyReady():
self.pause()
yield 1
•
yield 1
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
104. Step 4: Split into input & output
•
• We then join these back together in a pipeline:
•
from Kamaelia.Chassis.Pipeline import Pipeline
•
Pipeline(
•
VideoCaptureSource(),
•
SurfaceDisplayer(),
•
).run()
•
Demo!
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
105. Step 5: Recording
• Working back...
•
• We want to write to a file... (SimpleWriter)
• We want to write dirac encoded video DiracEncoder
• That expects YUV Frames (ToYUV420_planar)
• That expects RGB data + metadata
PureTransformation of somr RGB data
• Which needs a picture source
•
Which is where we started.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
106. Step 5: Recording
• Our imports therefore need to add:
•
• from Kamaelia.File.Writing import SimpleFileWriter
• from Kamaelia.Codec.Dirac import DiracEncoder
• from Kamaelia.Video.PixFormatConversion import ToYUV420_planar
• from Kamaelia.Util.PureTransformer import PureTransformer
•
• And we delete everything related to display
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
107. Step 5: Recording
• We then need to join it all together:
•
Pipeline(
•
VideoCaptureSource(),
PureTransformer(lambda F :
{"rgb" : pygame.image.tostring(F, "RGB"),
"size" : (352, 288),
"pixformat" : "RGB_interleaved",
}),
ToYUV420_planar(),
DiracEncoder(preset="CIF", encParams={"num_L1":0}),
SimpleFileWriter("X.drc"),
• ).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
108. Step 5: Recording
•
• This then records dirac encoded video, which we can now
playback with a simple dirac player!
•
•
Demo !
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
109. Questions?
• ... before we break?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
110. End of Part 1
•
• Before the second half, download and install Kamaelia
from the link below, if you haven't already.
•
• http://tinyurl.com/kot49x
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
111. Kamaelia:
Pragmatic Concurrency
A Tutorial
Michael Sparks
Europython '09, Birmingham UK
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
112. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems, embedding and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
113. Questions?
• ... before we carry on?
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
114. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
115. More advanced Stuff
• Have seen how to build a basic component and a basic
system - covers a wide set of problems. Now we
broaden the scope.
•
•
If we have time we'll come back to Kamaelia's STM model &
concepts of services, and (experimental) multiple process
support.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
116. Well behaved Shutdown
• Components should expect to receive these messages on their
“control” inbox, and behave as follows:
• Axon.Ipc.shutdownMicroprocess – if you receive this, you are
expected to shutdown immediately, and to pass this message on.
• Axon.Ipc.producerFinished – if you receive this, someone sending
you data has shutdown. You may want to shutdown depending on your
application's logic. You may process all outstanding messages in this
case. You may wish to pass this message on, or change it to
shutdownMicroprocess if it was unexpected.
• Due to different component needs no syntactic sugar exists for this,
but common cases are being discussed at present.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
122. Pipelines
• Seen how Pipelines can join components together.
• However also can do:
• Pipeline(
Lsystem(),
ConsoleEchoer(tag="n", forwarder=True),
Damage(),
circular = True,
).run()
•
To enable a feedback loop. (Demo)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
125. Graphlines
• Graphlines are like pipelines, but any shape. For
example, a simple presentation tool:
• Graphline(
CHOOSER = Chooser(items = files),
IMAGE = Image(size=(800,600), position=(8,48)),
NEXT = Button(caption="Next", msg="NEXT", position=(72,8)),
PREVIOUS = Button(caption="Previous", msg="PREV", position=(8,8)),
FIRST = Button(caption="First", msg="FIRST",position=(256,8)),
LAST = Button(caption="Last", msg="LAST",position=(320,8)),
linkages = {
("NEXT","outbox") : ("CHOOSER","inbox"),
("PREVIOUS","outbox") : ("CHOOSER","inbox"),
("FIRST","outbox") : ("CHOOSER","inbox"),
("LAST","outbox") : ("CHOOSER","inbox"),
("CHOOSER","outbox") : ("IMAGE","inbox"),
}
).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
126. Graphlines
•
• * Swiss Army Knife of Kamaelia components
• * Allows almost any topology
• - BUT an outbox may only link to one inbox
• - many outbox may link to one inbox
• * Can contain any component – including pipelines and
graphlines
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
132. Backplanes
•
• * For where you want 1 to many or many to many
• * Declare a backplane
• * Components can publish data to it
• * Components may subscribe to it.
• * Subscribers get a copy of all data sent
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
133. Backplanes
• Can be useful for updating a central state.
•
• For example players locations can be posted here:
• Backplane("PLAYERS").activate()
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
134. Backplanes
• Players Post their updates like this:
• Pipeline(
• MyGamesEventsComponent(up="p", down="l",
left="a", right="s"),
• BasicSprite("cat.png", name = "cat", border=40),
• PureTransformer(lambda x: ("Cat ", x)),
• PublishTo("PLAYERS"),
• ).activate()
•
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
135. Backplanes
• The system can analyse their updates like this:
• Pipeline(
• SubscribeTo("PLAYERS"),
• PlayerAnalyser(),
• Distancer(),
• ConsoleEchoer(),
• ).activate()
•
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
136. Backplanes & Servers
• When combined with a server, you instantly get a pub/
sub capable server, or splitter, or merger.
• Backplane(“SPLIT”).activate()
Pipeline(
DiracWebCam(),
PublishTo(“SPLIT”),
).activate()
def VideoProtocol(**argd): return SubscribeTo(“SPLIT”)
• ServerCore(protocol=VideoProtocol, port=1700).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
137. Backplanes & Servers
• When combined with a server, you instantly get a pub/
sub capable server, or splitter, or merger.
• Backplane(“CHAT”).activate()
def ChatProtocol(**argd):
return Pipelines(
SubscribeTo(“CHAT”)
NullComponent(),
PublishTo(“CHAT”)
)
• ServerCore(protocol=ChatProtocol, port=1700).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
138. PAR & Seq
• PAR & Seq are wrapper components with concepts
nabbed from Occam. (hence their names)
•
* Seq runs each component it's given, one after
another, wiring up inboxes/outboxes such that each
component handles it.
Useful in staged protocols (later)
•
* PAR runs all the components concurrently. Their
output is merged. Shutdown messages sent to PAR
are forwarded to all subcomponents – making it
useful for managing system shutdown.
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
139. Using PAR
• PAR can be used to simplify some systems:
• Pipeline(
PAR(
Button(caption="Next", msg="NEXT", position=(72,8)),
Button(caption="Previous", msg="PREV", position=(8,8)),
Button(caption="First", msg="FIRST" ,position=(256,8)),
Button(caption="Last", msg="LAST", position=(320,8)),
),
• Chooser(items = files),
Image(size=(800,600), position=(8,48)),
).run()
• This is the same slideshow as the previous Graphline...
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
140. Using PAR
• ... to run & shutdown multiple subsystems
• Pipeline(
timedShutdown(TTL=15),
PAR(
Pipeline(
ReadFileAdaptor(file, readmode="bitrate",
bitrate = 300000*8/5),
DiracDecoder(),
MessageRateLimit(framerate),
VideoOverlay(position=(260,48), size=(200,300)),
),
Pipeline(
ReadFileAdaptor(file, readmode="bitrate", bitrate = 2280960*8),
DiracDecoder(),
ToRGB_interleaved(),
VideoSurface(size=(200, 300), position=(600,48)),
),
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
142. Using Seq
• An example from the mobile reframer:
• Seq( "Decoding & separating frames...",
Graphline(
MAXF = DetermineMaxFrameNumber(edlfile),
DO = Carousel( ... ),
STOP = TriggeredOneShot(""),
...
),
"Processing edits...",
Graphline(
REFRAMING = ReframeVideo(edlfile...
SOUND = PassThroughAudio(edlfile...
ENCODING = ReEncode(outFileName...
...
),
"Cleaning up...",
StopSelector(),
).run()
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
143. Using Seq
• An example of passing state in a protocol handler:
• def CompositeBulletinBoardProtocol(**argd):
ConnectionInfo = {}
ConnectionInfo.update(argd)
return Seq(
Authenticator(State = ConnectionInfo),
UserRetriever(State = ConnectionInfo),
MessageBoardUI(State = ConnectionInfo),
StateSaverLogout(State = ConnectionInfo),
)
ServerCore(protocol=CompositeBulletinBoardProtocol, ...)
• Worth noting the similarity between this and wsgi.
(Except Seq can contain graphlines, pipelines, etc too)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
144. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
145. Larger Systems & Debugging
•
Large systems which are long running can
develop bugs which are awkward to debug.
•
Kamaelia has a collection of tools you can use
which we'll walk through / demonstrate next.
•
• Practical example.
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
148. Part 2
•
• Second part:
• - More advanced stuff (demo/etc 20 mins)
• - Larger systems and debugging. (20 mins)
• - Building a large(ish) system (exercise 40 mins)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
149. Building a Bulletin Board
•
We're going to build an “old school” bulletin board
system.
• * Someone telnets in & authenticates
• * Can get help, exit or read messages
• * Reading messages, they can read, reply,
exit reading, or get help
• * State is restored/saved at session start/end
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
150. Building a Bulletin Board
•
For practicality, we won't implement message posting,
nor message deletion, nor persistent user state. (Though
these would be useful exercises)
• All on-disk objects encoded as json objects:
* users
* Messages – stored in “folders” (directories) with
filenames == messageid
• * message fields: from, to, __body__, date, message,
reply-to, subject
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
151. Building a Bulletin Board
•
Getting started:
• * Start with basic server for request/response
Just echo initially - BB1.py
• * make restarting quicker for debugging - BB2.py
* abstract out “getting a line of data” - BB3.py
* Then use that abstraction - BB4.py
* Simplify “control” box handling - BB5.py
* Abstract out reusable bits from domain specific – BB6.py
•
We'll go through the code for these in the actual code
files rather than on these slides
•
(rather unwieldy as slides)
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
152. Building a Bulletin Board
•
Intermediate Plan
• This is the logic of the user protocol
• Seq(
•
Authenticator( <some shared state> ),
• UserRetriever( <some shared state> ),
• MessageBoardUI( <some shared state> ),
•
StateSaverLogout( <some shared state> ),
•
)
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
153. Building a Bulletin Board
•
Intermediate Plan
• * Tidy up control handling a touch more & add netPrint method
to reusable bit - BB7.py
• * Change Authenticator component to support the “passed on
state”, change protocol handler factory to create that Seq
pipeline. - BB8.py
•
* Change Authenticator to authenticate against a password file
and set “remoteuser” in the shared state - BB9.py
•
* Write stubs for UserRetriever & StateSaverLogout - BB10.py
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
154. Building a Bulletin Board
•
Final UI Plan
• * Change usage of waitMsg to yield self.waitMsg() -- BB11.py
• * Copy Authenticator's patten to create initial menu for
MessageBoardUI, including stub for messages menu - BB12.py
• * Use same pattern for messages menu, use waitMsg pattern
for logic. - BB13.py
• * Implement Folders to hold messages.
Numbers as filenames (msg 1, 2, 3, 4)
Messages as json encoded objects - BB14.py
•
* Clean up to add a line oriented buffer to handle partial lines.
(necessary for cross platform/real world) - BB15.py
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
155. Building a Bulletin Board
•
Final UI Plan
• * Change usage of waitMsg to yield self.waitMsg() -- BB11.py
• * Copy Authenticator's patten to create initial menu for
MessageBoardUI, including stub for messages menu - BB12.py
• * Use same pattern for messages menu, use waitMsg pattern
for logic. - BB13.py
• * Implement Folders to hold messages.
Numbers as filenames (msg 1, 2, 3, 4)
Messages as json encoded objects - BB14.py
•
* Clean up to add a line oriented buffer to handle partial lines.
(necessary for cross platform/real world) - BB15.py
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
156. Summary
•
We've covered
• * the 30,000 view of kamaelia
• * Building your own core
•
* Building components, including a video recording application
• * Tools for building systems
• * Tools for debugging systems
•
* Built a large(ish) system (an authenticated staged protocol)
•
* Illustrated the majority of this using real world systems
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com
157. Final Questions?
•
•
http://www.kamaelia.org/PragmaticConcurrency sparks.m@gmail.com