Historical view of process and channel oriented programming idioms: CSP 1978, Occam 1983, Pi-Calculus 1993 etc. How they map to Go and some examples of dynamic channel routing using Go to simulate peer-to-peer networks and microservices networks.
Event Stream Processing with Multiple ThreadsSylvain Hallé
We present an extension to the BeepBeep 3 event stream engine that allows the use of multiple threads during the evaluation of a query. Compared to the single-threaded version of BeepBeep, the allocation of just a few threads to specific portions of a query provides improvement in terms of throughput.
Event Stream Processing with Multiple ThreadsSylvain Hallé
We present an extension to the BeepBeep 3 event stream engine that allows the use of multiple threads during the evaluation of a query. Compared to the single-threaded version of BeepBeep, the allocation of just a few threads to specific portions of a query provides improvement in terms of throughput.
Just about all of my current technical content in one 364 slide mega-deck. Source files at https://github.com/adrianco/slides
Sections on:
Scene Setting
State of the Cloud
What Changes?
Product Processes
Microservices
State of the Art
Segmentation
What’s Missing?
Monitoring
Challenges
Migration
Response Times
Serverless
Lock-In
Teraservices
Wrap-Up
Opening talk at Monitorama, talks about the problems of monitoring, challenges of creating monitoring tools and why monitoring vendors keep getting disrupted. Ended with a discussion of simulation testing and serverless architectures - Monitorless.
Microservices: What's Missing - O'Reilly Software Architecture New YorkAdrian Cockcroft
Assuming you have already figured out microservices, what else do you need to figure out to get them to work properly. This talk skips my usual introduction to why and what, and goes deeper on how.
Full slide deck for day long discussion of microservices topics. Why use microservices, what options exist and how to migrate to them and address common problems.
Discussion of how microservices are being applied across both web scale and enterprise/government use cases to help speed up development.
Video available at http://www.ustream.tv/recorded/86151804
There are many ways to manage whether a service can talk to another service. It can be tempting to over-use one segmentation mechanism to implement policy when the real problem is how to coordinate and manage many mechanisms in the physical, cloud and container spaces. This talk summarizes the problem space and opportunities rather than offers solutions.
Presented at the Docker Palo Alto meetup Feb 16th 2016 http://www.meetup.com/Docker-Palo-Alto/events/228277181/
A rough and researchy presentation where I tried out some new material in front of a local audience. Skipped the usual introduction and talked about some of the problems people run into when they do microservices and miss a few things. More refined version of this talk to be shown at O'Reilly Software Architecture Conference in New York in April.
Microxchg Analyzing Response Time Distributions for MicroservicesAdrian Cockcroft
Research oriented presentation @Microxchg Berlin Feb 5th 2016. New code to collect histograms of response time and export them to monte-carlo simulation spreadsheet via getguesstimate.com
Sildes of an internal talk given at Twitter similar to a previous webinar for Redhat with the same title.
Speeding up development is a key concern, cloud and technology improvements like Docker speed up key steps that make continuous delivery possible. Breaking up the work into many separate microservices and datastores with stable APIs allows teams to make progress independently so that the organization scales. Monolithic apps are preferred for small projects, built by small teams and when very low latency and high efficiency is the primary requirement. Monitoring microservices is currently a challenge with solutions starting to emerge.
Summary of fast development and cloud native architecture along with cost optimization techniques. Presented as opening keynote at the Utility and Cloud Computing 2014 as part of the Cloud Control Workshop.
Updated slides for 2016 presentation on innovation in large organizations, why microservices and Docker can be useful, thoughts on monitoring for large complex architectures, some discussion of new topics - serverless architectures AWS Lambda and teraservices.
It's clear that Docker speeds up development and makes testing and deployment more efficient. As Docker moves into production new use cases and patterns are emerging that address availability and security concerns. With microservices, safety is part of the architecture that developers need to understand and build for. It's no longer good enough to wrap a firewall around an entire app when it goes to production, and have a cold standby in case it breaks.
Businesses are speeding up development and automating operations to remain competitive and to get large organizations to scale. Project based monolithic application updates are replaced by product teams owning containerized microservices. This puts developers on call, responsible for pushing code to production, fixing it when it breaks, and managing the cost and security aspects of running their microservices. In this world operations skill-sets are either embedded in the microservices development teams, or building and operating API driven platforms. The platform automates stress testing, canary based deployment, penetration testing and enforces availability and security requirements. There are no meetings or tickets to file in the delivery process for updating a containerized microservice, which can happen many times a day, and takes seconds to complete. The role of site reliability engineering moves from firefighting and fixing outages to buiding tools for finding problems and routing those problems to the right developers. SREs manage the incident lifecycle for customer visible problems, and measure and publish availability metrics. This may sound futuristic but Werner Vogels described this as “You build it, you run it” in 2006.
Python programming language provides the following types of loops to handle looping requirements:
1. While
2. Do While
3. For loop
Python provides three ways for executing the loops. While all the ways provide similar basic functionality, they differ in their syntax and condition-checking time.
Just about all of my current technical content in one 364 slide mega-deck. Source files at https://github.com/adrianco/slides
Sections on:
Scene Setting
State of the Cloud
What Changes?
Product Processes
Microservices
State of the Art
Segmentation
What’s Missing?
Monitoring
Challenges
Migration
Response Times
Serverless
Lock-In
Teraservices
Wrap-Up
Opening talk at Monitorama, talks about the problems of monitoring, challenges of creating monitoring tools and why monitoring vendors keep getting disrupted. Ended with a discussion of simulation testing and serverless architectures - Monitorless.
Microservices: What's Missing - O'Reilly Software Architecture New YorkAdrian Cockcroft
Assuming you have already figured out microservices, what else do you need to figure out to get them to work properly. This talk skips my usual introduction to why and what, and goes deeper on how.
Full slide deck for day long discussion of microservices topics. Why use microservices, what options exist and how to migrate to them and address common problems.
Discussion of how microservices are being applied across both web scale and enterprise/government use cases to help speed up development.
Video available at http://www.ustream.tv/recorded/86151804
There are many ways to manage whether a service can talk to another service. It can be tempting to over-use one segmentation mechanism to implement policy when the real problem is how to coordinate and manage many mechanisms in the physical, cloud and container spaces. This talk summarizes the problem space and opportunities rather than offers solutions.
Presented at the Docker Palo Alto meetup Feb 16th 2016 http://www.meetup.com/Docker-Palo-Alto/events/228277181/
A rough and researchy presentation where I tried out some new material in front of a local audience. Skipped the usual introduction and talked about some of the problems people run into when they do microservices and miss a few things. More refined version of this talk to be shown at O'Reilly Software Architecture Conference in New York in April.
Microxchg Analyzing Response Time Distributions for MicroservicesAdrian Cockcroft
Research oriented presentation @Microxchg Berlin Feb 5th 2016. New code to collect histograms of response time and export them to monte-carlo simulation spreadsheet via getguesstimate.com
Sildes of an internal talk given at Twitter similar to a previous webinar for Redhat with the same title.
Speeding up development is a key concern, cloud and technology improvements like Docker speed up key steps that make continuous delivery possible. Breaking up the work into many separate microservices and datastores with stable APIs allows teams to make progress independently so that the organization scales. Monolithic apps are preferred for small projects, built by small teams and when very low latency and high efficiency is the primary requirement. Monitoring microservices is currently a challenge with solutions starting to emerge.
Summary of fast development and cloud native architecture along with cost optimization techniques. Presented as opening keynote at the Utility and Cloud Computing 2014 as part of the Cloud Control Workshop.
Updated slides for 2016 presentation on innovation in large organizations, why microservices and Docker can be useful, thoughts on monitoring for large complex architectures, some discussion of new topics - serverless architectures AWS Lambda and teraservices.
It's clear that Docker speeds up development and makes testing and deployment more efficient. As Docker moves into production new use cases and patterns are emerging that address availability and security concerns. With microservices, safety is part of the architecture that developers need to understand and build for. It's no longer good enough to wrap a firewall around an entire app when it goes to production, and have a cold standby in case it breaks.
Businesses are speeding up development and automating operations to remain competitive and to get large organizations to scale. Project based monolithic application updates are replaced by product teams owning containerized microservices. This puts developers on call, responsible for pushing code to production, fixing it when it breaks, and managing the cost and security aspects of running their microservices. In this world operations skill-sets are either embedded in the microservices development teams, or building and operating API driven platforms. The platform automates stress testing, canary based deployment, penetration testing and enforces availability and security requirements. There are no meetings or tickets to file in the delivery process for updating a containerized microservice, which can happen many times a day, and takes seconds to complete. The role of site reliability engineering moves from firefighting and fixing outages to buiding tools for finding problems and routing those problems to the right developers. SREs manage the incident lifecycle for customer visible problems, and measure and publish availability metrics. This may sound futuristic but Werner Vogels described this as “You build it, you run it” in 2006.
Python programming language provides the following types of loops to handle looping requirements:
1. While
2. Do While
3. For loop
Python provides three ways for executing the loops. While all the ways provide similar basic functionality, they differ in their syntax and condition-checking time.
F# code, code below If i = 1 the branch jumps to line 2, so the pr.pdfalphaagenciesindia
F# code, code below
If i = 1 the branch jumps to line 2, so the program prints 20, 21, 20, 21, 20, 21, . . . and does not
terminate.
If i = 2 the branch jumps to line 12, so the program prints 20, 21.
The file:
// Abstract syntax
type branchId = int
type expr =
| Num of int // integer
| Var of string // variable
| Op of string * expr * expr // e1 op e2
type stmt =
| Print of expr // print e;
| Assign of string * expr // x = e;
| Block of stmt list // block { stmt1; ...; stmtN }
| Loop of stmt list // loop { stmt1; ...; stmtN }
| Br of branchId // br i;
| BrIf of expr * branchId // if (e) br i;
// Two more examples
// block {
// x = 0
// loop {
// print x;
// x = x + 1;
// if (x < n) br 0;
// }
// }
let countTo n =
Block [
Assign ("x", Num 0)
Loop [
Print (Var "x")
Assign ("x", Op ("+", Var "x", Num 1))
BrIf (Op ("<", Var "x", Num n), 0)
]
]
// block {
// x = 10
// y = 0
// block {
// loop {
// if (x < 2) br 1;
// x = x - 2;
// y = y + 1;
// br 0;
// }
// }
// print y;
// }
let div2 =
Block [
Assign ("x", Num 10)
Assign ("y", Num 0)
Block [
Loop [
BrIf (Op ("<", Var "x", Num 2), 1)
Assign ("x", Op ("-", Var "x", Num 2))
Assign ("y", Op ("+", Var "y", Num 1))
Br 0
]
]
Print (Var "y")
]
// Prettyprinter
let paren b s = if b then "(" + s + ")" else s
let rec prettyExpr (prec : int) (e : expr) : string =
match e with
| Var x -> x
| Num i -> sprintf "%d" i
| Op (op, e1, e2) ->
let e1 = prettyExpr 1 e1
let e2 = prettyExpr 1 e2
sprintf "%s %s %s" e1 op e2 |> paren (0 < prec)
let rec prettyStmtI (indent : string) (stmt : stmt) : string =
let extraIndent = indent + " "
match stmt with
| Print e ->
let e = prettyExpr 1 e
sprintf "%sprint %s;" indent e
| Assign (x, e) ->
let e = prettyExpr 0 e
sprintf "%s%s = %s;" indent x e
| Block stmts ->
let stmts = prettyStmtsI extraIndent stmts
sprintf "%sblock {\n%s%s}" indent stmts indent
| Loop stmts ->
let stmts = prettyStmtsI extraIndent stmts
sprintf "%sloop {\n%s%s}" indent stmts indent
| Br i -> sprintf "%sbr %i;" indent i
| BrIf (e, i) ->
let e = prettyExpr 0 e
sprintf "%sif (%s) br %i;" indent e i
and prettyStmtsI (indent : string) (stmts : stmt list) : string =
List.map (fun stmt -> prettyStmtI indent stmt + "\n") stmts |> String.concat ""
let prettyStmt (stmt : stmt) : string =
prettyStmtI "" stmt
// Execution of statements
type naivestore = Map
let emptystore : Map = Map.empty
let getSto (store : naivestore) x = store.Item x
let setSto (store : naivestore) (k, v) = store.Add(k, v)
type result =
| Continue of naivestore
| Branch of branchId * naivestore
let rec eval e (store : naivestore) : int =
match e with
| Num i -> i
| Var x -> getSto store x
| Op (op, e1, e2) ->
let i1 = eval e1 store
let i2 = eval e2 store
match op with
| "*" -> i1 * i2
| "+" -> i1 + i2
| "-" -> i1 - i2
| "==" -> if i1 = i2 then 1 else 0
| "<" -> if i1 < i2 then 1 else 0
| _ -> failwith "unknown primitive"
let rec exec (stmt : stmt) (store : naivestore) : result =
match stmt with
| Print e ->
let v = eval e store
printfn "%d" v
C.
This presentation is part of a talk by Martin j. Logan on the essentials of programming in the erlang language. The talk covers:
Data Types
Modules and Functions
State Management
Distribution
Fault Tolerance
The code for this talk can be found at github:
git://github.com/martinjlogan/ErlangBootstrap.git
https://github.com/martinjlogan/ErlangBootstrap
These questions are prepared by Classical Programming Experts and are asked during job interviews.The Solution to the given programs are prepared by Programming Experts and are often asked in job interviews. Knowing solution to these problems will help you clear your concepts.
In this chapter we will examine the loop programming constructs through which we can execute a code snippet repeatedly. We will discuss how to implement conditional repetitions (while and do-while loops) and how to work with for-loops. We will give examples of different possibilities to define loops, how to construct them and some of their key usages. Finally, we will discuss the foreach-loop construct and how we can use multiple loops placed inside each other (nested loops).
Keynote at Dockercon Europe Amsterdam Dec 4th, 2014.
Speeding up development with Docker.
Summary of some interesting web scale microservice architectures.
Please send me updates and corrections to the architecture summaries @adrianco
Thanks Adrian
Monitorama - Please, no more Minutes, Milliseconds, Monoliths or Monitoring T...Adrian Cockcroft
Monitorama opening keynote talk on the challenges of Monitoring in a world where we need to deal with continuous delivery, cloud, and automated control feedback loops.
Hack Kid Con - Learn to be a Data Scientist for $1Adrian Cockcroft
Attempt to inspire some kids to pay attention in Math and Science classes so they can get a good job and help fill the skills gap in the years to come.
Slides originally written in April 2013 for a private conference and internal use at Netflix. Publishing now since Heartbleed is another example of an epidemic failure mode.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
A Comprehensive Look at Generative AI in Retail App Testing.pdfkalichargn70th171
Traditional software testing methods are being challenged in retail, where customer expectations and technological advancements continually shape the landscape. Enter generative AI—a transformative subset of artificial intelligence technologies poised to revolutionize software testing.
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
Software Engineering, Software Consulting, Tech Lead.
Spring Boot, Spring Cloud, Spring Core, Spring JDBC, Spring Security,
Spring Transaction, Spring MVC,
Log4j, REST/SOAP WEB-SERVICES.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Navigating the Metaverse: A Journey into Virtual Evolution"Donna Lenk
Join us for an exploration of the Metaverse's evolution, where innovation meets imagination. Discover new dimensions of virtual events, engage with thought-provoking discussions, and witness the transformative power of digital realms."
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I ...Juraj Vysvader
In 2015, I used to write extensions for Joomla, WordPress, phpBB3, etc and I didn't get rich from it but it did have 63K downloads (powered possible tens of thousands of websites).
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Enterprise Resource Planning System includes various modules that reduce any business's workload. Additionally, it organizes the workflows, which drives towards enhancing productivity. Here are a detailed explanation of the ERP modules. Going through the points will help you understand how the software is changing the work dynamics.
To know more details here: https://blogs.nyggs.com/nyggs/enterprise-resource-planning-erp-system-modules/
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Globus
The U.S. Geological Survey (USGS) has made substantial investments in meeting evolving scientific, technical, and policy driven demands on storing, managing, and delivering data. As these demands continue to grow in complexity and scale, the USGS must continue to explore innovative solutions to improve its management, curation, sharing, delivering, and preservation approaches for large-scale research data. Supporting these needs, the USGS has partnered with the University of Chicago-Globus to research and develop advanced repository components and workflows leveraging its current investment in Globus. The primary outcome of this partnership includes the development of a prototype enterprise repository, driven by USGS Data Release requirements, through exploration and implementation of the entire suite of the Globus platform offerings, including Globus Flow, Globus Auth, Globus Transfer, and Globus Search. This presentation will provide insights into this research partnership, introduce the unique requirements and challenges being addressed and provide relevant project progress.
2. What does @adrianco do?
@adrianco
Technology Due
Diligence on Deals
Presentations at
Companies and
Conferences
Tech and Board
Advisor
Support for
Portfolio
Companies
Consulting and
Training
Networking with
Interesting PeopleTinkering with
Technologies
Vendor
Relationships
Previously: Netflix, eBay, Sun Microsystems, Cambridge Consultants, City University London - BSc Applied Physics
3. What does @adrianco do?
@adrianco
Technology Due
Diligence on Deals
Presentations at
Companies and
Conferences
Tech and Board
Advisor
Support for
Portfolio
Companies
Consulting and
Training
Networking with
Interesting PeopleTinkering with
Technologies
Vendor
Relationships
Previously: Netflix, eBay, Sun Microsystems, Cambridge Consultants, City University London - BSc Applied Physics
4. Agenda
1978 Communicating Sequential Processes
1983 Occam
How Channels Work
1992 Pi-Calculus
2013 The Life of Occam-Pi
2006 Occam-Pi based simulation
Pi-Calculus ideas in Go
Go Applications
6. “…the concepts and notations
introduced in this paper (although
described in the next section in the
form of a programming language
fragment) should not be regarded
as suitable for use as a
programming language, either for
abstract or for concrete programming.
They are at best only a partial solution
to the problems tackled.”
8. [
SIEVE(i: 1..100):: // create 100 sieve processes
p,mp:integer;
SIEVE(i - 1)?p; // first thing read is prime
print!p; // send it to be printed
mp := p; // mp is a multiple of p
*[m:integer; SIEVE(i - 1)?m ! // for loop wait for next
*[m > mp ! mp := mp + p]; // for m > mp { mp += p }
[m = mp ! skip [] m < mp ! SIEVE(i + 1)!m] // if nonprime skip else send next
]
||SIEVE(0)::print!2; n:integer; n := 3; // start here with 2
*[n < 10000 ! SIEVE(1)!n; n := n + 2] // for n=3;n<10000;n+=2 {send n}
||SIEVE(101)::*[n:integer;SIEVE(100)?n ! print!n] // last prime in line
||print::*[(i:0..101) n:integer; SIEVE(i)?n ! …] // print all the things somehow…
]
I warned you there was a test!
Prime Number Sieve in CSP
9. [
SIEVE(i: 1..100):: // create 100 sieve processes
p,mp:integer;
SIEVE(i - 1)?p; // first thing read is prime
print!p; // send it to be printed
mp := p; // mp is a multiple of p
*[m:integer; SIEVE(i - 1)?m ! // for loop wait for next
*[m > mp ! mp := mp + p]; // for m > mp { mp += p }
[m = mp ! skip [] m < mp ! SIEVE(i + 1)!m] // if nonprime skip else send next
]
||SIEVE(0)::print!2; n:integer; n := 3; // start here with 2
*[n < 10000 ! SIEVE(1)!n; n := n + 2] // for n=3;n<10000;n+=2 {send n}
||SIEVE(101)::*[n:integer;SIEVE(100)?n ! print!n] // last prime in line
||print::*[(i:0..101) n:integer; SIEVE(i)?n ! …] // print all the things somehow…
]
I warned you there was a test!
SIEVE(0) SIEVE(1) SIEVE(100) SIEVE(101)
print
…
2
p=3
3
p=5
m m
p
n
Prime Number Sieve in CSP
10. [
SIEVE(i: 1..100):: // create 100 sieve processes
p,mp:integer;
SIEVE(i - 1)?p; // first thing read is prime
print!p; // send it to be printed
mp := p; // mp is a multiple of p
*[m:integer; SIEVE(i - 1)?m ! // for loop wait for next
*[m > mp ! mp := mp + p]; // for m > mp { mp += p }
[m = mp ! skip [] m < mp ! SIEVE(i + 1)!m] // if nonprime skip else send next
]
||SIEVE(0)::print!2; n:integer; n := 3; // start here with 2
*[n < 10000 ! SIEVE(1)!n; n := n + 2] // for n=3;n<10000;n+=2 {send n}
||SIEVE(101)::*[n:integer;SIEVE(100)?n ! print!n] // last prime in line
||print::*[(i:0..101) n:integer; SIEVE(i)?n ! …] // print all the things somehow…
]
Parallel ||
Repetitive *
read ?p
Alternate []
send !p
I warned you there was a test!
SIEVE(0) SIEVE(1) SIEVE(100) SIEVE(101)
print
…
2
p=3
3
p=5
m m
p
n
Prime Number Sieve in CSP
Sequential ;
12. Occam’s Razor
Non sunt multiplicanda entia sine necessitate
"Entities must not be multiplied beyond necessity”
William of Ockham (c. 1287–1347)
13. David May’s Occam Language
Extremely simple and elegant
implementation of CSP as a language
Adds named channels
Designed as the assembly language for Transputer hardware
15. Subset of Occam Syntax
PROC delay(CHAN stop, in, out)
VAR going, v, d:
SEQ
going := TRUE
WHILE going
ALT
in?v
d := NOW + 100
WAIT AFTER d
out!v
stop?ANY
going := FALSE
:
PROC pipeline(CHAN in, out)
CHAN c[101]:
PAR
VAR x:
SEQ
in?x
c[0]!x
PAR i=[1 FOR 100]
VAR y:
SEQ
c[i-1]?y
c[i]!y+1
VAR z:
SEQ
c[100]?z
out!z
:
Disclaimer: This version of the Occam language was written in Pascal on VAX/VMS. These code examples have only been checked by manual inspection
16. Subset of Occam SyntaxDefinitions
scoped to
block
following :
PROC delay(CHAN stop, in, out)
VAR going, v, d:
SEQ
going := TRUE
WHILE going
ALT
in?v
d := NOW + 100
WAIT AFTER d
out!v
stop?ANY
going := FALSE
:
Reserved
words in
upper case
Only has
VAR and
CHAN
types
Indent
based
structure
PROC pipeline(CHAN in, out)
CHAN c[101]:
PAR
VAR x:
SEQ
in?x
c[0]!x
PAR i=[1 FOR 100]
VAR y:
SEQ
c[i-1]?y
c[i]!y+1
VAR z:
SEQ
c[100]?z
out!z
:
ALTernate
blocks on
multiple input
or time guards
Only has
PROC hence
non-functional
programming!
Time is a
built in
concept
Disclaimer: This version of the Occam language was written in Pascal on VAX/VMS. These code examples have only been checked by manual inspection
17. Subset of Occam SyntaxDefinitions
scoped to
block
following :
PROC delay(CHAN stop, in, out)
VAR going, v, d:
SEQ
going := TRUE
WHILE going
ALT
in?v
d := NOW + 100
WAIT AFTER d
out!v
stop?ANY
going := FALSE
:
Reserved
words in
upper case
Only has
VAR and
CHAN
types
Indent
based
structure
PROC pipeline(CHAN in, out)
CHAN c[101]:
PAR
VAR x:
SEQ
in?x
c[0]!x
PAR i=[1 FOR 100]
VAR y:
SEQ
c[i-1]?y
c[i]!y+1
VAR z:
SEQ
c[100]?z
out!z
:
ALTernate
blocks on
multiple input
or time guards
Only has
PROC hence
non-functional
programming!
PARallel
runs every
line in
parallel
and waits
FOR
replicator
can be
used with
SEQ, PAR,
ALT, IF
Time is a
built in
concept
IF (not
shown) like
ALT with
test guards
Disclaimer: This version of the Occam language was written in Pascal on VAX/VMS. These code examples have only been checked by manual inspection
18. PROC eratosthenes(CHAN output)
CHAN sieve[101], print[102]:
PAR
SEQ // process 0
print[0]!2 // start primes with 2
SEQ n = [1 FOR 4999] // for n=3;n<10000;n+=2 {send n}
sieve[0]!(n+n)+1 // explicit evaluation order
sieve[0]!-1 // send -1 as terminator
PAR i = [1 FOR 100] // create 100 sieve processes
var p, mp, m:
SEQ
sieve[i-1]? p // first thing read is prime
print[i]! p // send it to be printed
mp := p // mp is a multiple of p
WHILE m <> -1 // look for terminator
SEQ
sieve[i-1]? m // read from previous
WHILE m > mp
mp := mp + p
IF
m = mp
SKIP
m < mp
sieve[i]! m // pass to next
var n: // process 101
SEQ
sieve[100]? n
print[101]! n // last prime in line
sieve[100]? ANY // discard terminator
var n: // printer process
SEQ i= [0 FOR 101]
SEQ
sieve[i]? n
output! n
:
Prime Number
Sieve Translated
from CSP to
Occam
Naming shifts from
processes to channels
19. PROC eratosthenes(CHAN output)
CHAN sieve[101], print[102]:
PAR
SEQ // process 0
print[0]!2 // start primes with 2
SEQ n = [1 FOR 4999] // for n=3;n<10000;n+=2 {send n}
sieve[0]!(n+n)+1 // explicit evaluation order
sieve[0]!-1 // send -1 as terminator
PAR i = [1 FOR 100] // create 100 sieve processes
var p, mp, m:
SEQ
sieve[i-1]? p // first thing read is prime
print[i]! p // send it to be printed
mp := p // mp is a multiple of p
WHILE m <> -1 // look for terminator
SEQ
sieve[i-1]? m // read from previous
WHILE m > mp
mp := mp + p
IF
m = mp
SKIP
m < mp
sieve[i]! m // pass to next
var n: // process 101
SEQ
sieve[100]? n
print[101]! n // last prime in line
sieve[100]? ANY // discard terminator
var n: // printer process
SEQ i= [0 FOR 101]
SEQ
sieve[i]? n
output! n
:
Prime Number
Sieve Translated
from CSP to
Occam
0 1 100 101
printer
…
print[0]
sieve[0] sieve[1] sieve[99] sieve[100]
print[101]
Naming shifts from
processes to channels
21. PROC main(chan out)
VAR x,y:
SEQ
x := 1
y := x
out!y
:
Comparing Occam and Go
Implementation of simple
assignment is shown first
as a baseline example
func main() {
var y int
x := 1
y = x
fmt.Println(y)
}
22. VAR x,y:
SEQ
x := 1
y := x
How does assignment work?
Implementation of simple
assignment
Program Counter
Stack Pointer
Address Value
x 1
y
&x
&y
SP
push &x
push &y
copy // *(SP+2) to *(SP+1)
Occam assumes a simple
stack machine
Stack Points to an empty location that can be used to
save the PC if this proc pauses
23. Comparing Occam and Go
Parallel Channel Assignment
func main() {
var x, y int
x = 1
c := make(chan int)
var wg sync.WaitGroup
wg.Add(2)
go func() { defer wg.Done(); c <- x }()
go func() { defer wg.Done(); y = <-c }()
wg.Wait()
fmt.Println(y)
}
PROC main(CHAN out)
VAR x,y:
SEQ
x := 1
CHAN c:
PAR
c!x
c?y
out!y
:
24. VAR x,y:
SEQ
x := 1
CHAN c:
PAR
c!x
c?y
How does a CHAN work?
Implementation of parallel
channel assignment
Program Counter X
Stack Pointer X
Address Value
x 1
y
c SPX
&x
SPX PCX
&y
SPY
push &x
c==nil so c=SPX, *SPX=PCX, sleep
Program Counter Y
Stack Pointer Y
push &y
c!=nil so read *c+1 // into &y
run *c, c=nil
First proc pushes &x then test-and-sets the
channel, which is empty, so it writes stack
pointer into channel, writes program
counter to stack and sleeps
Second proc pushes &y then test-and-sets
channel, which is full, so read other stack
pointer from channel, grab x from it and
store to y. Put other proc stack on run
queue and clear channel for next time.
Disclaimer: This is intended to be a useful mental model for how channels work, not an exact implementation. Transputers did this in hardware!
26. We present the π-calculus, a
calculus of communicating
systems in which one can
naturally express processes
which have changing
structure. Not only may the
component agents of a
system be arbitrarily linked,
but a communication
between neighbours may
carry information which
changes that linkage.
27. It’s easy to show that…
This paper is
incomprehensible!
A triumph of notation
over comprehension.
Simple equations such as…
28. The Life of Occam-Pi
Peter Welch 2013
(a wonderful and easy to read history of
parallel languages)
29. …looks at the history of
occam, its underlying
philosophy (Ockham’s
Razor), its semantic
foundation on Hoare’s
CSP, its principles of
process oriented design
and its development over
almost three decades into
occam-π (which blends in
the concurrency dynamics
of Milner’s π-calculus).
30. Simulation of Skype Peer-to-peer Web Services
Choreography Using Occam-Pi
Adrian Cockcroft 2006
https://github.com/adrianco/spigo/blob/master/SkypeSim07.pdf
31. Occam-Pi was used to create an actor-
like protocol simulator with supervisor
pattern and dynamic channel based
connections between nodes
37. Dynamic Channel Protocol
https://github.com/adrianco/spigo/tree/master/tooling/gotocol
Imposition/Intention? https://en.wikipedia.org/wiki/Promise_theory
ch <-gotocol.Message{gotocol.GetRequest, listener, now, ctx, "why?"}
// Message structure used for all messages, includes a channel of itself
type Message struct {
Imposition Impositions // request type
ResponseChan chan Message // place to send response messages
Sent time.Time // time at which message was sent
Ctx Context // message context
Intention string // payload
}
38. Dynamic
Channel
Protocol
“Impositions”
Elements of
DHCP + DNS +
HTTP + Routing +
Payment
// Impositions is the promise theory term for requests made to a service
type Impositions int
// Constant definitions for message types to be imposed on the receiver
const (
// Hello ChanToParent Name Initial noodly touch to set identity
Hello Impositions = iota
// NameDrop ChanToBuddy NameOfBuddy Here's someone to talk to
NameDrop
// Chat - ThisOften Chat to buddies time interval
Chat
// GoldCoin FromChan HowMuch
GoldCoin
// Inform loggerChan text message
Inform
// GetRequest FromChan key Simulate http inbound request
GetRequest
// GetResponse FromChan value Simulate http outbound response
GetResponse
// Put - "key value" Save the key and value
Put
// Replicate - "key value" Save a replicated copy
Replicate
// Forget - FromBuddy ToBuddy Forget link between two buddies
Forget
// Delete - key Remove key and value
Delete
// Goodbye - name - tell supervisor and exit
Goodbye // test assumes this is the last and exits
numOfImpositions
)
39. Spigo Actor Pattern
func Start(listener chan gotocol.Message) {
...
for {
select {
case msg := <-listener:
flow.Instrument(msg, name, hist)
switch msg.Imposition {
case gotocol.Hello: // get named by parent
...
case gotocol.NameDrop: // someone new to talk to
...
case gotocol.Put: // upstream request handler
...
outmsg := gotocol.Message{gotocol.Replicate, listener, time.Now(),
msg.Ctx.NewParent(), msg.Intention}
flow.AnnotateSend(outmsg, name)
outmsg.GoSend(replicas)
}
case <-eurekaTicker.C: // poll the service registry
...
}
}
}
Skeleton code for replicating a Put message
Instrument incoming requests
Instrument outgoing requests
update trace context
40. Partitioned/Scoped Service Registries
// Package eureka is a service registry for the architecture configuration (nodes) as it evolves
// and passes data to edda for logging nodes and edges
package eureka
func Start(listener chan gotocol.Message, name string) {
// lots of code removed
for {
msg = <-listener
collect.Measure(hist, time.Since(msg.Sent))
switch msg.Imposition {
// lots more cases removed
case gotocol.GetRequest:
for n, ch := range microservices {
// respond with all the online names that match the service component
if names.Service(n) == msg.Intention {
// if there was an update for the looked up service since last check
if metadata[n].registered.After(lastrequest[callback{n, msg.ResponseChan}]) {
if metadata[n].online {
gotocol.Message{gotocol.NameDrop, ch, time.Now(), gotocol.NilContext,
n}.GoSend(msg.ResponseChan)
} else {
gotocol.Message{gotocol.Forget, ch, time.Now(), gotocol.NilContext,
n}.GoSend(msg.ResponseChan)
}
}
// remember for next time
lastrequest[callback{n, msg.ResponseChan}] = msg.Sent
}
}
}
}
}
41. Partitioned/Scoped Service Registries
// Package eureka is a service registry for the architecture configuration (nodes) as it evolves
// and passes data to edda for logging nodes and edges
package eureka
func Start(listener chan gotocol.Message, name string) {
// lots of code removed
for {
msg = <-listener
collect.Measure(hist, time.Since(msg.Sent))
switch msg.Imposition {
// lots more cases removed
case gotocol.GetRequest:
for n, ch := range microservices {
// respond with all the online names that match the service component
if names.Service(n) == msg.Intention {
// if there was an update for the looked up service since last check
if metadata[n].registered.After(lastrequest[callback{n, msg.ResponseChan}]) {
if metadata[n].online {
gotocol.Message{gotocol.NameDrop, ch, time.Now(), gotocol.NilContext,
n}.GoSend(msg.ResponseChan)
} else {
gotocol.Message{gotocol.Forget, ch, time.Now(), gotocol.NilContext,
n}.GoSend(msg.ResponseChan)
}
}
// remember for next time
lastrequest[callback{n, msg.ResponseChan}] = msg.Sent
}
}
}
}
}
Service registry lookup handler
GetRequest “genericname”
Service registry lookup returns
NameDrop chan “instancename” or
Forget chan “instancename”
Only respond if
there’s a change
since this client
last asked
Registry is named at startup by supervisor
42.
43. What else could a golang actor-like
simulator be useful for?
44. What else could a golang actor-like
simulator be useful for?
(slides from my Microservice talks)
46. Some tools can show
the request flow
across a few services
47. Interesting
architectures have a
lot of microservices!
Flow visualization is
a big challenge.
See http://www.slideshare.net/LappleApple/gilt-from-monolith-ruby-app-to-micro-service-scala-service-architecture
48. Simulated Microservices
Model and visualize microservices
Simulate interesting architectures
Generate large scale configurations
Eventually stress test real tools
Code: github.com/adrianco/spigo
Simulate Protocol Interactions in Go
Visualize with D3
See for yourself: http://simianviz.surge.sh
Follow @simianviz for updates
ELB Load Balancer
Zuul
API Proxy
Karyon
Business Logic
Staash
Data Access Layer
Priam
Cassandra Datastore
Three
Availability
Zones
Denominator
DNS Endpoint
55. Two Region Riak IoT
IoT Ingestion Endpoint
Stream Endpoint
Analytics Endpoint
East Region Ingestion
West Region Ingestion
Multi Region TS Analytics
See for yourself: http://simianviz.surge.sh/riak
59. Conclusions
CSP is too limited
π-Calculus syntax is incomprehensible
Occam-Pi makes CSP and π-Calculus readable
Go concurrency syntax is clumsy in places but works
Showed some useful channel based Go-π idioms
Pass channels over channels for dynamic routing
Go works well for actor like simulation