SlideShare a Scribd company logo
Concurrency with Go
Natural Design in a Technical Language
Frank Müller
•Oldenburg, Germany


•Born 1965


•Working at Kubermatic


•Team Lead Development


•@themue
Introduction Frank Müller
Our world is concurrent
• Individuals populate this world


• They act sometimes independent of each other, sometimes
dependent, sometimes together


• Communication and signals enable their coexistence
It's a natural principle Frank Müller
World of plants
World of animals
People at work
People doing sports
People at conferences
People of everyday life
It's our daily life
Modern systems are more powerful
• Computer architectures are changing


• Growth via CPUs, cores and hyper-threads


• Manual usage via threads is complex and error-prone


• Concurrent runtime environments enable fine-grained usage
One motivation is the hardware Frank Müller
Distribution of computing power Frank Müller
Process Process Process Process
Process Process Process Process
Runtime Envirunment
Core Core Core Core
Process
Process
Process
Process
Processes can work alone ...
... or as a team together
• Encapsulation of the state in the process


• Communication via messages


• Sequential processing


• Atomic state changes


• OOP in the real sense
Structure is more important motivation Frank Müller
❞ –Rob Pike
Parallelis
m

Programming as the simultaneous execution
of (possibly related) computations
.

Concurrenc
y

Programming as the composition of
independently executing processes.
• Actor Model


‣ 1973


‣ Carl Hewitt, Peter Bishop und Richard Steiger


• Communicating Sequential Processes


‣ 1978


‣ Tony Hoare
Ideas long known Frank Müller
• Concurrent processes communicate via one or more channels


• Processes, unlike channels, are anonymous


• Data is not sent until the receiver is ready to receive it


• Incoming data is processed sequentially
Communicating Sequential Processes Frank Müller
Communicating Sequential Processes Frank Müller
Process
Process
Process
Process Process
Process Process
Process
Process
Process
Technologies need their time
Examples of use
Processes provide services Frank Müller
Service
Provider
Client
Client Client
Client
Active
Waiting
Waiting
Waiting
Processes manage ressources Frank Müller
Client
Client
Client
Manager Resource B
Resource A
Resource C
Active
Waiting
Waiting
Read / Write
Processes manage parallel requests Frank Müller
Worker
Worker
Worker
Client
Client
Client
Master
Request A
Request B
Request C
Request A
Request B
Request C
Reply
Processes handle events Frank Müller
Event
Manager
Emitter
Emitter
Emitter
Subscriber
Subscriber
Subscriber
Events
Events
Events
Events
Processes monitor each other Frank Müller
Supervisor
Process A Process B
Process B'
Starts A Starts B
Monitors A Monitors B
Monitors B'
Restarts B'
Processes support powerful ETL Frank Müller
Sender
Sender
Sender
Transformator(s) Loader
Extractor Receiver
Receiver
Receiver
Transformator(s)
Transformator(s)
Transformed
Data
Transformed
Data
Transformed
Data
Raw
Data
Raw
Data
Raw
Data
Examples in Go
• Development started 2007 by Google


• Designed by Rob Pike, Ken Thompson, and Robert Griesemer


• Initial release 2012


• Looks imperative, but is multi-paradigm


• Types with methods, interfaces, function types, concurrency


• Concurrency realized by goroutines and channels
Google Go Frank Müller
• Goroutines run lightweight in a thread pool


• Functions spawned with the keyword go


• Large simultaneous number possible


• Channels are typed and run synchronous or buffered


• They are used for communication and synchronization


• select statements allow parallel processing of multiple channels


• Use in for loops enables continuous processing
Concurrency in Go Frank Müller
Example "Background Job"
// processMyData processes the data in the background.


func processMyData(data Data) { ... }


// startProcessing pre-processes the data and starts the goroutine.


func startProcessing() {


var data Data


data = ...


// Spawn goroutine.


go processMyData(data)


// Do something else.


...


}
Simple in own function Frank Müller
// startProcessing pre-processes the data and starts the goroutine to process


// it in the background.


func startProcessing() {


var data Data


data = ...


// Spawn goroutine, function uses outer data.


go func() {


...


}()


// Do something else.


...


}
Simple in embedded function Frank Müller
// startProcessing pre-processes the data and starts the goroutine to process


// a copy in the background.


func startProcessing() {


var data Data


data = ...


// Spawn goroutine with a data copy, same name is no problem.


go func(data Data) {


...


}(data)


// Do something else using the original data.


...


}
Embedded function with data copy Frank Müller
// process pre-processes the data, starts the goroutine and waits until it's


// done.


func process() {


data := ...


var wg sync.WaitGroup // sync.WaitGroup allows counting of activities.


wg.Add(1) // In example here we wait for only one processing.


go processData(&wg, data)


...


wg.Wait() // Wait until it's done.


}
Waiting for processing of data Frank Müller
// processData processes the passed data and signals its ending via


// the sync.WaitGroup.


func processData(wg *sync.WaitGroup, data Data) {


// Deferred function call tells wait group that one processing is done.


defer wg.Done()


// Process data data.


...


}
Tell waiter that work is done Frank Müller
// processDatas starts the goroutines to process the individual datas and waits


// until all are done.


func processDatas(datas []Data) {


var wg sync.WaitGroup


for _, data := range datas {


// Add one per each data to process.


wg.Add(1)


// Spawn processing like in last example.


go processData(&wg, data)


}


wg.Wait() // Wait until they are done.


}
Start a number of background jobs Frank Müller
Example "Streaming"
// processDatas processes all datas it receives via the data channel.


// Loop ends when the channel is closed.


func processDatas(dataChan <-chan Data) {


for data := range dataChan {


// Process the individual data sequentially.


...


}


}


Process all datas received from channel Frank Müller
// Create data channel.


dataChan := make(chan Data)


// Spawn processor with data channel.


go processDatas(dataChan)


// Send datas.


dataChan <- dataA


dataChan <- dataB


dataChan <- dataC


// Close channel.


close(dataChan)
Use the data processor Frank Müller
// processAtoB processes all A datas it receives via the in channel. Results of


// type B are written to the out channel. That will be closed if the function


// ends working a.k.a. the in channel has been closed.


func processAtoB(inChan <-chan A, outChan chan<- B) {


defer close(outChan)


for a := range inChan {


b := ...


outChan <- b


}


}


// processBtoC works similar to processAtoB, only with B and C datas.


func processBtoC(inChan <-chan B, outChan chan<- C) {


...


}
Piping Frank Müller
// Create buffered channels.


aChan := make(chan A, 5)


bChan := make(chan B, 5)


cChan := make(chan C, 5)


// Spawn processors.


go processAtoB(aChan, bChan)


go processBtoC(bChan, cChan)


go processCs(cChan)


// Write A data into A channel, then close.


...


close(aChan)
Use the piping Frank Müller
Example "Service"
• Structure with data and channels


• Function New() as constructor


• Method loop() or backend() for loop with select statement


• Public methods to access the instance


• Requests with return values need explicit channel


• Multiple return values need helper types
Often found pattern Frank Müller
// MyService simply provides a string and an integer field and allows to add


// them if possible.


type MyService struct {


a string


b int


// Many channels needed this way.


setAChan chan string


getAChan chan chan string


setBChan chan int


getBChan chan chan int


addChan chan addResp


}
Structure Frank Müller
// New create a new instance of MyService. The backend goroutine is controlled


// by the given context.


func New(ctx context.Context) *MyService {


ms := &MyService{


setAChan: make(chan string),


getAChan: make(chan chan string),


setBChan: make(chan int),


getBChan: make(chan chan int),


addChan: make(chan addReq),


}


go ms.backend(ctx)


return ms


}
Constructor Frank Müller
// SetA simply sends the data via the channel.


func (ms *MyService) SetA(a string) {


ms.setAChan <- a


}


// GetA sends a buffered channel and receives the result via it.


func (ms *MyService) GetA() string {


// Buffered to allow backend continue working.


respChan := make(chan string, 1)


ms.getAChan <- respChan


return <-respChan


}
Setter and getter Frank Müller
// addResp is a private transporter for the sum and a possible error.


type addResp struct {


sum int


err error


}


// Add sends a buffered channel for a transporter.


func (ms *MyService) Add() (int, error) {


respChan := make(chan addResp, 1)


ms.addChan <- retChan


resp := <-retChan


if resp.err != nil { return 0, resp.err }


return resp.sum, nil


}
Sometimes even more effort needed Frank Müller
// backend runs as goroutine and serializes the operations.


func (ms *MyService) backend(ctx context.Context) {


// Endless loop with for.


for {


// Select to switch between the channels.


select {


case <-ctx.Done():


// Terminate backend.


return


case ...:


...


}


}


}
Backend structure Frank Müller
select {


...


case respChan := <-ms.getBChan:


respChan <- ms.b


case respChan := <-ms.addChan:


var resp addResp


i, err := strconv.Atoi(ms.a)


if err != nil {


resp.err = err


} else {


resp.sum = i + ms.b


}


respChan <- resp


}
Other cases Frank Müller
• Much extra effort with typed channels and helper types


• Example here does not care if context is cancelled


• Public methods and business logic are separated


• Without private helpers the select statement may grow too much


• But thankfully there's a more simple way to do it
Summary Frank Müller
Example "Actor"
// Actor only needs two extra fields.


type Actor struct {


ctx context.Context


actChan chan func()


// Here own data.


...


}


func New(ctx context.Context) *Actor {


act := &Actor{ ... }


go backend()


return act


}
Structure and constructor Frank Müller
// backend is only executing the received functions.


func (act *Actor) backend() {


for {


select {


case <-act.ctx.Done():


return


case action := <-act.actChan:


// Execute received function.


action()


}


}


}
Backend Frank Müller
// do always keeps an eye on the context when sending.


func (act *Actor) do(action func()) error {


select {


case <-act.ctx.Done():


if ctx.Err() != nil {


return ctx.Err()


}


return errors.New("actor has been stopped")


case act.actChan <- action:


return nil


}


}
Safe sending of a function Frank Müller
// Add works like the Add() from example before.


func (act *Actor) Add() (i int, err error) {


// Send logic to the backend.


if aerr := act.do(func() {


fi, ferr := strconv.Atoi(act.a)


if ferr != nil {


err = ferr


return


}


i = fi + act.b


}); aerr != nil {


// Looks like the actor already has been stopped.


return 0, aerr


}


}
Business logic Frank Müller
• Less code


• Actor can be implemented in one package and always reused


• Additional buffered channel and doAsync() support pure setters
and callers without return values


• do() and doAsync() also could get optional timeout for fault
tolerant behavior
Summary Frank Müller
Example "Supervisor"
// Supervisor helps monitoring and restarting concurrent functions.


type Supervisor struct {


mu sync.Mutex


workers map[string]func() error


spawnChan chan string


}


// New starts the supervisor in the background.


func New(ctx context.Context) *Supervisor {


s := &Structure{ ... }


go s.backend(ctx)


return s


}
Structure and constructor Frank Müller
// Spawn tells backend to spawn the given worker.


func (s *Supervisor) spawn(id string, worker func() error) error {


s.mu.Lock()


defer s.mu.Unlock()


_, ok := s.workers[id]


if ok {


return errors.New("double worker ID")


}


s.workers[id] = worker


s.spawnChan <- id


return nil


}
Start a worker Frank Müller
// wrap takes care for errors of the worker. In case of an error it notifies


// the backend to re-spawn.


func (s *Supervisor) wrap(id string) {


worker := s.workers[id]


if err := worker(); err != nil {


// Log the error and re-spawn the worker.


log.Printf("worker %q terminated with error: %v", id, err)


s.spawnChan <- id


return


}


// Delete successful terminated worker.


s.mu.Lock()


delete(s.workers, id)


s.mu.Unlock()


}
Wrapper to check if worker error Frank Müller
// backend wraps workers and spawns them.


func (s *Supervisor) backend(ctx context.Context) {


for {


select {


case <-ctx.Done():


return


case id := <-s.spawnChan:


go s.wrap(id)


}


}


}
Backend Frank Müller
Pitfalls
• Channels work synchron or buffered


• Working reader could lead to blocking writes, buffers may be
filled


• Most times just waiting, but overload resource may lead to
instability


• If needed check for parallelized pre-processing steps, e.g. by
caller, and keep the need for serialization small


• Alternatively assistent workers for the backend may help
Blocked channels Frank Müller
• Avoid overlapping read and write access with external
modification


• E.g. use IncrBy() instead of Get(), local addition, and Set()


• Alternatively read value together with timed handle for update


• Other modifiers get an error during this time


• An unused handle must be released again
Race conditions Frank Müller
• Concurrent updates of coherent data may lead to invalid states


• Avoid too fine granular access


• Assure changes of all related data en bloc
Non-atomic updates Frank Müller
Final summary
• Power of concurrency seems to be complex


• Possibilities are manifold


• As is often the case, only a few patterns make up almost all use
cases


• Own or 3rd party packages reduce work here


• Design of elastic software requires a more natural rethink
Final summary Frank Müller
Thanks a lot and


have a nice


evening


Image Sources


123RF


Pexels


iStockphoto


Own photos

More Related Content

What's hot

Go on!
Go on!Go on!
Go on!
Vadim Petrov
 
Coding in GO - GDG SL - NSBM
Coding in GO - GDG SL - NSBMCoding in GO - GDG SL - NSBM
Coding in GO - GDG SL - NSBM
Raveen Perera
 
Mpi in-python
Mpi in-pythonMpi in-python
Mpi in-python
A Jorge Garcia
 
Go Containers
Go ContainersGo Containers
Go Containers
jgrahamc
 
Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;
Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;
Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;
Tzung-Bi Shih
 
Protocol handler in Gecko
Protocol handler in GeckoProtocol handler in Gecko
Protocol handler in Gecko
Chih-Hsuan Kuo
 
Go Concurrency Patterns
Go Concurrency PatternsGo Concurrency Patterns
Go Concurrency Patterns
ElifTech
 
.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund
.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund
.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund
Karel Zikmund
 
Concurrency in Go by Denys Goldiner.pdf
Concurrency in Go by Denys Goldiner.pdfConcurrency in Go by Denys Goldiner.pdf
Concurrency in Go by Denys Goldiner.pdf
Denys Goldiner
 
Rust
RustRust
Fundamental concurrent programming
Fundamental concurrent programmingFundamental concurrent programming
Fundamental concurrent programming
Dimas Prawira
 
Concurrency in Python4k
Concurrency in Python4kConcurrency in Python4k
Concurrency in Python4k
Rodolfo Carvalho
 
Deep Dive async/await in Unity with UniTask(EN)
Deep Dive async/await in Unity with UniTask(EN)Deep Dive async/await in Unity with UniTask(EN)
Deep Dive async/await in Unity with UniTask(EN)
Yoshifumi Kawai
 
Ownership System in Rust
Ownership System in RustOwnership System in Rust
Ownership System in Rust
Chih-Hsuan Kuo
 
Python Async IO Horizon
Python Async IO HorizonPython Async IO Horizon
Python Async IO Horizon
Lukasz Dobrzanski
 
Storm
StormStorm
3 rd animation
3 rd animation3 rd animation
3 rd animation
divyalakshmi77
 
Scapy
ScapyScapy
The async/await concurrency pattern in Golang
The async/await concurrency pattern in GolangThe async/await concurrency pattern in Golang
The async/await concurrency pattern in Golang
Matteo Madeddu
 
Router Queue Simulation in C++ in MMNN and MM1 conditions
Router Queue Simulation in C++ in MMNN and MM1 conditionsRouter Queue Simulation in C++ in MMNN and MM1 conditions
Router Queue Simulation in C++ in MMNN and MM1 conditions
Morteza Mahdilar
 

What's hot (20)

Go on!
Go on!Go on!
Go on!
 
Coding in GO - GDG SL - NSBM
Coding in GO - GDG SL - NSBMCoding in GO - GDG SL - NSBM
Coding in GO - GDG SL - NSBM
 
Mpi in-python
Mpi in-pythonMpi in-python
Mpi in-python
 
Go Containers
Go ContainersGo Containers
Go Containers
 
Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;
Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;
Global Interpreter Lock: Episode III - cat &lt; /dev/zero > GIL;
 
Protocol handler in Gecko
Protocol handler in GeckoProtocol handler in Gecko
Protocol handler in Gecko
 
Go Concurrency Patterns
Go Concurrency PatternsGo Concurrency Patterns
Go Concurrency Patterns
 
.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund
.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund
.NET Core Summer event 2019 in Brno, CZ - Async demystified -- Karel Zikmund
 
Concurrency in Go by Denys Goldiner.pdf
Concurrency in Go by Denys Goldiner.pdfConcurrency in Go by Denys Goldiner.pdf
Concurrency in Go by Denys Goldiner.pdf
 
Rust
RustRust
Rust
 
Fundamental concurrent programming
Fundamental concurrent programmingFundamental concurrent programming
Fundamental concurrent programming
 
Concurrency in Python4k
Concurrency in Python4kConcurrency in Python4k
Concurrency in Python4k
 
Deep Dive async/await in Unity with UniTask(EN)
Deep Dive async/await in Unity with UniTask(EN)Deep Dive async/await in Unity with UniTask(EN)
Deep Dive async/await in Unity with UniTask(EN)
 
Ownership System in Rust
Ownership System in RustOwnership System in Rust
Ownership System in Rust
 
Python Async IO Horizon
Python Async IO HorizonPython Async IO Horizon
Python Async IO Horizon
 
Storm
StormStorm
Storm
 
3 rd animation
3 rd animation3 rd animation
3 rd animation
 
Scapy
ScapyScapy
Scapy
 
The async/await concurrency pattern in Golang
The async/await concurrency pattern in GolangThe async/await concurrency pattern in Golang
The async/await concurrency pattern in Golang
 
Router Queue Simulation in C++ in MMNN and MM1 conditions
Router Queue Simulation in C++ in MMNN and MM1 conditionsRouter Queue Simulation in C++ in MMNN and MM1 conditions
Router Queue Simulation in C++ in MMNN and MM1 conditions
 

Similar to Concurrency with Go

Windows Phone 8 - 3.5 Async Programming
Windows Phone 8 - 3.5 Async ProgrammingWindows Phone 8 - 3.5 Async Programming
Windows Phone 8 - 3.5 Async Programming
Oliver Scheer
 
Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...
Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...
Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...
gdgsurrey
 
Apache Flink Deep Dive
Apache Flink Deep DiveApache Flink Deep Dive
Apache Flink Deep Dive
Vasia Kalavri
 
Go and Uber’s time series database m3
Go and Uber’s time series database m3Go and Uber’s time series database m3
Go and Uber’s time series database m3
Rob Skillington
 
Flow based programming in golang
Flow based programming in golangFlow based programming in golang
Flow based programming in golang
Anton Stepanenko
 
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 KeynoteAdvanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
StreamNative
 
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving SystemsPRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
NECST Lab @ Politecnico di Milano
 
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream AnalysisLWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
Jonas Traub
 
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
Flink Forward
 
running Tensorflow in Production
running Tensorflow in Productionrunning Tensorflow in Production
running Tensorflow in Production
Matthias Feys
 
Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, NutanixGuaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
HostedbyConfluent
 
J unit
J unitJ unit
[FFE19] Build a Flink AI Ecosystem
[FFE19] Build a Flink AI Ecosystem[FFE19] Build a Flink AI Ecosystem
[FFE19] Build a Flink AI Ecosystem
Jiangjie Qin
 
GopherCon 2017 - Writing Networking Clients in Go: The Design & Implementati...
GopherCon 2017 -  Writing Networking Clients in Go: The Design & Implementati...GopherCon 2017 -  Writing Networking Clients in Go: The Design & Implementati...
GopherCon 2017 - Writing Networking Clients in Go: The Design & Implementati...
wallyqs
 
Writing Networking Clients in Go - GopherCon 2017 talk
Writing Networking Clients in Go - GopherCon 2017 talkWriting Networking Clients in Go - GopherCon 2017 talk
Writing Networking Clients in Go - GopherCon 2017 talk
NATS
 
Programming using MPI and OpenMP
Programming using MPI and OpenMPProgramming using MPI and OpenMP
Programming using MPI and OpenMP
Divya Tiwari
 
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Data Con LA
 
Open Source XMPP for Cloud Services
Open Source XMPP for Cloud ServicesOpen Source XMPP for Cloud Services
Open Source XMPP for Cloud Services
mattjive
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data Analysis
J Singh
 
Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016
Yuta Iwama
 

Similar to Concurrency with Go (20)

Windows Phone 8 - 3.5 Async Programming
Windows Phone 8 - 3.5 Async ProgrammingWindows Phone 8 - 3.5 Async Programming
Windows Phone 8 - 3.5 Async Programming
 
Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...
Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...
Certification Study Group -Professional ML Engineer Session 2 (GCP-TensorFlow...
 
Apache Flink Deep Dive
Apache Flink Deep DiveApache Flink Deep Dive
Apache Flink Deep Dive
 
Go and Uber’s time series database m3
Go and Uber’s time series database m3Go and Uber’s time series database m3
Go and Uber’s time series database m3
 
Flow based programming in golang
Flow based programming in golangFlow based programming in golang
Flow based programming in golang
 
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 KeynoteAdvanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
Advanced Stream Processing with Flink and Pulsar - Pulsar Summit NA 2021 Keynote
 
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving SystemsPRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
PRETZEL: Opening the Black Box of Machine Learning Prediction Serving Systems
 
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream AnalysisLWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
LWA 2015: The Apache Flink Platform for Parallel Batch and Stream Analysis
 
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
William Vambenepe – Google Cloud Dataflow and Flink , Stream Processing by De...
 
running Tensorflow in Production
running Tensorflow in Productionrunning Tensorflow in Production
running Tensorflow in Production
 
Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, NutanixGuaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
Guaranteed Event Delivery with Kafka and NodeJS | Amitesh Madhur, Nutanix
 
J unit
J unitJ unit
J unit
 
[FFE19] Build a Flink AI Ecosystem
[FFE19] Build a Flink AI Ecosystem[FFE19] Build a Flink AI Ecosystem
[FFE19] Build a Flink AI Ecosystem
 
GopherCon 2017 - Writing Networking Clients in Go: The Design & Implementati...
GopherCon 2017 -  Writing Networking Clients in Go: The Design & Implementati...GopherCon 2017 -  Writing Networking Clients in Go: The Design & Implementati...
GopherCon 2017 - Writing Networking Clients in Go: The Design & Implementati...
 
Writing Networking Clients in Go - GopherCon 2017 talk
Writing Networking Clients in Go - GopherCon 2017 talkWriting Networking Clients in Go - GopherCon 2017 talk
Writing Networking Clients in Go - GopherCon 2017 talk
 
Programming using MPI and OpenMP
Programming using MPI and OpenMPProgramming using MPI and OpenMP
Programming using MPI and OpenMP
 
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
Big Data Day LA 2016/ Big Data Track - Portable Stream and Batch Processing w...
 
Open Source XMPP for Cloud Services
Open Source XMPP for Cloud ServicesOpen Source XMPP for Cloud Services
Open Source XMPP for Cloud Services
 
High Throughput Data Analysis
High Throughput Data AnalysisHigh Throughput Data Analysis
High Throughput Data Analysis
 
Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016
 

More from Frank Müller

JAX 2023 - Cloud Provider APIs
JAX 2023 - Cloud Provider APIsJAX 2023 - Cloud Provider APIs
JAX 2023 - Cloud Provider APIs
Frank Müller
 
JAX 2023 - Generics in Go
JAX 2023 - Generics in GoJAX 2023 - Generics in Go
JAX 2023 - Generics in Go
Frank Müller
 
Let The Computer Do It
Let The Computer Do ItLet The Computer Do It
Let The Computer Do It
Frank Müller
 
2021 OOP - Kubernetes Operatoren
2021   OOP - Kubernetes Operatoren2021   OOP - Kubernetes Operatoren
2021 OOP - Kubernetes Operatoren
Frank Müller
 
DevOpsCon - Verteilte Entwicklung in Go
DevOpsCon - Verteilte Entwicklung in GoDevOpsCon - Verteilte Entwicklung in Go
DevOpsCon - Verteilte Entwicklung in Go
Frank Müller
 
Devs@Home - Einführung in Go
Devs@Home - Einführung in GoDevs@Home - Einführung in Go
Devs@Home - Einführung in Go
Frank Müller
 
Fun with functions
Fun with functionsFun with functions
Fun with functions
Frank Müller
 
Ein Gopher im Netz
Ein Gopher im NetzEin Gopher im Netz
Ein Gopher im Netz
Frank Müller
 
Blockchains - Mehr als nur digitale Währungen
Blockchains - Mehr als nur digitale WährungenBlockchains - Mehr als nur digitale Währungen
Blockchains - Mehr als nur digitale Währungen
Frank Müller
 
Spaß an der Nebenläufigkeit
Spaß an der NebenläufigkeitSpaß an der Nebenläufigkeit
Spaß an der Nebenläufigkeit
Frank Müller
 
Go - Googles Sprache für skalierbare Systeme
Go - Googles Sprache für skalierbare SystemeGo - Googles Sprache für skalierbare Systeme
Go - Googles Sprache für skalierbare Systeme
Frank Müller
 
Cloud Provisioning mit Juju
Cloud Provisioning mit JujuCloud Provisioning mit Juju
Cloud Provisioning mit Juju
Frank Müller
 
Juju - Scalable Software with Google Go
Juju - Scalable Software with Google GoJuju - Scalable Software with Google Go
Juju - Scalable Software with Google Go
Frank Müller
 
RESTful Web Applications with Google Go
RESTful Web Applications with Google GoRESTful Web Applications with Google Go
RESTful Web Applications with Google Go
Frank Müller
 
Clouds, leicht beherrschbar
Clouds, leicht beherrschbarClouds, leicht beherrschbar
Clouds, leicht beherrschbar
Frank Müller
 
Skalierbare Anwendungen mit Google Go
Skalierbare Anwendungen mit Google GoSkalierbare Anwendungen mit Google Go
Skalierbare Anwendungen mit Google Go
Frank Müller
 
WTC 2013 - Juju - Mit etwas Magie zur perfekten Cloud
WTC 2013 - Juju - Mit etwas Magie zur perfekten CloudWTC 2013 - Juju - Mit etwas Magie zur perfekten Cloud
WTC 2013 - Juju - Mit etwas Magie zur perfekten Cloud
Frank Müller
 
Juju - Google Go in a scalable Environment
Juju - Google Go in a scalable EnvironmentJuju - Google Go in a scalable Environment
Juju - Google Go in a scalable Environment
Frank Müller
 
OOP 2013 - Weltweite Entwicklung von Open Source Software
OOP 2013 - Weltweite Entwicklung von Open Source SoftwareOOP 2013 - Weltweite Entwicklung von Open Source Software
OOP 2013 - Weltweite Entwicklung von Open Source Software
Frank Müller
 
Beauty and Power of Go
Beauty and Power of GoBeauty and Power of Go
Beauty and Power of Go
Frank Müller
 

More from Frank Müller (20)

JAX 2023 - Cloud Provider APIs
JAX 2023 - Cloud Provider APIsJAX 2023 - Cloud Provider APIs
JAX 2023 - Cloud Provider APIs
 
JAX 2023 - Generics in Go
JAX 2023 - Generics in GoJAX 2023 - Generics in Go
JAX 2023 - Generics in Go
 
Let The Computer Do It
Let The Computer Do ItLet The Computer Do It
Let The Computer Do It
 
2021 OOP - Kubernetes Operatoren
2021   OOP - Kubernetes Operatoren2021   OOP - Kubernetes Operatoren
2021 OOP - Kubernetes Operatoren
 
DevOpsCon - Verteilte Entwicklung in Go
DevOpsCon - Verteilte Entwicklung in GoDevOpsCon - Verteilte Entwicklung in Go
DevOpsCon - Verteilte Entwicklung in Go
 
Devs@Home - Einführung in Go
Devs@Home - Einführung in GoDevs@Home - Einführung in Go
Devs@Home - Einführung in Go
 
Fun with functions
Fun with functionsFun with functions
Fun with functions
 
Ein Gopher im Netz
Ein Gopher im NetzEin Gopher im Netz
Ein Gopher im Netz
 
Blockchains - Mehr als nur digitale Währungen
Blockchains - Mehr als nur digitale WährungenBlockchains - Mehr als nur digitale Währungen
Blockchains - Mehr als nur digitale Währungen
 
Spaß an der Nebenläufigkeit
Spaß an der NebenläufigkeitSpaß an der Nebenläufigkeit
Spaß an der Nebenläufigkeit
 
Go - Googles Sprache für skalierbare Systeme
Go - Googles Sprache für skalierbare SystemeGo - Googles Sprache für skalierbare Systeme
Go - Googles Sprache für skalierbare Systeme
 
Cloud Provisioning mit Juju
Cloud Provisioning mit JujuCloud Provisioning mit Juju
Cloud Provisioning mit Juju
 
Juju - Scalable Software with Google Go
Juju - Scalable Software with Google GoJuju - Scalable Software with Google Go
Juju - Scalable Software with Google Go
 
RESTful Web Applications with Google Go
RESTful Web Applications with Google GoRESTful Web Applications with Google Go
RESTful Web Applications with Google Go
 
Clouds, leicht beherrschbar
Clouds, leicht beherrschbarClouds, leicht beherrschbar
Clouds, leicht beherrschbar
 
Skalierbare Anwendungen mit Google Go
Skalierbare Anwendungen mit Google GoSkalierbare Anwendungen mit Google Go
Skalierbare Anwendungen mit Google Go
 
WTC 2013 - Juju - Mit etwas Magie zur perfekten Cloud
WTC 2013 - Juju - Mit etwas Magie zur perfekten CloudWTC 2013 - Juju - Mit etwas Magie zur perfekten Cloud
WTC 2013 - Juju - Mit etwas Magie zur perfekten Cloud
 
Juju - Google Go in a scalable Environment
Juju - Google Go in a scalable EnvironmentJuju - Google Go in a scalable Environment
Juju - Google Go in a scalable Environment
 
OOP 2013 - Weltweite Entwicklung von Open Source Software
OOP 2013 - Weltweite Entwicklung von Open Source SoftwareOOP 2013 - Weltweite Entwicklung von Open Source Software
OOP 2013 - Weltweite Entwicklung von Open Source Software
 
Beauty and Power of Go
Beauty and Power of GoBeauty and Power of Go
Beauty and Power of Go
 

Recently uploaded

Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
bhumivarma35300
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
jhkdigitalmarketing
 
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
ThousandEyes
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
Jersey (CHE-PING) Su
 
Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.
shivamt017
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
Nextskill Technologies
 
Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …
908dutch
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
AsifKarimJim
 
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
bahubalikumar09988
 
11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf
kalichargn70th171
 
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to KnowThe Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
onemonitarsoftware
 
To Avoid Mistakes When Using Online Attendance Sheets
To Avoid Mistakes When Using Online Attendance SheetsTo Avoid Mistakes When Using Online Attendance Sheets
To Avoid Mistakes When Using Online Attendance Sheets
Task Tracker
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
akshesh doshi
 
Top Chinese Government-backed APT Groups
Top Chinese Government-backed APT GroupsTop Chinese Government-backed APT Groups
Top Chinese Government-backed APT Groups
SOCRadar
 
Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...
Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...
Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...
norina2645
 
Il Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazioneIl Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
jealousviolet
 
Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech
 
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction InnovationNYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS Construction ERP Software
 

Recently uploaded (20)

Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
Independent Girls call Service Pune 000XX00000 Provide Best And Top Girl Serv...
 
Odoo E-commerce website development guides
Odoo E-commerce website development guidesOdoo E-commerce website development guides
Odoo E-commerce website development guides
 
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
Cisco Live Announcements: New ThousandEyes Release Highlights - July 2024
 
當測試開始左移
當測試開始左移當測試開始左移
當測試開始左移
 
Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.Shivam Pandit working on Php Web Developer.
Shivam Pandit working on Php Web Developer.
 
ERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in CoimbatoreERP Software Solutions Provider in Coimbatore
ERP Software Solutions Provider in Coimbatore
 
Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …Prada Group Reports Strong Growth in First Quarter …
Prada Group Reports Strong Growth in First Quarter …
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
 
welcome to presentation on Google Apps
welcome to   presentation on Google Appswelcome to   presentation on Google Apps
welcome to presentation on Google Apps
 
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Ranchi  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Ranchi 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
 
11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf11 Top Cross Browser Testing Tools to Know About.pdf
11 Top Cross Browser Testing Tools to Know About.pdf
 
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to KnowThe Ultimate Guide to Phone Spy Apps: Everything You Need to Know
The Ultimate Guide to Phone Spy Apps: Everything You Need to Know
 
To Avoid Mistakes When Using Online Attendance Sheets
To Avoid Mistakes When Using Online Attendance SheetsTo Avoid Mistakes When Using Online Attendance Sheets
To Avoid Mistakes When Using Online Attendance Sheets
 
ThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and DjangoThaiPy meetup - Indexes and Django
ThaiPy meetup - Indexes and Django
 
Top Chinese Government-backed APT Groups
Top Chinese Government-backed APT GroupsTop Chinese Government-backed APT Groups
Top Chinese Government-backed APT Groups
 
Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...
Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...
Celebrity Girls Call Mumbai 🛵🚡9910780858 💃 Choose Best And Top Girl Service A...
 
Il Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazioneIl Data Streaming per un’AI real-time di nuova generazione
Il Data Streaming per un’AI real-time di nuova generazione
 
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
VVIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 i...
 
Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.Mobile App Development Company in Noida - Drona Infotech.
Mobile App Development Company in Noida - Drona Infotech.
 
NYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction InnovationNYGGS 360: A Complete ERP for Construction Innovation
NYGGS 360: A Complete ERP for Construction Innovation
 

Concurrency with Go

  • 1. Concurrency with Go Natural Design in a Technical Language Frank Müller
  • 2. •Oldenburg, Germany •Born 1965 •Working at Kubermatic •Team Lead Development •@themue Introduction Frank Müller
  • 3. Our world is concurrent
  • 4. • Individuals populate this world • They act sometimes independent of each other, sometimes dependent, sometimes together • Communication and signals enable their coexistence It's a natural principle Frank Müller
  • 12. Modern systems are more powerful
  • 13. • Computer architectures are changing • Growth via CPUs, cores and hyper-threads • Manual usage via threads is complex and error-prone • Concurrent runtime environments enable fine-grained usage One motivation is the hardware Frank Müller
  • 14. Distribution of computing power Frank Müller Process Process Process Process Process Process Process Process Runtime Envirunment Core Core Core Core Process Process Process Process
  • 15. Processes can work alone ...
  • 16. ... or as a team together
  • 17. • Encapsulation of the state in the process • Communication via messages • Sequential processing • Atomic state changes • OOP in the real sense Structure is more important motivation Frank Müller
  • 18. ❞ –Rob Pike Parallelis m Programming as the simultaneous execution of (possibly related) computations . Concurrenc y Programming as the composition of independently executing processes.
  • 19. • Actor Model ‣ 1973 ‣ Carl Hewitt, Peter Bishop und Richard Steiger • Communicating Sequential Processes ‣ 1978 ‣ Tony Hoare Ideas long known Frank Müller
  • 20. • Concurrent processes communicate via one or more channels • Processes, unlike channels, are anonymous • Data is not sent until the receiver is ready to receive it • Incoming data is processed sequentially Communicating Sequential Processes Frank Müller
  • 21. Communicating Sequential Processes Frank Müller Process Process Process Process Process Process Process Process Process Process
  • 24. Processes provide services Frank Müller Service Provider Client Client Client Client Active Waiting Waiting Waiting
  • 25. Processes manage ressources Frank Müller Client Client Client Manager Resource B Resource A Resource C Active Waiting Waiting Read / Write
  • 26. Processes manage parallel requests Frank Müller Worker Worker Worker Client Client Client Master Request A Request B Request C Request A Request B Request C Reply
  • 27. Processes handle events Frank Müller Event Manager Emitter Emitter Emitter Subscriber Subscriber Subscriber Events Events Events Events
  • 28. Processes monitor each other Frank Müller Supervisor Process A Process B Process B' Starts A Starts B Monitors A Monitors B Monitors B' Restarts B'
  • 29. Processes support powerful ETL Frank Müller Sender Sender Sender Transformator(s) Loader Extractor Receiver Receiver Receiver Transformator(s) Transformator(s) Transformed Data Transformed Data Transformed Data Raw Data Raw Data Raw Data
  • 31. • Development started 2007 by Google • Designed by Rob Pike, Ken Thompson, and Robert Griesemer • Initial release 2012 • Looks imperative, but is multi-paradigm • Types with methods, interfaces, function types, concurrency • Concurrency realized by goroutines and channels Google Go Frank Müller
  • 32. • Goroutines run lightweight in a thread pool • Functions spawned with the keyword go • Large simultaneous number possible • Channels are typed and run synchronous or buffered • They are used for communication and synchronization • select statements allow parallel processing of multiple channels • Use in for loops enables continuous processing Concurrency in Go Frank Müller
  • 34. // processMyData processes the data in the background. func processMyData(data Data) { ... } // startProcessing pre-processes the data and starts the goroutine. func startProcessing() { var data Data data = ... // Spawn goroutine. go processMyData(data) // Do something else. ... } Simple in own function Frank Müller
  • 35. // startProcessing pre-processes the data and starts the goroutine to process 
 // it in the background. func startProcessing() { var data Data data = ... // Spawn goroutine, function uses outer data. go func() { ... }() // Do something else. ... } Simple in embedded function Frank Müller
  • 36. // startProcessing pre-processes the data and starts the goroutine to process 
 // a copy in the background. func startProcessing() { var data Data data = ... // Spawn goroutine with a data copy, same name is no problem. go func(data Data) { ... }(data) // Do something else using the original data. ... } Embedded function with data copy Frank Müller
  • 37. // process pre-processes the data, starts the goroutine and waits until it's 
 // done. func process() { data := ... var wg sync.WaitGroup // sync.WaitGroup allows counting of activities. wg.Add(1) // In example here we wait for only one processing. go processData(&wg, data) ... wg.Wait() // Wait until it's done. } Waiting for processing of data Frank Müller
  • 38. // processData processes the passed data and signals its ending via // the sync.WaitGroup. func processData(wg *sync.WaitGroup, data Data) { // Deferred function call tells wait group that one processing is done. defer wg.Done() // Process data data. ... } Tell waiter that work is done Frank Müller
  • 39. // processDatas starts the goroutines to process the individual datas and waits // until all are done. func processDatas(datas []Data) { var wg sync.WaitGroup for _, data := range datas { // Add one per each data to process. wg.Add(1) // Spawn processing like in last example. go processData(&wg, data) } wg.Wait() // Wait until they are done. } Start a number of background jobs Frank Müller
  • 41. // processDatas processes all datas it receives via the data channel. // Loop ends when the channel is closed. func processDatas(dataChan <-chan Data) { for data := range dataChan { // Process the individual data sequentially. ... } } Process all datas received from channel Frank Müller
  • 42. // Create data channel. dataChan := make(chan Data) // Spawn processor with data channel. go processDatas(dataChan) // Send datas. dataChan <- dataA dataChan <- dataB dataChan <- dataC // Close channel. close(dataChan) Use the data processor Frank Müller
  • 43. // processAtoB processes all A datas it receives via the in channel. Results of // type B are written to the out channel. That will be closed if the function // ends working a.k.a. the in channel has been closed. func processAtoB(inChan <-chan A, outChan chan<- B) { defer close(outChan) for a := range inChan { b := ... outChan <- b } } // processBtoC works similar to processAtoB, only with B and C datas. func processBtoC(inChan <-chan B, outChan chan<- C) { ... } Piping Frank Müller
  • 44. // Create buffered channels. aChan := make(chan A, 5) bChan := make(chan B, 5) cChan := make(chan C, 5) // Spawn processors. go processAtoB(aChan, bChan) go processBtoC(bChan, cChan) go processCs(cChan) // Write A data into A channel, then close. ... close(aChan) Use the piping Frank Müller
  • 46. • Structure with data and channels • Function New() as constructor • Method loop() or backend() for loop with select statement • Public methods to access the instance • Requests with return values need explicit channel • Multiple return values need helper types Often found pattern Frank Müller
  • 47. // MyService simply provides a string and an integer field and allows to add // them if possible. type MyService struct { a string b int // Many channels needed this way. setAChan chan string getAChan chan chan string setBChan chan int getBChan chan chan int addChan chan addResp } Structure Frank Müller
  • 48. // New create a new instance of MyService. The backend goroutine is controlled // by the given context. func New(ctx context.Context) *MyService { ms := &MyService{ setAChan: make(chan string), getAChan: make(chan chan string), setBChan: make(chan int), getBChan: make(chan chan int), addChan: make(chan addReq), } go ms.backend(ctx) return ms } Constructor Frank Müller
  • 49. // SetA simply sends the data via the channel. func (ms *MyService) SetA(a string) { ms.setAChan <- a } // GetA sends a buffered channel and receives the result via it. func (ms *MyService) GetA() string { // Buffered to allow backend continue working. respChan := make(chan string, 1) ms.getAChan <- respChan return <-respChan } Setter and getter Frank Müller
  • 50. // addResp is a private transporter for the sum and a possible error. type addResp struct { sum int err error } // Add sends a buffered channel for a transporter. func (ms *MyService) Add() (int, error) { respChan := make(chan addResp, 1) ms.addChan <- retChan resp := <-retChan if resp.err != nil { return 0, resp.err } return resp.sum, nil } Sometimes even more effort needed Frank Müller
  • 51. // backend runs as goroutine and serializes the operations. func (ms *MyService) backend(ctx context.Context) { // Endless loop with for. for { // Select to switch between the channels. select { case <-ctx.Done(): // Terminate backend. return case ...: ... } } } Backend structure Frank Müller
  • 52. select { ... case respChan := <-ms.getBChan: respChan <- ms.b case respChan := <-ms.addChan: var resp addResp i, err := strconv.Atoi(ms.a) if err != nil { resp.err = err } else { resp.sum = i + ms.b } respChan <- resp } Other cases Frank Müller
  • 53. • Much extra effort with typed channels and helper types • Example here does not care if context is cancelled • Public methods and business logic are separated • Without private helpers the select statement may grow too much • But thankfully there's a more simple way to do it Summary Frank Müller
  • 55. // Actor only needs two extra fields. type Actor struct { ctx context.Context actChan chan func() // Here own data. ... } func New(ctx context.Context) *Actor { act := &Actor{ ... } go backend() return act } Structure and constructor Frank Müller
  • 56. // backend is only executing the received functions. func (act *Actor) backend() { for { select { case <-act.ctx.Done(): return case action := <-act.actChan: // Execute received function. action() } } } Backend Frank Müller
  • 57. // do always keeps an eye on the context when sending. func (act *Actor) do(action func()) error { select { case <-act.ctx.Done(): if ctx.Err() != nil { return ctx.Err() } return errors.New("actor has been stopped") case act.actChan <- action: return nil } } Safe sending of a function Frank Müller
  • 58. // Add works like the Add() from example before. func (act *Actor) Add() (i int, err error) { // Send logic to the backend. if aerr := act.do(func() { fi, ferr := strconv.Atoi(act.a) if ferr != nil { err = ferr return } i = fi + act.b }); aerr != nil { // Looks like the actor already has been stopped. return 0, aerr } } Business logic Frank Müller
  • 59. • Less code • Actor can be implemented in one package and always reused • Additional buffered channel and doAsync() support pure setters and callers without return values • do() and doAsync() also could get optional timeout for fault tolerant behavior Summary Frank Müller
  • 61. // Supervisor helps monitoring and restarting concurrent functions. type Supervisor struct { mu sync.Mutex workers map[string]func() error spawnChan chan string } // New starts the supervisor in the background. func New(ctx context.Context) *Supervisor { s := &Structure{ ... } go s.backend(ctx) return s } Structure and constructor Frank Müller
  • 62. // Spawn tells backend to spawn the given worker. func (s *Supervisor) spawn(id string, worker func() error) error { s.mu.Lock() defer s.mu.Unlock() _, ok := s.workers[id] if ok { return errors.New("double worker ID") } s.workers[id] = worker s.spawnChan <- id return nil } Start a worker Frank Müller
  • 63. // wrap takes care for errors of the worker. In case of an error it notifies // the backend to re-spawn. func (s *Supervisor) wrap(id string) { worker := s.workers[id] if err := worker(); err != nil { // Log the error and re-spawn the worker. log.Printf("worker %q terminated with error: %v", id, err) s.spawnChan <- id return } // Delete successful terminated worker. s.mu.Lock() delete(s.workers, id) s.mu.Unlock() } Wrapper to check if worker error Frank Müller
  • 64. // backend wraps workers and spawns them. func (s *Supervisor) backend(ctx context.Context) { for { select { case <-ctx.Done(): return case id := <-s.spawnChan: go s.wrap(id) } } } Backend Frank Müller
  • 66. • Channels work synchron or buffered • Working reader could lead to blocking writes, buffers may be filled • Most times just waiting, but overload resource may lead to instability • If needed check for parallelized pre-processing steps, e.g. by caller, and keep the need for serialization small • Alternatively assistent workers for the backend may help Blocked channels Frank Müller
  • 67. • Avoid overlapping read and write access with external modification • E.g. use IncrBy() instead of Get(), local addition, and Set() • Alternatively read value together with timed handle for update • Other modifiers get an error during this time • An unused handle must be released again Race conditions Frank Müller
  • 68. • Concurrent updates of coherent data may lead to invalid states • Avoid too fine granular access • Assure changes of all related data en bloc Non-atomic updates Frank Müller
  • 70. • Power of concurrency seems to be complex • Possibilities are manifold • As is often the case, only a few patterns make up almost all use cases • Own or 3rd party packages reduce work here • Design of elastic software requires a more natural rethink Final summary Frank Müller
  • 71. Thanks a lot and have a nice evening Image Sources 123RF Pexels iStockphoto Own photos