Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
digitalocean.com
Go Generate Pipelines
Pipelines?
Pipelines
• https://blog.golang.org/pipelines
• concurrency pattern
• “stages connected by channels"
Pipelines
• produce a channel of stuff
• creates at least one goroutine
• source -> [middle] -> [middle] -> sink
Example
• https://play.golang.org/p/4Awrr2PgFo
Pattern
• directional channels
• goroutine
• close what you create
• for/range to completion (previous stage closes)
Concurrency
• increase the concurrency by stage
• easy to add
• https://play.golang.org/p/_qQVZ0Rh73
Generators vs Pipeline
• generators create values when you ask for them
• pipeline produce values before you ask for them
...
Real Program Concerns
• leaking goroutines
• cancelation
• errors
Leaking Goroutines
• every goroutine must terminate!
• pipelines create many goroutines
• https://play.golang.org/p/jUZIv6...
Cancelation Strategies
• exit program
• close source stage let downstream drain naturally
• close all stages
Exit program
• easy
• really really easy
Close source & drain downstream
• source closing trickles down to consumers
• all goroutines can exit
• https://play.golan...
Close all stages
• `out <-` must be interruptible everywhere
• every channel producing function provided “done”
• https://...
Errors
• can happen anywhere
• second output channel
• https://play.golang.org/p/aUv5MrD7Bj
Errors Take 2
• https://godoc.org/golang.org/x/net/context
• need a “pipeline context”
• signal done and place to write er...
Pipelines
• `out <-` always interruptible by `done`
• close what you create
• error channel/pipeline context
• often repet...
Go Generate
Go Generate
• since go1.4
• “comments” in your go project
• //go:generate [command] [args…]
Just a command
• generate say hi (for fun)
• generate rm -fr / (for evil)
• generate printenv (for understanding)
Not a shell…
• go:generate echo $EDITOR > /tmp/nope
• go:generate cat /tmp/file | grep something
But shellish…
• $GOFILE
• $GOPACKAGE
• $GOARCH
• $GOOS
• go:generate something -out=$GOARCH_something.go
Meant to write files
• stderr for communicating to the user
• silent in proper operation
• side effect (file) then committed...
Pipeliner
My common pipelines
• batch(<-chan Event) (<-chan []Event)
• flatMap(<-chan gzipfile) (<-chan Event)
• from([]Event) (<-chan...
Pipeliner
• side-project
• saves me time
• may annoy you (bad error messages)
• github.com/supershabam/pipeliner
digitalocean.com
close(talk)
digitalocean.com
questions?
Upcoming SlideShare
Loading in …5
×

go generate pipeliner

341 views

Published on

pipelines in golang

Published in: Software
  • Be the first to comment

go generate pipeliner

  1. 1. digitalocean.com Go Generate Pipelines
  2. 2. Pipelines?
  3. 3. Pipelines • https://blog.golang.org/pipelines • concurrency pattern • “stages connected by channels"
  4. 4. Pipelines • produce a channel of stuff • creates at least one goroutine • source -> [middle] -> [middle] -> sink
  5. 5. Example • https://play.golang.org/p/4Awrr2PgFo
  6. 6. Pattern • directional channels • goroutine • close what you create • for/range to completion (previous stage closes)
  7. 7. Concurrency • increase the concurrency by stage • easy to add • https://play.golang.org/p/_qQVZ0Rh73
  8. 8. Generators vs Pipeline • generators create values when you ask for them • pipeline produce values before you ask for them • https://play.golang.org/p/R_XHMu1jBz
  9. 9. Real Program Concerns • leaking goroutines • cancelation • errors
  10. 10. Leaking Goroutines • every goroutine must terminate! • pipelines create many goroutines • https://play.golang.org/p/jUZIv6H4l2
  11. 11. Cancelation Strategies • exit program • close source stage let downstream drain naturally • close all stages
  12. 12. Exit program • easy • really really easy
  13. 13. Close source & drain downstream • source closing trickles down to consumers • all goroutines can exit • https://play.golang.org/p/pRpNspMG2z
  14. 14. Close all stages • `out <-` must be interruptible everywhere • every channel producing function provided “done” • https://play.golang.org/p/Rd-hEVic0O
  15. 15. Errors • can happen anywhere • second output channel • https://play.golang.org/p/aUv5MrD7Bj
  16. 16. Errors Take 2 • https://godoc.org/golang.org/x/net/context • need a “pipeline context” • signal done and place to write errors • WIP idea • https://play.golang.org/p/-sn3QGpc7X
  17. 17. Pipelines • `out <-` always interruptible by `done` • close what you create • error channel/pipeline context • often repetitive…
  18. 18. Go Generate
  19. 19. Go Generate • since go1.4 • “comments” in your go project • //go:generate [command] [args…]
  20. 20. Just a command • generate say hi (for fun) • generate rm -fr / (for evil) • generate printenv (for understanding)
  21. 21. Not a shell… • go:generate echo $EDITOR > /tmp/nope • go:generate cat /tmp/file | grep something
  22. 22. But shellish… • $GOFILE • $GOPACKAGE • $GOARCH • $GOOS • go:generate something -out=$GOARCH_something.go
  23. 23. Meant to write files • stderr for communicating to the user • silent in proper operation • side effect (file) then committed to repository • can also generate _test.go file
  24. 24. Pipeliner
  25. 25. My common pipelines • batch(<-chan Event) (<-chan []Event) • flatMap(<-chan gzipfile) (<-chan Event) • from([]Event) (<-chan Event)
  26. 26. Pipeliner • side-project • saves me time • may annoy you (bad error messages) • github.com/supershabam/pipeliner
  27. 27. digitalocean.com close(talk)
  28. 28. digitalocean.com questions?

×