Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Job Queue in Golang

8,329 views

Published on

1. what is the different unbuffered and buffered channel?
2. how to implement a job queue in golang?
3. how to stop the worker in a container?
4. Shutdown with Sigterm Handling
5. Canceling Workers without Context
6. Graceful shutdown with worker
7. How to auto-scaling build agent?
8. How to cancel the current Job?

Published in: Technology
  • Copas Url to Read This eBook === http://bestadaododadj.justdied.com/B07BSMHFVQ-la-ou-les-chiens-aboient-par-la-queue.html
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Job Queue in Golang

  1. 1. Job Queue in Golang Bo-Yi Wu 2019.10.19
  2. 2. About me • Software Engineer in Mediatek • Member of Drone CI/CD Platform • Member of Gitea Platform • Member of Gin Golang Framework • Maintain Some GitHub Actions Plugins. • Teacher of Udemy Platform: Golang + Drone
  3. 3. Queue (Open Source) RabbitMQ NSQ
  4. 4. Why not use Open Source?
  5. 5. Single Service • Build in SQLite (LDAP, Job List) • Build in Cache (LRU) • Build for single binary • No third-party dependency
  6. 6. Before talking about Job Queue
  7. 7. Buffered vs Unbuffered http://bit.ly/understand-channel
  8. 8. Goroutine func main() { go func() { fmt.Println("GO GO GO") }() time.Sleep(1 * time.Second) }
  9. 9. Unbuffered make(chan bool)
  10. 10. Unbuffered Channel func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") c <- true }() <-c }
  11. 11. Unbuffered Channel func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") c <- true }() <-c }
  12. 12. Unbuffered Channel func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") <-c }() c <- true }
  13. 13. func main() { c := make(chan bool) go func() { fmt.Println("GO GO GO") c <- true c <- true }() <-c time.Sleep(1 * time.Second) } Unbuffered Channel
  14. 14. buffered make(chan bool, 1)
  15. 15. Buffered channel func main() { c := make(chan bool, 1) go func() { fmt.Println("GO GO GO") <-c }() c <- true }
  16. 16. Buffered channel func main() { c := make(chan bool, 1) go func() { fmt.Println("GO GO GO") <-c }() c <- true }
  17. 17. Buffered channel func main() { c := make(chan bool, 1) go func() { fmt.Println("GO GO GO") c <- true }() <-c }
  18. 18. How to implement Job Queue in Go
  19. 19. Sometimes you don’t need A job queue go process("job01")
  20. 20. func worker(jobChan <-chan Job) { for job := range jobChan { process(job) } } // make a channel with a capacity of 1024. jobChan := make(chan Job, 1024) // start the worker go worker(jobChan) // enqueue a job jobChan <- job
  21. 21. func worker(jobChan <-chan Job) { for job := range jobChan { process(job) } } // make a channel with a capacity of 1024. jobChan := make(chan Job, 1024) // start the worker go worker(jobChan) // enqueue a job jobChan <- job
  22. 22. Block if there already are 1024 jobs jobChan := make(chan Job, 1024)
  23. 23. Enqueue without blocking
  24. 24. func Enqueue(job Job, jobChan chan<- Job) bool { select { case jobChan <- job: return true default: return false } } if !Enqueue(job, job100) { Error( http.StatusServiceUnavailable, "max capacity reached", ) return }
  25. 25. Stopping the worker?
  26. 26. func main() { ch := make(chan int, 2) go func() { ch <- 1 ch <- 2 }() for n := range ch { fmt.Println(n) } }
  27. 27. func main() { ch := make(chan int, 2) go func() { ch <- 1 ch <- 2 close(ch) }() for n := range ch { fmt.Println(n) } }
  28. 28. func main() { ch := make(chan int, 2) go func() { ch <- 1 ch <- 2 }() go func() { for n := range ch { fmt.Println(n) } }() time.Sleep(1 * time.Second) }
  29. 29. Setup Consumer
  30. 30. type Consumer struct { inputChan chan int jobsChan chan int } const PoolSize = 2 func main() { // create the consumer consumer := Consumer{ inputChan: make(chan int, 1), jobsChan: make(chan int, PoolSize), } }
  31. 31. func (c *Consumer) queue(input int) { fmt.Println("send input value:", input) c.jobsChan <- input } func (c *Consumer) worker(num int) { for job := range c.jobsChan { fmt.Println("worker:", num, " job value:", job) } } for i := 0; i < PoolSize; i++ { go consumer.worker(i) }
  32. 32. consumer.queue(1) consumer.queue(2) consumer.queue(3) consumer.queue(4) blocking by poll size = 2
  33. 33. Output (Poll Size = 2) send input value: 1 send input value: 2 send input value: 3 worker: 0 job value: 1 send input value: 4 worker: 0 job value: 2 worker: 0 job value: 3 worker: 0 job value: 4
  34. 34. rewrite queue func func (c *Consumer) queue(input int) bool { fmt.Println("send input value:", input) select { case c.jobsChan <- input: return true default: return false } }
  35. 35. Output (Poll Size = 2) send input value: 1 send input value: 2 send input value: 3 send input value: 4 worker: 0 job value: 1 worker: 0 job value: 2
  36. 36. Shutdown with Sigterm Handling
  37. 37. func WithContextFunc(ctx context.Context, f func()) context.Context { ctx, cancel := context.WithCancel(ctx) go func() { c := make(chan os.Signal) signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) defer signal.Stop(c) select { case <-ctx.Done(): case <-c: f() cancel() } }() return ctx }
  38. 38. func WithContextFunc(ctx context.Context, f func()) context.Context { ctx, cancel := context.WithCancel(ctx) go func() { c := make(chan os.Signal) signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) defer signal.Stop(c) select { case <-ctx.Done(): case <-c: f() cancel() } }() return ctx }
  39. 39. func (c Consumer) startConsumer(ctx context.Context) { for { select { case job := <-c.inputChan: if ctx.Err() != nil { close(c.jobsChan) return } c.jobsChan <- job case <-ctx.Done(): close(c.jobsChan) return } } }
  40. 40. Cancel by ctx.Done() event func (c *Consumer) worker(num int) { for job := range c.jobsChan { fmt.Println("worker:", num, " job value:", job) } }
  41. 41. Canceling Workers without Context
  42. 42. cancelChan := make(chan struct{}) go worker(jobChan, cancelChan) func worker(jobChan <-chan Job, cancelChan <-chan struct{}) { for { select { case <-cancelChan: return case job := <-jobChan: process(job) } } } // to cancel the worker, close the cancel channel close(cancelChan) Create a cancel channel
  43. 43. cancelChan := make(chan struct{}) go worker(jobChan, cancelChan) func worker(jobChan <-chan Job, cancelChan <-chan struct{}) { for { select { case <-cancelChan: return case job := <-jobChan: process(job) } } } // to cancel the worker, close the cancel channel close(cancelChan) Create a cancel channel close(cancelChan)
  44. 44. Graceful shutdown with worker sync.WaitGroup
  45. 45. wg := &sync.WaitGroup{} wg.Add(numberOfWorkers) // Start [PoolSize] workers for i := 0; i < PoolSize; i++ { go consumer.worker(i) }
  46. 46. WaitGroup WaitGroup WaitGroup WaitGroup
  47. 47. func (c Consumer) worker(wg *sync.WaitGroup) { defer wg.Done() for job := range c.jobsChan { // handle the job event } }
  48. 48. Add WaitGroup after Cancel Function
  49. 49. func WithContextFunc(ctx context.Context, f func()) context.Context { ctx, cancel := context.WithCancel(ctx) go func() { c := make(chan os.Signal) signal.Notify(c, syscall.SIGINT, syscall.SIGTERM) defer signal.Stop(c) select { case <-ctx.Done(): case <-c: cancel() f() } }() return ctx } Add WaitGroup after Cancel Function
  50. 50. wg := &sync.WaitGroup{} wg.Add(numberOfWorkers) ctx := signal.WithContextFunc( context.Background(), func() { wg.Wait() close(finishChan) }, ) go consumer.startConsumer(ctx)
  51. 51. End of Program select { case <-finished: case err := <-errChannel: if err != nil { return err } }
  52. 52. How to auto-scaling build agent?
  53. 53. Communicate between server and agent
  54. 54. Jobs Schema
  55. 55. r := e.Group("/rpc") r.Use(rpc.Check()) { r.POST("/v1/healthz", web.RPCHeartbeat) r.POST("/v1/request", web.RPCRquest) r.POST("/v1/accept", web.RPCAccept) r.POST("/v1/details", web.RPCDetails) r.POST("/v1/updateStatus", web.RPCUpdateStatus) r.POST("/v1/upload", web.RPCUploadBytes) r.POST("/v1/reset", web.RPCResetStatus) } Check RPC Secret
  56. 56. /rpc/v1/accept Update jobs set version = (oldVersion + 1) where machine = "fooBar" and version = oldVersion
  57. 57. Create multiple worker
  58. 58. if r.Capacity != 0 { var g errgroup.Group for i := 0; i < r.Capacity; i++ { g.Go(func() error { return r.start(ctx, 0) }) time.Sleep(1 * time.Second) } return g.Wait() }
  59. 59. Break for and select loop func (r *Runner) start(ctx context.Context, id int64) error { LOOP: for { select { case <-ctx.Done(): return ctx.Err() default: r.poll(ctx, id) if r.Capacity == 0 { break LOOP } } time.Sleep(1 * time.Second) } return nil }
  60. 60. How to cancel the current Job?
  61. 61. Context with Cancel or Timeout ctx, cancel := context.WithCancel(context.Background()) defer cancel() timeout, cancel := context.WithTimeout(ctx, 60*time.Minute) defer cancel() Job03 context
  62. 62. Context with Cancel or Timeout ctx, cancel := context.WithCancel(context.Background()) defer cancel() timeout, cancel := context.WithTimeout(ctx, 60*time.Minute) defer cancel() Job03 context Job05 context
  63. 63. Watch the Cancel event go func() { done, _ := r.Manager.Watch(ctx, id) if done { cancel() } }()
  64. 64. Handle cancel event on Server subscribers: make(map[chan struct{}]int64), cancelled: make(map[int64]time.Time),
  65. 65. User cancel running job c.Lock() c.cancelled[id] = time.Now().Add(time.Minute * 5) for subscriber, build := range c.subscribers { if id == build { close(subscriber) } } c.Unlock()
  66. 66. Agent subscribe the cancel event for { select { case <-ctx.Done(): return false, ctx.Err() case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil } case <-subscriber: return true, nil } }
  67. 67. case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil }
  68. 68. case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil } 1 Cancel
  69. 69. case <-time.After(time.Minute): c.Lock() _, ok := c.cancelled[id] c.Unlock() if ok { return true, nil } 1 2 Reconnect Server Cancel
  70. 70. https://www.udemy.com/course/golang-fight/?couponCode=GOLANG201911
  71. 71. https://www.udemy.com/course/devops-oneday/?couponCode=DEVOPS201911
  72. 72. Any Question?

×