Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

神に近づくx/net/context (Finding God with x/net/context)


Published on

How do you manage a multi-package API in Go? An overview of guregu/kami and x/net/context

Published in: Technology
  • Be the first to comment

神に近づくx/net/context (Finding God with x/net/context)

  1. 1. 神に近づくx/net/context Finding God with x/net/context 11 March 2015 Gregory Roseberry Gunosy
  2. 2. ※ This talk has nothing to do with religion. ※ Go Gopher by Renee French
  3. 3. Let's make an API server
  4. 4. Attempt #1: Standard library Everyone told me to use the standard library, let's use it. Index page says hello Secret message page requires key func main() { http.HandleFunc("/", indexHandler) http.HandleFunc("/secret/message", requireKey(secretMessageHandler)) http.ListenAndServe(":8000", nil) } func indexHandler(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "hello world") }
  5. 5. Secret message func secretMessageHandler(w http.ResponseWriter, r *http.Request) { io.WriteString(w, "42") } Here's a way to write middleware with just the standard library: func requireKey(h http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { if r.FormValue("key") != "12345" {if r.FormValue("key") != "12345" { http.Error(w, "bad key", http.StatusForbidden) return } h(w, r) } } In main.go: http.HandleFunc("/secret/message", requireKey(secretMessageHandler))
  6. 6. There's been a change of plans... We were hard-coding the key, but your boss says now we need to check Redis. Let's just make our Redis connection a global variable for now... var redisDB *redis.Client func main() { redisDB = ... // set up redis } func requireKeyRedis(h http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { key := r.FormValue("key") userID, err := redisDB.Get("auth:" + key).Result()userID, err := redisDB.Get("auth:" + key).Result() if key == "" || err != nil {if key == "" || err != nil { http.Error(w, "bad key", http.StatusForbidden) return } log.Println("user", userID, "viewed message") h(w, r) } }
  7. 7. Just one quick addition... We need to issue temporary session tokens for some use cases, so we need to check if either a key or a session is provided. func requireKeyOrSession(h http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { key := r.FormValue("key") // set key from db if we have a session if session := r.FormValue("session"); session != "" {if session := r.FormValue("session"); session != "" { var err error if key, err = redisDB.Get("session:" + session).Result(); err != nil {if key, err = redisDB.Get("session:" + session).Result(); err != nil { http.Error(w, "bad session", http.StatusForbidden) return } } userID, err := redisDB.Get("auth:" + key).Result() if key == "" || err != nil { http.Error(w, "bad key", http.StatusForbidden) return } log.Println("user", userID, "viewed message") h(w, r) } }
  8. 8. By the way... Your boss also asks: Can we also check the X-API-Key header? Can we restrict certain keys to certain IP addresses? Can we ...? There's too much to shove into one middleware: so we make an auth package. package auth type Auth struct { Key string Session string UserID string } func Check(r *http.Request) (auth Auth, ok bool) { // lots of complicated checks }
  9. 9. What about Redis? We need to reference the DB from our new auth package as well. Should we pass the connection to Check? func Check(redisDB *redis.Client, r *http.Request) (Auth, bool) { ... } What happens we need to check MySQL as well? func Check(redisDB *redis.Client, archiveDB *sql.DB, r *http.Request) (Auth, bool) { ... } Your boss says MongoDB is web scale, so that gets added too. func Check(redisDB *redis.Client, archiveDB *sql.DB, mongo *mgo.Session, r *http.Request) (Auth, bool) { ... This isn't going to work...
  10. 10. How about an init method? Making a global here too? var redisDB *redis.Client func Init(r *redis.Client, ...) { redisDB = r } That doesn't solve our arguments problem. Let's shove them in a struct. package config type Context struct { RedisDB *redis.Client ArchiveDB *sql.DB ... } Init with this guy? auth.Init(appContext) Who inits who? What about tests?
  11. 11. Just one more thing... Your boss says it's vital that we log every request now, and include the key and user ID if possible. It's easy to write logging middleware, but how can we make our logger aware of our Auth credentials?
  12. 12. Session table Let's try making a global map of connections to auths. var authMap map[*http.Request]*auth.Auth Then populate it during our check. func requireKeyOrSession(h http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { ... a, ok := auth.Check(dbContext, r) authMapMutex.Lock() authMap[r] = &a ... } } Should work, but will our *http.Requests leak? We need to make sure to clean them up. What happens when we need to keep track of more than just Auth? How do we coordinate this data across packages? What about concurrency? (This is kind of how gorilla/sessions works)
  13. 13. There's got to be another way...
  14. 14. Attempt #2: Goji Goji is a popular web micro-framework. Goji handlers take an extra parameter called web.C (probably short for Context). c.Env is a map[interface{}]interface{} for storing arbitrary data — perfect for our auth token! This used to be a map[string]interface{}, more on this later. Let's rewrite our auth middleware for Goji: func requiresKey(c *web.C, h http.Handler) http.Handler { fn := func(w http.ResponseWriter, r *http.Request) { a := c.Env["auth"] if a == nil { http.Error(w, "bad key", http.StatusForbidden) return } h.ServeHTTP(w, r) } return http.HandlerFunc(fn) }
  15. 15. Goji groups We can set up groups of routes: package main import ( "" "" ) func main() { ... secretGroup := web.New() secretGroup.Use(requiresKey) secretGroup.Get("/secret/message", secretMessageHandler) goji.Handle("/secret/*", secretGroup) goji.Serve() } This will run our checkAuth for all routes under /secret/.
  16. 16. Goji benefits Fast routing Middleware groups Request context Einhorn support (zero-downtime deploys)
  17. 17. Downside: Goji-flavored context Let's say we want to re-use our auth package elsewhere, like a batch process. Do we want to put our database connections in web.C, even if we're not running a web server? Should all of our internal packages be importing Goji? package auth func Check(c web.C, session, key string) bool { // How do we call this if we're not using goji? redisDB, _ := c.Env["redis"].(*redis.Client) // kind of ugly... } Having to do a type assertion every time we use this DB is annoying. Also, what happens when some other library wants to use this "redis" key?
  18. 18. Downside: Groups need to be set up once, in main.go Defining middleware for a group is tricky. What happens if you have code like... package addon func init() { goji.Get("/secret/addon", addonHandler) // will secretGroup handle this? } Everything works will if your entire app is set up in main.go, but in my experience it's very finicky and hard to reason about handlers that are set up in other ways.
  19. 19. There's got to be another way...!
  20. 20. Attempt #3: kami & x/net/context What is x/net/context? It's an almost-standard package for sharing context across your entire app. Includes facilities for setting deadlines and cancelling requests. Includes a way to store data similar to Goji's web.C. Immutable, must be replaced to update Check out this official blog post, which focuses mostly on x/net/context for cancellation: Quick example: ctx := context.Background() // blank context ctx = context.WithValue(ctx, "my_key", "my_value") fmt.Println(ctx.Value("my_key").(string)) // "my_value"
  21. 21. kami kami is a mix of HttpRouter, x/net/context, and Goji, with a very simple middleware system included. package main import ( "fmt" "net/http" "" "" ) func hello(ctx context.Context, w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Hello, %s!", kami.Param(ctx, "name")) } func main() { kami.Get("/hello/:name", hello) kami.Serve() }
  22. 22. Example: sharing DB connections import "" I made a simple package for storing DB connections in your context. At Gunosy, we use something similar. db.OpenSQL() returns a new context containing a named SQL connection. func main() { ctx := context.Background() mysqlURL := "root:hunter2@unix(/tmp/mysql.sock)/myCoolDB" ctx = db.OpenSQL(ctx, "main", "mysql", mysqlURL)ctx = db.OpenSQL(ctx, "main", "mysql", mysqlURL) defer db.Close(ctx) // closes all DB connectionsdefer db.Close(ctx) // closes all DB connections kami.Context = ctxkami.Context = ctx kami.Get("/hello/:name", hello) kami.Serve() } kami.Context is our "god context" from which all request contexts are derived.
  23. 23. Example: sharing DB connections (2) Within a request, we use db.SQL(ctx, name) to retrieve the connection. func hello(ctx context.Context, w http.ResponseWriter, r *http.Request) { mainDB := db.SQL(ctx, "main") // *sql.DBmainDB := db.SQL(ctx, "main") // *sql.DB var greeting string mainDB.QueryRow("SELECT content FROM greetings WHERE name = ?", kami.Param(ctx, "name")). Scan(&greeting) fmt.Fprintf(w, "Hello, %s!", greeting) }
  24. 24. Tests For tests, you can put a mock DB connection in your context. main_test.go: import _ "" testhelper/testhelper.go: import ( "" "" _ "" ) func init() { ctx := context.Background() // use mogi for tests ctx = db.OpenSQL("main", "mogi", "") kami.Context = ctx }
  25. 25. How does it work? Because context.Value() takes an interface{}, we can use unexported type as the key to "protect" it. This way, other packages can't screw with your data. In order to interact with a database, you have to use the exported functions like OpenSQL, and Close. package db import ( "database/sql" "" ) type sqlkey string // lowercase! // SQL retrieves the *sql.DB with the given name or nil. func SQL(ctx context.Context, name string) *sql.DB { db, _ := ctx.Value(sqlkey(name)).(*sql.DB) return db } BTW: This is why Goji switched its web.C from a map[string]interface{} to map[interface{}]interface{}.
  26. 26. Middleware kami has no concept of middleware "groups". Middleware is strictly hierarchical. For example, a request for /secret/message would run the middleware registered under the following paths in order: / /secret/ /secret/message This means that you can define your paths anywhere and still get predictable middleware behavior. kami.Use("/secret/", requireKey)
  27. 27. Middleware (2) kami.Middleware is defined as: type Middleware func(context.Context, http.ResponseWriter, *http.Request) context.Context The context you return will be used for the next middleware or handler. Unlike Goji, you don't have control of how the next handler will be called. But, you can return nil to halt the execution chain.
  28. 28. Middleware (3) import "" func init() { kami.Use("/", doAuth) kami.Use("/secret/", requiresKey) } // doAuth returns a new context with the appropiate auth object inside func doAuth(ctx context.Context, w http.ResponseWriter, r *http.Request) context.Context { if a, err := auth.ByKey(ctx, r.FormValue("key")); err == nil { // put auth object in context ctx = auth.NewContext(ctx, a) } return ctx } // requiresKey stops the request if we don't have an auth object func requiresKey(ctx context.Context, w http.ResponseWriter, r *http.Request) context.Context { if _, ok := auth.FromContext(ctx); !ok { http.Error(w, "bad key", http.StatusForbidden) return nil // stop request } return ctx }
  29. 29. Hooks kami provides special hooks for logging and recovering from panics, kami.LogHandler and kami.PanicHandler. Handling panics. kami.PanicHandler = func(ctx context.Context, w http.ResponseWriter, r *http.Request) { err := kami.Exception(ctx) a, _ := auth.FromContext(ctx) log.Println("panic", err, a) } Logging request statuses. Notice how the function signature is different, it takes a writer proxy that includes the status code. kami.LogHandler = func(ctx context.Context, w mutil.WriterProxy, r *http.Request) { a, _ := auth.FromContext(ctx) log.Println("access", w.Status(), r.URL.Path, "from:", a.Key, a.UserID) } LogHandler will run after PanicHandler, unless LogHandler is the one panicking.
  30. 30. Graceful This is the "Goji" part of kami. Literally copy and pasted from Goji. kami.Serve() // works *exactly* like goji.Serve() Supports Einhorn for graceful restarts. Thank you, Goji.
  31. 31. Downsides kami isn't perfect. It is rather inflexible and may not fit your needs. You can't define separate groups of middleware, or separate groups of handlers, everything is global. You could mount kami.Handler() outside of "/" and use another router... You can't register middleware under wildcard paths: kami.Use("/user/:id/profile", middleware) won't work. Register it under /user/ and do your best. I will probably fix these issues eventually. Might have to fork HttpRouter... Pull requests are always welcome.
  32. 32. Production ready! We use kami to power the Gunosy API and it works just fine! Switching to x/net/context eliminates nearly all global variables. No more somepkg.Init() madness. Easy to test: just put mocks inside your context. Check it out!
  33. 33. Thank you Gregory Roseberry Gunosy