Your SlideShare is downloading. ×
0
Painless Data
Storage with
MongoDB & Go
• Author of Hugo, Cobra,
Viper & More
• Chief Developer
Advocate for MongoDB
• Gopher
@spf13
Why Go?
Why Another Language?
• Software is slow
• Sofware is hard to write
• Software doesn’t scale well
Go is Fast
• Go execution speed is close to C
• Go compile time rivals dynamic
interpretation
Go is Friendly
• Feels like a dynamic language in many ways
• Very small core language, easy to
remember all of it
• Singl...
Go is Concurrent
• Concurrency is part of the language
• Any function can become a goroutine
• Goroutines run concurrently...
MongoDB
Why Another Database?
• Databases are slow
• Relational structures don’t fit well
with modern programming
(ORMs)
• Database...
MongoDB is Fast
• Written in C++
• Extensive use of memory-mapped files 

i.e. read-through write-through memory caching.
•...
MongoDB is Friendly
• Ad Hoc queries
• Real time aggregation
• Rich query capabilities
• Traditionally consistent
• Geospa...
MongoDB is “Web Scale”
• Built in sharding support distributes data
across many nodes
• MongoS intelligently routes to the...
Document Database
• Not for .PDF & .DOC files
• A document is essentially an associative array
• Document == JSON object
• ...
Data Serialization
• Applications need persistant data
• The process of translating data structures
into a format that can...
BSON
• Inspired by JSON
• Cross language binary serialization format
• Optimized for scanning
• Support for richer types
MongoDB &
Go
Go’s Data Types
• Go uses strict & static typing
• 2 Types are similar to a BSON document
• Struct
• Map
bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Pri...
bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Pri...
bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Pri...
bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Pri...
bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Pri...
bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Pri...
!
type Project struct {
Name string `bson:"name"`
ImportPath string `bson:"importPath"`
}
project := Project{name, path}
!...
mgo (mango)
• Pure Go
• Created in late 2010 

("Where do I put my Go data?")
• Adopted by Canonical and MongoDB Inc.
itse...
Connecting
• Same interface for server, replica set, or shard
• Driver discovers and maintains topology
• Server added/removed, failo...
• Sessions are lightweight
• Sessions are copied (settings preserved)
• Single management goroutine for all copied session...
• Saves typing
• Uses the same session over and over
Convenient Access
projects := session.DB("OSCON").C("projects")
Writing
type Project struct {
Name string `bson:"name,omitempty"`
ImportPath string `bson:"importPath,omitempty"`
}
Defining Our Ow...
var projectList = []Project{
{"gocheck", "gopkg.in/check.v1"},
{"qml", "gopkg.in/qml.v0"},
{"pipe", "gopkg.in/pipe.v2"},
{...
type M map[string]interface{}
!
change := M{"$set": Project{ImportPath: "gopkg.in/
qml.v1"}}
!
err = projects.Update(Proje...
Querying
var project Project
!
err = projects.Find(Project{Name: "qml"}).One(&project)
if err != nil {
return err
}
!
fmt.Printf("P...
iter := projects.Find(nil).Iter()
!
var project Project
for iter.Next(&project) {
fmt.Printf("Project: %vn", project)
}
!
...
m := map[string]interface{}{
"name": "godep",
"tags": []string{"tool", "dependency"},
"contact": bson.M{
"name": "Keith Ra...
type Contact struct {
Name string
Email string
}
!
type Project struct {
Name string
Tags []string `bson:",omitempty"`
Con...
• Compound
• List indexing (think tag lists)
• Geospatial
• Dense or sparse
• Full-text searching
Indexing
// Root field
e...
Concurrency
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: na...
• Find 1 issued

• Doc 1 returned
• Find 2 issued

• Doc 2 returned
A Common Approach
Find 1 Find 2 DB
}
}
• Find 1 issued
• Find 2 issued

• Doc 1 returned
• Doc 2 returned
Concurrent Queries
Find 1 Find 2 DB
}
}
• Loads 200 results at a time
• Loads next batch with (0.25 * 200) results left to process
Concurrent Loading
session.SetB...
• Each Copy uses a different connection
• Closing session returns socket to the pool
• defer runs at end of function
Handl...
• Shares a single connection
• Still quite efficient thanks to concurrent capabilities of go + mgo
Handler With Single Sess...
GridFS
GridFS
• Not quite a file system
• Really useful for local file storage
• A convention, not a feature
• Supported by all dri...
gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file....
gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file....
gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file....
gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file....
gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file....
gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file....
gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file....
Full Featured
Features
• Transactions (mgo/txn experiment)
• Aggregation pipelines
• Full-text search
• Geospatial support
• Hadoop
In Conclusion
Getting Started
• 1. Install MongoDB
• 2. go get gopkg.in/mgo.v2
• 3. Start small
• 4. Build something great
Learning More
• MongoDB Manual
• Effective Go
• labix.org/mgo
Workshop Using mgo on spf13.com
on spf13.com
• @spf13
• Author of Hugo, Cobra,
Viper & More
• Chief Developer
Advocate for MongoDB
• Gopher
Thank You
Upcoming SlideShare
Loading in...5
×

Painless Data Storage with MongoDB & Go

10,415

Published on

This presentation will give developers an introduction and practical experience
of using MongoDB with the Go language. MongoDB Chief Developer Advocate &
Gopher Steve Francia presents plainly what you need to know about using MongoDB
with Go.

As an emerging language Go is able to start fresh without years of relational database dependencies. Application and library developers are able to build applications using the excellent Mgo MongoDB driver and the reliable go sql package for relational database. Find out why some people claim Go and MongoDB are a “pair made in heaven” and “the best database driver they’ve ever used” in this talk by Gustavo Niemeyer, the author of the mgo driver, and Steve Francia, the drivers team lead at MongoDB Inc.

We will cover:

Connecting to MongoDB in various configurations
Performing basic operations in Mgo
Marshaling data to and from MongoDB
Asynchronous & Concurrent operations
Pre-fetching batches for seamless performance
Using GridFS
How MongoDB uses Mgo internally

Published in: Technology
0 Comments
18 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,415
On Slideshare
0
From Embeds
0
Number of Embeds
45
Actions
Shares
0
Downloads
179
Comments
0
Likes
18
Embeds 0
No embeds

No notes for slide

Transcript of "Painless Data Storage with MongoDB & Go "

  1. 1. Painless Data Storage with MongoDB & Go
  2. 2. • Author of Hugo, Cobra, Viper & More • Chief Developer Advocate for MongoDB • Gopher @spf13
  3. 3. Why Go?
  4. 4. Why Another Language? • Software is slow • Sofware is hard to write • Software doesn’t scale well
  5. 5. Go is Fast • Go execution speed is close to C • Go compile time rivals dynamic interpretation
  6. 6. Go is Friendly • Feels like a dynamic language in many ways • Very small core language, easy to remember all of it • Single binary installation, no dependencies • Extensive Tooling & StdLib
  7. 7. Go is Concurrent • Concurrency is part of the language • Any function can become a goroutine • Goroutines run concurrently, communicate through channels • Select waits for communication on any of a set of channels
  8. 8. MongoDB
  9. 9. Why Another Database? • Databases are slow • Relational structures don’t fit well with modern programming (ORMs) • Databases don’t scale well
  10. 10. MongoDB is Fast • Written in C++ • Extensive use of memory-mapped files 
 i.e. read-through write-through memory caching. • Runs nearly everywhere • Data serialized as BSON (fast parsing) • Full support for primary & secondary indexes • Document model = less work
  11. 11. MongoDB is Friendly • Ad Hoc queries • Real time aggregation • Rich query capabilities • Traditionally consistent • Geospatial features • Support for most programming languages • Flexible schema
  12. 12. MongoDB is “Web Scale” • Built in sharding support distributes data across many nodes • MongoS intelligently routes to the correct nodes • Aggregation done in parallel across nodes
  13. 13. Document Database • Not for .PDF & .DOC files • A document is essentially an associative array • Document == JSON object • Document == PHP Array • Document == Python Dict • Document == Ruby Hash • etc
  14. 14. Data Serialization • Applications need persistant data • The process of translating data structures into a format that can be stored • Ideal format accessible from many languages
  15. 15. BSON • Inspired by JSON • Cross language binary serialization format • Optimized for scanning • Support for richer types
  16. 16. MongoDB & Go
  17. 17. Go’s Data Types • Go uses strict & static typing • 2 Types are similar to a BSON document • Struct • Map
  18. 18. bob := &Person{ Name: "Bob", Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf("Data: %qn", data)
 ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf("Person: %vn", person) Serializing with BSON
  19. 19. bob := &Person{ Name: "Bob", Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf("Data: %qn", data)
 ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf("Person: %vn", person) Serializing with BSON
  20. 20. bob := &Person{ Name: "Bob", Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf("Data: %qn", data)
 ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf("Person: %vn", person) Serializing with BSON
  21. 21. bob := &Person{ Name: "Bob", Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf("Data: %qn", data)
 ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf("Person: %vn", person) Serializing with BSON
  22. 22. bob := &Person{ Name: "Bob", Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf("Data: %qn", data)
 ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf("Person: %vn", person) Serializing with BSON
  23. 23. bob := &Person{ Name: "Bob", Birthday: time.Now(), } ! data, err := bson.Marshal(bob) if err != nil { return err } fmt.Printf("Data: %qn", data)
 ! var person Person err = bson.Unmarshal(data, &person) if err != nil { return err } fmt.Printf("Person: %vn", person) Serializing with BSON Data: "%x00x00x00x02name x00x04x00x00x00Bob x00tbirthdayx00x80rx97| ^x00x00x00x00"
 ! Person: {Bob 2014-07-21 18:00:00 -0500 EST}
  24. 24. ! type Project struct { Name string `bson:"name"` ImportPath string `bson:"importPath"` } project := Project{name, path} ! ! ! project := map[string]string{"name": name, "importPath": path} ! ! ! project := bson.D{{"name", name}, {"importPath", path}} Equal After Marshaling Struct Custom Map Document Slice
  25. 25. mgo (mango) • Pure Go • Created in late 2010 
 ("Where do I put my Go data?") • Adopted by Canonical and MongoDB Inc. itself • Sponsored by MongoDB Inc. from late 2011
  26. 26. Connecting
  27. 27. • Same interface for server, replica set, or shard • Driver discovers and maintains topology • Server added/removed, failovers, response times, etc Connecting session, err := mgo.Dial("localhost") if err != nil { return err }
  28. 28. • Sessions are lightweight • Sessions are copied (settings preserved) • Single management goroutine for all copied sessions Sessions func (s *Server) handle(w http.ResponseWriter, r *http.Request) { session := s.session.Copy() defer session.Close() // ... handle request ... }
  29. 29. • Saves typing • Uses the same session over and over Convenient Access projects := session.DB("OSCON").C("projects")
  30. 30. Writing
  31. 31. type Project struct { Name string `bson:"name,omitempty"` ImportPath string `bson:"importPath,omitempty"` } Defining Our Own Type
  32. 32. var projectList = []Project{ {"gocheck", "gopkg.in/check.v1"}, {"qml", "gopkg.in/qml.v0"}, {"pipe", "gopkg.in/pipe.v2"}, {"yaml", "gopkg.in/yaml.v1"}, } ! for _, project := range projectList { err := projects.Insert(project) if err != nil { return err } } fmt.Println("Okay!") Insert Okay!
  33. 33. type M map[string]interface{} ! change := M{"$set": Project{ImportPath: "gopkg.in/ qml.v1"}} ! err = projects.Update(Project{Name: "qml"}, change) if err != nil { return err } ! fmt.Println("Done!") Update Done!
  34. 34. Querying
  35. 35. var project Project ! err = projects.Find(Project{Name: "qml"}).One(&project) if err != nil { return err } ! fmt.Printf("Project: %vn", project) Find Project: 
 {qml gopkg.in/qml.v0}
  36. 36. iter := projects.Find(nil).Iter() ! var project Project for iter.Next(&project) { fmt.Printf("Project: %vn", project) } ! return iter.Err() Iterate Project: {gocheck gopkg.in/check.v1} Project: {qml gopkg.in/qml.v0} Project: {pipe gopkg.in/pipe.v2} Project: {yaml gopkg.in/yaml.v1}
  37. 37. m := map[string]interface{}{ "name": "godep", "tags": []string{"tool", "dependency"}, "contact": bson.M{ "name": "Keith Rarick", "email": "kr@nospam.com", }, } ! err = projects.Insert(m) if err != nil { return err } fmt.Println("Okay!") Nesting Okay!
  38. 38. type Contact struct { Name string Email string } ! type Project struct { Name string Tags []string `bson:",omitempty"` Contact Contact `bson:",omitempty"` } ! err = projects.Find(Project{Name: "godep"}).One(&project) if err != nil { return err } ! pretty.Println("Project:", project) Nesting II Project: main.Project{ Name: "godep", Tags: {"tool", "dependency"}, Contact: {Name:"Keith Rarick", Email:"kr@XZY.com"}, }
  39. 39. • Compound • List indexing (think tag lists) • Geospatial • Dense or sparse • Full-text searching Indexing // Root field err = projects.EnsureIndexKey("name") ... ! // Nested field err = projects.EnsureIndexKey("author.email") ...
  40. 40. Concurrency
  41. 41. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  42. 42. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  43. 43. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  44. 44. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  45. 45. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  46. 46. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  47. 47. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent
  48. 48. func f(projects *mgo.Collection, name string, done chan error) { var project Project err := projects.Find(Project{Name: name}).One(&project) if err == nil { fmt.Printf("Project: %vn", project) } done <- err } ! done := make(chan error) ! go f(projects, "qml", done) go f(projects, "gocheck", done) ! if err = firstError(2, done); err != nil { return err } Concurrent Project: {qml gopkg.in/qml.v1} Project: {gocheck gopkg.in/ check.v1}
  49. 49. • Find 1 issued
 • Doc 1 returned • Find 2 issued
 • Doc 2 returned A Common Approach Find 1 Find 2 DB } }
  50. 50. • Find 1 issued • Find 2 issued
 • Doc 1 returned • Doc 2 returned Concurrent Queries Find 1 Find 2 DB } }
  51. 51. • Loads 200 results at a time • Loads next batch with (0.25 * 200) results left to process Concurrent Loading session.SetBatch(200) session.SetPrefetch(0.25) ! for iter.Next(&result) { ... }
  52. 52. • Each Copy uses a different connection • Closing session returns socket to the pool • defer runs at end of function Handler With Session Copy func (s *Server) handle(w http.ResponseWriter, r *http.Request) { session := s.session.Copy() defer session.Close() ! // ... handle request ... }
  53. 53. • Shares a single connection • Still quite efficient thanks to concurrent capabilities of go + mgo Handler With Single Session func (s *Server) handle(w http.ResponseWriter, r *http.Request) { session := s.session ! // ... handle request ... }
  54. 54. GridFS
  55. 55. GridFS • Not quite a file system • Really useful for local file storage • A convention, not a feature • Supported by all drivers • Fully replicated, sharded file storage
  56. 56. gridfs := session.DB("OSCON").GridFS("fs") ! file, err := gridfs.Create("cd.iso") if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started)) GridFS
  57. 57. gridfs := session.DB("OSCON").GridFS("fs") ! file, err := gridfs.Create("cd.iso") if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started)) GridFS
  58. 58. gridfs := session.DB("OSCON").GridFS("fs") ! file, err := gridfs.Create("cd.iso") if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started)) GridFS
  59. 59. gridfs := session.DB("OSCON").GridFS("fs") ! file, err := gridfs.Create("cd.iso") if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started)) GridFS
  60. 60. gridfs := session.DB("OSCON").GridFS("fs") ! file, err := gridfs.Create("cd.iso") if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started)) GridFS
  61. 61. gridfs := session.DB("OSCON").GridFS("fs") ! file, err := gridfs.Create("cd.iso") if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started)) GridFS
  62. 62. gridfs := session.DB("OSCON").GridFS("fs") ! file, err := gridfs.Create("cd.iso") if err != nil { return err } defer file.Close() ! started := time.Now() ! _, err = io.Copy(file, iso) if err != nil { return err } ! fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started)) GridFS ! Wrote 470386961 bytes in 7.0s
  63. 63. Full Featured
  64. 64. Features • Transactions (mgo/txn experiment) • Aggregation pipelines • Full-text search • Geospatial support • Hadoop
  65. 65. In Conclusion
  66. 66. Getting Started • 1. Install MongoDB • 2. go get gopkg.in/mgo.v2 • 3. Start small • 4. Build something great
  67. 67. Learning More • MongoDB Manual • Effective Go • labix.org/mgo
  68. 68. Workshop Using mgo on spf13.com
  69. 69. on spf13.com
  70. 70. • @spf13 • Author of Hugo, Cobra, Viper & More • Chief Developer Advocate for MongoDB • Gopher Thank You
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×