The document discusses using MongoDB and Go together for painless data storage. Some key points: - Go is a good language for working with MongoDB as it is fast, friendly to use, and supports concurrency well out of the box. - MongoDB is also fast and scales well. It uses a flexible document model that maps well to Go's data types. - The mgo driver makes it easy to connect to MongoDB from Go, perform CRUD operations, and work with data concurrently. Data can be serialized and deserialized to BSON format efficiently. - Features like GridFS allow storing files within MongoDB, providing replicated file storage.

1. Painless Data
Storage with
MongoDB & Go

2. • Author of Hugo, Cobra,
Viper & More
• Chief Developer
Advocate for MongoDB
• Gopher
@spf13

3. Why Go?

4. Why Another Language?
• Software is slow
• Sofware is hard to write
• Software doesn’t scale well

5. Go is Fast
• Go execution speed is close to C
• Go compile time rivals dynamic
interpretation

6. Go is Friendly
• Feels like a dynamic language in many ways
• Very small core language, easy to
remember all of it
• Single binary installation, no dependencies
• Extensive Tooling & StdLib

7. Go is Concurrent
• Concurrency is part of the language
• Any function can become a goroutine
• Goroutines run concurrently, communicate
through channels
• Select waits for communication on any of a
set of channels

8. MongoDB

9. Why Another Database?
• Databases are slow
• Relational structures don’t fit well
with modern programming
(ORMs)
• Databases don’t scale well

10. MongoDB is Fast
• Written in C++
• Extensive use of memory-mapped files
i.e. read-through write-through memory caching.
• Runs nearly everywhere
• Data serialized as BSON (fast parsing)
• Full support for primary & secondary indexes
• Document model = less work

11. MongoDB is Friendly
• Ad Hoc queries
• Real time aggregation
• Rich query capabilities
• Traditionally consistent
• Geospatial features
• Support for most programming languages
• Flexible schema

12. MongoDB is “Web Scale”
• Built in sharding support distributes data
across many nodes
• MongoS intelligently routes to the correct
nodes
• Aggregation done in parallel across nodes

13. Document Database
• Not for .PDF & .DOC files
• A document is essentially an associative array
• Document == JSON object
• Document == PHP Array
• Document == Python Dict
• Document == Ruby Hash
• etc

14. Data Serialization
• Applications need persistant data
• The process of translating data structures
into a format that can be stored
• Ideal format accessible from many
languages

15. BSON
• Inspired by JSON
• Cross language binary serialization format
• Optimized for scanning
• Support for richer types

16. MongoDB &
Go

17. Go’s Data Types
• Go uses strict & static typing
• 2 Types are similar to a BSON document
• Struct
• Map

18. bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Printf("Data: %qn", data)
!
var person Person
err = bson.Unmarshal(data, &person)
if err != nil {
return err
}
fmt.Printf("Person: %vn", person)
Serializing with BSON

19. bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Printf("Data: %qn", data)
!
var person Person
err = bson.Unmarshal(data, &person)
if err != nil {
return err
}
fmt.Printf("Person: %vn", person)
Serializing with BSON

20. bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Printf("Data: %qn", data)
!
var person Person
err = bson.Unmarshal(data, &person)
if err != nil {
return err
}
fmt.Printf("Person: %vn", person)
Serializing with BSON

21. bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Printf("Data: %qn", data)
!
var person Person
err = bson.Unmarshal(data, &person)
if err != nil {
return err
}
fmt.Printf("Person: %vn", person)
Serializing with BSON

22. bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Printf("Data: %qn", data)
!
var person Person
err = bson.Unmarshal(data, &person)
if err != nil {
return err
}
fmt.Printf("Person: %vn", person)
Serializing with BSON

23. bob := &Person{
Name: "Bob",
Birthday: time.Now(),
}
!
data, err := bson.Marshal(bob)
if err != nil {
return err
}
fmt.Printf("Data: %qn", data)
!
var person Person
err = bson.Unmarshal(data, &person)
if err != nil {
return err
}
fmt.Printf("Person: %vn", person)
Serializing with BSON
Data: "%x00x00x00x02name
x00x04x00x00x00Bob
x00tbirthdayx00x80rx97|
^x00x00x00x00"
!
Person: {Bob 2014-07-21
18:00:00 -0500 EST}

24. !
type Project struct {
Name string `bson:"name"`
ImportPath string `bson:"importPath"`
}
project := Project{name, path}
!
!
!
project := map[string]string{"name": name, "importPath": path}
!
!
!
project := bson.D{{"name", name}, {"importPath", path}}
Equal After Marshaling
Struct
Custom Map
Document Slice

25. mgo (mango)
• Pure Go
• Created in late 2010
("Where do I put my Go data?")
• Adopted by Canonical and MongoDB Inc.
itself
• Sponsored by MongoDB Inc. from late 2011

26. Connecting

27. • Same interface for server, replica set, or shard
• Driver discovers and maintains topology
• Server added/removed, failovers, response times, etc
Connecting
session, err := mgo.Dial("localhost")
if err != nil {
return err
}

28. • Sessions are lightweight
• Sessions are copied (settings preserved)
• Single management goroutine for all copied sessions
Sessions
func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
session := s.session.Copy()
defer session.Close()
// ... handle request ...
}

29. • Saves typing
• Uses the same session over and over
Convenient Access
projects := session.DB("OSCON").C("projects")

30. Writing

31. type Project struct {
Name string `bson:"name,omitempty"`
ImportPath string `bson:"importPath,omitempty"`
}
Defining Our Own Type

32. var projectList = []Project{
{"gocheck", "gopkg.in/check.v1"},
{"qml", "gopkg.in/qml.v0"},
{"pipe", "gopkg.in/pipe.v2"},
{"yaml", "gopkg.in/yaml.v1"},
}
!
for _, project := range projectList {
err := projects.Insert(project)
if err != nil {
return err
}
}
fmt.Println("Okay!")
Insert
Okay!

33. type M map[string]interface{}
!
change := M{"$set": Project{ImportPath: "gopkg.in/
qml.v1"}}
!
err = projects.Update(Project{Name: "qml"}, change)
if err != nil {
return err
}
!
fmt.Println("Done!")
Update
Done!

34. Querying

35. var project Project
!
err = projects.Find(Project{Name: "qml"}).One(&project)
if err != nil {
return err
}
!
fmt.Printf("Project: %vn", project)
Find
Project:
{qml gopkg.in/qml.v0}

36. iter := projects.Find(nil).Iter()
!
var project Project
for iter.Next(&project) {
fmt.Printf("Project: %vn", project)
}
!
return iter.Err()
Iterate
Project: {gocheck gopkg.in/check.v1}
Project: {qml gopkg.in/qml.v0}
Project: {pipe gopkg.in/pipe.v2}
Project: {yaml gopkg.in/yaml.v1}

37. m := map[string]interface{}{
"name": "godep",
"tags": []string{"tool", "dependency"},
"contact": bson.M{
"name": "Keith Rarick",
"email": "kr@nospam.com",
},
}
!
err = projects.Insert(m)
if err != nil {
return err
}
fmt.Println("Okay!")
Nesting
Okay!

38. type Contact struct {
Name string
Email string
}
!
type Project struct {
Name string
Tags []string `bson:",omitempty"`
Contact Contact `bson:",omitempty"`
}
!
err = projects.Find(Project{Name: "godep"}).One(&project)
if err != nil {
return err
}
!
pretty.Println("Project:", project)
Nesting II
Project: main.Project{
Name: "godep",
Tags: {"tool",
"dependency"},
Contact: {Name:"Keith
Rarick", Email:"kr@XZY.com"},
}

39. • Compound
• List indexing (think tag lists)
• Geospatial
• Dense or sparse
• Full-text searching
Indexing
// Root field
err = projects.EnsureIndexKey("name")
...
!
// Nested field
err = projects.EnsureIndexKey("author.email")
...

40. Concurrency

41. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent

42. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent

43. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent

44. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent

45. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent

46. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent

47. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent

48. func f(projects *mgo.Collection, name string, done chan error) {
var project Project
err := projects.Find(Project{Name: name}).One(&project)
if err == nil {
fmt.Printf("Project: %vn", project)
}
done <- err
}
!
done := make(chan error)
!
go f(projects, "qml", done)
go f(projects, "gocheck", done)
!
if err = firstError(2, done); err != nil {
return err
}
Concurrent
Project: {qml gopkg.in/qml.v1}
Project: {gocheck gopkg.in/
check.v1}

49. • Find 1 issued
• Doc 1 returned
• Find 2 issued
• Doc 2 returned
A Common Approach
Find 1 Find 2 DB
}
}

50. • Find 1 issued
• Find 2 issued
• Doc 1 returned
• Doc 2 returned
Concurrent Queries
Find 1 Find 2 DB
}
}

51. • Loads 200 results at a time
• Loads next batch with (0.25 * 200) results left to process
Concurrent Loading
session.SetBatch(200)
session.SetPrefetch(0.25)
!
for iter.Next(&result) {
...
}

52. • Each Copy uses a different connection
• Closing session returns socket to the pool
• defer runs at end of function
Handler With Session Copy
func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
session := s.session.Copy()
defer session.Close()
!
// ... handle request ...
}

53. • Shares a single connection
• Still quite efficient thanks to concurrent capabilities of go + mgo
Handler With Single Session
func (s *Server) handle(w http.ResponseWriter, r *http.Request) {
session := s.session
!
// ... handle request ...
}

54. GridFS

55. GridFS
• Not quite a file system
• Really useful for local file storage
• A convention, not a feature
• Supported by all drivers
• Fully replicated, sharded file storage

56. gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file.Close()
!
started := time.Now()
!
_, err = io.Copy(file, iso)
if err != nil {
return err
}
!
fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started))
GridFS

57. gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file.Close()
!
started := time.Now()
!
_, err = io.Copy(file, iso)
if err != nil {
return err
}
!
fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started))
GridFS

58. gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file.Close()
!
started := time.Now()
!
_, err = io.Copy(file, iso)
if err != nil {
return err
}
!
fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started))
GridFS

59. gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file.Close()
!
started := time.Now()
!
_, err = io.Copy(file, iso)
if err != nil {
return err
}
!
fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started))
GridFS

60. gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file.Close()
!
started := time.Now()
!
_, err = io.Copy(file, iso)
if err != nil {
return err
}
!
fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started))
GridFS

61. gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file.Close()
!
started := time.Now()
!
_, err = io.Copy(file, iso)
if err != nil {
return err
}
!
fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started))
GridFS

62. gridfs := session.DB("OSCON").GridFS("fs")
!
file, err := gridfs.Create("cd.iso")
if err != nil {
return err
}
defer file.Close()
!
started := time.Now()
!
_, err = io.Copy(file, iso)
if err != nil {
return err
}
!
fmt.Printf("Wrote %d bytes in %sn", file.Size(), time.Since(started))
GridFS
!
Wrote 470386961 bytes in 7.0s

63. Full Featured

64. Features
• Transactions (mgo/txn experiment)
• Aggregation pipelines
• Full-text search
• Geospatial support
• Hadoop

65. In Conclusion

66. Getting Started
• 1. Install MongoDB
• 2. go get gopkg.in/mgo.v2
• 3. Start small
• 4. Build something great

67. Learning More
• MongoDB Manual
• Effective Go
• labix.org/mgo

68. Workshop Using mgo on spf13.com

69. on spf13.com

70. • @spf13
• Author of Hugo, Cobra,
Viper & More
• Chief Developer
Advocate for MongoDB
• Gopher
Thank You