Mongo db - How we use Go and MongoDB by Sam Helman
Upcoming SlideShare
Loading in...5
×
 

Mongo db - How we use Go and MongoDB by Sam Helman

on

  • 3,109 views

In this talk, we'll hear from Sam Helman, Software Engineer at 10gen, on how 10gen is integrating Go into our new and existing cloud tools. Some of the tools leveraging Go include the backup ...

In this talk, we'll hear from Sam Helman, Software Engineer at 10gen, on how 10gen is integrating Go into our new and existing cloud tools. Some of the tools leveraging Go include the backup capabilities in MongoDB Management Service and a continuos integration tool. This was an opportunity to experiment with new technologies and create a better product for end users.

Programming the in Go and the mgo driver has been extremely satisfying. Between the lightweight syntax, the first-class concurrency and the well documented, idiomatic libraries such as mgo, Go is a great choice for writing anything from small scripts to large distributed applications. In this talk, Sam will go through how the team has integrated Go and why Go and MongoDB are a great match for cloud services.

Statistics

Views

Total Views
3,109
Views on SlideShare
1,995
Embed Views
1,114

Actions

Likes
3
Downloads
11
Comments
0

12 Embeds 1,114

http://g33ktalk.com 979
http://www.hakkalabs.co 82
https://hakka.herokuapp.com 29
https://hakkalabs.herokuapp.com 9
http://www.feedspot.com 4
http://cloud.feedly.com 4
http://www.newsblur.com 2
http://plus.url.google.com 1
http://digg.com 1
http://localhost 1
http://newsblur.com 1
http://feeds.feedburner.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • MongoDB Management Service (MMS) is a cloud-based suite of services for managing MongoDB deployments. MMS provides monitoring and backup capabilities, helping users optimize clusters, automate tasks and mitigate operational risk. For customers that prefer to manage MongoDB on their own infrastructure, 10gen also offers MMS MonitoringOn-Prem as part of MongoDB Enterprise (backup features coming soon).
  • Install lightweight agent written in Go so it’s compiled nativelyPerform an initial syncOplog tails going forward – performance impact similar to adding another secondaryEncrypted in motion
  • 6 hour snaps for 48Dailys kept for 1 week1 snap kept 1 monthMonthlys kept for a yearDatacenters on east coast
  • Get the discount code from Meghan or Rhea

Mongo db - How we use Go and MongoDB by Sam Helman Mongo db - How we use Go and MongoDB by Sam Helman Presentation Transcript

  • Go at 10gen Sam Helman, Software Engineer
  • What is Go? “Go is an open source programming environment that makes it easy to build simple, reliable, and efficient software” - Google
  • What is Go? - A programming language (Golang)
  • Language features - Code organized into packages - Distinction b/t pointers and values - Structs instead of objects
  • Language features - First-class functions - Multiple return values - Concurrency via goroutines
  • Development features - Compiles quickly - Easy to learn, small - Strong standard libraries - http - subprocess - time - Nice plugins for editors - sublime text, vim, emacs
  • What else is Go? - A collection of development tools
  • mgo (mango) - Go Driver - labix.org/v2/mgo - Uses standard API for MongoDB drivers - A little syntactic sugar (UpdateId, Upsert) - Connection pooling, cluster discovery, read preference, auth, GridFS
  • mgo's BSON Implementation - labix.org/v2/mgo/bson - Fully to spec - Supports marshalling, unmarshalling, ordered, unordered
  • Built-in ODM - Uses struct tags to define db->struct mappings - Tags aid mgo in unmarshalling documents
  • How we came to use Go
  • MMS Backup - Hosted backups for MongoDB - Lots of flexibility o Periodic snapshots o Point-in-time restore o Support for sharded clusters
  • How it works (in a nutshell) - Backup agent o Runs on customer’s machine o Syncs data, then tails oplog - Backend storage ("Mothership") o Collection of oplog operations (sent from agent) o Deduped block storage of periodic snapshots
  • Backup Agent - Originally written in Java - However, needs to live on customer's machines - Needs to be easily deployable - Java relies on customer's JVM, installed libraries, etc.
  • Benefits of Go - Go is compiled and statically linked o Doesn't rely on anything on target machine o Large binaries, but no dependencies - Can be cross-compiled easily o http://www.laher.net.nz/goxc/ o GOOS, GOARCH
  • Mongo Continuous Integration (MCI) - Our home-brewed CI system - Used to test MongoDB cross-platform - Built for a number of considerations o Parallelization o Flexibility o Ease of tuning o Speed - (Will be open-source soon)
  • How it works - One master machine o Stores config info o Polls github for changes o Divides steps (compile, test) among workers o Sends agent process over to workers - Numerous worker machines o Run steps (via agent), report back to master
  • MCI Agent - Independent of master process - Runs on worker machines - Needs to run on all supported platforms o With minimal maintenance - Want to minimize worker machine setup o Many are dynamically spun up o As few dependencies as possible
  • Benefits of Go - No dependencies o Worker machines need only the agent binary - Cross-platform o Build on master machine, scp to workers - Single binary to copy to workers - Go dynamicallycompiles quickly o Agent code changes a lot o Rebuilt
  • MongoDB Management Service: Backup Overview
  • 21 What is MongoDB Management Service (MMS)?
  • How It Works
  • 23 Overview
  • 24 • From the initial sync, we rebuild your data in our datacenters and take a snapshot • We take snapshots every 6 hours • Oplog is stored for 48 hours Under the Hood
  • 25 • Balancer paused every 6 hours • A no-op token is inserted across all shards, mongoses and config servers • Oplog applied to replica sets until point in which token was inserted • Provides a consistent state of database across shards Sharded Clusters
  • Key Benefits
  • 27 • Oplog stored for 48 hours • Restore your replica set to any point-in-time in the last 48 hours by creating a custom snapshot Point-In-Time
  • 28 • Pull from custom URL • Push via SCP Easy to Restore
  • 29 • Confidence in your restore process • Build development, QA, analytics environments without impacting production Unlimited, Free Restores
  • 30 • Created by the engineers that build MongoDB • No need to write or maintain custom backup scripts Fully Managed
  • Getting Started
  • 32 1. Create an account at mms.10gen.com 2. Install MMS Monitoring Agent on your deployment 3. Register at mms.10gen.com/backup 4. Install MMS Backup Agent on your deployment 5. Start initial sync 6. Rest easy! Getting Started
  • Free Month – Register Using Coupon Code “MUG_NYC”