MongoDB training for java software engineers

© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
MongoDBfor Java Software Engineers
Copyrights © Moshe Kaplan
moshe.kaplan@brightaqua.com

MongoDB
for Java Software Engineers
Moshe Kaplan
Scale Hacker
http://top-performance.blogspot.com
http://blogs.microsoft.co.il/vprnd

© All rights reserved: Moshe Kaplan© All rights reserved: Moshe Kaplan MongoDB for Java Software Engineers
About Me: It’s all About
3
Scale

NOSQL. ANSWER A NEED
Introduction
4

5

The Consumer Revolution
6
http://topyaps.com/wp-content/uploads/2013/03/You-are-the-
product.-You-feeling-something.jpg

At the fraction of the cost…
7

8
http://lifehacker.com/5697167/if-youre-not-paying-for-it-
youre-the-product

Transportation
9

Moovit
10

The Medical Market Opportunities
11

MediSafe
12

13

Askem
14

Major Enablers:
Mobile, Cloud and IT Commoditization
15

The Prime Suspect
16

17
Assumptions…

Where did it Fail?
Get an Answer, Fast and Cheap

Where did it Fail?
I Just Want “Class Persistency
Storage” and Changing Schema on
Demand

Where did it Fail?
Be Always Available, Even w/ an Old
Answer

Where did it Fail?
Get Me Fast and Good Enough
Answer

Where did it Fail?
Data is Too Big, and Storage is $$$
But CPU and Network are Even More
http://www.powerbyte.com/Isilon.html

Software Providers
23

It is all great, but…
I Need to Meet Compliance
http://www.vision7.com/app_system/lib/image/content/PCI_compliance.jpg

I Need a Vendor

I Need Reporting
http://www.novell.com/communities/node/5851/get-ready-sentinel-61

I Need Transactions
http://www.novell.com/communities/node/5851/get-ready-sentinel-61

We Need Training for the Data Analysts
db.article.aggregate(
{ $group : {
_id : "$author",
docsPerAuthor : { $sum : 1 },
viewsPerAuthor : { $sum : "$pageViews" }
}}
);
< SUM(pageViews)
< SUM(1) = N
< GROUP BY author

NOSQL MARKET
Introduction
29

When Should I Choose NoSQL?
• Eventually Consistent
• Document Store
• Key Value
30
http://guyharrison.squarespace.com/blog/tag/nosq

Key Value Store
• insert
• get
• multiget
• remove
• truncate
31
<Key, Value>
http://wiki.apache.org/cassandra/API

Redis
• Very simple protocol (SMTP like)
• Amazing Performance (60Kqps ops on 1 CPU machine)
• Persistency to disk
• Very little security

Column Family Stores:
Key Value Store (with benefits)
• insert
• get
• multiget
• remove
• truncate
33
http://wiki.apache.org/cassandra/API

Cassandra
• Simple protocol
• Very Good Performance
• You have indexes (but limited)
• Data Model is a pain
• You need to design you data for queries:
“Table per Query”

Document Databases
var mydoc = {
_id: ObjectId("5099803df3f4948bd2f98391"),
name: { first: "Alan", last: "Turing" },
birth: new Date('Jun 23, 1912'),
death: new Date('Jun 07, 1954'),
contribs: [
"Turing machine",
"Turing test",
"Turingery"
],
views : NumberLong(1250000)
}
35

Database for Software Engineers
Class
Subclass
Document
Subdocument

37
MapReduce
http://blogs.microsoft.co.il/blogs/vprnd

HELLO. MY NAME IS MONGODB
Introduction
38

39
#5 Most Popular DB Engine
http://db-engines.com/en/ranking

Who is Using mongoDB?

Who is Behind mongoDB

Why MongoDB?
What? Why?
JSON End to End
No Schema “No DBA”, Just Serialize
Write 10K Inserts/sec on virtual machine
Read Similar to MySQL
HA 10 min to setup a cluster
Sharding Out of the Box
GeoData Great for that
No Schema None: no downtime to create new columns
Buzz Trend is with NoSQL
42

What mongoDB is Made of?
43
http://www.10gen.com/products/mongodb

Installation: Give Yourself 5min
• Add to /etc/yum.repos.d/10gen.repo
• [10gen]
• name=10gen Repository
• baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64
• gpgcheck=0
• enabled=1
• yum –y install mongo-10gen mongo-10gen-server
• The Packages:
• mongo-10gen: tools
• mongo-10gen-server: mongod and mongos

The Ubuntu Way
• sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv
7F0CEB10
echo "deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.0
multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.0.list
sudo apt-get -y update
sudo apt-get install -y mongodb-org

Installation w/ Authentication
• /etc/mongod.conf
• > mongo
• use admin
db.createUser(
{
user: "siteUserAdmin",
pwd: “Pss0rdxxx",
roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
} )

Mastering a New Query
Language

Connect to the Database
• Connect:
• > mongo
• Show current database:
• >> db
• Show Databases
• >> show databases;
• Show Collections
• >> show collections; or show tables;

Databases Manipulation: Create & Drop
• Change Database:
• >> use <database>
• Create Database
• Just switch and create an object…
• Delete Database
• > use mydb;
• > db.dropDatabase();

Collections Manipulation
• Create Collcation
>db.createCollection(collectionName)
• Delete Collection
> db.collectionName.drop()
Or just insert to it

SELECT: No SQL, just ORM…
• Select All
• db.things.find()
• WHERE
• db.posts.find({“comments.email” : ”b@c.com”})
• Pattern Matching
• db.posts.find( {“title” : /mongo/i} )
• Sort
• db.posts.find().sort({email : 1, date : -1});
• Limit
• db.posts.find().limit(3)

NoSQL and Data Modeling
What is the Difference

Same Terminology
• Database  Database
• Table  Collection
• Row  Document

A Blog Case Study in MySQL
http://www.slideshare.net/nateabele/building-apps-with-mongodb

as a SW Engineer would like it to be…

Migration from RDBMS to NoSQL
How to do that?

Data Migration
• Map the table structure
• Export the data and Import It
• Add Indexes
58
http://igcse-geography-lancaster.wikispaces.com/1.2+MIGRATION

Selected Migration Tool
59

Usage Details> Install ruby
> gem install mongify
… Modify the code to your needs
… Create configuration files
> mongify translation db.config >
translation.rb
> mongify process db.config translation.rb
60

Date Functions
• Year(), Month()… function included
• … buy only in the JavaScript engine
• Solution: New fields!
• [original field]
• [original field]_[year part]
• [original field]_[month part]
• [original field]_[day part]
• [original field]_[hour part]
61

NO SCHEMA IS A GOOD THING BUT…
Schemaless
62

Default Values
• No Schema
• No Default Values
• App Challenge
• Timestamps…
No single source of truth
63

Casting and Type Safety
• No Schema
• No …
• App Challenge
64

Auto Numbers
• Start using _id
{
"_id" : 0,
"health" : 1,
"stateStr" : "PRIMARY",
"uptime" : 59917
}
• Counter tables
• Dedicated database
• 1:1 Mapping
• Counter++ using findAndModify
65

ODM Solution
66

Data Analysts
67
http://www.designersplayground.com/pr/internet-meme-list/data-analyst-2/

Data Analysts
68
Pentaho
RockMongoMongoVUE RoboMongo
MongoChef

No Joins
• Do in the application
• Leverage the power of NoSQL
69

Limited Resultset
70
• 16MB document size
• GridFS

Bottom Line
• Powerful tool
• Embrace the Challenge
• Schema-less limitations: counters, data types
• Tools for Data Scientists
• Data design
71

Specific fields
Select All
db.users.find(
{ },
{ user_id: 1, status: 1, _id: 0 }
)
1: Show; 0: don’t show

WHERE
• != “A” { $ne: "A" }
• > 25 { $gt: 25 }
• > 25 AND <= 50 { $gt: 25, $lte: 50 }
• Like ‘bc%’ /^bc/
• < 25 OR >= 50 { $or : [ { $lt: 25 }, { $gte : 50 } ] }

Join
• Wrong Place…
• Or Map Reduce

80
 db.article.aggregate(
 { $group : {
 _id : { author : "$author“, name : “$name” },
 docsPerAuthor : { $sum : 1 },
 viewsPerAuthor : { $sum : "$pageViews" }
 }}
 );
GROUP BY
< GROUP BY author, name
< SUM(pageViews)
< SUM(1) = N

81
db.Movie.aggregate([
{$match:
{SeriesType : "F", MovieID : {$in : arrMovies}}
},
{$project:
{MovieID: "$MovieID", SeriesType: "$SeriesType",
Genres: "$Genres"}
},
{$unwind : "$Genres" },
{$group : { _id : "$Genres" , count : { $sum : 1 } } },
{$sort : { count: -1 }}
GROUP BY
WHERE
Keep some fields
Genres is an array
Counting and sorting

Aggregation Framework Operators
Operator Description
$project Adding/Removing fields
$match WHERE
$redact Changes document based on Doc content/structure
$limit First N documents
$skip Skips N docs
$unwind Turns array into a multiple documents
$group Group
$sort Sort
$geoNear Geo spatial
$out Write Output to collection

83
db.posts.update(
{“comments.email”: ”b@c.com”},
{$set : {“comments.email”: ”d@c.com”}}
}
SET age = age + 3
• db.users.update(
• { status: "A" } ,
• { $inc: { age: 3 } },
• { multi: true }
• )
UPDATE

84
j = { name : "mongo" }
k = { x : 3 }
db.things.insert( j )
db.things.insert( k )
INSERT

85
db.users.remove(
{ status: "D" }
)
DELETE

86
Every operation on a document is atomic
Two Phase Commit implementation is up to
you
Atomic Transactions: Single Row

87
 Multiple documents at once
db.foo.update(
{ status : "A" , $isolated : 1 },
{ $inc : { count : 1 } },
{ multi: true }
)
 Disclaimers:
• Sharding is not supported
• Not all or nothing (no roll back on failure)
Atomic Transactions: $isolated

88
 t = db.transactions.findAndModify({
query: {
state: "initial“
},
update: {
$set: {
state: "pending"
},
$currentDate: { lastModified: true }
},
new: true
})
Atomic Transactions: findAndModify

89
 If it is about complex transactions.
 Simplify the case.
 or Consider keeping w/ RDBMS
Atomic Transactions: Bottom Line

90
 Failure and order:
• db.collection.initializeOrderedBulkOp()
• db.collection.initializeUnorderedBulkOp()
 1000 ops/bulk:
var bulk = db.items.initializeUnorderedBulkOp();
bulk.insert( { item: "abc123", defaultQty: 100, status: "A", points: 100 } );
bulk.insert( { item: "ijk123", defaultQty: 200, status: "A", points: 200 } );
bulk.insert( { item: "mop123", defaultQty: 0, status: "P", points: 0 } );
bulk.execute();
Bulk Operations

91
 Create a new project
 Get the Maven configuration for MongoDB Java Driver
• http://mongodb.github.io/mongo-java-driver/
Project Setup

92
List l = new ArrayList();
/**** Insert ****/
// create a document to store key and value
for (int i = 1; i < 1000000; ++i) {
Document document = new Document()
.append("name", "Moshe Kaplan")
.append("age", 36 + i)
.append("createdDate", new Date());
l.add(document);
}
table.insertMany(l);
Bulk Ops in Java

93
List<String> continentList = Arrays.asList(new String[]{"Africa", "Europe", "Asia"});
DBObject match = new BasicDBObject("$match", new BasicDBObject("continent.name", new BasicDBObject("$in",
continentList)));
DBObject projectFields = new BasicDBObject("continent.name", 1);
projectFields.put("area", 1);
projectFields.put("_id", 0);
DBObject project = new BasicDBObject("$project", projectFields );
DBObject groupFields = new BasicDBObject( "_id", "$continent.name");
groupFields.put("average", new BasicDBObject( "$avg", "$area"));
DBObject group = new BasicDBObject("$group", groupFields);
List agList = new ArrayList();
agList.add(match);
agList.add(project);
agList.add(group);
MongoCursor<Document> cursor = countries.aggregate(agList).iterator();
while (cursor.hasNext()) {
System.out.println(cursor.next());
}
Aggregation Framework in Java

Performance Tuning
Make a Change

MONGODB TUNING
95

journalCommitInterval = 300:
Write to disk: 2ms <= t <= 300ms
Default 100ms, increase to 300ms to save resources
Disk
The Journal
96
Memory
Journal Data
1 2

RAM Optimization:
dataSize + indexSize < RAM
97
OS
Data Index
Journal

PROFILING AND SLOW LOG
98

Profiling Configuration
• Enable:
• mongod --profile=1 --slowms=15
• db.setProfilingLevel([level] , [time])
• How much:
• 0 (none)  1 (slow queries only)  2 (all)
• 100ms: default
• Where:
• system.profile collection @ local db
99

Profiling Results Analysis
• Last 5 >1ms: show profile
• w/o commands:
db.system.profile.find( { op: { $ne : 'command' } } ).pretty()
• Specific database:
db.system.profile.find( { ns : 'mydb.test' } ).pretty()
• Slower than:
db.system.profile.find( { millis : { $gt : 5 } } ).pretty()
• Between dates:
db.system.profile.find({ts : {
$gt : new ISODate("2012-12-09T03:00:00Z") ,
$lt : new ISODate("2012-12-09T03:40:00Z")
}}).pretty()
100

Explain
> db.courses.find().explain();
{ "cursor" : "BasicCursor",
"isMultiKey" : false,
"n" : 11, “nscannedObjects" : 11, "nscanned" : 11,
"nscannedObjectsAllPlans" : 11, "nscannedAllPlans" : 11,
"scanAndOrder" : false, "indexOnly" : false,
"nYields" : 0,
"nChunkSkips" : 0,
"millis" : 0,
"indexBounds" : {},
"server" : "primary.domain.com:27017"
}
101

INDEXES
102

Index Management
• Regular Index
• db.users.createIndex( { user_id: 1 } )
• db.users.ensureIndex( { user_id: 1 } )
• Multiple + DESC Index
• db.users.ensureIndex( { user_id: 1, age: -1 } )
• Sub Document Index
• db.users.ensureIndex( { address.zipcode: 1 } )
• Unique Index
• db.users.ensureIndex( { address.zipcode: 1 } , { unique : true } )
• List Indexes
• db.users.getIndexes()
• Drop Indexes
• db.users.dropIndex(“indexName”)
103

Known Index Issues
• Bound filter should be the last (in the index as well).
• BitMap Indexes not really working
• You should design your indexes carefully
104

Dex: The Index Analyzer
• Installation:
• sudo apt-get -y install python-pip
sudo pip install dex
• Running:
• dex [mongodb_uri] (-f <logfile_path> | -p) [<options>]
• dex -w -p -n "testdb.*" mongodb://127.0.0.1/testdb -f
/var/log/mongodb/mongod.log

mtools: Visualize and Analyze Logs

Capped Collections
• Fixed size collections
• Circular buffers like
• High throughput operations
• Order guarantee
db.createCollection("mycoll", {capped: true, size:100000})
db.cappedCollection.find().sort( { $natural: -1 } )
• Case studies:
• Logs
• Cache

TTL
• Remove Old Data Automatically
• db.log_events.createIndex(
{ "createdAt": 1 }, { expireAfterSeconds: 3600 }
)
• db.log_events.insert( {
"expireAt": new Date('July 22, 2013 14:00:00'),
"logEvent": 2,
"logMessage": "Success!“
} )

ENVIRONMENT TUNING
109

TTL
• # For SSD only
• blockdev --setra 16 /dev/sdb
• blockdev --setra 16 /dev/dm-2
• # For all cluser mongod & mongos
• for i in /sys/kernel/mm/*transparent_hugepage/enabled;
do echo never > $i; done
• for i in /sys/kernel/mm/*transparent_hugepage/defrag;
do echo never > $i; done

STATS &
SCHEMA DESIGN
111

Sparse Matrix? I don’t Think so
• mongostat
• > db.stats();
• > db.collectionname.stats();
• Fragmentation if storageSize/size > 2
• db.collectionanme.runCommand(“compact”)
• Padding (wrong design) if paddingFactor > 2
112

High Availability
Going Real Time

(Do Not) Master/Slave

• In mongo.conf
• # Replication Options
• replSet=myReplSet
• > rs.initiate()
• > rs.conf()
• > rs.add(“host:port")
• rs.reconfig()
Replication Set
115

• rs.addArb(“host:port")
• Also:
• Low Priority
• Hidden
• (Weighted) Voting
Arbiter
116

Show Status: rs.status();
• {"set" : “myReplSet", "date" : ISODate("2013-02-05T10:23:28Z"),
• "myState" : 1,
• "members" : [
• {
• "_id" : 0, "name"
: "primary.example.com:27017",
• "health" : 1, "state" :
1,
• "stateStr" : "PRIMARY",
"uptime" : 164545,
• "optime" : Timestamp(1359901753000, 1),
• "optimeDate" : ISODate("2013-02-
03T14:29:13Z"), "self" : true
• },
• {
• "_id" : 1, "name"

Replica Set Recovery
• Create a new mongod
• Either install a plain vanilla
• Or duplicate existing mongod (better)
• Connect to the system
• Use the previous machine IP
• Or change configuration to remove old and add new

Read Preferences
• primary
• primaryPreferred
• secondary
• secondaryPreferred
• Nearest
mongo.setReadPreference(ReadPreference.nearest());

Write Concern: Unacknowledged

Write Concern: Acknowledged
mongo.setWriteConcern(WriteConcern.NORMAL);

Write Concern: Journaled

Write Concern: Replica Aacknowledged

Sharding and Scale out:
Make a big Change
Map Reduce and Aggregation

Secondary Read Enabling

The Strategy : Sharding

MongoDB Implementation

Step 1: Create a Config ReplicaSet
• mkdir /data/configdb
• mongod --configsvr --dbpath /data/configdb --port 27019

Step 2: Install Mongos
• mongos --configdb config01:27019, config02:27019, config03:27019

Step 3: Add Shards
• Connect a mongos
• Add Shard
• sh.addShard( "rs1/mongodb0.example.net:27017" )
• sh.addShard( "mongodb0.example.net:27017" )

Step 4: Enable Sharding
• sh.enableSharding("<database>")

Step 5: Sharding Colleciton
• sh.shardCollection("<database>.<collection>", shard-key-pattern)
• sh.shardCollection("records.people", { "zipcode": 1, "name": 1 } )
• Keys:
• High Cardinality to enable split
• Use common query field
• Use Compound indexes for sharding

BACKUP AND MONITORING
133

First Option – Single Server
Logical Backup Physical Backup
Method mongodump Point in time snapshot (using LVM tools) or disk image/copy
(using AWS or Azure “external” tools)
Pros Low costs Low costs
Cons • Downtime: Long;
• Duration: Long (slow backup since logical data needs to be
extracted);
• Performance impact: High (slows the disks and may stuck the
machine on heavy used machines);
• Data consistency: Intact;
• Differential: Supported;
• Sharding: Supported;
• Downtime: OS and/or infrastructure depended;
• Duration: Short (faster backup since only data blocks are
copied);
• Performance impact: Unknown (depends on OS and/or
infrastructure);
• Data consistency: Unknown state;
• Differential: Infrastructure depended;
• Sharding: Unsupported;
134
Sharding: is a type of database partitioning that separates very large databases
the into smaller, faster, more easily managed parts called data shards. The word
shard means a small part of a whole..

SECOND Option – REPLICA SET
Logical Backup Physical Backup
Method mongodump Stop slave and copy its disk
Pros • Downtime: None (backup is performed using Slave server –
Master server is always up);
• Duration: Not significant (backup is performed using Slave
server);
• Performance impact: None (backup is performed using Slave
server – Master server is not impacted);
• Data consistency: Intact;
• Differential: Supported;
• Sharding: Supported;
• Downtime: None
• Duration: Not significant
Cons Very high costs – requires two additional servers. A slave
server of the same type and size as the master server; and a
small arbiter server (used as a secondary verification for
Master server availability tests and “voting”).
• Costs: Requires a dedicated server per replica set
135

THIRD OPTION - MongoDB MMS
• Part of the MongoDB Enterprise Edition or as a Cloud Service
• The Cloud Service offer
• $50/month/node
• $2.5/GB/Month backup.
• A valid go to market way of MongoDB
for upsale
• MMS Features
• Point in time recovery
• Daily snapshots
• Detailed monitoring
• Alerts
136

How to Enable Incremental Backup
• In Backup
• Use the --oplog flag when doing mongodump
• Dump each hour the local.oplog collection
• In recovery
• mongorestore --oplogReplay
• applyOps to implement hourly dump
137

mongostat

mongotop

db.serverStatus()

db.stats() and db.collection.stats()

rs.status()

STORAGE ENGINES
143

MMAPv1

MongoDB 3.0 and WiredTiger
• MongoDB version 3.0 supports new storage engine
(WiredTiger):
• Disk Compression
• Heavy write
• Document level locking
• File per collection
• Server wide selection:
• config.yaml
• launch w/ --storageEngine = wiredTiger
145

MongoDB Pluggable Architecture

Engines Comparison

YAML Based Configuration
storage:
dbPath: "/var/lib/mongodbwt"
directoryPerDB: true
engine: "wiredTiger"
wiredTiger:
engineConfig:
cacheSizeGB: 16
journalCompressor: zlib
directoryForIndexes: true
collectionConfig:
blockCompressor: zlib
indexConfig:
prefixCompression: true
systemLog:
destination: file
path: "/var/log/mongodb/mongod.log"
logAppend: true
timeStampFormat: iso8601-local
processManagement:
fork: true
pidFilePath: "/var/run/mongodb.pid"
#security:
# keyFile: "/etc/mongo.key"
# authorization: "enabled"
replication:
replSetName: "arp0"

SECURITY
149

Providing Permissions
• use admin
db.createUser( {
user: "siteUserAdmin", pwd: "password",
roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
} )
• use records
db.createUser( {
user: "recordsUserAdmin", pwd: "password",
roles: [ { role: "userAdmin", db: "records" } ]
} )

Roles
Read
readWrite
dbAdmin
dbOwner
userAdmin
clusterAdmin, clusterManager, …
backup, restore
readAnyDatabase, readWriteAnyDatabase
root

Granular Actions
use admin
db.createRole(
role: "manageOpRole",
privileges: [
{ resource: { cluster: true }, actions: [ "killop", "inprog" ] },
{ resource: { db: "", collection: "" }, actions: [ "killCursors" ] }
],
roles: []
}
)

Thank You !
Moshe Kaplan
moshe.kaplan@brightaqua.com
054-2291978

MongoDB training for java software engineers

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (10)

Similar to MongoDB training for java software engineers

Similar to MongoDB training for java software engineers (20)

More from Moshe Kaplan

More from Moshe Kaplan (20)

Recently uploaded

Recently uploaded (20)

MongoDB training for java software engineers