Por qué MongoDB: caso real de tráfico marítimo

Spring Data y MongoDB
en un proyecto real.

David Gómez
@dgomezg

sábado 18 de febrero de 12

David Gómez
rchitect & Trainer
Sw A mas.com
siste
extrema-
dgomez@
zg
@dgome


Spring CORE

Rich Web Applications
with Spring

Enterprise Integration


#springio
#extrema


Hablaremos de

Un caso real
¿Por qué NoSQL? ¿Por qué MongoDB?
MongoDB ‘a pelo’
Spring Data al rescate
¡Cuidado!


Un caso real.
Control de tráﬁco marítimo


Emisor AIX
3-5 min


Receptor GPS

Radar (3 s)
Goniometro

Estación Meteo,
Emisor AIX
Cámaras,
3-5 min
...


El problema: La escalabilidad


Por cada barco:
Recepción y procesamiento de señales (nº)
Transacciones,
Cálculo de alarmas
Gestión de Históricos


Señales
(1 AIS x 3 min + 1 radar x 3 seg) x 100 barcos
Alarmas
Persistencia de Líneas de Varada
(10.000+ ptos x 100+ líneas)
40 Alarmas
Zonas, Elementos Navegación
Persistencia: Total: 87 tablas + históricos(x2)


¿Por qué NoSQL?

Teorema CAP (Eric Brewer)

Source: Nathan Hurst’s Blog


¿Por qué NoSQL?


Consistency

C



¿Por qué NoSQL?


Consistency Availability

C A



¿Por qué NoSQL?



C A

P
Partition Tolerance



¿Por qué NoSQL?



C A
Sólo
2

P
Partition Tolerance



¿Por qué NoSQL?



A
Oracle, MySQL,
C
Sólo
2

P
Partition Tolerance



¿Por qué NoSQL?



A
Oracle, MySQL,
C
Sólo
M Hyp is, M
on er em

2
go Ta c
Re

DB ble ac
d

, D , H heD
at Ba B
aS se
to
re
,

P
Partition Tolerance



¿Por qué NoSQL?
RDBMS
Orientadas a documento
Teorema CAP (Eric Brewer) orientadas a columna
Key-Value


A
Oracle, MySQL,
C
Sólo
M Hyp is, M

,
on er em

uc ra, ort
2

aK
go Ta c

Co nd m
Re

Ri
DB ble ac

B, a de
d

B,
D ass ol
, D , H heD

hD
ple C o,V
at Ba B

am
aS se
to

yn
re

D
,

P
Partition Tolerance Sim



Pero... ¿NoSQL?

(cc) Photo by mwin - http://www.ﬂickr.com/photos/wingler/3429634150


Pero... ¿NoSQL?

NoSQL !== no SQL
Not only SQL

(cc) Photo by mwin - http://www.ﬂickr.com/photos/wingler/3429634150


NoSQL: ACID vs BASE

Atomicity, Basically
Available,
Consistency,
Soft state,
Isolation,
Eventual
Durability consistency

(cc) Photo by zhouxuan12345678 - http://www.ﬂickr.com/photos/53921113@N02/5645102295/


Tipos.

orientadas a Columna orientadas a Documento

Key-value orientadas a Grafos


¿Por qué MongoDB?

Orientada a documento
Almacenados en Colecciones
(bajo una clave)
Los documentos pueden ser heterogéneos

Formatos textuales: JSON & BSON

(cc) Photo by Ampersand Duck - http://www.ﬂickr.com/photos/ampersandduck/4941185476


MongoDB

Escrita en C++
Orientada a documento
Formato JSON (o BSON)
Un poco SQL (queries, índices, Referencias
externas)
Particionado horizontal (sharding)
Consultas Javascript

(cc) Photo by Ampersand Duck - http://www.ﬂickr.com/photos/ampersandduck/4941185476


Orientadas a Documento

Documentos formateados en JSON


Orientadas a Documento

Documentos formateados en JSON
{ "_id" : "224000999",
"_class" : "com.vts.model.Vessel",
"flag" : "ALBANIA",
"name" : "Sample NOT_AVAILABLE Vessel 224000999",
"callsign" : "SV224000999",
"toBow" : 25,
"toStern" : 5,
"toPort" : 5,
"toStarboard" : 5,
"comments" : "Sample vessel created automatically for test purposes"
}


El API

Consola y API JavaScript:


El API

beleriand:bin dgomez$ ./mongo
MongoDB shell version: 1.8.1
connecting to: test
>


El API

connecting to: test
> use vts
switched to db vts
>


El API

connecting to: test
> use vts
switched to db vts
> show collections
>
Event
WeatherData
system.indexes
vessel
>


El API

connecting to: test
> use vts
switched to db vts
> show collections
>
Event
WeatherData
system.indexes
vessel
> db.Event.find()
>
{ "_id" : ObjectId("4e0b5b211446446f6be3bb1a"), "_class" :
"com.vts.model.events.SystemEvent", "timestamp" :
ISODate("2011-06-29T17:04:33.039Z") }
{ "_id" : ObjectId("4e0b5b3d144676f49946443f"), "_class" :
"com.vts.model.events.SystemEvent", "timestamp" :
ISODate("2011-06-29T17:05:01.394Z") }
>


El API

> show collections
Event
WeatherData
system.indexes
vessel
>


El API

> show collections
Event
WeatherData
system.indexes
vessel
> db.alarms.save({"type" : "speed", "severity" : "warn", "vesselSSID" :
>
"224000999"})
>


El API

> show collections
Event
WeatherData
system.indexes
vessel
>
"224000999"})
> show collections
>
Event
WeatherData
alarms
system.indexes
vessel
>


El API

> show collections
Event
WeatherData
system.indexes
vessel
>
"224000999"})
> show collections
>
Event
WeatherData
alarms
system.indexes
vessel
> db.alarms.find()
>
{ "_id" : ObjectId("4f3acca3ed4cc8078fd1a4be"), "type" : "speed", "severity" :
"warn", "vesselSSID" : "224000999" }
>


Documentos

Identiﬁcados por un “_id” (generado o asignado)
Textual (JSON)
Binario (BSON)*
Documentos grandes divididos en
chunks


Programación de clientes

Multitud de lenguajes (drivers)
Transformación a JSON realizada por el driver




{ "_id" : "224000999",
"flag" : "ALBANIA",
"name" : "Sample NOT_AVAILABLE Vessel 224000999",
"callsign" : "SV224000999",
"toBow" : 25,
"toStern" : 5,
"toPort" : 5,
"toStarboard" : 5,
"comments" : "Sample vessel created automatically for test purposes" }

sample.py




{ "_id" : "224000999",
{ "_id" => "224000999",
"flag""flag" => "ALBANIA",
: "ALBANIA",
"name""name" => "Sample NOT_AVAILABLE 224000999",
: "Sample NOT_AVAILABLE Vessel Vessel 224000999",
"callsign" : "SV224000999",
"callsign" => "SV224000999",
"toBow" : 25, => 25,
"toBow"
"toStern" : 5, => 5,
"toStern"
"toPort" : 5, => 5,
"toPort"
"toStarboard" : 5, => 5,
"toStarboard"
"comments" : "Sample"Sample createdcreated automatically for test purposes" }
"comments" => vessel vessel automatically for test purposes" }

sample.py Ruby




{ "_id" : "224000999",
{ "_id" => "224000999", "224000999",
array( "_id" =>
"flag""flag" "flag" => "ALBANIA",
: "ALBANIA",
=> "ALBANIA",
"name""name" "name" => "Sample NOT_AVAILABLE Vessel 224000999",
: "Sample"Sample NOT_AVAILABLE 224000999",
=> NOT_AVAILABLE Vessel Vessel 224000999",
"callsign" : "callsign" => "SV224000999",
"callsign" => "SV224000999",
"SV224000999",
"toBow" : 25, => 25, => 25,
"toBow""toBow"
"toStern" : 5, => 5, => 5,
"toStern"
"toStern"
"toPort" : 5, => 5, => 5,
"toPort"
"toPort"
"toStarboard" : 5, => 5, => 5,
"toStarboard"
"toStarboard"
"comments" : "comments" => "Sample vessel created automatically for test purposes"}
"comments" => "Sample createdcreated automatically for test purposes" }
"Sample vessel vessel automatically for test purposes" }

sample.py sample.php
Ruby




BasicDBObject doc = new BasicDBObject();
{ "_id" : "224000999",
{ "_id" => "224000999", "224000999",
array( "_id" => doc.put("flag", vessel.getFlag());
"flag""flag" "flag" => "ALBANIA",
: "ALBANIA",
=> "ALBANIA",doc.put("name", vessel.getName());
"name""name" "name" => "Sample NOT_AVAILABLE Vessel 224000999",
: "Sample"Sample NOT_AVAILABLE 224000999",
=> NOT_AVAILABLE Vessel Vessel 224000999",
doc.put("callsign", vessel.getCalSign());
"callsign" : "callsign" => "SV224000999",
"callsign" => "SV224000999",
"SV224000999",
doc.put("toBow", vessel.getBow());
"toBow" : 25, => 25, => 25,
"toBow""toBow"
"toStern" : 5, => 5, => 5,
"toStern"
"toStern" doc.put("toPort", vessel.getStern());
"toPort" : 5, => 5, => 5,
"toPort"
"toPort" doc.put("toStern", vessel.getPort());
"toStarboard" : 5, => 5, doc.put("toStartboard", vessel.getStarboard());
"toStarboard"
"toStarboard" => 5,
"comments" : "comments" => doc.put("comments", vessel.getComments()); purposes"}
"comments" => "Sample createdcreated automatically for test purposes" }
"Sample vessel vessel vessel created automatically for test
"Sample automatically for test purposes" }

sample.py sample.php
Ruby
sample.java


Cliente Java con
MongoDB
‘a pelo’


1. Obtener el Driver

Importar el driver


1. Obtener el Driver

Importar el driver

<dependency>
<groupId>org.mongodb</groupId>
<artifactId>mongo-java-driver</
artifactId>
<version>2.7.0-rc1</version>
</dependency>


2. Conexión

Obtener la referencia a la BD y la conexión


2. Conexión

Obtener la referencia a la BD y la conexión

Mongo m = new Mongo( "127.0.0.1" , 27017 );
DB db = m.getDB( "playground" );


3. Inserciones
Crear e insertar un objeto


3. Inserciones
Crear e insertar un objeto
BasicDBObject doc = new BasicDBObject();
doc.put("flag", vessel.getFlag());
doc.put("name", vessel.getName());
doc.put("callsign", vessel.getCalSign());
doc.put("toBow", vessel.getBow());
doc.put("toPort", vessel.getStern());
doc.put("toStern", vessel.getPort());
doc.put("toStartboard", vessel.getStarboard());
doc.put("comments", vessel.getComments());

DBCollection coll = db.getCollection("vessel");
coll.insert(doc);


4. Consultas (I)

Obtener todos los documentos de una colección


4. Consultas (I)

DBCollection vessels = db.getCollection("vessel");

DBCursor cursor = vessels.find();

List<Vessel> vessels = new ArrayList<>();
while (cursor.hasNext()) {
DBObject jsonVessel = cursor.next();
Vessel vessel = new Vessel();
vessel.setMmsi(jsonVessel.get(“mmsi”));
vessel.setLength(jsonVessel.get(“length”));
// ...

vessels.add(vessel);
}


4. Consultas (I)


DBCursor cursor = vessels.find();

List<Vessel> vessels = new ArrayList<>();
while (cursor.hasNext()) { class DBCollection {
public abstract
DBObject jsonVessel = cursor.next();
public final DBCursor find(){
Vessel vessel = new Vessel();
public final DBCursor find( DBObject ref )
vessel.setMmsi(jsonVessel.get(“mmsi”));
public final DBCursor find( DBObject ref , DBObject keys )
vessel.setLength(jsonVessel.get(“length”));
// ...
public final DBObject findOne( Object obj )
vessels.add(vessel); DBObject findOne( Object obj, DBObject fields )
public final
}
public DBObject findAndModify( DBObject query , DBObject update )

}


4. Consultas (II)
Consultas “SQL-like”


4. Consultas (II)
En RDBMS:
SELECT * FROM VESSEL WHERE FLAG = “ALBANIA”;

Las consultas se hacen también en JSON


4. Consultas (II)
En RDBMS:

{ “flag” : “ALBANIA” }


4. Consultas (II)
En RDBMS:


connecting to: test
> use vts
switched to db vts
> db.vessel.find( { "flag" : "ALBANIA" });
{ "_id" : "224312999", "_class" : "com.vts.model.Vessel", "flag" : "ALBANIA",
"name" : "Sample TOWING Vessel 224312999", "callsign" : "SV224312999", “length”:
30, “beam” : “10”, "toBow" : 25, "toStern" : 5, "toPort" : 5, "toStarboard" : 5,
>


4. Consultas (II)
En RDBMS:


connecting to: test
> use vts
switched to db vts DBObject query = new BasicDBObject();
query.put("flag", "ALBANIA");
> db.vessel.find( { "flag" : "ALBANIA" });
{ "_id" : "224312999", "_class" : "com.vts.model.Vessel", "flag" : "ALBANIA",
DBCursor albanianShips = vessels.find(query);
"name" : "Sample TOWING Vessel 224312999", "callsign" : "SV224312999", “length”:
30, “beam” : “10”, "toBow" : 25, "toStern" : 5, "toPort" : 5, {
while (albanianShips.hasNext()) "toStarboard" : 5,
// ...
> }


Operadores de consulta

Otros operadores


Operadores de consulta

Otros operadores
$all
$and
$exists
$gt
$in
$lt
$mod
$ne
$nin
$nor
$or
$size
$type


Consultas complejas
Concatenación de consultas
RDBMS:


Consultas complejas
SELECT * FROM VESSELS WHERE length < 90 AND length > 30
RDBMS: AND (flag = “ALBANIA” or flag = ”SAN MARINO”);


Consultas complejas

{
“length” : { $lt : 90, $gt : 30},
Mongo: $or : [
{ “flag” : “ALBANIA”},
{ “flag” : “SAN MARINO”} ]
}


Consultas complejas

{
“length” : { $lt : 90, $gt : 30},
Mongo: $or : [
{ “flag” : “ALBANIA”},
{ “flag” : “SAN MARINO”} ]
}

DBObject query = new BasicDBObject();
DBObject condition = new BasicDBObject("$lt",90);
condition.put("$gt",30);
query.put("length", condition);

DBObject [] flags = {
Mongo new BasicDBObject("flag", "ALBANIA"),
new BasicDBObject("flag", "SAN MARINO")
};
Driver: query.put("$or", flags);

talks.find(query);


DBObjects desde JSON

com.mongo.util.JSON


DBObjects desde JSON

com.mongo.util.JSON
Object query = JSON.parse("{ 'length' : { $lt : 90, $gt : 30}, " +
'$or' : [ { 'flag' : 'ALBANIA}, " +
"
" { 'flag' : 'SAN MARINO'} ] }");

DBCursor cursor = db.find((DBObject) query);


Además...

Gestión de excepciones
Gestión recursos
DB, Collección, Cursores, DBObjects...

Transformaciones Objeto JSON


Spring Data MongoDB


Spring Data
Objetivo:
Proporcionar un mecanismo homogéneo y
completo para el acceso a BD NoSQL
Multiples proyectos:
Soportados:
MongoDB, Redis, Neo4j, Hadoop, GemFire,
RiaK
En proceso:
CouchDB, Hbase, Cassandra


Spring Data y MongoDB

XML namespace para conﬁgurar driver Mongo
MongoTemplate
Conversión de excepciones automática
Conversión Conﬁgurable
JMX monitoring


Importando Spring Data


Importando Spring Data

<repository>
<id>spring-release</id>
<name>Spring Maven Release Repository</name>
<url>http://repo.springsource.org/libs-release</url>
</repository>

<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-mongodb</artifactId>
<version>1.0.1.RELEASE</version>
</dependency>


Namespace


Namespace
<?xml
version="1.0"
encoding="UTF-‐8"?>
<beans
xmlns="http://www.springframework.org/schema/beans"

xmlns:xsi="http://www.w3.org/2001/XMLSchema-‐instance"

xmlns:mongo="http://www.springframework.org/schema/data/mongo"

xmlns:context="http://www.springframework.org/schema/context"

xsi:schemaLocation="http://www.springframework.org/schema/beans

http://www.springframework.org/schema/beans/spring-‐beans.xsd

http://www.springframework.org/schema/data/mongo

http://www.springframework.org/schema/data/mongo/spring-‐mongo-‐1.0.xsd

http://www.springframework.org/schema/context

http://www.springframework.org/schema/context/spring-‐context.xsd">

</beans>


Namespace
<?xml
version="1.0"
<beans










<mongo:db-‐factory
dbname="vts"/>

</beans>


Namespace
<?xml
version="1.0"
<beans










<mongo:db-‐factory
dbname="vts"/>

<bean
id="mongoTemplate"
class="org.springframework.data.mongodb.core.MongoTemplate">

<constructor-‐arg
name="mongoDbFactory"
ref="mongoDbFactory"
/>

</bean>

</beans>


MongoTemplate
<bean
id="mongoTemplate"

<constructor-‐arg
name="mongoDbFactory"
ref="mongoDbFactory"
/>
</bean>

Identiﬁca la colección
Convierte query a JSON
Convierte respuesta (“_class”)
Gestiona las conexiones y cursores
Convierte las excepciones


MongoTemplate - Consultas



public class MongoTemplate implements MongoOperations, ApplicationContextAware {

public <T> T findOne(Query query, Class<T> entityClass)

public <T> List<T> find(Query query, Class<T> entityClass)

public <T> T findById(Object id, Class<T> entityClass)

public <T> List<T> findAll(Class<T> entityClass)

}




public <T> T findOne(Query query, Class<T> entityClass)

public <T> List<T> find(Query query, Class<T> entityClass)

public <T> T findById(Object id, Class<T> entityClass)

public <T> List<T> findAll(Class<T> entityClass)

}
public <T> T findOne(Query query, Class<T> entityClass, String collectionName)
....


Consultas
RDBMS: SELECT * FROM VESSEL WHERE FLAG = “ALBANIA”;


Consultas

Mongo: { “flag” : “ALBANIA” }


Consultas



query.put("flag", "ALBANIA");

Mongo DBCursor albanianShips = vessels.find(query);
while (albanianShips.hasNext()) {
Driver: DBObject result = albanianShips.next();
// Create Vessel object from DBObject
}


Consultas


Query query = new Query(Criteria.where("flag").is("ALBANIA");

List<Vessel> vessels = mongoTemplate.find(query, Vessel.class);

Utilizado para
Identiﬁcación de la colección
Serialización JSON


Serialización automática


Serialización automática

{
"_id"
:
"224000999",

"_class"
:
"com.vts.model.Vessel",

"flag"
:
"ALBANIA",

"name"
:
"Sample
NOT_AVAILABLE
Vessel
224000999",

"callsign"
:
"SV224000999",

"toBow"
:
25,

"toStern"
:
5,

"toPort"
:
5,

"toStarboard"
:
5,

"comments"
:
"Sample
vessel
created
automatically
for
test
purposes"
}


Consultas complejas


Consultas complejas
ORDER BY lastUpdate;


Consultas complejas

{
“length” : { $lt : 90, $gt : 30}
}


Consultas complejas

{
“length” : { $lt : 90, $gt : 30}
}

DBObject condition = new BasicDBObject("$lt",90);
condition.put("$gt",30);
query.put("length", condition);

DBObject [] flags = {
new BasicDBObject("flag", "ALBANIA"),
new BasicDBObject("flag", "SAN MARINO")
};
query.put("$or", flags);

vessels.find(query);


Consultas complejas

{
“length” : { $lt : 90, $gt : 30}
}

Criteria criteria = Criteria.where("length");
criteria.gte(from);
criteria.lte(to);

Query query = new Query(criteria);
query.sort().on("lastUpdate", Order.ASCENDING);

List<Vessel> vessels = mongoTemplate.find(query, Vessel.class);


MongoTemplate - inserciones


MongoTemplate - inserciones


public void save(Object objectToSave)

public void save(Object objectToSave, String collectionName)

}


AbstractRepository


AbstractRepository

public abstract class AbstractMongoRepository<T> {

@Inject
protected MongoTemplate mongoTemplate;

/** Clase que utiliza Mongo para identificar el nombre de la coleccion */
private Class<T> persistentClass;

protected AbstractService(Class<T> persistentClass) {
this.persistentClass = persistentClass;
}

public String determineCollection() {
return persistentClass.getSimpleName();
}
}


AbstractRepository
public abstract class AbstractMongoRepository<T> {

public void save(T e) {
mongoTemplate.save(e, determineCollection());
}

public void save(Collection<T> entities) {
for (T e : entities) {
save(e);
}
}

public void removeById(String id) {
Object e = findById(id);
if (e != null) {
mongoTemplate.remove(e, determineCollection());
}
}

public T findById(String id) {
T t = mongoTemplate.findById(id, persistentClass);
if (t == null) {
throw new ObjectNotFoundException(persistentClass, id);
}
return t;
}
}


VesselRepository


VesselRepository

public class VesselRepository extends AbstractMongoRepository<Vessel> {

public VesselRepository() {
super(Vessel.class);
}

}


A tener en cuenta


El caso de BitSet
Cuidado con los campos transient


El caso de BitSet
Cuidado con los campos transient
public class BitSet implements Cloneable, java.io.Serializable {

/** The internal field corresponding to the serialField "bits". */
*/
private long[] words;

/** The number of words in the logical size of this Bitset */
private transient int wordsInUse = 0;

private void expandTo(int wordIndex) {
int wordsRequired = wordIndex+1;
if (wordsInUse < wordsRequired) {
ensureCapacity(wordsRequired);
wordsInUse = wordsRequired;
}
}
}


Conversión Manual


Conversión Manual
public
class
BitSetReadConverter
implements
Converter<DBObject,
BitSet>
{

@Override

public
BitSet
convert(DBObject
source)
{

BasicDBList
words
=
(BasicDBList)source.get("words");

BitSet
bitset
=
new
BitSet();

int
index
=
0;

for
(Object
word
:
words)
{

long
value
=
(Long)word;

while
(value
!=
0L)
{

if
((value
&
1L)
!=
0)
{

bitset.set(index);

}

++index;

value
=
value
>>>
1;

}

}

return
bitset;

}
}


Conversión Manual
Porque a veces lo automático no funciona


Conversión Manual
Porque a veces lo automático no funciona
<?xml version="1.0" encoding="UTF-8"?>
<beans>

<bean id="mongoTemplate"
<constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
<constructor-arg name="mongoConverter" ref="mappingConverter" />
</bean>

<mongo:db-factory dbname="vts"/>

<mongo:mapping-converter>
<mongo:custom-converters>
<mongo:converter>
<bean class="com.vts.db.BitSetReadConverter"/>
</mongo:converter>
</mongo:custom-converters>
</mongo:mapping-converter>

</beans>


Queries con multiples valores



Query query = new Query(
Criteria
.where("length").gte(20)
.and("length").lt(60));



Criteria
.where("length").gte(20)
.and("length").lt(60));

DEBUG MongoTemplate - find using query: { "length" : { "$lt" : 60} }



Criteria
.where("length").gte(20).lte(60));



Criteria
.where("length").gte(20).lte(60));

DEBUG MongoTemplate - find using query: { "toBow" : { "$gte" : 20 , "$lte" : 60} }


Conclusiones
Reducción del número de tablas (87x2 vs 5)
Optimización en el rendimiento de escrituras
Simpliﬁcación de los DAOs con Spring-Data

Otras ventajas:
Documentos en formato de Respuesta REST
interfaz Javascript en la consola


Q&A
David Gómez
@dgomezg


Por qué MongoDB: caso real de tráfico marítimo

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

More from David Gómez García

More from David Gómez García (20)

Recently uploaded

Recently uploaded (13)

Por qué MongoDB: caso real de tráfico marítimo