The document discusses using Cassandra and Celery to solve big data problems. Cassandra is described as a distributed, fault-tolerant, and scalable NoSQL database that stores data as key-value pairs. Celery is presented as a tool for distributing tasks across multiple machines by adding tasks to queues. An example architecture is proposed using RabbitMQ to connect an application to Celery workers that interface with the Cassandra database. Configuration and usage of Cassandra, Celery, and their integration are outlined.

1. Big Data amb Cassandra i Celery
#bbmnk novembre 2013
Santi Camps Taltavull
@santicamps
@socialvane

2. La Problemàtica (Big Data)
➲
➲
➲
➲
➲
Gran volum d'informació (TeraBytes)
Informació no estructurada
Poca densitat d'informació útil
Altíssima capacitat de processament
Poca pasta

3. Les solucions aplicades
➲
➲
➲
➲
➲
➲
➲
➲
➲
➲
➲
➲
BBDD distribuida Cassandra
Gestor de tasques distribuides Celery
Gestor de missatgeria RabbitMQ
Aplicació --> RabbitMQ --> Celery <--> Cassandra
4 servidors inicials
12 TB de capacitat
208 GB de RAM
44 nuclis de CPU
Tolerant a fallades
Redundant
Molt Fàcilment Escalable
I Barat !!

4. Cassandra
➲
➲
➲
➲
➲
➲
Neix dins de Facebook i s'allibera
L'adopta la fundació Apache
Twitter també l'empra
Està escrit amb Java
És una BBDD NO SQL
Les dades es guarden com a clau -> valor

5. Cassandra - Avantatges
➲
➲
➲
➲
➲
BBDD distribuida
Redundància configurable
Tolerant a fallades
Preparada per WAN
Totalment Escalable

6. Cassandra - Inconvenients
➲
➲
➲
➲
No té gestió de transaccions
Es coordina amb timestamps
En mode RandomPartitioner no permet ordenar
En mode RandomPartitioner filtrar es fa difícil

7. Cassandra - Característiques
➲
➲
➲
➲
➲
Name Space = BBDD
Column Family = Taula
Cada Registre pot tenir columnes diferents
Un Registre pot tenir milions de columnes
Tots es guarda com a clau -> valor

8. Cassandra - Exemple
create column family user_item with key_validation_class = 'UTF8Type' and comparator =
'UTF8Type' and default_validation_class = 'UTF8Type'
and column_metadata=[
{ column_name: source,
validation_class: UTF8Type, index_type: KEYS},
{column_name: user_name,
validation_class: UTF8Type, index_type: KEYS},
{column_name: type,
validation_class: UTF8Type},
{column_name: last_update,
validation_class: UTF8Type},
{column_name: id,
validation_class: UTF8Type},
{column_name: profile_image_url, validation_class: UTF8Type},
{column_name: name,
validation_class: UTF8Type},
{column_name: friends_count,
validation_class: UTF8Type},
{column_name: followers_count, validation_class: UTF8Type},
{column_name: location,
validation_class: UTF8Type},
{column_name: description,
validation_class: UTF8Type},
{column_name: lang,
validation_class: UTF8Type},
{column_name: geo_latitude,
validation_class: FloatType, index_type: KEYS},
{column_name: geo_longitude,
validation_class: FloatType, index_type: KEYS},
{column_name: geo_radious,
validation_class: FloatType},
];

9. Cassandra - Exemple
get user_item['facebook.santi.camps.58']
... ;
=> (name=description, value=Me dedico a ..., timestamp=1383782405981374)
=> (name=followers_count, value=0, timestamp=1383782405981374)
=> (name=friends_count, value=, timestamp=1383782405981374)
=> (name=geo_latitude, value=4.264729, timestamp=1383782405981374)
=> (name=geo_longitude, value=39.88943, timestamp=1383782405981374)
=> (name=geo_radious, value=8.976159, timestamp=1383782405981374)
=> (name=id, value=100000444843078, timestamp=1383782405981374)
=> (name=lang, value=en_GB, timestamp=1383782405981374)
=> (name=last_update, value=2013-11-07T01:00:05.981352, timestamp=1383782405981374)
=> (name=location, value=Mahón, Islas Baleares, Spain, timestamp=1383782405981374)
=> (name=name, value=Santi Camps, timestamp=1383782405981374)
=> (name=profile_image_url, value=https://graph.facebook.com/santi.camps.58/picture,
timestamp=1383782405981374)
=> (name=profile_url, value=https://www.facebook.com/santi.camps.58,
timestamp=1383782405981374)
=> (name=source, value=facebook, timestamp=1383782405981374)
=> (name=type, value=user, timestamp=1383782405981374)
=> (name=user_name, value=santi.camps.58, timestamp=1383782405981374)

10. Cassandra - Indexació
get user_follower_index['santicamps58.facebook.current'];
=> (name=2013-10-29T11:09:01.979083, value=santicamps58.facebook.100000561127539,
timestamp=1381823950979106)
=> (name=2013-10-27T09:59:07.980314, value=santicamps58.facebook.1810751517,
timestamp=1381823950980330)
=> (name=2013-10-11T07:50:10.980547, value=santicamps58.facebook.100002326398873,
timestamp=1381823950980559)
...
get user_follower_item['santicamps58.facebook.100002326398873'];
=> (name=fetch_date, value=2013-10-15, timestamp=1381823950980662)
=> (name=friend_count, value=134, timestamp=1381823950980662)
=> (name=id, value=100002326398873, timestamp=1381823950980662)
=> (name=lang, value=, timestamp=1381823950980662)
=> (name=name, value=Diego Izquierdo Carranza, timestamp=1381823950980662)
=> (name=profile_image_url, value=https://graph.facebook.com/diego.izquierdocarranza/picture,
timestamp=1381823950980662)
=> (name=profile_url, value=https://www.facebook.com/diego.izquierdocarranza,
timestamp=1381823950980662)
=> (name=source, value=facebook, timestamp=1381823950980662)
=> (name=start_date, value=2013-10-15, timestamp=1381823950980662)
=> (name=user_name, value=diego.izquierdocarranza, timestamp=1381823950980662)

11. Cassandra - Indexació
get mention_tag_source_index['803.possitive'];
...
=> (name=2013-11-08T02:00:27.361445, value=803__-UzkY7psQTYJ,
timestamp=1383876396514768)
=> (name=2013-11-08T06:53:57, value=803__twitter.398704931630481408,
timestamp=1383894677856944)
=> (name=2013-11-08T06:54:38, value=803__twitter.398705100648382464,
timestamp=1383894677646453)
=> (name=2013-11-08T06:57:51, value=803__twitter.398705909511503872,
timestamp=1383894677313681)
...
get mention_tag_source_index['803.possitive.google'];
=> (name=2012-12-01T00:00:00.395260, value=803__YfOIKwVseDkJ,
timestamp=1381830781423739)
=> (name=2012-12-01T00:00:00.420936, value=803__YfOIKwVseDkJ,
timestamp=1381867147942586)
=> (name=2012-12-01T00:00:00.633055, value=803__YfOIKwVseDkJ,
timestamp=1381830436666804)
=> (name=2013-06-14T00:00:00.055140, value=803__5Bv2Eu9qk04J,
timestamp=1381867142254676)

12. Cassandra - Indexació
get mention_item['803__twitter.398705909511503872'];
=> (name=body, value=@SocialVane INTERESANTÍSIMA HERRAMIENTA DE ANÁLISIS PARA
REDES SOCIALES, timestamp=1383894677307778)
=> (name=body_norm, value=your_brand interesante herramienta analisis red your_brand,
timestamp=1383894677307778)
=> (name=brand, value=103, timestamp=1383894677307778)
=> (name=checked, value=false, timestamp=1383894677307778)
=> (name=emissor, value=SebastianCamps, timestamp=1383894677307778)
=> (name=emissor_id, value=234140801, timestamp=1383894677307778)
=> (name=emissor_name, value=Sebastián Camps , timestamp=1383894677307778)
=> (name=geo, value=None, timestamp=1383894677307778)
=> (name=id, value=398705909511503872, timestamp=1383894677307778)
=> (name=in_reply_to_id, value=, timestamp=1383894677307778)
=> (name=interest, value=, timestamp=1383894677307778)
=> (name=interest_checked, value=False, timestamp=1383894677307778)
=> (name=lang, value=es, timestamp=1383894677307778)
=> (name=like_action_count, value=0, timestamp=1383894677307778)
=> (name=probability, value=0.482361909795, timestamp=1383894677307778)
=> (name=query, value=803, timestamp=1383894677307778)
=> (name=reply_action_count, value=0, timestamp=1383894677307778)
=> (name=retweeted, value=False, timestamp=1383894677307778)
=> (name=share_action_count, value=0, timestamp=1383894677307778)
=> (name=source, value=twitter, timestamp=1383894677307778)
=> (name=tag, value=possitive, timestamp=1383894677307778)
=> (name=time, value=2013-11-08T06:57:51, timestamp=1383894677307778)

13. Celery
➲
➲
➲
➲
➲
Es configuren cues d'execució
S'engeguen N workers a M màquines escoltant cada cua
Les tasques distribuibles es marquen al codi
Es defineix la cua d'execució de cada tasca
Es poden cridar síncronament o asíncrona
➲
➲
➲
Molt senzill d'implantar
Molt fàcil d'escalar
Cal vigilar la concurrència

14. Celery Exemple
CELERY_ROUTES = {
"celeryutils.track_all_users_followers": {"queue": "slow", "routing_key": "slow_task"},
"userfollowers.bulk_insert": {"queue": "slow", "routing_key": "slow_task"},
"extract_mentions_from_website": {"queue": "slow", "routing_key": "slow_task"},
"LeadsClassifier.classify_untagged": {"queue": "cpu", "routing_key": "cpu_task"},
...
@task(name = 'extract_mentions_from_website', time_limit=300)
def extract_mentions_from_website(brand, query,...):
...
# CRIDA LOCAL
extract_mentions_from_website(params)
# CRIDA DISTRIBUIDA ASÍNCRONA
extract_mentions_from_website.delay(params)
# CRIDA DISTRIBUIDA SÍNCRONA
extract_mentions_from_website.delay(params).get()