Cascalog présenté par Bertrand Dechoux

  • 1,449 views
Uploaded on

 

More in: Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
1,449
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
4
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

Transcript

  • 1. HUG France #1 - 15 Mars 2012 Cascalog des MapReduces concisBertrand DECHOUX@BertrandDechoux
  • 2. Casca... quoi? Map/Reduce : pertinent mais insuffisant Des alternatives : Pig, Hive... et Cascading ! Cascalog : Cascading + Datalog + Clojure
  • 3. Cascading : les concepts tuple : liste de valeurs nommées tap : source et sink de données ▶ par défaut : LFS et HFS ▶ par la communauté : Avro, HBase, JDBC, JSON, Kryo, Memcached, Membase, ElasticSearch, SimpleDB, Solr, Thrift... flow : tap(s) source + des opérations + tap(s) sink
  • 4. Cascading : les opérations (Cascading 1.2 - User Guide)
  • 5. Clojure un dialect de Lisp sur la JVM ▶ environnement connu ▶ interoperabilité avec Java ▶ homoiconicité ou code-as-data : fait pour les DSLs
  • 6. Clojure : syntaxe 1/2 (http://lisperati.com/syntax.html)
  • 7. Clojure : syntaxe 2/2 (http://lisperati.com/syntax.html)
  • 8. Cascalog : select ... from ... (?<- (stdout) [?person] (person ?person)) (?<- (stdout) [?person ?age] (age ?person ?age)) (?<- (stdout) [?age] (age _ ?age))
  • 9. Cascalog : select ... from ... where ... (?<- (stdout) [?person ?age] (age ?person ?age) (< ?age 30))
  • 10. Cascalog : select ... as ... from ... (?<- (stdout) [?person ?junior] (age ?person ?age) (< ?age 30 :> ?junior))
  • 11. Cascalog : select count(*) from ... group by ... (?<- (stdout) [?count] (age _ _) (c/count ?count)) (?<- (stdout) [?junior ?count] (age _ ?age) (< ?age 30 :> ?junior) (c/count ?count)) disponible : count, sum, min, max
  • 12. Cascalog : select ... from ... join ... (?<- (stdout) [?person ?age ?gender] (age ?person ?age) (gender ?person ?gender))
  • 13. Cascalog : select ... from ... (select ...) (let [many-follows (<- [?person] (follows ?person _) (c/count ?count) (> ?count 2))] (?<- (stdout) [?personA ?personB] (many-follows ?personA) (many-follows ?personB) (follows ?personA ?personB)))
  • 14. Cascalog : définir vos fonctions (defn toUpperCase [person] (.toUpperCase person)) (?<- (stdout) [?PERSON] (person ?person) (toUpperCase ?person :> ?PERSON))
  • 15. Plus d’informations? Cascalog ▶ github ▶ wiki ▶ google group  Clojure et cie ▶ midje, framework de test ▶ Pallet, DevOps pour la JVM
  • 16. #1 HUG France 15 Mars 2012 Cascalog Merci pour votre attentionBertrand DECHOUX@BertrandDechoux