Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

What is Hydra?

2,551 views

Published on

Presented by Joel Westberg

Published in: Technology
  • Be the first to comment

What is Hydra?

  1. 1. WHAT IS HYDRA? Findability Day 2012
  2. 2. Hydra is technology
  3. 3. Hydra brings structure What is unstructured data? •  A linguistic excuse? News articles Plain text that contains invaluable metadata for search, such as: •  Title •  Author byline •  Lead paragraph
  4. 4. Hydra is about your data •  Enrich your documents with metadata, to power your search •  Language  detec+on   •  Sen+ment  analysis   •  Headline  extrac+on   •  Regular  expression  matching  and  extrac+on   •  Filter out unwanted documents •  Collect statistics •  Export to Staging environments
  5. 5. Before Hydra
  6. 6. Before Hydra
  7. 7. Hydra scales
  8. 8. Hydra Design Objectives Scalability •  Possible to connect any number of processing machines Fault tolerance •  Failiure of a stage affects only a single document •  Failiures can be automaticly detected Robustness •  Stages and nodes are completely independent (no domino- effect) Development ease •  Allow test driven pipeline development  
  9. 9. What about Hadoop and Big Data? Usecases for document enrichment •  Pagerank   •  Analy+cs   Hadoop & Map/Reduce advantages •  Huge  scalability   •  Ability  to  work  on  en+re  document  set  at  once   Hadoop & Map/Reduce drawbacks •  Batch  processing   •  Time-­‐to-­‐index  
  10. 10. Hydra integrated with Hadoop     Blue – First round of indexing only Red – Second round of indexing Purple – All documents
  11. 11. Hydra in summary Hydra •  can chew through almost anything •  has many heads •  regenerates •  scales
  12. 12. Hydra is Open Source •  Other committers •  The role of Findwise For more information: •  http://www.findwise.com/hydra •  http://findwise.github.com/Hydra •  Email: joel.westberg@findwise.com
  13. 13. Joel Westberg joel.westberg@findwise.com @joelwes

×