The original vision of Nutch, 14 years later: Building an open source search engine

•

3 likes•552 views

Sylvain Zimmer

This talk was presented at Apache BigData Europe 2016 in Sevilla, Spain on Nov 15th.

Technology

The original vision of Nutch, 14 years later:
Building an open source search engine
Apache Big Data Europe 2016
sylvain@sylvainzimmer.com
@sylvinus

/usr/bin/whoami
• Jamendo (Founder & CTO, 2004-2011)
• TEDxParis (Co-founder, 2009-2012)
• dotConferences (Founder, 2012-)
• Pricing Assistant (Co-founder & CTO, 2012-)

"The original motivation for the Nutch project was to provide
a transparent alternative to the growing power of a
handful of private search services over most users’ view of
the Web.
CommerceNet Labs Technical Report, Nov 2004
However, as Nutch has been adopted with greater
enthusiasm by smaller organizations, the Nutch
Organization has de-emphasized operating a multi-
billion-page index in the public interest."

https://explain.commonsearch.org/?q=python&g=en

Agenda
• Values & tech choices
• Search engine components
• Challenges
• Opportunities

Radical transparency
• Open source (Apache License v2)
• Open data
• (Governance)

Privacy
• Results can be tailored by language/country, but
NOT by user/cookie/sessionid
• o/ Cache everything!
• Tor service: http://comsearchl2zlnre.onion

Participation & Pragmatism
• Use high-level languages as much as possible
(Python, Go)
• Embrace active communities (Spark, Elasticsearch)
• Use mainstream participation platforms, even if they
are nonfree (GitHub, Slack)

http://infolab.stanford.edu/~backrub/google.html
The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998)
Crawler
Indexer
Database
SearcherRanker

Specs
• HTML parsing & analysis
• Tokenization / NLP
• Static rankings
• Language detection
• I/O from crawls to databases

Common Search Pipeline
Doc sources
Common Crawl,
WARC ﬁles,
URLs ...
Filter
plugins
Document
parsing
Output
plugins
Data output
Database, ﬁle,
HDFS, S3, ...

HTML parsers
• BeautifulSoup & friends
• lxml
• html5lib
• Gumbo!

Gumbocy
• Use Cython instead of ctypes
• Smaller API
• Tree traversal on the Cython side with basic
boilerplate/visibility support
https://github.com/commonsearch/gumbocy

https://github.com/commonsearch/urlparse4

Ranking formula
rank = f( static_score , dynamic_score( query ) )
Alexa
DMOZ
Blacklists
PageRank
...
ElasticSearch & Lucene
TF-IDF
BM25
...

https://about.commonsearch.org/developer/get-started

Specs
• Send user query to databases
• Search-as-you-type
• HTML & JSON endpoints
• High performance

https://github.com/commonsearch/cosr-front

http://infolab.stanford.edu/~backrub/google.html
The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998)
Crawler
Parser
Index
SearcherRanker

Funding / Scale
• Frugalism
• Caching
• In-kind services
• Individual donations / Foundation grants
• General economic incentives

Spam
• Email spam
• Wikipedia vandalism
• Algorithm complexity & scale
• Given enough eyeballs, all spam is shallow?

Relevance
• Exhaustivity
• Rescoring
• Evaluation
• More at 4:30pm ;-)

More search dimensions
• Realtime search
• Local search
• Universal search

Semantic search
• Wikidata
• YAGO
• Conversational / Voice search

Outreach
• Easy onboarding & docs
• Making people care believe

Decentralization
• YaCy
• Extremely high technical & social cost!
• Transparency?

Research
• More people should know how to build search
engines
• Spam, Relevance, Large-scale data processing
• We need more open datasets!

Make the Web a better place!
• SEO
• Transparency
• Inﬂuence of money
• Public service

Questions?
https://about.commonsearch.org/contributing
https://github.com/commonsearch
contact@commonsearch.org
slack.commonsearch.org

What's hot

Liferay's new search engine is extremely powerful. This session will show how the new OSGi infrastructure in DXP made it possible for us to introduce several new modular extension points in Search so customers can add their own search engine mappings, build custom queries for individual fields, programmatically fine tune boosting and relevance, and go beyond the Search Portlet with new Search Pages that can be assembled from both out-of-the-box portlets and brand new components built from scratch - for the ultimate customer experience. André "Arbo" Oliveira joined Liferay in early 2014 as a senior engineer and leads the Search Infrastructure team. He's been writing code for a living for 22 years, 14 of them as a Java developer and architect. Ever since discovering Elasticsearch, he's vowed never to write another SQL WHERE clause again.

Search Intelligently - Liferay Symposium North America 2016, Chicago, USA

André Ricardo Barreto de Oliveira

NoSQL: what does it mean, how did we get here, and why should I care? - Hugo ...

South London Geek Nights

RDF Graph Data Management in Oracle Database and NoSQL Platforms

Graph-TA

Graphs, Graphs everywhere - Lucene powered relation exploration

Zbyszko Papierski

There is a lot of confusion out there about the various kinds of NoSQL, and NewSQL, technologies. Document stores, graph databases, columnar databases, graph databases, and the list goes on. This confusion has lead to a good deal of less than optimal deployments, pain, and, ultimately, antipathy. In this talk, Dan will walk us through a high-level explanation of the various NoSQL technologies available to us, how they work, and provide some dos and don'ts for their implementation.

NoSQL Roundup

Daniel Fields

Deriving an Emergent Relational Schema from RDF Data

Graph-TA

Managing RDF data with graph databases

Graph-TA

Introduction to Graph Databases

Max De Marzi

Graph Databases try to make it easy for developers to leverage huge amounts of connected information for everything from routing to recommendations. Doing that poses a number of challenges on the implementation side. In this talk we want to look at the different storage, query and consistency approaches that are used behind the scenes. We’ll check out current and future solutions used in Neo4j and other graph databases for addressing global consistency, query and storage optimization, indexing and more and see which papers and research database developers take inspirations from.

How Graph Databases efficiently store, manage and query connected data at s...

jexp

Riding the Semantic Web

Matthias Vandermaesen

Infrastructure crossroads... and the way we walked them in DKPro

openminted_eu

Neo4j in Depth

Max De Marzi

Query-time Nonparametric Regression with Temporally Bounded Models - Patrick ...

Lucidworks

Webinar: Event Processing & Data Analytics with Lucidworks Fusion

Lucidworks

What's hot (14)

Search Intelligently - Liferay Symposium North America 2016, Chicago, USA

NoSQL: what does it mean, how did we get here, and why should I care? - Hugo ...

RDF Graph Data Management in Oracle Database and NoSQL Platforms

Graphs, Graphs everywhere - Lucene powered relation exploration

NoSQL Roundup

Deriving an Emergent Relational Schema from RDF Data

Managing RDF data with graph databases

Introduction to Graph Databases

How Graph Databases efficiently store, manage and query connected data at s...

Riding the Semantic Web

Infrastructure crossroads... and the way we walked them in DKPro

Neo4j in Depth

Query-time Nonparametric Regression with Temporally Bounded Models - Patrick ...

Webinar: Event Processing & Data Analytics with Lucidworks Fusion

Similar to The original vision of Nutch, 14 years later: Building an open source search engine

We've known for years that data-driven content was a 'thing' when we'd produce simple infographics that shared a few statistics and they'd get easy traction for us online. The game has lifted and consumers are becoming more and more obsessed with data and are now demanding higher quality and more complex data-driven content. The challenge for us now as "T-Shaped" marketers is that there are increasing demands for us to learn new skills to produce this content but we don't have the time to do this amongst the other things we need to be expert at. This presentation is going to give you specific help on how to produce data-driven content without any programming skill. After watching this presentation you'll have the confidence to build your own data-driven content with the knowledge of: - blueprints for data-driven content ideas - scraping tools, frameworks and methodologies - how to brief in a data scraping project to your in-house team or a freelancer - how to turn your data into visually appealing content - channels for promoting data-driven content to ensure it gets traction

Jeremy cabral search marketing summit - scraping data-driven content (1)

Jeremy Cabral

Meetup SF - Amundsen

Philippe Mizrahi

Ordering the chaos: Creating websites with imperfect data

Andy Stretton

Strata sf - Amundsen presentation

Tao Feng

One of the most compelling additions to a SharePoint practitioner’s toolbox is hybrid search. Although hybrid search capabilities were already around for a few years, with the introduction of the “Cloud Search Service Application” things got a lot more interesting. This demo-heavy session will focus on the technical implementation details and their prerequisites, as well as the typical hurdles that you’ll face in your first hybrid search project.

I2 - SharePoint Hybrid Search Start to Finish - Thomas Vochten

SPS Paris

The Web of data and web data commons

Jesse Wang

Drupal and Apache Stanbol

Alkuvoima

Resource sync overview and real-world use cases for discovery, harvesting, an...

openminted_eu

ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...

Martin Klein

"A Toolkit for Digital Research" - CNI 2013

Kaitlin Thaney

2018 09-03 aOS Aachen - SharePoint demystified - Thomas Vochten

aOS Community

DSpace is a free, open-source software application for creating institutional repositories. Out of the box DSpace is a rather simple, uninspired repository, but with a bit of work it can be turned into a workhorse for disseminating institutional knowledge. By committing to using DSpace as the canonical location for institutional outputs, you can focus on standardizing metadata taxonomies and carefully curating content, then leverage application program interfaces (APIs) to integrate it with other services. This strategy is more efficient, reduces duplication of outputs, and increases the potential impact of institutional knowledge through syndication, harvesting, etc.

Maximizing the Impact of Institutional Knowledge Using DSpace

AIMS (Agricultural Information Management Standards)

Building a Big Data Pipeline

Jesus Rodriguez

Data council sf amundsen presentation

Tao Feng

Open Data and Web API

Sammy Fung

Fc3 integration strategies

GabrieleSani3

This slide deck has been prepared for a workshop on Linked Data Publishing and Semantic Processing using the Redlink platform (http://redlink.co). The workshop delivered at the Department of Information Engineering, Computer Science and Mathematics at Università degli Studi dell'Aquila aimed at providing a general understanding of Semantic Web Technologies and how these can be used in real world use cases such as Salzburgerland Tourismus. A brief introduction has been also included on MICO (Media in Context) a European Union part-funded research project to provide cross-media analysis solutions for online multimedia producers.

What do we want computers to do for us?

Andrea Volpini

Disrupting Data Discovery

markgrover

Fried dallas spug

Jeff Fried

CKAN - the open source data portal platform

Maurizio Napolitano

Similar to The original vision of Nutch, 14 years later: Building an open source search engine (20)

Jeremy cabral search marketing summit - scraping data-driven content (1)

Meetup SF - Amundsen

Ordering the chaos: Creating websites with imperfect data

Strata sf - Amundsen presentation

I2 - SharePoint Hybrid Search Start to Finish - Thomas Vochten

The Web of data and web data commons

Drupal and Apache Stanbol

Resource sync overview and real-world use cases for discovery, harvesting, an...

ResourceSync - Overview and Real-World Use Cases for Discovery, Harvesting, a...

"A Toolkit for Digital Research" - CNI 2013

2018 09-03 aOS Aachen - SharePoint demystified - Thomas Vochten

Maximizing the Impact of Institutional Knowledge Using DSpace

Building a Big Data Pipeline

Data council sf amundsen presentation

Open Data and Web API

Fc3 integration strategies

What do we want computers to do for us?

Disrupting Data Discovery

Fried dallas spug

CKAN - the open source data portal platform

Recently uploaded

Enterprise Knowledge’s Urmi Majumder, Principal Data Architecture Consultant, and Fernando Aguilar Islas, Senior Data Science Consultant, presented "Driving Behavioral Change for Information Management through Data-Driven Green Strategy" on March 27, 2024 at Enterprise Data World (EDW) in Orlando, Florida. In this presentation, Urmi and Fernando discussed a case study describing how the information management division in a large supply chain organization drove user behavior change through awareness of the carbon footprint of their duplicated and near-duplicated content, identified via advanced data analytics. Check out their presentation to gain valuable perspectives on utilizing data-driven strategies to influence positive behavioral shifts and support sustainability initiatives within your organization. In this session, participants gained answers to the following questions: - What is a Green Information Management (IM) Strategy, and why should you have one? - How can Artificial Intelligence (AI) and Machine Learning (ML) support your Green IM Strategy through content deduplication? - How can an organization use insights into their data to influence employee behavior for IM? - How can you reap additional benefits from content reduction that go beyond Green IM?

Driving Behavioral Change for Information Management through Data-Driven Gree...

Enterprise Knowledge

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

🐬 The future of MySQL is Postgres 🐘

RTylerCroy

Tata AIG General Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

[2024]Digital Global Overview Report 2024 Meltwater.pdf

hans926745

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Delhi Call girls

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

08448380779 Call Girls In Friends Colony Women Seeking Men

Delhi Call girls

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

Slack Application Development 101 Slides

praypatel2

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

What are drone anti-jamming systems? The drone anti-jamming systems and anti-spoof technology protect against interference, jamming, and spoofing of the UAVs. To protect their security, countries are beginning to research drone anti-jamming systems, also known as drone strike weapons. The anti-jam and anti-spoof technology protects against interference, jamming and spoofing. A drone strike weapon is a drone attack weapon that can attack and destroy enemy drones. So what is so unique about this amazing system?

What Are The Drone Anti-jamming Systems Technology?

Antenna Manufacturer Coco

Handwritten Text Recognition for manuscripts and early printed texts

Maria Levchenko

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

Recently uploaded (20)

Driving Behavioral Change for Information Management through Data-Driven Gree...

A Domino Admins Adventures (Engage 2024)

🐬 The future of MySQL is Postgres 🐘

Tata AIG General Insurance Company - Insurer Innovation Award 2024

08448380779 Call Girls In Civil Lines Women Seeking Men

How to Troubleshoot Apps for the Modern Connected Worker

[2024]Digital Global Overview Report 2024 Meltwater.pdf

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

08448380779 Call Girls In Diplomatic Enclave Women Seeking Men

Scaling API-first – The story of a global engineering organization

A Year of the Servo Reboot: Where Are We Now?

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

08448380779 Call Girls In Friends Colony Women Seeking Men

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

Slack Application Development 101 Slides

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Axa Assurance Maroc - Insurer Innovation Award 2024

What Are The Drone Anti-jamming Systems Technology?

Handwritten Text Recognition for manuscripts and early printed texts

Boost Fertility New Invention Ups Success Rates.pdf

The original vision of Nutch, 14 years later: Building an open source search engine

1. The original vision of Nutch, 14 years later: Building an open source search engine Apache Big Data Europe 2016 sylvain@sylvainzimmer.com @sylvinus

2. /usr/bin/whoami • Jamendo (Founder & CTO, 2004-2011) • TEDxParis (Co-founder, 2009-2012) • dotConferences (Founder, 2012-) • Pricing Assistant (Co-founder & CTO, 2012-)

3. "The original motivation for the Nutch project was to provide a transparent alternative to the growing power of a handful of private search services over most users’ view of the Web. CommerceNet Labs Technical Report, Nov 2004 However, as Nutch has been adopted with greater enthusiasm by smaller organizations, the Nutch Organization has de-emphasized operating a multi- billion-page index in the public interest."

5. again?

10. transparency reproducibility

11.

12. https://uidemo.commonsearch.org

13. https://explain.commonsearch.org/?q=python&g=en

14. Agenda • Values & tech choices • Search engine components • Challenges • Opportunities

15. Values & tech choices

16.

17. Radical transparency • Open source (Apache License v2) • Open data • (Governance)

18. Privacy • Results can be tailored by language/country, but NOT by user/cookie/sessionid • o/ Cache everything! • Tor service: http://comsearchl2zlnre.onion

19. Participation & Pragmatism • Use high-level languages as much as possible (Python, Go) • Embrace active communities (Spark, Elasticsearch) • Use mainstream participation platforms, even if they are nonfree (GitHub, Slack)

20. Search engines

21. http://infolab.stanford.edu/~backrub/google.html The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998) Crawler Indexer Database SearcherRanker

22. Crawler

23. http://commoncrawl.org

24. Today at 3:30pm!

25. http://scrapy.org

26. http://github.com/cocrawler/cocrawler

27. Indexer

28. Specs • HTML parsing & analysis • Tokenization / NLP • Static rankings • Language detection • I/O from crawls to databases

29.

30. Common Search Pipeline Doc sources Common Crawl, WARC ﬁles, URLs ... Filter plugins Document parsing Output plugins Data output Database, ﬁle, HDFS, S3, ...

31. HTML parsers • BeautifulSoup & friends • lxml • html5lib • Gumbo!

32. https://github.com/google/gumbo-parser

33. Gumbocy • Use Cython instead of ctypes • Smaller API • Tree traversal on the Cython side with basic boilerplate/visibility support https://github.com/commonsearch/gumbocy

34. https://github.com/commonsearch/urlparse4

35. Database(s)

36.

37. http://lucene.apache.org/

38.

39. Ranker

40. Ranking formula rank = f( static_score , dynamic_score( query ) ) Alexa DMOZ Blacklists PageRank ... ElasticSearch & Lucene TF-IDF BM25 ...

41.

42. https://about.commonsearch.org/developer/get-started

43. Today @ 4:30pm ;-)

44. Searcher / Frontend

45. Specs • Send user query to databases • Search-as-you-type • HTML & JSON endpoints • High performance

46.

47. https://github.com/commonsearch/cosr-front

48. http://infolab.stanford.edu/~backrub/google.html The Anatomy of a Large-Scale Hypertextual Web Search Engine (1998) Crawler Parser Index SearcherRanker

49. Challenges

50. Funding / Scale • Frugalism • Caching • In-kind services • Individual donations / Foundation grants • General economic incentives

51. Spam • Email spam • Wikipedia vandalism • Algorithm complexity & scale • Given enough eyeballs, all spam is shallow?

52. Relevance • Exhaustivity • Rescoring • Evaluation • More at 4:30pm ;-)

53. More search dimensions • Realtime search • Local search • Universal search

54. Semantic search • Wikidata • YAGO • Conversational / Voice search

55. Outreach • Easy onboarding & docs • Making people care believe

56. Opportunities

57. Decentralization • YaCy • Extremely high technical & social cost! • Transparency?

58. Research • More people should know how to build search engines • Spam, Relevance, Large-scale data processing • We need more open datasets!

59. https://about.commonsearch.org/blog/

60. Make the Web a better place! • SEO • Transparency • Inﬂuence of money • Public service

61. Questions? https://about.commonsearch.org/contributing https://github.com/commonsearch contact@commonsearch.org slack.commonsearch.org

The original vision of Nutch, 14 years later: Building an open source search engine

Recommended

Recommended

More Related Content

What's hot

What's hot (14)

Similar to The original vision of Nutch, 14 years later: Building an open source search engine

Similar to The original vision of Nutch, 14 years later: Building an open source search engine (20)

More from Sylvain Zimmer

More from Sylvain Zimmer (12)

Recently uploaded

Recently uploaded (20)

The original vision of Nutch, 14 years later: Building an open source search engine