Get involved with the Apache Software Foundation Shalin Shekhar Mangar shalin [at] apache [dot] org

Who am I?

History 1996 – A ”patchy” web server 1999 – The Apache Software Foundation, Tomcat, Lucene 2002 – Nutch 2006 – Solr, Hadoop 2008 – Mahout

1996 – A ”patchy” web server

1999 – The Apache Software Foundation, Tomcat, Lucene

2002 – Nutch

2006 – Solr, Hadoop

2008 – Mahout

Today Apache HTTPD powers 65% of all servers and serves 100 million websites! Lucene powers search on thousands of web sites Hadoop powers AOL, Yahoo, Facebook. Runs on thousand node clusters So many projects! Thousands of active contributors

Apache HTTPD powers 65% of all servers and serves 100 million websites!

Lucene powers search on thousands of web sites

Hadoop powers AOL, Yahoo, Facebook. Runs on thousand node clusters

So many projects!

Thousands of active contributors

Why work on Apache/OSS? Work on what you like, when you like Development in the ”real” world Learn from the best Build a publicly verifiable resume Companies will find you!

Work on what you like, when you like

Development in the ”real” world

Learn from the best

Build a publicly verifiable resume

Companies will find you!

Problems we're solving Fast full-text search Application servers & frameworks Processing petabytes of data on thousands of unreliable commodity servers Crawling the web Scalable machine learning algorithms Data mining & analytics

Fast full-text search

Application servers & frameworks

Processing petabytes of data on thousands of unreliable commodity servers

Crawling the web

Scalable machine learning algorithms

Data mining & analytics

High performance, scalable, full text search library Focus: Indexing + Searching Documents 100% Java, no dependencies No crawlers or document parsing Users: Wikipedia, Technorati, SourceForge, … Applications: Eclipse, Jira, Nutch, Solr, many commercial products

High performance, scalable, full text search library

Focus: Indexing + Searching Documents

100% Java, no dependencies

No crawlers or document parsing

Users: Wikipedia, Technorati, SourceForge, …

Applications: Eclipse, Jira, Nutch, Solr, many commercial products

Lucene Inverted Index

Lucene Components Inverted Index Write once – merge in the background Query Types – Term, Boolean, Prefix, Range Scoring – TF, IDF, Length, Constant, Function Filtering

Inverted Index

Write once – merge in the background

Query Types – Term, Boolean, Prefix, Range

Scoring – TF, IDF, Length, Constant, Function

Filtering

Lucene – Towards the future Near real-time search – Many engineering challenges Flexible indexing – Alternate file formats, data structures Updates – Common values & per-document Query Optimization Better language support

Near real-time search – Many engineering challenges

Flexible indexing – Alternate file formats, data structures

Updates – Common values & per-document

Query Optimization

Better language support

Search server built on Lucene Schema HTTP APIs Replication Distributed Search Caching Extensible with plugins

Search server built on Lucene

Schema

HTTP APIs

Replication

Distributed Search

Caching

Extensible with plugins

Solr – Towards the Future Near Real-Time Search & Replication Scale to hundreds of servers Scale to thousands of indexes on a single box Update documents Faster auto-complete component Field Collapsing Clustering, Spell Suggestions, Clickstream feedback

Near Real-Time Search & Replication

Scale to hundreds of servers

Scale to thousands of indexes on a single box

Update documents

Faster auto-complete component

Field Collapsing

Clustering, Spell Suggestions, Clickstream feedback

Distributed File System – HDFS Map/Reduce Job scheduler Reliably store petabytes of data Compute in parallel Detect/handle failures

Distributed File System – HDFS

Map/Reduce

Job scheduler

Reliably store petabytes of data

Compute in parallel

Detect/handle failures

Map/Reduce map(key1,value) -> list<key2,value2> reduce(key2, list<value2>) -> list<value3> A large number of problems can be solved in this functional way Sort, Word Count, PageRank, Deduplication Data mining, co-occurence analysis

map(key1,value) -> list<key2,value2>

reduce(key2, list<value2>) -> list<value3>

A large number of problems can be solved in this functional way

Sort, Word Count, PageRank, Deduplication

Data mining, co-occurence analysis

Hadoop Map/Reduce

Hadoop – Towards the Future Better job scheduling, resource sharing Hadoop Workflow systems Hbase – Large databases in the cloud Performance improvements Hundreds more!

Better job scheduling, resource sharing

Hadoop Workflow systems

Hbase – Large databases in the cloud

Performance improvements

Hundreds more!

How do I start? Choose your project Join the mailing list or forum Check out the code Find open issues and feature requests Ask the developers on what you can work on

Choose your project

Join the mailing list or forum

Check out the code

Find open issues and feature requests

Ask the developers on what you can work on

Contributing Ideas! Features & Bug fixes Unit tests Documentation Performance benchmarks

Ideas!

Features & Bug fixes

Unit tests

Documentation

Performance benchmarks

Do's and Don'ts dnt rite sms lingo! Be courteous Don't be an island. Collaborate. Learn from your mistakes Persevere. It takes time.

dnt rite sms lingo!

Be courteous

Don't be an island. Collaborate.

Learn from your mistakes

Persevere. It takes time.

Questions? Shalin Shekhar Mangar shalin [at] apache [dot] org http://twitter.com/shalinmangar http://shalinsays.blogspot.com