Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (17)
Similar to Big data - Solr Integration
Similar to Big data - Solr Integration (20)
More from rkulandaivel
Recently uploaded
Recently uploaded (20)
Big data - Solr Integration
- 1. BIG DATA Kulandaivel Ramalingam ESB Architect
- 2. Solr create directory eg: velsearch create directory solr_vel under velsearch copy solrconfig.xml and schema.xml to solr_vel Create Instance Directory eg: vel-search : solrctl instancedir --create vel-search solr_vel/ The below output would appear: Uploading configs from solr_vel//conf to <hostname>:<port>/solr. This may take up to a minute.
- 3. Solr Create collection: solrctl collection --create vel-search -s 1 Now data directory created in HDFS: Instance: /var/lib/solr/vel-search_shard1_replica1 Data: hdfs://<hostname>:<port>/solr/vel-search/core_node1/data Index: hdfs://<hostname>:<port>/solr/vel-search/core_node1/data/index
- 4. Solr Generate data in csv format from Hive database Make sure that "" in each column is appeared as in the example below: id,name,location "1","Vel","India" "2","Ram","US" "3","Kul","UK"
- 5. Solr Upload it to HDFS Ensure that reviews.conf has the below values: commands : [ { readCSV { separator : "," columns : [id,name,location] ignoreFirstLine : true quoteChar : """ trim : true charset : UTF-8 } }
- 6. Solr Run the below indexing command: hadoop jar /usr/lib/solr/contrib/mr/search-mr-*-job.jar org.apache.solr.hadoop.MapReduceIndexerTool -D 'mapred.child.java.opts=-Xmx500m' --log4j /usr/share/doc/search*/examples/solr-nrt/ log4j.properties --morphline-file reviews.conf -- output-dir hdfs://<hostname>:<port>/tmp/load -- verbose --go-live --zk-host <hostname>:<port>/solr -- collection vel-search hdfs://<hostname>:<port>/user/cloudera/query_result_ copy.csv
- 7. Solr The below message wil appear at the end of the executionof the above command: 37097 [main] INFO org.apache.solr.client.solrj.impl.HttpClientUtil - Creating new http client, config: 37119 [main] INFO org.apache.solr.common.cloud.ConnectionManager - Waiting for client to connect to ZooKeeper 37121 [main-EventThread] INFO org.apache.solr.common.cloud.ConnectionManager - Watcher org.apache.solr.common.cloud.ConnectionManager@dd606a name:ZooKeeperConnection Watcher:localhost:2181/solr got event WatchedEvent state:SyncConnected type:None path:null path:null type:None 37121 [main] INFO org.apache.solr.common.cloud.ConnectionManager - Client is connected to ZooKeeper 37122 [main] INFO org.apache.solr.common.cloud.ZkStateReader - Updating cluster state from ZooKeeper... 37957 [main] INFO org.apache.solr.hadoop.GoLive - Done committing live merge 37958 [main] INFO org.apache.solr.hadoop.GoLive - Live merging of index shards into Solr cluster took 2.032 secs 37958 [main] INFO org.apache.solr.hadoop.GoLive - Live merging completed successfully 37958 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Succeeded with job: jobName: org.apache.solr.hadoop.MapReduceIndexerTool/MorphlineMapper, jobId: job_1409686197310_0004 37958 [main] INFO org.apache.solr.hadoop.MapReduceIndexerTool - Success. Done. Program took 38.025 secs. Goodbye.
- 8. Solr Go to Solr admin for testing: http://192.168.137.134:8983/solr/#/vel-search_ shard1_replica1/query Give the primary key as query for search:
Editor's Notes
- This template can be used as a starter file for presenting training materials in a group setting. Sections Right-click on a slide to add sections. Sections can help to organize your slides or facilitate collaboration between multiple authors. Notes Use the Notes section for delivery notes or to provide additional details for the audience. View these notes in Presentation View during your presentation. Keep in mind the font size (important for accessibility, visibility, videotaping, and online production) Coordinated colors Pay particular attention to the graphs, charts, and text boxes. Consider that attendees will print in black and white or grayscale. Run a test print to make sure your colors work when printed in pure black and white and grayscale. Graphics, tables, and graphs Keep it simple: If possible, use consistent, non-distracting styles and colors. Label all graphs and tables.
- Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
- Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
- Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
- Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
- Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
- Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
- Give a brief overview of the presentation. Describe the major focus of the presentation and why it is important. Introduce each of the major topics. To provide a road map for the audience, you can repeat this Overview slide throughout the presentation, highlighting the particular topic you will discuss next.
- Microsoft Confidential