• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Building specialized industry applications using Solr, and migration from FAST ESP
 

Building specialized industry applications using Solr, and migration from FAST ESP

on

  • 817 views

Uchida Spectrum, Inc. is a leader in the Japan search market. USI provides SMART InSight, a search application used by many Fortune 500 companies for specialized industry applications like R&D and ...

Uchida Spectrum, Inc. is a leader in the Japan search market. USI provides SMART InSight, a search application used by many Fortune 500 companies for specialized industry applications like R&D and quality assurance for manufacturing, claims and customer management etc.

Statistics

Views

Total Views
817
Views on SlideShare
817
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Building specialized industry applications using Solr, and migration from FAST ESP Building specialized industry applications using Solr, and migration from FAST ESP Presentation Transcript

    • Building  specialized  applica/ons  using  Solr;     Migra/on  from  FAST  ESP   Rahul  Agarwalla   Head  of  Interna/onal  Business   Uchida  Spectrum  Inc.  ©2011 Uchida Spectrum, Inc. All rights reserved.
    • Uchida  Spectrum  Overview   SoDware  License  Business   1995  ~   •  So)ware  License  Sales   •  License  Management  Repor:ng   •  License  Procurement  System   •  License  Adjustment  Consul:ng   Network  Technology  Services   Enterprise  Search  Business   1997  ~   •  Network  System  Consul:ng  Services   2002  ~   ― Ac:ve  Directory  Network       •  Enterprise  Intelligence  Applica:on   ― Exchange  Messaging  Network     ―  SMART  InSight  G2  Enterprise   •  License  Management  System  Consul:ng   ―  SMART  InSight  G2  Professional   ― So)ware  Management  Server     •  Search  PlaRorm  Consul:ng  &  Support   •  Portal  System  Consul:ng   ―  FAST  ESP   ― Share  Point  Portal  Server     ―  Lucene/Solr   ― Websphere  Portal  Server     ―  Lucid  Works  Enterprise    ©2011 Uchida Spectrum, Inc. All rights reserved. Page-2 Page-2
    • Some  of  Uchida  Spectrum’s  customers  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-3 Page-3
    • SMART/InSight  History   Customers  in  Japan,  China  &  India:     •  2  of  top  3  Japanese  car  manufacturers   •  Top  consumer  electronics  company   •  Large  financial  ins8tu8ons   •  China’s  biggest  eCommerce  firm   2005:     SMART  InSight  1.1   2004:  PlaRorm  for   custom  solu:ons   2003:     FAST  Alliance  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-4 Page-4
    • What  is  today’s  buzz  word?   Smart Phone • Extreme  scalability   • Flexibility  &  Extensibility   • Feature  rich  search  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-5 Page-5
    • What  I  learnt  from  the   Japan  catastrophe  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-6 Page-6
    • The  power  of  community   Japanese  Government   Japanese  People   [Closed/big  brother]   [Open  community]   •  Slow,  behind  the  curve   •  Quick  response   •  Legacy/CYA   •  Disclose  /  Share   •  Confusion   •  Prac:cal  Impact   Power shift Driver  of  innova/on  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-7 Page-7
    • Lessons  from  FAST  ESP  Migra/on:  advantage  LWE/Solr   •  Key  Issues:   1.  Smaller  record  and  index  size  enable  faster  index  maintenance   2.  #  of  records  per  node:  rule  of  thumb  10m  vs.  2m   3.  Licensing  &  Maintenance  Cost:  less  than  ½   •  Scalability:  5x   •  Cost  Performance:  10x   •  High  Flexibility   •  Lower  Opera/ons  Cost   •  Faster  Innova/on  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-8
    • Enterprise  Search  expecta/ons   •  Big  data  scale   •  Security  is  important   •  Disparate  data:  geography,  systems,   languages,  format,  structures   •  KM  is  good  to  have,  databases  are   cri:cal   •  Support  different  users  &  usage:   department,  role,  tasks   •  High  recall  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-9 Page-9
    • Lessons  from  FAST  ESP  Migra/on:  Filling  the  gaps   •  Security   •  ACL  security:  complex  requirements     •  File  System:  file  &  folder  level  control   •  CRM/ERP…  :  Keeping  ACLs  up-­‐to-­‐date   •  Content  aggrega/on   •  Connectors   •  Normaliza:on   •  Open  source  op:ons  for  ESP  pipeline   •  Openpipeline   •  Pypes  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-10
    • Building  specialized  applica/ons:  Content  fusion   •  Content  fusion  from  disparate  data:     •  Single  index  ≠  integra:on   •  Modeling  of  content  rela:onships  is  essen:al  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-11
    • Virtual  integra/on  based  on  search   Applica/on  layer   Content  sets  and  inter-­‐rela/onships     Content store Big  table,  flat  index   Search Index Search Index Search Index©2011 Uchida Spectrum, Inc. All rights reserved. Page-12
    • Virtual  integra/on  based  on  search…2   Search  Service   Content   Append  Pipeline   Tagging  Pipeline   Result  Pipeline   Query  Pipeline   Security   •  Data  transforma:on:       .  .  .  .  .  .   - key:key,  key:value,  field  names   Boos:ng   •  Query  &  Result  transforma:on   Transform   •  Boos:ng  /  Relevancy  algorithm   •  Security   •  Mul:-­‐Language  support   LWE  Adapter   SolrAdapter   ……   Other   •  Federa:on  &  mashups     Search Index ……   LWE Solr©2011 Uchida Spectrum, Inc. All rights reserved. Page-13
    • Building  specialized  applica/ons:  Personaliza/on   •  Applica/on  flow  depends  on  the  task     •  Data  Personaliza/on  increases  produc/vity   •  SMART  InSight  approach:  Task  based  UI   •  Schema  independent  widgets  for  analy:cs  &   visualiza:on     •  Portalized   •  Personalized:  widgets,  func:ons,  content,  fields  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-14
    • Knowledge  Center:  made  possible  by  Solr   Scalability  and  low  TCO  gives  us  ability  to  build  new  features   •  Knowledge  Centre  has  logs  of  all  user  ac:vity  in  SMART  InSight   •  This  would  be  too  costly  with  a  commercial  Search  Engine  and  would     not  be  feasible  in  a  database   Using  this  rich  data  we  can:   •  Profile  users,  groups  and  networks   •  Personalize  Recommenda:ons   •  Create  social  ranking  algorithms   •  Usage  analy:cs  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-15
    • Overview  of  SMART  InSight  for  Automo/ve Task  based  UIs   NHTSA   Internet   Page Widgets Ajax  Portal Personaliza/on Benchmarking   EDR Virtual  Integra/on   Convergent  Knowledge   Repair   Framework Framework   Dealers   Knowledge   SA   Contents  Set Centre   Recommend Data  Chain SA   Design Claims Profiling Parts  Catalog     SA   Engineering Metadata Analysis PLM   SA   Claims Specs Knowledge  Log CAD   Internal   Management  &  Security Early  Defect    Warning   Content  Model   Claim  Analysis  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-16
    • :  Interac/ve  Click  Log  Analysis  System • >  $50  Billion  sales  /  year   • >  800  Million  Items   • >  370  Million  Users     • Billions  of  clicks  per  day   Access  Log Solr Hadoop Solr,  Hadoop  +  SMART/InSight  G2 xxxxxxxx Xxxx Xxxx Xxxx Xxxx xxxx ©2011 Uchida Spectrum, Inc. All rights reserved. Page-17
    • :  Global  Research  Community   •  Top  Academic  Ins/tutes:     •  Faculty,  Research  Fellows  &  Post   graduate  students   •  Govt.  Departments  &  Corporate  R&D   •  Scien:sts  and  researchers   Research  Discovery  &   Collec/ve  Intelligence   (Knowledge  Centre)   •  >  270  content  sources:  Socie/es,   Broadcast   Associa/ons,  Publishers  &  Open   Search   •  IEEE,  ACM…   •  Elsevier,  Wiley,  Springer…   Dynamic  Result  Merging   Real  /me  indexing   Solr ©2011 Uchida Spectrum, Inc. All rights reserved. Page-18
    • Demonstra/on  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-19
    • Contact  Details   Rahul  Agarwalla   Head  –  Interna/onal  Business   rahul@spectrum.co.jp   www.spectrum.co.jp  ©2011 Uchida Spectrum, Inc. All rights reserved. Page-20