This document discusses enhancing site search with cognitive APIs. It describes taking structured and unstructured data from RSS feeds and indexing it in Cloudant for search. The Alchemy Language API is used to extract entities and taxonomy from article text to further index the data. This enriched indexing allows for more semantic search capabilities beyond keyword search. A demo news website is built with Node-RED to demonstrate the full workflow of ingesting RSS data, indexing with Cloudant and Alchemy, and building a front end for search and browsing.

1. Enhanced Site Search with
Cognitive APIs
Glynn Bird
Developer Advocate @ IBM Cloud Data Services
glynn.bird@uk.ibm.com
@glynn_bird

2. ●What is search?
●Simple Search
●Adding some "cognitive"
Agenda

3. @glynn_bird
Primary search

4. @glynn_bird
In-site search

5. @glynn_bird
Elasticsearch
• Stores JSON Documents
• Search based on Apache Lucene
• Provides HTTP search API
• Pay per-GB on compose.com

6. @glynn_bird
Cloudant
• Stores JSON Documents
• Based on Apache CouchDB
• Search based on Apache Lucene
• Provides HTTP search API
• PAYG/Dedicated-as-a-service or Local

7. @glynn_bird
Get started - Simple Search Service
https://developer.ibm.com/clouddataservices/simple-search-service/

8. @glynn_bird
Game of Thrones search demo
http://sss-got-theme.mybluemix.net/

9. @glynn_bird
Structured vs Unstructured Data
Structured Data
● known schema
● predictable
● indexable
Unstructured Data
● unknown schema
● difficult to parse and
index
DB

10. @glynn_bird
Example data
{
"url": "http://www.bbc.co.uk/news/business-37742991",
"title": "AT&T announces it will buy Time Warner",
"description": "US telecoms giant AT&T announces it will buy entertainment group Time Warner",
"date": "2016-10-22T23:44:03.000Z",
"image_url": "http://c.files.bbci.co.uk/_91950162_breaking_image_large-3-1.png"
}

11. @glynn_bird
Structured data
{
"url": "http://www.bbc.co.uk/news/business-37742991",
"title": "AT&T announces it will buy Time Warner",
"description": "US telecoms giant AT&T announces it will buy entertainment group Time Warner",
"date": "2016-10-22T23:44:03.000Z",
"image_url": "http://c.files.bbci.co.uk/_91950162_breaking_image_large-3-1.png"
}

12. @glynn_bird
Unstructured data
{
"url": "http://www.bbc.co.uk/news/business-37742991",
"title": "AT&T announces it will buy Time Warner",
"description": "US telecoms giant AT&T announces it will buy entertainment group Time Warner",
"date": "2016-10-22T23:44:03.000Z",
"image_url": "http://c.files.bbci.co.uk/_91950162_breaking_image_large-3-1.png"
}

13. @glynn_bird
Let's build news website
● take RSS feeds
● put the data into a database
● index it
○ newest articles first
○ keyword search

14. @glynn_bird
Node-RED
● visual programming tool
● https://nodered.org/

15. @glynn_bird
Indexing data in Cloudant - MapReduce
function(doc) {
emit(doc.date, doc.title);
}
● Build index sort articles by date
● Create custom 'map' function

16. @glynn_bird
Indexing data in Cloudant - MapReduce

17. @glynn_bird
Front end

18. @glynn_bird
Indexing data in Cloudant - Search
function(doc) {
index('default', doc.title);
index('default', doc.description);
}
● Build full-text index
● Create custom 'map' function

19. @glynn_bird
Cloudant Search
● Punctuation removal
● Word splitting/stemming
● Stop-word removal
● Full-text indexing using Apache Lucene

20. @glynn_bird
Front end

21. @glynn_bird
Front end

22. @glynn_bird
Summary so far...

23. @glynn_bird
But can we do better?

24. @glynn_bird
Watson Alchemy Language API
● Feed it text or a URL
● Returns:
○ entities - people/places/companies
○ taxonomy

25. @glynn_bird
Watson Alchemy Language API
Entities
Country: US
Company: AT&T
Company: Time Warner
JobTitle: Telecoms
Taxonomy
/art and entertainment
/technology and computing/internet technology/isps
/business and industrial/company/merger and acquisition

26. @glynn_bird
How can we use Alchemy in our workflow?

27. @glynn_bird
How can we use Alchemy in our workflow?

28. @glynn_bird
More indexing
● Index the Alchemy entities
○ e.g. Country:US
● Index the Alchemy taxonomy
○ e.g. ["Finance","Investing"]

29. @glynn_bird
Front end

30. @glynn_bird

31. @glynn_bird

32. @glynn_bird

33. @glynn_bird
Demo
https://glynnbird.github.io/alchemy-news/

34. @glynn_bird
It's not just language...

35. @glynn_bird
Watson saw….

36. @glynn_bird
Just one more

37. @glynn_bird
Watson saw...

38. @glynn_bird

39. @glynn_bird

40. @glynn_bird

41. Summary
● Node-RED
● Cloudant
● Alchemy Language API
Bluemix: https://www.ibm.com/cloud-computing/bluemix/
Simple Search Service: https://developer.ibm.com/clouddataservices/simple-search-service/
News Demo: https://glynnbird.github.io/alchemy-news/

42. Developer Advocate
glynn.bird@uk.ibm.com
Thanks
Glynn Bird
Blog: www.glynnbird.com
Twitter: @glynn_bird