Trend Detection and Visualization and Custom Search Applications

2,483 views

Published on

This seminar deals with Trend detection in numbers and text and its visualization. In the second part, it focuses on Custom Search Application, Apache Solr, Semantic search and Linked data approach.

Published in: Travel, Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,483
On SlideShare
0
From Embeds
0
Number of Embeds
651
Actions
Shares
0
Downloads
30
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Datalog is a query and rule language for deductive databases that syntactically is a subset of Prolog
  • Trend Detection and Visualization and Custom Search Applications

    1. 1. Trend Detection and Visualization and Custom Search Applications Seminar for PG PUSHPIN Pranav Kadam (6641525) Universität Paderborn January 12, 2012
    2. 2. Overview • Trend Detection Trend Detection in Numbers Trend Detection in Text Trend Visualization • Custom Search Applications Apache Solr Semantic Search Linked Data ApproachTrend Detection and Visualization and Custom Search Applications 2
    3. 3. Overview • Prototypes • Q&ATrend Detection and Visualization and Custom Search Applications 3
    4. 4. Trend DetectionTrend Detection and Visualization and Custom Search Applications 4
    5. 5. Trend Detection What is a trend? • A general direction in which something is changing • An inclination • A pattern of gradual change in a condition over time • A trend is always associated with time often described using ‘time series‘ • Long term change in the mean level of a ‘time series‘.Trend Detection and Visualization and Custom Search Applications 5
    6. 6. Trend Detection Trend Analysis • Practice of collecting information and trying to detect trend in it • Process of identifying pattern in behavior of a time series by minimising noise • Useful in forecasting future events • Science of studying changes in social patterns E.g. Google Trends, Youtube Trends, trendwatching.com, Facebook Insights, Tag Cloud(on PG PUSHPIN blog)Trend Detection and Visualization and Custom Search Applications 6
    7. 7. Trend Detection Trend Detection in NumbersTrend Detection and Visualization and Custom Search Applications 7
    8. 8. Trend Detection in Numbers Time series and statistical methods • Time series: ordered sequence of values at equally spaced time intervals • Trend detection in numbers: Statistical methods to interpret time series and determine behavior • Assumption: pattern in past data can be used to forecast future data points • Models: AutoRegressive(AR), Integrated(I), Moving Average(MA)Trend Detection and Visualization and Custom Search Applications 8
    9. 9. Trend Detection in Numbers Moving Average • Average of time series data taken at consecutive periods • New data in, old data out as the series progresses E.g. MA of temperature for six months: Temp from January to June, February to July, March to August, and so on. • Minimizes temporal fluctuations • Establishes trend, distinguishes any value above or below trendline • Applications in fields of Financial analysis, Trade, Economics, MathematicsTrend Detection and Visualization and Custom Search Applications 9
    10. 10. Trend Detection in Numbers Moving Average • Simple Moving Average: Plain average of data points over specific no. of periods • Period selected can be short, medium or long according to interest (E.g. standard periods of SMA for stock market analysis is 50 days or 200 days) • Longer the period gives smoother curve but increases the lag • SMA always lags behind the latest data pointTrend Detection and Visualization and Custom Search Applications 10
    11. 11. Trend Detection in Numbers Moving Average • Exponential Moving Average: Weight applied to the data pointa to reduce the lag • Weight decreases exponentially and never reaches zero • EMA has less lag and is more sensitive to the changes in data points • SMA vs EMA: Though difference is apparent, either one cannot be stated as better over the other MA preference depends on objectives & time horizonTrend Detection and Visualization and Custom Search Applications 11
    12. 12. Trend Detection Trend Detection in TextTrend Detection and Visualization and Custom Search Applications 12
    13. 13. Trend Detection in Text Trend detection system • Emerging Trend: Topic area growing in interest and utility over time • Study of emerging trend dependent on automated process • TD system processes collection of textual data and identifies upward(growing), downward(falling) or sideway(constant) tendency • TD then highlights the emerging topics in trial periodTrend Detection and Visualization and Custom Search Applications 13
    14. 14. Trend Detection in Text Trend detection system • Trend detection methods can be classified as: Fully-automatic Semi-automatic • Fully-automatic systems: It generates a list of emerging topics from the input(collection of texual data) Reviewer examines data & evidence provided to conclude actual emerging trends Results supported with graphical visualizationTrend Detection and Visualization and Custom Search Applications 14
    15. 15. Trend Detection in Text Trend detection system • Semi-automatic: User inputs a topic System outputs the evidence that helps to determine that the topic is emerging or not Evidence provided either as a summary or a descriptive reportTrend Detection and Visualization and Custom Search Applications 15
    16. 16. Trend Detection in Text Useful models, schemes and tools • Term-Document Matrix • Scheme: Term Frequency – Inverse Document Frequency (tf-idf) • Latent Semantic Analysis • Science Citation Index or Web of Science database • Inspec, Compendex databaseTrend Detection and Visualization and Custom Search Applications 16
    17. 17. Trend Detection in Text Approches for Trend Detection 1. Tracing a trend via citation linkages: Determine a potential trend or select a topic of interest Find recent documents on the topic Examine whether they really discuss the topic Extract keywords Fetch abstract of the documents those are frequently referenced using citation information Examine abstract to verify relation with topicTrend Detection and Visualization and Custom Search Applications 17
    18. 18. Trend Detection in Text Approches for Trend Detection 1. Tracing a trend via citation linkages: Examine the references used above and make a subset where author names are referenced in more than, say, 3 documents As an improvement, query the repositories of citation linkage information and other sources Graph document frequency, repeated authors and no. of venues by yearTrend Detection and Visualization and Custom Search Applications 18
    19. 19. Trend Detection in Text Approches for Trend Detection 1. Tracing a trend via citation linkages: Years with overall higher document frequency are likely to have points where trend is emerging  Finally, to determine trend, apply a series of thresholds like atleast one repeated author, atleast 10 venues present, etc.Trend Detection and Visualization and Custom Search Applications 19
    20. 20. Trend Detection in Text Approches for Trend Detection 2. Using web resources: Select a main topic area first Knowledge in this area is essential to identify trends in later stages Validate it as a possible research area using sources like Inspec database Search workshop websites and technical papers for discussions on the main topic areaTrend Detection and Visualization and Custom Search Applications 20
    21. 21. Trend Detection in Text Approches for Trend Detection 2. Using web resources: Search web using helper terms like most recent contribution, hot topic, cutting edge strategy, etc Again search an indexing database with main topic ‘AND‘ newly found candiate trend from year of origin to current yearTrend Detection and Visualization and Custom Search Applications 21
    22. 22. Trend Detection in Text Approches for Trend Detection 2. Using web resources:  If document frequency increases over the years, the candidate trend is a genuine trend x If documents from same author appear in different years its not a trendTrend Detection and Visualization and Custom Search Applications 22
    23. 23. Trend Detection Trend VisualizationTrend Detection and Visualization and Custom Search Applications 23
    24. 24. Trend Visualization Trend visualization techniques • Trends can be visualized using Line graphs Bar graphs Word clouds Frequency tables Sparklines HistogramsTrend Detection and Visualization and Custom Search Applications 24
    25. 25. Trend Visualization Other ways to visualize trends • ThemeRiver Visualizes thematic variations over time Changing widths depict changes in thematic strength of the associated documents Flow represents time Colors represent themes Vertical section represents an ordered time sliceTrend Detection and Visualization and Custom Search Applications 25
    26. 26. Trend Visualization Other ways to visualize trends • ThemeRiverTrend Detection and Visualization and Custom Search Applications 26
    27. 27. Trend Visualization Other ways to visualize trends • ThemeRiver Assigning same color group to related themes simplify its trackingTrend Detection and Visualization and Custom Search Applications 27
    28. 28. Trend Visualization Other ways to visualize trends • SparkClouds SparkClouds= Sparklines + Tag Clouds Sparkline, characterized by small size and high data density, visualize trends and variations in a simple condensed wayTrend Detection and Visualization and Custom Search Applications 28
    29. 29. Trend Visualization Other ways to visualize trends • SparkClouds Tag clouds are text based visualizations showing frequency, popularity or importance of wordsTrend Detection and Visualization and Custom Search Applications 29
    30. 30. Trend Visualization Other ways to visualize trends • SparkClouds Sparklines are added to tag clouds to represent trend across series of tag clouds Overview of trends provided in limited space Its compact and aestheticTrend Detection and Visualization and Custom Search Applications 30
    31. 31. Custom Search ApplicationsTrend Detection and Visualization and Custom Search Applications 31
    32. 32. Custom Search Application Apache Solr • Open source search platform from Apache Lucene project • Provides full text search, faceted search, dynamic clustering, database integration, rich document handling, geo-spatial search • High scalability, distributed search • The core of search and navigation engine of some of the world‘s largest internet sitesTrend Detection and Visualization and Custom Search Applications 32
    33. 33. Custom Search Application Apache Solr • Written in Java, runs as a standalone search server within a servlet container like Jetty or Tomcat • REST-like API eases its use with any prog. language • Input: XML, JSON or binary over HTTP(GET) • Output: XML, JSON or binary • Highly customizableTrend Detection and Visualization and Custom Search Applications 33
    34. 34. Custom Search Application Apache Solr • Operations: Indexing data Updating data Deleting data Querying data Sorting Higlighting Faceted searchTrend Detection and Visualization and Custom Search Applications 34
    35. 35. Custom Search Application Semantic Web • An extension to current Web • Information is given well-defined meaning • Goes beyond media objects to link people, places, events, organizations, etc. • Resources connected by multiple relations • Data modeled using directed labeled graph • Based on W3C‘s RDF, it does quering and exchanging instance data in RDF using SOAPTrend Detection and Visualization and Custom Search Applications 35
    36. 36. Custom Search Application Semantic Web 9°C temp located in type USA City San Francisco Apple Inc. birth place Steve Jobs type Company Businessman died on Pixar February 24, 1955 October 5, 2011Trend Detection and Visualization and Custom Search Applications 36
    37. 37. Custom Search Application Semantic Search • Context-based search results • Can possibly enhance, but cannot replace the traditional navigational search • Disambiguation • Data divided as ontological data and instance data • Determines meaning of every word and establishing a context between them to achieve coherence for a sentenceTrend Detection and Visualization and Custom Search Applications 37
    38. 38. Custom Search Application Semantic Search • Search Methodologies: RDF Path Traversal Keyword Concept Mapping Graph Patterns Logics Fuzzy Concepts, Fuzzy Relations, Fuzzy Logics • Examples Hakia, SenseBot, DeepDyveTrend Detection and Visualization and Custom Search Applications 38
    39. 39. Custom Search Application Linked Data Approach • Linked data: method of publishing structured data that can be interlinked • Based on HTTP and URIs, extended to be read by computers • Components: URIs HTTP RDF Serialization formats (RDFa, RDF/XML, N3)Trend Detection and Visualization and Custom Search Applications 39
    40. 40. Custom Search Application Linked Data Approach • KiWi – a Linked Media Framework • Easy to setup server application bundling Semantic Web technologies • Consists of LMF core and LMF modulesTrend Detection and Visualization and Custom Search Applications 40
    41. 41. Custom Search Application Linked Data Approach • KiWi LMF core: Use URIs as names for things. Use HTTP URIs, so that people can look up those names. When someone looks up a URI, provide useful information, using the standards (RDF, SPARQL). Include links to other URIs, so that they can discover more things.Trend Detection and Visualization and Custom Search Applications 41
    42. 42. Custom Search Application Linked Data Approach • KiWi LMF module: LMF Semantic Search(highly configurable Semantic Search service based on Apache SOLR) LMF Linked Data Cache (implements a cache to the Linked Data Cloud) LMF Reasoner (implements a rule-based reasoner that allows to process Datalog-style rules over RDF triples)Trend Detection and Visualization and Custom Search Applications 42
    43. 43. PrototypesTrend Detection and Visualization and Custom Search Applications 43
    44. 44. Questions and AnswersTrend Detection and Visualization and Custom Search Applications 44
    45. 45. Thank you!Trend Detection and Visualization and Custom Search Applications 45

    ×