In the talk I describe two approaches for improve the recall and precision of an enterprise search engine using machine learning techniques. The main focus is improving relevancy with ML while using your existing search stack, be that Luce, Solr, Elastic Search, Endeca or something else.
Personalized Search at Sandia National LabsLucidworks
Clay Pryor, R&D S&E, Computer Science & Ryan Cooper, Sandia National Labs. Presentation from ACTIVATE 2019, the Search and AI Conference hosted by Lucidworks. http://www.activate-conf.com
Slides: Concurrent Inference of Topic Models and Distributed Vector Represent...Parang Saraf
Abstract: Topic modeling techniques have been widely used to uncover dominant themes hidden inside an unstructured document collection. Though these techniques first originated in the probabilistic analysis of word distributions, many deep learning approaches have been adopted recently. In this paper, we propose a novel neural network based architecture that produces distributed representation of topics to capture topical themes in a dataset. Unlike many state-of-the-art techniques for generating distributed representation of words and documents that directly use neighboring words for training, we leverage the outcome of a sophisticated deep neural network to estimate the topic labels of each document. The networks, for topic modeling and generation of distributed representations, are trained concurrently in a cascaded style with better runtime without sacrificing the quality of the topics. Empirical studies reported in the paper show that the distributed representations of topics represent intuitive themes using smaller dimensions than conventional topic modeling approaches.
For more information, please visit: http://people.cs.vt.edu/parang/ or contact parang at firstname at cs vt edu
Dice.com Bay Area Search - Beyond Learning to Rank TalkSimon Hughes
This talk describes how to implement conceptual search (semantic search) within a modern search engine using the word2vec algorithm to learn concepts. We also cover how to auto-tune the search engine parameters using black box optimization techniques, and the problems of feedback loops encountered when building machine learning systems that modify the user behavior used to train the system.
Personalized Search at Sandia National LabsLucidworks
Clay Pryor, R&D S&E, Computer Science & Ryan Cooper, Sandia National Labs. Presentation from ACTIVATE 2019, the Search and AI Conference hosted by Lucidworks. http://www.activate-conf.com
Slides: Concurrent Inference of Topic Models and Distributed Vector Represent...Parang Saraf
Abstract: Topic modeling techniques have been widely used to uncover dominant themes hidden inside an unstructured document collection. Though these techniques first originated in the probabilistic analysis of word distributions, many deep learning approaches have been adopted recently. In this paper, we propose a novel neural network based architecture that produces distributed representation of topics to capture topical themes in a dataset. Unlike many state-of-the-art techniques for generating distributed representation of words and documents that directly use neighboring words for training, we leverage the outcome of a sophisticated deep neural network to estimate the topic labels of each document. The networks, for topic modeling and generation of distributed representations, are trained concurrently in a cascaded style with better runtime without sacrificing the quality of the topics. Empirical studies reported in the paper show that the distributed representations of topics represent intuitive themes using smaller dimensions than conventional topic modeling approaches.
For more information, please visit: http://people.cs.vt.edu/parang/ or contact parang at firstname at cs vt edu
Dice.com Bay Area Search - Beyond Learning to Rank TalkSimon Hughes
This talk describes how to implement conceptual search (semantic search) within a modern search engine using the word2vec algorithm to learn concepts. We also cover how to auto-tune the search engine parameters using black box optimization techniques, and the problems of feedback loops encountered when building machine learning systems that modify the user behavior used to train the system.
Slides from Enterprise Search & Analytics Meetup @ Cisco Systems - http://www.meetup.com/Enterprise-Search-and-Analytics-Meetup/events/220742081/
Relevancy and Search Quality Analysis - By Mark David and Avi Rappoport
The Manifold Path to Search Quality
To achieve accurate search results, we must come to an understanding of the three pillars involved.
1. Understand your data
2. Understand your customers’ intent
3. Understand your search engine
The first path passes through Data Analysis and Text Processing.
The second passes through Query Processing, Log Analysis, and Result Presentation.
Everything learned from those explorations feeds into the final path of Relevancy Ranking.
Search quality is focused on end users finding what they want -- technical relevance is sometimes irrelevant! Working with the short head (very frequent queries) has the most return on investment for improving the search experience, tuning the results, for example, to emphasize recent documents or de-emphasize archive documents, near-duplicate detection, exposing diverse results in ambiguous situations, using synonyms, and guiding search via best bets and auto-suggest. Long-tail analysis can reveal user intent by detecting patterns, discovering related terms, and identifying the most fruitful results by aggregated behavior. all this feeds back into the regression testing, which provides reliable metrics to evaluate the changes.
By merging these insights, you can improve the quality of the search overall, in a scalable and maintainable fashion.
This tutorial gives an overview of how search engines and machine learning techniques can be tightly coupled to address the need for building scalable recommender or other prediction based systems. Typically, most of them architect retrieval and prediction in two phases. In Phase I, a search engine returns the top-k results based on constraints expressed as a query. In Phase II, the top-k results are re-ranked in another system according to an optimization function that uses a supervised trained model. However this approach presents several issues, such as the possibility of returning sub-optimal results due to the top-k limits during query, as well as the prescence of some inefficiencies in the system due to the decoupling of retrieval and ranking.
To address this issue the authors created ML-Scoring, an open source framework that tightly integrates machine learning models into Elasticsearch, a popular search engine. ML-Scoring replaces the default information retrieval ranking function with a custom supervised model that is trained through Spark, Weka, or R that is loaded as a plugin in Elasticsearch. This tutorial will not only review basic methods in information retrieval and machine learning, but it will also walk through practical examples from loading a dataset into Elasticsearch to training a model in Spark, Weka, or R, to creating the ML-Scoring plugin for Elasticsearch. No prior experience is required in any system listed (Elasticsearch, Spark, Weka, R), though some programming experience is recommended.
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...S. Diana Hu
Search engines have focused on solving the document retrieval problem, so their scoring functions do not handle naturally non-traditional IR data types, such as numerical or categorical. Therefore, on domains beyond traditional search, scores representing strengths of associations or matches may vary widely. As such, the original model doesn’t suffice, so relevance ranking is performed as a two-phase approach with 1) regular search 2) external model to re-rank the filtered items. Metrics such as click-through and conversion rates are associated with the users’ response to items served. The predicted selection rates that arise in real-time can be critical for optimal matching. For example, in recommender systems, predicted performance of a recommended item in a given context, also called response prediction, is often used in determining a set of recommendations to serve in relation to a given serving opportunity. Similar techniques are used in the advertising domain. To address this issue the authors have created ML-Scoring, an open source framework that tightly integrates machine learning models into a popular search engine (SOLR/Elasticsearch), replacing the default IR-based ranking function. A custom model is trained through either Weka or Spark and it is loaded as a plugin used at query time to compute custom scores.
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch
The hallmark of a great search experience is always delivering the most relevant results, quickly, to every user. The difficulty lies behind the scenes in making that happen elegantly and at a scale. From App Search’s intuitive drag and drop interface to the advanced relevance capabilities built into the core of Elasticsearch — Elastic offers a range of tools for developers to tune relevance ranking and create incredible search experiences. In this session, we’ll explore some of Elasticsearch’s advanced relevance ranking features, such as dense vector fields, BM25F, ranking evaluation, and more. Plus we’ll give you some ideas for how these features are being used by other Elastic users to create world-class, category defining search experiences.
Introduction to Enterprise Search. A two hour class to introduce Enterprise Search. It covers:
The problems enterprise search can solve
History of (web) search
How we search and find?
Current state of Enterprise Search + stats
Technical concept
Information quality
Feedback cycle
Five dimensions of Findability
DATA SCIENCE AND BIG DATA ANALYTICSCHAPTER 2 DATA ANA.docxrandyburney60861
DATA SCIENCE AND BIG DATA
ANALYTICS
CHAPTER 2:
DATA ANALYTICS LIFECYCLE
DATA ANALYTICS LIFECYCLE
• Data science projects differ from BI projects
• More exploratory in nature
• Critical to have a project process
• Participants should be thorough and rigorous
• Break large projects into smaller pieces
• Spend time to plan and scope the work
• Documenting adds rigor and credibility
DATA ANALYTICS LIFECYCLE
• Data Analytics Lifecycle Overview
• Phase 1: Discovery
• Phase 2: Data Preparation
• Phase 3: Model Planning
• Phase 4: Model Building
• Phase 5: Communicate Results
• Phase 6: Operationalize
• Case Study: GINA
2.1 DATA ANALYTICS
LIFECYCLE OVERVIEW
• The data analytic lifecycle is designed for Big Data problems and
data science projects
• With six phases the project work can occur in several phases
simultaneously
• The cycle is iterative to portray a real project
• Work can return to earlier phases as new information is uncovered
2.1.1 KEY ROLES FOR A
SUCCESSFUL ANALYTICS
PROJECT
KEY ROLES FOR A
SUCCESSFUL ANALYTICS
PROJECT
• Business User – understands the domain area
• Project Sponsor – provides requirements
• Project Manager – ensures meeting objectives
• Business Intelligence Analyst – provides business domain
expertise based on deep understanding of the data
• Database Administrator (DBA) – creates DB environment
• Data Engineer – provides technical skills, assists data
management and extraction, supports analytic sandbox
• Data Scientist – provides analytic techniques and modeling
2.1.2 BACKGROUND AND OVERVIEW
OF DATA ANALYTICS LIFECYCLE
• Data Analytics Lifecycle defines the analytics process and
best practices from discovery to project completion
• The Lifecycle employs aspects of
• Scientific method
• Cross Industry Standard Process for Data Mining (CRISP-DM)
• Process model for data mining
• Davenport’s DELTA framework
• Hubbard’s Applied Information Economics (AIE) approach
• MAD Skills: New Analysis Practices for Big Data by Cohen et al.
https://en.wikipedia.org/wiki/Scientific_method
https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining
http://www.informationweek.com/software/information-management/analytics-at-work-qanda-with-tom-davenport/d/d-id/1085869?
https://en.wikipedia.org/wiki/Applied_information_economics
https://pafnuty.wordpress.com/2013/03/15/reading-log-mad-skills-new-analysis-practices-for-big-data-cohen/
OVERVIEW OF
DATA ANALYTICS LIFECYCLE
2.2 PHASE 1: DISCOVERY
2.2 PHASE 1: DISCOVERY
1. Learning the Business Domain
2. Resources
3. Framing the Problem
4. Identifying Key Stakeholders
5. Interviewing the Analytics Sponsor
6. Developing Initial Hypotheses
7. Identifying Potential Data Sources
2.3 PHASE 2: DATA PREPARATION
2.3 PHASE 2: DATA
PREPARATION
• Includes steps to explore, preprocess, and condition
data
• Create robust environment – analytics sandbox
• Data preparation tends to be t.
Modern Perspectives on Recommender Systems and their Applications in MendeleyKris Jack
Presentation given for one of Pearson's Data Research teams. It motivates the use of recommender systems, describes common approaches to building and evaluating them and gives examples of how they are used in Mendeley. Thanks to Maya Hristakeva for creating some of the slides.
We live in a world of silos - separate systems each with data essential to our daily work. No organization has all its important information in one place - 61% of knowledge workers regularly access 4 or more systems to get the information they need to do their jobs, and 15% need 11 or more systems. Integration to provide a unified view across these systems is very valuable, but it has been difficult to accomplish - even between different Microsoft products. This seminar will show you how to bridge across these silos using a search-based approach that is both quick and powerful.
Solving Real World Challenges with Enterprise SearchSPC Adriatics
Enterprise Search is complex, even in theory. But when you implement your search solution and everything turns to reality, you’ll find some new, never-seen challenges. In this session, I’ll collect the best, biggest and most exciting challenges from my experience, including real world customer scenarios and solutions. Regardless of the SharePoint version you use (SharePoint 2010, FAST Search for SharePoint, SharePoint 2013), this session is for you if you want to prepare for these “unexpected” scenarios.
Evolving the Optimal Relevancy Ranking Model at Dice.comSimon Hughes
This is a talk about gathering a golden test set of relevancy judgements, either using manual annotators or search log mining, to use in either an automated or manual relevancy tuning process. We also discuss the dangers of positive feedback loops when building closed-loop machine learning models for search and recommendation.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
More Related Content
Similar to Enhancing Enterprise Search with Machine Learning - Simon Hughes, Dice.com
Slides from Enterprise Search & Analytics Meetup @ Cisco Systems - http://www.meetup.com/Enterprise-Search-and-Analytics-Meetup/events/220742081/
Relevancy and Search Quality Analysis - By Mark David and Avi Rappoport
The Manifold Path to Search Quality
To achieve accurate search results, we must come to an understanding of the three pillars involved.
1. Understand your data
2. Understand your customers’ intent
3. Understand your search engine
The first path passes through Data Analysis and Text Processing.
The second passes through Query Processing, Log Analysis, and Result Presentation.
Everything learned from those explorations feeds into the final path of Relevancy Ranking.
Search quality is focused on end users finding what they want -- technical relevance is sometimes irrelevant! Working with the short head (very frequent queries) has the most return on investment for improving the search experience, tuning the results, for example, to emphasize recent documents or de-emphasize archive documents, near-duplicate detection, exposing diverse results in ambiguous situations, using synonyms, and guiding search via best bets and auto-suggest. Long-tail analysis can reveal user intent by detecting patterns, discovering related terms, and identifying the most fruitful results by aggregated behavior. all this feeds back into the regression testing, which provides reliable metrics to evaluate the changes.
By merging these insights, you can improve the quality of the search overall, in a scalable and maintainable fashion.
This tutorial gives an overview of how search engines and machine learning techniques can be tightly coupled to address the need for building scalable recommender or other prediction based systems. Typically, most of them architect retrieval and prediction in two phases. In Phase I, a search engine returns the top-k results based on constraints expressed as a query. In Phase II, the top-k results are re-ranked in another system according to an optimization function that uses a supervised trained model. However this approach presents several issues, such as the possibility of returning sub-optimal results due to the top-k limits during query, as well as the prescence of some inefficiencies in the system due to the decoupling of retrieval and ranking.
To address this issue the authors created ML-Scoring, an open source framework that tightly integrates machine learning models into Elasticsearch, a popular search engine. ML-Scoring replaces the default information retrieval ranking function with a custom supervised model that is trained through Spark, Weka, or R that is loaded as a plugin in Elasticsearch. This tutorial will not only review basic methods in information retrieval and machine learning, but it will also walk through practical examples from loading a dataset into Elasticsearch to training a model in Spark, Weka, or R, to creating the ML-Scoring plugin for Elasticsearch. No prior experience is required in any system listed (Elasticsearch, Spark, Weka, R), though some programming experience is recommended.
RecSys 2015 Tutorial – Scalable Recommender Systems: Where Machine Learning...S. Diana Hu
Search engines have focused on solving the document retrieval problem, so their scoring functions do not handle naturally non-traditional IR data types, such as numerical or categorical. Therefore, on domains beyond traditional search, scores representing strengths of associations or matches may vary widely. As such, the original model doesn’t suffice, so relevance ranking is performed as a two-phase approach with 1) regular search 2) external model to re-rank the filtered items. Metrics such as click-through and conversion rates are associated with the users’ response to items served. The predicted selection rates that arise in real-time can be critical for optimal matching. For example, in recommender systems, predicted performance of a recommended item in a given context, also called response prediction, is often used in determining a set of recommendations to serve in relation to a given serving opportunity. Similar techniques are used in the advertising domain. To address this issue the authors have created ML-Scoring, an open source framework that tightly integrates machine learning models into a popular search engine (SOLR/Elasticsearch), replacing the default IR-based ranking function. A custom model is trained through either Weka or Spark and it is loaded as a plugin used at query time to compute custom scores.
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch
The hallmark of a great search experience is always delivering the most relevant results, quickly, to every user. The difficulty lies behind the scenes in making that happen elegantly and at a scale. From App Search’s intuitive drag and drop interface to the advanced relevance capabilities built into the core of Elasticsearch — Elastic offers a range of tools for developers to tune relevance ranking and create incredible search experiences. In this session, we’ll explore some of Elasticsearch’s advanced relevance ranking features, such as dense vector fields, BM25F, ranking evaluation, and more. Plus we’ll give you some ideas for how these features are being used by other Elastic users to create world-class, category defining search experiences.
Introduction to Enterprise Search. A two hour class to introduce Enterprise Search. It covers:
The problems enterprise search can solve
History of (web) search
How we search and find?
Current state of Enterprise Search + stats
Technical concept
Information quality
Feedback cycle
Five dimensions of Findability
DATA SCIENCE AND BIG DATA ANALYTICSCHAPTER 2 DATA ANA.docxrandyburney60861
DATA SCIENCE AND BIG DATA
ANALYTICS
CHAPTER 2:
DATA ANALYTICS LIFECYCLE
DATA ANALYTICS LIFECYCLE
• Data science projects differ from BI projects
• More exploratory in nature
• Critical to have a project process
• Participants should be thorough and rigorous
• Break large projects into smaller pieces
• Spend time to plan and scope the work
• Documenting adds rigor and credibility
DATA ANALYTICS LIFECYCLE
• Data Analytics Lifecycle Overview
• Phase 1: Discovery
• Phase 2: Data Preparation
• Phase 3: Model Planning
• Phase 4: Model Building
• Phase 5: Communicate Results
• Phase 6: Operationalize
• Case Study: GINA
2.1 DATA ANALYTICS
LIFECYCLE OVERVIEW
• The data analytic lifecycle is designed for Big Data problems and
data science projects
• With six phases the project work can occur in several phases
simultaneously
• The cycle is iterative to portray a real project
• Work can return to earlier phases as new information is uncovered
2.1.1 KEY ROLES FOR A
SUCCESSFUL ANALYTICS
PROJECT
KEY ROLES FOR A
SUCCESSFUL ANALYTICS
PROJECT
• Business User – understands the domain area
• Project Sponsor – provides requirements
• Project Manager – ensures meeting objectives
• Business Intelligence Analyst – provides business domain
expertise based on deep understanding of the data
• Database Administrator (DBA) – creates DB environment
• Data Engineer – provides technical skills, assists data
management and extraction, supports analytic sandbox
• Data Scientist – provides analytic techniques and modeling
2.1.2 BACKGROUND AND OVERVIEW
OF DATA ANALYTICS LIFECYCLE
• Data Analytics Lifecycle defines the analytics process and
best practices from discovery to project completion
• The Lifecycle employs aspects of
• Scientific method
• Cross Industry Standard Process for Data Mining (CRISP-DM)
• Process model for data mining
• Davenport’s DELTA framework
• Hubbard’s Applied Information Economics (AIE) approach
• MAD Skills: New Analysis Practices for Big Data by Cohen et al.
https://en.wikipedia.org/wiki/Scientific_method
https://en.wikipedia.org/wiki/Cross_Industry_Standard_Process_for_Data_Mining
http://www.informationweek.com/software/information-management/analytics-at-work-qanda-with-tom-davenport/d/d-id/1085869?
https://en.wikipedia.org/wiki/Applied_information_economics
https://pafnuty.wordpress.com/2013/03/15/reading-log-mad-skills-new-analysis-practices-for-big-data-cohen/
OVERVIEW OF
DATA ANALYTICS LIFECYCLE
2.2 PHASE 1: DISCOVERY
2.2 PHASE 1: DISCOVERY
1. Learning the Business Domain
2. Resources
3. Framing the Problem
4. Identifying Key Stakeholders
5. Interviewing the Analytics Sponsor
6. Developing Initial Hypotheses
7. Identifying Potential Data Sources
2.3 PHASE 2: DATA PREPARATION
2.3 PHASE 2: DATA
PREPARATION
• Includes steps to explore, preprocess, and condition
data
• Create robust environment – analytics sandbox
• Data preparation tends to be t.
Modern Perspectives on Recommender Systems and their Applications in MendeleyKris Jack
Presentation given for one of Pearson's Data Research teams. It motivates the use of recommender systems, describes common approaches to building and evaluating them and gives examples of how they are used in Mendeley. Thanks to Maya Hristakeva for creating some of the slides.
We live in a world of silos - separate systems each with data essential to our daily work. No organization has all its important information in one place - 61% of knowledge workers regularly access 4 or more systems to get the information they need to do their jobs, and 15% need 11 or more systems. Integration to provide a unified view across these systems is very valuable, but it has been difficult to accomplish - even between different Microsoft products. This seminar will show you how to bridge across these silos using a search-based approach that is both quick and powerful.
Solving Real World Challenges with Enterprise SearchSPC Adriatics
Enterprise Search is complex, even in theory. But when you implement your search solution and everything turns to reality, you’ll find some new, never-seen challenges. In this session, I’ll collect the best, biggest and most exciting challenges from my experience, including real world customer scenarios and solutions. Regardless of the SharePoint version you use (SharePoint 2010, FAST Search for SharePoint, SharePoint 2013), this session is for you if you want to prepare for these “unexpected” scenarios.
Evolving the Optimal Relevancy Ranking Model at Dice.comSimon Hughes
This is a talk about gathering a golden test set of relevancy judgements, either using manual annotators or search log mining, to use in either an automated or manual relevancy tuning process. We also discuss the dangers of positive feedback loops when building closed-loop machine learning models for search and recommendation.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
2. Who Am I?
• Chief Data Scientist at DHI (Dice.com) under Yuri Bykov
• Dice.com – leading US job board for IT professionals
• PhD Candidate DePaul University (NLP and Machine Learning)
• Twitter handle: https://twitter.com/hughes_meister
• Email: simon.hughes@dice.com
• Main Data Science Projects
• Dice Job and Talent Search
• Dice Recommender Engines (e.g. Similar Positions)
• Dice Salary Predictor - https://www.dice.com/salary-calculator
• Dice Career Paths Page - https://www.dice.com/career-paths
• Dice Skills Pages - https://www.dice.com/skills
7. Measuring Search Relevancy
• Recall - How many of the relevant documents were returned?
• Precision - How relevant were the results returned?
Retrieved DocumentsRelevant Documents PrecisionRecall
Retrieved Relevant Documents
8. Relevancy Optimization
• Improving Recall – Conceptual Search*, Blind Feedback
• Improving Precision – Query Optimization*, Query Classification, LTR
• Optimizing for precision is easier – correct mistakes in the current
search results
• Optimizing for recall is harder – need to know which relevant
documents in the index don’t get retrieved
9. Conceptual Search
• A.K.A. Semantic Search
• Two key challenges with keyword matching:
• Polysemy: Words have multiple meanings
• E.g. engineer – mechanical engineer? Programmer? automation engineer?
• Synonymy: Many different words have the same or similar meaning
• E.g. QA, quality assurance, tester; VB, Visual Basic, VB.Net
• Other related challenges –
• Typos, Spelling Errors, Idioms
• Conceptual search attempts to solve these problems by learning
concepts from words
• Attempts to improve recall
10. Conceptual Search
Senior Hadoop* Developer
At least eight years of database/application
development experience in an complex enterprise
environment. Experience writing in SQL, stored
procedures, query performance tuning preferably
on SQL Server. Strong familiarity with working in a
Linux and Windows environment which includes
shell and power shell scripting. At least two years of
hands on experience designing and implementing
data pipelines in production using tools from the
Hadoop* ecosystem such as MapReduce, Hive,
HBase, Spark*, Sqoop, Oozie, and Pig. Broad
knowledge of software development including
software architecture, functional and non-
functional aspects, CI/CD, principles and tools
Java
Technologies*
Big Data
Databases
Software
Architecture
System
Admin
*items are also java technologies
11. Conceptual Search
• Conceptual search allows us to retrieve documents by how similar the
concepts in the query are to the concepts in a document
• Concepts are automatically learned from documents using machine
learning
• Traditional techniques (LSA, LDA) are based on factorizing large
matrices and don’t scale well
• Word2vec – learns vector representations of words based on context
- an iterative algorithm, scales much better
12. Word2vec
• Learns vector representations of words by predicting surrounding words
• Similar words get similar vector representations
• Finds interesting relationships between words - e.g. ‘word math’
13. Word2vec Pros and Cons
• Works much better if common phrases are treated as single tokens
• e.g. java developer=>java_developer, sql server=>sql_server
• Advantages
• Effective at learning related terms /phrases
• e.g. java developer, j2ee developer, java engineer, java architect, hadoop engineer
• Disadvantages
• Doesn’t handle word sense disambiguation well
• Sees antonyms as similar as appear in similar contexts:
• Black and white, up and down, hot and cold, Trump and Clinton, Democrat and Republican
• If the keywords in your domain are noun phrases, typically less of an issue
• Often aggregating concepts over an entire document can solve a lot of these
issues provided query is disambiguated
14. Using Word2vec In Search
Search engines use inverted indexes - work with terms and not vectors. Approaches:
• Query Expansion
• Expand user’s query with most similar word2vec terms/phrases
• Doesn’t require modifying the search index
• Can boost expansion terms using word2vec similarity score
• Clustering
• Cluster word2vec terms and create separate fields mapping terms into their clusters
• Easy to implement using standard synonym files
• Create different sized clusters to get broader / finer grain matching
• Re-Ranker
• Re-rank the top n documents of a query using the word2vec vector similarity
• More complicated to implement
• Can be used as features for a LTR model
15. Learned Clusters
Pre-processing - Colocation (phrase) detection using PMI, word2vec over
phrases and top keywords, then k-means clustering
• Natural Languages: bi lingual, bilingual, chinese, fluent, french, german,
japanese, korean, lingual, localized, portuguese, russian, spanish, speak,
speaker
• Apply Programming Languages: cocoa, swift
• Search Engine Technologies: apache solr, elasticsearch, lucene, lucene solr,
search, search engines, search technologies, solr, solr lucene
• Microsoft .Net Technologies: c# wcf, microsoft c#, microsoft.net, mvc web,
wcf web services, web forms, webforms, windows forms, winforms, wpf wcf
16. Learned Clusters – Soft Skills
Attention / Attitude:
• attention, attentive, close attention, compromising, conscientious,
conscious, customer oriented, customer service focus, customer service
oriented, deliver results, delivering results, demonstrated commitment,
dependability, dependable, detailed oriented, diligence, diligent, do
attitude, ethic, excellent follow, extremely detail oriented, good
attention, meticulous, meticulous attention, organized, orientated,
outgoing, outstanding customer service, pay attention, personality,
pleasant, positive attitude, professional appearance, professional
attitude, professional demeanor, punctual, punctuality, self motivated,
self motivation, superb, superior, thoroughness
17. Conceptual Search In Action
• Only conceptual search matches shown
– all keyword matches are excluded
• These are documents that would not be
returned by regular keyword search
18. Conceptual Search In Action
• Only conceptual search matches shown
– all keyword matches are excluded
• These are documents that would not be
returned by regular keyword search
19. Relevancy Tuning
• Search engines provide a lot of different knobs that can be used to
improve relevancy
• These include the weight (or ‘boost’) given to each field in a search
query, the minimum number of terms required for a match, what type of
queries are executed (disjunction max, best fields, etc), and document
quality scores (e.g. google’s page rank)
• Often these knobs are tuned manually by the search engineer to
optimize their view of the optimal search experience
• Focus is primarily on precision as easier to judge
• Can we do better?
20. Golden Test Collection
• We really need a set of high quality relevancy judgements
• Two Main Sources:
1. Manual Annotations
• Expert users rate results for common queries
• Costly to collect
• May not reflect judgements of your users
• Active learning can be used to improve annotation efficiency if used in LTR
2. Search Logs / Click Stream Data
• Collect data from search logs that indicate which documents seem to be relevant
• Reflects how your users view relevancy
• Relies on implicit signals which can be noisy – documents clicked, viewed
• Hard to get explicit feedback from users
21. Manual Annotations
• Users rate each document
based on how relevant it is to
the query
• Important that the ratings
differ for a query, otherwise
no useful information is
provided to the algorithm
22. Machine Learning Approaches
• Often we can’t optimize search engine relevancy directly as the scoring
functions are not differentiable
• Evaluating relevancy can be very costly – running thousands of queries
against the search engine to evaluate each parameter configuration
• Instead we can use black-box optimization algorithms to optimize the
parameters, typically this is more efficient than random search
• Most companies also using machine learning to train a re-ranking model
to re-rank the top N results
• However it is better to first optimize the search engine’s settings so that
the top N results are more likely to contain the most relevant documents
23. Information Retrieval Metrics
• Precision alone is not a great metric as it is insensitive to the ordering of the
documents returned
• Objective – maximize preferred information retrieval metric:
1. Normalized Discounted Cumulative Gain (NDCG)
• Discounts relevancy scores by their ranking in the results
2. Mean Average Precision (MAP)
• Average of the precision at the location of each relevant document returned
3. Precision at k
• Precision at the top k documents (usually 10)
• Insensitive to the ordering of documents within top k
• NDCG is used when you have ratings, MAP and ‘Precision at k’ are used for
binary relevant/irrelevant judgements or click data
24. Black Box Optimization Algorithms
1. Genetic Algorithms
• Standard GA
• Evolutionary Strategies
• Genetic Programming – for evolving new scoring equations
• E.g Python DEAP package
2. Bayesian Optimization
• As it searches the parameter space, focuses more on areas of uncertainty (using LCB and
similar variants from reinforcement learning)
• E.g. Python scikit-optimize package
3. Coordinate Ascent/Descent
• Very simple algorithm – use a line search to find the optimal value for each parameter
while keeping all others fixed
• Can get stuck in local maxima/minima
• Searches more efficiently than more random approaches
25. Test NDCG Improvements on MLT Task
• Tried different algorithms for
optimizing Elastic Search
MoreLikeThis queries
• Parameters – relative boosts on title
and skills, number of terms
extracted, min doc freq per term
• Coordinate ascent produced the
largest improvement in the training
and test data
• 8.2% Improvement on test data set
26. Test NDCG Improvements on Talent Search
• Tried different algorithms for
optimizing Talent search queries
• Parameters – relative boosts on
different fields, phrase vs term
matching
• GA produced the highest test score
at the end, but GBT had highest test
score overall – early stopping?
• 0.64% Improvement on test data set
– much smaller but ratings quality
much lower
27. Summary
• There are many ways you can apply machine learning to improve your user’s
search experience
• I have gone over two ways in which you can improve the recall and relevancy
of your search engine
• Using conceptual search to learn synonyms and improve recall
• Using black box optimization algorithms to automate relevancy tuning
• Many other approaches for applying machine learning to improve search:
• Learning to Rank (LTR)
• Query Classification
• Query Parsing
• Personalization
This talk will cover conceptual search and query optimization techniques. Query classification and LTR (Learning To Rank) are other common approaches to improving precision which won’t be covered in this talk.
Map words to concepts
Words can map to multiple concepts, e.g. the java technologies above, a number of terms map to that.
Labels in bold are manually assigned for interpretability
