KNO.E.SIS Approach to Impactful Research,
Creating Exceptional Careers & Economic Development
Amit Sheth
LexisNexis Ohio Eminent Scholar & Exec. Director,
The Ohio Center of Excellence in Knowledge-enabled Computing
& BioHealth Innovations (Kno.e.sis)
Wright State, USA
Presentation template by SlidesCarnival
Photographs by Unsplash1

“
Given the record so far,
what is it possible in the
next decade?
EDUCATION RESEARCH &
INNOVATIONS
TECH DEV & REAL
WORLD APPLICATIONS
● Tech Transfer
● Incubation
● Commercialization
● Economic &
Societal Impact
2

AS AN EDUCATOR:
STUDENT SUCCESS
28+~30+>10 >1800
11/11>40%
~25 1036
KNO.E.SIS “Gurukul”

AS AN EDUCATOR
❖ Graduated 28 PhDs and about the same number of MS-Thesis
students. Six employed as TT faculty in academia (NCSU, CWRU,
GMU,...), majority in top industry research labs (IBM Almaden, IBM
Watson, Samsung Research, Bosch Research,...) and top high tech
companies (Amazon, LinkedIn, Target Labs, Walmart,...), and some
are successful entrepreneurs. MS at the higher end of packages.
All recent postdocs are also in good academic positions.
❖ Exceptional Accomplishments: ~15 publications, member of PCs-
often for top conferences, workshop organization, tutorial, majority
have patents, exceptional internships, networking and social skills.
Avg citation of first 19 PhDs > 1800 (majority > 1000).
❖ Exceptional Diversity: > 40% are women
❖ Typically funded ~25 researchers at a time (~15 PhD students, 1-2
postdocs, 2-3 software engineers, rest: MS thesis and BS students).
During the 10 years, funded 1,036 months of GRA, (~300
quarter/semesters of GRA support).
4

PhD Advisees elected Outstanding
Student for each of the 11 years
Sanjaya’s research on understanding rapidly
growing Emojis, and development of EmojiNet,
the first machine-readable emoji dictionary has
garnered significant international attention and
acclaim. Kaggle.com inducted EmojiNet as a
featured dataset and open-sourced his research.
His work has been featured in nine media articles.
He will be featured in an upcoming documentary
on emoji, directed by Emmy winning film
producers. His 15 co-authored peer-reviewed
publications, including several in top venues, have
already exceeded 100 citations. He has given two
invited talks, served on eight programming
committees, is co-organizing an international
workshop and editing a special issue of a top
journal.
Advisor: Amit Sheth
Cory is an exceptional representative of our
accomplishments in preparing world-class
graduates, represented by several unparalleled
achievements: excellent publication record that
has translated into exceptionally high citation (h-
index 11, nearly 700 citations); co-editing an
broadly used report for W3C resulting from an
international activity; keynote at a major
international event in Venice, Italy, along with 4
other invited talks - two at Dagstuhl Seminars; PC
co-chair of an international workshops and
serving on 19 Program Committees including top
conferences like ESWC 2010-2013; guest co-
editing a special issues of high quality journal and
a top international magazine; coordinating
funded projects and mentoring students.
Advisor: Amit Sheth
5

Student Success: Our Most Important Performance Measure
Exceptional First Jobs
George Mason U., CWRU, IBM Research,
LinkedIn, Twitter, Amazon; 10 of Raymer
and Sheth’s advisee are/were faculty.
Exceptional Starting Salaries
e.g., $150-250K for a PhD; six figures
for M.S.
Exceptional Internships
IBM Research Almaden/Watson,
Samsung Research, Bosch R&D, GE
Research, Google, Mayo, NLM, QCRI, etc.
Major Awards
2015-2016 George Thomas Post-Graduate
Fellowship, Eric & Wendy Schmidt Data
Science for Social Good Fellowship, USAID
and ICT4Peace Fellowship, ITU Telecom
World Young Innovators, etc.
Exceptional Publication Impact
11 out of Sheth’s first 20 PhDs have 1,000+
citations each (3 over 5,000); Average
citation for Sheth’s first 18 PhDs: 1,800+
Exceptional Professional Services
Invited talks; PC membership of 10-20
conferences, usually involving top
conferences in their fields.
50% of the Sheth’s PhDs have filed at least 2 patents; 1 former PhD has filed 30+ patents.
6

Tenured/ Tenure Track
WHERE ARE ADVISEE EMPLOYED?
7

SHETH Group: All Funded
8

DIVERSITY
We encourage women in computing!
9

AS A RESEARCHER:
IMPACT
27+/25+/9
57/75+/2+2
IEEE
98/95/<100
AAAI
10

AS A RESEARCHER
❖ IEEE fellow
❖ AAAI fellow
❖ 27+ million in research funds as a PI (4 NIH R01s, 25+ NSF,
DARPA, AFRL, AFOSR, ARL, …; significant industry gifts
(MSR, HP-R, IBM/IBM Faculty awards)
❖ 57 keynotes, 75+ international conferences/workshops
organizations incl numerous PC/General Chairs, 250+ PCs,
2 EICs, 2 Book Series
❖ h-index = 98, i100 = 95, among top 100 scientist worldwide
in CS (2018 and last several years)
11

AS AN ENTREPRENEUR:
ECONOMIC DEVELOPMENT & JOBS
Taalee/Semagix
Cognovi Labs
OFC Commercialization Award [Runner-up]
InfoHarness,
METEOR, ASEMR,...
ezDI
6+m
12

NSF IIS-0842129
III-SGER: Spatio-Temporal-
Thematic Queries of Semantic
Web Data: a Study of
Expressivity and Efficiency
$163,800, 09/2008-08/2011
IIS-1143717
III: EAGER - Expressive Scalable
Querying over Integrated
Linked Open Data
$141,828, 09/2011-08/2013
NSF IIS-1111182 [Main Research Funding]
SoCS: Collaborative Research:
Social Media Enhanced
Organizational Sensemaking
in Emergency Response
$518,408, 09/2011-08/2015
IIS-1343041
I-Corps: Towards Commercialization
of Twitris - A System for Collective Intelligence
$50,000, 08/2013 - 01/2014
From NSF Funded Research to Economic Dev
IIP-1542911 [This project]
PFI: AIR-TT: Market-driven
Innovations
and Scaling up of Twitris - A
System for Collective Social
Intelligence
$200,000 (+ $12,000 REU)
09/01/2015 - 08/30/2017
Trial license with iKove Venture
Partners and the launch of
Cognovi Labs in Feb 2016.
Exclusive license Oct 2016.
$2.3m in funding.
$492,500 for cluster/private cloud from AFOSR that supports scalable computing for Twitris.
Additional $6, research funds from AFRL, NIH for applications and application supported development.
NIH R01 DA039454-01
Trending: Social Media Analysis to Monitor
Cannabis and Synthetic Cannabinoid Use
(eDrugTrends)
Total: $1,689,019: 08/2014 – 08/2017
Development & Applications
14

Sample Media Coverage of Twitris Technology/Startup
Details at: http://knoesis.org/amit/media [65+ since 2017]
Dec 7, 2015
Chennai floods: How social media
and crowdsourcing helps
people on ground
Sep 9, 2014
Digital soldiers emerge
heroes in Kashmir
flood rescue
June 27, 2013
Using crisis mapping to
aid Uttarakhand
Feb 17, 2012
Web App Analyzes Tweets
in Real Time for a Record of
Historic Events
Sept 26, 2016
Cognovi Labs will Study the
Presidential Debates Tonight
with Their Emotional
Sentiment Tool
June 29, 2016
The Twitris Sentiment Analysis
Tool by Cognovi Labs
Predicted the Brexi hours
Earlier Than Polls
15

Example TechCrunch Coverage of Twitris & Cognovi Labs
TechCrunch is the most viewed technology web site.
16

LEADERSHIP
Institutional development, inclusive collaborations
COE-4 colleges 10,000+ 864/17/18TB/435
17

How We Measure SUCCESS
KNO.E.SIS is highly successful in its primary mission of
★ Exceptional student outcomes
★ Exceptional publication
★ Real-world impact (http://j.mp/kimpact)
★ Our PhDs compete with their counterparts from top 10 schools
for initial jobs in research universities, top industry research labs
and highly competitive companies, while a few have been
successful entrepreneurs.
This is enabled by our vision to keep ahead of the competition,
interdisciplinary faculty, world-class computing and physical
infrastructure, exceptional funding, extensive collaborations with
academia and industry.
18

DoD & Industry
• Metabolomics &
Proteomics
• Medical Info Decisions
• Human Detection
on Synthetic FMV
• Sensor & Information
• Material Genomics
• Cardiology Semantic
Analysis
NIH
• kHealth - Asthma
• eDrug Trends
• Depression on Social Media
• Drug Abuse Early Warning
NSF
• Harassment on Social Media
• Citizen & Physical Sensing
• Twitris - Collective Intelligence
• Aerial Surveillance
• Visual Experience
• Web Robot Traffic
Kno.e.sis’ research in World Wide Web ranks Wright State University among the top 10 organizations in the world based
on 10-yr impact. Its total budget for currently active projects is $13+ million, with $6+ million for new projects starting
after July 2015. The significant majority of funds are highly competitive federal grants. World-class research is
complemented by exceptional student outcomes and commercialization with local economic impact.
As an Ohio COE on Bio Health
Innovation, Kno.e.sis conducts
research leading to building
intelligent systems for clinical,
biomedical, policy, and
epidemiological applications.
Example clinical/healthcare
applications include major diseases
such as asthma, depression,
cardiology, dementia and GI.
This is complemented by social and
development challenges such as
marijuana legalization policy,
harassment on social media,
gender-based violence, and
disaster coordination.
60+ Funded
Students
• 40 PhD
• 16 MS
• 5 BS
Kno.e.sis:
Ohio Center of Excellence in Knowledge-enabled Computing
19

Additional Funded Projects (when Kno.e.sis faculty is a PI/jointPI*)
NMR-Based Urinary Metabolomics in Rats Exposed to Burn Pit Emissions and Respirable Sand, $241K, Reo, Raymer
PFI: AIR-TT: Market-driven Innovations and Scaling up of Twitris - A System for Collective Social Intelligence; 200K,
Sheth, Mackay
CRII: CSR: Towards Understanding and Mitigating the Impact of Web Robot Traffic on Web Systems; 174K, Doran
Medical Information Decision Assistance and Support; 25K, Prasad, Sheth
Choose Ohio First: Growing the STEMM Pipeline in the Dayton Region FY2016/FY2017; Raymer
Westwood Partnership to Prevent Juvenile Repeat Violent Offenders; $200K, Sheth, Doran, Dustin
Semantic Web-based Data Exchange and Interoperability for OEM-Supplier Collaboration; 89K, Prasad, Sheth
NIDA National Early Warning System Network (iN3): An Innovative Approach; 299K, Carlson, Sheth, Boyer,
Daniulaityte, Nahas
CUTE: Instructional Laboratories for Cloud Computing Education; 200K, Chen, Wang, Mateti
SemMat: Federated Semantic Services Platform for Materials Science and Engineering; 315K, Sheth, Prasad,
Srinivasan
Materials Database Knowledge Discovery and Data Mining; 190K, Sheth, Prasad, Srinivasan
* Grants with Kno.e.sis faculty as coPI or investigator not included
20

Tanvi Banerjee Robert Carlson Pratik Parikh Keke Chen Raminta Daniulaityte
Guozhu Dong Derek Doran John Flach Michael Raymer T.K. Prasad
Nick ReoValerie Shalin Amit Sheth Maninder Kalra Thomas Wischgoll
Funded students: > 60
• 30+ PhD
• 20+ MS
• Growing number of BS
Kno.e.sis Faculty:
Practically all have active funds, multidisciplinary collaborations
William Romine
21

Clinical Research
Biomedical Research
Materials Engineering
Cognitive Science
Computer Sc. & Engineering
AI (ML, NLP, KR/SW), IR, DM, BI, CC, Vis, …
Biomed Industrial
HF Engineering
Epidemiology
Clinical Psych
Biological Sc. & Bio Stats
Numerous orgs in humanitarian
and crisis space
22

KNO.E.SIS [Partial]
23

24

25

We are World Class….(last available data, but not updated recently by MAS)
26

Kno.e.sis: VISIONS AND THEMES
01
Computing for
Human Experience 02 Physical-Cyber-Social
Big & Smart Data
03
Semantic,
Cognitive and
Perceptual
Computing
04
Personalized digital
health/connected
health and medicine,
Epidemiology,
public/community
health
Looking forward: Contextualized & Personalized AI
27

Success Defined by OUTCOMES and IMPACT
A key reason for Kno.e.sis' success is its unique work culture involving
teamwork to solve complex problems. Practically all our work involves
real world challenges, real-world data, interdisciplinary collaborators,
path-breaking research to solve the real-world challenges, real-world
deployments, real world use, and measurable real world impact.
These are demonstrable in diverse forms such as policy impact (e.g.,
FDA advisory), operational real-time use of our tools to save lives
during disasters, open source data and tools, and licensing/tech
transfer and commercialization of our technology used to launch a
local startup.
28

Artificial
Intelligence
Ambient
Intelligence
Augmenting
Human Intellect
Human-Computer
Symbiosis
Computing for
Human Experience
Machine-centric Human-centric
John McCarthy Mark Weiser Douglas Engelbart Joseph C.R. Licklider
Figure: Views along the spectrum of machine-centric to human-centric computing.
At the far right is our work on Computing for Human Experience, which explores paradigms such as
Semantic, Cognitive, and Perceptual Computing. http://bit.ly/SCP-Magazine
Kno.e.sis Center
http://bit.ly/k-Che,
http://slidesha.re/k-che
Machine-centric to Human-centric Computing
29

Social Media Big Data
Twitris, eDrugTrends
Let’s talk Big Data @ Kno.e.sis
Sensor/ IoT Big Data
CityPulse, kHealth
Healthcare Big Data
kHealth, EMR, Prediction
Biomedical Big Data
Biomarker from NextGen,
Sequencing and Proteomics,
SCOONER
Big and Smart
Data Certificate
Kno.e.sis private cloud: 864 CPU cores, 18TB RAM, 17TB SSD, 435TB disk
30

Health Related Studies at KNO.E.SIS [Overview]
HealthChallenges
(Also, Dementia,
Obesity,
Parkinson’s,
Liver Cirrhosis,
ADHF ) Public Policy/ Population Epidemiology Personalized Health
PCS + EMR
kHealth
Asthma in Children
Bariatric Surgery
Physical(IoT)/Cyber/
Social (PCS)+ EMR
Marijuana Social
Drug Abuse Social
Mental Health
Depression (Suicide) Social + Public + EMR
Health
Knowledge Graph
Services
Social + Clinical Data
...and infrastructure technologies:
Context-aware KR (SP),
KG development,
Smart Data from PCS Big Data,
Twitris
31

kHealth - Asthma
Principal Investigators: Amit P. Sheth
Co-Investigators: Krishnaprasad Thirunarayan , Maninder Kalra
Other Faculty: Tanvi Banerjee
Students: Utkarshini Jaimini, Hong Yung (Joey) Yip, Revathy Venkataramanan,
Vaikunth Sridharan, Dipesh Kadaria
Grant Number: 1 R01 HD087132-01
Project Title: KHealth: Semantic Multisensory Mobile
Approach to Personalized Asthma Care
Timeline: 07/01/2016 – 06/30/2019
Award Amount: $938,725
32

APH for Asthma and Bariatrics: Patient-centric drivers
This not only prevent the disease, but also enhances the patient’s health
BariatricsAsthma
33

Example 1: kHealth Asthma
Data Sources
Heterogeneous data and collection
method (1852 data points/ patient /day)
Semantic, Cognitive, Perceptual
Computing Framework
http://bit.ly/SCPComputing
Smarter conversations with
actionable meaningful information.
http://bit.ly/kHealth-Asthma
Example 1: kHealth Asthma
34

Context-Aware Harassment
Detection on Social Media
Principal Investigators: Prof. Amit P. Sheth
Co-Investigators: Valerie Shalin, Krishnaprasad Thirunarayan
Other Faculty: Debra Steele-Johnson, Dr. Jack L. Dustin
PhD Students: Lu Chen, Wenbo Wang, Monireh Ebrahimi, Kathleen Renee Wylds
MS Students: Pranav Karan, Rajeshwari Kandakatla
Collaboration with Beavercreek High School
NSF Award#: CNS 1513721
TWC SBE: Medium: Context-Aware Harassment Detection on
Social Media
Timeline: 01 Sep. 2015 - 31 Aug. 2018
Award Amount: $925,104 + $16,000 (REU)
35

Social and Physical Sensing Enabled Decision Support for
Disaster Management and Response
Principal Investigators: Prof. Amit P. Sheth, Prof. Srinivasan Parthasarathy (OSU)
Co-Principal Investigators: Densheng Liu (OSU), Ethan Kubatko (OSU), Valerie
Shalin, Krishnaprasad Thirunarayan
PhD Students: Sarasi Lalithsena, Pavan Kapanipathi, Hussein Al-Olimat
MS Student: Siva Kumar
Postdoctoral Researcher: Tanvi Banerjee
NSF Award#: EAR 1520870
Hazards SEES: Social and Physical Sensing Enabled Decision
Support for Disaster Management and Response
Timeline: 01 Jul. 2015 - 31 Jul. 2019
Award Amount: $1,975,000 (WSU: $787,500)
36

Example 1: kHealth Asthma
Timely delivery of right resources
and information to the right people
at right location!
Chain of Plausibility (CoP) Pipeline
https://bit.ly/2HI9ydy
37

kHealth - Dementia
Principal Investigators: Tanvi Banerjee
Mentors: Amit Sheth, Larry Lawhorne
Students: ….
Grant Number: 1K01LM012439-01
Project Title: Managing Dementia through Multisensory
SmartPhone Approach to Support Aging in Place
Timeline: 09/01/2016 – 08/30/2019
Award Amount: $509,909
38

eDrug Trends
Principal Investigators: Prof. Amit P. Sheth, Prof. Raminta Daniulaityte
Co-Investigators: Robert Carlson, Krishnaprasad Thirunarayan, Ramzi Nahhas,
Silvia Martins (Columbia), Edward W. Boyer (U. Mass.)
PhD Students: Farahnaz Golroo, Sanjaya Wijeratne, Lu Chen, Adarsh Alex
MS Student: Adarsh Alex
Postdoctoral Researcher: Francois Lamy
Software Engineer: Gary Smith
NIH Award#: 5 R01 DA039454-02
Trending: Social media analysis to monitor cannabis and
synthetic cannabinoid use
Timeline: 15 Sep. 2014 - 14 Sep. 2018
Award Amount: $1,689,019 + $162,505
39

eDrug Trends Project Overview
40

Modeling Social Behavior for
Healthcare Utilization in Depression
Principal Investigators: Prof. Amit P. Sheth, Prof. Jyotishman Pathak (Cornell)
Co-Investigators: Krishnaprasad Thirunarayan, Tanvi Banerjee,
William V. Bobo (Mayo Clinic), Nilay D Shah (Mayo Clinic), Lila J Rutten (Mayo Clinic),
Jennifer B McCormick (Mayo Clinic), Gyorgy Simon (Mayo Clinic)
Other Faculty: Debra Steele-Johnson, Jack Dustin
PhD Students: Ashutosh Jadhav, Amir Hossein Yazdavar, Hussein Al-Olimat,
SoonJye Kho
Master Student: Surendra Marupudi
NIH Award#: 1 R01 MH105384-01A1
Modeling Social Behavior for Healthcare Utilization in
Depression
Timeline: 1 Jul. 2015 - 30 Jun. 2019
Award Amount: $1,934,525 (WSU: $505,600)
41

Depression Modeling Project Overview
42

43
Problem Solved (Project) DL Technique Used Data Used Outcome Summary
Diagnostic Mental Health
Classification (Alchemy for
Health)
Autoencoder Reddit, Twitter
Our evaluations show that compared to baseline system (SVM & CNN), our proposed
approach yields 15.27% and 4.17% improvements in F-Score on the curated blog dataset.
Medical-Severity
Identification (Depression,
Alchemy for Health)
CNN, LSTM, CNN-
SVM
Blog-post, Twitter,
Reddit
Identifying Street Gang
Members on Twitter
Word Embeddings Twitter
A 6.39% improvement in F1-score compared to the baseline models which were not
trained using word embeddings.
Emoji Sense
Disambiguation
Word Embeddings
Twitter, Google
News
180.41% and 111.69% improvements in F1-score compared to the baseline models
which were based on overlap-based emoji sense disambiguation methods, when
compared with Twitter and Google News-based word embedding models, respectively.
Emoji Similarity
Word Embeddings
and Emoji
Embeddings
Twitter, Google
News
Our emoji embedding models learned over emoji sense labels perform best in a
benchmark sentiment analysis task, outperforming the previous best emoji embedding
models with an improvement of 7.73%.
Smart Crowd Selection Word embeddings Twitter
Our approach was able to select crowds that outperformed, based on a wisdom score
measure, crowds selected at random as well as based on expertise defined by users' their
previous performance. The approach is able to select such a smart crowd without using
their current/previous judgement data.
Adverse Drug Reactions
discovery from
Unstructured Data
Bidirectional
LSTM,sequence
labeling
Forum data In progress
Embeddings for Ontological
Concepts
DBpedia/Wikipedia Submitted

Accomplishments & Accolades (a sample)
From NSF Program Manager of recently completed project that lead to outcomes (# and
quality of PhDs/MS/UG; publications, participation/use in real world disaster events,
extensive national and international media coverage,....).
“Thank you for your wonderfully
informative final report for your
collaborative project.”
“I am pleased that your work has been
reported in publications that will hopefully
lead to an even broader adoption of your
techniques and tools.”
“I am glad to see your research
results in developing an effective
Twitris platform for use by PIOs that
utilizes social media for
coordination of efforts for an
emergency response.”
“I appreciate your efforts in broadening
the impacts of your project via
collaboration with emergency personnel
and including them in the final testing of
Twitris. It is commendable that your tool
has been successfully used in past year’s
major disaster events.”
44

Among top universities in the world in World Wide Web (cf: 10-yr impact, Microsoft Academic
Search: among top 10 – 2013 data).
Among the largest academic groups in the US in Semantic Web + Social/Sensor Webs,
Mobile/Cloud/Cognitive Computing, Big Data, IoT, Health/Clinical &
Biomedicine Applications.
Exceptional student success: internships and jobs at top salary (IBM Watson/Research, MSR,
Amazon, CISCO, Oracle, Yahoo!, Samsung, research universities, NLM, startups ).
100 researchers including 15 World Class faculty (> 3K citations/faculty avg) and > 40 funded
PhD students.
Extensive research for largely multidisciplinary projects; world class resources; industry
sponsorships/collaborations (Google, IBM, …)
45

❖Created Twitris, a Commercial Grade software, which had significant NSF and
AFRL research funds, followed by NSF I-CORP and NSF-PFI-AIR funding;
currently VC/entrepreneurs are evaluating for potential licensing/start up.
❖One recent patent awarded, two filed.
❖Many more patents filed by companies where Kno.e.sis students intern.
❖ezDI has funded Sponsored Research for the fifth year in the row and now has
major successful products.
❖A local entrepreneur has just signed Sponsored Research that is essentially
incubating his company in Kno.e.sis.
❖On going SBIR/STTR funding and participating; faculty (Raymer) led company.
Economic & IP Development
46

KNO.E.SIS is the shining example of world-class
research and student outcomes.
Information…Meaning…Actions…Impact
http://knoesis.org http://facebook.com/Kno.e.sis
47

Discussions
48

49
Visit us at:
www.knoesis.org