Reseval Mashup Platform Talk at SECO

•Download as PPTX, PDF•

1 like•370 views

We are going to represent a Mashup platform for the research evaluation. This talk was given at 2nd Search computing workshop in Como, italy on 27-may-2010.

An Open Mashup Platform ForResearch Impact Evaluation By: Muhammad Imran

EXISTING SOLUTIONS: Web of Science Publish or Perish Tool DBLP

PRIMARY USERS OF RESEARCH EVALUATION External Entities Government agencies/funding organizations ,[object Object]

Govt. policy for future investmentUniversity Management Management, including committees, ,[object Object],University Departments Institutional research, academic affairs, tech transfer, etc. ,[object Object],Individuals Faculty, Staff, Students ,[object Object],[object Object]

What's hot

Database-Centric Guidelines for Building a Scholarly Metrics Information Syst...

Aravind Sesagiri Raamkumar

Academy Owned Peer Review

Micah Altman

Ontologies for music from a digital library practitioner's perspective

Jenn Riley

Metadata for Assessment resources

chessmu

Transfer & Advising Presentation

smscheurer

Ipn metodika konference_pardubice_sivertsen

MEYS, MŠMT in Czech

Costas-data metrics-nfdp13

DataDryad

V.3 poster current citations and a future with linked data

Iliadis Dimitrios

Responsible metrics: One size doesn't fit all

Ludo Waltman

The HR Acuity Title IX Survey for Institutions of Higher Education gathers data on the impact Title IX investigations are having on higher education. How much are institutions spending on Title IX-related expenses? What technology platform are campuses using to document and track allegations? What does the Title IX investigation process look like and what are the top 3 challenges of managing it?

Title IX: A Survey for Institutions of Higher Education

HR ACUITY LLC

Reference & Instruction Assessment_FACRL 2016

Elizabeth Fairall

This study aimed to find out how STEM collage students' use of different course resources in an LMS predict their final grades. They used Logistic Regression Models to estimate assosciation between resourse use and final grade. The results showed that there is a positively significant association between exam-related sources use and final grade. On the other hand, when students use lecture-related resources, then they are less likely to have good grades. Waddington, R. J., Nam, S., Lonn, S., Teasley, S. D. (2016). Improving early warning systems with categorized course resource usage. Journal of Learning Analytics, 3(3), 263–290.

Prediction-Improving Early Warning Systems With Categorized Course Resource U...

Beste Ulus

Towards Ranking in Folksonomies for Personalized Recommender Systems in E-Lea...

Mojisola Erdt née Anjorin

Reference Question Data Mining

Joshua Finnell

Over the recent years, there has been a growing interest in developing new scientometric measures that could go beyond the traditional citation-based bibliometric measures. This interest is motivated on one side by the wider availability or even emergence of new information evidencing research performance, such as article downloads, views and twitter mentions, and on the other side by the continued frustrations and problems surrounding the application of citation-based metrics to evaluate research performance in practice. Semantometrics are a new class of research evaluation metrics which build on the premise that full text is needed to assess the value of a publication. This talk will present the results of an investigation into the properties of the semantometric contribution measure (Knoth & Herrmannova, 2014). We will provide a comparative evaluation of the contribution measure with traditional bibliometric measures based on citation counting.

Semantometrics: Towards Fulltext-based Research Evaluation

petrknoth

Graduate & Post-graduate students' awareness about plagiarism

Ata Rehman

Information Literacy

Danielle Carlock

Resource sharing opportunities among academic libraries

Khalid Mahmood

What's hot (18)

Database-Centric Guidelines for Building a Scholarly Metrics Information Syst...

Academy Owned Peer Review

Ontologies for music from a digital library practitioner's perspective

Metadata for Assessment resources

Transfer & Advising Presentation

Ipn metodika konference_pardubice_sivertsen

Costas-data metrics-nfdp13

V.3 poster current citations and a future with linked data

Responsible metrics: One size doesn't fit all

Title IX: A Survey for Institutions of Higher Education

Reference & Instruction Assessment_FACRL 2016

Prediction-Improving Early Warning Systems With Categorized Course Resource U...

Towards Ranking in Folksonomies for Personalized Recommender Systems in E-Lea...

Reference Question Data Mining

Semantometrics: Towards Fulltext-based Research Evaluation

Graduate & Post-graduate students' awareness about plagiarism

Information Literacy

Resource sharing opportunities among academic libraries

Viewers also liked

Domain Specific Mashups

Muhammad Imran

Mivestabelenpresentation

Andrea Fuentes

art of the_americas

Andrea Fuentes

Technology And Education

finkp

Prospectvision

Andrew Wise Manager, EMEA Customer Success Team

14thcenteuropresentation

Andrea Fuentes

Amate bark painting presentation

Andrea Fuentes

This paper addresses the baffling problem of name disambiguation in the context of digital libraries that administer bibliographic citations. The problem emanates when multiple authors share a common name or when multiple name variations of an author appear in citation records. Name disambiguation is not trivial to solve, and most of the digital libraries do not provide an efficient way to accurately identify the citation records of an author. Furthermore, lack of complete meta-data information in digital libraries hinders the existence of generic algorithm that can be applicable on any dataset. We propose a heuristic-based, unsupervised and adaptive method that also embraces users’ interaction to count users’ feedback in disambiguation process. Moreover, the method exploits important features associated with an author and citation records such as co-authors, affiliation, publication title, venue etc., and contrives a conspicuous multilayer hierarchical clustering algorithm, which tunes it- self according to the available information and form clusters of unambiguous records. Our experiments on a set of researchers that are contemplated to be highly ambiguous de- cisively produced high precision and recall results and affirm the viability of our algorithm.

A Real-time Heuristic-based Unsupervised Method for Name Disambiguation in Di...

Muhammad Imran

Keynote slides for ISCRAM 2016. "Social Media platforms such as Twitter are invaluable sources of time-critical information. Information on social media communicated during emergencies convey timely and actionable information. For rapid crisis response, real-time insights are important for emergency responders. Although, many humanitarian organizations would like to use this information, however they struggle due a number of issues such as information overload, information vagueness, less credible and misinformation. In this talk, I will describe the role of social media and potential artificial intelligence computational techniques useful for humanitarian organizations and decision makers to make sense of social media data for rapid crisis response."

The Role of Social Media and Artificial Intelligence for Disaster Response

Muhammad Imran

Creative Commons

leenamarsh

An emerging paradigm for the processing of data streams involves human and machine computation working together, allowing human intelligence to process large-scale data. We apply this approach to the classification of crisis-related messages in microblog streams. We begin by describing the platform AIDR (Artificial Intelligence for Disaster Response), which collects human annotations over time to create and maintain automatic supervised classifiers for social media messages. Next, we study two significant challenges in its design: (1) identifying which elements must be labeled by humans, and (2) determining when to ask for such annotations to be done. The first challenge is selecting the items to be labeled by crowdsourcing workers to maximize the productivity of their work. The second challenge is to schedule the work in order to reliably maintain high classification accuracy over time. We provide and validate answers to these challenges by extensive experimentation on real- world datasets.

Coordinating Human and Machine Intelligence to Classify Microblog Communica0o...

Muhammad Imran

We present AIDR (Artificial Intelligence for Disaster Response), a platform designed to perform automatic classification of crisis-related microblog communications. AIDR enables humans and machines to work together to apply human intelligence to large-scale data at high speed. The objective of AIDR is to classify messages that people post during disasters into a set of user-defined categories of information (e.g., “needs”, “damage”, etc.) For this purpose, the system continuously ingests data from Twitter, processes it (i.e., using machine learning classification techniques) and leverages human-participation (through crowdsourcing) in real-time. AIDR has been successfully tested to classify informative vs. non-informative tweets posted during the 2013 Pakistan Earthquake. Overall, we achieved a classification quality (measured using AUC) of 80%. AIDR is available at http://aidr.qcri.org/.

Artificial Intelligence for Disaster Response

Muhammad Imran

This work describes our work presented at the ISCRAM-2013 conference. We presented Tweet4act system, which is used to detect and classify crisis-related messages communicated over a microblogging platform. Our system relies on extracting content features from each message. These features and the use of an incident-specific dictionary allow us to determine the period type of an incident that each message belongs to.

Tweet4act: Using Incident-Specific Profiles for Classifying Crisis-Related Me...

Muhammad Imran

During mass convergence events such as natural disasters, microblogging platforms like Twitter are widely used by affected people to post situational awareness messages. These crisis related messages disperse among multiple categories like infrastructure damage, information about missing, injured, and dead people etc. The challenge here is to extract important situational updates from these messages, assign them appropriate informational categories, and finally summarize big trove of information in each category. In this paper, we propose a novel framework which first assigns tweets into different situational classes and then summarize those tweets. In the summarization phase, we propose a two stage summarization framework which first extracts a set of important tweets from the whole set of information through an Integer-linear programming (ILP) based optimization technique and then follows a word graph and content word based abstractive summarization technique to produce the final summary. Our method is time and memory efficient and outperforms the baseline in terms of quality, coverage of events, locations et al., effectiveness, and utility in disaster scenarios.

Summarizing Situational Tweets in Crisis Scenario

Muhammad Imran

This presentation describes our work presented at the 10th International Conference on Information Systems on Crisis Response and Management (ISCRAM) in Baden-Baden, Germany. The work shows the importance of microblogging websites such as Twitter, and huge number of informative messages that can contribute to situational awareness at the time of disasters. Specifically, the work shows the classification, and information extractions of those valuable, actionable informative messages that people post during emergencies.

Extracting Information Nuggets from Disaster-Related Messages in Social Media

Muhammad Imran

Introduction to Machine Learning: An Application to Disaster Response

Muhammad Imran

Viewers also liked (16)

Domain Specific Mashups

Mivestabelenpresentation

art of the_americas

Technology And Education

Prospectvision

14thcenteuropresentation

Amate bark painting presentation

A Real-time Heuristic-based Unsupervised Method for Name Disambiguation in Di...

The Role of Social Media and Artificial Intelligence for Disaster Response

Creative Commons

Coordinating Human and Machine Intelligence to Classify Microblog Communica0o...

Artificial Intelligence for Disaster Response

Tweet4act: Using Incident-Specific Profiles for Classifying Crisis-Related Me...

Summarizing Situational Tweets in Crisis Scenario

Extracting Information Nuggets from Disaster-Related Messages in Social Media

Introduction to Machine Learning: An Application to Disaster Response

Similar to Reseval Mashup Platform Talk at SECO

Unveiling the Ecosystem of Science: How can we characterize and assess divers...

Nicolas Robinson-Garcia

In the rapidly changing world of research and scholarly communications, researchers are faced with a fast growing range of options to publicly disseminate, review, and discuss research—options which will affect their long-term reputation. Early career scholars must be especially thoughtful in choosing how much effort to invest in dissemination and communication, and what strategies to use. Dr. Micah Altman briefly reviews a number of bibliometric and scientometric studies of quantitative research impact, a sampling of influential qualitative writings advising this area, and an environmental scan of emerging researcher profile systems. Based on this review, and on professional experience on dozens of review panels, Dr. Altman suggests some steps early career researchers may consider when disseminating their research and participating in public reviews and discussion.

Reputation Management for Early Career Researchers

Micah Altman

Providing Tools for Author Evaluation - A case study

inscit2006

What is bibliometrics and how does it work?

Ciarán Quinn

Bosman-Kramer Changing Research Workflows

National Information Standards Organization (NISO)

Introduction to Altmetrics for Medical and Special Librarians

Linda Galloway

Needs Assessment and Program Planning

Lynda Milne

Falk-Krzesinski, "Administrator (Institutional Use of the Data): Data-informe...

National Information Standards Organization (NISO)

Alternative metrics to evaluate scientific impact

Iara Vidal Pereira de Souza

With the enormous growth of data, retrieving information from the Web became more desirable and even more challenging because of the Big Data issues (e.g. noise, corruption, bad quality…etc.). Expert seeking, defined as returning a ranked list of expert researchers given a topic, has been a real concern in the last 15 years. This kind of task comes in handy when building scientific committees, requiring to identify the scholars’ experience to assign them the most suitable roles in addition to other factors as well. Due to the fact the Web is drowning with plenty of data, this opens up the opportunity to collect different kinds of expertise evidence. In this paper, we propose an expert seeking approach with specifying the most desirable features (i.e. criteria on which researcher’s evaluation is done) along with their estimation techniques. We utilized some machine learning techniques in our system and we aim at verifying the effectiveness of incorporating influential features that go beyond publications

TOWARDS A MULTI-FEATURE ENABLED APPROACH FOR OPTIMIZED EXPERT SEEKING

csandit

1430 delasalle uksg 2013

UKSG: connecting the knowledge community

2016 AAUDE

Danielle Pokusa Connelly

LITA’s Altmetrics and Digital Analytics Interest Group is proud to present Heather Coates, Richard Naples, and Lauren Collister in our second free webinar of the season. Heather will introduce the concept of altmetrics with a quick "Altmetrics 101," Richard will discuss the Smithsonian's implementation of Altmetric, and Lauren will share the University of Pittsburgh's experience with Plum Analytics.

Altmetrics 101 - Altmetrics in Libraries

IUPUI

CHiR presentation measuring scholarly and public impact

Plethora121

Scholarly Metrics in Specialized Settings

Elaine Lasda

Slides from Keynote presentation at the University of Southern California's 2015 Teaching with Technology annual conference. "9:15 am – ANN Auditorium Key Note: What Do We Mean by Learning Analytics? Leah Macfadyen, Director for Evaluation and Learning Analytics, University of British Columbia Executive Board, SoLAR (Society for Learning Analytics Research) Leah Macfadyen will define and explore the emerging and interdisciplinary field of learning analytics in the context of quantified and personalized learning. Leah will use actual examples and case studies to illustrate the range of stakeholders learning analytics may serve, the diverse array of questions they may be used to address, and the potential impact of learning analytics in higher education."

Macfadyen usc tlt keynote 2015.pptx

Leah Macfadyen

Early Career Tactics to Increase Scholarly Impact

Elaine Lasda

Enhancing DMPTool: Further Streamlineing Data Mangement Planning Process

University of California Curation Center

Sparc-Japan-Slow-revolution-in-scholarly-communication

hierohiero

Methods for measuring citizen-science impact

Luigi Ceccaroni

Similar to Reseval Mashup Platform Talk at SECO (20)

Unveiling the Ecosystem of Science: How can we characterize and assess divers...

Reputation Management for Early Career Researchers

Providing Tools for Author Evaluation - A case study

What is bibliometrics and how does it work?

Bosman-Kramer Changing Research Workflows

Introduction to Altmetrics for Medical and Special Librarians

Needs Assessment and Program Planning

Falk-Krzesinski, "Administrator (Institutional Use of the Data): Data-informe...

Alternative metrics to evaluate scientific impact

TOWARDS A MULTI-FEATURE ENABLED APPROACH FOR OPTIMIZED EXPERT SEEKING

1430 delasalle uksg 2013

2016 AAUDE

Altmetrics 101 - Altmetrics in Libraries

CHiR presentation measuring scholarly and public impact

Scholarly Metrics in Specialized Settings

Macfadyen usc tlt keynote 2015.pptx

Early Career Tactics to Increase Scholarly Impact

Enhancing DMPTool: Further Streamlineing Data Mangement Planning Process

Sparc-Japan-Slow-revolution-in-scholarly-communication

Methods for measuring citizen-science impact

More from Muhammad Imran

Millions of people use social media to share information during disasters and mass emergencies. Information available on social media, particularly in the early hours of an event when few other sources are available, can be extremely valuable for emergency responders and decision makers, helping them gain situational awareness and plan relief efforts. Processing social media content to obtain such information involves solving multiple challenges, including parsing brief and informal messages, handling information overload, and prioritizing different types of information. These challenges can be mapped to information processing operations such as filtering, classifying, ranking, aggregating, extracting, and summarizing. This work highlights these challenges and presents state of the art computational techniques to deal with social media messages, focusing on their application to crisis scenarios.

Processing Social Media Messages in Mass Emergency: A Survey

Muhammad Imran

Rapid access to situation-sensitive data through social media networks creates new opportunities to address a number of real-world problems. Damage assessment during disasters is a core situational awareness task for many humanitarian organizations that traditionally takes weeks and months. In this work, we analyze images posted on social media platforms during natural disasters to determine the level of damage caused by the disasters. We employ state-of-the-art machine learning techniques to perform an extensive experimentation of damage assessment using images from four major natural disasters. We show that the domain-specific fine-tuning of deep Convolutional Neural Networks (CNN) outperforms other state-of-the-art techniques such as Bag-of-Visual-Words (BoVW). High classification accuracy under both event-specific and cross-event test settings demonstrate that the proposed approach can effectively adapt deep-CNN features to identify the severity of destruction from social media images taken after a disaster strike.

Damage Assessment from Social Media Imagery Data During Disasters

Muhammad Imran

We present an end-to-end social media image processing system called Image4Act. The system aims at collecting, denoising, and classifying imagery content posted on social media platforms to help humanitarian organizations in gaining situational awareness and launching relief operations. The system combines human computation and machine learning techniques to process high-volume social media imagery content in real time during natural and human-made disasters. To cope with the noisy nature of the social media imagery data, we use a deep neural network and perceptual hashing techniques to filter out irrelevant and duplicate images. Furthermore, we present a specific use case to assess the severity of infrastructure damage incurred by a disaster. The evaluations of the system on existing disaster datasets as well as a real-world deployment during a recent cyclone prove the effectiveness of the system.

Image4Act: Online Social Media Image Processing for Disaster Response

Muhammad Imran

Real-Time Processing of Social Media Content for Social Good

Muhammad Imran

AIDR Tutorial (Artificial Intelligence for Disaster Response)

Muhammad Imran

An emerging challenge in the online classification of social media data streams is to keep the categories used for classification up-to-date. In this paper, we propose an innovative framework based on an Expert-Machine-Crowd (EMC) triad to help categorize items by continuously identifying novel concepts in heterogeneous data streams often riddled with outliers. We unify constrained clustering and outlier detection by formulating a novel optimization problem: COD-Means. We design an algorithm to solve the COD-Means problem and show that COD-Means will not only help detect novel categories but also seamlessly discover human annotation errors and improve the overall quality of the categorization process. Experiments on diverse real data sets demonstrate that our approach is both effective and efficient.

A Robust Framework for Classifying Evolving Document Streams in an Expert-Mac...

Muhammad Imran

More from Muhammad Imran (6)

Processing Social Media Messages in Mass Emergency: A Survey

Damage Assessment from Social Media Imagery Data During Disasters

Image4Act: Online Social Media Image Processing for Disaster Response

Real-Time Processing of Social Media Content for Social Good

AIDR Tutorial (Artificial Intelligence for Disaster Response)

A Robust Framework for Classifying Evolving Document Streams in an Expert-Mac...

Reseval Mashup Platform Talk at SECO

1. An Open Mashup Platform ForResearch Impact Evaluation By: Muhammad Imran

2. EXISTING SOLUTIONS: Web of Science Publish or Perish Tool DBLP

5. Groups

6. Artifacts

7. Universities

9. Future Policy

10. Finding Expert

11. Hiring

12.

13. DBLP

14. Scopus

15. ACM

16.

17. G-Index

18. C-Index

19.

20. Diff normalization

21. Self-Citation

22. Filters…

23. DIMENSIONS INRESEARCH EVALUATION Different Things Different Purposes Different Data Sources Different Metrics Different Algorithms Research Evaluation

24.

25. Number of and value of Grants awarded

26. Number of awards (e.g. Nobel Prizes)

27. Number of patents

28. Number of post-graduate researchers

29. Publication counts

30. Citation counts

31. Bibliometric metrics

32. Voting

33. Reputation, Reviews, Rating (Liquid Journal)

34. And many more …

35. Peer Evaluation

36.

37. Complex Metric Computation Logic Diversity Autonomy Variability

38. PROPOSED SOLUTION Putting everything together Capability to define complex logic User Personalized metrics support User Input Data Presentation Mashup Magic Evaluate individual, groups … Data Data Common platform for various data sources

39. ARCHITECTURE MASHUP UI Mashup Exec Logic ResEval Components ResEval Domain Model ResEval Mashup Language Mashup Engine Resource Space Management System

40. ResEval Mashup 1 (the vision)

41. ResEval Mashup 2

42. ResEval Mashup 3

43. CONCLUSION & FUTURE WORK We proposed architecture for Personalized metrics definition Resource-Oriented Provide Impact evaluation for individual & groups Future work Mashup platform for complex metric logic Components as web service

44. Thank you.

Editor's Notes

General Problems:Data incompleteness (data sources)Predefined metrics for evaluationOnly allowed fixed interface based queries
Common platform to access various kind of scientific resources.Support the definition of personalized metrics.Capability to define complex queries using Research Evaluation Mash Queries.Capable to evaluate individual, contribution and group of researchers.Defining personalized metricsAnswering the complex queriesCustomization of user queryComprising different data sourcesGroup formation & composition

Reseval Mashup Platform Talk at SECO

Recommended

Recommended

More Related Content

What's hot

What's hot (18)

Viewers also liked

Viewers also liked (16)

Similar to Reseval Mashup Platform Talk at SECO

Similar to Reseval Mashup Platform Talk at SECO (20)

More from Muhammad Imran

More from Muhammad Imran (6)

Reseval Mashup Platform Talk at SECO

Editor's Notes