The document outlines a presentation on identifying misinformation. It includes slides on introducing the project, research methods used, data collected, the design process, a demonstration of the final product, lessons learned, and potential changes for redoing the project. Research methods would include qualitative and quantitative approaches. Data was collected from various platforms, repositories, and scraping methods. The final product design process involved stakeholder input. The presentation would show screenshots and demos of the designed system. Reflections on lessons learned and potential changes would be shared.
This project aimed to create a series of models for the extraction of Named Entities (People, Locations, Organizations, Dates) from news headlines obtained online. We created two models: a traditional Natural Processing Language Model using Maximum Entropy , and a Deep Neural Network Model using pre-trained word embeddings. Accuracy results of both models show similar performance, but the requirements and limitations of both models are different and can help determine what type of model is best suited for each specific use case.
This project aimed to create a series of models for the extraction of Named Entities (People, Locations, Organizations, Dates) from news headlines obtained online. We created two models: a traditional Natural Processing Language Model using Maximum Entropy , and a Deep Neural Network Model using pre-trained word embeddings. Accuracy results of both models show similar performance, but the requirements and limitations of both models are different and can help determine what type of model is best suited for each specific use case
Temporal and semantic analysis of richly typed social networks from user-gene...Zide Meng
We propose an approach to detect topics, overlapping communities of interest, expertise, trends and activities in user-generated content sites and in particular in question-answering forums such as StackOverflow. We first describe QASM (Question & Answer Social Media), a system based on social network analysis to manage the two main resources in question-answering sites: users and content. We also introduce the QASM vocabulary used to formalize both the level of interest and the expertise of users on topics. We then propose an efficient approach to detect communities of interest. It relies on another method to enrich questions with a more general tag when needed. We compared three detection methods on a dataset extracted from the popular Q&A site StackOverflow. Our method based on topic modeling and user membership assignment is shown to be much simpler and faster while preserving the quality of detection. We then propose an additional method to automatically generate a label for a detected topic by analyzing the meaning and links of its bag of words. We conduct a user study to compare different algorithms to choose a label. Finally we extend our probabilistic graphical model to jointly model topics, expertise, activities and trends. We performed experiments with real-world data to confirm the effectiveness of our joint model, studying user behaviors and topic dynamics.
http://www-sop.inria.fr/members/Zide.Meng/
We address major challenges in searching temporal document collections. In such collections, documents are created and/or edited over time. Examples of temporal document collections are web archives, news archives, blogs, personal emails and enterprise documents. Unfortunately, traditional IR approaches based on term-matching only can give unsatisfactory results when searching temporal document collections. The reason for this is twofold: the contents of documents are strongly time-dependent, i.e., documents are about events happened at particular time periods, and a query representing an information need can be time-dependent as well, i.e., a temporal query. Our contributions are different time-aware approaches within three topics in IR: content analysis, query analysis, and retrieval and ranking models. In particular, we aim at improving the retrieval effectiveness by 1) analyzing the contents of temporal document collections, 2) performing an analysis of temporal queries, and 3) explicitly modeling the time dimension into retrieval and ranking.
Leveraging the time dimension in ranking can improve the retrieval effectiveness if information about the creation or publication time of documents is available. We analyze the contents of documents in order to determine the time of non-timestamped documents using temporal language models. We subsequently employ the temporal language models for determining the time of implicit temporal queries, and the determined time is used for re-ranking search results in order to improve the retrieval effectiveness. We study the effect of terminology changes over time and propose an approach to handling terminology changes using time-based synonyms.
In addition, we propose different methods for predicting the effectiveness of temporal queries, so that a particular query enhancement technique can be performed to improve the overall performance. When the time dimension is incorporated into ranking, documents will be ranked according to both textual and temporal similarity. In this case, time uncertainty should also be taken into account. Thus, we propose a ranking model that considers the time uncertainty, and improve ranking by combining multiple features using learning-to-rank techniques. Through extensive evaluation, we show that our proposed time-aware approaches outperform traditional retrieval methods and improve the retrieval effectiveness in searching temporal document collections.
With the rise of Web 2.0, API-based software has appeared. This article examines the API-based search tool created for the Korean search engine Naver: Webonaver (Webometrics Tool for Naver). The software is able to collect large amounts of data automatically and can easily distinguish between different types of information on the web, which was impossible before. In particular, Internet researchers can improve efficiency of data analysis within a specified timeframe using this tool. This paper illustrates how to use WeboNaver and tries to verify the usability and reliability through several case studies. In this article, Korean National Assembly Members’ web presence was analyzed, as was the web presence of the term H1N1.
Web 2.0의 도래와 함께 Open API를 응용한 소프트웨어 프로그램이 등장하면서 더 이상 사용자들은 웹에서 정보를 수동으로 검색하면서 일일이 살펴보는 번거로움을 겪지 않아도 된다. 공개된 API를 활용해 몇 번의 간단한 조작으로 방대한 데이터를 체계적으로 수집하고 관리할 수 있다. 본 논문은 Open API를 응용해 개발한 검색전문 프로그램 WeboNaver(Webometrics Tool for Naver)를 소개한다. 이는 한국에서 가장 영향력 있는 검색엔진 중의 하나인 네이버를 이용해 방대한 데이터를 카테고리별로 자동수집하여 저장해주는 프로그램이다. 연구자들은 이를 활용해 데이터 관리와 처리, 분석 과정에 정확성과 고도의 효율성을 기할 수 있을 것이다. 논문의 목적은 WeboNaver의 사용을 원하는 학생, 일반인, 연구자의 이해를 돕고자 실제 사례들을 통하여 분석절차를 구체적으로 제시해 그 유용성을 입증하는 것이다. 이 프로그램을 사용하여 18대 국회의원 292명의 웹가시성을 조사하였다. 또한 신종플루와 관련된 단어들의 웹 가시성을 분석하였다.
Slides for the first meeting of the course 'Big Data and Automated Content Analysis' at the Department of Communication Science, University of Amsterdam
This project aimed to create a series of models for the extraction of Named Entities (People, Locations, Organizations, Dates) from news headlines obtained online. We created two models: a traditional Natural Processing Language Model using Maximum Entropy , and a Deep Neural Network Model using pre-trained word embeddings. Accuracy results of both models show similar performance, but the requirements and limitations of both models are different and can help determine what type of model is best suited for each specific use case.
This project aimed to create a series of models for the extraction of Named Entities (People, Locations, Organizations, Dates) from news headlines obtained online. We created two models: a traditional Natural Processing Language Model using Maximum Entropy , and a Deep Neural Network Model using pre-trained word embeddings. Accuracy results of both models show similar performance, but the requirements and limitations of both models are different and can help determine what type of model is best suited for each specific use case
Temporal and semantic analysis of richly typed social networks from user-gene...Zide Meng
We propose an approach to detect topics, overlapping communities of interest, expertise, trends and activities in user-generated content sites and in particular in question-answering forums such as StackOverflow. We first describe QASM (Question & Answer Social Media), a system based on social network analysis to manage the two main resources in question-answering sites: users and content. We also introduce the QASM vocabulary used to formalize both the level of interest and the expertise of users on topics. We then propose an efficient approach to detect communities of interest. It relies on another method to enrich questions with a more general tag when needed. We compared three detection methods on a dataset extracted from the popular Q&A site StackOverflow. Our method based on topic modeling and user membership assignment is shown to be much simpler and faster while preserving the quality of detection. We then propose an additional method to automatically generate a label for a detected topic by analyzing the meaning and links of its bag of words. We conduct a user study to compare different algorithms to choose a label. Finally we extend our probabilistic graphical model to jointly model topics, expertise, activities and trends. We performed experiments with real-world data to confirm the effectiveness of our joint model, studying user behaviors and topic dynamics.
http://www-sop.inria.fr/members/Zide.Meng/
We address major challenges in searching temporal document collections. In such collections, documents are created and/or edited over time. Examples of temporal document collections are web archives, news archives, blogs, personal emails and enterprise documents. Unfortunately, traditional IR approaches based on term-matching only can give unsatisfactory results when searching temporal document collections. The reason for this is twofold: the contents of documents are strongly time-dependent, i.e., documents are about events happened at particular time periods, and a query representing an information need can be time-dependent as well, i.e., a temporal query. Our contributions are different time-aware approaches within three topics in IR: content analysis, query analysis, and retrieval and ranking models. In particular, we aim at improving the retrieval effectiveness by 1) analyzing the contents of temporal document collections, 2) performing an analysis of temporal queries, and 3) explicitly modeling the time dimension into retrieval and ranking.
Leveraging the time dimension in ranking can improve the retrieval effectiveness if information about the creation or publication time of documents is available. We analyze the contents of documents in order to determine the time of non-timestamped documents using temporal language models. We subsequently employ the temporal language models for determining the time of implicit temporal queries, and the determined time is used for re-ranking search results in order to improve the retrieval effectiveness. We study the effect of terminology changes over time and propose an approach to handling terminology changes using time-based synonyms.
In addition, we propose different methods for predicting the effectiveness of temporal queries, so that a particular query enhancement technique can be performed to improve the overall performance. When the time dimension is incorporated into ranking, documents will be ranked according to both textual and temporal similarity. In this case, time uncertainty should also be taken into account. Thus, we propose a ranking model that considers the time uncertainty, and improve ranking by combining multiple features using learning-to-rank techniques. Through extensive evaluation, we show that our proposed time-aware approaches outperform traditional retrieval methods and improve the retrieval effectiveness in searching temporal document collections.
With the rise of Web 2.0, API-based software has appeared. This article examines the API-based search tool created for the Korean search engine Naver: Webonaver (Webometrics Tool for Naver). The software is able to collect large amounts of data automatically and can easily distinguish between different types of information on the web, which was impossible before. In particular, Internet researchers can improve efficiency of data analysis within a specified timeframe using this tool. This paper illustrates how to use WeboNaver and tries to verify the usability and reliability through several case studies. In this article, Korean National Assembly Members’ web presence was analyzed, as was the web presence of the term H1N1.
Web 2.0의 도래와 함께 Open API를 응용한 소프트웨어 프로그램이 등장하면서 더 이상 사용자들은 웹에서 정보를 수동으로 검색하면서 일일이 살펴보는 번거로움을 겪지 않아도 된다. 공개된 API를 활용해 몇 번의 간단한 조작으로 방대한 데이터를 체계적으로 수집하고 관리할 수 있다. 본 논문은 Open API를 응용해 개발한 검색전문 프로그램 WeboNaver(Webometrics Tool for Naver)를 소개한다. 이는 한국에서 가장 영향력 있는 검색엔진 중의 하나인 네이버를 이용해 방대한 데이터를 카테고리별로 자동수집하여 저장해주는 프로그램이다. 연구자들은 이를 활용해 데이터 관리와 처리, 분석 과정에 정확성과 고도의 효율성을 기할 수 있을 것이다. 논문의 목적은 WeboNaver의 사용을 원하는 학생, 일반인, 연구자의 이해를 돕고자 실제 사례들을 통하여 분석절차를 구체적으로 제시해 그 유용성을 입증하는 것이다. 이 프로그램을 사용하여 18대 국회의원 292명의 웹가시성을 조사하였다. 또한 신종플루와 관련된 단어들의 웹 가시성을 분석하였다.
Slides for the first meeting of the course 'Big Data and Automated Content Analysis' at the Department of Communication Science, University of Amsterdam
Hoaxy is a tool to visualize the spread of URLs consisting low-credible web-documents. We use features related to propagation . dynamics to classify the duplicates of low-credible claims.
The Materials Data Facility: A Distributed Model for the Materials Data Commu...Ben Blaiszik
Presentation given at the UIUC Workshop on Materials Computation: data science and multiscale modeling. Materials Data Facility data publication, discovery, Globus, and associated python and REST interfaces are discussed. Video available soon.
Crowdsourced query augmentation through the semantic discovery of domain spec...Trey Grainger
Talk Abstract: Most work in semantic search has thus far focused upon either manually building language-specific taxonomies/ontologies or upon automatic techniques such as clustering or dimensionality reduction to discover latent semantic links within the content that is being searched. The former is very labor intensive and is hard to maintain, while the latter is prone to noise and may be hard for a human to understand or to interact with directly. We believe that the links between similar user’s queries represent a largely untapped source for discovering latent semantic relationships between search terms. The proposed system is capable of mining user search logs to discover semantic relationships between key phrases in a manner that is language agnostic, human understandable, and virtually noise-free.
Slides for the first meeting of the course 'Big Data and Automated Content Analysis' at the Department of Communication Science, University of Amsterdam
Querylog-based Assessment of Retrievability Bias in a Large Newspaper CorpusMyriam Traub
This is the set of slides for my presentation at JCDL 2016 in Newark, USA.
Bias in the retrieval of documents can directly influence the information access of a digital library. In the worst case, systematic favoritism for a certain type of document can render other parts of the collection invisible to users. This potential bias can be evaluated by measuring the retrievability for all documents in a collection. Previous evaluations have been performed on TREC collections using simulated query sets. The question remains, however, how representative this approach is of more realistic settings. To address this question, we investigate the effectiveness of the retrievability measure using a large digitized newspaper corpus, featuring two characteristics that distinguishes our experiments from previous studies: (1) compared to TREC collections, our collection contains noise originating from OCR processing, historical spelling and use of language; and (2) instead of simulated queries, the collection comes with real user query logs including click data.
First, we assess the retrievability bias imposed on the newspaper collection by different IR models. We assess the retrievability measure and confirm its ability to capture the retrievability bias in our setup. Second, we show how simulated queries differ from real user queries regarding term frequency and prevalence of named entities, and how this affects the retrievability results.
Thomas Heinis is a post-doctoral researcher in the database group at EPFL. His research focuses on scalable data management algorithms for large-scale scientific applications. Thomas is a part of the "Human Brain Project" and currently works with neuroscientists to develop the data management infrastructure necessary for scaling up brain simulations. Prior to joining EPFL, Thomas completed his Ph.D. in the Systems Group at ETH Zurich, where he pursued research in workflow execution systems as well as data provenance.
Exploiting temporal information in retrieval of archived documents (doctoral ...Nattiya Kanhabua
In a text retrieval community, many researchers have shown a good quality of searching a current snapshot of the Web. However, only a small number have demonstrated a good quality of searching a long-term archival domain, where documents are preserved for a long time, i.e., ten years or more. In such a domain, a search application is not only applicable for archivists or historians, but also in a context of national library and enterprise search (searching document repositories, emails, etc.). In the rest of this paper, we will explain three problems of searching document archives and propose possible approaches to solve these problems. Our main research question is: How to improve the quality of search in a document archive using temporal information?
FINDING OUT NOISY PATTERNS FOR RELATION EXTRACTION OF BANGLA SENTENCESkevig
Relation extraction is one of the most important parts of natural language processing. It is the process of extracting relationships from a text. Extracted relationships actually occur between two or more entities of a certain type and these relations may have different patterns. The goal of the paper is to find out the noisy patterns for relation extraction of Bangla sentences. For the research work, seed tuples were needed containing two entities and the relation between them. We can get seed tuples from Freebase. Freebase is a large collaborative knowledge base and database of general, structured information for public use. But for Bangla language, there is no available Freebase. So we made Bangla Freebase which was the real challenge and it can be used for any other NLP based works. Then we tried to find out the noisy patterns for relation extraction by measuring conflict score.
Building a semantic search system - one that can correctly parse and interpret end-user intent and return the ideal results for users’ queries - is not an easy task. It requires semantically parsing the terms, phrases, and structure within queries, disambiguating polysemous terms, correcting misspellings, expanding to conceptually synonymous or related concepts, and rewriting queries in a way that maps the correct interpretation of each end user’s query into the ideal representation of features and weights that will return the best results for that user. Not only that, but the above must often be done within the confines of a very specific domain - ripe with its own jargon and linguistic and conceptual nuances.
This talk will walk through the anatomy of a semantic search system and how each of the pieces described above fit together to deliver a final solution. We'll leverage several recently-released capabilities in Apache Solr (the Semantic Knowledge Graph, Solr Text Tagger, Statistical Phrase Identifier) and Lucidworks Fusion (query log mining, misspelling job, word2vec job, query pipelines, relevancy experiment backtesting) to show you an end-to-end working Semantic Search system that can automatically learn the nuances of any domain and deliver a substantially more relevant search experience.
Slides for the second meeting of the course 'Big Data and Automated Content Analysis' at the Department of Communication Science, University of Amsterdam
Hoaxy is a tool to visualize the spread of URLs consisting low-credible web-documents. We use features related to propagation . dynamics to classify the duplicates of low-credible claims.
The Materials Data Facility: A Distributed Model for the Materials Data Commu...Ben Blaiszik
Presentation given at the UIUC Workshop on Materials Computation: data science and multiscale modeling. Materials Data Facility data publication, discovery, Globus, and associated python and REST interfaces are discussed. Video available soon.
Crowdsourced query augmentation through the semantic discovery of domain spec...Trey Grainger
Talk Abstract: Most work in semantic search has thus far focused upon either manually building language-specific taxonomies/ontologies or upon automatic techniques such as clustering or dimensionality reduction to discover latent semantic links within the content that is being searched. The former is very labor intensive and is hard to maintain, while the latter is prone to noise and may be hard for a human to understand or to interact with directly. We believe that the links between similar user’s queries represent a largely untapped source for discovering latent semantic relationships between search terms. The proposed system is capable of mining user search logs to discover semantic relationships between key phrases in a manner that is language agnostic, human understandable, and virtually noise-free.
Slides for the first meeting of the course 'Big Data and Automated Content Analysis' at the Department of Communication Science, University of Amsterdam
Querylog-based Assessment of Retrievability Bias in a Large Newspaper CorpusMyriam Traub
This is the set of slides for my presentation at JCDL 2016 in Newark, USA.
Bias in the retrieval of documents can directly influence the information access of a digital library. In the worst case, systematic favoritism for a certain type of document can render other parts of the collection invisible to users. This potential bias can be evaluated by measuring the retrievability for all documents in a collection. Previous evaluations have been performed on TREC collections using simulated query sets. The question remains, however, how representative this approach is of more realistic settings. To address this question, we investigate the effectiveness of the retrievability measure using a large digitized newspaper corpus, featuring two characteristics that distinguishes our experiments from previous studies: (1) compared to TREC collections, our collection contains noise originating from OCR processing, historical spelling and use of language; and (2) instead of simulated queries, the collection comes with real user query logs including click data.
First, we assess the retrievability bias imposed on the newspaper collection by different IR models. We assess the retrievability measure and confirm its ability to capture the retrievability bias in our setup. Second, we show how simulated queries differ from real user queries regarding term frequency and prevalence of named entities, and how this affects the retrievability results.
Thomas Heinis is a post-doctoral researcher in the database group at EPFL. His research focuses on scalable data management algorithms for large-scale scientific applications. Thomas is a part of the "Human Brain Project" and currently works with neuroscientists to develop the data management infrastructure necessary for scaling up brain simulations. Prior to joining EPFL, Thomas completed his Ph.D. in the Systems Group at ETH Zurich, where he pursued research in workflow execution systems as well as data provenance.
Exploiting temporal information in retrieval of archived documents (doctoral ...Nattiya Kanhabua
In a text retrieval community, many researchers have shown a good quality of searching a current snapshot of the Web. However, only a small number have demonstrated a good quality of searching a long-term archival domain, where documents are preserved for a long time, i.e., ten years or more. In such a domain, a search application is not only applicable for archivists or historians, but also in a context of national library and enterprise search (searching document repositories, emails, etc.). In the rest of this paper, we will explain three problems of searching document archives and propose possible approaches to solve these problems. Our main research question is: How to improve the quality of search in a document archive using temporal information?
FINDING OUT NOISY PATTERNS FOR RELATION EXTRACTION OF BANGLA SENTENCESkevig
Relation extraction is one of the most important parts of natural language processing. It is the process of extracting relationships from a text. Extracted relationships actually occur between two or more entities of a certain type and these relations may have different patterns. The goal of the paper is to find out the noisy patterns for relation extraction of Bangla sentences. For the research work, seed tuples were needed containing two entities and the relation between them. We can get seed tuples from Freebase. Freebase is a large collaborative knowledge base and database of general, structured information for public use. But for Bangla language, there is no available Freebase. So we made Bangla Freebase which was the real challenge and it can be used for any other NLP based works. Then we tried to find out the noisy patterns for relation extraction by measuring conflict score.
Building a semantic search system - one that can correctly parse and interpret end-user intent and return the ideal results for users’ queries - is not an easy task. It requires semantically parsing the terms, phrases, and structure within queries, disambiguating polysemous terms, correcting misspellings, expanding to conceptually synonymous or related concepts, and rewriting queries in a way that maps the correct interpretation of each end user’s query into the ideal representation of features and weights that will return the best results for that user. Not only that, but the above must often be done within the confines of a very specific domain - ripe with its own jargon and linguistic and conceptual nuances.
This talk will walk through the anatomy of a semantic search system and how each of the pieces described above fit together to deliver a final solution. We'll leverage several recently-released capabilities in Apache Solr (the Semantic Knowledge Graph, Solr Text Tagger, Statistical Phrase Identifier) and Lucidworks Fusion (query log mining, misspelling job, word2vec job, query pipelines, relevancy experiment backtesting) to show you an end-to-end working Semantic Search system that can automatically learn the nuances of any domain and deliver a substantially more relevant search experience.
Slides for the second meeting of the course 'Big Data and Automated Content Analysis' at the Department of Communication Science, University of Amsterdam
Experfy.com - This Big Data training gives one the background necessary to start doing analyst work on Big Data. It covers - areas like Big Data basics, Hadoop basics and tools like Hive and Pig - which allows one to load large data sets on Hadoop and start playing around with SQL Like queries over it using Hive and do analysis and Data Wrangling work with Pig.
The Big Data online course also teaches Machine Learning Basics and Data Science using R and also covers Mahout briefly - a Recommendation, Clustering Engine on Large data sets. The course includes hands-on exercises with Hadoop, Hive , Pig and R with some examples of using R to do Machine Learning and Data Science work
See more at: https://www.experfy.com/training/courses/big-data-analyst
Responsible conduct of research: Data ManagementC. Tobin Magle
A presentation for the Food and Nutrition Science Responsible conduct of research class on data management best practices. Covers material in the context of writing a data management plan.
The scientific and economic value of research data is enormous. To ensure successful subsequent usage, the scientific community needs efficient access to data, the data has to be reliable and persistent, and the quality of the data has to be proved.
One solution to these preconditions is to apply the techniques of today’s scientific publishing to research data. Besides its publication in a data repository together with some metadata, the data should undergo a transparent public peer-review using a publication platform.
The presentation discusses two approaches. On the one hand, the data can be the basis for a research article and undergoes a review parallel to the review of the manuscript. The data is then a reviewed supplement to a scientific publication. On the other hand, the data itself can be the subject of a publication whose quality is then assured by peers.
The presentation provides practical experience, especially with the latter strategy, realized through an established open access journal.
Citi Global T4I Accelerator Data and Analytics PresentationMarquis Cabrera
Presented on data and analytics for the Citi T4I Global Social Good Accelerator, which is an open innovation initiative seeking to source tech solutions that promote integrity around the world.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
Opendatabay - Open Data Marketplace.pptxOpendatabay
Opendatabay.com unlocks the power of data for everyone. Open Data Marketplace fosters a collaborative hub for data enthusiasts to explore, share, and contribute to a vast collection of datasets.
First ever open hub for data enthusiasts to collaborate and innovate. A platform to explore, share, and contribute to a vast collection of datasets. Through robust quality control and innovative technologies like blockchain verification, opendatabay ensures the authenticity and reliability of datasets, empowering users to make data-driven decisions with confidence. Leverage cutting-edge AI technologies to enhance the data exploration, analysis, and discovery experience.
From intelligent search and recommendations to automated data productisation and quotation, Opendatabay AI-driven features streamline the data workflow. Finding the data you need shouldn't be a complex. Opendatabay simplifies the data acquisition process with an intuitive interface and robust search tools. Effortlessly explore, discover, and access the data you need, allowing you to focus on extracting valuable insights. Opendatabay breaks new ground with a dedicated, AI-generated, synthetic datasets.
Leverage these privacy-preserving datasets for training and testing AI models without compromising sensitive information. Opendatabay prioritizes transparency by providing detailed metadata, provenance information, and usage guidelines for each dataset, ensuring users have a comprehensive understanding of the data they're working with. By leveraging a powerful combination of distributed ledger technology and rigorous third-party audits Opendatabay ensures the authenticity and reliability of every dataset. Security is at the core of Opendatabay. Marketplace implements stringent security measures, including encryption, access controls, and regular vulnerability assessments, to safeguard your data and protect your privacy.
1. Slide 1 -- very brief introduction to your project. (Note, this is to help your classmates refresh their memory about your project, which should be very
short, one or two sentences highlight) Jerry
Slide 2 -- all research methods you have used to complete this project(for each method, use justone sentence to justify why it's necessary to adopt this
method)
Slide 3 -- all data you havecollected (a list of types of the data, including number of the email correspondence, number of the interviews, pages of
documents you reviewed, etc.) {平台，数据仓库（github、kaggle）和canvasoet里提供的数据 （covid，ukrane，伊朗），调查平台数据是否可
用}（Ziyan），爬数据方法（jerry）
清理数据 (Jerry)，数据可视化分析（SBS(lingyu)，power bi（））
Slide 4 ~ x -- final productdesign process (This should be the focus, tell us how your interaction with the sponsors, users, etc. informed your design
thinking, and how you came up with the design ideas) 每周任务，不过也可以放我们的dashboard设计，如何沟通 (lingyu)
Slide x+1 ~ y -- how your final productlooks like? (Note, a list of screen shots would be helpful, or a live demo but make sure your designed websiteworks
properly. You only have about 20 minutes in total, so don't wastetime on searching, finding, or fixing websitepages) 介绍wireframe（Jerry）
Slide y+1 -- Take away points (Whatyou havelearned from doing this project, shareyour valuable experiences)（ziyan）
Final slide -- if you are given an opportunity to re-do the project, whatmay you change??? (Jerry）
2. ITC 6040 Capstone
Final Report
Strategies for Identifying
Mis-/Disinformation
Team 2：
Ouyang Zhaode, Lingyu Hu, Ziyan Yan, Zixun Zhou
4. Mikhail Oet, PhD
Professor in Commerce and Economic
Development (CED) program
Northeastern University
Our Sponsors:
Mission:
To get the rightinformation to the
right people at the right time
Research
• PlatformResearch
• Data Repositories Research
• Data Scraping Methods Research
• U.S., China, Russian Research
DataVisualization
• Dashboard Design
DataAnalysis
• Data Cleaning
• Sentiment Analysis
• Word Semantic Analysis
What Are We Doing? Help Identifying Fake News
How We Identify?
6. Research Method---
Qualitative Research
1. Identify Research Questions：
• How to collect data?
• How to use a data repository?
• How to analyze a dataset?
2. Case Study
3. Research Report
9. Data Source (1)
Gi t Hub
❑ Provider of hosting program and it could offer the research results of fake news
❑ The results were not used, but we use the dataset
❑ Use the keywords
❑ Datasetsourced from Weibo about the false information of COVID-19
K a ggl e
❖ Datasetwebsite owned by Google
❖ Offer scientific topics
❖ Provides data on the issue of fake news about COVID-19
10. Data Source (2)
01
COVID-19
Source: Weibo, Twitter
Topic: Misinformation of COVID-19
02 Ukraine
Event Registry Ukraine-English Dataset
Event Registry Ukraine-Russian Dataset
From Twitter
03 Iran
The theme of Iran will be from Tweet by inputting keywords
13. Data We Collected
Qualitative
Quantitative
Primary
Secondary
Information gather from the guest speakers &
stakeholders(professors, sponsors, and other teams)
Articles and reports we read
Data we scrap on the social media and news websites
Data our sponsor provided&
data repositories we found
Primary
Secondary
COUNT
3 Guest Sections
10+Zoom Recording
15+ Meetings
30+ Emails
30+ Articles,
Reports & Videos
3 Experiment
Web Scrapings
5+ Data
Repositories We
Found
14. Data Scraping Methods Research
RSS feed to CSV (Online Converting Tools) DataCollectors (Octopus, BrightData)
Web Scraping (Python)
Methods Use Cases Difficulty
RSS feed to CSV Websites Providing RSS Feeds Low
Data Collectors PopularSocial Medias Medium
Web Scraping Static Websites High
16. Data Cleaning
MS Excel Power Query
• For CSV format
• Easy-to-use
• For ad-hocanalysis (One-time use)
* Limitations:
• Data should less than 1 million row
• Data should less than 1GB
Python
• For JSON or other data format
• Can cut a large data into many smaller files
• Cleaning as scale
• For data pipeline use (continuously data streaming)
* No limitation, but take more time and more effort
We can use
Google Sheet
to do batch
translation
17. Power BI
1. Data visualization
2. Data query
3. Data Modeling
4. Key data analysis
20. Ukraine – Russian
SBS Analysis
• All words content are Russian
• All records are news
• “Russian” and “Ukraine” appeared mostin
the dataset
• Specific words do not appear too much
21. Ukraine – Russian SBS Analysis
• Ukraine, Russian, and Putin care Topic 1 most
• NATO, Russian, and USA care Topic 2 most
• Xi and Biden care about Topic 5 most
22. Ukraine – Russian SBS Analysis
• T6 has a strong relationship with T2
• T5 has the second strong relationship
with T2
Conclusions:
• Ukraine, Russian and Putin care Winter Olympicsmost
• Russian, NATO and USA care potentialmilitaryactivities
• Xi and Biden care relationshipwith other countries
• Covid has stronger relationshipwith potentialmilitary activities
• Relationship incountries could influence the potential militaryactivities
23. Twitter Transparency
Project
Power BI Analysis
• [Hanya Kamu]:Only you
• Most hashtags are meaningless
• Tweet numbers in 2012
• Most tweets appeared in June and August
• Trend is unstable
25. User Input
External Data - Revenpack
News Articles
Dashboard
Fake Score
Sentiment
polyfact
propublica
Local Check
Based on
Historical Data
External Data - GDI
Contribution by Country
26. Dashboard - Data Prerequisite
News Article Dataset
External Data
31. Data Collection
Plan
1. Develop a Data Collection Plan
2. Us
1. What are we going to solve?
e.g., A list of issues
2. What consider success?
e.g., A Service Level Agreements (SLA)
3. What dataavailable?
4. What form does that data come in?
5. Where the datawill be collectedfrom?
6. Whether to measure a sample or the whole population?
7. What format the datawill be displayed?
We Did
We Missed
35. Week 1- 2 : Platform Exploration
Exploring platforms where the data can be crawled.
Platforms in Russian, English and Chinese.
If possible, crawl the data by learning new tools.
36. Week 3 – 4: Learning New Tools
Clean and filter the data.
1
Learn new tools: SBS (and its format),
Power BI, sentiment analysis
2
Provide some basic findings.
3
37. Week 5 – 6: Data Visualization
• Use data visualization tools to analyze
data
• Determine the final tools: SBS, Power
BI.
• Providing some findings.
38. Week 7 – 8: Dashboard learn and design
• Keep using SBS to analyze the data
• Design dashboard by learning from
Ravenpack
• Combine and provide sample
designed dashboard
Sample designed dashboard
39. Process to Final Product
1. Ask Questions
2. Create
3. Feedback
4. Research
5. Revise
6. Feedback
7. Continue Revise
