The document outlines the data collection and text processing methods utilized by a data science team, including monitoring user-generated content from various online sources. It explains techniques such as tokenization and case folding for text normalization, along with the calculation of TF-IDF (term frequency-inverse document frequency) to assess the relevance of words within a text corpus. The emphasis is on handling semi-structured data and employing effective representation methods for text analysis.

1. NADAR SARASWATHI
COLLEGE OF ARTS AND
SCIENCE
DATA SCIENCE & ANALYTICS
A.Uthra Devi
II M.Sc Computer Science

2. COLLECT RAW TEXT
 The data science team investigates the problem,
understands the necessary data source, and formulates
initial hypotheses
 Data must be collected before anything can happen
 The data science tam starts by actively monitoring
various websites for user generated contents. the user
generated contents being collected being collected
could be related articles from news portals and blogs.
Comments on ACME’s products from online shops or
reviews sites, or social media posts that contain
keywords phone or be book.

3.  Regardless of data source, the team would deal with
semi structured data such as HTML web pages.
Really simple syndication feeds. XML, or JavaScript
object notation files.
 Enough structure needs to be imposed to find the part
of the raw text that the team really cares about.
 Many news portals and blogs provide data feeds that
are in an open standard format. Such as RSS or XML.
 Regular expressions can find words and strings that
match particular patterns in the text effectively and
efficiently.

4. REPRESENT TEXT
 In this data representation step, raw text is first
transformed with text normalization techniques:
 Tokenization
 Case folding
Tokenization or tokenizing is the task of separating
words from the body of text.
Raw text is converted into collections of tokens after
the tokenization, where each token is a word.
A common approach is tokenizing on space. For
example, with the tweet shown previously.

5.  TOKENIZATION:
A common approach is tokenizing on spaces.
Example: text analysis sometimes called text analysis
Another way is to tokenize the text based on
punctuation marks and spaces.
Data science and big data analytics, ”has
become well accepted across academia and the
industry.

6.  CASE FOLDING:
It reduces all letters to lowercase
Example: text analysis sometimes called text
analytics=text analysis sometimes called text
analytics.
One needs to be cautious applying case folding
to tasks such as information extraction, sentiment
analysis, and machine translation.
If case folding must be present. One way to
reduce problems is to create a lookup table of words
not to be case folded.

7. TF-IDF
TF-IDF stands for Term Frequency Inverse Document
Frequency of records.
It can be defined as the calculation of how relevant a
word in a series or corpus is to a text.
 The meaning increases proportionally to the number
of times in the text a word appears but is compensated
by the word frequency in the corpus (data-set).

8. TF-IDF FORMULA

9. IDF
 Besides stop words, words that are more general in
meaning tend to appear more often, thus having
higher term frequencies.
 The additional variable should reduce the effect of the
term frequency as the term appears in more
documents.
 The highest corpus-wide term frequencies(TF)
 The highest document frequencies(DF)
 The highest inverse document frequencies(IDF)

10. THANK
YOU