Download as PDF, PPTX
![Emails, Blogs, Tweets, Online Chat,……
Chat Logs
g
[12:51:13 PM] Geetha: alrite
[12:52:01 PM] Richa: id has valid pw not expired
[12:52:49 PM] Geetha: can't get to theh site
can t
[12:53:04 PM] Richa: network connection may be slow
[12:54:39 PM] Geetha: ok Im able to now
[12:54:53 PM] Richa: should I reset the password
13](https://image.slidesharecdn.com/bigdataveracitychallenges-140119101624-phpapp01/85/Big-data-veracity-challenges-13-320.jpg)
![Finding Patterns with Gaps
Need for
N d f patterns capturing variations in expressions
i
i i
i
i
Have you rented a car from us before
Have you rented a car before
Have you rented a car from <Rent_Agency> before
<Rent Agency>
Mining regular expression patterns over tokens or entity types
Each tt
E h pattern represented as a t k sequence
t d
token
[rented car before]
Token sequences mined efficiently using extension of apriori algorithm
2
8](https://image.slidesharecdn.com/bigdataveracitychallenges-140119101624-phpapp01/85/Big-data-veracity-challenges-28-320.jpg)
![Bayesian Fusion
In the Bayesian framework the degrees of belief are
represented by a priori, conditional and a posteriori
conditional,
probabilities.
Usually, decisions are made on a posteriori probabilities P(θn | yi ),
where yi i the input coming from source I,
h
is th i
t
i f
I
xi = P(θn | yi ) represents statistics of each source to be combined
(data, outputs of classifiers).
Fusion is
F i i performed by the Bayesian rule, which under the
f
d b th B
i
l
hi h d th
condition of source independence is reduced to a product:
Fn(x1,...,xI)|y =Fn(P)|yi =P(θn)∏[P(θn |yi)/P(θn)], n
This fusion operator is conjunctive and assumes total reliability
of the sources
61](https://image.slidesharecdn.com/bigdataveracitychallenges-140119101624-phpapp01/85/Big-data-veracity-challenges-61-320.jpg)
Uploaded byPrayukth K V
Big data veracity challenges
The document discusses the challenges of big data related to volume, velocity, variety, and veracity, highlighting that traditional data processing tools are insufficient for analyzing the vast, diverse, and often uncertain data generated today. Key issues include the prevalence of unreliable data and the rapid growth of data sources, such as social media, which complicates trust in information. It emphasizes the need for new approaches and solutions to effectively manage and extract meaningful insights from extensive datasets.
More Related Content
Similar to Big data veracity challenges
Big data veracity challenges
- 1. Big Data andVeracity Challenges Text Mining Workshop, ISI Kolkata L. Venkata Subramaniam L V k t S b i IBM Research India Jan 8, 2014 1
- 2. The Four Dimensionsof Big Data Volume l Velocity l i Data at Rest Data in Motion Terabytes to exabytes of existing data to process Streaming data, milliseconds to seconds to respond Variety i Data in Many Forms Structured, unstructured, text, multimedia Veracity* i * Data in Doubt Uncertainty due to data inconsistency & incompleteness, incompleteness ambiguities, latency, deception, model approximations * Truthfulness, accuracy or precision, correctness 2 2
- 3. We’ve Moved intoa New Era of Computing ! In order to realize new opportunities, you need to think beyond traditional sources of data The term Big Data applies to information that can’t be processed or analyzed using traditional processes or tools 12+ terabytes of Tweets eets created daily. 100’s of different types of data. 5+million trade events per second. Transactional & Application Data Machine Data Social Data Enterprise Content Volume Velocity Variety Veracity Only 1 in 3 decision makers trust their information. • Volume • Velocity • Variety • Variety • Str ct red Structured Semi• Semi structured • Highl Highly unstructured • Highl Highly unstructured • Ingestion • Veracity • Volume • Throughput 3
- 4. Volume is growingso are Veracity issues By 2015, 80% of all available data will be uncertain 2015 By 2015 the number of networked devices will be double the entire global population. All sensor data has uncertainty. 8000 100 90 7000 80 6000 70 5000 60 4000 50 3000 40 30 2000 Aggregat Uncertainty % te Glob Data Volume in Exaby bal ytes 9000 20 1000 The total number of social media accounts exceeds the entire global population. This data is highly uncertain in both its expression and content. Data quality solutions exist for enterprise data like p customer, product, and address data, but this is only a fraction of the total enterprise data. p 10 0 2005 4 Multiple sources: IDC Cisco IDC,Cisco 2010 2015
- 5. What is BigData? Big Data applies to information that can’t be processed or analyzed using traditional processes or tools Telco Profiles Data Vo olume, Ve elocity, Var riety Call Detail Records Market Trends Smart Grid Smarter Weather Cities Sensor Modeling Data Smarter Smarter Traffic Water Portfolio Risk Market Feeds Credit Card Transactions Medical Transcription Electronic Data Interchange CRM Customer Records Traditional Data & Processing Precise, authoritative, well formed 5 Text, Audio, Video Contact Centers Retail Fraud SWIFT Account Management Homeland Security Uncertainty (1/veracity) Disease Progression Patient P ti t Records Predictive Modeling of Outcomes Social Network Data Services S i Data Uncertainty at Scale Inconsistent, imprecise, uncertain, unverified, s pontaneous, ambiguous, deceptive
- 6. Social media usersin India India (No. of Users In Million) 50 45 40 India (No. of Users In India (No of Users In Platforms Million) 35 30 Facebook India (No. of Users In Million) 20 Twitter 15 Million Linkedin 25 45 Million 15 Million 15 10 5 0 Facebook Twitter Linkedin Youtube Google Plus 6
- 7. Veracity issues arisedue to: Process Uncertainty Processes contain “randomness” “ d ” Data Uncertainty Data input is uncertain All modeling is approximate Actual Spelling Intended y Spelling Text Entry p g ? ? ? Uncertain travel times Model Uncertainty GPS Uncertainty Fitting a curve to data ?? ? Testimony {Paris Airport} Ambiguity g y Semiconductor yield 7 Contaminated? Rumors {John Smith, Dallas} {John Smith, Kansas} Conflicting Data Forecasting a hurricane (www.noaa.gov) ( )
- 8.
- 9. Upto 10 0,000 timesmore noisy Big Data, Fast Data, Noisy Data Social Media Communication is meant for Friends 30% world population on the internet and increasing fast Type of Text WER SMS (texting) 50% Tweets 35% ASR 30% Web queries 15% OCR 5% Newswire Text (WSJ, Reuters, NYT) 0.005% 55 million illi Tweets per day Lead Generation, Disaster Tracking g Large Dimensional, uncertain, unverified I’ll see ya tomo RIP Jackson J k I’m lookie out 4 a car 2 burn rubber on the streets of LA What should I buy?? A mini laptop with Windows OR a Apple MacBook!??! Noisy, Informal, Noisy Informal Implicit and Contextual Conversations There are more social networking accounts t ki t than people in the world Social Networking overtakes Search: Facebook becomes the most visited website ahead of Google Big Data: More video content was uploaded onto YouTube in the past two months than all the new content ABC, ABC CBS and NBC have been entering 24/7 since 1948.” 9
- 10. SMS 0 there –there 1 aint – are not 2 no – no 3 doubt – doubt 4 there – there 5 hon – honey 6 im – I am 7 gonna – going 8 be – be 9 takin – taking 10 it – it 11 4 – for 12 life – life 13 u – You 14 wont – wont 15 b – be 16 rida – rid of 17 me – me 18 lol – laugh out loud 19 Ray – (NAME) Texting Language: Over 50% of the words are written in non standard ways Spontaneous Language: Use of slang, ungrammatical, no punctuations, no case information Mixing of Languages: Many SMS contain text in a mix of two or more languages Type of Noise % Deletion of Characters 48% Phonetic Substitution 33% Abbreviations 5% Dialectical Usage 4% 52% words were non standard Deletion of Words 1.2% (Contractor et al., 2010) 101 SMSes 10
- 11. Speech Recognition SPEAKER 1:windows thanks for calling and you can learn yes i don't mind it so then i went to SPEAKER 2: well and ok bring the machine front Recognition Errors: 10-40% Word Error Rates end loaded with a standard um and that's um it's a desktop machine and i did that everything was working wonderfully um I went ahead connected into my my network um so i i changed my network settings to um to my home network so i i can you know it's showing me for my workroom um and then it is said it had to reboot in order for changes to take effect so i rebooted and now it's asking me for a password which i never i never said anything up SPEAKER 1: ok just press the escape key i can Spontaneous Language: Use of slang, use of fillers slang like um and ah, ungrammatical, false starts, starts no punctuations, no punctuations case information doesn't do anything can you pull up so that i mean Mixing f L Mi i of Languages: Contain words from two or more languages 11
- 12. Historical Text Non StandardSpellings: No notion of the importance of having a single spelling for each word. Letters would be added or removed to ease line justification. New words: New words, words that are variants of present vocabulary words Different Language Style: Different grammar, language g g y g , g g model. OCR: Character substitution errors, missed punctuations. Baron et al. 2009 al 12
- 13. Emails, Blogs, Tweets,Online Chat,…… Chat Logs g [12:51:13 PM] Geetha: alrite [12:52:01 PM] Richa: id has valid pw not expired [12:52:49 PM] Geetha: can't get to theh site can t [12:53:04 PM] Richa: network connection may be slow [12:54:39 PM] Geetha: ok Im able to now [12:54:53 PM] Richa: should I reset the password 13
- 14. What is NoisyText? Any kind of difference in the surface form of an electronic text from the intended, correct or original text (Knoblock et al., 2007) Noise can be at the lexical level {b4 before befour} {b4, before, Resulting in substitution, insertion, deletion, transposition, run-on, and split. Noise can be at morphological, syntactic, discourse level {I can hear u, I can hear you, I can here you} Resulting in substitution, insertion, deletion, transposition of words and the introduction of out of vocabulary words. d 14
- 15. Classifying Noise Lexical Errors(Subramaniam et al., 2009) Missing characters {before > bef} Extra characters {raster > raaster} Phonetic substitution {before > b4, late > l8} , } Abbreviations {laugh out loud > lol, United Nations > UN} Syntactical Errors (Kukich, 1992; Foster et al., 2007) Missing Word {What are the subjects? > What the subjects?} Extra word {Was that in the summer? >Was that in the summer it?} Real word spelling errors {She could not comprehend > She could no comprehend. comprehend.} Agreement {She steered Melissa round a corner > She steered corner. Melissa round a corners.} Dialectical usage {I’m going to be there > I’ gonna b there} th I’m be th } 15
- 16. Techniques for AutomaticallyDetecting Lexical Errors (Kukich 92) Efficient methods to detect strings that do not appear in a given word list, dictionary or lexicon Nonword error d t ti N d detection Two approaches N-gram Look up each n-gram in an input string in a precompiled table to ascertain either its existence or its frequency. Nonexistent or infrequent n-grams (shj, i ) are identified as possible misspellings. hj iqn id tifi d ibl i lli Good for identifying errors made by OCR devices But unusual/foreign language valid words will be marked and nicelooking mistakes will be marked valid ill ma ked alid Dictionary based Input string appears in a dictionary? If not, the string is f f flagged as a misspelled word. But nearly two-thirds of the words in a dictionary did not appear in an eight million word corpus of New York Times text and conversely two text, and, twothirds of the words in the text were not in the dictionary (1986 study) 16
- 17. Techniques for automaticallyDetecting Incorrect (Syntax) Grammar (Foster et al., 2007) Efficient methods to detect word sequences that do not form a Effi i h d d d h d f grammatical sentence Three Approaches N-gram Classifies a sentence as ungrammatical if it contains an unusual part of speech sequence Precision-grammar Classifies a sentence using a parser and a broadcoverage hand-written grammar Probabilistic-parsing Probabilistic parsing Finds sentences with parsing error 17
- 18. Quantifying Noise (Subramaniamet al., 2009) Quantifying Lexical Errors {Before, b4, befour, befor, bfore} Edit Di Edi Distance Good for measuring surface level deviation from original Perplexity e p e ty Good for measuring deviation from underlying language structure at character level Quantifying Semantic Errors {I came to LA yesterday. I am still jet lagged., Came la yester day still jetlagged, Came 2 LA ystrday stil jetl8d} WER Good for measuring real word errors (speech recognition errors) Perplexity Good for measuring deviation from “proper” BLEU Good for comparing a candidate translation against multiple reference translations 18
- 19. Spelling Correction (Kukich,1992) Isolated Word Correction Minimum edit distance techniques Similarity key techniques Probabilistic techniques N-gram-based N gram based techniques Rule-based techniques Will not catch typos resulting in correctly spelled words {form, from} yp g y p , Estimates put real word errors at 30% of all word errors Context-Dependent Word Correction Parsing Language models Can errors be ignored and still meaningful interpretation be done? {I am coming with you, I comes with you} 19
- 20. SMS Text Normalization disis n eg 4 txtin lang This is an example for Texting language Extreme corruption of words and sentences Models for SMS language are lacking Tomorrow never dies!!! 2moro (9) ( ) tomoz (25) tomoro (12) tomrw (5) tom (2) tomra (2) tomorrow (24) tomora ( ) (4) tomm (1) ( ) tomo (3) tomorow (3) 2mro (2) morrow (1) tomor (2) tmorro (1) moro ( ) (1) Occurrence in a 1000 sms corpus 20
- 21. Finding Canonical Sets(Acharyya, 2009) Learn mappings costmer, castumar, kustamar, customer coustomber How can we do it in an unsupervised way ? Find some invariant, that does not change in spite of corruptions Buckets of context seem invariant! <..Back Bucket....> sceam <..Front Bucket...> sceam : sms(2) new(5) recharge(4) t l h tel-provider(2) about(3) id b t <..Back Bucket...> scheme <..Front Bucket...> scheme : sms(4) new(2) activate(3) tel-provider(2) about(1) recharge(1) 21
- 22. SMS Based FAQRetrieval (Kothari et al., 2009) SMS Question FAQ how 2 actvate romng on me hanset Database How do I activate Roaming Dial *567*2# from your handset What are the rates for roaming within India Roaming rates on prepaid connections are 60 Paise per minute SMS Answer Dial *567*2# from your handset Goal is to find the Question Q* that best matches the SMS S •A scoring function Score(Q) assigns a score to each question Q in the FAQ dataset. The score measures how closely the question matches the SMS string S S. 22
- 23. FAQ Retrieval ProblemFormulation SMS is treated as a sequence of tokens S=s1,s2,…,sn Let Θ denote the questions in the FAQ corpus where each question Q ∈ Θ is treated as a set of tokens Goal is to find the question Q* that best matches the SMS S 23
- 24. Method M th d For Feach t k si , a li t Li consisting of all t h token list i ti f ll terms f from th di ti the dictionary that are variants of si are constructed. Variants are sorted in the descending order of their weight This space is searched to find the closest matching FAQ question. 24
- 25. Extracting Dialog Models(Negi et al 2009) al., Huge number of repetitive calls at contact centers Building t k i t d di l B ildi task oriented dialog systems t Task specific information – concepts, subtasks Task structure - manual encoding g Using large amounts of human to human conversation data Extracting dialogue models using human-to-human conversations E t ti di l d l i h t h ti 25
- 26. Example Conversation: CarRental Domain 26
- 27. Overview Transcribed Calls Normalized Calls Utterance No a ato Normalization Subtasks Mining of Subtasks Chat-bot AIML Co ve s o Conversion 27
- 28. Finding Patterns withGaps Need for N d f patterns capturing variations in expressions i i i i i Have you rented a car from us before Have you rented a car before Have you rented a car from <Rent_Agency> before <Rent Agency> Mining regular expression patterns over tokens or entity types Each tt E h pattern represented as a t k sequence t d token [rented car before] Token sequences mined efficiently using extension of apriori algorithm 2 8
- 29. Association Analysis Total numberof possible itemsets is exponential (2N) Brute-force technique infeasible Support filtering is necessary • • To eliminate spurious patterns To avoid exponential search - Support has anti-monotone property: X ⊆ Y implies σ(Y) ≤ σ(X) Efficient algorithms have been designed to exhaustively find all itemsets/patterns with sufficiently high support Given d items, there are 2d possible candidate itemsets ibl did t it t 29
- 30. Utterance Normalization Identify concepts NamedEntity Annotation Rule based annotator for annotations such as location, date, car model, model and amount “I want to pick it up from <location> on <date>” Grouping of utterances Find patterns with gaps and represent each utterance by them along with unigrams and bi-grams Agent and customer utterances are clustered separately using an offg p y g the shelf clustering algorithm 30
- 31. Finding Subtasks andordering Customer and agents engage in similar kinds of interactions to accomplish an objective Represent each call with agent utterance and customer utterance cluster labels Subtasks Patterns of cluster labels (agents) with possible gaps Lot of variability in customer utterances Vertical pattern mining C1 C1 C2 C3 C3 Cn 31
- 32. Subtask Preconditions Utterance pre-conditions U dii Customer utterances that indicate start of a subtask “please make this booking for “make payment” subtask please booking” make payment Frequent features from customer utterances Flow pre-conditions Only logical orders of subtasks are allowed “make “ k payment” subtask cannot b executed unless “ th t” bt k t be t d l “gather pick-up information” subtask has been executed. Collection of all the subtasks that precede the subtask p 32
- 33.
- 34. Data Fusion Problem Given multipledata points about an entity, create a single p p y, g object representation while resolving conflicting data values Difficulties Null values: Subsumption and complementation Contradictions in data values Uncertainty & truth: Discover the true value and model u ce ta ty this process uncertainty in t s p ocess Metadata: Preferences, recency, correctness Lineage: Keep original values and their origin Implementation in DBMS: SQL, extended SQL, UDFs, etc. SQL SQL UDFs etc 34
- 35. 360 Context Analyze socialdata in the context of enterprise data to build entity and event profiles and establish linkages between them for online and offline analysis Entity (people, products, events) Insights The problem Solution What are the key product interests of person A? Over time learn about the person’s product interests from her social media postings p g What is the location and trajectory of person B? List significant events like marriage, birth of a child, relocation, etc. What are the events of interest happening in a given location? Lists the top events in a given geography What is the sentiment g product? on a given p Gives the sentiment on a product p Understand customers wants and needs better Gives the current location and locations in the past What life events happened in person A’s life in the past x months? Key Sustained Value Factor: intent to purchase for customers Social Data Smarter Commerce real-time public safety events Enterprise Databases User Domains What MDM 360 does? propensities/ sentiment/intent • event Detection • entity Linkages • sentiment core customer view/transactions • event Profiles • entity Profiles Smarter Cities Application Domains Builds an entity’s complete profile by aggregating data about the entity from social and enterprise data ld ’ l fl b d b h f l d d sources. Here an entity refers to people, products, brands and events. 35 35 IBM Confidential
- 36. Extraction Challenges: Stagesof intent Stage Example Wishing for an event “I just want to graduate, get a job, get a car, and live with my boyfriend” Anticipating an event “Im getting a car for graduation yay!!!!!” During an event “At disneyworld :D” Post event / continuous state “Apparently I got a raise at work three months ago and didn't know? Sweeeeeeeeeet” Hobby “Loves to fish, travel and frequent concerts. Down to earth, athletic, professional 40 and single. earth athletic professional, single Loves the outdoors, working out, travel and younger fit guys for dating.” 36
- 37. Extraction Challenges: Detectingfiltering conditions Filter Example Spam “Need a New #Credit Card for your #Business or online #Ebay store? Compare and Apply Online. http://retweet.it/r/We0iai” Sarcasm, jokes “I thought I was having a stroke this afternoon but it turns out it was too many Starbucks Refreshers plus my leg falling asleep.” Resolve ambiguous meaning “In the words of @LNSmooth23 I'm retiring from the nightlife” Non-personal “My mom is buying a house, but why in Willingboro” 37
- 38. 360-degree Profiles fromSocial Media g Personal Attributes Event Detection • Identifiers: what, where, when….. • Attributes: severity, urgency… Social Media based 360-degree Event and Individual Profiles Timely Insights on Events Ti l I i ht E t • Event Detection • Public Safety Events • Plans for public disturbances • Sentiment around events • Citizen sentiment • Identifiers: name, address, age, gender, occupation… • Interests: sports, pets cuisine sports pets, cuisine… • Life Cycle Status: marital, parental • Relationships: family, friends, co-workers, work and interest network Timely Insights on Individuals • Intent to participate in public events • Instigation for causing public damage • Sentiment on events, govt policies • Current Location • Hate messages Personal Interests P lI • Personal preferences or political leanings • Activity History Intent We must support the movement, I am going to the rally at Jantar Mantar tomorrow Anna Hazare has a point when he says politicians are corrupt and need to be taught a lesson. The rally starts at 10. Public Safety Events Mamta Deedi is also joining Anna, Ramdev, n Kejriwal. She is going to do Anshan at Delhi Jantar mantar. Ye sab public ki kahke le rahe hain. So Its Mamata's day out tomorrow at #JantarMantar. #Rally. Location announcements I'm at Karir Square http://4sq.com/fYReSj 38
- 39. More data: Customerintent extracted from social media provides context Go for the best, DP2000 Buying a DSLR today ! Buying DSLR today! Thrza gr8 deal on ZX 550 @ ZX-550 the mall Prior Business Social Transactions Data Entity Extraction, Fact Discovery, Intent & Sentiment Influencers Intent 450M+ tweets/day Millions of tweets yield one company-specific fact Customer ready to buy a DSLR camera today today, possibly at a nearby mall Michael’s online friends offer lots of advice Text Analytics used to extract intent from Social Media Married, Male, Spouse , , p Birthdate, Gift Type, Intent to Purchase, Timeframe Wifey’s birthday tomorrow, looking for a killer dslr Sarcasm, Wishful Thinking Potential Locations and Activity Maybe I should buy her that purple roadster, while I’m at it. ;-) lol Intent to Purchase, Gift Type? In NYC area this w/e, any good malls nearby? Region & City Location, Timeframe, Intent to Shop Resultant fact base contains billions of facts, and is incrementally updated Fact segmentation or clustering is rapid enough to drive a business decision 3939
- 40. Matching Twitter profilesto Corporate Data • Linking Social Media profiles with Employee database • Several extensions are possible, for example, linking with Citizens and Security databases Social media profiles (name, address, gender, age gender age, employment, relationship, …) Employment filter Social media profiles of IBM employees p y and their network Name: first, last Name, work location, job description Current Demo focused on Name and Location matching, as well as EmployeeOf information Choice of social media profile attributes for linking constrained by availability of IBM BluePage attributes Twitter: 45M profiles Resolution Semantic Name Variations Bill Chamberlin vs. Chamberlain, William H. C. Mohan vs. Mohan Chandrasekaran (Mohan) Employee Directory: 460K entries p y y Name: (first, middle, last, preferred) Geo Proximity Home l H location: city, ( t t ) country ti it (state), t Employment: company + role Saratoga, CA vs. San Jose, CA New Jersey vs. New York Job Role Disambiguation “Soft a e sales manage at IBM…” vs. Software manager IBM s “Managing SPSS Sales for Canada…” 40 Work l W k location: ( i state, zip, country) i (city, i ) Job description
- 41. Example Result • Semanticname variations: Twitter name is a close variation of the IBM names • Geo Proximity: Work locations are within 25mi of the Twitter location • Job Role Disambiguation : description in Twitter profile matches HR role 41
- 42. Common D tP bl C Data Problems • Lack of information standards t d d Ashok Kumar A Kumar • Data misplaced in the database Four sixteen Street 8 Anand Niketan Delhi 8, Niketan, Mr. Ashok Kr #416 Anand Niketan, N Delhi, 21 110021 • Different formats & structures across different systems Data surprises in individual fields 416 Anand Niketan, New Delhi, India 110021 Email Tax ID Telephone 91,,,, 228-02-1975 6173380300 ranivrgeoi@yahoo.co.in i i@ h i 025 37 1888 025-37-1888 415 392 2000 415-392-2000 ,CYRUS_DASTUR@HOTMAIL.COM 34-2671434 3380321 HP 15 State St. 508-466-1200 Orlando • Special characters in the data • The redundancy nightmare • Duplicate records with a lack of standards 90328574 90328575 01456 90238495 90233479 90233489 90345672 IBM I.B.M. Inc. 187 N.Pk. Str. Salem NH 01456 187 N.Pk. St. Salem NH Int. Bus. Machines International Bus. M. Inter-Nation Consults I.B. I B Manufacturing 187 No. Park St Salem NH 04156 187 Park Ave Salem NH 04156 15 Main Street Andover MA 02341 Park Blvd Bostno MA 04106 Blvd. 42
- 43. Address Variations… Variations • Spellingvariations, hyphenation, abbreviations • I 344 I-344 | Sarojini Nagar | N Delhi | 23 • 344 Block J | Sarojni Ngr | New Delhi | 110023 • 344 Block I | Sarojni Ngr | New Delhi | 110023 • Multiple Ways of writing the same field • 13B | Link Road | Versova | Mumbai • 18 Block M | Bandra Versova Link Rd | Versova | Mumbai Rd. • Missing Address Fields • 4 Block C | ISID Campus I 4, I V. Kunj I New Delhi | 110070 V • 4C I ISID Campus | Institutional Area| V. Kunj | New Delhi | 110070 • Errors • 4C I ISID Campus | Institutional Area| V. Kunj, New Delhi | 110007 43
- 44. Regional variations inAddresses across India Addresses in different regions contain words of the local language even when the addresses are written in English Ex : The commonly used word to describe a street type is “Gali” in Northern India whereas “Beedhi/Veedhi” is the commonly used term in Southern India Street Intersections and Street Information containing multiple Street Type Identifiers like Cross and Main are extensively found in the Southern Indian regions Ex : “3rd Main, 4th B Cross” , Sector and Pocket Information are found primarily in North Indian Addresses Ex : “Sector 5, Pocket 2A 2nd Block” Regional differences in writing addresses necessitate bifurcation of standardization rules based on regions. 44
- 45. Investigating the Data g g Takethe Example: 123 St. Virginia St. Parsing: Separates multi-valued fields into individual pieces Lexical A l i L i l Analysis: Determines business significance of individual pieces Context Sensitive: Identifies various data structures and content 123 Number 123 Number 123 St. Virginia Street Type Alpha St. Virginia Street Type St. Street Name Street Type St. Virginia St. “The instructions for handling the data are inherent within the data itself.” 45 St.
- 46. Sample Standardized Output SampleAddress Input: “SANT KRUPA BUILDING, 2ND FLOOR, CHHEDA RD, NR S V JOSHI HIGH SCHOOL, DOMBIVALI (E), THANE. INDIA.” Standardization Output: St d di ti O t t DoorNo Floor Value Building Name Building Type Street Name Street Type 20 2nd FLOOR SANT KRUPA BUILDING CHHEDA ROAD Landmark Position Landmark Area City District State NEAR S V JOSHI HIGH DOMBIVALLI SCHOOL EAST THANE THANE MAHARASHTRA 46
- 47. Input Addresses vsStandardized Addresses Sr.No Standardized address Highlights 1 A38/91 KONIA . . VARANASI INDIA A38/91 KONIA VARANASI VARANASI UTTARPRADESH INDIA Autopopulation of state 2 VILL BASUDEVPUR PO KHANJANCHAK DURGACHAK HALDIA TAMLUK INDIA DURGACHAK ,HALDIA,VILLAGEBASUDEVPUR PO-KHANJANCHAK PO KHANJANCHAK TAMLUK EAST MIDNAPORE WESTBENGAL INDIA Rural address Handling 3 NEAR RAJGHAR GIRLS SCHOOL LACHIT NAGAR HOUSE NO 5 ULUBARI GUWAHATI ASSAM GUWAHATI INDIA 5 NEAR RAJGHAR GIRLS SCHOOL Maintaining a ULUBARI LACHIT NAGAR GUWAHATI standard format KAMRUP ASSAM INDIA across addresses (house no preceeds Landmark information) 4 1/15, PREMJYOTI CO OP HSG 1/15 PREMJYOTI COOPERATIVE SOC., RAMBAUG - 5, KALYAN HOUSING SOCIETY,RAMBAUG 5 (W), MAHARASHTRA 421301 KALYAN WEST BHIWANDI THANE BHIWANDI INDIA MAHARASHTRA 421301 Standardization of Tokens 5 4 7 Input address 3/2,FIRINGI DANGA ROAD, P.O.MALLICKPARA SERAMPORE-3 CALCUTTA INDIA Standardization of tokens 3/2,FIRINGI DANGA ROAD, SERAMPORE-3 P.O.MALLICKPARA KOLKATA WESTBENGAL INDIA
- 48. Two Methods toDecide a Match Are these two records a match? RHITU K KAZANGIAN RITU KUMAR B B KAZANGIAN +5 +2 A +20 128 MAIN ST 02111 12/8/62 128 MAINE RD 02110 12/8/62 / / A B D B A = BBAABDBA +3 +4 -1 +7 +9 = +49 Deterministic Decisions Tables: • Fields are compared • Letter grade assigned g g • Combined letter grades are compared to a vendor delivered file • Result: Match; Fail; Suspect Probabilistic Record Linkage: • Fields are evaluated for degree-of-match • Weight assigned: represents the “information content” by value • Weights are summed to derived a total score • Result: Statistical probability of a match 48
- 49. A Closer Lookat Probabilistic Matching C ose oo obab st c atc g RHITU K KAZANGIAN 128 MAIN RITU KUMAR KAZANGIAN 128 MAINE RD 02110 12/8/62 +5 +2 +20 +3 ST +4 02111 12/8/62 -1 +7 +9 = 49 Histogram of Weights 4000 3500 The weighted score is a p probability of a match; it y ; expresses the amount of information content for all of the fields compared 3000 # of Pairs f relative measure of the The CUTOFF is the score above which good matches are found 2500 2000 UnMatched 1500 1000 500 Matched 0 -50 49 -40 -30 -20 -10 0 10 20 30 40 50 60 49
- 50. The Value ofInformation Content Information Content is measured both at the field and at the field value level and is calculated automatically Discriminating Value represents the significance of one field versus another in contributing to a match For example a Gender Code contributes less information than a Tax-Id Number Frequency represents the significance of one value in a field over another value q y p g For example in a Last-Name Field, “SMITH” contributes less information than “ROUTZAHN” Probabilistic Matching uses the automatically generated measures of Information Content to achieve the highest match rates possible utilizing a scientifically-justifiable C h h h h h bl l f ll f bl methodology 50
- 51. Data Framework aroundthe Individual • Logins (User credentials) • Profile • Expertise • External & internal unstructured data linked to individuals • S Social activity l Big Data Individual Social Presence • • • • BLOG Comment Opinion “Like”s Community Individual Credentials • to Person • Communities • to Company: Roles, History • IBM Linkage Individual Core Personal data: • Name, Address • Phone, eMail • Behavioral Preference / permissions Transactions involving i l i the Individual • Tech Support Call • O Opportunity & Orders t it Od • Responses to Marketing Campaigns Relationships with the individual i di id l Interactions with the Individual • • • • • • Digital Phone eMail F2F Social Web traffic 51
- 52. Analytics steps Text Analytics •Analyze and extract consumer attributes from individual messages Intent Entity Integration • Integrate information about a consumer within a single social media source over ti di time Entity Resolution • Link social media profiles with customer data t d t • Link and integrate information about a consumer across multiple social media sources All I really want is the Disney Visa card from chase with the castle on it Life Events Looks like we'll be moving to New Orleans sooner than I thought. Personal Attributes I am a engineer, mom, and wife Relationships Social Profiles of Consumers Master Data on Customers In fact I'm looking forward to the new month. Both myself and the wife have our th B th lf d th if h graduation ceremonies 52
- 53. Person Information acrossDocuments Who Is James Dimon? Do these filings refer to the same person ? variability in the person’s name, lack of a key identifier supporting attributes vary depending on the context (form type) All these facts need to be linked and integrated 53 53 Signatures Biographies Insider Transactions Committee memberships
- 54. Entity & RelationshipAnalytics from Big Data Entity Views Crawl Extract / Text Analytics Entity Resolution Map/Fuse /Aggregate Entities Relationships: E ii &R l i hi Object-centric view Unstructured data sources Untrusted View Challenge Construct and maintain comprehensive profiles of entities and relationships from unstructured data sources Main Problem: Assemble an entity view, where each entity aggregates data from thousands of different documents Multiple stages of complex processing: – Information extraction • – From each unstructured d F h t t d document, extract relevant structured records t t t l t t t d d Entity resolution • Link records (possibly across documents) that are about the same real-world “entity” Entity Integration – Entity population: mapping / fusion / aggregation • 54 Collect all the facts about the same entity into one rich object with clean values and relationships to other entities
- 55. The Complete EntityView Current purchase intentions expressed by the consumer Location-based information about a consumer (where they plan to travel, events they are going to attend) Purchase history for a consumer Life events (relocation, home purchase, wedding, graduation) Related people based on social networking data Comments/complaints expressed about various products and services Customer identity information (e.g., name, location) obtained from profiles and content of posts Micro-segmentation information about individual consumers (e.g., gender, age range, profession) 360 degree 360-degree profile of a customer City State Age Range Gender Houston TX 30-39 Female San Jose CA ? Male Marital Status ? Married Number of kids Employment Status Occupation ? Employed Journalist 2 Employed Software Engineer … … … Aggregate attributes from multiple sources Filter to obtain a segmentation Analyze to obtain “Similar Populations” Adding more input data gives better predictive power 55
- 56. Attribute fusion example:Inferring location from multiple clues Metadata Name: Tracy Guida Sc ee a e @ acygu da Screen name: @tracyguida Location: Tampa Description: just a Nor-Cal gal trying to fall in love with Florida Social Media Profile Screen name : @tracyguida Location: Tampa, FL Name: Tracy Guida Disambiguation, fusion of partial information Permanent location Fusion libraries: • Confidence: metadata vs. content Messages Gotta love Florida football #hot #humid http://instagr.am/p/QOHPqhKdYt/ Check out my blog about #food in #TampaBay h k bl b f d http://www.myothercitybythebay.com Textual clues Temporary location I'm at Tracy's Seat At Micah's (Tampa, FL) http://4sq.com/SZ4yjj http //4sq com/SZ4 jj I'm at S.o.G (Tampa, Florida) http://4sq.com/UDweM5 Check-ins I'm at Eats American Grill (Tampa, FL) http://4sq.com/O1a1Jm Who's Wh ' watching the #presidential #debate tonight? hi h # id i l #d b i h? (from 27.97989014,-82.54825406) Fusion libraries: • Confidence: place mentions vs. g geo-codes • Analysis of location time-series Geo-located G l d documents 56
- 57. The Reliability (Veracity)Challenge Θ = {θ1,...,θ N } - a set of hypotheses (frame of discernment, universe of discourse) {xni } - probability, possibility, belief in hypothesis {θn} of source i {Oi } - input data (social media, enterprise information) F(x1,...,xI ) – Fusion operator {O1} Environment Environment 57 {OI } Source 1 (source belief model model, source characteristics) Source I (source belief model, source characteristics) {x1} Fusion Fusion operator operator { xI } F ( x1,..., x I )
- 58. Typical Reliability Settings Itis possible to assign a numerical degree of reliability to each source A subset of sources is reliable but we do not know which one Reliabilities of the sources can be ordered but no precise reliability values are known Reliability dependent on context too During Mumbai Mantralaya fire a few tens of tweets on this event on Twitter Same day there is a match and there are several thousand tweets “Miami Miami on Fire” 58
- 59. Strategies for UtilizingReliability Strategies explicitly utilizing reliability of sources Reliability is used to modify beliefs of each model before fusion and then use transformed beliefs (separable case) Strategies for modifying the fusion process to account for the reliability of the sources (non separable case) (non-separable case). Strategies identifying reliability of data input to fusion processes and eliminating the sources of poor reliability Combination of strategies mentioned above F(x1,...,xI ) FR (x1,...,xI ) F - i a context d is t t dependent operator, which depends on the d t t hi h d d th strategy selected and defined within the framework used for uncertainty representation R 59
- 60. Reliability Coefficients Reliability coefficientsrepresent trust in each belief model. They introduce the second level of uncertainty and represent a measure of y p the adequacy of the model used, the reality of the environment, and source characteristics Ri = Ri (Mi, γ ,Υ) - reliability of source i (reliability of source i and , ) y ( y hypothesis j : Rij) Mi - model of source i γ parameters characterizing external environment (context) Υ -parameters characterizing the internal environment of source I (tuning parameters) Relative eli bilit Rel ti e reliability : ∑iIRi =1 1 May be replaced with max Ri = 1 i 60
- 61. Bayesian Fusion In theBayesian framework the degrees of belief are represented by a priori, conditional and a posteriori conditional, probabilities. Usually, decisions are made on a posteriori probabilities P(θn | yi ), where yi i the input coming from source I, h is th i t i f I xi = P(θn | yi ) represents statistics of each source to be combined (data, outputs of classifiers). Fusion is F i i performed by the Bayesian rule, which under the f d b th B i l hi h d th condition of source independence is reduced to a product: Fn(x1,...,xI)|y =Fn(P)|yi =P(θn)∏[P(θn |yi)/P(θn)], n This fusion operator is conjunctive and assumes total reliability of the sources 61
- 62. Weighted Average If thesources are not totally reliable, several fusion rules within the framework of the probability theory have been proposed in the literature A majority of the weighted average methods are based on consensus th theory, which involves general procedures of hi h i l l d f combining single source probability distributions while decisions are based on Bayesian decision theory Fn(x1,...,xI,R1,...,RI)|yi =Fn(P,R)|yi =∑iP(θn |yi) Ri where R is reliability associated with the sources in the global membership function expressing quantitatively the goodness of each source i 62
- 63. Incorporation of Contextual Information Thismethod integrates contextual information The Th method is based on the fact that, in a given context, th d i b d th f t th t i i t t only a subset J of a set N of all sources to be combined is valid or reliable (i.e. their belief model adequately represents reality) Fn ( x1 ,..., x I , R1 ,..., R I ) | y = ∑P(θ |y1,...,yn,AJ ) P(AJ ) where P(AJ) is the probability of validity of the subset J of inputs. This probability is calculated thanks to the reliability Ri of the individual inputs 63
- 64. Biographical and Biometricfusion for Person Identification Many modern data repositories record both biographical and biometric information Motor Vehicle Licensing Authority, Passport, Identify cards etc Unique Identification number (www.uidai.gov.in) Fusing information from multiple sources bring value in Data integration: Creating single view of citizen, person, customer Identification of the person using Biometric information and biographical information Scaling person identification for large number of customer records – – Biographical data is abundant, easy to match, scales to millions of records but can be noisy and uncertain. Biometric data is noise free and gives high precision for identification but does not scale to large number of records – Both stream contain complimentary information which can be exploited by fusing together Fusion for Person Identification can be done at two levels – Decision fusion: Each matcher provides the decision which are then fused to produce the final decision. – Score fusion: Each matcher provides score which is used for producing a score for decision making. 64
- 65. Score Fusion usingBiometric and Biographical matcher Consider M matchers operating on a database containing N records which have both biographical and biometric information. For query q if all the records are equally likely for the identifier than the posterior of the score given records is given by There can b multiple biometric as well as biographical Th be lti l bi ti ll bi hi l matchers Each query q will generate N x M scores i.e. M dimensional scores for N records Genuine match score density We model the scores as being generated from a probability distribution. b bilit di t ib ti Score is fused using a joint distribution from different sources The probability distribution under reasonable assumption is the posterior distribution of scores given a query The genuine and imposter match scores are assumed to be identically distributed The posterior distribution is modeled as a Gaussian mixture model. The model is built for both genuine match distribution and imposter distribution The query is assigned an identity of n0 only if Models are learnt from training data. The algorithms is Which simplifies to 65 Imposter match score density
- 66. Results DataSets Biometrics: NIST Datasetconsisting of match scores of right and left index finger Biographical : Electoral records of citizens in an emerging economy Consists of Names and Address Total of 6000 people were associated with the biometrics and the biographical data. Here M = 4, 2: Biometric, 2: Biographical (Name & Address) Experimental Setup Accuracy for different modalities Half of the dataset was used for training the probability densities for both the imposter and genuine match score distribution was estimated The number of Gaussians components was 5 The remaining records was used for testing. Experimental Results. E i t lR lt Score is fused using a joint distribution from these four different sources The name modality has the lowest accuracy where the biometric modality has high accuracy The fused accuracy is much higher than the individual localities The accuracy increases when all the modalities are combined thus validating the usefulness of fusion 66 Identification accuracy for fusion of modalities
- 67. Social listening formonitoring the Philippine general elections 2013 • Online and offline analysis of social media messages around election debates and election chatter for ABS‐CBN TV Channel • Analysis of English and Filipino chatter to determine buzz and reaction on candidates, campaigns, parties, topics and events campaigns, parties, topics and events • Analysis of over 6 million election related Twitter and Facebook posts • Comparison with Pulse Asia Election Survey Real time and offline monitoring of social g media conversations about parties and POE, GRACE candidates Mar 13 50% 45% 40% 35% 30% 25% Positive and negative sentiments for candidates 20% 15% 10% 5% 0% Mar 08 Mar 09 Mar 10 Mar 11 Mar 12 Mar 13 Mar 14 Grace Poe released her TV ad which drew flak from viewers. This was also the time that 3 candidates (Legarda Poe Escudero) of the (Legarda, Poe, Liberal Party who were also "guest" candidates of UNA were dropped by UNA as the President forbade them to attend UNA's soirees. Escudero felt really, really bad about being dropped by UNA (led by former president Estrada). Grace (l d b f id E d ) G Poe offered to mediate between Escudero and Estrada. ZUBIRI, MIGZ (UNA) VILLAR,CYNTHIA HANEPBUHAY (NP) VILLANUEVA, BRO.EDDIE (BP) TRILLANES, ANTONIO IV (NP) SEÑERES, CHRISTIAN (DPP) POE, GRACE PENSON, RICARDO MAGSAYSAY, RAMON JR. (LP) MAGSAYSAY, MITOS (UNA) MADRIGAL, JAMBY (LP) MACEDA, MANONG ERNIE (UNA) LLASOS, MARWIL (KPTRAN) LEGARDA, LOREN (NPC) ( ) HONTIVEROS, RISA (AKBAYAN) HONASAN, GRINGO (UNA) HAGEDORN, ED FALCONE, BAL (DPP) ESCUDERO, CHIZ ENRILE, JUAN PONCE JR.(NPC) EJERCITO ESTRADA, JV (UNA) DELOS REYES,JC (KPTRAN) DAVID, LITO (KPTRAN) COJUANGCO, TINGTING (UNA) , ( ) CAYETANO, ALAN PETER (NP) CASIÑO, TEDDY BINAY, NANCY (UNA) BELGICA, GRECO (DPP) AQUINO, BENIGNO BAM (LP) ANGARA, EDGARDO (LDP) ALCANTARA, SAMSON (SJS) 0.00% 10.00% 20.00% 30.00% 40.00% 50.00% 60.00% 70.00% 80.00% 67 90.00% 100.00%
- 68. Worldwide leads: Intentto buy, relocation to India Venu Nair: Male, Atlanta, USA: Looking for good investment in Indian real estate market Kiran Singh: Female, IT g , professional, Gurgaon: Any good 2 BR in Sohna Road? 5 12 14 2 Data from Dec 3-4, 2013 68
- 69. Sample leads Name Sex Location Profession Interest where Param_star M India Media 2BHK India KiranSingh F Gurgaon, India IT 2BHK Sohna Road, Gurgaon Venu Nair M Atlanta, US Muhammad Faiz M Singapore IT Hooker India - Bangalore Real Estate Apartm Bangalore Bangalore, ent India Apartm India ent 2 and 3 Noida, India BHK 69
- 70. Crowd sensing g • The “powerof the crowd” – a lot of information in a timely manner from everywhere • People already use the social media to share public safety and law enforcement information • Gain deep situational awareness • Emergencies, call for help Enable proactive actions by augmenting traditional law enforcement methods Police Monitoring Limited coverage Analytics and fusion in nearreal-time Crowd sensors 70 Rich events & KPIs
- 71. Drinking in theOpen Come to South City 2, in evening, its a regular scene there since last 4 years, people drink in open and food is served by restaurants in their cars khandsa road per sunrise hospital se aage tekho ke pass rehari waale sharab pilaate hai, jinki wajah se waha aane jaane wale log pareshaan ho rahe hai even shaam ko to PCR ka bhi unhe darr ni hai kirpa hai, hai, karke inhe waha se hataiya Gurgaon Police I also have a complaint to register. We have an alcohal drinking menace in front of our commercial complex anand ganga comlex at complex, comlex, sohna chowk, on the main road. Police Harassment These two Constables (Davinder Singh & his Colleague) were at their worst behaviour...when they found all documents ok in the Car. I couldn't understand the reason for harrasment...opp Wrong Parking this is the main way from sadar bazar to bhuteshwar mandir. I dnt p park vehicles both the way y think y this road exist. It is the best place to p are used to park vehicles no action have been taken from years. I think HUDA or MCG is not serious abt matter. 71
- 72. Event detection andmapping 72
- 73. Conclusions Noise is anunavoidable fact of real life communication Communication meant for human consumption can be C i ti tf h ti b noisy for computers and vice versa Due to ubiquitous sensors (GPS, Accelerometer), easy of use apps (Facebook, Twitter, YouTube), and higher internet connectivity, the key characteristics of raw data is changing. This new data can be characterized by 4Vs Volume, Velocity, Variety and Veracity For example, during a Football match, some people will Tweet about Goals Penalties etc while in addition there may be other Goals, Penalties, etc. reports in news channels. The data describes the same event Fusion should create a single object representation Different sources may have different reliability and it is necessary to account for this fact to avoid decreasing in p performance of fusion results 73 Reliability and context should be taken into account during fusion
- 74. Conclusions Noise can bedefined as any kind of difference in the surface form of an electronic text from the intended, correct or original text Noise N i can b in the form of errors arising from uncertainty in be i h f f i i f i i language and communication and recognition errors 74
- 75.