Cultural compliance and quantifying behavior
•
0 likes•521 views
In my talk, Cultural Compliance and Quantifying Behavior, I discuss ways in which Natural Language Processing and Psychometric Analysis can potentially enhance the detection of bad actors
Recommended
Recommended
More Related Content
Similar to Cultural compliance and quantifying behavior
Similar to Cultural compliance and quantifying behavior (20)
Recently uploaded
Recently uploaded (20)
Cultural compliance and quantifying behavior
- 3. Identifying Bad Actors • Bad actors can result in huge losses and fines! • Can we identify them? • What would it be worth to financial institutions if we flagged a few before they happened? Harry.mendell@gmail.com 917-609-5179
- 4. How important is it to identify bad actors? Over $204 billion has been lost since 2009 • Company Total settlments Sums paid ($billions) • Bank of America 34 $77.09 • JPMorgan Chase 26 $40.12 • Citigroup 18 $18.39 • Wells Fargo 10 $10.24 • BNP Paribas 1 $8.90 • UBS 8 $6.54 • Deutsche Bank 4 $5.53 • Morgan Stanley 7 $4.78 • Barclays 7 $4.23 • Credit Suisse 4 $3.74 • Source: Keffe, Bruyette & Woods (settlements exceeding $100 million) Harry.mendell@gmail.com 917-609-5179
- 5. Can we quantify employee behavior and point to compliance issues by analyzing email and texts? • The SEC expects banks to monitor emails but has no specific requirements. • "It's not just email, it's about using key words to monitor social media as well. Flag activities that may be illegal or may represent insider trading." • A segment on CNBC, revealed that Goldman Sachs checks employee emails and texts for 180 phases • Source: http://www.cnbc.com/2016/06/15/you-wont-believe-what-gets-an-email-flagged-at-goldman- cnbc-has-the-list.html Harry.mendell@gmail.com 917-609-5179
- 6. • a sure {bet}|{thing} • adjust your account|losses|profits • against my expressed[?] wishes • answer {your}|{the} %ANY%[0,3] phone • are {not responsive}|unresponsive • bad to worse • charge in excessive amount • charged {too much}|{excessively} • close|end|terminate my %ANY% [1,5] relationship with GS|Goldman|{this firm} • Clowns {managing|running} the fund|show|portfolio|account|{my money} • concern* %ANY% [1,5] safety of my money|fund|account • cover {your}|{our} losses • didn't authorize the sale • didn't|didnt|{did not} explain to me|us • disturbs|troubles me|us • don't worry i'll take care of it • don't you f*cking understand • done|{did this} without %ANY% [1,5] calling|emailing|contacting me|us|anyone • embezzled the account • extremely|really|quite|very unhappy|disappointed Harry.mendell@gmail.com 917-609-5179 • failed to execute {our}|{my}instructions • fix the {trade}|{trades}|{commissions} • fix|adjust|change the trade*|commission* • formally|formal complain|complaint • found numerous|several errors|mistakes • give you a piece of {the}|{my} commission • how could this happen again[?] • How could you|GS|Goldman possibly[?] lose so|this|that much • I %ANY% [0,4] {losing}|{lost} patience with {you}|{GS}|{Goldman} • I am not a happy camper • I didn't {authorize}|{agree} • I expect {a|an|your}[?] {answer*|response} {today|now|asap} • I|we have lost|{run out of}|{ran out of} confidence|faith|trust|patience • I have raised %ANY% [1,5] at least %ANY%[1,3] times • I have raised %ANY% [1,5] so many times • I lost {exorbitant|enormous amounts of}|{so|too much} money • I told you %ANY% {days|weeks|months} ago • I told you %ANY%[0,1] {days|weeks|months} ago • i want the %ANY%[0,2] trade reversed • Paying fees {through|thru} the {nose|a--|butt} • phone {calls}|{call}e-mail{have}|{has} not been answered • piece of sh*t • pissed|pisses me off • poor|terrible|crappy {fund|account|portfolio}[?] results|performance • really %ANY%[0,2] pissed|{PO'd} • rebate|refund my|your loss* • rebate|refund what I lost • register that|this as a complaint • remedy the situation • report the matter to the {sec}|{nasd}|{nyse} • reverse the commissions • reverse {this}|{the} %ANY%[0,5] {trade}|{transaction} • screw*|f*ck* it up • so frustrat* • something {went}|{is really}|{will go} wrong • stock will {fly}|{soar}|{dive}|{tank} • supposed to be the top|best financial company • surprised|concerned|frustrated|angry that you didn't|{did not} contact|call|email me • take care of any fees|commissions One third of Goldman’s search phrases
- 10. Yes! We can do better • Goldman method can be expanded by extracting more phrases from a large corpus of emails from known compliance violators. • The sensitivity can be increased by using synonyms for key words used in all of the phrases. • Deep Learning combined with vector space techniques called word embedding can capture meaning! • Google has open sourced word2vec and pre-trained vectors based on the Google News dataset (about 100 billion words). • We can also analyze emails and text for emotional content. Harry.mendell@gmail.com 917-609-5179
- 11. word2vec • Use corpus to train a neural network model (NNM) maximizing the conditional probability of context given the word • Input each word to the trained NNM to get the word’s vector • But what do we mean by a word’s vector? Harry.mendell@gmail.com 917-609-5179
- 13. Word Vectors (Richard Socher CS244 Stanford 4/1/15) Co-occurrence matrix: in this example word has to be adjacent (in most applications 5 to 10 words on each side or more). I like deep learning I like NLP I enjoy flying Counts I like enjoy deep learning NLP flying . I 0 2 1 0 0 0 0 0 like 2 0 0 1 0 1 0 0 enjoy 1 0 0 0 0 0 1 0 deep 0 1 0 0 1 0 0 0 learning 0 0 0 1 0 0 0 1 NLP 0 1 1 0 0 0 0 1 flying 0 0 1 0 0 0 0 1 . 0 0 0 0 1 1 1 0 Harry.mendell@gmail.com 917-609-5179
- 14. Word Vectors (Richard Socher CS244 Stanford 4/1/15) Problems with simple co-occurrence vectors • Increase in size with vocabulary • Very high dimensional: require a lot of storage • Subsequent classification models have sparsity issues • Models are less robust • Solution: Low dimensional vectors • Idea: store “most” of the important information in a small, dense vector • Usually around 25-1000 dimensions • One solution is to take Singular Value Decomposition of the coocurrence matrix • But computational costs scale quadratically Harry.mendell@gmail.com 917-609-5179
- 15. How word2vec works • It uses a neural net to directly produce the dense low dimensional word vectors. • It does so by having the neural net train to predict the probability of finding other words that are usually in its proximity. • The word2vec training methods and algorithms have been highly optimized to make it orders of magnitude faster then the first attempts with neural nets. • Large text corpora are crucial for good performance. Harry.mendell@gmail.com 917-609-5179
- 17. Word2vec neural net (tensorflow.org) Harry.mendell@gmail.com 917-609-5179 On the other hand, for feature learning in word2vec we do not need a full probabilistic model. The CBOW and skip-gram models are instead trained using a binary classification objective logical regression to discriminate the real target words wt from k imaginary (noise) words w~, in the same context. We illustrate this for a CBOW model. For skip-gram the direction is simply inverted.
- 18. Similarity with Word2Vec example • For example, if you enter 'france', distance will display the most similar words and their distances to 'france', which should look like: • Word Cosine distance • spain 0.678515 • belgium 0.665923 • netherlands 0.652428 • italy 0.633130 • switzerland 0.622323 • luxembourg 0.610033 • portugal 0.577154 • russia 0.571507 • germany 0.563291 • catalonia 0.534176 Harry.mendell@gmail.com 917-609-5179
- 28. Just the tip of the iceberg • More complex phrases can be used • Analogies representing non-compliance can be expressed • In addition to training on news, model can be enhanced by further training on emails and text that give more targeted examples Harry.mendell@gmail.com 917-609-5179
- 33. Identify employees that have behavioral traits that could lead to compliance violations? • You can tell a lot about a person by the words they use to express themselves • There is a long history of psychological research using word analysis to determine personality traits • There are linguistic libraries that have been widely tested that analyze text and report a feature vector of behavioral traits Harry.mendell@gmail.com 917-609-5179
- 34. WordNet-Affect Examples A-Labels Examples EMOTION noun anger#1, verb fear#1 MOOD noun animosisy#1, adjective amiable#1 TRAIT noun aggressiveness#1, adjective competitive#1 COGNITIVE STATE noun confusion#2, adjective dazed#2 PHYSICAL STATE noun illness#1, adjective all in#1 HEDONIC SIGNAL noun hurt#3, noun suffering#4 EMOTION-ELICITING SITUATION noun awkwardness#3, adjective out of danger#1 EMOTIONAL RESPONSE noun cold sweat#1, verb tremble#2 BEHAVIOUR noun offense#1, adjective inhibited#1 ATTITUDE noun intolerance#1, noun defensive#1 SENSATION noun coldness#1, verb feel#3 Harry.mendell@gmail.com 917-609-5179
- 35. EmoLex • Crowdsourcing a Word-Emotion Association Lexicon Mohammad and Turney 2013 • Crowd sourcing was used to generate a large, high-quality, word– emotion and word–polarity association lexicon. • EmoLex, is an order of magnitude larger than the WordNet Affect Lexicon. Harry.mendell@gmail.com 917-609-5179
- 38. Lets see what we get from psychometric analysis of the Enron email corpus • Available at https://www.cs.cmu.edu/~./enron/ • This dataset was collected and prepared by the CALO Project (A Cognitive Assistant that Learns and Organizes). It contains data from about 150 users, mostly senior management of Enron, organized into folders. The corpus contains a total of about 0.5M messages. This data was originally made public, and posted to the web, by the Federal Energy Regulatory Commission during its investigation. Harry.mendell@gmail.com 917-609-5179
- 47. Harry.mendell@gmail.com 917-609-5179 PCA of social media posts categorized by our LIWC based model
- 48. Ending comments • NLP and Deep learning tools such as word2vec can greatly improve key word search systems presently used to analyze email and text messages for compliance. • It would be very interesting to try Psychometric analysis on a major banks email and text corpus. • Please let me know if anyone wants to try J Harry.mendell@gmail.com 917-609-5179
- 49. Thank you! • Please contact me if you have any questions. • harry.mendell@gmail.com • 917-609-5179 Harry.mendell@gmail.com 917-609-5179