This whitepaper presents a case study that taps into the textual content of financial news articles/blogs at regular intervals of time, process it, store the features extracted into a database, build the model, and finally predict stock movement for a defined period ahead. The article is passed through a data processing pipeline which consists of multiple steps like Named Entity Extraction, Sentiment Analysis and a customized algorithm.

1. PredictiveAnalyticsthrough
SentimentAnalysis

2. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Abstract
Introduction
Solution
Implementation
NamedEntityExtraction
SentimentAnalysis
Model
Conclusion
Reference
AuthorInfo
3
3
4
5
5
5
66
7
7
7
TableofContents

3. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Mediainthe21stcenturyhasbecomeverydiverseandwiththeadventofthesocialmediarevolutioninthe
lastdecade,ithasgrowninstaturetoo.Thelayersofdifferentmediacontentscoupledwiththeexpressions
inthesocialmediahasbecomeacauldronofemotions,thoughtprocessesandsentiments.Thisdatahas
becomethefocusofmanygroups/companiesasalotcanbeforeseenthroughthis.Theavailabilityofdata
andimprovementoftechnologyhaveallowedustofindanincredibleamountofvalue,whichafewyearsago
wouldhavebeendeemedimpossible.Thequestionweattemptedtosolveiswhetherweminevalueoutof
commoncommonday-to-daydataanduseitforpredictiveanalysis.Thiscasestudyisourattemptistoshowcasethe
useofdailyfinancialnewsarticlestopredictstockmarketmovements,tohighlightthedetailedMLexperi-
mentsandresultsachieved.
StockMarketpredictionshavebeengenerallybasedonstochasticmodels.ThealgorithmsusedareExponen-
tialMovingAverage,andHead&Shoulders.ArtificialNeuralNetworksandGeneticAlgorithmsarealsoused
heavily.ManyanalystsusemoretraditionaltechniquessuchasP/ERatiotoo.Allthesetechniquesusedstock
marketprices,stockvolumestradedanddividendspaidtomodelthepredictions.Howeverourattemptwas
tohighlighthowmarketandpublicsentimentcanbeharnessedfrom financialnewsarticlesandusedforpre-
dictingstockmovementswithouttheregularlyusedentities/features.
PredictiveAnalyticswith‘UnstructuredData’hasbecomeoneofthecornerstonesofBigDataAnalytics.This
usecaseshowcasespredictiveanalyticsworkflow/pipelinewithunstructureddataandhowsimplywecanput
ittogetherusingsomeopensourcetools.
ThekeychallengesofBigDataaccordingtoGartnerareInformationStrategy,DataAnalyticsandEnterprise
InformationManagement.Italsoclaims“Through2015,85%ofFortune500organizationswillbeunableto
exploitbigdataforcompetitiveadvantage”.Webelievethefactorslike‘useofrighttechniques’,‘extractionof
rightparameters’and“discoveringunknownvalues”contributeforthesame.Theendeavorwastoinvestigate
thesefactorsinourcasestudy.
TheTheideawastotapintothetextualcontentoffinancialnewsarticles/blogsatregularintervaloftime,process
it,storethefeaturesextractedintoadatabase,buildthemodel,andfinallypredictstockmovementfora
definedperiodahead.Aswepredictthemovementofthestockstotheusers,weingestedthecurrent‘stock
marketfeed’toverifyourprediction.Thefeedbackwasusedbythesystem tofinetunethemodel.The
end-to-endworkflowofthesystem canbeseenbelow.
Abstract
Introduction
PredictiveAnalyticsthroughSentimentAnalysis|3

4. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
NewsFeed
StockMarketFeed
Users
Application
Prediction
Database
ETL
Prediction
Feedback
Letustakeastepbackandunderstandtheintuition.Considertwosentences“Robertisagoodstudent”and
“Monicaisthebeststudentintheclass”.Whatallinformationwecanextractfrom thefirstsentence?Firstthe
principalentityofthesentenceis“Robert”,thegenderoftheentityis“Male”andthesentencesayssomething
positiveaboutentity.Thereremainsaquestionthough‘isitmeasurable?’From thesecondsentencewecan
easilyinterpretthat“Monica”isafemalestudentandthissentencetoosayssomethingpositiveaboutthe
entity.Thereisnoquantifiableattributeswhichcanbeassociatedwiththesesentences.Nowifwestartto
quantifyquantifythepositiveornegativemood/sentimentsaccordingtosomeweightsitcanbemeasured.Hencelets
consider‘good’=4and‘best’=5.Thenwecanlabelthetwosentencesasinthetablebelow.
Whatdidwejustdohere?WetransformedUnstructuredDataandshapeditintoStructuredData.Allthiswas
sosimple,wasn’tit?Humansaretrainedtoidentifyanddistinguishbetweenpartsof,toneofasentence,and
interpretationsoflanguages.Nowthequestioniswhetheramachinecandoitornot.Yes,itcan,withthehelp
ofTextMiningtools/technologies,thetechniquesusedareEntityExtractionandSentimentAnalysis.
PredictiveAnalyticscanbeappliedtofindanswerstotheunknown.Themovementofthestocksinthefuture
iswhatwearelookingfor.Thestocknewshasalotofunstructuredinformationaboutstocksymbols,compa-
nies,profit/loss,andlastlytheoverallsentimentaboutthecompanies/market.Thestatisticalalgorithmsused
thestructuredinformationextractedfrom thefinancialnewsarticlestobuildamodel.Thismodelisinturn
usedtopredictthestockmovementofthecompanyassearchedbytheuser.
MainElement-Robert
Gender-Male
SentimentIndex-4
MainElement-Monica
Gender-Female
SentimentIndex-5
Figure1.System Workflow
Solution
PredictiveAnalyticsthroughSentimentAnalysis|4

5. ©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
NewsarticlesrelatedtoNASDAQStockExchangewereused.WehaveusedaRSSNewsFeedtotapintothat.
ThefirststepwastoharvesttheRSSFeedforthenewscontent.Aftercleansing,thearticlesareparsedto
separatecoretext.Eacharticleisthenpassedthroughadataprocessingpipelinewhichconsistsofmultiple
stepsandsub-steps.ThemajorstepsareNamedEntityExtraction,SentimentAnalysisandourownalgorithm.
WehaveusedexternallylibrarylikeGATEandreferencedresearchworkbyFinnArupNielsenfrom the
UniversityofDenmarkforthispipeline.
GATEisusedfortheextractionoridentificationoftheseentities.GATEhasapre-designedworkflowknown
asANNIE.ANNIEisanInformationExtractionsystemandcomprisesofcoupleofstepsthatcanbeordered,
and/orremoved/added.ThisconsistsofTokenizer,Gazetter,SentenceSplitter,POSTagger,NETran-
sreducerandOrthomatcher.Thetokenizerandsentencesplitterareusedtosplitthewordsandsentences.
TheroleoftheGazetteeristoidentifyentitiesinthetextbasedonpre-definedlists.TheOrthomatcher
moduleaddsidentityrelationsbetweennamedentitiesfoundbythesemantictagger,inordertoperform
co-reference.co-reference.ThePOSTaggerisusedtotagtheentitiesinthepieceoftext.WetrainedGATEtoidentifynew
STOCKCODE/SYMBOLSbylearningfrom existingcodeswhichwasavailablewithus.
Twodifferentapproacheswereusedforcalculationofsentimentswhicharethebasicfeaturesofthemodel
tobebuilt.
TheprimaryfeaturesareEnglishKeywordsSentiment,StockKeywordsSentimentandContextualSenti-
ment.WedecidedtowritethealgorithminthelinesofresearchdonebyFinnArupNielsenfromtheUniver-
sityofDenmarktocalculatethesentiments.HeusesalistofEnglishkeywordsratedfrom +5to-5known
asAFINN.
Thekeywords,whichwillbematchedfrom theAFFINlist,wouldbe‘good’withascaleof5and‘more’with
ascaleof4.Thedistanceofeachentityinthissentencefrom thesetwokeywordswillbecalculated.Further
isakeywordfrom anentitytheeffectofthewordisdiminishedandvice-versa.Onthebasisofthisweget
theEnglishsentimentscorewhichmaybepositive,neutralornegative.
WeWefurtherresearchedandscaledalistofabout3500keywords,whicharerelatedwiththestockmarkets,
thesewereusedtocalculatethestocksentimentscoreinaccordancewiththestockmarketmood.Someof
thekeywordscannotbeattributedtoeitheroftheclassesabovearecontextualinnature;wecalculateda
contextualsentimentscoreforstockkeywordstoo.
Hereisanexampleforillustration:
“Rohitisagoodstudenthealwaysgetsmorethan90%score”nowwiththehelpofNamedEntityExtraction
wewillgetthefollowingentities.
<Name>Ram<Name>isagoodstudenthealwaysgetsmorethan<percentage>90%<percentage>score.
Implementation
NamedEntityExtraction
SentimentAnalysis
PredictiveAnalyticsthroughSentimentAnalysis|5

6. Model:
©2015,HCLTechnologies.ReproductionProhibited.ThisdocumentisprotectedunderCopyrightbytheAuthor,allrightsreserved.
Oncethefeaturesandtheirvalueswereextractedwebuildacoupleofmodelswiththestatisticalcomput-
ingmodelR.Theclassificationmodelsusedwere‘Random Forest’and‘NativeBayes’.Oncethemodelwas
availableweusedthesametopredictthemovements(UP/DOWN)ofastocksymbol.Simultaneouslywe
ingestedthe‘StockMarketFeed’forthatparticularsymbolandusedthatfeedbacktorecalibratethe
weightageofthekeywordsandhencerefinethemodel.
Afewrowsfrom Table1wereusedforfinetuningofthemodel.Theresultsachievedwereabout80%
accurateaswecanseeinthepie-chartsinFigure2.
ResultsAchieved
Down
54%
Down
24%
Up
76%
Up
46%
Up
46%
Down
26%
Down
54%
Up
46%
Actual
NativeBayes Random Forest
Error19.3%Error21.8%
Predicted Actual Predicted
Table1.Predictiontable
Figure2.Pie-charts:Actualv/sPredicted
PredictiveAnalyticsthroughSentimentAnalysis|6

7. Formoredetailscontact:ers.info@hcl.com
Followusontwitter:http://twitter.com/hclersand
Ourbloghttp://www.hcltech.com/blogs/engineering-and-rd-services
Visitourwebsite:http://www.hcltech.com/engineering-rd-services
Hello,I’m from HCL’sEngineeringandR&DServices.Weenabletechnologyledorganizationstogotomarketwithinnovativeproducts
andsolutions.Wepatnerwithourcustomersinbuildingworldclassproductsandcreatingassociatedsolutiondeliveryecosystems
to help bringmarketleadership.Wedevelop engineeringproducts,solutionsand platformsacrossAerospaceand Defense,
Automotive,ConsumerElectronics,Software,Online,IndustrialManufacturing,MedicalDevices,NetworkingandTelecom,Office
Automation,SemiconductorandServers&Storageforourcustomers.
ThiswhitepaperispublishedbyHCLEngineeringandR&DServices.
Theviewsandopinionsinthisarticleareforinformationalpurposesonlyandshouldnotbeconsideredasasubstituteforprofessional
businessadvice.TheusehereinofanytrademarksisnotanassertionofownershipofsuchtrademarksbyHCLnorintendedtoimply
anyassociationbetweenHCLandlawfulownersofsuchtrademarks.
FormoreinformationaboutHCLEngineeringandR&DServices,
Pleasevisithttp://www.hcltech.com/engineering-rd-services
Copyright@ HCCopyright@ HCLTechnologies
Allrightsreserved.
KinnarKumarSen
HCLEngineeringandR&DServices
Conclusion
Reference
AuthorInfo
http://www.gartner.com/technology/test/big-data.jsp
http://fnielsen.posterous.com/simplest-sentiment-analysis-in-python-with-af
http://r-project.org
http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010
http://nlp.standford.edu/software
Theaim ofthiscasestudywasnottopredictthestockmarketwith100%accuracybuttobringoutthefact
thattherearebusinessvalueshiddeninthedatawhichwesee,touchandexperienceeveryday,andtech-
niquesareavailablewhichcanbeputtogethertominethosevalues.Asforimprovementofthepredictionis
concerned,thereareseveralareaswhichcanbeattendedto,suchasinclusionofRealTimeSocialNetwork
feeds,refiningthealgorithmsused,andattunethem tostockmarkets,introducingthemarketsegments
featuresofthestocksymbols.
PredictiveAnalyticsthroughSentimentAnalysis|7