Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cist2014 slides

216 views

Published on

This is my presentation for CIST 2014 at Informs in San Francisco.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Cist2014 slides

  1. 1. MODELING CUSTOMER SATISFACTION FROM UNSTRUCTURED DATA USING A BAYESIAN APPROACH M o h s e n F a r h a d l o o , R a y m o n d P a t t e r s o n a n d E r i k R o l l a n d U n i v e r s i t y o f C a l i f o r n i a , M e r c e d a n d A l b e r t a S c h o o l o f B u s i n e s s . 1 mfarhadloo@ucmerced.edu rayp@ualberta.ca erolland@ucmerced.edu
  2. 2. AGENDA  Introduction  Customer satisfaction study  Challenges  Our method  Experimental results  Future work  Questions 2
  3. 3. INTRODUCTION  Sentiment analysis/opinion mining  Computational and automatic study of people’s opinions expressed in written language.  Focuses on subjective part of text  identify opinionated information rather than retrieval of factual information.  Sentiment analysis brings together various fields of research: text mining, Natural Language Processing, Data mining.  Ubiquitous Internet is a very good source of people’s opinions and has triggered research in this field. 3
  4. 4. WHY OPINIONS ARE IMPORTANT?  Opinions are key influencers of our behaviors.  Whenever we need to make a decision we often seek out others’ opinions.  True for both individuals and organizations.  It is simply the “human nature”.  We want to express our opinions.  We also want to hear others’ opinions. 4
  5. 5. CUSTOMER SATISFACTION STUDY  Traditional questionnaires.  Expensive to run.  May not be available (California law prohibits collection of CSP- Cust Sat data).  Analyze the publically available free-form text reviews. 5
  6. 6. DEFINITIONS  Contributor: Contributor is the person or organization who is expressing his/her/its opinion in written language or text.  Object: An object is an entity which can be a product, service, person, event, organization, or topic.  Review: Review is a contributor-generated text that contains the opinions of the contributor about some aspects of the object.  Aspect: Aspect is the component, attribute or feature of the object that contributor has commented on.  Opinion: An opinion on an aspect is a positive, neutral or negative view, attitude, emotion or appraisal on that aspect from a contributor. 6
  7. 7. CHALLENGES  Dealing with unstructured data. Transforming the unstructured data into semi-structured. Using a framework for aspect-level sentiment analysis [FarhadlooRolland13].  Discovering the relative importance of aspects of each object from the contributors perspective. Modeling the overall satisfaction in a Bayesian framework. 7
  8. 8. 1. TRANSFORMING THE UNSTRUCTURED DATA INTO SEMI-STRUCTURED DATA A S P E C T I D E N T I F I C A T I O N S E N T I M E N T I D E N T I F I C A T I O N 8 For each contributor’s review: Count the frequency of positive, neutral and negative sentiments of each aspect. (K aspects  3K-dimensional vector)
  9. 9. 2. PROBABILISTIC MODELING OF OVERALL SATISFACTION  Our probabilistic approach addresses the following issues: Generation of a single rating for each aspect using positive, neutral and negative sentiment counts. Discovering the relative importance of each aspect from the contributor’s perspective. 9
  10. 10. 2. PROBABILISTIC MODELING OF OVERALL SATISFACTION  Assumptions  The aspect rating depends on how many times that aspect has been associated with positive, neutral and negative sentiments in a review.  An overall rating assigned to each review.  The overall rating is a weighted sum of all individual aspect ratings.  The weights in this linear combination reflect the relative importance of the aspect. 10
  11. 11. PROBABILISTIC MODEL  For each review d  adij : 3K-dim vector containing the normalized positive, neutral, negative frequency counts associated with each aspect.  rd: the overall rating.  wdi: weight of the ith aspect in document d.  αj: combination coefficients for combining the sentimental frequencies. 11  To take into account any other factors and uncertainty  Different contributors may have different preferences over aspect.
  12. 12. HOW TO USE THE MODEL?  Given the parameters of the model θ= {μ, Σ, δ2, α} .  Calculate the aspect rating of each individual aspect 12  Find the most probable linear combination weights for each review  The MAP estimator for the importance weights is Subject to
  13. 13. MODEL PARAMETER ESTIMATION  There are 4 sets of parameters to be estimated (θ):  μ: mean vector of the prior distribution over wdi.  Σ: covariance matrix of the prior distribution over wdi.  δ2: variance of the normal distribution of rd.  α: mixing coefficients of sentimental frequency counts.  We follow an EM-style algorithm to estimate θ 13  EM-style algorithm: initialize from θ0 and alternate between E- and M- steps:  E-step: Using the current parameters estimate, calculate the hidden variables.  For each review d  Calculate aspects’ rating ad  Calculate wd  M-step: Update the parameters using the calculations done in E-step.
  14. 14. PARAMETER UPDATES 14 Get the update for δ2 by maximizing the probability defined for rd
  15. 15. EXPERIMENTAL RESULTS  Data  Reviews from TripAdvisor.com.  Reviews of 30 state parks: 1,791 reviews with 13,466 sentences .  Labels have been provided manually using a 3-person Delphi method.  Test set comprised of 21 independently managed state parks, containing 448 reviews with 3,289 sentences.  Discard terms that occur fewer than 10 times.  Size of BOW wordlist 1,296 and size of BON wordlist 718. 15
  16. 16. EXPERIMENTAL RESULTS  Aspect identification  Using the BON representation of the sentences in the training set and the aspect identification subsystem, the aspects that the users have expressed their opinions about were identified as ``beach", ``camp", ``hike", ``history", ``nature", ``park", ``ranger", ``road", ``shop", ``shower", ``trail", ``tour" and ``view".  Train the Bayesian model using above K =13 aspects. 16 ML estimate of mixing coefficients.
  17. 17. EXPERIMENTAL RESULTS  Important aspects from contributors' perspectives. Apply the training set aspects and MLE mixing coefficients (α) to the test set to obtain a frequency weighted value for each aspect of a particular California State Park.  Important test set aspects are those whose aggregated frequency weighted values exceed expectations established in the training set. 17
  18. 18. EXPERIMENTAL RESULTS  Aspect importance using ground truth sentiments. Park# Beach Camp Hike History Nature Park Ranger Road Shop Shower Trail Tour View 1 0.06854 0.16711 0.07501 0.06601 0.06523 0.05703 0.10531 0.05736 0.07483 0.04984 0.05287 0.05851 0.10235 2 0.16780 0 0.15154 0 0 0.11721 0.06755 0 0.38667 0.01146 0 0 0.09776 3 0.04964 0.07117 0.09431 0.09908 0.07391 0.06426 0.08473 0.06299 0.0762 0.05377 0.10709 0.06816 0.09469 4 0.02975 0.03076 0.06617 0.24449 0.1345 0.05226 0.0827 0.02876 0.051 0.04814 0.12603 0.02502 0.08044 5 0.01272 0.00784 0.17289 0.09766 0.00754 0.13984 0.06573 0.00926 0.04382 0.07618 0.00836 0.27037 0.08782 6 0.02019 0.0369 0.01776 0.00203 0.12903 0.02451 0.08 0.0144 0.01200 0.04833 0.297 0.23142 0.0865 7 0.01513 0.01323 0.07175 0.24644 0.16251 0.01423 0.059 0.1735 0.02701 0.06170 0.00866 0.0495 0.09739 8 0.0721 0.25708 0.04696 0.05469 0.06721 0.04701 0.12725 0.04112 0.03512 0.04987 0.05659 0.03950 0.10548 9 0.0352 0.03825 0.08665 0.25090 0.11918 0.06350 0.08019 0.04376 0.06830 0.05053 0.03938 0.03962 0.08457 10 0.05266 0.07661 0.09505 0.07665 0.06857 0.07030 0.08351 0.04956 0.08856 0.05202 0.05456 0.13584 0.09610 11 0.0518 0.10176 0.08040 0.23005 0.10207 0.07029 0.09735 0.01514 0.07755 0.04612 0.017 0.02374 0.08675 12 0.12267 0 0 0 0.1644 0.00648 0.0478 0 0.28654 0 0 0.273 0.09911 13 0.09258 0.37424 0.01126 0.02362 0.06832 0.03172 0.15434 0.02607 0.02488 0.04327 0.01838 0.01873 0.11259 14 0.03924 0.01871 0.06006 0.32673 0.16236 0.05012 0.08021 0.01764 0.09201 0.03801 0.01909 0.01838 0.07745 15 0.05389 0.0816 0.04387 0.03567 0.12593 0.03655 0.06888 0.03458 0.12432 0.03562 0.02943 0.28867 0.09445 16 0.07868 0.17650 0.07031 0.05894 0.06548 0.05531 0.10841 0.05074 0.09509 0.04485 0.04238 0.05032 0.10299 17 0.08006 0.19414 0.04314 0.03607 0.08313 0.04334 0.11181 0.03162 0.09282 0.03917 0.03027 0.11125 0.10314 18 0.06538 0.05459 0.10385 0.04463 0.06359 0.05515 0.07721 0.10724 0.11741 0.05122 0.11195 0.04605 0.10174 19 0.06667 0.18113 0.07811 0.09793 0.06325 0.06508 0.11012 0.04160 0.06648 0.05074 0.03077 0.04785 0.10027 20 0.05997 0.22014 0.03022 0.10810 0.10394 0.03826 0.11860 0.02781 0.03174 0.0460 0.02747 0.08797 0.09977 21 0.06356 0.14984 0.04899 0.07503 0.09796 0.03996 0.10063 0.05112 0.07422 0.04428 0.04894 0.10537 0.10012 18
  19. 19. EXPERIMENTAL RESULTS  Aspect importance using predicted sentiments. 19 p a r k # Beach Camp Hike History Nature Park Ranger Road Shop Shower Trail Tour View 1 0.06647 0.11719 0.0974 5 0.05638 0.05636 0.0604 0 0.09176 0.0889 4 0.09225 0.0532 9 0.069 95 0.0461 9 0.103 36 2 0.16911 0 0.1493 4 0 0 0.1232 1 0.06396 0 0.38617 0.0114 3 0 0 0.096 76 3 0.06661 0.05977 0.1232 4 0.08691 0.04596 0.0831 1 0.08189 0.0632 6 0.12716 0.0511 1 0.059 54 0.0547 6 0.096 69 4 0.03660 0.03130 0.0745 9 0.17366 0.11148 0.0580 1 0.08147 0.0293 8 0.06549 0.0485 7 0.145 06 0.0605 0 0.083 89 5 0.02551 0.01053 0.1653 2 0.09011 0.01444 0.1286 5 0.06667 0.0234 1 0.06987 0.0703 3 0.011 60 0.2334 8 0.090 08 6 0.02455 0.03885 0.0242 9 0.00671 0.11728 0.0299 1 0.08564 0.0063 0 0.01322 0.0482 2 0.363 42 0.1575 8 0.084 01 7 0.03781 0.02803 0.0598 0 0.13169 0.14107 0.0205 5 0.06435 0.1244 2 0.07648 0.0498 6 0.025 43 0.1416 4 0.098 87 8 0.07117 0.24194 0.0509 5 0.03417 0.06393 0.0501 9 0.12192 0.0396 1 0.04392 0.0498 3 0.040 21 0.0861 9 0.105 98 9 0.03137 0.02964 0.0816 2 0.29391 0.13363 0.0624 6 0.07989 0.0338 7 0.06565 0.0488 0 0.028 89 0.0292 5 0.081 03 1 0 0.05198 0.08459 0.0913 4 0.07080 0.06721 0.0748 0 0.08592 0.0293 6 0.08654 0.0509 4 0.041 43 0.1701 9 0.094 89 1 1 0.03269 0.05527 0.0905 4 0.26035 0.11104 0.0730 8 0.08500 0.0264 9 0.05663 0.0525 8 0.023 94 0.0489 5 0.083 44 1 2 0 0 0 0 0.17179 0.0245 3 0.03663 0 0.00915 0.0519 9 0 0.6124 3 0.093 47 1 3 0.09336 0.32923 0.0255 8 0.03212 0.06704 0.0370 3 0.14412 0.0327 4 0.05035 0.0421 1 0.017 93 0.0176 3 0.110 77 1 4 0 0 0 0.52977 0.26704 0.0147 3 0.07620 0 0.01310 0.0356 6 0 0 0.063 51 5 0.05398 0.03651 0.0468 9 0.04147 0.12133 0.0392 6 0.07152 0.0337 5 0.11929 0.0368 9 0.034 08 0.2706 4 0.094 40 1 6 0.06300 0.15706 0.0711 0 0.07061 0.07117 0.0553 6 0.10379 0.0526 7 0.06564 0.0503 7 0.072 95 0.0658 7 0.100 40 1 7 0.07485 0.15336 0.0626 5 0.06409 0.07930 0.0512 3 0.10311 0.0432 0 0.09980 0.0422 0 0.042 53 0.0830 7 0.100 61 1 8 0.04069 0.03634 0.1344 6 0.03462 0.05277 0.0463 5 0.06043 0.2139 0 0.06193 0.0735 7 0.095 69 0.0366 9 0.112 55 1 9 0.08601 0.03928 0.1099 9 0.05068 0.05486 0.0734 0 0.07912 0.0533 6 0.18299 0.0381 5 0.073 63 0.0619 4 0.096 58 2 0 0.06617 0.20466 0.0440 4 0.05372 0.08443 0.0430 4 0.11314 0.0406 7 0.05269 0.0467 6 0.040 62 0.1070 2 0.103 03 2 1 0.06513 0.11917 0.0694 2 0.09824 0.08981 0.0492 9 0.09438 0.0605 6 0.09318 0.0452 6 0.046 66 0.0701 9 0.098 70
  20. 20. EXPERIMENTAL RESULTS  Quantify the error rate for identifying the significant aspects.  Performed a diagnostic test.  Bolded aspects as positive and not-bolded aspects as negative (above tables). 20 Diagnostic test results. If sentiment analysis is used, we are 88.3% accurate with respect to identifying the most important aspects contributing to overall customer satisfaction, compared to when we evaluate every sentence by hand.
  21. 21. FUTURE WORK  Do the identified aspects explain overall customer satisfaction ratings?  Use ordinal logistic regression to estimate R2 values 21 R2 values from ordinal logistic regression on significant aspects. R2 values from ordinal logistic regression on all aspects.
  22. 22. DISCUSSION  Managers can use our methodology to assess the importance of aspects that drive the overall customer satisfaction.  Using our methodology managers are able to discover the important aspects for each object, and may then efficiently reallocate resources to improve overall customer satisfaction.  Researchers in marketing may extend our methodology to have a recommendation system that recommends products/services that have positive reviews.  Researchers can evaluate substantially larger amount of data with this method than with traditional surveys, and the cost to do so would be much less. 22
  23. 23. 23
  24. 24. QUESTIONS Thank you 24

×