The document outlines a presentation on leveraging social media analytics to gain customer insights through data extraction and analysis from various platforms like Twitter. Key topics include planning analytics projects, using data services for querying tweets, processing and analyzing data, and refining search results for meaningful insights. It also provides a guide on setting up and utilizing data services blueprints for text data processing in SAP Data Services.

1. © 2014 Method360, Inc. All rights reserved.
Leverage Social Media Data to
Glean Actionable Customer
Insight
PRESENT ED BY :
James Hanck | Enterprise Information Management

2. © 2014 Method360, Inc. All rights reserved.
Method360
Method360
Method360 Presenter
JAMES HANCK
Vice President
Enterprise Information Management
Practice Lead

3. © 2014 Method360, Inc. All rights reserved.
Method360
Agenda
•Planning a Social Media Analytics Project
•Leveraging Data Services to Search for Tweets
•Process the Results Using Data Services Transforms
•Analyze the Results and Refine your Search
•Data Services Blueprints

4. © 2014 Method360, Inc. All rights reserved.
Method360
What is Social Media Analytics?
SOCIAL MEDIA ANALYTICS
•Extracting and analyzing data from social media sites, blogs, customer reviews, etc., with the possible goal being: Measuring Customer SentimentUnderstanding Brand LoyaltyIncreasing your Competitive AdvantageImproving Product Acceptance RatesDeveloping your Company’s Thought LeadershipAnd much more!
•Should be more comprehensive than monitoring or counting “likes” and “tweets”
•Includes in-depth analysis and assessment of the web presence of an organization
•Should generate greater understanding of the image and effectiveness of your organization’s web presence

5. © 2014 Method360, Inc. All rights reserved.
Method360
Questions to Ask
•Questions to ask when beginning a social media analytics project: What is the goal of the project? What data sources will you leverage? What are the data parameters? How will you present your analysis? What are your usability and access requirements?
SOCIAL MEDIA ANALYTICS

6. © 2014 Method360, Inc. All rights reserved.
Method360
Plan it Out
•Utilize Data Services to search Twitter
•Search for Tweets about your company and products
•Determine the sentiment of the returned Tweets (positive or negative)
•Load structured search data into SQL Server for further analysis
•Two data flows will be needed to accomplish these goals
Goal
Source
Parameters
FindProductReviews
Twitter
New ProductsOnly, Determine Sentiment
Tools
Requirements
Data Services & SQL Server
Administrative Level Access
SOCIAL MEDIA ANALYTICS

7. © 2014 Method360, Inc. All rights reserved.
Method360
Agenda
•Planning a Social Media Analytics Project
•Leveraging Data Services to Search for Tweets
•Process the Results Using Data Services Transforms
•Analyze the Results and Refine your Search
•Data Services Blueprints

8. © 2014 Method360, Inc. All rights reserved.
Method360
DATA SERVICES TO SEARCH TWEETS
Twitter
1.The Twitter Search Data Flow
2.Steps in the Search Data Flow
3.Twitter REST API
4.Utilizing a Configuration File for Easy Search Editing
1.User Defined Transform to Read the Configuration File
2.User Defined Transform that Creates the REST API query
5.Sample Twitter Search Results

9. © 2014 Method360, Inc. All rights reserved.
Method360
Twitter_SearchData Flow
DATA SERVICES TO SEARCH TWEETS

10. © 2014 Method360, Inc. All rights reserved.
Method360
Method360
Steps in the Search Data Flow
Read ConfigFile
Create REST Query
Query Twitter
Transform Unstructured Data
Load Database Tables
1.Extract search terms and other attributes from configuration file
2.Create query to be used by REST Twitter API
3.Query Twitter
4.Process and transform the unstructured Twitter data that is returned
5.Load tables in database:
1.Search Term Table
2.Search Information Table
3.Twitter Data Table
DATA SERVICES TO SEARCH TWEETS

11. © 2014 Method360, Inc. All rights reserved.
Method360
Twitter REST API
•Returns a set of Tweets given certain search parameters
•Queries against indices of recent or popular Tweets
•Not a full collection of all Tweets, normally includes between 6-9 days of Tweets
•Example: Return popular Tweets that contain the hashtag #Method360Query: https://api.twitter.com/1.1/search/tweets.json?q=%23Method360&result_type=Popular
*Need OAuthkeys added to query for authorized access to the API
•Operators modify the query to return certain Tweets:
oWith one particular word
oWith an exact phrase
oWith one word OR another
oWith one word BUT NOT another
oWith #hashtags
oThat are sent from certain people
oThat reference certain people (@user)
oSent before or after certain dates
oSent from certain locations
oWritten in particular languages
oAnd more…
DATA SERVICES TO SEARCH TWEETS

12. © 2014 Method360, Inc. All rights reserved.
Method360
Configuration File
•Utilize a configuration file that can be accessed within the data flow
•Make it simple and easy to change search attributes without editing the data flow:
oSearch Terms
oDate of Tweets
oLanguage
oNumber of Tweets to be Returned
oAnd more…
•A transform turns attributes into a query that can be used by the REST Twitter API
•Users can edit the configuration file and then execute the job to search for Tweets
DATA SERVICES TO SEARCH TWEETS

13. © 2014 Method360, Inc. All rights reserved.
Method360
Get_Search_Tasks
•User Defined Transform
•Created with the Python programming language
•Reads the configfile and extracts the data needed for the REST Twitter API query
•Transforms the data to Unicode, making it ready for the next transformation
DATA SERVICES TO SEARCH TWEETS

14. © 2014 Method360, Inc. All rights reserved.
Method360
Search_Twitter_Transform
•Another user defined transform
•Turns the output from previous step into the REST Twitter API
•Runs the query and collects the resulting Tweets
•Use the Data Services Python Expression Editor to edit the transform
DATA SERVICES TO SEARCH TWEETS

15. © 2014 Method360, Inc. All rights reserved.
Method360
Data Services Job Log
DATA SERVICES TO SEARCH TWEETS

16. © 2014 Method360, Inc. All rights reserved.
Method360
Twitter Data Table Results
DATA SERVICES TO SEARCH TWEETS

17. © 2014 Method360, Inc. All rights reserved.
Method360
Agenda
•Planning a Social Media Analytics Project
•Leveraging Data Services to Search for Tweets
•Process the Results Using Data Services Transforms
•Analyze the Results and Refine your Search
•Data Services Blueprints

18. © 2014 Method360, Inc. All rights reserved.
Method360
DATA SERVICES TRANSFORMS
Data Services Transforms
1.The Process Tweets Data Flow
2.Steps in the Process Data Flow
3.The Entity Extraction Transform
1.Configuring Entity Extraction
2.Entity Extraction Options and Rules
4.Parse, Merge, and Average Sentiments
5.Sample Entity Extraction Output
6.Sample Sentiment Table Output

19. © 2014 Method360, Inc. All rights reserved.
Method360
Process Tweets Data Flow
DATA SERVICES TRANSFORMS

20. © 2014 Method360, Inc. All rights reserved.
Method360
Method360
Steps in the Process Data Flow
Read Search Results
Find Recent Extractions
Entity Extraction Transform
Determine Sentiment Score
Load Database Tables
1.Extract data from database tables containing output from the search data flow
2.Determine which Tweets are from the most recent search
3.Utilizes the Entity Extraction transform to analyze the data for sentiments
4.Calculates a sentiment score for every Tweet where sentiment data was found
5.Load tables in database
1.Entity Extraction Data
2.Sentiment Data
DATA SERVICES TRANSFORMS

21. © 2014 Method360, Inc. All rights reserved.
The Entity Extraction Transform Method360
• Text data processing in SAP Data Services is
performed by the Entity Extraction transform
• Provides:
Linguistic Analysis
Natural Language Processing
Entity and Fact Extraction
• Offers support for multiple languages
• The Entity Extraction transform can be found
with the other transforms in the object library
under Text Data Processing • Drag and drop the transform to add it to a data flow
• Connect an input schema and configure
DATA SERVICES TRANSFORMS

22. © 2014 Method360, Inc. All rights reserved.
Method360
Configuring Entity Extraction
•Possible Input Fields:
oText
oText_ID
oLanguage
•Possible Output Fields:
oID
oParent_ID
oStandard_Form
oType
oSource_Form
oSource
oOffset
oLength
oParagraph_ID
oSentence_ID
oConverted_Text
oLanguage
DATA SERVICES TRANSFORMS

23. © 2014 Method360, Inc. All rights reserved.
Method360
Entity Extraction Options and Rules
Common
oRun as Separate Process
Languages
oDefault Language
oFilter by Entity Type
Processing Options
oDictionary Only
oAdvanced Parsing
oProcessing Timeout
oDocument Properties
Dictionaries
oDictionary File
oFilter by Entity Types
RulesRule FileFilter by Rule Names
DATA SERVICES TRANSFORMS

24. © 2014 Method360, Inc. All rights reserved.
Method360
Parse, Merge, and Average Sentiments
•Transforms the entity extraction output into more structured and usable form
•Parse out the different sentiment and Sentiment Types within the Tweets
•Many Tweets contain more than one different type of sentiment
•Merge and average the determined sentiments to create an average sentiment score
•Loads the newly structured data into two SQL Server tables:
1.Extraction Data (one row per sentiment)
2.Sentiment Data (average sentiment score)
DATA SERVICES TRANSFORMS

25. © 2014 Method360, Inc. All rights reserved.
Method360
Average Sentiment Transform
Entity Extraction Table:
Sentiment Data Table:
Data Services cannot automatically sense sarcasm in Tweets
The average sentiment score can help solve for false positives as in the example above
DATA SERVICES TRANSFORMS

26. © 2014 Method360, Inc. All rights reserved.
Method360
Extraction Table Output
DATA SERVICES TRANSFORMS

27. © 2014 Method360, Inc. All rights reserved.
Method360
Sentiment Table Output
DATA SERVICES TRANSFORMS

28. © 2014 Method360, Inc. All rights reserved.
Method360
Agenda
•Planning a Social Media Analytics Project
•Leveraging Data Services to Search for Tweets
•Process the Results Using Data Services Transforms
•Analyze the Results and Refine your Search
•Data Services Blueprints

29. © 2014 Method360, Inc. All rights reserved.
Method360
Analyze Results
ANALYZE AND REFINE
•Analyze the Results from Job: 1500 Results (max allowed as per configfile) Many non-company related resultsOnly 12 results relating to the iPhone productNot very useful results

30. © 2014 Method360, Inc. All rights reserved.
Method360
Edit ConfigFile to Refine Query
•Add “iPhone” to search term in the configfile
•Returns 1500 Tweets containing the terms “Apple” and “iPhone”
•129 new rows added to Sentiment table
•Easily edit the configfile for different results
ANALYZE AND REFINE

31. © 2014 Method360, Inc. All rights reserved.
Method360
Agenda
•Planning a Social Media Analytics Project
•Leveraging Data Services to Search for Tweets
•Process the Results Using Data Services Transforms
•Analyze the Results and Refine your Search
•Data Services Blueprints

32. © 2014 Method360, Inc. All rights reserved.
Method360
DATA SERVICES BLUEPRINTS
Agenda
1.Text Data Processing Blueprints
2.Download and Install the Blueprints
3.Set-up the Text Data Twitter Job

33. © 2014 Method360, Inc. All rights reserved.
Method360
DATA SERVICES BLUEPRINTS
Text Data Processing Blueprints
SAP Data Services Text Data Processing Blueprints
•Available on the SAP Community Networkhttp://scn.sap.com/docs/DOC-8820
•Contain jobs that demonstrate different ways Data Services can be used to find and process unstructured text
•Includes a Twitter job that utilizes the REST Twitter API to query recent and/or popular Tweets using user-defined search terms
•SAP documentation will help walk you through the blueprint installation and configuration

34. © 2014 Method360, Inc. All rights reserved.
Method360
Method360
DATA SERVICES BLUEPRINTS
Review and Download
Browse the available blueprints and select the one that best matches your version of Data Services
Save and Unzip
Save the .zip file to the Tutorial Files folder in your installed Data Services path and extract the files to that folder
Import the .atlFile into Designer
Using Data Services Designer, import the tdp_blueprints_english.atlfile which will create a Text Data Processing project in your object library
Configure the Jobs
The Text Data Processing project contains eight different end-to-end jobs that can be run after making some configurations
Download and Install the Blueprints
Save and Unzip
Import Project
Configure and Run Jobs
Download Blueprints

35. © 2014 Method360, Inc. All rights reserved.
Method360
Set-up the Text Data Twitter Job
DATA SERVICES BLUEPRINTS
•The TdpBlueprintEn_Twitterjob consists of two different data flows which will search Twitter for user specified terms and then summarize the data in your database
Learn what people are saying about your products, your competitors, or anything!
•The first step in configuring the Twitter job is to edit the TextDataProcessingBlueprintsdatastore to connect to your database
•Ensure that the [$$SamplesInstall] substitution parameter is set to you Data Services installation directory
•Execute the create.sqlscript in the database used for the blueprints to create the five tables needed for the Twitter job
•Copy the two Python library files into the appropriate Data Services folder
•Edit the search.cfgconfiguration file to specify search terms and authorization credentials
•Search.cfgfile located in the Tutorial FilesText Data Processing SamplesEnglishSocial folder

36. © 2014 Method360, Inc. All rights reserved.
Method360
Wrap Up
ANY QUESTIONS

37. © 2014 Method360, Inc. All rights reserved.
Method360
Stay Tuned for….
ENTERPRISE INFORMATION MANAGEMENT
JIM HANCK
415.694.8229
jhanck@method360.com
SAP Data Services book by
COMING IN WINTER CATALOG

38. © 2014 Method360, Inc. All rights reserved.
Method360
Thank You
END-TO-END SERVICES. HIGH VALUE SOLUTIONS.