1. HOW TO AUTOMATE YOUR
KEYWORD RESEARCH
Paul Shapiro – Organic Search Director at Catalyst
SearchLove Boston – May 4th, 2016

2. @fighto 2
Without Automation

3. @fighto 3
With Automation

4. @fighto
WHO AM I?
4

5. @fighto 5
Tool Data Sources Visualization
+ +

6. @fighto
WHAT YOU WILL LEARN
• The problems with the standard keyword research process
• How you can save time on keyword research with automation
• How to automate keyword research with a powerful, but easy-to-use
tool
• How APIs work and how to leverage them
• What data sources are useful for keyword research automation
• How to use data visualization to make keyword research decisions
6

7. MANUAL KEYWORD
RESEARCH IS
OFTEN BLAND &
TIME CONSUMING
7

8. @fighto 8

9. @fighto 9
Most Keyword Research Looks Like This

10. @fighto 10
That’s just a
Google Keyword
Planner Export

11. @fighto
TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH
5
8
12
21 21
6
10
18
22
28
0
5
10
15
20
25
30
Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages) Large (250-499 pages) Extra Large (>500 pages)
Hours to Complete Keyword Research by Site Size
Average (Low End) Average (High End)
11

12. @fighto 12
AIN’T NOBODY GOT TIME FOR MANUAL
KEYWORD RESEARCH!

13. @fighto
SIZE OF THE DATA SET
13
vs.

14. @fighto 14
We can do better!

15. WE CAN SAVE TIME WITH
AUTOMATION

16. @fighto
TYPICAL TIME INVESTMENT FOR KEYWORD RESEARCH
5
8
12
21 21
6
10
18
22
28
0
5
10
15
20
25
30
Micro (0-49 pages) Small (50-99 pages) Medium (100-249 pages) Large (250-499 pages) Extra Large (>500 pages)
Hours to Complete Keyword Research by Site Size
Average (Low End) Average (High End)
16

17. @fighto 17
If we do keyword research
once per month, we will save
around 24 to 45 full days
over a 5-year period using
automation.
http://xkcd.com/1205/

18. WE CAN DO BETTER THAN AN
EXCEL SPREADSHEET

19. @fighto
THAT’S MORE LIKE IT!
19
h/t @victorpan

20. @fighto
HOW TO USE FOR KEYWORD SELECTION / TARGETING
20
This top-right quadrant contains
keywords with:
• Low competition
• Good growth
Larger bubbles show higher
search volumes.
You can alternatively use
current rank on the x-axis to
signal organic market share like
a traditional growth-share
matrix.

21. @fighto 21
That’s much better!
actionability.

22. MY KEYWORD RESEARCH
AUTOMATION PROCESS

23. @fighto 23
https://www.knime.org
It starts with a tool…

24. @fighto
WHY KNIME?
• Singular tool for data collection,
manipulation, analysis, and reporting
• Easy to use and no development skills
necessary
• If you are a developer, it provides faster
prototyping
• Large library of built-in “nodes”
• Free/Open Source
• Run on Windows/Mac/Linux
• Infinitely expandable
24

25. THE BASICS OF KNIME

26. @fighto
WHAT’S A NODE?
• Nodes are prebuilt, drag and drop modules designed perform a singular task
• Nodes are strung together like a chain to accomplish larger, more complex tasks
• Nodes can be grouped together into “meta-nodes”, which can be configured in unison
26

27. @fighto
HOW DO YOU ADD NODES & HOW DO THEY CONNECT?
How do you add nodes to your “workflow”?
How do you string nodes together?
27

28. @fighto
HOW DO YOU CONFIGURE & RUN NODES?
Configuring Nodes
Running Workflows
28
OR

29. APIs 101
29

30. @fighto
EASY & APPLICABLE API: GOOGLE AUTOSUGGEST
http://suggestqueries.google.com/complete/search?output=toolbar&hl=en&q=ninja%20turtles
30
Simple API using “GET Request”
This is a variable
that changes

31. @fighto 31
Output = XML

32. A KNIME-API EXAMPLE
32

33. @fighto
STEP 1: LIST OF KEYWORDS
33

34. @fighto
STEP 2: ENCODE SPACES IN KEYWORD PHRASES
34
Space
Character

35. @fighto
STEP 3: CONCATENATE API URL + KEYWORDS
35

36. @fighto
STEP 4: DOWNLOAD API OUTPUT
36

37. @fighto
STEP 6: LOOP THROUGH EACH KEYWORD
37

38. @fighto
STEP 7: PARSE THE XML OUTPUT FORM THE API
38

39. @fighto 39
Auto-Complete
Suggestions

40. @fighto 40

41. @fighto
FYI: I’M NOT A KNIME SALESMAN
41

42. DATA INPUTS / SOURCES

43. @fighto 43
# of primary data sources to aggregate

44. @fighto 44
http://searchwilderness.com/gwmt-data-python/
Google Search Analytics – Historical MySQL
Database via Python Script

45. @fighto 45
AdWords - Search Query Report API or CSV

46. @fighto 46
SEMRush – API (if <100,000)
or Domain vs. Domain:
Competitor Keywords (Advanced Mode)
https://www.semrush.com/info/domain_vs_domain

47. @fighto 47
GrepWords API – For Google Keyword Planner
related query suggestions & search volumes
http://grepwords.com/

48. @fighto 48
Google Autosuggest API
or Emulation with Selenium

49. @fighto 49
Rank Checking/Full SERP API - GetSTAT
https://getstat.com/

50. @fighto 50
MarketMuse – Semantic Keyword
Recommendations
https://www.marketmuse.com/
Alternative: http://pshapi.ro/SemanticKWR

51. @fighto 51
Moz API + Moz Nodes
https://tech.knime.org/book/mmi-labs-nodes
https://moz.com/products/mozscape/access

52. @fighto 52
Google Trends – Unofficial API
https://github.com/GeneralMills/pytrends
Script soon to follow: http://pshapi.ro/slopetrends

53. @fighto 53
More data sources:
http://pshapi.ro/kwmetrics

54. THE DATA-TO-KNIME
PROCESS
54

55. @fighto
Filtering +
Data
Manipula-
tions
ONE BIG KEYWORD LIST
Seed Keywords
- List
- GWMT
- SEMRush Comp. KWs
- SQR Keywords
Keyword Planner
Suggestions (via
GrepWords)
Google Autocomplete
Semantic Keyword
Recommendations
(via MarketMuse)
Google Autocomplete
SEMRush
Domain vs. Domain
Keywords
Google Autocomplete
55

56. @fighto
STRING ‘EM ALL TOGETHER AND THEN…
56

57. @fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess
competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic
Competition
Search Volume
Keyword Trends
57

58. @fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess
competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic
Competition
• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume
Keyword Trends
58

59. @fighto
DATA MANIPULATIONS / CALCULATIONS
• Get top 10 results from rank checking API (i.e., GetSTAT)
• Use Moz API nodes and find average PA to assess
competiveness.
• Optionally, use SEMRush’s Keyword Difficulty API
Organic
Competition
• Get Search Volumes via SEMRush API or via GrepWords APISearch Volume
• Use 2 Years of Google Trends data to calculate slope and
determine growing/declining keywordsKeyword Trends
59

60. @fighto 60

61. @fighto 61
Reusable
5-Minutes

62. @fighto 62
You can use this data to
filter keywords and reduce
massive list... OR, you can
leverage data visualization!

63. VISUALIZING KEYWORD
RESEARCH

64. @fighto 64
We have a
few options…

65. @fighto
OPTION: EXCEL + BUBBLE CHART
65

66. @fighto
OPTION: TABLEAU
66

67. @fighto
OPTION: PYTHON + BOKEH CHARTS IN KNIME
67

68. @fighto
THEY’RE INTERACTIVE…
68

69. @fighto 69
Sample Python Code:
http://pshapi.ro/SLBokeh

70. @fighto
SO WHAT…
• Free up time spend doing mundane, repetitive work and spend more doing impactful, strategic work
• Free up staffing resources and utilize them in other ways
• Extract and analyze a larger pool of data for greater data pool
70
From this… To this…

71. @fighto
SUMMARY
• You should rethink how your currently do keyword research.
• Keyword research is often bland, time consuming, and not as actionable as it can be.
• KNIME provides an easy-to-use interface for automating your keyword research
• APIs are easier to use than they sound; Use them
• There are great data sources you can leverage via APIs/built-in KNIME nodes
• Visualize your keywords using a growth-share matrix like-graph to make keyword usage
decisions
• Emojis are awesome
71

72. @fighto
RESOURCES: DATA SOURCES > KNIME
72
Data Source KNIME Node(s)
Google Search Analytics Database Table Connector/Reader
AdWords Search Query Report XLS Reader
SEMRush Domain vs. Domain (or API if less
than 100,000 keyword)
XLS/CSV Reader or GET Resource & Read
REST Representation
Google Keyword Planner/GrepWords GET Resource & Read REST Representation
Google Autosuggest
GET Resource & Read REST Representation
(or Selenium nodes)
Rank Checking (via GetSTAT) GET Resource & Read REST Representation
Semantic Keywords/MarketMuse POST Resource
MOZ Moz API Connector/URL Metrics nodes
Google Trends TBD – Either CSV or Python Script nodes

73. @fighto
ADDITIONAL RESOURCES
• KNIME Version to install: “KNIME Analytics Platform + all free extensions”
• To install KNIME Add-ons, Choose Help->Install New Software
• Moz nodes
• Should come with Nightly Builds. If not can add http://update.knime.org/community-
contributions/3.1 as a repository.
• Add-on Selenium nodes
• http://seleniumnodes.com/
• Search Engine Land article addressing additional data inputs: http://pshapi.ro/kwmetrics
• Information about Growth-share matrices:
https://en.wikipedia.org/wiki/Growth%E2%80%93share_matrix
• Sanple Python + Bokeh script for an interactive keyword research visualization in KNIME:
http://pshapi.ro/SLBokeh
73

74. @fighto
Questions?
74

75. THANK YOU!
@fighto
http://searchwilderness.com
http://catalystdigital.com