[BrightonSEO April 2019] This talk shares the detailed methodology used to restructure a job aggregator site, which, like many large websites, had huge problems with indexability, and the rules used to direct robot crawl, taking the site from over 2.5 million URLs to just 20,000. It covers crawling and indexing issues and dive into the case study aided by flow charts to explain the full approach used to restructure the site and how to implement it.

1. slideshare.net/areejabuali
RESTRUCTURING WEBSITES TO
IMPROVE INDEXABILITY
AREEJ ABUALI // VERVE SEARCH //
@areej_abuali

2. slideshare.net/areejabuali

3. I’m here to talk to you about a framework
that I came up with to fix a client’s
indexability challenge

4. But more importantly…
(And I only realised this after I’d finished preparing this talk)

5. I’m also here to talk to you about how
“Technical Problems are People Problems”

6. I first met this client back in 2017

7. They’re a job aggregator site

8. My CEO introduced me, saying:
“Areej is the bestTech SEO you’ll find.”

9. Needless to say, the pressure was on!

10. They wanted to work with us because
we’re known to produce cool
creative campaigns

11. And we’re good at outreach
and link building

12. But we said that from some quick analysis,
it was too early for that…

13. “Your website is struggling”

14. Here’s where you are…

15. 89% decrease in organic visibility

16. Organic Traffic
2017 X
2016 Y
Change -45.6%

17. You barely rank…

18. And your site is so massive;
we struggled to crawl it

19. On the other hand, here’s where
your competitors are…

22. To make this work, we need to fix the
fundamentals first

23. We need to fix your tech

24. So, let’s spend 6 months purely on tech

25. Once that’s sorted, then we can work
on creative campaigns together

27. This talk is about my 18-month
relationship with this client

28. It’s about what worked, what didn’t work
and what I would have done differently

29. The Initial Findings

30. Links Tech Content
I started working on a comprehensive audit

31. I ended up with a 70-page document

32. There was a total of 50 recommendations

33. Some of the main things included were…

34. 72% of backlinks came from only
3 referring domains

35. Their on-page content was full of
duplication and missing the basics

36. There were NO sitemaps

37. Canonical tags were not set up correctly

38. And their internal linking structure
was a nightmare

39. Every recommendation was outlined using a
traffic light system

40. We had a half-day audit handover meeting
where I walked them through all
of our recommendations

41. Everyone was in good spirits

42. Yet I couldn’t help but feel it
was not enough…

43. Something was missing

44. Everything I recommended up till
now was solid

45. But I had a gut feeling that due to the
nature of their site…

46. These recommendations wouldn’t
quite cut it

47. I had to go back to the drawing board

48. The Supplementary Findings

49. Also Known As…

50. The Findings I Should’ve Found The First Time
Round But Didn’t So I’m Choosing To Call It
Supplementary Findings To Sound
Like An Expert.

51. They’re a job aggregator site – in essence, they’re a
job search engine

52. That means that every single search
conducted could, potentially, be
crawled and/or indexed
if it wasn’t built right

53. That’s equivalent to an infinite number
of potential searches!

54. I needed to go back to basics

55. In my first six months in SEO,
I used the terms ‘Crawling’ and ‘Indexing’
interchangeably

56. Until someone on Twitter called me
out on it, and said…

57. “You do realise they’re not the same thing,
right?”

59. So let’s make sure we’re all on
the same page

60. Crawling (V)
The process of gathering information
from billions of webpages through
their link structure.

61. Indexing (V)
To add crawled webpages to
the search engine index.

62. Which part was their site
not getting right?

63. I knew that it was impossible to crawl

64. The one time I tried to fully crawl the site,
it returned over 2.5 million URLs

65. I could only crawl it by excluding
massive sections

67. And if their pages couldn’t be crawled,
then they would never be
indexed properly…

68. And they won’t rank

69. So there were three problems that
needed to be fixed

70. Crawling Indexing Ranking

71. It was apparent that there were no rules in
place to help direct robots

72. This can create a potentially-unlimited
number of URLs

73. Google was wasting crawl budget by
crawling duplicate thin pages and
attempting to index them

74. The ‘Aha!’ Moment

75. It was apparent that there were no rules in
place to help direct robots

76. I needed to create a customised framework
that instructs bots on what to do and
what not to do

77. The job aggregator industry seemed to be
doing one of two things

78. Limiting indexable pages
à miss out on ranking opportunity

79. Limiting indexable pages
à miss out on ranking opportunity
Not limiting indexable pages
à wind up with reduced link equity

80. My framework was going to do neither

81. Instead it would use search volume data
to determine which pages are
valuable to search engines

82. So how exactly would this work?

83. It starts off by passing
the search query
through a keyword
combination script

84. This script outputs different combinations
for the search conducted

85. It does that by changing the order of
keywords to see all possible combinations

86. Digital Marketing Manager London

87. Digital Marketing Manager London
London Digital Marketing Manager
Digital Marketing London Manager

88. 4! = 24 combinations

89. These combinations will increase based
on the search query and filters applied

90. Even though Google will regard most
of these combinations as the same…

91. The script will help avoid duplicating
pages that have different versions
of the same thing

92. It then searches the
database to see
if this job is available

93. If (Job # Available)
Load a page stating so
and that is no-index
and no-crawl

94. If (Job = Available)
Search the keyword
database for all
keyword combinations
from the script

95. Fetch search volume
data for these
keyword combinations

96. For search volume data, we recommended
using keywordtool.io API

97. If (SearchVolume > 50)
A page is then created
using the highest SV
keyword combination
that is both indexed
and crawled

98. If (SearchVolume < 50)
Load page for users
but no-index and
no-crawl it

99. Your search volume cut-off can be updated
at anytime and is based on what makes
sense for your industry

101. There is always the possibility of
errors occurring

102. What if there’s a tie-breaker?
(Several keyword combinations having the
same search volume)

103. Create an indexed crawlable page based
on the keyword used in the
user search query

104. What if the API was down?
(And you’re unable to generate search volume data)

105. Load a no-index no-crawl page for
usability and don’t store the
query in the database

106. But I still knew that this was not enough…

107. What about their internal linking structure?

108. This was the status of their header

109. This is what we recommended

110. We also provided internal linking
recommendations for their footer and
job advertisement pages

111. And an exact breakdown of their filter
system on job search pages

112. As for their sitemaps – they
had none

113. We recommended creating and
splitting them up

114. Blog Sitemap
Job Advertisements Sitemap
Job Results Sitemap
Ancillary Sitemap

115. The final step was to help them sort out
their content

116. Even if they were only indexing high
search volume pages; their
content was very thin

117. Their chances to rank would
still be minimal

118. They had the same H1,Title Tag & Meta
Description for every filtered
indexed page

119. This was creating thin pages and partially
duplicated content

120. Most competitors automatically generate
optimised meta tags

121. They needed to do the same for
indexable filter pages

122. They also needed to create pages targeting
specific category search terms

123. Other than a company page and a
handful of blog posts, there were
no core content pages

124. So we performed in-depth keyword research
and opportunity analysis to see what
content generates traffic

125. And we provided a content audit
and strategy to go with it

126. My ‘Aha!’ Moment felt complete

127. I was happy, I was pleased, I was ecstatic!

128. This was the piece of the puzzle
that was missing

129. Now all that was left was for
the client to implement it!

130. The Implementation Phase

131. Four months later, the client confirmed that
they implemented everything

133. By that point, we had started working on
creative campaigns and outreach

134. The first thing that caught my attention
was their site went from over
2.5M crawled pages to 20K

135. Which initially seemed like good news…

136. Until I realised that their traffic had
declined…

137. Remember this?
Organic Traffic
2017 X
2016 Y
Change -45.6%

138. Organic Traffic
2018 X
2017 Y
2016 Z
Change -85.7%

140. The Mini Audit Findings

141. Also Known As…

142. The Findings I’m Rushing To Find In A Panic
To Prove That They Haven’t Implemented
My Recommendations Accurately,
Hence I Am Still An Expert.

143. I went through the original list of
50 recommended actions

144. At that point, there were 29 that had
not been implemented

145. I also discovered ten new issues that were
affecting their indexability

146. Google was choosing to only index
20% of the submitted sitemap

147. Crawling is one of the most expensive
things that Google do

148. Crawl budget is precious – it should
not be wasted

149. Googlebot will choose to visit their site less
if their indexability is not in check

150. The client had added canonical tags and
felt that was enough

151. They’re relying on Googlebot to:
1) Crawl the pages

152. They’re relying on Googlebot to:
1) Crawl the pages
2) Find the canonical tags

153. They’re relying on Googlebot to:
1) Crawl the pages
2) Find the canonical tags
3) Then choose to ignore them

154. Canonical tags are simply hints for bots

155. Google might decide to ignore the
tags and pick other pages to index

156. Canonical tags do not preserve
crawl budget

157. Almost 80K pages had been indexed
despite not being submitted
via the sitemap

158. And some pages were included in the
sitemap but not indexed

159. /job/finance-manager-liverpool

160. /job/finance-manager-liverpool?index=
57271b7c~4&utm_campaign=job-
detail&utm_source=search-
result&utm_medium=website

161. Because this page was the one
discovered via internal links

162. Over 5K similar pages with parameters
were getting indexed

163. Your main goal is to maximise crawl budget

164. You cannot use canonical tags as a
sticking plaster to fix that

165. This implementation was still incomplete

166. I had to change my way of conveying
this message

167. I put a stop to the endless stream of emails
and scheduled a face-to-face meeting

168. We reviewed each and every single
remaining task and discussed them in detail

169. We re-prioritised tasks and put
estimated completion dates

170. It was not an easy meeting but
it felt productive

171. Where are we now?

172. On a personal level, I discovered that I
suffer from Imposter Syndrome

173. This was my constant state of mind

174. I was working closely with the CTO
throughout this project

175. I felt he didn’t trust me or my knowledge

176. No matter how much I tried to assure him that
he’s in safe hands

177. The fact of the matter is,
this site is his baby

178. And I was constantly attacking his baby

179. So, what would I have done differently?

180. If I could go back in time, I would realise
what the actual problem was

181. All technical problems are people problems

182. The SEO recommendations were solid

183. Getting it implemented was the hard part

184. As a tech SEO, the most you can do
is to influence priorities

185. You have no control

186. In this instance, I didn’t manage to
persuade him to implement
the recommendations

187. I also learned that the way I’ve been
doing SEO audits is plain wrong

188. I always focused on delivering a set
of comprehensive actions

189. Instead, maybe I should just deliver
a SINGLE recommendation

190. And once that’s implemented…

191. Then, and only then, will I recommend
another

192. And maybe I shouldn’t recommend
Nice-To-Do’s…

193. Until there are only Nice-To-Do’s
left to do

194. Because they are simply a distraction from
the main problem

195. Over the past year, we created
6 creative campaigns for them

196. These campaigns generated 261 links and
had 1.6M estimated views

197. We got coverage on all of these sites and more…

198. Yet it barely had an impact…

199. On a daily basis, I was being asked:
“Why aren’t our rankings improving?”

200. On a daily basis, I was constantly saying…

201. “The core reason your rankings aren’t
improving is due to incomplete
tech actions…”

202. “Let’s work together to get these
implemented ASAP as they will
directly impact your traffic and rankings.”

203. This talk doesn’t have a happy ending…

204. It isn’t a successful case study…

205. It isn’t a successful case study…
YET

206. There’s no upward visibility graph and
page one rankings for me to show off…

207. There’s no upward visibility graph and
page one rankings for me to show off…
YET

208. This talk is about real life…

209. It’s about a long-overdue project where I
learned a lot about working
alongside CTOs

210. It’s about how I created a framework that
fixes indexability issues that I’m proud of
and I know in my gut *works*

211. So I wanted to share it with you

212. Because I can see this applied across
many sites in plenty of industries

213. And I’d love forYOU to implement it

214. So I’m going to share my full
methodology with you

215. bit.ly/brightonseo-areej

216. And just remember…

217. Getting the basics right is so fundamental

218. If you can do nothing else,
just do the tech.

219. Areej AbuAli
Head of SEO |Verve Search
Slides: slideshare.net/areejabuali
Questions: areej@vervesearch.com
Tweet things @areej_abuali