Try: Fail, Try: Succeed by Tim Grant

1. 11
Try: Fail, Try: Succeed
The story of how we built North’s QA team from the ground up.
QA or the Highway - Feb 2019

2. 22
- Startup based in Kitchener, Ontario
- Offices in Toronto, San Francisco
- Retail stores in Toronto and Brooklyn
- Current over 400 employees
- 75 employees 3 years ago.
- In October 2018 launched Focals
Who is North?

3. Messages
Stay in touch with the
people you care
about. Instantly read
incoming messages
and quickly respond
(or compose new
messages) with smart
replies, emojis or
voice-to-text.
Go & Uber
Search and navigate
to your favorite
locations and points
of interest with
turn-by-turn
directions, or request
an Uber directly from
your glasses.
Calendar, Weather,
& Reminders
See the time and
date, your calendar,
weather forecast, and
select notifications.
Or use Do Not
Disturb to put it all on
pause and focus on
the task at hand.
Amazon Alexa
Talk to Alexa
anywhere, with visual
responses or audio
from the speaker
embedded in Focals.
Discreetly scan the
news, set timers,
control smart home
devices and more.
Key Focals Experiences:

4. 44
- Joined North in May 2016 as QA Manager
- Grown from 1 co-op to 18 full-time & 2 co-ops
- 45% female
- Team is structured into 2 communities
- Automation Development & Wearable Test
- Testing: apps, mobile, backend, web, retail &
manufacturing
Growth of QA @ North

5. 55
About Me
2002 2008 20162001
G
raduated
from
U
niversity
ofW
aterloo,Com
p.Eng.
ElectricalEngineer:M
icrotronix
D
atacom
EE
->
SW
M
anager:N
etIntegration
->
IBM
TechnicalProjectLead:Christie
D
igital
Q
A
M
anager:N
orth

6. 66
All the things I wished I’d known before I started at
North for the first time.
- Tools
- Processes
- Testing Strategies (the dreaded Test Plan word)
- Regression
- Interviewing & Hiring
What am I here to talk about?

7. 77
So many things I “could” do:
- What tools are we using?
- Do we have any documented processes?
- Is there a test plan for what we going to test?
- Probably many others I didn’t even consider.
It’s day #1, what do I do?
So many other things to understand:
- What’s the timeline/roadmap?
- How does the product work, what does it do?
- How realistic is any of this (personal filter)?

8. 88
✓ Bug tracking
✓ Requirements Tracking
✓ Continuous Integration Server (build system)
- Test Case Management
✓ Method to deploy builds to devices.
✓ A way to capture specifications/documentation
✓ Code Repository
- Probably others I didn’t even realize
Let’s start with tools

9. 99
But what about the process?

10. 1010
- With a small team everyone is going
to be called on to help out
- Especially when you’re up against
a tight deadline.
- When the pressure is on the process
is the only thing they can rely on.
Why is process important early on?

11. 1111
- Make sure we have a process for testing.
- Make sure it is well documented and people know it.
- Make sure everyone is familiar with the product and staying
up to date.
- Use templates so people know what to fill out.
- Keep that process lightweight so it’s easy to update.
- Keep updating it as you learn and grow.
Lessons Learned

12. 1212
- When should we prepare one?
- When is it too early to write one?
- How do we decide on the level of detail and where to break it up?
- Lessons learned:
- Pay attention to the state of the program, especially what
state are you at for everyday internal use?
- Probably too early.
- If product is talking about “learning how people might use this
technology” (Alpha program)
- Probably too early.
Test Plans

13. 1313
- As we defined our MVP for release, functionality was covered as
we verified stories.
- Testers additionally focused on the NFR (security, load,
performance, compatibility, etc).
- QA lead focused on the product release strategy and how we’re
going to “regress” on the features.
Test Plans - What did work well

14. 1414
Problem: How do you regress on the feature set when:
1. Still in flux and going through high churn?
2. Automation can’t keep up given the churn?
3. You have 1 day and are doing it weekly?
What can we learn from all those key demos we did?
Our Solution: Regression Testing → Release Testing
- Build out testing scripts that help direct exploratory testing.
- Checklists at the end of the script to verify key features.
Speaking about that regression?

15. 1515
Background:
Shirley is a teacher who wears Focals all day…
Narrative:
During a history lesson a student asks a question to which she doesn’t know the
answer.
During lunch she remembers she needs to pick up milk and adds that to her
grocery list.
At the end of the day she gets a notification to remind her to stop by the
grocery store.
On the way home she gets stuck behind a train, so sends a quick message to
let her family know she will be late.
Example

16. 1616
What does Focals look like?

17. 1717

18. 1818
Hiring

19. 1919
Interview Process
Phone
Interview
(HR)
In Person
Interview
Take
Home
Challenge
Cultural
Interview +
Founder
- Programming Challenge
- Fit on the team
- Testing Mindset/Technical Skills
- 3-4 ppl 30 min each

20. 2020
- No one ever “failed” the phone interview.
- Effectiveness of the programming challenge varied.
- A lot of pressure on one person during the interview.
- Candidates mentioned they got the same question multiple times.
- Too generic as we looked to hire automation developers.
- Learned we needed to tailor the process for the candidates skills.
What challenges did we have with this?

21. 2121
What do we do today?
Phone Interview
(HR)
In Person
Interview
Take Home
Challenge
Founder
Hackerrank
Coding Challenge
Phone Interview
(HR)
In Person
Interview
Take Home
Challenge
Founder
Hackerrank
Coding Challenge
Tester Automation Dev
? based on
CV
30m Fit
30m Culture
45m Test
45m Tech
30m Fit
30m Culture
45m Design
45m Tech

22. 2222
- Should only take 4-8 hours to complete.
- The intent is for the candidate to really show off to us.
- Tailored to the role and their skill set.
- Examples:
- Testing RESTful APIs with an automated tool.
- Testing our corporate website.
- Architect a system to automate UI testing on Focals.
- Bluetooth on a previous product.
Take home challenge?

23. 2323
- Develop a hackerrank assessment for testers
- Develop stronger test questions to have the candidate work
through a scenario.
- See if we can make the phone interview more valuable to help
screen candidates.
What do we want to do next?

24. 2424
What’s one question you can’t ask a candidate,
but probably should when working on super
secret smart glasses?
A quick aside

25. 2525
The Team

26. 2626
- Had a ton of QA experience including as Director of QA.
- Concerns: Was this role was “beneath” him?
- Positives:
○ His testing experience & his leadership experience.
- 2 important parts of his application/interview:
1. He researched salaries and had a reasonable expectation.
2. He focused on his testing skills not his management skills.
The Experienced Tester

27. 2727
- Challenges:
○ I had to remember where he was coming from and provide
big picture context.
- What made him special?
○ As things popped up he could jump across areas.
○ He could react to things quickly given his strong focus/time
management skills.
○ He was a natural to train/teach others.
The Experienced Tester

28. 2828
- Since day 1 we have always had at least 1 co-op student.
- Great way to get bright, enthusiastic team members.
- A way to build up your recruitment pipeline.
Lesson #1:
- Make sure the person mentoring the co-op makes time for the
student.
- Consider the mentor to be someone other than their manager.
Lesson #2:
- When looking at schools consider from where you might hire
full-time employees.
- Where will you get older/experienced students?
Co-ops Students

29. 2929
Take advantage of their strengths
- Jr developer with little formal QA experience, BUT….
- Previously held a role called Community Support Associate
- Interview => shows a strong interest in QA @ North.
- Initially had him focus on automating a smoke test suite.
- Strengths: strong Python skills, good design practices.
- Weaknesses: struggled with the hardware and C/C++ coding.
- What to do after that was complete….. OS?
- Joined our backend project as tester.
- He became a strong advocate for quality in that team.
- Tests are 100% automated.

30. 3030
- Come up with guidance to write lightweight test cases.
- Expand the knowledge across our team on our core technology.
- Expand our automated regression to compliment release testing.
- Maintain the exploratory nature of our release testing.
- Come up with ways to measure quality in the field that resonates
with Px.
- Look to drive quality practices left into the development stream.
Challenges that lay ahead

31. 3131
Questions?
?