Machine Learning in Practice, presented at CTO Summit Chicago 2019. This is a set of lessons and high level info on what we've learnt over the years at Rainforest QA.

1. Machine Learning in Practice

2. Rainforest

3. QA as Infrastructure

4. Powered by:
60k+ humans

5. Powered by:
60k+ humans
AI

6. TLDR:
modern, fast QA

7. Today:
Our use
Things we’ve learnt
When to use / not
Getting started

8. AI second

9. (data from 2016, didn’t have time to repull)

10. Tester Quality

11. Tester Quality
● Aim - make sure testers are doing the right thing.
● Journey:
● Pure code rules
● anti-fraud
● Rule based system
● Anti-tiredness / laziness
● Classifiers for various actions
● Clicks
● Scroll
● Drag
● Typing
● etc

12. Automation

13. Automate…automation
Watching what testers do, and repeating.
Assistive “things” to a rule-based system:
● Tried: visual QA, image recognition, OCR, segmentation for actions
● Using: OCR, image recognition
● New: image rec + segmentation for objects, WISP

14. Automate…automation
Training data is hard
● Don’t “know” what the tests are doing. NLP = hard
● WISP - standardizing language

17. Things we’ve learnt
(usually the hard way)

18. https://hackernoon.com/the-ai-hierarchy-of-needs-18f111fcc007

19. Team matters:
1 x Data scientist

20. Team matters:
1 --> 3 x Data scientist

21. Team matters:
3 x Data scientist
1 x Production Engineer

22. Team matters:
6 x Data scientist
2 x Production Engineer
1 x Data Engineer

23. High level SLDC for AI
1. Figure out problem
2. Get data
3. Get / make training data
4. Build prototype model
5. Train model (can take non-trivial time)
6. Test using training data
7. Go to #1 (bad data) or #4 (hope-of-good-data + “bad model”)
8. Deploy
9. Test in production
10. Iterate (aka, go to #1 or #4)

25. mechanical turk?

28. Amazon mechanical turk

29. mturk
• Super early AWS service; public since 2005, invented < 2001
24 x 7, on-demand, programmatic interface to do Human
Intelligence Tasks
• “Automate” the un-automatable

30. mturk
Pay (lots of) humans to do (lots of) things. Classic things:
• Extract data from receipts
• Identify things in photos
• Search for data for you (find the phone number of XYZ
restaurant)
• Transcribe audio
• Data science - ground truths for ml / ai

31. mturk
◦ It’s kinda hard to use right
◦ Single-Purpose APIs make this easier

32. Testing?

33. What happens when your data
changes?

34. 🔥

35. What happens ppl use them
unexpectedly?

37. What happens when they fail?

40. What happens when ppl mess with
you?

42. GDPR 🔥

43. Tooling

44. Tooling / Ops
● How do you debug?
● How do you deploy?
● How do you monitor?

45. Ethics / Bias

46. –Automated Inference on Criminality using Face Images
Xiaolin Wu, Xi Zhang
“We study, for the first time, automated inference on criminality
based solely on still face images, which is free of any biases of
subjective judgments of human observers.”

47. Criminals
Non-criminals

48. No smile
Smile

49. T-shirt
Formal shirt

50. Perception
● Logical and calculating
● Almost never wrong, reliable
● Mistakes are absurd when they do happen
● Trustworthy
● Objective and free of any biases

51. Example

52. “Anything we build using data is going to reflect the biases and
decisions we make when collecting that data.”
Fred Benenson, ex-VP Data @ Kickstarter

53. Getting started

54. OpenAI Gym

56. Machine Learning:
The high interest credit card
of technical debt
https://ai.google/research/pubs/pub43146

57. Changing Anything Changes Everything.

58. Summary
● Be pragmatic; use the easiest thing
● Data quantity and quality really matters
● Bias and Ethics is a thing
● Testing is hard
● It’s mostly still 1999 ops wise

60. Questions?