The growing demand for personalization, the benefits of automation, and the big data accessibility all lead to ML projects becoming commonplace in the modern business world. And while more people learn how to build successful ML models, there’s still a lot of misconceptions about supporting tasks. Specifically, data annotation might look like a small and simple job that, in reality, will take up a lot of your time and resources. Besides, it’s ridden with hidden pitfalls that depend on the complexity of the project, the available resources, and security risks. I would like to lead you through the biggest of these problems and offer recommendations to avoid them. With the final checklist I’ll provide, you’ll be able to build a smart and effective data labeling strategy, whether you decide to annotate the data in-house or find an outsourcing partner.

1. &

2. 6+ Insights to
Overcome the
Hidden Pitfalls of
Data Annotation
Karyna Naminas
CEO at Label Your Data

3. The place of data annotation in the era of big data
Problems and hidden pitfalls of data annotation
Real-life annotation cases
Recommendations
Speaking Points

4. Businesses & Big Data
Expected growth of
global AI market
by 2027
$267
billion

5. Preparing the Data Can Take Up to 80% of Your
Time
Data Collection 10%
Data Unification 10%
Data Cleaning 25%
Data Augmentation 10%
Data Annotation 25%
Designing ML Algorithm 3%
ML Model Training 10%
ML Model Tuning 5%
ML Maintenance 2%

6. Common Data Annotation Tasks
NLP
Natural Language Programming
NER
Text Classification
Intent & Sentiment Analysis
Comparison
Audio-to-Text Transcription
CV
Computer Vision
2D Boxes
Semantic Segmentation
Polygons
Image Categorization
Video Annotation & Object Tracking
LiDAR, RADAR
OCR

7. PROBLEMS AND
RECOMMENDATIONS

8. Team Strategy
In-house
Internal
Freelance project
Outsourcin

9. Project context
Control
In-house Hiring
CONS
HR resources
Financial and time cost
PROS

10. Internal Delegation
PROS CONS
Project context
Control
Lack of challenge
Decreasing quality
Distraction from main
tasks

11. Freelance Project Work
PROS CONS
Saving money
Project facilitation
Poor quality
Missed deadlines
Lack of control
Preliminary work on
instructions
Understandable software

12. Outsourcing
PROS CONS
Saving money
High quality
Professional tools
Trained experts
Decreased control
Minimum commitment
project

13. Recommendation
PoC projects are best to be kept in-house
The same goes for large continuous projects (that
require labeling in small portions)
For a short-term ML project, look for an outsourcing
option without minimum commitment

14. Managing a Large Labeling Team
Annotating big or
continuous datasets

15. Objective instructions vs
subjective interpretation
Consider all limitations and
available resources
Recommendation
Re-checker to catch
potential errors
HR, Finance, and Legal
constraints

16. Ensuring the High Quality of Annotations
What method works
best for your project?

17. Quality Check Types
Cross-reference method Overlap method
Separate Re-checker teams Labelers’ background

18. Keep in Mind
Make sure your dataset and type of annotation fit your project’s goals!
Detailed instructions Training sessions
Ask for a pilot Test questions

19. Data Transfer to Outsourcing Companies
Look for a simple and
secure solution

20. Recommendation
Outsourcing company may offer:
Integration with cloud storages (Google Drive, Dropbox,
S3)
Transferring via an FTP server
Transferring via API

21. Data leaks and data
breaches
Data Protection Compliances

22. Recommendation
CCPA
California
Consumer Privacy Act

23. 2qq2
Great
annotation
requires
professional
software
Choosing the Right Labeling Tool

24. Recommendation
Free open-source tools
Paid ready solutions
Developing a basic annotation tool
State-of-the-art software
In most cases, use ready solutions to save time and efforts

25. Open-source Data Annotation Tools

26. A Few More Pro Tips
100 clients
worldwide
Keep in mind
potential growth
and model
validation
Create the point of
communication
with your
annotation partner
Keep in mind that
every project is
unique, so a unique
approach is required

27. Pre-Annotation Checklist
Do you have enough HR resources to build an in-house team?
Are you prepared to manage an in-house team?
Do you use cross-referencing or an overlap method to ensure the high quality of annotation?
Do you have a separate review/Re-checker team?
Have you considered your annotators’ background?
Do you have detailed instructions for labelers?
Do you schedule training sessions for your annotators?
Do you use tool integration or FTP/API data transfer?
Can you ensure the security of sensitive data?
Do you have the right annotation tool?

28. v
THANK YOU!

29. Q&A
session
🍿

30. 120/4 Kozatska Str.,
Kyiv, 03118, Ukraine
Hi! My name is Karyna,
I’m CEO at LabelYourData.
Contact me directly to:
karyna@labelyourdata.com
+38 (050) 046-2825
“
”
1521 Concord Pike,
Wilmington, DE 19803 USA
“Label Your Data” is part of SupportYourApp
group of companies with offices in Ukraine,
and presence in the US and over 700
employees worldwide.
Inquiries: team@labelyourdata.com