The 2014-born child of crowdsourcing is crowdshaping. It means using personal data and user behavior to shape and reshape a product or a service. One example in the online translation industry, is when website content is machine translated first and based on page views, bounce rate and engagement information, the most important content is post-edited or translated by human translators in a later phase. How can we harvest user or customer data and feedback to offer the right level of quality? How to measure usability and customer behavior? How to interpret the results and translate them into Business Intelligence? In several TAUS conferences and workshops in the past, the importance of linking figures to ROI was emphasized. Quality and productivity scores in isolation of the bigger business picture are meaningless.

1. Business Intelligence
How to translate different measurements and scores into
Business Intelligence
Tom Shaw (Capita TI)
Paul Mangell (Alpha CRC)
Scott Carothers (Kinetic TheTechnologyAgency)

2. The Goal – Improve our Industry using Business Intelligence
?
What Data? How to capture?
What’s relevant? To whom?
How to interpret?
Adequacy
Fluency
TM Leveraging
User Sentiment
MT Leveraging
Accuracy
Style / Readability
Translator Quality
User Engagement
TER, Bleu, Meteor
Per Word
Per Hour CTR
PE Distance
TAT

3. Translation Quality – Business Intelligence from a LSP perspective
Tier 3 content
Tier 2 content
Tier 1 content
Support content, non-publishable
user driven or reference
Publishable content for user
assistance or product support related
Highly visible content, such as
Marketing or software localization as
well as legal

4. Quality Evaluation Triangle
QUALITY
Users
LSPBuyer
The user experience should be equal or
better of that achieved in Source language
Customising metrics and QA
process in coordination with the
buyer will assure expectations are
measured adequately
The buyer’s “internal clients” play a
key role in setting up quality
expectations and measuring success

5. Quality Evaluation - LSP and Buyer
Content type specific QA
process and metricsLSP
Evaluation Criteria / Model:
Error Typology / Adequacy / Fluency
Territory specific requirements
and preferences
Buyer
Internal Clients

6. Quality Evaluation - Users
Users
User feedback and content
engagement levels
Evaluation Criteria / Model:
User Activity / Engagement

7. Quality Evaluation Models - Thoughts
1) Quality Evaluation Models have to be closely linked to purpose / end use
2) Standard models not always good enough. Customised to specific needs
3) Metrics / Benchmarks should be set with end user in mind
……………………. Who should drive? LSP<>Buyer, Buyer<>Users….

8. Common Metrics the Buyers Maintain
Most common metrics:
• Which LSP has the assignment
• Delivery Time
• How long did post-editing take?

9. Most Valuable Metrics for the Buyer
1. Aggregate volume
2. Turnaround time
3. TM Analysis
4. LSP ratings
5. Cost tracking

10. Enterprise Metrics Dashboard

11. Checks and balances in collecting and sharing data
• How will the data be collected, analysed, presented?
• Is all data good data?
• Who needs to know which data set?
• What is the cost of collecting, analysing, and presenting data?
• What us the impact of negative or positive data on the enterprise, and the
resource/supplier?
• What should be done in the light of the new data?
• Who will take the actions implied/required?
• Who needs to agree that the data strategy is correct? Buy in v apathy/antipathy

12. Challenges / Considerations / Points to Discuss?
• What is a quality level? Who sets / defines it?
• All supply chain to work towards the same goal
• Linking QE results from LSP for different levels with End User Feedback
• External factors affecting the results – e.g. translation quality directly linked to # sales?
• Not all languages can be treated equally – e.g. acceptance of MT in Japan vs Spain
• High quality / perfect really always required?
• Variable quality levels – impact on brand?
• How do high QA scores correlate to user engagement?
• How to customise your metrics based on the communication objectives
• Professional translators offering a ‘less than perfect’ service