This document summarizes a framework for reconciling strings to structured identifiers. It is a two-stage process that first standardizes data using transformers, then matches the standardized data using matchers. The framework includes over 40 transformers and 20 matchers. It has been used to create an open source plant name reconciliation service that standardizes strings to identifiers from a particular source. The service exposes endpoints that allow flexible querying and is compatible with Open Refine. Future work may expand it to support additional data types like specimens.

1. Strings to things: a user-friendly
framework for data reconciliation
Nicky Nicolson, RBG Kew
@nickynicolson
Biodiversity Information Standards (TDWG) annual meeting
Nairobi, Kenya / 28th September – 1 October 2015

2. Reconciliation
• Turns a string representation of an entity into
an actionable identifier.
e.g.:
Tahina spectabilis
Will reconcile to:
http://ipni.org/urn:lsid:ipni.org:names:77086615-1

4. Maximise reuse, two stage process
1. Standardise data
- Package of 40 plus “transformers”
- All accept a string input, produce a string
output

5. Examples of transformers

6. Open Refine screenshot

7. Open source
http://github.com/RBGKew/StringTransformers

8. Maximise reuse, two stage process
2. Match the data
- Package of 20 plus “matchers”
- All accept two inputs and return a flag if they
match

9. Configuring a service
1) Read tabular data (file or DB)
2) Configure transformers
3) Configure matchers

10. Run it…
1) Service description
2) Three service endpoints
3) Javascript query interface

11. IPNI Reconciliation Service

12. 3 service endpoints

13. IPNI Reconciliation Service

14. Flexible web service
• Open Refine compatible
• But underneath it’s JSON over HTTP
• … so call it from any programming language

15. Service metadata

16. Service call

17. Service response

18. List of reconciliation services
https://github.com/OpenRefine/OpenRefine/wi
ki/Reconcilable-Data-Sources

19. Open source
https://github.com/RBGKew/Reconciliation-and-Matching-Framework

20. What we’ll work on in the future

21. Reconciliation services on different
data types
• Specimens
– Add DwCA as a readable data store
– Collections focussed transformers & matchers
– Resolve & link specimen duplicates
• People
• Trait glossaries

22. Integration with github

24. Thanks to:
• Biodiversity Informatics team (Abigail Barker,
Matt Blissett, James Crowe, John Iacona, Rob
Turner, Alecs Gueder)
• Plant & fungal name curation team (Christine
Barker / Irina Belyaeva / Katherine Challis /
Rafael Govaerts / Paul Kirk / Heather Lindon /
Emma Williams)
• Data improvement team (Anna Lynch, Rachel
Witherow, Malin Rivers, Esther Wainwright-Deri)

25. @nickynicolson / n.nicolson@kew.org
http://bit.ly/k-names-service
http://github.com/RBGKew
Biodiversity Information Standards (TDWG) annual meeting
Nairobi, Kenya / 28th September – 1 October 2015