How community software supports language     documentation and data analysis                    Peter Bouda     Centro Int...
What is „open“ in software?• Open Source license (but be careful about  restrictions!)• Make participation easy  – Documen...
Community• Software projects are not only source code:  – Feedback from users  – Write documentation  – Test!!! Report bug...
Examples•   EOPAS•   LingPy and qlc•   NLTK (Natural Language Toolkit)•   Poio and PyAnnotation•   Scientific Python
EOPAS• „Ethnographic E-Research Online Presentation  System for Interlinear Text”• Present interlinear text online• Suppor...
EOPAS
Why EOPAS is open• Published on Github• EOPAS is community software:  – Based on a modern web framework (Ruby on Rails)  –...
Poio and PyAnnotation• Started during my internship in DoBeS project  „Minderico - An endangered language in  Portugal“• I...
Poio and PyAnnotation• PyAnnotation provides access to different file  formats• Provides access to data programmatically (...
Poio Editor and Analyzer• Start: Poio Editor as an „add-on“ to Elan• Open Elan transcription and add morpho-  syntactic an...
Live Demo
PyAnnotation• Parses LD files:   – Elan   – Toolbox   – Kura• Unified data access through API• Modify data structure and w...
Scientific Python• Python programming language• Collection of tools and scientific libraries:   – IPython   – NumPy and Sc...
Live Demo
CLARIN• „Common Language Resources and  Technology Infrastructure“• Current projects of CLARIN-D:  – Weblicht: SOA to crea...
Framework for Fieldwork Data• Improve annotation and analysis tasks based on  documentation data• Build a bridge between L...
Library to access LD data
Generic representation• GrAF: Graph Annotation Framework• Based on annotation graphs• Developed at American National Corpu...
Morpho-syntactic structure
Custom structures• Morpho-syntactic vs. GRAID
Development• Library is developed at CIDLeS• Web API and app is developed at  CCeH/University of Cologne• Coordination at ...
Support Open Software!• Use existing project whenever possible and  contribute by giving feedback• In LD: data drives deve...
Thank you for your attention!• pbouda@cidles.eu• Become a member of CIDLeS to support our  software development:          ...
Links• EOPAS: http://www.eopas.org/• Poio and PyAnnotation:  http://www.cidles.eu/ltll/poio• Python(x,y):  http://code.goo...
Links• Apache UIMA: http://uima.apache.org/• CLARIN-D: http://de.clarin.eu/index.php/en/
Upcoming SlideShare
Loading in …5
×

How community software supports language documentation and data analysis

510 views

Published on

Field linguists have increasingly adopted the latest technologies and tools for language documentation. Their needs have led to remarkable developments in software and archiving, exemplified by work at the MPI in Nijmegen, which leads the innovation cycles that take place in the digital working environments of field linguists. The next step in research is now the analysis and theoretical exploitation of the huge amount of data that has been collected in numerous language documentation projects that use these environments. This research will also rely on computer-based strategies, as data is instantly available in digital formats.
In this talk I will introduce some of the lesser known tools and software packages for annotation and analysis tasks. Some of these tools were created within DOBES projects and/or as community projects by small teams; they can be combined with well-known tools like ELAN or Toolbox to give researchers access to their data. I will focus on how a combination of simple, special purpose tools makes researchers more productive and how existing software libraries allow scientific projects to create their own, task-specific software tools that they can tailor to their own needs.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
510
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

How community software supports language documentation and data analysis

  1. 1. How community software supports language documentation and data analysis Peter Bouda Centro Interdisciplinar de Documentação Linguística e Social Minde/Portugal
  2. 2. What is „open“ in software?• Open Source license (but be careful about restrictions!)• Make participation easy – Documentation – Transparent development process (e.g. discuss features publically) – Attract programmers (code quality, make „giving back“ easy, online meetings, code sprints, …)• Try to create and support a community from the beginning, otherwise nobody will use your code
  3. 3. Community• Software projects are not only source code: – Feedback from users – Write documentation – Test!!! Report bugs – In our case: provide data for tests – Propose features• Best code and software quality• Websites for community development (Github, Bitbucket, …)
  4. 4. Examples• EOPAS• LingPy and qlc• NLTK (Natural Language Toolkit)• Poio and PyAnnotation• Scientific Python
  5. 5. EOPAS• „Ethnographic E-Research Online Presentation System for Interlinear Text”• Present interlinear text online• Supported files: – Elan – Transcriber – Toolbox interlinear glossed text
  6. 6. EOPAS
  7. 7. Why EOPAS is open• Published on Github• EOPAS is community software: – Based on a modern web framework (Ruby on Rails) – clear and documented deployment strategy (how to use it on your own) – easy to maintain, low entry level, good code quality – several options for participation listed on website• Publish your data with EOPAS and support the development!
  8. 8. Poio and PyAnnotation• Started during my internship in DoBeS project „Minderico - An endangered language in Portugal“• Ideas and support by Prof. Johannes Helmbrecht• Support by Institute for General Linguistics and Language Typology at University of Munich• Support by Institute for General Linguistics at University of Bamberg
  9. 9. Poio and PyAnnotation• PyAnnotation provides access to different file formats• Provides access to data programmatically (API)• Poio is graphical user interface (GUI) on top of PyAnnotation• Two software packages: – Poio Editor – Poio Analyzer
  10. 10. Poio Editor and Analyzer• Start: Poio Editor as an „add-on“ to Elan• Open Elan transcription and add morpho- syntactic annotations• Analyzer to search in Elan and Toolbox files• Now: adding support for GRAID (Grammatical Relations and Animacy in Discourse) and any other annotation types• Goal: a highly customizable desktop software for diverse annotation and analysis scenarios / sparse annotation
  11. 11. Live Demo
  12. 12. PyAnnotation• Parses LD files: – Elan – Toolbox – Kura• Unified data access through API• Modify data structure and write Elan files again – good for batch processing• Combines well with Scientific Python for analysis
  13. 13. Scientific Python• Python programming language• Collection of tools and scientific libraries: – IPython – NumPy and SciPy, scikit-learn, networkx, … – An easy installer: Python(x,y)• Alternative to Matlab, R, and other mathematical tools• But: general usage for software development
  14. 14. Live Demo
  15. 15. CLARIN• „Common Language Resources and Technology Infrastructure“• Current projects of CLARIN-D: – Weblicht: SOA to create annotated corpora – Virtual Language Observatory• “Kurationsprojekt” to develop software framework to access fieldwork data• Among others: based on code of PyAnnotation
  16. 16. Framework for Fieldwork Data• Improve annotation and analysis tasks based on documentation data• Build a bridge between LD and NLP data formats and technology – Lexan, UIMA, …• DoBeS corpus as central resource• Develop a basic software library• Web API and web app as reference implementation
  17. 17. Library to access LD data
  18. 18. Generic representation• GrAF: Graph Annotation Framework• Based on annotation graphs• Developed at American National Corpus• Common representation helps to process and analyze data from different sources• Map LD data (Elan, Toolbox, …) to GrAF• Users work with „structures“ (GRAID, Morph- Syntax, POS, …) that can be mapped to a GUI
  19. 19. Morpho-syntactic structure
  20. 20. Custom structures• Morpho-syntactic vs. GRAID
  21. 21. Development• Library is developed at CIDLeS• Web API and app is developed at CCeH/University of Cologne• Coordination at Institute for Linguistics, University Cologne• August/September 2012 – July 2013
  22. 22. Support Open Software!• Use existing project whenever possible and contribute by giving feedback• In LD: data drives development, developers need files to test• Share your code as soon as possible• Use existing infrastructure like Github to share
  23. 23. Thank you for your attention!• pbouda@cidles.eu• Become a member of CIDLeS to support our software development: www.cidles.eu
  24. 24. Links• EOPAS: http://www.eopas.org/• Poio and PyAnnotation: http://www.cidles.eu/ltll/poio• Python(x,y): http://code.google.com/p/pythonxy/• LingPy: http://lingulist.de/lingpy/• QLC: https://github.com/pbouda/qlc• NLTK: http://nltk.org/
  25. 25. Links• Apache UIMA: http://uima.apache.org/• CLARIN-D: http://de.clarin.eu/index.php/en/

×