How community software supports language documentation and data analysis
Upcoming SlideShare
Loading in...5
×
 

How community software supports language documentation and data analysis

on

  • 549 views

Field linguists have increasingly adopted the latest technologies and tools for language documentation. Their needs have led to remarkable developments in software and archiving, exemplified by work ...

Field linguists have increasingly adopted the latest technologies and tools for language documentation. Their needs have led to remarkable developments in software and archiving, exemplified by work at the MPI in Nijmegen, which leads the innovation cycles that take place in the digital working environments of field linguists. The next step in research is now the analysis and theoretical exploitation of the huge amount of data that has been collected in numerous language documentation projects that use these environments. This research will also rely on computer-based strategies, as data is instantly available in digital formats.
In this talk I will introduce some of the lesser known tools and software packages for annotation and analysis tasks. Some of these tools were created within DOBES projects and/or as community projects by small teams; they can be combined with well-known tools like ELAN or Toolbox to give researchers access to their data. I will focus on how a combination of simple, special purpose tools makes researchers more productive and how existing software libraries allow scientific projects to create their own, task-specific software tools that they can tailor to their own needs.

Statistics

Views

Total Views
549
Views on SlideShare
549
Embed Views
0

Actions

Likes
0
Downloads
1
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

How community software supports language documentation and data analysis How community software supports language documentation and data analysis Presentation Transcript

  • How community software supports language documentation and data analysis Peter Bouda Centro Interdisciplinar de Documentação Linguística e Social Minde/Portugal
  • What is „open“ in software?• Open Source license (but be careful about restrictions!)• Make participation easy – Documentation – Transparent development process (e.g. discuss features publically) – Attract programmers (code quality, make „giving back“ easy, online meetings, code sprints, …)• Try to create and support a community from the beginning, otherwise nobody will use your code
  • Community• Software projects are not only source code: – Feedback from users – Write documentation – Test!!! Report bugs – In our case: provide data for tests – Propose features• Best code and software quality• Websites for community development (Github, Bitbucket, …)
  • Examples• EOPAS• LingPy and qlc• NLTK (Natural Language Toolkit)• Poio and PyAnnotation• Scientific Python
  • EOPAS• „Ethnographic E-Research Online Presentation System for Interlinear Text”• Present interlinear text online• Supported files: – Elan – Transcriber – Toolbox interlinear glossed text
  • EOPAS
  • Why EOPAS is open• Published on Github• EOPAS is community software: – Based on a modern web framework (Ruby on Rails) – clear and documented deployment strategy (how to use it on your own) – easy to maintain, low entry level, good code quality – several options for participation listed on website• Publish your data with EOPAS and support the development!
  • Poio and PyAnnotation• Started during my internship in DoBeS project „Minderico - An endangered language in Portugal“• Ideas and support by Prof. Johannes Helmbrecht• Support by Institute for General Linguistics and Language Typology at University of Munich• Support by Institute for General Linguistics at University of Bamberg
  • Poio and PyAnnotation• PyAnnotation provides access to different file formats• Provides access to data programmatically (API)• Poio is graphical user interface (GUI) on top of PyAnnotation• Two software packages: – Poio Editor – Poio Analyzer
  • Poio Editor and Analyzer• Start: Poio Editor as an „add-on“ to Elan• Open Elan transcription and add morpho- syntactic annotations• Analyzer to search in Elan and Toolbox files• Now: adding support for GRAID (Grammatical Relations and Animacy in Discourse) and any other annotation types• Goal: a highly customizable desktop software for diverse annotation and analysis scenarios / sparse annotation
  • Live Demo
  • PyAnnotation• Parses LD files: – Elan – Toolbox – Kura• Unified data access through API• Modify data structure and write Elan files again – good for batch processing• Combines well with Scientific Python for analysis
  • Scientific Python• Python programming language• Collection of tools and scientific libraries: – IPython – NumPy and SciPy, scikit-learn, networkx, … – An easy installer: Python(x,y)• Alternative to Matlab, R, and other mathematical tools• But: general usage for software development
  • Live Demo
  • CLARIN• „Common Language Resources and Technology Infrastructure“• Current projects of CLARIN-D: – Weblicht: SOA to create annotated corpora – Virtual Language Observatory• “Kurationsprojekt” to develop software framework to access fieldwork data• Among others: based on code of PyAnnotation
  • Framework for Fieldwork Data• Improve annotation and analysis tasks based on documentation data• Build a bridge between LD and NLP data formats and technology – Lexan, UIMA, …• DoBeS corpus as central resource• Develop a basic software library• Web API and web app as reference implementation
  • Library to access LD data
  • Generic representation• GrAF: Graph Annotation Framework• Based on annotation graphs• Developed at American National Corpus• Common representation helps to process and analyze data from different sources• Map LD data (Elan, Toolbox, …) to GrAF• Users work with „structures“ (GRAID, Morph- Syntax, POS, …) that can be mapped to a GUI
  • Morpho-syntactic structure
  • Custom structures• Morpho-syntactic vs. GRAID
  • Development• Library is developed at CIDLeS• Web API and app is developed at CCeH/University of Cologne• Coordination at Institute for Linguistics, University Cologne• August/September 2012 – July 2013
  • Support Open Software!• Use existing project whenever possible and contribute by giving feedback• In LD: data drives development, developers need files to test• Share your code as soon as possible• Use existing infrastructure like Github to share
  • Thank you for your attention!• pbouda@cidles.eu• Become a member of CIDLeS to support our software development: www.cidles.eu
  • Links• EOPAS: http://www.eopas.org/• Poio and PyAnnotation: http://www.cidles.eu/ltll/poio• Python(x,y): http://code.google.com/p/pythonxy/• LingPy: http://lingulist.de/lingpy/• QLC: https://github.com/pbouda/qlc• NLTK: http://nltk.org/
  • Links• Apache UIMA: http://uima.apache.org/• CLARIN-D: http://de.clarin.eu/index.php/en/