Your SlideShare is downloading. ×
Mediabase Ready and First Analysis Report
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Mediabase Ready and First Analysis Report

1,572
views

Published on

European Commission Seventh Framework Project (IST-257822) …

European Commission Seventh Framework Project (IST-257822)
Mediabase Ready and First Analysis Report
Deliverable D4.3
Editor: Michael Derntl (RWTH Aachen University)
Contributors: Adam Cooper, Manh Cuong Pham, Ralf Klamma, Dominik Renzel
Dissemination level: Public
Delivery date: 2011-09-30

Published in: Technology, Education

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,572
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
25
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Coordination and Support Action European Commission Seventh Framework Project (IST-257822)Mediabase Ready and First Analysis ReportDeliverable D4.3Editor: Michael Derntl (RWTH Aachen University)Contributors: Adam Cooper, Manh Cuong Pham, Ralf Klamma, Dominik RenzelDissemination level: PublicDelivery date: 2011-09-30Work Package WP4: Weak Signals Analysis – Emerging RealityDissemination Level PublicStatus Version 1.0 — FinalDate September 30, 2011
  • 2. Amendment History Version Date Editor Description/Comments 1.0 30 Sept. 2011 Michael Derntl Final version Contributors Name Institution RoleMichael Derntl RWTH Aachen University Editor/AuthorAdam Cooper University of Bolton (CETIS) AuthorRalf Klamma RWTH Aachen University AuthorManh Cuong Pham RWTH Aachen University AuthorDominik Renzel RWTH Aachen University AuthorPaul Lefrere The Open University (OU) ReviewerLampros Stergioulas Brunel University ReviewerChristian Voigt Zentrum für Soziale Innovation (ZSI) ReviewerDeliverable description in the DoW: The deliverable will describe the continuation of the established PROLEARN Mediabase equipped with new tools combining the existing social network analysis with topic mining. This will realize a structural-semantic analysis of signals from the Web 2.0 strongly related to technology enhanced learning. Results from the analysis will be reported here but can be obtained continuously from the Web interfaces of the Mediabase afterwards.
  • 3. Contents1 Introduction .......................................................................................................... 12 The TEL-Map Mediabase ........................................................................................2 2.1 Conceptual Model of the TEL-Map Mediabase ........................................................................... 3 2.2 Components Overview .................................................................................................................. 4 2.3 Analysis Approach ......................................................................................................................... 7 2.4 Potential Questions ..................................................................................................................... 103 Analysis of the European TEL Project Landscape .................................................. 12 3.1 Data Set ........................................................................................................................................ 12 3.2 TEL Projects as Social Networks ................................................................................................ 14 3.3 Project Consortium Progression................................................................................................. 15 3.3.1 FP7 Projects ....................................................................................................................... 15 3.3.2 All TEL Projects – FP6, FP7, and eContentplus .............................................................. 16 3.3.3 Identifying Project Clusters .............................................................................................. 17 3.4 Organizational Collaborations .................................................................................................... 19 3.4.1 Collaborations in FP7 projects ......................................................................................... 19 3.4.2 Collaborations in all TEL Projects: FP6, FP7, and eContentplus ................................... 21 3.4.3 Dynamic SNA of the TEL Project Landscape .................................................................. 25 3.5 Geo-Mapping TEL Projects.........................................................................................................284 Analysis of TEL Publication Outlets ...................................................................... 29 4.1 Data Set ........................................................................................................................................ 29 4.2 Social Network Analysis of TEL Venues and Papers ................................................................. 31 4.3 Co-Authorship Network Analysis ............................................................................................... 32 4.3.1 Formal Foundations.......................................................................................................... 32 4.3.2 Overview ............................................................................................................................ 32 4.3.3 Dynamic SNA .................................................................................................................... 34 4.3.4 Most Prolific Authors and Their Topics ........................................................................... 35 4.3.5 Overall TEL Co-authorship Network ............................................................................... 37 4.3.6 Central Authors in the Co-Authorship Network..............................................................38 4.4 Structural-Semantic Analysis: SNA and Topic Mining Combined ........................................... 39 4.5 Citation Network Analysis .......................................................................................................... 435 Analysis of the TEL Social Web .............................................................................44 5.1 Social Web Data Set .................................................................................................................... 45 5.2 Formal Foundations .................................................................................................................... 46 5.3 Results.......................................................................................................................................... 47 5.3.1 TEL Blog Network and Most Central Blogs ..................................................................... 47
  • 4. 5.3.2 TEL Blog Clusters .............................................................................................................. 49 5.3.3 Bursts ................................................................................................................................. 516 Embeddable Interactive Visualizations and Queries ............................................. 527 Key Findings for Weak Signals ............................................................................. 55 7.1 TEL Projects................................................................................................................................. 55 7.2 TEL Papers................................................................................................................................... 56 7.3 TEL Social Web............................................................................................................................ 578 Conclusion........................................................................................................... 57References ................................................................................................................. 58Appendix A: TEL Projects — Timeline ........................................................................ 60Appendix B: TEL Projects — SNA Metrics .................................................................... 61
  • 5. FiguresFigure 1: Concept map underlying of the TEL-Map Mediabase metamodel. ............................................. 4Figure 2: TEL-Map Mediabase components overview model. .................................................................... 5Figure 3: Data model of TEL projects. ........................................................................................................ 12Figure 4: Word clouds of project descriptions. .......................................................................................... 14Figure 5: FP7 TEL projects graph visualization. ........................................................................................ 15Figure 6: Project consortium progression between FP6, FP7, and eContentplus projects. .................... 17Figure 7: Visualization of the FP7 collaboration graph.............................................................................. 19Figure 8: Center region cut-out of the FP7 collaboration graph. ..............................................................20Figure 9: Word cloud of the 20 word stems with highest frequency in the FP7 project descriptions .... 21Figure 10: Partner collaborations spanning FP6, FP7, and eContentplus projects. ................................ 22Figure 11: Local clustering of organizations plotted against (a) PageRank and (b) degree. .................... 24Figure 12: Overall development of collaboration network since 2004. .................................................... 26Figure 13: Impact of newly launched projects the collaboration network................................................ 26Figure 14: Impact of organizations on collaboration. ................................................................................ 27Figure 15: Development of the ratio of projects coordinated by novice organizations ............................ 27Figure 16: Google Map overlay with organizations involved in TEL projects. .........................................28Figure 17: Data model for TEL papers and events. ....................................................................................30Figure 18: Word cloud of most frequent terms in TEL conference paper titles. ...................................... 31Figure 19: Development model for conference communities.................................................................... 32Figure 20: Cumulative annual (co-)author figures of selected TEL conferences over the last 10 years. 33Figure 21: Co-authorship network visualization for the TEL conferences. .............................................. 33Figure 22: Co-authorship network measures of five conferences in TEL................................................. 35Figure 23: Most frequent terms in papers of top TEL authors in 2010. ................................................... 37Figure 24: Complete co-authorship network in the core TEL venues. ..................................................... 37Figure 25: Co-authorship network of the “inner circle” of authors in the core TEL venues. ..................38Figure 26: Citation network measures of five conferences in TEL. .......................................................... 44Figure 27: Relational model of the TEL blogosphere. ............................................................................... 45Figure 28: Number of blogs added to and blog entries indexed in the TEL-Map Mediabase. ................ 46Figure 29: TEL blogs link network visualization, excluding self-references. ........................................... 47Figure 30: Top 100 word stems appearing in 2011 blog entries of the top 20 blogs................................ 49Figure 31: Colored TEL blog clusters. .........................................................................................................50Figure 32: Bursty terms appearing only in 2011. ....................................................................................... 51Figure 33: Bursty terms with rising frequency over the last three years. ................................................. 52Figure 34: Visualization of the same SQL query as a table (left) and as a graph (right). ........................ 53Figure 35: SQL query visualization as an annotated timeline. .................................................................. 54
  • 6. TablesTable 1: Uses of social network analysis and topic mining in the TEL-Map Mediabase. ........................ 10Table 2: Overview of the 77 TEL Projects in the TEL-Map Mediabase..................................................... 13Table 3: TEL project clusters in FP6, FP7, and eContentplus (ECP) and the word clouds of their projectdescriptions. ................................................................................................................................................. 18Table 4: Top 30 organizations involved in TEL projects by PageRank. The numbers in square bracketsnext to the values represent the rank of that value among all 604 organizations.................................... 23Table 5: Strongest partnership bonds over all TEL projects in FP6, FP7 and eContentplus. ................. 25Table 6: Selection of conferences relevant to the TEL community. .......................................................... 31Table 7: Fifteen most prolific authors at conferences and journals with a broad TEL scope. Namesmarked with an asterisk (*) indicate authors currently based in Europe. ................................................ 36Table 8: Top 15 TEL authors by betweenness centrality. .......................................................................... 39Table 9: Top ten co-author pairs in core TEL venues. ............................................................................... 39Table 10: Betweenness centrality of authors of papers identified in D4.1. ...............................................40Table 11: Summary of structural-semantic analysis: themes and matching papers. ............................... 41Table 12: Top twenty blog sources by PageRank. The number in square bracket indicates the blog’soverall rank for the respective metric. .......................................................................................................48Table 13: Clusters of TEL blogs indexed in Mediabase.............................................................................. 49
  • 7. 1 IntroductionThe European Framework Programmes (FP) for Research and Technological Development are a keypillar of the European research area and act as the primary vehicle for the European Union to createand sustain growth, employment and global competitiveness [3]. FPs are complex frameworksdefining the specific research programmes and challenges to be tackled over a seven-year period with amulti-billion Euro budget. In FP7, the Cooperation programme, which also hosts the TechnologyEnhanced Learning (TEL) thematic area, received the largest share of the total FP7 funds. For thetwenty-six partly completed and partly running TEL projects in FP7 the European Commission hasprovided or will provide a total funding of more than one hundred million Euro. People andorganizations with a stake in TEL research and development are likely to be interested in knowingwhere this enormous amount of money went and what impact it has generated and is generating onthe TEL landscape. First and foremost, the European Commission itself is interested in what impactthe spending has generated over the years. In addition, there are many organizations and individualsin Europe that have a stake in TEL, e.g. technology providers, technology adopters, and highereducation institutes, to name a few (see [16] for a more comprehensive list of TEL-Map stakeholders).To address the issue of generating such information based on strong and weak signals in a variety of(web) sources, one core threads pursued in TEL-Map includes the application of social networkanalysis and visualization as well as topic mining.This deliverable reports on social network analysis and topic mining work performed in WP4, “WeakSignal Analysis—Emerging Reality,” to support weak signal analysis and the mapping of the currentTEL landscape. To achieve this, the deliverable outlines the conceptual foundations of the TEL-MapMediabase, where all underlying data sources were stored, and presents first results of the analyses.The main task underlying the work reported in this deliverable is Task 3 in WP4, which comprises thefollowing objectives: • TEL-Map Mediabase: Based on the PROLEARN Mediabase the aim was to develop a TEL- Map Mediabase, which shall contain social media artifacts and related resources to support the mapping of the TEL landscape and complement the Delphi-based weak signal analysis approach reported in D4.1 [23]. The focus in the TEL-Map Mediabase shall be on issues, topics, and structures of relevance to TEL. This required a filtering of existing Mediabase content, an extension of the sources fed into Mediabase with TEL-related content and development of new tools to support analyses of these extended sources. The TEL-Map Mediabase is presented in Section 2. • Social Network Analysis: One of the pillars of the analysis methodology in WP4 is social network analysis (SNA) of actors involved in TEL and their relationships. “Actor” is meant as an abstract concept in this context, which can refer to various kinds of entities like people, conferences, projects, publications, and so forth. SNA offers highly effective methods for obtaining interactive visualizations and network metrics for these social networks, allowing the identification of the most important actors from a wide range of perspectives. In this deliverable the focus is on analysis of TEL projects and organizations involved in those projects (Section 3); TEL papers, authors and publication outlets (Section 4); and TEL social media sources (Section 5). • Topic Mining: In addition to the network-metrics and structural analysis approach taken in SNA, the analysis methodology shall be complemented with a topic mining approach. The objective is to identify bursty topics, shifts in topics, emerging/declining topics from a variety of sources in TEL, achieving a structural-semantic analysis of signals. This is tackled in combination with SNA in Sections 3 through 5. –1–
  • 8. • Continuous analysis: As indicated in the title, this deliverable was conceived to present a first analysis report, while TEL stakeholders shall be supported in continuously obtaining up- to-date analysis results from the Mediabase web interfaces. This requires web-based tools for continuous analysis of TEL sources (dealt with in Section 6) and an online resource page where data sets and data processing components can be accessed and/or downloaded. For the latter goal, a resource page was set up on the TEL-Map homepage. This D4.3 resource page is available at http://telmap.org/?q=content/d4.3 and will be continuously updated with pointers to results obtained, tools developed, and analyses performed in WP4—Task 3, which will continue to run until the end of the project.In regard to the embedding of this work into TEL-Map’s overall WP structure, the WP4’s mission—i.e.the identification of weak signals that can inform the overall road-mapping process—also requires usto propel the convergence of different analytical methods. For instance, this can be achieved in WP4 byfeeding results from one analytical method into another one in order to cross-validate and enrichexisting findings, but it also needs to happen between WPs, e.g. by informing WP5’s gap analysis andWP3’s scenario building. Gap analysis aims to explore why some technologies seem to be much moreprominent in TEL research than in TEL practice (e.g. consider the uptake of 3D worlds) and othertechnologies are slowly becoming mainstream with no matching amount of research available (e.g.laptops in schools or social media at the workplace). Here, weak signals can inform an in-depthanalysis of specific technologies by considering the spread of awareness of that technology acrossvarious communities as well as the use of synonyms referring to the same set of issues but underdifferent labels. Likewise, scenario building events (WP3) can be informed through weak signals asthey are early indicators of change that have the potential to alter the future of TEL adopters and TELproviders. In this context, scenarios that consciously consider weak signals increase their robustness,leading to better strategic planning processes.This deliverable is structured as follows. In Section 2 we introduce the TEL-Map Mediabase,containing data relevant to TEL in terms of projects, publications, and social media. Each of thesubsequent sections presents first analyses performed and results obtained in the TEL-Map Mediabasesources, i.e. TEL projects in Section 3, TEL publications in Section 4, and TEL blogosphere inSection 5. An embeddable, widget-based toolkit for enabling stakeholders to query and visuallyinteract with the data contained in the TEL-Map Mediabase is presented in Section 6. Section 7 drawskey findings from the analysis for weak signals collection from the core analysis sections, and Section 8wraps up the deliverable with a discussion of limitations and an outlook on upcoming work in WP4.2 The TEL-Map MediabaseTEL-Map Mediabase is an evolution of the established PROLEARN Mediabase. In this section we firstdescribe the original idea and concept of Mediabase and continue with detailing the structure, content,and meta-model of the enhanced TEL-Map Mediabase.In the PROLEARN project1, a TEL project funded by the European Commission under FP6, one coreeffort was the creation and maintenance of a media base for TEL in Europe, providing different targetaudiences like scientists, policy makers, and communities of practice with digital information obtainedfrom mailing lists, newsletters, blogs, RSS/Atom feeds, websites, and so forth [10]. In addition tocollecting large amounts of data, one key objective was the provision of easy-to-use end-user tools forextracting and presenting relevant information contained in the Mediabase, e.g. for cross-media socialnetwork analysis, self-observation and self-modeling of communities [18], collaborativeadministration and retrieval of media artifacts, etc. The key concepts in the metamodel of thePROLEARN Mediabase are (cf. [10], p. 248-9):1 http://www.prolearn-project.org –2–
  • 9. • Community as a sub-network of the whole network, representing trustful relations among its members; • Process as a value-adding set of activities performed by community members, e.g. acquisition, retrieval, monitoring; • Actor as humans, users or groups of humans/users performing and being affected by processes; • Medium as an artifact produced or consumed by processes.For the development of the TEL-Map Mediabase, particular emphasis was put on the TELblogosphere, which is being observed and continuously retrieved using special-purpose crawlers (cf.[9]); the blogosphere sources in the Mediabase were extended by the TEL-Map members. In addition,the artifacts stored and indexed in the Mediabase were extended with digital information on EuropeanTEL projects as well as publications in TEL-related conferences and journals.2.1 Conceptual Model of the TEL-Map MediabaseTEL-Map aims to empower stakeholders to find relevant projects and useful outputs as well as newcollaborators for TEL projects; it also aims at giving a rich overview of different types of actorsinvolved in the TEL domain (see DoW, p. 17-18). WP4 in particular focuses on analyses andvisualizations from social media items gathered and automatically crawled from relevant sources. Torealize these ambitious objectives, we have enhanced and extended the metamodel and the content ofthe existing PROLEARN Mediabase. This enhanced TEL-Map Mediabase additionally includesinformation on TEL projects and participants funded by the European Commission, as well as authorsand their papers published in TEL-related conferences and journals.The conceptual model of TEL-Map Mediabase is displayed in Figure 1. It exposes three main areas: • TEL Social Media: blogs, feeds, and blog entries; currently focusing on the blogosphere that includes TEL-related blog sources. • TEL Projects: information on projects funded by the European Commission under FP6, FP7, and eContentplus, including information on participating organizations. • TEL Papers: information on papers published in TEL-related journals, conferences, and workshops.For each of these three areas there is a dedicated database schema. These schemas are described indetail in the relevant sections. There are several components (crawlers, importers, exporters, and end-user tools) which were developed to obtain the relevant data, to feed the data into the database, as wellas to extract and interact with the data. These are described in Section 2.2.Limitations. While the TEL-Map Mediabase databases contain an enormous amount of data, thereare several concepts and their links in Figure 1 which are currently not or only partly represented inthe data. These include: • Meeting and Project Meeting: While we have data on conference and workshop events in the TEL Papers database, we do not yet have data on project meetings (some of which are collocated with other events). This information is missing since there we do not yet have mechanisms of automatically obtaining these data. • Deliverable: Project deliverables are also not yet included. This can be done in the future by crawling the web pages of the projects stored in our TEL Projects database. However, we expect that manual editing will be required, since the deliverable pages are not uniform across different projects. For some projects, the deliverables cannot be found at all on the project website. –3–
  • 10. TEL Social Media Blogosphere part of has Blog Comment has TEL Papers refs refs published at Entry Publication Paper Venue refs post is a has is a has author of Person Author associated is a with TEL Projects Journal Organization take Deliverable part in consortium member Conference Workshop produce Project Project meeting organize Meeting is a collocated with Figure 1: Concept map underlying of the TEL-Map Mediabase metamodel. • Person: The concept “person” is actually the glue between the three different databases, since a person can be an author of a paper in the TEL Papers data, the owner of a blog in the TEL Media data, and a member of an organization participating in a project indexed in the TEL Projects data. We do currently not have an automated procedure that is capable of matching and obtaining data related to persons, mostly because the data is not readily available (e.g. some blogs do not contain personal information on their author, and most projects do not provide detailed information on the persons involved). We aim to work toward this integration in upcoming WP4 work.2.2 Components OverviewThe components of TEL-Map Mediabase are conceptually arranged in different groups or layers (seeFigure 2): the information to be used for weak signal analysis in the context of Mediabase is containedin many different web data sources. To collect and filter the relevant information in structured format,a set of importers and crawlers were deployed, which ingest the relevant data into different databases(or database schemas). To process the data for analysis, visualization or any other kind of interaction,a set of exporters enables end-user applications to obtain and present the data. The layers and theircomponents are described in detail below.Importers. This layer includes services and processes that obtain relevant data from web sources andtransform these data into a structured, relational database format. • Blog Crawler: The blog crawler is deployed as a cron job, which runs every night. It crawls the RSS/Atom feeds and the websites of indexed sources and extracts new entries and ingests –4–
  • 11. Web Data Sources European Community DBLP Publisher Information Pages Blogosphere Bibliography Pages LearningFrontiers Importers Portal Projects DBLP Abstracts Blog Feed Feed Crawler Importer Crawler Crawler Importer Aggregator Mediabase Databases Commander TEL Projects TEL Papers TEL Media Exporters CSV Data GraphML Visualization Exporter Exporter Widget Creator Legend Service / Data Processing Apps Process Graph Visualization Query Widgets and Analysis Apps Database R Excel Query Query yEd Graphviz Visualizer Explorer End-User Application Matlab ... Gephi ... Data Flow Figure 2: TEL-Map Mediabase components overview model. those into the database. Upon ingestion it not only stores the raw HTML of the entries; it also extracts a plain-text, non-markup version of the content, the comments associated with each blog entry, the URLs it references, and it computes burstiness of terms occurring in blog entries. The blogs scheduled for indexing are entered in two ways: (1) directly through the Mediabase Commander on the Learning Frontiers portal, or (2) indirectly through the Feed Aggregator, which is installed on the Learning Frontiers portal to collect links to relevant RSS or Atom feeds. These feeds are automatically ingested into the TEL Media database by the Feed Importer. • Abstracts Crawler: The TEL Papers database contains data like title, authors and citations on TEL-related papers. Since DBLP, the data source of the TEL papers database, does not contain abstracts and keywords, the goal of this crawler is to enhance the basic paper information with abstracts and keywords. The following conferences were crawled: ECTEL, ICWL, ICALT, ITS, DIGITEL and WMTE. Since the crawler supports the abstract pages of springerlink.com (Springer Verlag), computer.org and IEEExplore, the crawler can be used to crawl many more conferences. The crawler is written in Ruby using the Mechanize Library for extracting the information from the HTML pages. The crawler does not directly interact with the TEL papers database. Instead, desired information from the database has to be exported and imported as CSV data. • Feed Importer: One objective of TEL-Map is to analyze the voices in TEL to detect weak signals. This required enriching the Mediabase with TEL-related social media artifacts2. On the Learning Frontiers portal, we installed the aggregator module, which allows registered2 See task 3 in the description of WP4 in the DoW, p. 39: “We will integrate current RSS aggregators to enhance the contents of the Mediabase.” –5–
  • 12. users to provide links to their favorite TEL-related feeds, either RSS or Atom feeds. This module offers several forms of access to the aggregated feeds, e.g., directly through Drupal’s mysql relational database or through a machine-processible OPML file that contains all RSS or Atom feed sources, or through the Learning Frontiers portal front-end, which will display the recent feed entries to the user as an HTML page. To integrate the aggregated feeds into Mediabase, we developed a module that fetches all feeds from the feed aggregator that were not yet ingested into Mediabase; for each matching feed, the module then creates a blogwatcher project entry (including the feed’s tag associations) in Mediabase. Once a day, a blog crawler processes the blogs and adds all blog entries to Mediabase (including older entries that do not show up in the current RSS/Atom feed). • DBLP Importer: The records in the papers database were obtained from DBLP, a free and open bibliography mainly for computer science and its sub-disciplines. DBLP data is valuable since it includes information on conference series and journals, authors, and the papers published in the conferences and journals. Importing the data is done via an XML file that includes all DBLP records. The DBLP importer extracts these records and stores them in a relational database schema. In addition it is capable of extracting citation information on the imported papers using the CiteSeerX database. • Projects Crawler: In order to collect information about the running (or completed) TEL projects, we developed a crawler that automatically scrapes data from the project factsheets on the CORDIS website (for FP6 and FP7 projects), as well as from the eContentplus pages. All projects funded under TEL-related calls were scraped. The extracted information contains data like project description, start and end dates, project participants, funding and cost, project coordinator, etc. The data from these fact sheets were in a first step transformed to an XML-based format, which can be used by XML-processing applications like the project landscape story on the Learning Frontiers portal3. In a second step, the data was fed into a relational database schema to be used e.g. by the Drupal installation that is hosting the Learning Frontiers portal4. Analyses performed using the projects data obtained by this crawler are reported in Section 3.Databases. The TEL-Map Mediabase database consists of a collection of three relational databaseschemas, which are used to store and index TEL-related projects, papers, and social media artifacts(currently mainly blogs). • TEL Projects: This database includes details on TEL projects funded under FP6, FP7, and eContentplus programmes. It includes detailed information on the projects like start and end dates, cost, EC funding, coordinator, and consortium members. The TEL projects database is fed by the Projects Crawler. Details on the project data set are given in Section 3.1. • TEL Papers: This database includes information on TEL-related conference series, conference events, journals, authors, and papers published in the conferences, workshops and journals. It is fed by the DBLP Importer. Details on the papers data set are given in Section 4.1. • TEL (Social) Media: This database includes TEL-related blogs, including the blog entries, comments and analytical information like length, words occurrences, and word burst for certain entries. Details on the blogosphere data set are given in Section 5.1.Exporters. To enable analysis of the TEL-Map Mediabase data, the data are accessible either nativelyvia clients that connect to the database(s) using the database drivers, or via exporters. The exportersease the process of obtaining data for analysis by providing a set of predefined export formats.3 http://learningfrontiers.eu/?q=story/tel-project-landscape4 http://learningfrontiers.eu/?q=project_space –6–
  • 13. • CSV Data Exporter: Includes a set of scripts that export data contained in the databases into CSV format (CSV = comma separated values). These CSV files are supported by most data processing applications like Excel, R, SPSS, and so forth. • GraphML Exporter: Data can also be exported as graphs for social network analysis. The data is exported in the most common graph exchange format, i.e. the XML-based GraphML language. These GraphML files can be imported, visualized, and analyzed in graph visualization and analysis applications like yEd, Gephi, or the igraph library for R. For many other graph visualization and analysis software packages, there are conversion tools from and to GraphML. • Query Visualizer and Query Explorer: interacting with social network visualizations reaches its limits when it comes to specific queries that focus on selected aspects of the data set or the network graphs. To enable efficient end-user interaction with the data, we implemented a set of query visualization widgets. These widgets can be embedded on any web page (e.g. in iGoogle) and allow direct querying of the databases using SQL. The unique feature of these widgets is that they can be used to visualize the query results in different formats (e.g. table, pie chart, timeline, or graph) and that they can export the visualization of any given query as a widget. Additionally, CSV and GraphML export (see above) of query results is supported by the explorer widget. More details in Section 6.Applications. End-users will mostly interact with the data through applications like Excel, R, and theLearning Frontiers portal. While Figure 2 includes many example applications, the following list onlyfocuses on those that were developed for TEL-Map: • Learning Frontiers Portal: The Learning Frontiers portal is the single-access-point portal to results generated in the TEL-Map project. It includes two apps that can be used to contribute to content generation in the TEL Media database: The Mediabase Commander enables adding blogs directly to the database, and the Feed Aggregator is a Drupal module that we installed to allow users to collect relevant feeds. The feeds are ingested into the database at regular intervals by the Feed Importer. Note that Mediabase Commander (MBC) is also available as a Firefox add-on. • Query Widgets: We developed a set of widgets that can be used to (a) query the TEL-Map Mediabase databases using SQL, (b) to automatically visualize the query results in different formats, (c) export the query result in different formats, and (d) to export a query visualization as a self-contained widget that can be embedded into any web site.2.3 Analysis ApproachThis deliverable reports on first results of using social network analysis (SNA) and topic mining on thedata stored in the TEL-Map Mediabase. SNA contributes to the structural analysis of actors and theirrelationships and topic mining contributes to the semantic analysis of actors and relationshipsbetween actors. The combination of SNA and topic mining thus enables the structural-semanticanalysis of TEL sources.Social Network Analysis (SNA) is one of the work threads pursued in WP4 of TEL-Map to detectweak signals [23, 6] indicating future directions and insight into collaboration and communicationnetworks in different types of media and settings. SNA constitutes a rather new field of research andits application to digital libraries is very promising in terms of knowledge discovery [19, 20]. SNAdefines techniques used to compute metrics of different actors in a social network. These metricstypically represent the importance of actors within their network or neighborhood, e.g. their centrality,connectedness, etc. –7–
  • 14. To enable the calculation of SNA metrics for the data in TEL-Map Mediabase, the entities stored in theMediabase need to be modeled as a social network. A social network is modeled as a graph = ,with being the set of vertices (or nodes) and being the set of edges connecting the vertices with oneanother [2]. Any “actor” entity in the Mediabase can be modeled as a vertex, if it is connected to otheractors through any relationship of interest (modeled as edges) that can be obtained from theMediabase data. For instance, consider the following social network graphs: • TEL projects can be modeled as nodes and overlaps in the consortia of any two projects can be modeled as edges; • Organizations can be modeled as nodes, while projects in which organizations collaborated can be modeled as edges; • Persons can be modeled as nodes, while co-authorships on papers relevant to TEL can be modeled as edges; • Papers can be modeled as nodes, while citations between papers can be modeled as edges; • Blogs can be modeled as nodes, while links between the blogs’ entries can be modeled as edges.There are several different, yet complementary methods of gaining insight into the modeled socialnetwork graphs:(1) Visual interaction: The graph can be visualized using graph visualization software (like yEd,Graphviz, or Gephi). Similar to maps software like Google Maps, graph visualization software typicallyallows the user to zoom (vertical filter) into the visualization and to pan the visualized graph(horizontal filter). In addition these tools often offer graph layout algorithms, which can be used toalign the vertices in a predefined shape (e.g. circular, organic, hierarchical, etc.). Graph visualizationgenerally provides a holistic, condensed view on the overall network.(2) Data querying: Interacting with graph visualizations will typically spawn more specific questionsand exploratory tasks [5]. Some of these explorations cannot be performed using the visualizationalone, e.g. the number of shortest paths through the network that lead through a particular node. Suchresults can be obtained by enabling querying into the graph data. We developed a web-based toolkit forenabling this (see Section 6).(3) SNA Metrics: SNA allows the computation of different metrics for the graph, its nodes and itsedges. In the SNA reported in this deliverable, we mainly focus on the following metrics: • Avg. shortest path length: this is a graph metric that represents the average length of all shortest paths through the network. Over time this metric will grow quickly initially, but slows down or may even shrink in “mature” graphs. • Diameter: This represents the length of the longest shortest path through the network. In isolation this value will not be very informative; it is useful however for comparing network development over time (see e.g. Section 3.4.3). • Largest connected component: This measure represents the number (or the share) of nodes that are connected with each other in the largest sub-network of the graph. The lower this value, the higher the fragmentation in the network. • Density: This metric represents the ratio between the number of existing connections in the graph and the number of possible connections. The higher this value, the higher the connectedness of the nodes. One observation of interest is the development of density over –8–
  • 15. time, when new nodes join the graph, to see whether these new nodes inter-connect tightly with the existing ones. • Betweenness centrality: The betweenness centrality of a node represents the share of shortest paths through the network that pass through that node. The betweenness centrality is typically higher for nodes that connect (“bridge”) two or more sub-networks (also called “connected components”) in the network. For instance, an author who works in the intersection of artificial intelligence and technology-enhanced learning is likely to have a higher betweenness centrality in a co-authorship network than a person in the same network who only publishes with members of the core artificial intelligence community. • Degree centrality: The degree of a node is represented by the number of its direct ties with other nodes, i.e. edges coming in and leading out of that node. Typically this value is normalized into a value between 0 and 1 by dividing the degree of a node by the number of other nodes in the graph. This is the simplest centrality measure for network analysis • Closeness centrality: This measure is used to determine how close a node is to all other nodes that are reachable via edges. The closeness centrality is obtained by computing the mean length of these (shortest) paths. Nodes with a favorable closeness centrality are important nodes in the sense that they can easily reach other nodes for collaboration, information, or influence. • PageRank: This measure became widely known through Google’s use of it for ranking web sites by importance [17]. The PageRank of a node depends on the PageRank of nodes connected to it. So a node being connected to another node that is important makes the source node more important, too. With increasing distance between nodes this “diffusion” of importance to other nodes is gradually reduced by a damping factor. • Clustering coefficient: The clustering of a node (local clustering) measures how strongly the neighborhood of the node tends towards forming a clique, where every two nodes are connected by an edge. The clustering coefficient of the whole network is obtained by computing the average local clustering coefficient of its nodes. • Authorities and Hubs: authorities refer to nodes that represent authoritative sources of information in the network that are being pointed to by good hubs; a good hub is a node that point to many good authorities [12]. Thus there is a circular dependency between these two metrics.Topic Mining is an approach for discovering knowledge from text sources. Typically topics aredescribed by word distributions and sometimes also time distributions (cf. [24]). In the context of thisdeliverable we use a simplified approach to topic mining that mainly focuses on term stems and theirfrequency of appearance in the content entities stored in the Mediabase (e.g. blog text, paper abstracts,project descriptions) at a particular point in time or in a particular time window. For the firststructural-semantic analyses reported in this deliverable, we focused on a “big picture” approach tocomplementing social network metrics with content analysis for different sources and actors in theTEL-Map Mediabase. This includes: • For illustrating topic distribution in large sources we filtered the sources by identifying sources that are linked to key actors in the community (e.g. central organizations in projects, entries of central blogs). Following this, we present the core topics represented in these sources either through word clouds or through analysis of rising and falling frequency of topic occurrence in the sources. –9–
  • 16. • Building on the topic mining approach of selected TEL conferences in D4.1, we filtered the results for sources that were contributed by key authors in these conferences’ co-authorship networks and extracted weak signals there.2.4 Potential QuestionsThe combined results of SNA and topic mining can give rich insight into the available data and be usedto detect and explore potential signals (both strong and weak ones) in the data. The matrix in Table 1gives a brief overview of questions addressed by using SNA and topic mining on the different datasources in the TEL-Map Mediabase. Table 1: Uses of social network analysis and topic mining in the TEL-Map Mediabase. Social Network Analysis Topic MiningTEL Papers • Most central authors in TEL • Rising and falling terms in TEL paper • Most frequent collaborations on TEL abstracts and keywords papers • Topics addressed by most important • Most important TEL conferences and TEL authors/papers journals • Development characteristics of authorship networks in TEL conferences.TEL Projects • Consortium progression between • Topic distribution and shifts in TEL projects project foci over time • Partner collaborations across TEL • Funding and partners related to topics projects in TEL projects • Most central organizations in TEL projects • Most central TEL projects • Development of SNA metrics in project collaboration network over timeTEL Media • Citation network in TEL blogs • Topic bursts in TEL blogs over time • Most central web sources referenced in • Recently appearing topics TEL blogs • Topics with a rising frequency over the • Authorities and hubs in the TEL last years blogosphere • Co-occurrence of words/bursts in blog entriesIn the following, we elaborate more on the objectives and potential signals that can be identified bytackling the questions outlined in Table 1.TEL Papers Social Network Analysis and Topic Mining: • Most central authors in (European) TEL: identifies authors that have a central position in the co-authorship and citation network of TEL papers; these authors are likely to have authority regarding the focus of current TEL research and directions for future TEL research, which can be analyzed using topic mining. • Most frequent collaborations in TEL: Since TEL research is collaborative work, the identification of most important authors is complemented with collaboration frequency to identify strong ties between authors and communities. – 10 –
  • 17. • Most important TEL conferences and journals: identifying the most important outlets for publishing TEL research results will indicate venues where TEL key people meet for exchange and collaboration. Knowing the core TEL conferences will facilitate researchers in finding relevant collaborators. • Development characteristics of TEL conferences: identifies patterns of development of authorship networks, which will reveal several insightful network characteristics, e.g. whether the TEL community is a fragmented community, whether TEL conferences develop like conferences in other disciplines, etc. • Rising and falling terms in TEL papers: analysis of these terms will reveal topics and topic shifts in published TEL research. Of course, published TEL research is only a fraction of the research actually performed, and typically conference papers are up to one year behind the actual research work. For journal papers this lag is even worse, since journal papers often appear only 2-3 years after submission of the manuscript. • Topics addressed by prolific authors: Prolific or otherwise central authors identified in the co- authorship networks of different (sets of) publication outlets can be used for revealing topics that likely have impact on current and future work.TEL Projects Social Network Analysis and Topic Mining: • Consortium progression between projects and partner collaborations across TEL projects: this will identify organizational collaboration between different (consecutive and concurrent) projects that sustain beyond the lifetime of one project’s consortium. Strong partnership ties between organizations on the one hand, and new project funding for participants of a project may indicate fruitful and successful collaboration in that project and can thus be considered as an indicator of project success. • Most central TEL projects: analysis of consortium progression will also identify the most central projects in terms of having the largest consortium overlap with other projects, connecting different succeeding and preceding projects, and similar centrality measures. • Most central organizations in TEL projects: SNA can be used to identify the most central organizations in the TEL collaboration network in terms of number of connections, closeness to other organizations in the network, and connections between different organizational clusters or sub-networks. • Development of SNA metrics in project collaboration network over time: dynamic analysis of the collaboration network in projects over different funding calls or years will identify several characteristics of development patterns in the European TEL “market”, including development of collaboration network characteristics over time, impact of new projects on the collaboration network (e.g. introduction new organizations introduced by new projects) over time, and impact of new organizations on the creation of new collaboration ties between organizations. • Topic distribution in projects can be analyzed using the descriptions of projects or project clusters which were previously identified by SNA.TEL Media Social Network Analysis and Topic Mining: • Citation network in TEL blogs: identifies the most central blogs and blog entries in the TEL blogosphere and can be used in combination with topic mining on those blogs to identify trending, upcoming, and declining topics. – 11 –
  • 18. • Most authoritative web sources referenced in TEL blogs: in addition to citing sources in the blogosphere, bloggers reference all sorts of sources on the web; analyzing these can help to identify the most authoritative (type of) sources on the web for TEL bloggers (this will be tackled in upcoming WP4 work) • Topic bursts in TEL blogs over time: based on frequently occurring words in social media sources we are able to identify newly emerging terms and topics as well as topics with rising or falling frequency. This analysis is enhanced by filtering for those blogs that have a central position in the blogosphere.3 Analysis of the European TEL Project LandscapeThere currently exists no readily available, structured data set on TEL projects funded in recentprogrammes, with the exception of HTML factsheets offered on the web by the European Commissionas well as a load of project websites and deliverables produced by the project consortia. Turninginformation overload into an opportunity is the driving vision of visual analytics [7], and this sectionaims to achieve this vision in the context of TEL projects funded under FP6, FP7 and eContentplusprogrammes by applying SNA and information visualization methods on projects and collaborationswithin project consortia.3.1 Data SetData Model. The database used for the analyses in this paper was scraped from publicly availableproject information pages on CORDIS [4], i.e. the Community Research and Development InformationService offered by the European Commission, and other European Community project informationpages. The scraped data was captured according to the data model presented in Figure 3 and fed into arelational database. The data scraping was focused on TEL-related projects funded under FP6, FP7and eContentplus. ROLE participate Organization has_location N 1 ID N 1 NAME COUNTRY Project Geolocation ID ID CONTRACT_NO TITLE ACRONYM LATITUDE DESCRIPTION DATE_START DATE_END TYPE LONGITUDE PROGRAMME CALL COST FUNDING PRECISION WEBSITE_URL FACTSHEET_URL RCN Figure 3: Data model of TEL projects. – 12 –
  • 19. Information that was not available in CORDIS includes the geographical coordinates of projectmembers. These locations were semi-automatically obtained by invoking the Google Maps API andYahoo Maps API using the partner names and countries provided in the factsheets. Since some of thepartner names produced ambiguous geographical results, the geographical coordinates will not becorrect for some institutions. Also, the spelling of organization names and country names wasinconsistent in the project fact sheets in many cases; this was corrected manually (which still does notguarantee correctness). Additionally, organizational name changes are not accounted for. For instance,Giunti Labs S.R.L. was rebranded to eXact Learning Solutions in 2010. In the data set, these—and allorganizations with similar rebrandings—are represented as separate entities. Likewise, organizationalmergers are not accounted for, e.g. ATOS Origin and Siemens Learning, which merged in 2011.Selection of TEL Projects. Table 2 includes the details on the 77 TEL projects used in the followinganalyses, and a visual timeline of these projects can be found in Appendix A. Table 2: Overview of the 77 TEL Projects in the TEL-Map Mediabase. Programme Call # Projects (acronyms) Call 2005 4 CITER, JEM, MACE, MELT COSMOS, EdReNe, EUROGENE, eVip, Intergeo, KeyToNature, Call 2006 7 eContenplus5 Organic.Edunet Call 2007 3 ASPECT, iCOPER, EduTubePlus Call 2008 5 LiLa, Math-Bridge, mEducator, OpenScienceResources, OpenScout CONNECT, E-LEGI, ICLASS, KALEIDOSCOPE, LEACTIVEMATH, IST-2002-2.3.1.12 a 8 PROLEARN, TELCERT, UNFOLD APOSDLE, ARGUNAUT, ATGENTIVE, COOPER, ECIRCUS, ELEKTRA, FP6 IST-2004-2.4.10 b 14 I-MAESTRO, KP-LAB, L2C, LEAD, PALETTE, PROLIX, RE.MATH, TENCOMPETENCE ARISE, CALIBRATE, ELU, EMAPPS.COM, ICAMP, LOGOS, LT4EL, IST-2004-2.4.13 c 10 MGBL, UNITE, VEMUS ICT-2007.4.1 d 6 80DAYS, GRAPPLE, IDSPACE, LTFLL, MATURE, SCY COSPATIAL, DYNALEARN, INTELLEO, ROLE, STELLAR, TARGET, ICT-2007.4.3 d 7 FP7 XDELIA ALICE, ARISTOTELE, ECUTE, GALA, IMREAL, ITEC, METAFORA, ICT-2009.4.2 b 13 MIROR, MIRROR, NEXT-TELL, SIREN, TEL-MAP, TERENCE Total: 77a … Technology-enhanced learning and access to cultural heritageb … Technology-Enhanced Learningc … Strengthening the Integration of the ICT research effort in an Enlarged Europed … Digital libraries and technology-enhanced learningTopics and topic shifts. To give an indication of the topic focuses in these projects, Figure 4presents for FP6, FP7, and eContentplus a word cloud of the funded projects’ descriptions. It revealsan interesting difference between FP6 and FP7 projects. In FP6, we find many meta-concepts in thedescriptions like project, development, research, European, while descriptions of TEL projects in FP7expose some concrete research and learning related topics like adaptive, social, design, process,activities, and so forth. It could be argued that during FP6 the TEL landscape was gradually beginningto take form, while in FP7 the research agenda already included several hot topics.5 For each eContentplus call, only projects funded under the “Educational content” category were considered. The project SHARE-TEC (call 2007) was excluded from the data, since there was no official fact sheet available. – 13 –
  • 20. Looking at eContentplus in comparison to FP6 and FP7, there is a strong emphasis on content andmetadata, while still including heavy use of educational and learning as terms. Content is a termfound also in FP6 with some frequency, but it is missing in the top term list of FP7, probably showingthat the eContentplus participants and the European Commission were targeting different foci. FP6 FP7 eContentplus All TEL projects Figure 4: Word clouds of project descriptions.3.2 TEL Projects as Social NetworksA TEL project—like any other collaborative type of project—can be modeled as a social network wherea number of partner organizations collaborate under coordination of a coordinating organization. Asocial network is modeled as a graph = , with being the set of vertices (or nodes) and beingthe set of edges connecting the vertices with one another [2].Let be the set of projects, and let be the set of organizations involved in these projects. Functionrepresents the membership of any organization ∈ in the consortium of any project ∈ and isdefined as follows: , if ∈ participated or particiaptes in ∈ ∶ → , otherwise . The data model and these formal foundations enable powerful analyses and visualizations includingthe project network, the organizational partnership network, temporal relationships between projectconsortia, and the geographical mapping of organizations involved in projects. A selection of theseanalyses is presented in the following sub-sections, focusing on these objectives: • Visualizing and analyzing project consortium progression. By progression we mean partnerships within project consortia that sustain beyond one single project. Investigating these dynamics can be used to identify successful and strongly connected organizations between consortia of different projects. This objective is tackled in Section 3.3. • Visualizing and analyzing organizational collaborations within projects. Repeated collaboration in projects will create strong ties between organizations. Computing social network metrics for those connections will reveal the most important organizations currently involved in TEL research. This objective is dealt with in Section 3.4. – 14 –
  • 21. • Interactive visualization of geographical distribution of project consortia to complement the social network metric-based approaches with geographical map overlays, identifying hotspots in the European TEL landscape. This objective is dealt with in Section 3.5.3.3 Project Consortium ProgressionThe project consortium progression graph =( , contains projects and their relationships witheach other based on overlapping consortia. The graph will show projects as nodes and an edge betweentwo nodes if there is any organization that has participated in both projects, i.e. = , and = , ∶ , ∈ ∧ ≠ ∧∃ ∈ ∶ , ∧ , " . can be modeled as a directed graph, which exposes the temporal progression of project consortia.Each edge in this graph represents a temporal relationship between two connected projects: the edgepoints from the project which started earlier to the project which started later.3.3.1 FP7 ProjectsA visualization of for the 26 FP7 projects is shown in Figure 5. The size of each node in thisvisualization is proportional to the betweenness centrality [2] of that node, and the weight of the edgewas determined by the number of partners that overlap between two project consortia. Thebetweenness centrality measure is an effective means of exposing nodes that act as “bridges” betweenotherwise distant nodes (or groups of nodes) by computing for each node the share of all shortestpaths through the network that lead through the node. COSPATIAL TERENCE INTELLEO METAFORA MIROR TEL-MAP MIRROR ITEC 80DAYS GALA STELLAR NEXT-TELL LTFLL DYNALEARN GRAPPLE XDELIA IMREAL ROLE ECUTE TARGET IDSPACE MATURE SIREN SCY ALICE ARISTOTELE Figure 5: FP7 TEL projects graph visualization. – 15 –
  • 22. The visualization of project connections in Figure 5 exposes one node that could be labeled as thecurrent “epicenter” of TEL projects in FP7. This node represents GALA, the network of excellence onserious games [29]. There are two main factors why this project is such a strong connector: 1. the consortium is extraordinarily large with 31 participating organizations6, and 2. the project has started only recently in October 2010, following the most recently closed TEL call in FP7 (see the projects timeline in Appendix A) .Obviously, a project which starts later than other projects has a higher chance of having organizationsin its consortium which were already part of previous project consortia. Other projects that carried onmultiple consortium members to the GALA consortium are TARGET, GRAPPLE, and STELLAR.Another strong, currently running project is ROLE, which is a harbor for project consortiumpartnerships from previous projects, and also has overlaps with succeeding project consortia. If we hadcomputed the betweenness centrality of the projects taking into account the direction of the edges,ROLE, STELLAR and MIRROR would be the most betweenness-central projects. Such a computationwould, however, statistically favor projects that have started in the middle between the begin date ofFP7 and the current date, since in this time window projects are more likely to have outgoingconsortium connections in addition the incoming ones.3.3.2 All TEL Projects – FP6, FP7, and eContentplusA graph of all TEL projects funded in FP6, FP7, and eContentplus is given in Figure 6. The graphincludes all 77 projects and a total of 712 connections between those projects. KALEIDOSCOPE is byfar the largest node, which can be attributed to the fact that this project had an extremely largeconsortium of 83 partner organizations, which is more than five times the typical consortium size. It isalso evident in this visualization that in addition to strong ties between FP6 and FP7 projects, theeContentplus projects have very strong connections to both FP6 and FP7. This can probably beexplained by the fact that eContentplus filled a “funding gap” in 2007 when FP6 funding was stallingfollowing the last FP6 projects launched in 2006, while FP7 funding was kicked off with the first TELprojects starting in 2008. In fact, in 2007 only eContentplus projects were launched with EC fundingin our data set (compare also the dynamic network analysis in Section 3.4.3, in particular Figure 13d).This kind of gap filling by eContentplus, where a large share of organizations funded under FP6 andFP7 engaged in e-content focused R&D projects, could be interpreted as evidence for a “researchfollows money” attitude of researchers involved in TEL. That is, if there had not been funding fromeContentplus, organizations would likely have looked for funding opportunities in TEL-relatedprogrammes with different focus between 2006 and 2008.A table with all projects displayed in Figure 6 along with their SNA metrics (and ranks) is given inAppendix B.6 See http://learningfrontiers.eu/?q=story/tel-project-landscape&proj=GALA and http://www.learningfrontiers.eu/?q=tel_project/GALA – 16 –
  • 23. FP7 ALICE SIREN MATURE ARISTOTELE NEXT-TELL IMREAL ECUTE TARGET MIROR 80DAYS COSPATIAL GALA METAFORA INTELLEO MIRROR DYNALEARN GRAPPLEIDSPACE XDELIA SCY ROLE TERENCE STELLAR ITEC TEL-MAP eContentplus LTFLL LiLa eViP FP6 I-MAESTRO mEducator ECIRCUS MACE EdReNe KeyToNature APOSDLE UNFOLD OpenScout ASPECT COOPER JEM Math-Bridge TELCERT RE.MATH iCOPER EUROGENE PROLEARN MELT KALEIDOSCOPE CONNECT EduTubePlus Intergeo COSMOS MGBL ARGUNAUTARISE ELEKTRA PROLIX Organic.Edunet TENCOMPETENCE OpenScienceResources CITER E-LEGI UNITE LEACTIVEMATH PALETTE LT4EL ICLASS VEMUS ICAMP ELU KP-LAB L2C LEAD LOGOS CALIBRATE ATGENTIVE EMAPPS.COM Figure 6: Project consortium progression between FP6, FP7, and eContentplus projects.3.3.3 Identifying Project ClustersThe project consortium progression graph was subjected to cluster analysis using the Louvainmethod described in [1]. This method first divides the nodes into local clusters, and then collapseseach clusters’ nodes into a single node. These two steps are applied repeatedly until the final set ofclusters is reached.There are 6 resulting clusters of projects as listed in Table 3: • Cluster C0 includes mostly FP7 projects, with some FP6 and eContentplus projects, which focus on learning, development, research and technology as evident form the word cloud extract from these projects’ descriptions. • Cluster C1 exposes the strongest thematic focus on learning (and education) of all clusters; there are no other terms that really stand out. The cluster includes a mix of all funding programmes. • Cluster C2 shows a strong topical emphasis on content, collaboration, knowledge and support; this cluster is well represented by projects from all funding schemes. – 17 –
  • 24. • Cluster C3 includes projects related development, content, competence, tools and testing. In this cluster there is the smallest gap between frequency of occurrence of learning and other terms. • Cluster C4 has a strong focus on science and education, and also school is a term that stands out. • Cluster C5 emphasizes mostly on content, development and technology. It has the strongest focus on content of all clusters; yet it includes not only eContentplus projects.It is evident that eContentplus projects are spread over all clusters, indicating that this fundingprogramme (a) did not disrupt collaboration structures in TEL and (b) was definitely relevant for atopic focus on educational content. Moreover, projects of all funding schemes are represented in allclusters, indicating a coherent research agenda since the first FP6 projects. Table 3: TEL project clusters in FP6, FP7, and eContentplus (ECP) and the word clouds of their project descriptions. ALICE [FP7], APOSDLE [FP6], COSPATIAL [FP7], ECIRCUS [FP6], ECUTE [FP7], eViP [ECP], GALA [FP7], I-MAESTRO [FP6],C0 IMREAL [FP7], KALEIDOSCOPE [FP6], MATURE [FP7], MIRROR [FP7], NEXT- TELL [FP7], SIREN [FP7], TARGET [FP7] 80DAYS [FP7], CITER [ECP], DYNALEARN [FP7], EduTubePlus [ECP], ELEKTRA [FP6], ICLASS [FP6], Intergeo [ECP],C1 LEACTIVEMATH [FP6], LiLa [ECP], LOGOS [FP6], METAFORA [FP7], MIROR [FP7], PALETTE [FP6], PROLEARN [FP6], PROLIX [FP6], RE.MATH [FP6] ATGENTIVE [FP6], E-LEGI [FP6], EUROGENE [ECP], ICAMP [FP6], iCOPER [ECP], INTELLEO [FP7], JEM [ECP], KP-C2 LAB [FP6], L2C [FP6], LEAD [FP6], LT4EL [FP6], LTFLL [FP7], mEducator [ECP], OpenScout [ECP], ROLE [FP7], STELLAR [FP7], TEL-MAP [FP7], XDELIA [FP7] COOPER [FP6], GRAPPLE [FP7], IDSPACE [FP7], MACE [ECP], Math-BridgeC3 [ECP], TELCERT [FP6], TENCOMPETENCE [FP6], UNFOLD [FP6] ARGUNAUT [FP6], ARISE [FP6], ARISTOTELE [FP7], CONNECT [FP6],C4 COSMOS [ECP], OpenScienceResources [ECP], Organic.Edunet [ECP], SCY [FP7], UNITE [FP6], VEMUS [FP6] ASPECT [ECP], CALIBRATE [FP6], EdReNe [ECP], ELU [FP6], EMAPPS.COMC5 [FP6], ITEC [FP7], KeyToNature [ECP], MELT [ECP], MGBL [FP6], TERENCE [FP7] – 18 –
  • 25. 3.4 Organizational CollaborationsIn addition to the project consortium progression network presented in the previous section, TELprojects can be viewed from another angle: the organizational collaboration graph $ = ( $ , $contains organizations and their collaborations in the project consortia. This graph showsorganizations as nodes and an edge between two nodes if there is any project where both organizationshave participated in, i.e. $ = and $ = , ∶ , ∈ $ ∧ ≠ ∧∃ ∈ ∶ , ∧ , " .3.4.1 Collaborations in FP7 projectsIn the collaboration graph for FP7 TEL projects, $ & , the number of participating organizations | $ & | % %is 211, and the number of collaborations | $ % &| between those organizations amount to 1,983. The % &visualization of $ is depicted in Figure 7. The size of each node was determined by the betweennesscentrality of the node, while edge weight was determined by the number of projects in which twoorganizations collaborate or had collaborated. The node arrangement was computed using the yEdgraph editor’s [28] organic layout algorithm, which tends toward a symmetric and clustereddistribution of nodes [27]. UNIVERSIDAD POLITECNICA DE MADRID F UNDAÇÃO UNIVERSIDADE DE BRASILIA UNIVERSIT AET F UER BODENKULTUR WIEN CENTRAL L ABORATORY OF GENERAL ECOL OGY - ZENTRALNA LABORATORIYA PO OBSCHTA EKOLOGIYA T EL AVIV UNIVERSITY UNIVERSITEIT VAN AMST ERDAM UNIVERSITY OF HULL NAT IONAL UNIVERSITY CORPORATION, KYOTO UNIVERSITY NATIONAL TECHNICAL UNIVERSITY OF ATHENS EXT RIM MINTIA SOLOUSIONS ET AIREIA PLIROFORIKIS KAI TILEPIKOINONION ETAIREIA PERIORISMENIS EFTHINIS GAKO HOJIN SEIKEI GAKUEN T HE REGENTS OF THE UNIVERSIT Y OF CALIFORNIA UNIVERSITAET AUGSBURG IT UNIVERSITY OF COPENHAGEN WAGENINGEN UNIVERSITEIT UNIVERSITY OF BATH MAKASH - ADVANCING CMC APPLICATIONS IN EDUCAT ION, CULTURE AND SCIENCE EDUCA.CH INST IT UT SUISSSE DES MEDIAS POUR L A FORMATION ET LA CULT URE UNIVERSITE PIERRE MENDES FRANCE JACOBS UNIVERSITY BREMEN GGMBH UNIVERSITY OF SUNDERLAND SVIETIMO INFORMACINIU TECHNOLOGIJU CENT RAS SPACE APPLICATIONS SERVICES NV KUNGLIGA TEKNISKA HOEGSKOLAN FACULT ES UNIVERSITAIRES NOTRE-DAME DE LA PAIX ASBL BUNDESMINISTERIUM FÜR UNTERRICHT, KUNST UND KUL TUR INST IT UTE F OR PARAL LEL PROCESSING OF THE BULGARIAN ACADEMY OF SCIENCES JYVASKYLAN YLIOPIST O UNIVERSITATEA POLITEHNICA DIN BUCUREST I BRUNEL UNIVERSITY HOGSKOLEN I OSLO SMART TECHNOL OGIES ( GERMANY) GMBH T HE UNIVERSITY OF MANCHESTER NATIONAL MINISTRY OF EDUCATION MORPHEUS SOFTWARE VOF GIUNT I L ABS S.R.L. LANDESINIT IAT IVE NEUE KOMMUNIKATIONSWEGE MECKLENBURG- VORPOMMERN E.V. TUNG UNIVERSITAT HILDESHEIM STIF SERIOUS GAMES INTERACT IVE EUN PARTNERSHIP AISBL ELFA S.R.O. INESC ID - INSTITUTO DE ENGENHARIA DE SISTEMAS E COMPUT ADORES, INVESTIGACAO E DESENVOLVIMENTO EM LISBOA TIIGRIHUPPE SIHTASUTUS THE MANCHEST ER METROPOLITAN UNIVERSITY AURUS KENNIS- EN TRAININGSSYST EMEN B.V. HERIOT -WATT UNIVERSITY THE UNIVERSITY OF BOLTON MINIST ERIO DA EDUCACAO EDUCATIO PUBLIC SERVICES NON-PROFIT L LC UNIVERSITY OF PIRAEUS RESEARCH CENTER UNIVERSIDAD COMPL UTENSE DE MADRID RHEINISCH-WESTFAELISCHE TECHNISCHE HOCHSCHULE AACHEN UNI-C, DANMARKS EDB-CENTER F OR FORSKNING OG UDDANNELSE FEST O LERNZENTRUM SAAR GMBH BIT MEDIA E-L EARNING SOL UTION GMBH AND CO KG UNIVERSITY OF T HE WEST OF SCOT LAND PLAYGEN LTD CYNT ELIX CORPORATION BV PROMETHEAN L IMITED ICODEON LIMIT ED EBERHARD KARL S UNIVERSITAET TUEBINGEN UNIVERSITEIT UTRECHT KNOWLEDGE MARKETS CONSULTING GMBH WIRT SCHAFT SUNIVERSITAET WIEN FUNDACION ESADE AALTO-KORKEAKOULUSAATIO UNIVERSIDAD DE VIGO AALBORG UNIVERSITET TT Y- SAATIO SHANGHAI JIAO TONG UNIVERSITY INDIRE ISTITUTO NAZIONALE DI DOCUMENT AZIONE PER LINNOVAZIONE E LA RICERCA EDUCATIVA UNIVERSIT Y OF CYPRUS MINISTERUL APARARII NAT IONALE THE BRITISH INSTITUTE FOR L EARNINGAND DEVEL OPMENT LBG EDUBIT VZ W T HE GOVERNING COUNCIL OF THE UNIVERSIT Y OF TORONTO UNIVERSITE PAUL SABAT IER TOULOUSE III INSTIT UTO DE EDUCAÇÃO DA UNIVERSIDADE DE LISBOA UNIVERSITETET I OSLO BIBA - BREMER INST ITUT FUER PRODUKTION UND LOGIST IK GMBH FUTURELAB EDUCAT ION U&I LEARNING NV ORT F RANCE NATO UNDERSEA RESEARCH CENT RE ZENTRUM F UER SOZIAL E INNOVATION SENTER FOR IKT I UTDANNINGEN POLITECNICO DI MIL ANO UNIVERSIT AET KOBLENZ-LANDAU CENT RE NATIONAL DE DOCUMENTATION PEDAGOGIQUE CONSIGLIO NAZ IONALE DEL LE RICERCHE THE OPEN UNIVERSITY F RAUNHOFER-GESEL LSCHAF T ZUR FOERDERUNG DER ANGEWANDTEN FORSCHUNG E.V UNIVERSITA DEGLI STUDI DI GENOVA KAT HOLIEKE UNIVERSITEIT LEUVEN TARTU ULIKOOL OPEN UNIVERSITEIT NEDERLAND CENTRE EUROPEEN DEDUCATION PERMANENTE VIRTECH L TD LEAN ENTERPRISE INSTITUTE POLSKA SPOLKA Z OGRANICZONA ODPOWIEDZIALNOSC IAA UNIVERSITET UPPSAL STICHTING TECHNASIUM ENOVAT E AS UNIVERSITY OF LEICESTER CYNTEL IX CORPORAT ION LIMITED COVENTRY UNIVERSITY EIDGENOESSISCHE TECHNISCHE HOCHSCHULE Z UERICH VOL KSWAGEN AG UNIVERSIT AET GRAZ INSTIT UT FÜR ANGEWANDT E SYSTEMTECHNIK BREMEN GMBH UNIVERSITY COLL EGE LONDON ALMA MATER STUDIORUM- UNIVERSITA DI BOLOGNA NOKIA OYJ T HE UNIVERSITY OF NOT TINGHAM TAKOMAT JOHNE, SCHNATMANN, SCHWARZ GBR FACULTY OF ORGANIZATIONAL SCIENCES, UNIVERSITY OF BEL GRADE UNIVERSIT ETET I BERGEN T ECHNISCHE UNIVERSITEIT DEL FT UNIVERSITEIT TWENT E ECOL E POLYTECHNIQUE FEDERALE DE LAUSANNE EESTI OPETAJATE LIITDE PRAKTIJK, NAT UURWETENSCHAPPELIJK ONDERWIJS V.O.F. INI DOO IMC INFORMATION MULT IMEDIA COMMUNICATION AG INOVACIJSKO-RAZ VOJNI INSTITUT UNIVERZ E V LJUBLJANI AKTIENGESELLSCHAFT OESTERREICH SIEMENS ATHABASCA UNIVERSIT Y ATOS ORIGIN SOCIEDAD ANONIMA ESPANOLA SONY FRANCE S.A. T ALLINNA ULIKOOL UNIVERSITE JOSEPH FOURIER GRENOBLE 1 ST IF TELSEN SINT EF T ECHNOLOGY & SOCIET Y ERASMUS UNIVERSIT EIT ROTTERDAM UNIVERSITAET DUISBURG-ESSEN NORGES T EKNISK-NATURVITENSKAPELIGE UNIVERSITET NTNU F RIEDRICH-AL EXANDER UNIVERSITAET ERLANGEN - NUERNBERG UNIVERSITY OF BRISTOL Z ENTRUM FUER GRAPHISCHE DATENVERARBEIT UNG E.V. GOETEBORGS UNIVERSITET COMPEDIA SOFTWARE & HARDWARE DEVELOPMENT LTD MOMA SPA TECHNISCHE UNIVERSITAET GRAZ THE CHANCELLOR, MAST ERS AND SCHOLARS OF THE UNIVERSIT Y OF CAMBRIDGE SAXO BANK AS EAR COMMUNICATION ASSOCIATES LIMITED - CCA CL SCIENTER SCRL FUNDACIO PER A LA UNIVERSITAT OBERT A DE CATALUNYA T HE UNIVERSITY OF EXETER AMIS DRUZBA ZA TELEKOMUNIKACIJE D.O.O. VRIJE UNIVERSITEIT BRUSSEL TECHNISCHE UNIVERSITEIT EINDHOVEN BL EKINGE TEKNISKA HOEGSKOLA UNIVERSIT Y OF SUSSEX CENT RO DI RICERCA IN MATEMAT ICA PURA ED APPLICATA - CONSORZ IO GOTTF RIED WIL HEL M LEIBNIZ UNIVERSITAET HANNOVER NATIONAL AND KAPODISTRIAN UNIVERSITY OF AT HENS ALBERT-L UDWIGS-UNIVERSITAET FREIBURG TESTALUNA SRL L UDWIG- MAXIMILIANS- UNIVERSITAET MUENCHEN UNIVERSITA DEGL I STUDI DI PADOVA HEALT HWARE SPA - PHI EXACT LEARNING SOLUTIONS SPA UNIVERSITE CATHOL IQUE DE LOUVAIN LIBERA UNIVERSITA DI BOL ZANO KOMPETENZZENTRUM FUER WISSENSBASIERTE ANWENDUNGEN UND SYSTEME F ORSCHUNGS - UND ENTWICKLUNGS GMBH UNIVERSITY OF L EEDS UNIVERSITAET PADERBORN EMPOWERTHEUSER LTD FONDAZIONE BRUNO KESSLER THE UNIVERSITY OF WARWICK UNIVERSITA DEL LA SVIZZERA ITALIANA CENTRE INT ERNACIONAL DE METODES NUMERICS EN ENGINYERIA MEDIEN IN DER BILDUNG ENGINEERING - INGEGNERIA INFORMATICA SPA UNIVERSITAET INNSBRUCK BOC INFORMATION TECHNOLOGIES CONSULTING SP. Z .O.O. F ORSCHUNGSZENT RUM INFORMATIK AN DER UNIVERSITAET KARLSRUHE REGOLA SRL UNIVERSITA DEGLI STUDI DI MIL ANO IMAGINARY SRL THE PROVOST FELLOWS AND SCHOLARS OF THE COLLEGE OF T HE HOLY AND UNDIVIDED TRINITY OF QUEEN EL IZABET H NEAR DUBLIN UNIVERSITA DEGLI STUDI DI VERONA INFOMAN AG NEUROLOGISCHE KLINIK GMBH BAD NEUSTADT UNIVERSITA DEGLI STUDI DELLAQUILA SIVECO ROMANIA SA SAP AG BAR ILAN UNIVERSITY RUHR-UNIVERSITAET BOCHUM DEUTSCHES FORSCHUNGSZENTRUM FUER KUENSTLICHE INTEL LIGENZ GMBH THE CITY UNIVERSITY TRACOIN QUALITY BV UNIVERSIT Y OF HAIF A PONTYDYSGU LTD BOC ASSET MANAGEMENT GMBH UNIVERSIDAD DE SALAMANCA F ACHHOCHSCHULE NORDWESTSCHWEIZ VEREIN OF FENES L ERNEN SOLUCIONES INTEGRAL ES DE FORMACION Y GESTION STRUCTURALIA, S.A T HE UNIVERSITY OF BIRMINGHAM MOHOL Y-NAGY MUVESZETI EGYETEM THE HEBREW UNIVERSITY OF JERUSALEM REGISTERED NURSING HOME ASSOCIAT ION LIMITED UNI RESEARCH AS AMNIN D.O.O CENTR Z A Z NANST VENO VIZUALIZACIJO INSTITUTE OF EDUCATION, UNIVERSITY OF LONDON COPENHAGEN BUSINESS SCHOOL BRITISH T ELECOMMUNICATIONS PUBL IC L IMITED COMPANY* L ONDON METROPOLITAN UNIVERSITY KAT HOLISCHE UNIVERSITAT EICHST ATT-INGOLSTADT MT O PSYCHOLOGISCHE FORSCHUNG UND BERATUNG GMBH JOANNEUM RESEARCH F ORSCHUNGSGESELLSCHAFT MBH Figure 7: Visualization of the FP7 collaboration graph. – 19 –
  • 26. % & A cut-out of the center region of the FP7 collaboration graph $ with legible node labels is shown in Figure 8. INESC ID - INSTITUTO DE ENGENHARIA DE SISTEMAS E COMPUTADORES, INVESTIGACAO E DESENVOLVIMENTO EM LISBOA TIIGRIHUPPE SIHTASUTUS- EN TRAININGSSYSTEMEN B.V. HERIOT-WATT UNIVERSITY THE UNIVERSITY OF BOLTON UNIVERSIDAD COMPLUTENSE DE MADRID RHEINISCH-WESTFAELISCHE TECHNISCHE HOCHSCHULE AACHEN FESTO LERNZENTRUM SAAR GMBH UNIVERSITY OF THE WEST OF SCOTLAND PLAYGEN LTD CYNTELIX CORPORATION BV UNIVERSITEIT UTRECHT KNOWLEDGE MARKETS CONSULTING WIRTSCHAFTSUNIVERSITAET WIEN FUNDACION ESADE AALTO-KORKEAKOULUSAATIO AALBORG UNIVERSITET TTY-SAATIO SHANGHAI JIAO TONG UNIVERSITY MINISTERUL APARARII NATIONALE THE BRITISH INSTITUTE FOR LEARNINGAND DEVELOPMENT LBG UNIVERSITE PAUL SABATIER TOULOUSE III BIBA - BREMER INSTITUT FUER PRODUKTION UND LOGISTIK GMBH U&I LEARNING NV ORT FRANCE NATO UNDERSEA RESEARCH CENTRE ZENTRUM FUER SOZIALE INNOVATION POLITECNICO DI MILANOVERSITAET KOBLENZ-LANDAU CONSIGLIO NAZIONALE DELLE RICERCHE THE OPEN UNIVERSITY UNIVERSITA DEGLI STUDI DI GENOVA KATHOLIEKE UNIVERSITEIT LEUVEN OPEN UNIVERSITEIT NEDERLAND CENTRE EUROPEEN DEDUCATION PERMANENTE VIRTECH LTD UPPSALA UNIVERSITET UNIVERSITY OF LEICESTER CYNTELIX CORPORATION LIMITED COVENTRY UNIVERSITY EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZUERICH UNIVERSITAET GRAZ UNIVERSITY COLLEGE LONDON NOKIA OYJ THE UNIVERSITY OF NOTTINGHAM TECHNISCHE UNIVERSITEIT DELFT ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE IMC INFORMATION MULTIMEDIA COMMUNICATION AG SIEMENS AKTIENGESELLSCHAFT OESTERREICH ATOS ORIGIN SOCIEDAD ANONIMA ESPANOLA STIFTELSEN SINTEF TECHNOLOGY & SOCIETY NORGES TEKNISK-NATURVITENSKAPELIGE UNIVERSITET NTNU UNIVERSITY OF BRISTOL TECHNISCHE UNIVERSITAET GRAZ THE CHANCELLOR, MASTERS AND SCHOLARS OF THE UNIVERSITY OF CAMBRIDG CLEAR COMMUNICATION ASSOCIATES LIMITED - CCA SCIENTER SCRL VRIJE UNIVERSITEIT BRUSSEL TECHNISCHE UNIVERSITEIT EINDHOVEN GOTTFRIED WILHELM LEIBNIZ UNIVERSITAET HANNOVER Figure 8: Center region cut-out of the FP7 collaboration graph. The visualizations show one core sub-network in the center, featuring organizations that have strong ties due to several shared projects. It also shows a strong sub-network towards the top-right of the center. The other sub-networks in the periphery of the visualization tend to expose consortia of projects whose members are not involved in multiple projects. The bonds between those sub-networks are established by organizations that are involved in collaboration clusters in both the periphery and the center of the network. These nodes represent connectors in the network. The strongest ties, of course, are built between organizations that collaborate in many different projects and/or that collaborate in projects with large consortia. The five top-ranked among those in FP7 are: 1. Technische Universität Graz, Austria (82 distinct connections in 7 projects) – 20 –
  • 27. 2. Open Universiteit Nederland, Netherlands (67 / 5) 3. Aalto-Korkeakoulusaatio, Finland (66 / 3) 4. Katholieke Universiteit Leuven, Belgium (63 / 4). 5. ATOS Origin Sociedad Anonima Espanola, Spain (59 / 4)This is actually a “multi-cultural” top-five list since the top-ranked organization is represented by apsychology department at a university of technology, the second-ranked is an open university, thethird-ranked is a multidisciplinary research unit, the fourth-ranked is a hypermedia and databasesdepartment at a Catholic university, and the fifth-ranked is a research and IT services company. This isa confirmative signal that TEL is an interdisciplinary research area which needs strong partners fromdifferent ends of the spectrum, both in terms of research area and organizational background.A visualization like this can, among other purposes, be used to identify the most prolific collaboratorsas an entry point to more detailed investigation. For instance, Technische Universität Graz, althoughbeing a university of technology, is primarily involved as a provider of the psychological expertise inthe projects. The word cloud of all 26 project descriptions in Figure 9—which displays the 20 wordstems with the highest frequency in the descriptions—clearly shows that TEL projects have a strongfocus on the human(istic) aspects of TEL, e.g. learn, knowledg[e], design, adapt[ation],collabor[ation], and so forth. A strong and well-connected department of psychology, like that ofTechnische Universität Graz, has an excellent standing in such a landscape; this indicates that partnersproviding expertise in non-technological, “cross-cutting” themes have great opportunities in getting apiece of the TEL funding cake.Figure 9: Word cloud of the 20 word stems with highest frequency in the FP7 project descriptions3.4.2 Collaborations in all TEL Projects: FP6, FP7, and eContentplusA collaboration graph for all TEL projects in FP6, FP7, and eContentplus programmes, $ , is given inFigure 10. It includes | $ | = 604 partner organizations and | $ | = 9,330 distinct pairs of partnercollaboration. Apart from the core network, there are no strongly connected sub-networks, whichmeans that in every project there exists one partner who is involved in at least one other project. – 21 –
  • 28. Figure 10: Partner collaborations spanning FP6, FP7, and eContentplus projects.We calculated SNA metrics and funding statistics for each participant in $ ; the resulting table of thetop 30 organizations is given in Table 4. The table is ordered by PageRank, a metric that not only takesinto account the number of edges of each node, but also the “importance” of the neighboring adjacentnodes. This means, an organization’s importance depends on the number of collaborations with otherorganizations and on the importance of the organization’s collaborators.The ranking clearly shows that there is a strong relationship between the most “profane” ranking (i.e.funding) and most of the other network metrics. One notable exception is the local clusteringcoefficient, which has an apparent negative correlation with all other metrics. We recall that clusteringof a node in a network refers to the connectedness of the node’s neighborhood. A organizationalcollaboration network like $ has some noteworthy properties in this respect: an organization which isinvolved in only one project will have a high clustering coefficient, since the neighborhood of thatorganization is strongly connected through the project. So this node will have low scores in centralitymetrics and a higher score on clustering. On the other hand, an organization that is involved in severalprojects will likely connect several sub-networks made up of organizations involved in fewer projects.The organization will thus have a lower clustering coefficient, but likely higher values for the centralitymetrics, like betweenness centrality. To illustrate this effect, Figure 13a plots the local clusteringcoefficient against the PageRank for each organization. The plot reveals and interesting characteristicof the collaboration network, namely that there are two quite clearly separated sets of organizations inthe plot. One set of organizations manages to have a significantly higher PageRank than the other set – 22 –
  • 29. at a similar level of local clustering. This separation is even more evident when plotting the degree(number of connections) against the local clustering coefficient (see Figure 13b). Table 4: Top 30 organizations involved in TEL projects by PageRank. The numbers in square brackets next to the values represent the rank of that value among all 604 organizations. Organization PR ▼ BC LC DC CC Funding*THE OPEN UNIVERSITY, UNITED KINGDOM .0125 [1] .1185 [1] .2151 [601] 219 [1] .6012 [1] 3.55 [3]KATHOLIEKE UNIVERSITEIT LEUVEN, BELGIUM .0090 [2] .0752 [2] .1716 [604] 148 [3] .5462 [5] 2.56 [6]OPEN UNIVERSITEIT NEDERLAND, NETHERLANDS .0086 [3] .0414 [6] .2161 [600] 133 [7] .5442 [6] 3.45 [4]JYVASKYLAN YLIOPISTO, FINLAND .0080 [4] .0667 [3] .3170 [588] 170 [2] .5657 [2] 1.26 [39]FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG DER .0068 [5] .0529 [4] .1833 [603] 111 [22] .5294 [18] 3.40 [5]ANGEWANDTEN FORSCHUNG E.V, GERMANYDEUTSCHES FORSCHUNGSZENTRUM FUER .0066 [6] .0390 [7] .1916 [602] 106 [27] .5280 [21] 3.68 [1]KUENSTLICHE INTELLIGENZ GMBH, GERMANYATOS ORIGIN SOCIEDAD ANONIMA ESPANOLA, SPAIN .0064 [7] .0236 [15] .4316 [565] 142 [5] .5482 [4] 1.33 [33]UNIVERSITAET GRAZ, AUSTRIA .0064 [8] .0230 [18] .4016 [573] 148 [3] .5552 [3] 2.03 [10]UNIVERSITEIT UTRECHT, NETHERLANDS .0061 [9] .0203 [23] .4323 [564] 139 [6] .5336 [11] 1.62 [19]INESC ID - INSTITUTO DE ENGENHARIA DE SISTEMAS E .0061 [10] .0368 [8] .4741 [552] 130 [8] .5280 [21] 1.68 [16]COMPUTADORES, INVESTIGACAO EDESENVOLVIMENTO EM LISBOA, PORTUGALTHE UNIVERSITY OF WARWICK, UNITED KINGDOM .0058 [11] .0333 [9] .4754 [551] 129 [10] .5374 [10] 1.68 [17]UNIVERSITY OF CYPRUS, CYPRUS .0057 [12] .0176 [26] .4668 [554] 130 [8] .5394 [7] 1.48 [25]THE UNIVERSITY OF NOTTINGHAM, UNITED KINGDOM .0057 [13] .0198 [24] .4902 [542] 129 [10] .5327 [13] 1.52 [22]IMC INFORMATION MULTIMEDIA COMMUNICATION AG, .0056 [14] .0102 [46] .3124 [590] 86 [57] .5071 [47] 2.35 [8]GERMANYUNIVERSITAET DUISBURG-ESSEN, GERMANY .0056 [15] .0258 [12] .4773 [547] 125 [12] .5285 [20] 1.82 [14]GOTTFRIED WILHELM LEIBNIZ UNIVERSITAET .0054 [16] .0186 [25] .2860 [592] 87 [54] .5115 [44] 2.41 [7]HANNOVER, GERMANYECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE, .0053 [17] .0111 [41] .5647 [506] 118 [18] .5322 [15] 1.66 [18]SWITZERLANDEUN PARTNERSHIP AISBL, BELGIUM .0052 [18] .0231 [16] .2617 [595] 87 [54] .4863 [97] 1.83 [13]KUNGLIGA TEKNISKA HOEGSKOLAN, SWEDEN .0052 [19] .0242 [14] .5335 [527] 121 [14] .5336 [11] .98 [69]THE PROVOST FELLOWS AND SCHOLARS OF THE .0051 [20] .0211 [20] .5344 [526] 119 [16] .5379 [9] 2.31 [9]COLLEGE OF THE HOLY AND UNDIVIDED TRINITY OFQUEEN ELIZABETH NEAR DUBLIN, IRELANDTECHNISCHE UNIVERSITAET GRAZ, AUSTRIA .0051 [21] .0257 [13] .2577 [598] 90 [50] .4971 [60] 3.56 [2]GIUNTI LABS S.R.L., ITALY .0051 [22] .0207 [21] .2644 [594] 82 [62] .5054 [49] 1.95 [12]CONSIGLIO NAZIONALE DELLE RICERCHE, ITALY .0050 [23] .0074 [58] .5687 [505] 119 [16] .5303 [17] 1.08 [60]RESEARCH ACADEMIC COMPUTER TECHNOLOGY .0050 [24] .0263 [11] .5512 [517] 114 [20] .5132 [43] 1.26 [40]INSTITUTE, GREECEEOTVOS LORAND TUDOMANYEGYETEM, HUNGARY .0049 [25] .0230 [17] .4986 [538] 124 [13] .5394 [7] .98 [68]UNIVERSITETET I OSLO, NORWAY .0049 [26] .0175 [27] .5171 [533] 121 [14] .5294 [18] 1.32 [34]UNIVERZA V LJUBLJANI, SLOVENIA .0046 [27] .0212 [19] .2468 [599] 78 [93] .4778 [102] 1.16 [50]ELLINOGERMANIKI AGOGI SCHOLI PANAGEA SAVVA AE, .0046 [28] .0118 [39] .2587 [597] 69 [96] .4483 [130] 1.11 [54]GREECEMEDIEN IN DER BILDUNG, GERMANY .0045 [29] .0122 [36] .6132 [494] 110 [23] .5262 [23] 1.39 [27]CENTRE NATIONAL DE LA RECHERCHE SCIENTIFIQUE, .0045 [30] .0117 [40] .6033 [498] 110 [23] .5239 [24] 1.37 [30]FRANCEPR = PageRank | BC = Betweenness centrality | LC = Local clustering coefficient | DC = Degree centrality | CC = Closeness centrality(all metrics were calculated using unweighted edges)* … Funding is stated in million Euro (EC contribution to the project cost). Note that CORDIS states the total funding for each project. The funding per consortium member for each project was computed by dividing the total EC contribution to that project by the number of consortium members. This should give a good estimate. – 23 –
  • 30. .014 250 .012 200 .010 PageRank 150 .008 Degree .006 100 .004 50 .002 .000 0 0.0 0.2 0.4 0.6 0.8 1.0 0.0 0.2 0.4 0.6 0.8 1.0 Local clustering cofficient Local clustering coefficient (a) (b) Figure 11: Local clustering of organizations plotted against (a) PageRank and (b) degree.Clearly, it is favorable to belong to the set of organizations that achieve a higher PageRank or degree ata given clustering value. This “bump” in PageRank seems to happen when an organization reaches aPageRank value range above .003. For degree, the bump seems to happen when an organization hasconnections to at least ~80 other organizations. To achieve these values, an organization needs toeither connect to new organizations in new projects and/or by connecting to other organizations with ahigh PageRank.A degree threshold of at least 80 was achieved by 92 organizations in the data set. Those projects inFP6, FP7 and eContentplus where at least half of the consortium is made up by these organizationsare: KALEIDOSCOPE (100%), RE.MATH (88%), STELLAR (81%), LEAD (67%), GRAPPLE, ALICE(60% each), SCY (58%), PROLEARN (54%), and TEL-Map (50%). This short list includes three of thefour TEL networks of excellence. GALA; the fourth TEL network of excellence is found at rank 25 with29%. This can be interpreted as evidence that participation in a network of excellence is highlybeneficial to the centrality of an organization in the collaboration network.The top partnership bonds across all TEL projects are displayed in Table 5. The table shows the 22pairs of organizations that have collaborated in at least 4 TEL projects. Assuming partnership is onlycontinued from successful projects, we can conjecture that those projects where the listedorganizations were involved can be flagged as having lasting impact, at least in terms of continuity inresearch collaborations. The most important of these projects, ordered by frequency of partnership inTable 5, are: PROLEARN (FP6; 16 pairs), ICOPER (eContentplus; 10 pairs), OpenScout (eContentplus;9 pairs), GRAPPLE (FP7; 8 pairs), STELLAR, ROLE (FP7; 5 pairs), and PROLIX (FP6, 5 pairs). It isevident that the PROLEARN network of excellence that co-kicked off FP6 succeeded in creating andsustaining strong partnerships, while the KALEIDOSCOPE network of excellence, which started at thesame time as PROLEARN, failed to achieve this despite having a much larger consortium. – 24 –
  • 31. Table 5: Strongest partnership bonds over all TEL projects in FP6, FP7 and eContentplus. Rank Partnership # GOTTFRIED WILHELM LEIBNIZ UNIVERSITAET 1. OPEN UNIVERSITEIT NEDERLAND with HANNOVER 6 - THE OPEN UNIVERSITY with KATHOLIEKE UNIVERSITEIT LEUVEN 6 3. THE OPEN UNIVERSITY with OPEN UNIVERSITEIT NEDERLAND 5 IMC INFORMATION MULTIMEDIA COMMUNICATION - THE OPEN UNIVERSITY with AG 5 - JYVASKYLAN YLIOPISTO with THE OPEN UNIVERSITY 5 6. OPEN UNIVERSITEIT NEDERLAND with KATHOLIEKE UNIVERSITEIT LEUVEN 4 DEUTSCHES FORSCHUNGSZENTRUM FUER - OPEN UNIVERSITEIT NEDERLAND with KUENSTLICHE INTELLIGENZ GMBH 4 IMC INFORMATION MULTIMEDIA COMMUNICATION GOTTFRIED WILHELM LEIBNIZ UNIVERSITAET - AG with HANNOVER 4 IMC INFORMATION MULTIMEDIA COMMUNICATION DEUTSCHES FORSCHUNGSZENTRUM FUER - AG with KUENSTLICHE INTELLIGENZ GMBH 4 IMC INFORMATION MULTIMEDIA COMMUNICATION - AG with OPEN UNIVERSITEIT NEDERLAND 4 - ATOS ORIGIN SOCIEDAD ANONIMA ESPANOLA with UNIVERSITAET GRAZ 4 FRAUNHOFER-GESELLSCHAFT ZUR FOERDERUNG - DER ANGEWANDTEN FORSCHUNG E.V with OPEN UNIVERSITEIT NEDERLAND 4 - POLITECNICO DI MILANO with OPEN UNIVERSITEIT NEDERLAND 4 IMC INFORMATION MULTIMEDIA COMMUNICATION - GIUNTI LABS S.R.L. with AG 4 - ECOLE POLYTECHNIQUE FEDERALE DE LAUSANNE with THE OPEN UNIVERSITY 4 IMC INFORMATION MULTIMEDIA COMMUNICATION - WIRTSCHAFTSUNIVERSITAET WIEN with AG 4 - WIRTSCHAFTSUNIVERSITAET WIEN with THE OPEN UNIVERSITY 4 NATIONAL CENTRE FOR SCIENTIFIC RESEARCH IMC INFORMATION MULTIMEDIA COMMUNICATION - DEMOKRITOS with AG 4 DEUTSCHES FORSCHUNGSZENTRUM FUER - TECHNISCHE UNIVERSITEIT EINDHOVEN with KUENSTLICHE INTELLIGENZ GMBH 4 - EUN PARTNERSHIP AISBL with UNIVERZA V LJUBLJANI 4 - EUN PARTNERSHIP AISBL with TIIGRIHUPPE SIHTASUTUS 4 THE PROVOST FELLOWS AND SCHOLARS OF THE - COLLEGE OF THE HOLY AND UNDIVIDED TRINITY with UNIVERSITAET GRAZ 4 OF QUEEN ELIZABETH NEAR DUBLIN3.4.3 Dynamic SNA of the TEL Project LandscapeThe previous figures all took the current status of collaborations and projects as a basis for calculatingSNA metrics and graph characteristics. To understand the dynamics of the projects and theirconsortium collaborations we now take a look at the development of SNA metrics of the collaborationnetwork on a yearly basis, starting from 2004 when the first FP6 projects were launched, up to theyear 2010 (inclusive). Year 2011 was omitted from the analyses since the year is not yet over, and nonew TEL projects were launched thus far.Figure 12 shows that in FP6 the first set of (eight) projects launched in 2004 introduced 4,199 distinctcollaboration connections among 157 organizations in the TEL landscape (Figure 12a). This massiveentry number is mainly due to the KALEIDOSCOPE network of excellence, which was launched in thatyear with an extremely large consortium of 83 partners (the average consortium size in FP6 was 14.5partners). In Figure 12b we see that the diameter of the network—i.e. the longest shortest path throughthe network—has reached its peak in 2006, after only 2 years; in 2010, the diameter shrunk to a valueof 4, which means that one or more projects have introduced direct connections between previously – 25 –
  • 32. distant partners. It also shows that the average path length has been stable at a value of around 2.5since 2006. This means that each organization is on average connected to each other organization byonly two intermediate organizations. This indicates that the collaboration network is extremely tightlyknit. One correlate of being tightly knit is that such a network tends to be hard for outsiders to join;this can inhibit the diffusion of information and innovation across a network boundary. 1,000 9,330 10,000 6 8,234 5 5 5 5 7,409 5 800 8,000 6,607 4 4 6,047 4 600 4,987 6,000 3 4,199 604 3 2.46 2.52 2.51 2.50 2.50 542 2.18 400 476 4,000 1.74 423 2 371 200 Organizations 2,000 Diameter 257 1 Collaborations Avg. Path Length 157 0 0 0 2004 2005 2006 2007 2008 2009 2010 2004 2005 2006 2007 2008 2009 2010 (a) (b) Figure 12: Overall development of collaboration network since 2004.Until the end of 2010, the number of organizations involved in TEL projects almost quadrupled (3.9-fold), while the number of project-based collaboration ties between those organizations slightly morethan doubled (2.2-fold) during the same time window (cf. Figure 12a). This gap can partly be explainedby Figure 13a, which shows that although there has been a steady flow of new projects, these projectshave added fewer and fewer new organizations to the picture, exposing a drop from 8.1 neworganizations per new project in 2006 to 4.8 new organizations per new project in 2010. 25 20 600 New Projects 525 New Projects New Organizations per New Project 500 20 New Collaborations per New Project 15 19.6 14 14 400 13 15 12 14 14 13 10 300 12 9 10 8 7 9 7 200 8 8.1 5 80 89 84 5 7.1 7.4 56 76 69 5.9 100 5.5 4.8 0 0 0 2004 2005 2006 2007 2008 2009 2010 2004 2005 2006 2007 2008 2009 2010 (a) (b) Ratio of New Organizations in Consortia of New Projects New projects per programme per year Avg. Consortium Size of New Projects 15 14 FP7 100% 50 85% 13 FP6 90% 45 12 eContentplus 80% 67% 40 11 66% 10 70% 62% 35 9 10 7 60% 30 8 42% 44% 7 14 50% 40% 25 13 40% 20 6 6 23.0 5 30% 15 4 8 3 7 20% 14.1 10 5 10% 10.9 12.1 12.0 12.6 11.9 5 2 4 3 1 0% 0 0 2004 2005 2006 2007 2008 2009 2010 2004 2005 2006 2007 2008 2009 2010 (c) (d) Figure 13: Impact of newly launched projects the collaboration network.Figure 13c demonstrates that the average size of the consortia of newly launched projects has beenrelatively stable since 2005, ranging between 10.9 and 14.1. In contrast, the average share of newlyintroduced organizations per launched project has dropped from 66% in 2005 to 40% in 2010. The – 26 –
  • 33. sharpest drop is evident for projects that started in the year 2008 (from 62% to 42%); this is the yearwhen the first 6 FP7 projects plus 3 new eContentplus projects were launched. So it appears that at thetransition from FP6 to FP7 and eContentplus, the project consortia resorted to building on anestablished core of members.In Figure 14 we see that the average number of new collaboration ties created by each organizationmaking its debut in TEL projects has, after an initial fall between 2004 and 2005, increased from 7.9in 2005 to 17.7 in 2010. Hence, starting to participate in TEL projects has an increasingly positiveeffect in terms of new collaborations with other organizations involved in TEL. 30 26.7 25 20 17.7 15.1 15 12.5 10.8 9.3 10 7.9 5 New Collaborations per New Organization 0 2004 2005 2006 2007 2008 2009 2010 Figure 14: Impact of organizations on collaboration.The project participation data shows that of the 34 TEL projects launched between 2008 and 2010,20% were coordinated by organizations which had not participated in any previous (or at that timerunning) TEL project. The development of this percentage over time is plotted in Figure 15. The sharpincrease in 2007 is likely due to eContentplus, where the focus shifted to e-content and metadata, andthus new organizations were introduced. The data shows that even for complete “newbieorganizations” in TEL it is absolutely feasible to write a successful project proposal in the coordinatorrole. 100% 88% 86% 90% 80% 70% 60% 50% 40% 50% 43% 30% 20% Ratio of Projects with Novice 25% 23% 10% Participants as Coordinator 11% 0% 2004 2005 2006 2007 2008 2009 2010 Figure 15: Development of the ratio of projects coordinated by novice organizationsHowever, the tendency evident in most of the figures in this section points in another direction; itappears that there is less and less demand for new organizations on the TEL market. One the onehand, this is understandable: if an organization launches a new project it is likely to resort to partnersit has already successfully collaborated with, particularly as more organizations are entering the“market” every year. On the other hand, it shows that project consortia and collaboration ties betweenorganizations behave like an inertial mass, which impedes the involvement of new and freshorganizations, and likely also new ideas and research foci. – 27 –
  • 34. 3.5 Geo-Mapping TEL ProjectsAnother interactive visualization we developed based on the geographically enhanced CORDIS datashows the TEL hotspots in Europe (see Figure 16). It allows selection of (multiple) FP7 TEL projects,which will result in the display of (a) the partners involved in those projects on a Google Map overlay, and (b) a word cloud of the descriptions of all selected projects.Clicking on a marker in the map will show a popup window with the name of the organization, the listof projects in which the organization is or was involved, and a link “Show partners”, which, upon beingclicked will display all the consortium members of all projects where this organization is or wasinvolved. An interactive online version of this is available at http://learningfrontiers.eu/?q=story/tel-project-landscape . Figure 16: Google Map overlay with organizations involved in TEL projects. – 28 –
  • 35. In combination with the centrality and degree metrics obtained from the analysis of $ thisvisualization enables in-depth understanding of collaboration structures and geographicalagglomeration of TEL organizations across Europe. The map exposes a strong north-south axisstarting in Scotland, via England, the Netherlands, Belgium, and West Germany, to northern Italy withstrong “outposts” in Switzerland and Austria.4 Analysis of TEL Publication OutletsIn general, peer reviewed publications are one major scientific indicator of the impact of research anddevelopment work. The same holds true for TEL, which is an emerging research area in severalestablished disciplines like psychology or computer science. Particularly in computer science,conferences have a dominant role in communication of research, with many conference venues havingthe same or even higher impact than well-ranked journals.The major gathering of European TEL researchers is the EC-TEL conference, an international refereedconference that was launched in the scope of the FP6 network of excellence PROLEARN. According toMicrosoft Academic Search, there are 57 conferences, compared to 19 journals in the computereducation category. Such domination raises questions about understanding the communities ofconferences and their development pattern in order to have an overview on the current research workof the TEL area. For researchers, understanding the community means getting to know the researchenvironment, which leads to self-adaptation and (hopefully) improvement in the field. For conferenceorganizers and other stakeholders, an overview of their communities is important for maintaining,cultivating and promoting the conferences and their communities.The structure of scientific collaboration can be researched in great detail by SNA of two distinct datasets: the co-authorship graph and the citation graph. The co-authorship graph reveals the contributionstructures of a scientific community by disclosing who has collaborated with whom in terms of co-authoring of papers. The citation graph discloses the influencing areas, conferences, and journals of aconference in terms of cited papers. Together, the two graphs allow a detailed analysis of theknowledge structure and flows within the scientific community but also the knowledge flows betweenadjacent scientific communities. SNA of TEL communities is only available on the level of singleconferences [8, 15, 21] or from the perspective of a project [26]. A systematic comparison of keyfeatures of scientific community shows that, depending on the duration of existence, differentconferences expose different development patterns.4.1 Data SetThe data set used in our study is the combination of DBLP and CiteSeerX digital libraries. We choosethese two because they cover most of the relevant sub-disciplines. We retrieved the publication list ofconferences from DBLP and used CiteSeerX to fill the citation list of publications in DBLP. This wasachieved by using the canopy clustering technique [13]. Overall, the matching algorithm gave us864,097 pairs of matched publications, of which only a subset is relevant to TEL, of course.The data was stored in the TEL-Map Mediabase in a relational database schema. The schema includesseveral dozen tables. The most important of those are displayed at a conceptual level in Figure 17. Thecentral entity is paper. Each paper has relationships with its authors, the keywords, abstract, and theproceedings in which it appeared. Proceedings are published for an event (e.g. EC-TEL 2011) and eachevent belongs to an event series (e.g. EC-TEL 2011 is the 6th event in the EC-TEL event series). Tosimplify storage and querying, the same concepts are used to represent journal publications. A journalis represented as an event series, each volume of the journal as an event, and each issue as aproceeding. Events can be related to other events, e.g. a workshop event held in conjunction with aconference event. – 29 –
  • 36. Keyword cite N N N has_keyword Paper authorship Author N N N 1 N N has_classification has_abstract appear_in Event Series 1 N 1 1 Classification Abstract Proceeding belong_to N 1 N N N belongs_to publish Event related_to 1 N Figure 17: Data model for TEL papers and events.From these data sets we extracted the co-authorship and the citation networks for conferences andjournals with a primary focus on TEL. The set of most relevant conferences was obtained by thefollowing procedure: • Starting with definitely relevant TEL journals indexed in DBLP like Educational Technology & Society (ETS), IEEE Transactions on Learning Technology (TLT), and Computers & Education (CE), we computed for authors who have published in these journals a list of conferences where those authors have published the most papers since 2005. A total of 1,135 conference series (repeat: not single conferences, but series) are in the candidate set, which shows that TEL researchers are active in a wide range of sub-disciplines. • From this result, ordered by papers per event, we filtered the top five conferences whose primary focus is on TEL related topics. Conferences that did not meet the filter were CHI (Computer Human Interaction), HCI (Human Computer Interaction), ICCE (International Conference on Computers in Education), HICSS (Hawaii International Conference on System Sciences) and CRIWG (International Workshop on Groupware). Some of those, while somehow relevant to European TEL, were removed because (a) their main focus is not in TEL, e.g. HICSS, which is a multi-track systems science conference; or (b) because they have a strong regional focus, e.g. ICCE focuses mainly on the Asia-Pacific region.The resulting set of our top five TEL conference series is listed in Table 6. The table clearly shows thatthe most relevant international conference is ICALT, which has published 768 papers by ET&S, CE andTLT authors since 2005. The most relevant European conference is ECTEL with 215 papers publishedby ET&S, CS and TLT authors since 2005. AIED and ITS are venues of primary relevance to work bythe artificial intelligence community as related to TEL. That is, the three core TEL conferences onDBLP appear to be ICALT, ECTEL and ICWL, since these have their primary focus in the TEL area. – 30 –
  • 37. Table 6: Selection of conferences relevant to the TEL community. Conference Series Acronym Series Events Relevancy*IEEE International Conference on Advanced Learning Annually 2001-2010 ICALT 768Technologies (except 2002)Artificial Intelligence in Education AIED Bi-annually 2005-2009 216European Conference on Technology Enhanced Learning ECTEL Annually 2006-2010 215 Bi-annually 1992-2010International Conference on Intelligent Tutoring Systems ITS 168 (except 1994)International Conference on Web-Based Learning ICWL Annually 2002-2010 90* … Total number of papers in the conference series since 2005, which were written by authors who have also published in ET&S, CE, or TLT. Note that we considered all proceedings associated on DBLP with the respective event series; this means e.g. that workshop proceedings associated with a particular conference event are included as part of the event series.The five conferences have published a total 3,291 papers between 2005 and 2010. The word cloud ofthose word stems that are mentioned at least 100 times in the titles is displayed in Figure 18. It isevident that the core word stem is learn, with system, support, design, model, collabor, web, educ,tutor, adapt in the vicinity. Figure 18: Word cloud of most frequent terms in TEL conference paper titles.4.2 Social Network Analysis of TEL Venues and PapersWe applied time series analysis on the networks to reveal the following SNA parameters from thenetworks over time: Densification law, clustering coefficient, maximum betweenness, largestconnected component, diameter, and average path length [25]. These parameters enable us to explainthe community building process that we proposed in [20], as depicted with adaptations in Figure 19.To interpret the shape of the community, one needs to use a combination of all of those parameters.We note that this is not the only model that explains the development pattern of every conference. Wepresent it here because, as we found in our previous study [20], it is a typical model that describes thecommunity building process of many conferences in different areas in computer science: Initially,there are few connections between authors. After some events, author groups become apparent in thenetwork (“Bonding” in Figure 19), which are—in the best case—gradually integrated throughpublications that involve authors from more than one group (“Emergence” in Figure 19). Over time, aconference a conference then typically reaches a state of development that either represents aninterdisciplinary, hierarchical or focused authorship network, the latter including a strongly connectedcore group of authors that is connected to other smaller groups. – 31 –
  • 38. Figure 19: Development model for conference communities.4.3 Co-Authorship Network Analysis4.3.1 Formal FoundationsCo-authorships on papers can be modeled as a social network. Let be the set of papers, let be theset of outlets relevant to TEL (conferences, journals, books, etc.), and let ( be the set of authors.Function ) represents the authorship of papers ∈ by author(s) ∈ ( and is defined as follows: , if ∈ ( is author of ∈ )∶ (→ , otherwise . Function * represents the appearance of papers ∈ in outlets ∈ and is defined as follows: , if ∈ appeared in ∈ *∶ → , otherwise .Assuming that ( and only contain relevant authors and outlets, respectively, we can define the co-authorship graph + = ( + , + with + = ( and + = (, ∶ , ∈ + ∧ ∧ ∃( , ∈ ∶ *( , ∧ )( , ∧ )( , " .4.3.2 OverviewAll five conference co-authorship networks are complex networks. In the years since 2003, these fiveconferences combined have published papers written by a relatively stable number of 1,350 (co-)authors each year. This is shown in Figure 20.For illustration, Figure 21 displays the current co-author network for each of the five conferences inthumbnail form. Each thumbnail represents the state of the co-authorship network as of end of year2010. – 32 –
  • 39. 1600140012001000 ICWL AIED 800 IST ECTEL 600 ICALT 400 200 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010Figure 20: Cumulative annual (co-)author figures of selected TEL conferences over the last 10 years. (a) ICALT (b) ECTEL (c) AIED (d) ITS (e) ICWL Figure 21: Co-authorship network visualization for the TEL conferences. – 33 –
  • 40. 4.3.3 Dynamic SNAIn the light of a typical development pattern as introduced in Figure 19, these conferences expose amixed picture. ICALT (a) and ECTEL (b) have well connected authors. For ECTEL, which is only halfas old as ICALT, this is a remarkable achievement that is likely based in the origins of ECTEL: theconference was started as an initiative out of the PROLEARN project in 2006, and to this day remainsa strongly EU TEL project focused presentation outlet and meeting venue. As we see from Figure 22d,ECTEL gathers about 30% of all authors in the largest connected sub-network, with several clearlydistinguishable sub-networks in the periphery. ICWL (e) on the other hand is as old as ICALT andseems to struggle with managing the transition from stage b to stage c in the development pattern inFigure 19. Less than 15% of all ICWL authors form the core circle of this conference. The remainingtwo conferences AIED (c) and ITS (d) expose very mature author communities, which is likely due tothe fact that these two conferences attract a strong core of artificial intelligence (AI) researchers. Inthat sense, they are probably difficult to compare with the other TEL conferences, since their core topicis AI rather than TEL.The density, i.e. the ratio between the number of edges and the number of nodes increases over timewith a coefficient larger than 1 and lower than 2 (Figure 22a). The clustering coefficient of all co-authorship networks is quite high and falling over the years (Figure 22b) but Figure 22d shows thatAIED and ECTEL have quickly growing largest connecting components (i.e. the core sub-network ofauthors) indicating a faster scientific community building process than for ICALT and ICWL.ITS has the largest core author group of all five conferences, but it needed longer to develop. Also themaximum betweenness of ITS is the biggest indicating that there are many active key members (i.e.those authors that connect different author communities through co-authoring of papers) contributingto the conference and the community development. ICALT and ICWL have not such a clear pattern,while AIED and ECTEL are developing very fast. Fast development of the community typicallyindicates that the conference has a tighter focus and/or the authors publishing at those conferencesalready had strong ties among each other. ECTEL, for example, is a European conference, so thecommunity is by definition smaller than that of ICALT or ICWL, which address TEL communitiesworldwide.For interdisciplinary conferences, it is common that there are several strongly connected clustersrepresenting the different disciplines, while there are connecting authors between those. Some TELconferences under analysis here expose characteristics of interdisciplinary networks (evident in theICWL network), while there are also cases of apparently focused authorship networks e.g. for ICALTand ECTEL. Still, the largest connected components (the core author group) for TEL conferences tendto include about one third of the authors. For strongly focused and “grown-up” conferences, e.g.SIGMOD or VLDB in computer science (which are beyond their 30th anniversary), the largestcomponent includes roughly two thirds of all authors who have published there.All diameters of the co-authorship networks are still growing; this indicates that the development ofthe communities of these conferences is not finished yet (Figure 22e). The diameter represents thelength of the longest shortest path through the network, so a peak in diameter growth would indicatelack of integrating new author communities into the conference community. Also the average pathlength is still growing indicating again that the overall TEL author network is still growing. – 34 –
  • 41. 4 (a) Densification law (b) Clustering Coefficient 10 1 0.95 Clustering coefficient Number of edges 3 10 0.9 1.1976 ICALT: 0.34889*x 1.0544 ICALT ICWL: 1.1149*x 0.85 10 2 ICWL 1.2415 ECTEL: 0.40338*x ECTEL ITS: 0.15818*x 1.3817 0.8 ITS 1 AIED: 1.0128*x 1.1197 AIED 10 0.75 10 1 10 2 10 3 10 4 1 2 3 4 5 6 7 8 9 Number of nodes Age (c) Maximum Betweenness (d) Largest connected component 0.08 0.7 ICALT ICALT Largest connected component ICWL 0.6 ICWLMaximum betweenness 0.06 ECTEL ECTEL 0.5 ITS ITS AIED 0.4 AIED 0.04 0.3 0.02 0.2 0.1 0 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Age Age (e) Diameter (f) Average Path Length 20 8 ICALT ICALT ICWL ICWL Average path length 15 6 ECTEL ECTEL Diameter ITS ITS 10 AIED 4 AIED 5 2 0 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Age Age Figure 22: Co-authorship network measures of five conferences in TEL.4.3.4 Most Prolific Authors and Their TopicsThe 15 most prolific authors in terms of published papers at the conferences and journals are displayedin Table 7. The ranking considers papers published at venues with a core TEL scope—that is, papers atICALT, ECTEL and ICWL conferences, as well as ET&S, TLT, and CE journals. To remove biasintroduced by conferences and journals with a longer publishing record only papers published sincethe year 2000 were considered.The table shows that by far the most prolific author has been Kinshuk, who is with AthabascaUniversity, Canada. The most prolific European author is Demetrios G. Sampson from University ofPiraeus, Greece. The ranking clearly shows that, although the conferences have visited all continentsand the journals are independent of author and venue location, TEL research (in terms of publishedoutcomes) seems mainly based in Europe and Asia. Only one author accounts for North America, i.e.Kinshuk. However, until 2006 Kinshuk was with Massey University in New Zealand, making theranking an almost exclusively “Eurasian” list. – 35 –
  • 42. Table 7: Fifteen most prolific authors at conferences and journals with a broad TEL scope. Names marked with an asterisk (*) indicate authors currently based in Europe. Rank Author Name # Papers 1. Kinshuk 59 2. Toshio Okamoto 38 Demetrios G. Sampson* 38 4. Rob Koper* 34 5. Alexandra I. Cristea* 33 6. Nian-Shing Chen 31 7. Maria Grigoriadou* 25 Dragan Gasevic 25 Erkki Sutinen* 25 Ralf Klamma* 25 Chin-Chung Tsai 25 12. Erik Duval* 23 Tak-Wai Chan 23 Gwo-Dong Chen 23 Wolfgang Nejdl* 23Top topics in top authors’ papers. Assuming that the most important TEL authors according tothis list also have an influence in the current focus and future direction of TEL topics, it is worthwhileto analyze the topics of the papers these authors have published. There are 21 paper title term stems7that have had a rising frequency between 2008 and 20108 (with no intermediate fall). The top-10 ofthose by ordered by absolute frequency in 2010 are mobil, comput, hypermedia, virtual, learners,engin, cognit, multimedia, study, style, and project. The ten most frequent term stems between 2007and 2010 without taking into account the frequency tendency are learn, adapt, web, model, educ,system, support, semant, design, and collabor.The most frequent term stems in 2010 that have not dropped in frequency compared to 2009 areadapt, semant, design, mobil, author, awar, manag, person, and knowledg (compare Figure 23). Thisword cloud represents the most recent upcoming work of the top authors published in 2010, andshould therefore be a good indicator of the current direction. It clearly indicates a way towardsadaptive, personalized and mobile learning, with an emphasis on semantics, management, authoring,and awareness.7 Word stemming was performed using Jon Abernathy’s PHP implementation of the Porter Stemming Algorithm, which can be found at http://www.chuggnutt.com/stemmer-source.php. Prior to that, a stop-word list was applied. A few overlapping word stems resulting from this algorithm were then merged manually.8 The paper data was last updated in October 2010, so the 2010 frequencies were multiplied by 4/3 to account for the missing three months in 2010. This will tend towards values that underestimate the real frequency, since the DBLP data is usually updated some time (sometimes days, sometimes weeks, sometimes months) after the publication date. – 36 –
  • 43. Figure 23: Most frequent terms in papers of top TEL authors in 2010.4.3.5 Overall TEL Co-authorship NetworkLooking at the combined co-authorship graph of the core TEL venues (ECTEL, ICALT, ICWL, TLT,ETS and CE), we obtained 14,689 distinct co-authorship relationships among 7,802 authors. Somenoteworthy facts about this graph are: • The largest connected component—i.e. the inner circle of co-authors—consists of 2,249 authors (6,251 co-authorship relations), which corresponds to 28.8% of all authors. Having almost one third of all authors in the inner co-authorship circle indicates that this core TEL author community (displayed in Figure 25) is tightly knit. However, as evident in Figure 24 there are 1,401 connected components in total in the graph, which means that there are a huge number of disconnected co-authorship circles that do not “integrate” with the core. Figure 24: Complete co-authorship network in the core TEL venues. – 37 –
  • 44. • The average degree centrality of the authors is 3.765, which means that each author has been writing papers with almost 4 different colleagues. • The clustering coefficient of the graph has a value of .718, which indicates a mature co- authorship network. • The average shortest path length is at 7.6, which is typical for a small-world network, in which most nodes can be reached from every other by a small number of steps, although the network may be very large (cf. Milgram’s small world experiment [14]). The diameter, i.e. the most distant connection between any two authors spans 28 nodes, which is quite a large extent. Figure 25: Co-authorship network of the “inner circle” of authors in the core TEL venues.4.3.6 Central Authors in the Co-Authorship NetworkThe top 15 authors in the TEL co-authorship network are listed in Table 8. The table is ordered bybetweenness centrality, since this measure is a very good indicator whether an author has integrateddifferent communities. In addition, the table lists the PageRank value as well as clustering coefficientand degree centrality for each author. According to this list the top author is Kinshuk, who weighs inwith 72 different co-authors and who is part of more than 3% of all shortest paths through thenetwork. He has by far the highest betweenness and PageRank values, indicating his central positionin the network. The most important European authors in this list are R. Koper, who is with OpenUniversiteit Nederland (a central organization in the European TEL project landscape; see Table 4 in – 38 –
  • 45. Section 3.4.2) and A. Cristea who has an affiliation history with University of Warwick and TUEindhoven, which are also both frequent collaborators in TEL projects. Table 8: Top 15 TEL authors by betweenness centrality9. Betweenness PageRank Clustering Degree Rank Author Centrality ×10-2 Coefficient Centrality 1. Kinshuk .0315 .2011 .0411 72 2. Rob Koper* .0235 .1191 .0968 63 3. Alexandra I. Cristea* .0164 .0869 .1028 32 4. Toshio Okamoto .0116 .1092 .0796 39 5. Yongwu Miao* .0113 .0221 .2051 13 6. Heinz Ulrich Hoppe* .0112 .0765 .1351 32 7. Hugh C. Davis* .0109 .0854 .1336 37 8. Daniel Burgos* .0092 .0260 .2053 20 9. Stephen J. H. Yang .0088 .0439 .1810 15 10. Shian-Shyong Tseng .0074 .0401 .2762 15 11. Chin-Chung Tsai .0073 .0868 .0746 32 12. Davinia Hernández Leo* .0073 .0597 .2080 27 13. Ralf Klamma* .0069 .0870 .1817 41 14. Nian-Shing Chen .0064 .1108 .0692 40 15. Jin-Tan Yang .0061 .0535 .1527 29The top-10 co-authorship pairs in the core TEL venues are listed in Table 9. Nine of those involveEuropean authors, demonstrating the strong collaboration ties within and with European TEL. Table 9: Top ten co-author pairs in core TEL venues. Rank Author Pair Papers 1. Jelena Jovanovic* with Dragan Gasevic 16 2. Juan I. Asensio-Pérez* with Yannis A. Dimitriadis* 14 Jon A. Elorriaga* with Ana Arruarte Lasa* 14 4. Kinshuk with Nian-Shing Chen 13 Ralf Steinmetz* with Christoph Rensing* 13 6. Manuel Caeiro* with Luis E. Anido-Rifón* 12 Pablo Moreno-Ger* with Baltasar Fernández-Manjón* 12 José Luis Sierra* with Baltasar Fernández-Manjón* 12 Yao-Ting Sung with Kuo-En Chang 12 10. Ignacio Aedo* with Paloma Díaz* 11 Ralf Klamma* with Marc Spaniol* 114.4 Structural-Semantic Analysis: SNA and Topic Mining CombinedMethodology. In D4.1, “Report on Weak Signals Collection” [23], a simple approach to miningconference abstracts for candidate weak signals was described along with the findings and initialinterpretation of the possible meaning of those findings. By combining this—i.e. the “semantic9 The metrics were computed using the igraph package in R. The betweenness centrality was calculated using unweighted co-authorship connections. An asterisk (*) next to an author’s name indicates that this author’s currently main affiliation is with a European organization. – 39 –
  • 46. analysis”—with social network analysis of the co-authorship network—i.e. the “structural analysis”—we believe it is possible to gain a better understanding of the significance of candidate weak signals.Author betweenness centrality was used to identify abstracts from the set of papers found to haveunexpected increases in topic prominence as judged by the increase in single-term occurrence (ornewly appearing terms) in D4.1. This centrality measure is expected to be a useful indicator of weaksignal significance since it shows the author’s importance in chains of communication betweenotherwise separate (or more-separated) groups of researchers. This can be viewed in two (reciprocal)ways: 1. An author with high betweenness centrality is likely to have more influence over the spread of the idea that appears in a conference paper. 2. An author with high betweenness centrality can be expected to be exposed to more diverse ideas and to identify their significance through a conference paper.For the purposes of the structural-semantic analysis, the same set of conferences considered in D4.1was used; betweenness centrality measures were therefore calculated for ICALT, ICWL and ECTELonly, in contrast to previous sections. About two thirds of the top 15 authors ranked by betweennesscentrality occur in both sets. The most central author was found to be Chen-Chung Liu (betweenness.00441). The ranked list of authors was inspected down to a betweenness of .00044, i.e. a centrality of1/10th relative to the top-ranked author.Key Authorships. Table 10 shows the authors of papers identified in D4.1 as potentially containingweak signals. It is remarkable that the most central author within this set of papers is at rank 36 andwith a betweenness of approximately 1/6th of the most central author (Kinshuk, with a betweennesscentrality of .026). While it would be premature to draw conclusions from this observation, it doesappear to call into question our approach of using betweenness centrality as a measure of weak signalsignificance. Consequently, rather than understanding this measure to give us an absolute indicator ofweak signal significance, we only use it to indicate the relative significance among the set of candidateweak signals. Table 10: Betweenness centrality of authors of papers identified in D4.1. Betweenness Relative # Papers Co-auth. Rank Author Centrality Centrality (D4.1) Group 36. Chen-Chung Liu .00441 1.0 2 A 39. Marcus Specht .00398 0.9 1 B 79. Gary B. Wills .00214 0.5 2 C 95. Marco Kalz .00168 0.4 1 B 109. Marcelo Milrad .00134 0.3 1 - 127. Yvonne Margaret Howard .00122 0.3 1 - 130. Chin-Yeh Wang .00116 0.3 1 A 144. Gwo-Dong Chen .00103 0.2 1 A 176. Hui-Chun Chuang .00068 0.2 1 A 195. Lester Gilbert .00060 0.1 2 C 220. Baw-Jhiune Liu .00048 0.1 1 AThe table also shows co-authorship groups within the selected papers. Although it is clear frominspection of the full data set that individuals in each of these co-authorship groups are part ofdifferent networks, it would be unwise to assume that the paper “Design and Evaluation of an AffectiveInterface of the E-learning Systems” by Hui-Chun Chuang (.0007), Chin-Yeh Wang (.0012), Gwo- – 40 –
  • 47. Dong Chen (.0010), Chen-Chung Liu (.0044), Baw-Jhiune Liu (.0005) published at ICALT 2010, has asignificance according to the sum of centrality measures.On the basis of this data, and recognizing that this is a qualitative judgment, we might nominate thefollowing as “researchers to pay attention to”: • Chen-Chung Liu • Marcus Specht • Gary B WillsKey Weak Signals. Four principle themes were synthesized in D4.1 on the basis of around 8prominent terms and for each, there is at least one related paper written by at least one of the authorsidentified in the previous Table 10. The intersection between the themes and author centrality issummarized in the following Table 11, in which underlining is used to highlight “researchers to payattention to”. Table 11: Summary of structural-semantic analysis: themes and matching papers. Theme and Comment Papers Matching Criteria“Affect” is a relatively diffuse topic, which was represented through “Design and Evaluation of an Affective Interface of the E-learningterms such as “girl”, “emotion”, “skin” and “negative”. Only one Systems” (Hui-Chun Chuang, Chin-Yeh Wang, Gwo-Dong Chen,paper matches the SNA criterion although all of the authors match Chen-Chung Liu, Baw-Jhiune Liu, ICALT 2010)it.“e-Assessment” and its principle technical counterpart, “QTI” also “A Formative eAssessment Co-Design Case Study” (David A.has one paper matching the structural-semantic criteria, this time Bacigalupo, W. I. Warburton, E. A. Draffan, Pei Zhang, Lesterwith two listed authors. Gilbert, Gary B. Wills, ICALT 2010)“Risk” was highlighted in the context of evidence-based design of “Towards an Ergonomics of Knowledge Systems: Improving thelearning environments. Only two papers were identified in D4.1 and Design of Technology Enhanced Learning” (David E. Millard andboth also match the SNA criterion, which matched three out of the Yvonne Howard, ECTEL 2010 )five authors (all three belong to the same department). “Towards a Competence Based System for Recommending Study Materials (CBSR)” (Athitaya Nitchot, Lester Gilbert, Gary B. Wills, ICALT2010)“Authentic learning” is manifested in different ways between the five “An Audio Book Platform for Early EFL Oral Reading Fluency” (Kuo-papers identified in D4.1. Three of these also match the SNA Ping Liu, Cheng-Chung Liu, Chih-Hsin Huang, Kuo-Chun Hung,criterion with four distinct authors from our list. Chia-Jung Chang, ICALT 2010) “Ambient Displays and Game Design Patterns” (Sebastian Kelle, Dirk Börner, Marco Kalz and Marcus Specht, ECTEL 2010) “Exploring the Benefits of Open Standard Initiatives for Supporting Inquiry-Based Science Learning” (Bahtijar Vogel, Arianit Kurti, Daniel Spikol and Marcelo Milrad, ECTEL 2010)From the point of view of a weak signals analysis, the weakest theme appears to be “e-Assessment”. InD4.1 the rise of interest in this theme was ascribed to a main-streaming of e-assessment leading topapers reflecting e-assessment in practice. This is somewhat in contrast to the areas of research thatare grouped under the other three themes, which are more aspirational.On the basis of the structural-semantic analysis, there seems no clear reason to give particularprominence to any of the other three themes; none of them has a markedly more influential set ofauthors than the others. This leads us to synthesize a vision of the future, latent in the chosen abstractsof the TEL research community, with aims such as: • to increase the degree to which TEL practice uses tools with evidence-based and principled design methodologies; – 41 –
  • 48. • to increase the support for authentic rather than overtly-educational activities through TEL; • to increase the degree to which affect and emotion are taken account of in learner interactions with TEL systems.This represents a vision that is progressive rather than conservative and grounded in pedagogicconcerns rather than technical or managerial matters.The specific weak signals falling within the three themes are best identified by reference to the subjectmatter of the papers, here expressed through the abstracts: • “Design and Evaluation of an Affective Interface of the E-learning Systems” Students affections in learning have a significant impact on engagement and learning outcomes. When students have negative emotions, they usually do not learn well. But current e-learning systems often lack many features of profound affection, and fail to provide suitable emotional interaction. In this paper, we evaluate some studies of affective interaction e-learning systems. We also proposed our approach to develop an emotionally interactive learning system. • “Towards an Ergonomics of Knowledge Systems: Improving the Design of Technology Enhanced Learning” As Technology Enhanced Learning (TEL) systems become more essential to education there is an increasing need for their creators to reduce risk and to design for success. We argue that by taking an ergonomic perspective it is possible to better understand why TEL systems succeed or fail, as it becomes possible to analyze how well they are aligned with their users and environment. We present three TEL case studies that demonstrate these ideas, and show how an ergonomic analysis can help frame the problems faced in a useful way. In particular we propose using a variant of ergonomics that emphasizes the expression, communication and use of knowledge within the system, we call this approach Knowledge System Ergonomics. • “Towards a Competence Based System for Recommending Study Materials (CBSR)” Most e-learning systems require intervention from a teacher. The development of adaptive hypermedia systems, such as intelligent tutoring systems, aimed to reduce the teachers task. However, such systems are still at risk of inconsistently modelling the user when estimating a learners knowledge level. We propose a system called CBSR (Competence based System for Recommending Study Materials) which recommends appropriate study materials from the Web without requiring teacher intervention, based upon a competency model. This has the benefit of an improved pedagogical approach to e-learning, and a more consistent profile of learners competences which can persist though their life. • “An Audio Book Platform for Early EFL Oral Reading Fluency” Oral reading fluency is essential to overall reading achievements and repeat reading has been found to be an effective strategy for oral reading fluency. Choral reading is the most authentic use of repeated readings in the EFL primary grades. However, teachers have neither sufficient time nor adequate expertise to deal with non-fluent readers. Hence, challenges with oral reading fluency and motivation have long been considered a common characteristic for teacher and students. Hence, this study proposed a one-to-one Audio Book Platform ... • “Ambient Displays and Game Design Patterns” In this paper we describe a social learning game we implemented to evaluate various means of ubiquitous learning support. Making use of game design patterns it was possible to implement information channels in such a way that we could simulate ubiquitous learning support in an authentic situation. The result is a prototype game in which one person is chosen randomly to become “Mister X”, and the other players have to find clues and strategies to find out who is the wanted person. In our scenario we used 3 different information channels to provide clues and compared them with respect to user appreciation and effectiveness. – 42 –
  • 49. • “Exploring the Benefits of Open Standard Initiatives for Supporting Inquiry-Based Science Learning” Mobile devices combined with sensor technologies provide new possibilities for embedding inquiry- based science learning activities in authentic settings. These technologies rely on various standards for data exchange what makes the development of interoperable mobile and sensor-based applications a challenging task. In this paper, we present our technical efforts related to how to leverage data interoperability using open standards. To validate the potential benefits of this approach, we developed a prototype implementation and conducted a trial with high school students in the field of environmental science. The initial results indicate the potential benefits of using open standards for data exchange in order to support the integration of various technological resources and applications.The key weak signals of recently-recognized R&D needs and associated capacities emerging are: • Emotionally interactive learning systems; • An ergonomic perspective to understanding why TEL systems succeed or fail; • Intelligent tutoring (broad sense) systems based on alternatives to cognitive models (e.g. competence); • Game design patterns for learning support using at-hand technologies; • Exploitation of sensor technologies in mobile devices for authentic learning activities.The European Commission demonstrated great sensibility for these weak signals, since they are wellrepresented in the target outcomes of the TEL objective in the upcoming ICT Call 8, which include TELsystems endowed with capabilities of human tutors with an affective and personalized design, as wellas tools for fostering creativity and non-standard thinking10.4.5 Citation Network AnalysisThe citation networks of the TEL conferences ECTEL, ICALT, ICWL, ITS, and AIED—like the co-authorship networks—are complex networks with a ratio between number of edges and number ofnodes still growing (greater 1 and less than 2 in Figure 26a). The clustering coefficients of allconferences are similar, with ICWL exposing a higher coefficient than the other four conferences(Figure 26b). But Figure 26d shows that the literature of ICWL and ICALT is much less connected thanthat of ITS, AIED and ECTEL, which indicates that the two former have a broader, moreinterdisciplinary scope than the three latter. This is supported by the maximum betweenness in Figure26c showing the existence of more common core references in these scientific communities. Thediameters of ECTEL and AIED are beginning to shrink very early indicating that the body of literatureof these community is quite stable and the themes of the communities are found. Also the path lengthdevelopment is supporting this indicator.10 See http://ec.europa.eu/research/participants/portal/page/cooperation?callIdentifier=FP7-ICT-2011-8 – 43 –
  • 50. (a) Densification law (b) Clustering Coefficient 4 10 0.5 ICALT: 0.072098*x 1.3285 Clustering co efficien t ICWL: 0.024549*x 1.612 0.4 Numbe r of edg es 3 1.4728 10 ECTEL: 0.030923*x 0.3 ITS:0. 10363*x 1.3992 1.5493 ICALT AIED: 0.042131*x 0.2 ICWL 2 10 ECTEL 0.1 ITS AIED 1 10 0 1 2 3 4 10 10 10 10 1 2 3 4 5 6 7 8 9 Number of nodes Age (c) Maximum Betweenness (d) Largest connected component La rgest co nne cted co mp one nt 0.1 1 ICALT ICALT Maximum b etwe enn ess 0.08 ICWL 0.8 ICWL ECTEL ECTEL 0.06 ITS 0.6 ITS AIED AIED 0.04 0.4 0.02 0.2 0 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Age Age (e) Diameter (f) Average Path Length 20 8 Averag e p ath len gth 15 6 Dia me ter 10 ICALT 4 ICALT ICWL ICWL 5 ECTEL 2 ECTEL ITS ITS AIED AIED 0 0 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Age Age Figure 26: Citation network measures of five conferences in TEL.5 Analysis of the TEL Social WebStarting in the PROLEARN project11, RWTH has been hosting the collection of social web artifactsrelated to professional learning and technology enhanced learning from different sources, includingblogs, news lists, podcasts and websites. TEL-Map intends to use the content of these social webartifacts for supporting weak signal analysis by identifying topics and topic shifts on the TEL socialweb. Today, probably the most relevant channels for researchers, practitioners, consumers, andcommentators are blogs and micro-blogs (tweets). Since blogs are the richest data set in Mediabase,we have extended the blog sources in the original PROLEARN Mediabase with additional blogs andfeeds relevant to TEL. These sources—blogs and feeds—were collected via community mobilization inthe Learning Frontiers portal feed aggregator (see Section 2.2). All sources provided in the feedaggregator are periodically ingested into the TEL-Map Mediabase.Sources in the TEL-Map Mediabase are visited each night by a web crawler process that visits allsources and looks for new entries in these sources, e.g. new blog posts. One problem with RSS/Atomfeeds is that they usually contain only a limited number of recent entries, typically not more than 20.In many blogs, particularly those kept by frequent bloggers or blogger communities, the feed reflects11 http://www.learningfrontiers.eu/?q=tel_project/PROLEARN – 44 –
  • 51. only a tiny fraction of the full body of entries posted by the blogger(s). To remedy this issue for newlyingested sources, the crawler also parses the HTML sources of the blog web pages or blog post pagesfor links to older entries that are not listed in the most recent feed. This is possible since most bloghosting platforms display links to the blog or website archive in a side pane of the page. All new entriesfound in the feed and the archived entries found via hyperlinks are retrieved from the web and storedin the Mediabase both in original markup form and plain non-markup text. In addition a word set isextracted from the text and stored in the database.The word sets are immediately processed by a burst detector process, which identifies bursty words(i.e. frequently occurring words) in the sources. Generally speaking, bursts refer to topics whichappear, gain popularity, and then fade [11]. For instance, if a blogger is writing blog posts about aconference that she is attending, it is likely that during the conference the blogger’s entries exposebursty words related to the topic of the conference. Each word burst is associated with a burst powervalue that allows ranking and visualization of bursts, e.g. via a word cloud.Another process extracts links (URLs) from the markup texts of the new entries found in the blogs andfeeds. All links are collected in the database and recorded with their original source. All other existingsources are scanned whether their URL matches one of the links. All matches are recorded in thedatabase. This way, we are able to maintain a structured, inter-linked representation of the artifacts inthe TEL blogosphere and the TEL web as represented by TEL-related RSS/Atom feeds in theMediabase.5.1 Social Web Data SetData Model. The data model of the indexed TEL blogosphere is presented in Figure 27. Its mainentities are described as follows. Blogs and websites are represented as sources that may containseveral entries (e.g. blog posts or news pages). Both sources and entries are a sub-concept of actor,which is an abstract concept introduced in the Mediabase to facilitate network analysis by representingsocial media artifacts. Actors, which may have community-provided tags, can be located via a URL.These URLs may be referenced by other entries. has Comment N 1 Entry entry_urls URL N N 1 N 1 actor_url Tag is_a Burst source_entries N 1 N 1 1 Actor actor_tag N Source is_a 1 1 Figure 27: Relational model of the TEL blogosphere.Data Set. At the time of writing this report, the TEL Media database of TEL-Map Mediabase contains804 TEL-related sources, which contain a total of 341,649 entries. Those entries point to more than1.08 million distinct URLs in their hypertext. 58,460 of these URLs (ca. 5%) represent sources and – 45 –
  • 52. entries indexed in the Mediabase; the rest refers to sources outside of the collected TEL sources. The1.08 million URLs are hyperlinked 1.86 million times in total from the different entries. That is, eachentry includes 5.5 hyperlinks on average.Data Collection. Collection of blog sources started in 2006 in the context of the PROLEARN project.The distribution of the number of blogs entered since then as well as the blog entries indexed by thecrawlers is displayed in Figure 28. Interestingly, the highest number of blog sources was added to theMediabase after the PROLEARN project ended (445 sources in 2008). In 2009 the activity almoststalled, while after the launch of the TEL-Map project in 2010 the figures are rising again (71 sourcesin 2010 and 82 sources in 2011). The development of the number of yearly blog entries indexeddeveloped similarly. The figure shows the entries indexed since the year 2000, although there aremany entries indexed that were published earlier. There is a discrepancy between blogs added in a yearand the entries in indexed in that year. The reason is that when a blog is added for instance in 2008,the crawler also indexes archived entries that were published before 2008, adding blog entries toprevious years. So the current increase in added blog sources in the coming year will lead to a“delayed” increase of indexed blog entries that were actually published in 2011, but not indexed yet.The figure suggests a lag of 2-3 years when considering the sharp increase in indexed blog entriesstarting in 2004, although the indexing only began two years later in 2006.Although the number of indexed entries is very large and a useful body to analyze, there are certainlymore than 804 relevant sources out there on the web for TEL. One focus in future TEL-Map worktherefore will lie in continuously extending the body of indexed sources. 80,000 500 70,000 450 400 60,000 350 50,000 300 40,000 250 Entries 200 Blogs 30,000 150 20,000 100 10,000 50 0 0 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 Figure 28: Number of blogs added to and blog entries indexed in the TEL-Map Mediabase.5.2 Formal FoundationsReference networks between sources and entries on the TEL social web can be modeled as socialnetworks. Let , be the set of social web sources (e.g. blogs, web pages), let be the set of distinctentries (e.g. blog posts, news posts) contained within these sources, and let ( = , ∪ be the set ofactors, representing a union of sources and their entries. Function . maps the entries ∈ to theircontaining sources ∈ , and is defined as follows: , if ∈ is contained in s ∈ , . ∶ , × → , otherwise . Function <= represents the links from entries ∈ to actors ∈ ( as follows: , if ∈ links to ∈ ( <= ∶ × ( → , otherwise . – 46 –
  • 53. Function <> represents a link structure similar to <= , however only links from entries to entries areconsidered, i.e. , if ∈ links to ∈ <> ∶ → , otherwise .Now we can define several graphs, of which we focus on the following two: 1. A directed link network graph ? = > ( ? , > > ? representing links between social web entries ? = , with ? = ( , ∶ , ∈ ? ∧ ∧ <> ( , " . > > > 2. A directed link network graph ? = ( ?@ , ? representing links between social web sources @ @ ? = ,, with an edge being established between any originating source @ and and any target source , if any entry A of source links directly to source , or indirectly via one of the entries B contained in source : ( , ∶ , ∈ ?@ ∧ ∧ ∃ ∈ : .( , ∧ @ =C J . ? EF∃A ∈ ?@ : A = ∧ <= ( , A G ∨ F∃B ∈ : .( , B ∧ <> ( , B GI 5.3 Results5.3.1 TEL Blog Network and Most Central Blogs @Hyperlink Graph. Graph ? , which is visualized in Figure 29, consists of 617 blogs (nodes) and7,303 directed links (edges) between the nodes. As defined above, each edge represents a hyperlinkappearing in the source blog that points to the target blog. The graph by definition only includes blogs Figure 29: TEL blogs link network visualization, excluding self-references. – 47 –
  • 54. that link to another blog or that are being linked to by another blog, therefore not all sources indexedin the TEL-Map Mediabase are represented in the network. However, the network includes 77% of allMediabase sources, which indicates a healthy, interlinked selection of sources for the Mediabase.The network exposes one single weakly connected network, i.e. the largest connected componentincludes 100% of the network’s nodes; network analysis reveals that there are a total of 217 stronglyconnected components, that is, sub-networks where all nodes are immediately connected to each otherthrough edges. The diameter of the network equals 9 and the average path length is 2.97, whichindicates a highly integrated network, where nodes are close to each other.Top Sources. Table 12 exposes the twenty most relevant blog sources in the TEL-Map Mediabaseordered by PageRank, which not only takes into account the weight of edges between nodes, but alsothe importance of the source node of a link. The “TEL” column indicates whether a blog is primarilyabout TEL related topics (e.g. e-learning, educational technology, etc.) It is evident that nodes with ahigh PageRank typically also have strong centrality characteristics in the network. The list includessome prominent blogs, like those of Stephen Downes (ranked 6th and 8th) and CogDogBlog (9th), whileit also includes references to sources that are not genuinely TEL related, but of high relevance to TEL,e.g. Creative Commos News (2nd) which deals with licensing issues, and Read/WriteWeb (1st) or GoogleOperating System (14th) which deal with (web) technology with no particular TEL focus. Since abouthalf of the blogs in the table are not genuinely about TEL, it appears that TEL blogs heavily link tosources that discuss cutting-edge technology. Table 12: Top twenty blog sources by PageRank. The number in square bracket indicates the blog’s overall rank for the respective metric. Between-TEL Source PageRank In-Degree Authority Hub ness Centr. Read/WriteWeb – http://www.readwriteweb.com .0449 [1] 178 [1] .0228 [1] .0257 [1] .1238 [1] Creative Commons » CC News – http://creativecommons.org .0400 [2] 144 [3] .0185 [3] .0212 [2] .0386 [4] apophenia – http://www.zephoria.org/thoughts .0205 [3] 99 [7] .0127 [7] .0152 [6] .0162 [13]X Weblogg-ed – http://weblogg-ed.com .0158 [4] 132 [4] .0169 [4] .0197 [3] .0384 [5] Joi Itos Web – http://joi.ito.com .0149 [5] 19 [117] .0025 [117] .0029 [109] .0015 [151]X Stephens Web – http://www.downes.ca .0143 [6] 150 [2] .0192 [2] .0000 [472] .0000 [390]X Cool Cat Teacher Blog – http://coolcatteacher.blogspot.com .0100 [7] 95 [8] .0122 [8] .0143 [7] .0092 [35] Half an Hour – http://halfanhour.blogspot.com .0098 [8] 112 [5] .0144 [5] .0163 [4] .0595 [3] CogDogBlog – http://cogdogblog.com .0098 [9] 78 [10] .0101 [10] .0117 [9] .0349 [6]X Moving at the speed of creativity – http://www.speedofcreativity.org .0093 [10] 86 [9] .0111 [9] .0129 [8] .0268 [9]X elearnspace – http://www.elearnspace.org/blog .0090 [11] 111 [6] .0143 [6] .0158 [5] .0798 [2] iterating toward openness – http://opencontent.org/blog .0085 [12] 45 [28] .0059 [28] .0069 [26] .0085 [37]X Ideas and Thoughts from an EdTech – http://ideasandthoughts.org .0084 [13] 68 [12] .0088 [12] .0105 [11] .0152 [14] Google Operating System – http://googlesystem.blogspot.com .0081 [14] 42 [32] .0055 [32] .0063 [31] .0026 [113]X DArcy Norman dot net – http://www.darcynorman.net .0080 [15] 56 [17] .0073 [17] .0085 [17] .0103 [29] Seths Blog – http://sethgodin.typepad.com .0079 [16] 68 [12] .0088 [12] .0095 [13] .0043 [76]X K12 Online Conference – http://k12onlineconference.org .0079 [17] 57 [16] .0074 [16] .0088 [15] .0010 [183]X The Rapid eLearning Blog – http://www.articulate.com/rapid-elearning .0078 [18] 43 [30] .0056 [30] .0062 [33] .0032 [97] Avant Game – http://avantgame.blogspot.com .0075 [19] 8 [232] .0011 [232] .0014 [195] .0029 [105] Michael Geist Blog – http://www.michaelgeist.ca .0074 [20] 30 [61] .0039 [61] .0046 [59] .0039 [81]Terms in Top Sources. The frequency of word appearance in the entries contained in the blog listedin Table 12 is displayed as a word cloud in Figure 30. Apart from Google, frequently appearing termsinclude for instance: access, copyright, internet, time, course, Canadian, people, search, learn.Unfortunately this list does not provide conclusive hints to potentially trending topics, probably withthe exception of access and copyright. – 48 –
  • 55. Figure 30: Top 100 word stems appearing in 2011 blog entries of the top 20 blogs.5.3.2 TEL Blog ClustersThe blog network includes six communities / clusters, which were identified using the Louvain method[1]. The clusters are listed in Table 13 along with frequency of word stems appearing in the entries ofthose blogs in 2011. In total, there are 16,371 blog entries indexed during the year 2011 to date12.Clusters C1 and C2 have a negligible count of blog entries, and cluster C2 in particular is representedmostly by blogs in German language. This language-based clustering indicates that bloggers tend topoint to sources in their own language. Table 13: Clusters of TEL blogs indexed in Mediabase. Color in # Blogs / Entries Top 20 Word Stems in 2011 Figure 31 160 blogs (19.9%)C0 Yellow 6,865 entries (41.9%) 13 blogs (1.6%)C1 Gray 30 entries (0%) 43 blogs (5.3%)C2 Pink (German stop-words only) 302 entries (1.8%) 133 blogs (16.5%)C3 Green 5,389 entries (32.9%) 117 blogs (14.6%)C4 Blue 1,770 entries (7.1%) 150 blogs (18.7%)C5 Brown 1,447 entries (8.8%)12 Note that the blog network in Figure 29 does not include all indexed blogs, therefore the figures in Table 13 do not add up to 100%. – 49 –
  • 56. The largest cluster, C0, does not have any form of learning or education in its top 20 word stems,indicating that this cluster represents a set of general technology related blogs of some relevance to theTEL community. Those clusters with a topic focus closely related to TEL appear to be C3, C4, and C5,accounting for roughly 50% of all blogs and blog entries. Frequent word stems appearing (apart fromlearning-related word stems) in these clusters include among others: technolog, mobil, social, design,develop, share, open, peopl, and content. One interpretation of these terms could be an emphasis on“designing social and mobile technologies for people.” Particularly the design and mobile aspects canalso be found in prolific authors’ papers as discussed in Section 4.3.Figure 31 presents the TEL blog network in a colored style, where each color represents one of theclusters, the nodes size corresponds to the PageRank of the node, and the edge thickness correspondsto the frequency of links from one to the other node. Each node was colored using the color of itscluster; each edge was colored using the color of the source node’s cluster. The node labels representthe node’s identifiers in the database so that interested readers can query the database for details usingthe TEL-Map Mediabase Query Widgets (see Section 6). The layout was generated using Gephi’sForceAtlas2 layout algorithm. The visualization exposes a few nodes that stand out in terms ofPageRank, with most of the important nodes belonging to cluster C0, i.e. the technology cluster. 783 85 104 892 787 914 834 130 881 147 558 820 917 379 929 785 600 439 237 255 136 883 609 906 576 901 853 426 617 516 102 874 526 882 519 425 146 885 807 563 597 21 215 524 465 258 638 628 886 140 858 591 8 810 636 305 514 487 483 565 490 304 795 329 594 711 786 183 137 551 614 335 24 624 453 108 771 854 710 265 653 103 913 654 878 469 675 272 122 12 474 708 765 370 308 652 704 239 100 310 434 300 101 836 716 269 254 482 438 712 325 595 307 773 262 687 309 224 139 312 169 279 238 347 685 396 372 22 133 649 602 400 391 306 96 518 596 241 755 705 669 9 290 324 534 26302 106 132 264 843 292 422 489 1 119 420 721 501 857 127 598 242 682 171 275 772 87 257 278 703 494 358 174 639 713 86 927 577 221 504 23 267 220 135 593 192 17 180 109 13 123 375 832 440 113 776 105 557 507 462 442 251 791 415 170 252 283 599 322 219 825 177 83 16 615 678 701 666 261 298 110 376 459 928 15 260 240 466 151 346 365 554 556 571 92 293 491 263 228 157 536 680 664 281 81 750 424 756 112 646 722 90 707 578 303 650 532 806 443 414 797 93 297 209 32 78 629 403 473 764 910 668 547 395 448 362 207 6 80 189 168 301 214 194 550 658 826 82 815 689 700 359 352 273 356 266 831 125 402 360 570 529 344 409 89 196 326 749 193 661 696 657 181 253 327 502 803 922 419 676 690 809 695 729 693 176 166 11 655 107 460 68 588 590 345 575 75 27 686 684 284 548 407 515 580 569 73 416 688 484 216 586 456 814 620 585 579 271 14 179 229 198 204 244 601 38 702 706 447 374 446 512412 458 643 871 217 662 384 718 184 39 339218 282 581 382 248 366475 496 268 227 540 505 341 259 497 508 389 545 478 161 566 399 185 393 582 91 296 288 154 549 451 561 77 294 651 397 608 564 560 715 367 640 4 343 663 759 535 589 697 751 665 461 225 371 210 357 330 485 411 291 380 510 559 930 418 642 660 404 354 337 348 613 436 383 607 493 60 699 647 758 95 517 468 525 506 498 435 387 634 641 671 250 150 40 464 280 406 299 542 121 363 143 388 25 531 681 674 583 839 230 222 353 530 351 452 373 398 338 152 544 511 546 428 572 28 69 203 433 153 405 648 552 683 445 385 835 714 775 837 539 417 333 495 287 522 467 454 74 610 476 486 503 645 538 631 477 131 622 679 931 863 423 361 562 527 42 274 757 450 492 212 541 289 332 141 568 852 844 64 523 368 43 386 145 156 410 378 331 499 644 401 667 148 114 827 246 120 659 924 849 165 128 637 5 621 149 134 29 98 41 437 876 233 231 431 441 677 286 413 155 692 635 247 470 328 444 182 488 848 349 574 838 813 Figure 31: Colored TEL blog clusters. – 50 –
  • 57. 5.3.3 BurstsWe have analyzed the word bursts appearing in blog sources using a dynamic topic mining approach.As described in Section 5.1, a burst occurs when particular words are used very frequently by bloggers.The “power value” of the burst represents the word frequency. Burstiness can then be analyzed in thecontext of particular time windows. For this analysis, the word bursts for all sources were grouped byyear and analyzed for tendencies of rising and falling frequency. Standard word stemming proceduresand stop-word lists were applied.There are 2,640 word stems that started to appear in 2011 and never had bursts before 2011. A wordcloud of the top 100 of those is displayed in Figure 32. The words refer to highly diverse topics fromvarious areas such as politics (e.g. gaddafi, mubarak), tools (e.g. bitcoin, itwin), hardware (e.g.chromebook) conferences (e.g. acmmm11, itsc11), websites (e.g. bestcollegesonline.com), and so forth.Some bursts of relevance to TEL include, among others: • Screencastcamp: a screencastcamp is “a gathering of screencasters and visual communication aficionados wanting to network, learn, and collaborate on the art of screencasting. The event relies on the passion and creativity of the attendees—all sessions, discussions, and demos are led by attendees sharing their knowledge” [http://screencastcamp.com]. • MobiMOOC: A MOOC is a massive open online course, i.e. it involves many participants and is open to anyone who wants to join. [http://www.mooc.ca]; MobiMOOC adopts this concept in a mobile learning context. • Edcamp: The edcamp model is based on the […] BarCamp model, which is an open ad-hoc gathering for sharing and learning including demos, discussions, and a focus on interaction. In edcamps, sessions are not planned or scheduled until the morning of the event using a scheduling board on which attendees can place an index card with their session on it. [http://www.edutopia.org/blog/about-edcamp-unconference-history] • Studyboost: “provides [an] interactive social media studying platform to enable learning beyond the classroom. StudyBoost allows students to study by answering self- or teacher- prepared questions, at home or while on the go, via two of the most highly used technologies Figure 32: Bursty terms appearing only in 2011. – 51 –
  • 58. amongst students: SMS text messaging and instant messaging. For teachers, StudyBoost empowers them to use popular technology to further engage their students beyond the classroom or to integrate technology for learning into their plans. For students, StudyBoost makes true mLearning a reality by leveraging technology they are intimately familiar with and that they carry around everywhere.” [https://studyboost.com/about_us]These four bursts in 2011 all point in to a model of leveraging web technology (in particular: mobiletechnology) for an open and inclusive approach to education and learning.In a further step we identified bursty words that have been rising in power over the last three years.3,641 word stems match this criterion, of which the top one hundred are displayed as a word cloud inFigure 33. Again, the figure exposes many terms with no particular TEL provenance. However, thereare mentions of technologies and trends that are definitely relevant to TEL (and appearing to pointinto a similar direction as the bursts exclusively appearing in 2011 as presented above). For instance: • Screencastcamp (see above); • ds106 (an open online course on digital storytelling; see http://ds106.us and transmedia (often used in conjunction with storytelling); • mobilelearn, mlearncon, and tablet (as a mobile device). Figure 33: Bursty terms with rising frequency over the last three years.6 Embeddable Interactive Visualizations and QueriesAs a typical result of interacting with information visualization artifacts, the exploration of the social-network view on TEL-Map Mediabase artifacts such as papers, blogs and projects as presented inprevious sections will spawn more specific questions and exploratory tasks (cf. [5]). It may triggerwishes for “zooming into” the data by obtaining more detailed information on contained nodes andedges, e.g. a list of the top funded organizations or the projects with the highest rate of consortiumprogression to or from other projects, or all papers that have specific words in the abstract, etc. – 52 –
  • 59. To achieve interaction with the data based on specific queries arising during the exploratory process,we developed a web-based toolkit for interactive SQL query visualization, which was developedspecifically for end-users.The toolkit was implemented as a set of inter-communicating Google Gadgets, allowing users toconnect with TEL-Map Mediabase databases and query these databases using SQL. The result of aquery is immediately visualized by simply choosing one of the predefined types of visualization, i.e.tables (the typical result presentation in database query applications), bar and pie charts, timelines,and graphs. There are several configuration options for the more complex visualizations, particularlyfor graphs.For instance, the screenshot in Figure 34 shows two different visualizations of a query that obtains anordered list of pairs of organizations that are involved in the same FP7 project consortia, restricted topairs that appeared together in more than one project consortium—i.e. a subset of the partnershipgraph $ introduced in Section 3.4. The left side of the figure shows the query results visualized as atable. With two mouse clicks the same query results can be visualized as a graph. For constructing thegraph on the right-hand side of Figure 34 the first column of the query result is interpreted as thesource node, the second column as the target node, and the third column as the edge weight. Thelayout algorithm for the graph visualization, the meaning of columns, and other parameters can beconfigured easily using dropdown boxes (the configuration portion was clipped from the screenshotsin Figure 34. Figure 34: Visualization of the same SQL query as a table (left) and as a graph (right).Figure 35 shows an example of a timeline-based visualization. The first column of the result isautomatically interpreted as a timestamp, and all other result columns are plotted against that value.The top-right corner displays the values for the current point in time, where the user hovers with themouse cursor. The visualization in Figure 35 shows a per-year breakdown of average and maximumfunding per project in FP6 and FP7 as well as sum of funding and number of projects which started inthat year. – 53 –
  • 60. Figure 35: SQL query visualization as an annotated timeline.Furthermore, it is possible to formulate queries with certain filter parameters and have thevisualization react to changed filter values. For example one might only be interested in a sub-networkof $ that restricts the resulting visualization to immediate collaborators of one particular partner. Onselection of a partner, the gadget refreshes its visualization based on the new query and its results.Once the query author decides to share the current query and its visualization with the public, oneclick (on the “Send to Gadget Creator” button) will generate a custom gadget: A web service in thetoolkit’s backend generates the complete code for a Google Gadget which will display the query resultvisualization. Once published the resulting gadget can be embedded into any web page, e.g. into thepersonal iGoogle homepage. By combining multiple gadgets produced with this toolkit, it is thenpossible to arrange complete interactive web-based data dashboards, where stakeholders can have areal-time visual presentation of data that is interesting and relevant to them.Currently we have links to the query widgets available on the TEL-Map website (see the D4.3 resourcepage at http://telmap.org/?q=content/d4.3), allowing either to open the widgets in a normal browserwindow or to embed them in the personal iGoogle homepage. To remove SQL-related barriers andsimplify the setup of such “intelligence dashboards” for different stakeholders in TEL to facilitate thestakeholders in observing different variables of interest at a glance, we are currently working onenhanced visualizations and stakeholder-tailored sets of predefined and refinable queries on thedifferent data sources represented in the TEL-Map Mediabase. These are planned to be offered via theLearning Frontiers portal as embeddable widgets. – 54 –
  • 61. 7 Key Findings for Weak SignalsIn this section we present a list the key findings drawn from the analyses and results reported mainlyin Sections 3–5. Those key findings that are related to or have implications for European Commissionpolicy in TEL are highlighted with a bounding box. The key findings are grouped by the three datasources in the TEL-Map Mediabase.7.1 TEL Projects“Multi-Culturality” The list of the most prolific organizations in FP7 exposes the diversity of toporganizations both in structure and location. The top-five list includes a distance university, a researchcompany, a traditional university, topped by a psychology department at a technical university, alllocated in different countries. The topic distribution also shows that particularly in FP7 there are manyaspects in the project descriptions that are both genuinely related to human needs, with strongsupportability by technology (e.g. adaptation).North-South Axis. The more than 200 organizations that have participated or are participating inTEL projects in FP7 to date are mostly aligned on a North-South axis across Europe starting inScotland/England and the Nordic countries, via Netherlands, Belgium, West Germany, Switzerlandand Austria, and practically terminating in Northern Italy. The Southern and Eastern regions ofEurope and some prominent countries like France appear underrepresented. There is a hugedevelopment potential for the countries in these regions, and for the European Commission as well, interms of funding and contribution in future TEL calls.eContentplus as Gap Filler. While there are strong ties between FP6 and FP7 in terms ofparticipating organizations, it was demonstrated that eContentplus acted as a broker between FP6 andFP7 project consortia. Particularly some Best Practice Networks like ASPECT or ICOPER, and alsoTargeted Projects like OpenScout, have many strong consortium overlaps with both preceding FP6projects and succeeding FP7 projects. This pattern is probably simply due to the fact that in 2007 therewere neither new project launches in FP6 nor in FP7. On the other hand it could also be attributed to aplain “research follows money” attitude. That is, if there had not been funding from eContentplus,organizations would likely have looked for funding opportunities in TEL-related programmes withdifferent focus between 2006 and 2008. Anyway, eContentplus apparently was supportive and non-disruptive for the organizational collaboration network in European TEL, since eContentplus projectsare found in all identified project clusters, despite having a quite “narrow” topic focus around e-content and metadata issues.Role of Project Type. In the social network analysis of TEL projects it was revealed that IntegratedProjects (IPs), Networks of Excellence (NoEs) and eContentplus Best Practice Networks (BPNs) arethe most central projects, whereby this cannot solely be ascribed to the typically larger size of theconsortia of these projects compared to e.g. STREPs. For instance, these projects typically also includepairs of organizations that appear in list of most frequent collaborators. Also these projects have asignificant share of their consortium made of organizations that have a highly favorable centrality vs.clustering ratio. This indicates that IPs and NoEs are very important not only for shaping the researchagenda, but also for creating the strong and sustained collaboration ties between TEL organizations.The TEL Family. With every new TEL project, relatively fewer organizations are penetrating theexisting overall collaboration network in TEL projects. Over the last three years, an average of 40% ofthe consortia of new projects was not previously involved in any TEL projects. The sharpest drop inthis number occurred for projects that started in the year 2008 (from 62% to 42%), when the first FP7TEL projects were launched. It appears that at the transition to FP7, the project consortia—and theEuropean Commission—resorted to building on and funding an established core of organizations, thusstrengthening existing collaboration bonds; this will eventually lead to a tightly knit “family” of TEL – 55 –
  • 62. organizations, an inertial mass that can impede the involvement of new organizations, and likely alsonew ideas and research foci. This is strengthened by the fact that of the 34 launched TEL projects since2008, 4 out of 5 are being coordinated by organizations that have already participated in at least oneprevious TEL project. Of course, from the EC’s viewpoint it seems reasonable to fund projects where alarge share of the consortium have previous experience in EC-funded TEL projects. Still, this appearsto be a policy issue that requires attention.7.2 TEL PapersSolid European Base. European researchers are extremely well represented in the most importantTEL conferences and journals, as well as by social network metrics in their co-authorship network. Forinstance, of the 15 most prolific authors in TEL outlets 8 are currently based in Europe; the same ratioapplies to the list of 15 most betweenness-central authors in the co-authorship network of TEL outlets.“Under Construction.” The five TEL-related conferences, which were subject to scrutiny inSection 4, have developed constantly, although at a different pace. Comparing this pattern with that ofestablished conferences in a field—e.g. in sub-disciplines of computer science like databases, datamining, etc.—we found that the TEL conferences expose a development pattern that is typical of“young” and interdisciplinary conferences. This means that while there is significant core of 20-30% ofthe authors in the “inner circle”, there are also dozens or even hundreds of very small sub-networkswithin the co-authorship network that are disconnected from each other. Nevertheless, we see thatconferences and journals in TEL are building their community in a way that shapes a clear core. In thissense, maintaining and promoting key members who play the role of gate keepers is important.EC-TEL. The EC-TEL conference series and its (collocated) events can be considered as a highlysuccessful community in terms of development pattern of the co-authorship network and citationnetwork. EC-TEL managed to achieve a tight collaboration network and stable citation network afteronly few years, while other TEL related conferences are struggling with coherence in their community.This rapid community development is certainly being propelled by the strong ties in EU TEL projects,and it is certainly a strong piece of evidence for a high-impact initiative that has its roots in the FP6network of excellence, PROLEARN.Interdisciplinarity. TEL is an interdisciplinary field of research, which is both evident from theproject descriptions, core organizations involved in the projects, and also from the publications. Forinstance, the authors who published in important TEL journals Educational Technology & Society,Transactions on Learning Technology, and Computers & Education, also have papers published inmore than one thousand different conference series since 2005 in our data set. That is, the authors arecoming from a multitude of different (sub-)disciplines. In terms of community development,interdisciplinarity has pros and cons. On the one hand, it attracts researchers from different areas to aconference. On the other hand, it slows down the process of building the core, as we saw in thedevelopment pattern of more focused conferences like AIED, ITS and ECTEL, versus conferences likeICWL and ICALT, which have a broader topic focus. The most important conference series for TELauthors on a global scale authors is probably ICALT: in the events since 2005, ICALT has published alarge number of papers, and it is also an important venue for authors who frequently publish in TEL-related journals.Intelligent Systems. The key weak signals that emerged from analysis of papers by central authorsat three main TEL conferences include emotionally interactive learning systems, intelligent tutoringand exploitation of sensor technologies and mobile devices for authentic learning activities. As saidabove in a different context, the European Commission demonstrated great sensibility for these weaksignals, since they are well represented in the target outcomes of the TEL objective in the upcomingICT Call 8, which include “TEL systems endowed with capabilities of human tutors” in an affective andpersonalized way, as well as “tools for fostering creativity and non-standard thinking”. – 56 –
  • 63. 7.3 TEL Social WebTechnology Sources as Authorities. From the analysis of the blog network, it became evident thatthe largest cluster of blogs being pointed to from entries in the TEL blogosphere are related totechnology news and trends in general and not to TEL in particular; TEL related blogs can be found inthree separate clusters, with one focusing on schools, and the other two focusing on learning,education, and technology aspects. Two of the TEL related clusters are rather hard to distinguish interms of content; more detailed analyses will have to be performed to identify what differentiates theseclusters.Open and Inclusive Education. The word bursts that have appeared only in 2011, as well as thosewith a rising frequency of occurrence in the last three years, indicate a clear emphasis in the mostcentral blog sources on leveraging web technology—in particular mobile technology—for open andinclusive approaches to education and learning, e.g. through MOOCs: massive open online courses.This emphasis could be an interesting thread to follow up in European R&D policy.8 ConclusionTEL-Map aims at providing the European Commission and other TEL stakeholders with detailedinformation (trend-indicators, statistics, analyses, policy implications) on the current and future TELlandscape. This deliverable reported first analyses and results obtained in WP4 involving theapplication of social network analysis and topic mining in several TEL data sources. For each source,the formal foundation for social network analysis was laid to enable a graph-based and network-basedanalysis of the current TEL landscape. In addition, stakeholders are facilitated in interacting with theTEL data sets and analysis results using intuitive, embeddable tools and flexible analyses. This willsupport stakeholders in perceiving, understanding and reasoning about complex data sources in TEL,which is one of the main goals of visual analytics [22].There are several limitations in the current data sources and the analyses, which will have to beaddressed in forthcoming WP4 work:Firstly, the scientific publications indexed in the TEL Papers database are not fully representative ofcurrent trends and current work. Also many highly successful innovative technologies and practices donot necessarily come with scientific papers. For instance, there was probably no scientific publicationthat introduced micro-blogging in education before Twitter was taken up by early-adopting teachers.In addition to that, our main data source is DBLP, which presents itself as a computer sciencebibliography. Although many important TEL venues are represented in DBLP, it only represents asmall subset of potentially relevant venues. In future work we plan to include additional sources.However, many conference and workshop publications are not indexed in any publicly accessiblebibliography, therefore presenting a serious obstacle to maintaining a comprehensive TELpublications database.Secondly, the blogosphere indexed in the TEL Social Media database have been collected with directinvolvement of the community (e.g. through the Mediabase Commander plugin for Firefox); thisdatabase does not constitute a centrally controlled set of sources. Therefore there are many sourcesthat are not explicitly or immediately related to TEL. For upcoming work we plan to come up with adata cleansing and sources filtering strategy to increase the focus on TEL.Finally, the TEL projects dataset exclusively contains FP6, FP7 and eContentplus projects relevant toTEL. There are many additional sources and projects that could be included, e.g. the Lifelong LearningProgramme, additional projects from the EC Policy Support Programme, the UK JISC funded projects,and many more. Also, we currently have descriptive metadata on the projects only; that is, we do nothave project deliverables in the database (which would significantly increase the text corpus for data – 57 –
  • 64. mining), nor do we have a unified, integrated view on persons as paper authors, as bloggers, and asmembers of organizations participating in TEL projects.By next year several new TEL projects will be funded in FP7 ICT call 8, thousands of new TEL-relatedpapers will be published, and the blogosphere indexed in the TEL-Map Mediabase will continue togrow to reflect those changes and subsequent changes. There is a dedicated page on the TEL-Mapwebsite [http://telmap.org/?q=content/d4.3] where the up-to-date data can be accessed, obtained andqueried using embeddable query visualization widgets. In addition, the core results of the first analysesreported in this deliverable, as well as upcoming analyses with extended and enhanced data sets andresults will be published as stories on the Learning Frontiers portal and accessible via the TEL-Mapwebsite.References1. Blondel, V.D., Guillaume, J.-L., Lambiotte, R., Lefebvre, E.: Fast unfolding of communities in large networks. Journal of Statistical Mechanics: Theory and Experiment 2008, P10008 (2008)2. Brandes, U., Erlebach, T. (eds.): Network Analysis: Methodological Foundations. Springer (2005)3. European Commission: FP7 in Brief. http://is.gd/fp7brief (2007)4. European Commission: Community Research and Development Information Service (CORDIS). http://cordis.europa.eu/home_en.html5. Fekete, J.-D., van Wijk, J., Stasko, J., North, C.: The Value of Information Visualization. In: Kerren, A., Stasko, J., Fekete, J.-D., North, C. (eds.) Information Visualization, pp. 1-18. Springer Berlin / Heidelberg (2008)6. Hiltunen, E.: Good Sources of Weak Signals: A Global Study of Where Futurists Look For Weak Signals. Journal of Futures Studies 12(4), 21–43 (2008)7. Keim, D., Andrienko, G., Fekete, J.-D., Görg, C., Kohlhammer, J., Melançon, G.: Visual Analytics: Definition, Process, and Challenges. In: Kerren, A., Stasko, J., Fekete, J.-D., North, C. (eds.) Information Visualization, pp. 154-175. Springer Berlin / Heidelberg (2008)8. Kienle, A., Wessner, M.: Analysing and cultivating scientific communities of practice. International Journal of Web Based Communities 2(4), 377 - 393 (2006)9. Klamma, R., Cao, Y., Spaniol, M.: Watching the Blogosphere: Knowledge Sharing in the Web 2.0. In: Nicolov, N., Glance, N., Adar, E., Hurst, M., Liberman, M., Martin, J.H., Salvetti, F. (eds.) Proceedings of the 1st International Conference on Weblogs and Social Media, Boulder, Colorado, USA, March 26-28, 2007, pp. 105 – 112 (2007)10. Klamma, R., Spaniol, M., Cao, Y., Jarke, M.: Pattern-Based Cross Media Social Network Analysis for Technology Enhanced Learning in Europe. In: Nejdl, W., Tochtermann, K. (eds.) Innovative Approaches for Learning and Knowledge Sharing, pp. 242-256. Springer Berlin Heidelberg, Berlin, Heidelberg (2006)11. Klamma, R., Spaniol, M., Denev, D.: PALADIN: A Pattern Based Approach to Knowledge Discovery in Digital Social Networks. Proceedings of I-KNOW ’06, 6th International Conference on Knowledge Management, pp. 457-464. Graz, Austria (2006)12. Kleinberg, J.M.: Hubs, authorities, and communities. ACM Computing Surveys 31, 5-es (1999)13. McCallum, A., Nigam, K., Ungar, L.H.: Efficient clustering of high-dimensional data sets with application to reference matching. Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining - KDD ’00, pp. 169-178. Boston, Massachusetts, United States (2000)14. Milgram, S.: The small world problem. Psychology Today 2, 60-67 (1967) – 58 –
  • 65. 15. Ochoa, X., Méndez, G., Duval, E.: Who We Are: Analysis of 10 Years of the ED-MEDIA Conference. World Conference on Educational Multimedia, Hypermedia and Telecommunications 2009, pp. 189-200 (2009)16. Olivier, B. (ed.): TEL-Map Deliverable D1.2: Conceptual Framework for Dynamic Roadmapping. http://is.gd/telmapD12 (2011)17. Page, L., Brin, S., Motwani, R., Winograd, T.: The PageRank Citation Ranking: Bringing Order to the Web. http://ilpubs.stanford.edu:8090/422/ (1999)18. Petrushyna, Z., Klamma, R.: No Guru, No Method, No Teacher: Self-classification and Self- modelling of E-Learning Communities. In: Dillenbourg, P., Specht, M. (eds.) Times of Convergence. Technologies Across Learning Contexts, pp. 354-365. Springer Berlin Heidelberg, Berlin, Heidelberg19. Pham, M.C., Klamma, R.: The Structure of the Computer Science Knowledge Network. 2010 International Conference on Advances in Social Networks Analysis and Mining (ASONAM), pp. 17-24. IEEE (2010)20. Pham, M.C., Klamma, R., Jarke, M.: Development of Computer Science Disciplines - A Social Network Analysis Approach. http://arxiv.org/abs/1103.1977 (2011)21. Reinhardt, W., Meier, C., Drachsler, H., Sloep, P.: Analyzing 5 years of EC-TEL proceedings - Who we are and what we publish. Proceedings of Sixth European Conference on Technology Enhanced Learning. Springer LNCS, Palermo, Italy (2011)22. Thomas, J.J., Cook, K.A. (eds.): Illuminating the Path: The Research and Development Agenda for Visual Analytics. IEEE Computer Society (2005)23. Voigt, C., Pirkkalainen, H. (eds.): TEL-Map Deliverable D4.1: Report on Weak Signals Collection. http://telmap.org/?q=content/deliverables (2011)24. Wang, X., Jin, X., Chen, M., Zhang, K., Shen, D.: Topic Mining over Asynchronous Text Sequences. IEEE Transactions on Knowledge and Data Engineering PrePrint, 1-15 (2010)25. Wasserman, S., Faust, K.: Social Network Analysis: Methods and Applications. Cambridge University Press (1994)26. Wild, F., Haley, D., Bülow, K.: CONSPECT: Monitoring Conceptual Development. In: Luo, X., Spaniol, M., Wang, L., Li, Q., Nejdl, W., Zhang, W. (eds.) Advances in Web-Based Learning – ICWL 2010, pp. 299-308. Springer Berlin Heidelberg, Berlin, Heidelberg (2010)27. yWorks: Organic Layout Style. http://is.gd/yedorganic28. yWorks: yEd Graph Editor. http://yworks.com/yed29. GaLA Home Page. http://www.galanoe.eu/ – 59 –
  • 66. Appendix A: TEL Projects — TimelineThis figure shows a timeline of TEL projects funded under FP6, FP7, and eContentplus. 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 KALEIDOSCOPE (FP6) PROLEARN (FP6) TELCERT (FP6) ICLASS (FP6) UNFOLD (FP6) LEACTIVEMATH (FP6) CONNECT (FP6) E-LEGI (FP6) EMAPPS.COM (FP6) ICAMP (FP6) I-MAESTRO (FP6) MGBL (FP6) VEMUS (FP6) CALIBRATE (FP6) ARGUNAUT (FP6) ATGENTIVE (FP6) COOPER (FP6) LEAD (FP6) LT4EL (FP6) PROLIX (FP6) TENCOMPETENCE (FP6) RE.MATH (FP6) ARISE (FP6) ELU (FP6) KP-LAB (FP6) PALETTE (FP6) UNITE (FP6) LOGOS (FP6) APOSDLE (FP6) ECIRCUS (FP6) ELEKTRA (FP6) L2C (FP6) JEM (eContentplus) MACE (eContentplus) MELT (eContentplus) CITER (eContentplus) EdReNe (eContentplus) eViP (eContentplus) KeyToNature (eContentplus) Organic.Edunet (eContentplus) Intergeo (eContentplus) EUROGENE (eContentplus) COSMOS (eContentplus) GRAPPLE (FP7) LTFLL (FP7) SCY (FP7) MATURE (FP7) 80DAYS (FP7) IDSPACE (FP7) EduTubePlus (eContentplus) iCOPER (eContentplus) ASPECT (eContentplus) TARGET (FP7) INTELLEO (FP7) STELLAR (FP7) DYNALEARN (FP7) ROLE (FP7) COSPATIAL (FP7) XDELIA (FP7) mEducator (eContentplus) Math-Bridge (eContentplus) LiLa (eContentplus) OpenScienceResources (eContentplus) OpenScout (eContentplus) ALICE (FP7) ARISTOTELE (FP7) MIRROR (FP7) METAFORA (FP7) MIROR (FP7) NEXT-TELL (FP7) ITEC (FP7) ECUTE (FP7) SIREN (FP7) IMREAL (FP7) GALA (FP7) TERENCE (FP7) TEL-MAP (FP7) – 60 –
  • 67. Appendix B: TEL Projects — SNA MetricsThis table lists the 77 projects in our data set including SNA metrics (ordered by project acronym). Progr- Start Page- Weighted Closeness Betweenness Local Clust- PROJECT Cluster Authority Hub Degree amm e Year Rank Degree Centrality Centrality ering Coeff.80DAYS FP7 2008 1 .0115 [28] .0121 [26] .0087 [52] 13 [51] 18 [49] .5315 [48] .0016 [52] .6667 [20]ALICE FP7 2010 0 .0102 [37] .0106 [32] .0076 [57] 11 [53] 14 [58] .5241 [52] .0015 [54] .7091 [18]APOSDLE FP6 2006 0 .0115 [28] .0106 [32] .0129 [26] 22 [22] 26 [29] .5672 [26] .0208 [13] .5238 [50]ARGUNAUT FP6 2005 4 .0089 [40] .0091 [37] .0123 [29] 22 [22] 27 [28] .5758 [23] .0082 [25] .5455 [46]ARISE FP6 2006 4 .0064 [53] .0076 [44] .0060 [63] 11 [53] 11 [61] .5067 [62] .0004 [66] .8545 [10]ARISTOTELE FP7 2010 4 .0115 [28] .0000 [75] .0059 [64] 8 [65] 10 [63] .4967 [67] .0008 [63] .5357 [48]ASPECT eContentplus 2008 5 .0281 [8] .0302 [7] .0246 [9] 30 [12] 59 [9] .6179 [12] .0284 [8] .5034 [53]ATGENTIVE FP6 2005 2 .0000 [73] .0015 [69] .0031 [73] 3 [72] 3 [72] .4153 [73] .0000 [72] 1.0000 [1]CALIBRATE FP6 2005 5 .0064 [53] .0060 [51] .0135 [23] 17 [34] 30 [23] .5507 [33] .0064 [31] .4485 [66]CITER eContentplus 2006 1 .0102 [37] .0106 [32] .0055 [68] 9 [62] 9 [67] .5033 [66] .0015 [53] .5000 [56]CONNECT FP6 2004 4 .0051 [57] .0045 [55] .0145 [20] 20 [26] 32 [21] .5630 [27] .0081 [26] .5158 [51]COOPER FP6 2005 3 .0051 [57] .0060 [51] .0111 [37] 16 [38] 26 [29] .5352 [45] .0009 [62] .8250 [12]COSMOS eContentplus 2007 4 .0115 [28] .0121 [26] .0119 [31] 15 [44] 25 [32] .5429 [38] .0050 [37] .5429 [47]COSPATIAL FP7 2009 0 .0089 [40] .0091 [37] .0058 [65] 9 [62] 10 [63] .5067 [62] .0011 [59] .6389 [27]DYNALEARN FP7 2009 1 .0089 [40] .0091 [37] .0056 [67] 8 [65] 9 [67] .5067 [62] .0020 [50] .4286 [70]ECIRCUS FP6 2006 0 .0038 [62] .0030 [61] .0087 [51] 10 [58] 15 [55] .5101 [60] .0017 [51] .6444 [25]ECUTE FP7 2010 0 .0064 [53] .0060 [51] .0065 [60] 6 [69] 10 [63] .4872 [69] .0003 [69] .8000 [14]EdReNe eContentplus 2007 5 .0000 [73] .0015 [69] .0027 [77] 2 [74] 2 [74] .3938 [74] .0000 [72] 1.0000 [1]EduTubePlus eContentplus 2008 1 .0077 [48] .0076 [44] .0089 [49] 12 [52] 18 [49] .5278 [51] .0041 [41] .4697 [61]E-LEGI FP6 2004 2 .0051 [57] .0045 [55] .0184 [13] 34 [9] 42 [13] .6441 [8] .0365 [5] .4385 [68]ELEKTRA FP6 2006 1 .0077 [48] .0076 [44] .0082 [55] 11 [53] 16 [54] .5170 [56] .0014 [55] .6364 [29]ELU FP6 2006 5 .0102 [37] .0106 [32] .0104 [40] 18 [31] 22 [37] .5588 [30] .0091 [24] .4444 [67]EMAPPS.COM FP6 2005 5 .0026 [69] .0030 [61] .0076 [58] 7 [68] 15 [55] .4634 [71] .0001 [71] .8571 [8]EUROGENE eContentplus 2007 2 .0153 [23] .0166 [21] .0126 [28] 22 [22] 29 [24] .5802 [21] .0058 [32] .6667 [20]eViP eContentplus 2007 0 .0077 [48] .0076 [44] .0055 [69] 9 [62] 9 [67] .5101 [60] .0011 [58] .6389 [27]GALA FP7 2010 0 .0536 [1] .0619 [1] .0332 [4] 42 [3] 79 [5] .6847 [4] .0580 [3] .3449 [75]GRAPPLE FP7 2008 3 .0217 [14] .0242 [11] .0278 [7] 35 [8] 69 [7] .6333 [9] .0220 [11] .4571 [64]ICAMP FP6 2005 2 .0038 [62] .0045 [55] .0110 [38] 15 [44] 25 [32] .5315 [48] .0030 [43] .5524 [43]ICLASS FP6 2004 1 .0038 [62] .0030 [61] .0155 [17] 24 [19] 35 [17] .5846 [20] .0151 [19] .4783 [59]iCOPER eContentplus 2008 2 .0332 [6] .0378 [5] .0347 [3] 39 [5] 91 [3] .6667 [7] .0218 [12] .4791 [58]IDSPACE FP7 2008 3 .0153 [23] .0166 [21] .0094 [46] 17 [34] 21 [40] .5507 [33] .0010 [61] .8235 [13]I-MAESTRO FP6 2005 0 .0000 [73] .0015 [69] .0028 [75] 2 [74] 2 [74] .3838 [76] .0000 [72] 1.0000 [1]IMREAL FP7 2010 0 .0230 [13] .0242 [11] .0106 [39] 18 [31] 20 [43] .5547 [32] .0152 [18] .5294 [49]INTELLEO FP7 2009 2 .0089 [40] .0091 [37] .0062 [61] 10 [58] 12 [60] .5170 [56] .0004 [67] .8000 [14]Intergeo eContentplus 2007 1 .0115 [28] .0136 [23] .0091 [48] 14 [49] 18 [49] .5241 [52] .0053 [35] .5934 [36]ITEC FP7 2010 5 .0255 [9] .0287 [8] .0164 [16] 22 [22] 37 [16] .5758 [23] .0174 [14] .5065 [52]JEM eContentplus 2006 2 .0026 [69] .0030 [61] .0028 [76] 2 [74] 2 [74] .3858 [75] .0000 [72] 1.0000 [1]KALEIDOSCOPE FP6 2004 0 .0000 [73] .0015 [69] .0612 [1] 61 [1] 150 [1] .8352 [1] .1630 [1] .2732 [76]Key ToNature eContentplus 2007 5 .0128 [25] .0121 [26] .0116 [33] 20 [26] 22 [37] .5758 [23] .0287 [7] .6474 [24]KP-LAB FP6 2006 2 .0089 [40] .0091 [37] .0113 [35] 16 [38] 22 [37] .5429 [38] .0136 [23] .4000 [73]L2C FP6 2006 2 .0089 [40] .0076 [44] .0085 [53] 16 [38] 17 [53] .5507 [33] .0046 [39] .8333 [11]LEACTIVEMATH FP6 2004 1 .0051 [57] .0045 [55] .0100 [41] 15 [44] 21 [40] .5352 [45] .0025 [46] .6952 [19]LEAD FP6 2005 2 .0038 [62] .0030 [61] .0072 [59] 10 [58] 14 [58] .5135 [58] .0012 [57] .6667 [20]LiLa eContentplus 2009 1 .0115 [28] .0121 [26] .0061 [62] 10 [58] 11 [61] .5205 [54] .0014 [56] .6444 [25]LOGOS FP6 2006 1 .0026 [69] .0030 [61] .0031 [72] 3 [72] 3 [72] .4294 [72] .0000 [72] 1.0000 [1]LT4EL FP6 2005 2 .0064 [53] .0060 [51] .0114 [34] 19 [30] 26 [29] .5630 [27] .0024 [48] .7485 [16]LTFLL FP7 2008 2 .0217 [14] .0242 [11] .0167 [15] 27 [15] 41 [15] .5984 [15] .0066 [28] .6325 [30]MACE eContentplus 2006 3 .0179 [18] .0211 [14] .0195 [12] 33 [10] 48 [12] .6333 [9] .0158 [16] .5000 [56]Math-Bridge eContentplus 2009 3 .0293 [7] .0332 [6] .0153 [18] 26 [16] 35 [17] .5891 [16] .0159 [15] .5508 [45]MATURE FP7 2008 0 .0089 [40] .0091 [37] .0097 [43] 16 [38] 19 [45] .5390 [41] .0041 [42] .5750 [39]mEducator eContentplus 2009 2 .0255 [9] .0287 [8] .0133 [24] 24 [19] 28 [27] .5891 [16] .0232 [10] .5652 [41]MELT eContentplus 2006 5 .0128 [25] .0136 [23] .0139 [22] 20 [26] 32 [21] .5588 [30] .0052 [36] .5842 [37]METAFORA FP7 2010 1 .0191 [16] .0211 [14] .0095 [45] 16 [38] 19 [45] .5390 [41] .0047 [38] .5833 [38]MGBL FP6 2005 5 .0000 [73] .0015 [69] .0028 [74] 1 [77] 2 [74] .3671 [77] .0000 [72] .0000 [77]MIROR FP7 2010 1 .0077 [48] .0076 [44] .0051 [70] 6 [69] 8 [70] .4967 [67] .0005 [65] .6667 [20]MIRROR FP7 2010 0 .0255 [9] .0287 [8] .0127 [27] 24 [19] 29 [24] .5802 [21] .0057 [33] .6123 [34]NEXT-TELL FP7 2010 0 .0179 [18] .0196 [17] .0099 [42] 15 [44] 20 [43] .5315 [48] .0030 [44] .6000 [35]OpenScienceResources eContentplus 2009 4 .0166 [21] .0181 [19] .0117 [32] 14 [49] 25 [32] .5390 [41] .0023 [49] .6154 [33]OpenScout eContentplus 2009 2 .0459 [2] .0514 [2] .0293 [6] 39 [5] 75 [6] .6726 [5] .0347 [6] .4656 [62]Organic.Edunet eContentplus 2007 4 .0128 [25] .0136 [23] .0119 [30] 17 [34] 25 [32] .5390 [41] .0055 [34] .4779 [60]PALETTE FP6 2006 1 .0089 [40] .0091 [37] .0077 [56] 11 [53] 15 [55] .5205 [54] .0024 [47] .4182 [71]PROLEARN FP6 2004 1 .0026 [69] .0015 [69] .0440 [2] 50 [2] 114 [2] .7451 [2] .0641 [2] .3682 [74]PROLIX FP6 2005 1 .0115 [28] .0121 [26] .0213 [10] 29 [13] 53 [10] .6080 [13] .0146 [20] .5025 [55]RE.MATH FP6 2005 1 .0038 [62] .0030 [61] .0093 [47] 11 [53] 19 [45] .5135 [58] .0030 [45] .4364 [69]ROLE FP7 2009 2 .0370 [5] .0423 [4] .0272 [8] 39 [5] 69 [7] .6726 [5] .0279 [9] .4521 [65]SCY FP7 2008 4 .0179 [18] .0196 [17] .0129 [25] 20 [26] 29 [24] .5630 [27] .0066 [27] .5579 [42]SIREN FP7 2010 0 .0077 [48] .0076 [44] .0049 [71] 6 [69] 7 [71] .4872 [69] .0006 [64] .7333 [17]STELLAR FP7 2009 2 .0395 [4] .0453 [3] .0318 [5] 42 [3] 81 [4] .6909 [3] .0385 [4] .4146 [72]TARGET FP7 2009 0 .0115 [28] .0121 [26] .0112 [36] 17 [34] 23 [36] .5468 [37] .0065 [29] .5515 [44]TELCERT FP6 2004 3 .0038 [62] .0030 [61] .0142 [21] 25 [17] 34 [20] .5891 [16] .0065 [30] .6267 [31]TEL-MAP FP7 2010 2 .0408 [3] .0000 [75] .0204 [11] 31 [11] 51 [11] .6230 [11] .0144 [21] .5032 [54]TENCOMPETENCE FP6 2005 3 .0115 [28] .0106 [32] .0176 [14] 25 [17] 42 [13] .5891 [16] .0141 [22] .5700 [40]TERENCE FP7 2010 5 .0242 [12] .0000 [75] .0096 [44] 18 [31] 21 [40] .5429 [38] .0043 [40] .6209 [32]UNFOLD FP6 2004 3 .0038 [62] .0045 [55] .0083 [54] 15 [44] 18 [49] .5352 [45] .0004 [68] .9048 [6]UNITE FP6 2006 4 .0191 [16] .0211 [14] .0152 [19] 28 [14] 35 [17] .6080 [13] .0157 [17] .4630 [63]VEMUS FP6 2005 4 .0051 [57] .0045 [55] .0057 [66] 8 [65] 10 [63] .5067 [62] .0001 [70] .8571 [8]XDELIA FP7 2009 2 .0166 [21] .0181 [19] .0088 [50] 16 [38] 19 [45] .5507 [33] .0010 [60] .8583 [7] – 61 –