• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Scratchpads: A standard implementation using Drupal

Scratchpads: A standard implementation using Drupal



An invited presentation at the National Biodiversity Network technical conference held in the Natural History Museum, London, UK. November 19, 2009.

An invited presentation at the National Biodiversity Network technical conference held in the Natural History Museum, London, UK. November 19, 2009.



Total Views
Views on SlideShare
Embed Views



2 Embeds 15

http://vsmith.info 12
http://www.slideshare.net 3



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • Going to give a introduction to the Scratchpad project which is based at the NHM, but part of the EDIT iniatitiave. Specifically I’m going to: Provide sound background on the origins of the Scratchpad project Say a little about Drupal - the content management system on which the Scratchpads are based Give a technical overview of how the Scratchpads work and how we manage them. Run through the functionalities of the Scratchpads And finally give you some perspectives on where the project is going and what the future is for the Scratchpad iniative.
  • Before I do this let me say a few words about the case for the Scratchpads: Why do we need them. - Taxonomy / biodiversity studies have lots of big questions which require large aggregations of data to address. Issue like climate change, changing patterns of land use, protection of food supplies, ecosystem services etc.
  • The problem is that the way we do taxonomy doesn’t make it possible to address these questions. There is no collective focus for the discipline. Instead we are all specilaists in our particyular field. We need some way of connecting the very small process of doing taxonomic research, with the very grand ambitions we have for the discipline as the founcatiion for much of biological science. And no matter how we do this, we have to do it in a way that functions both technically (providing some way of aggreggating all the diverse inputs into taxonomy) and sociologically, that fits the practical ways in which most taxonmists work.
  • The way that the NHM have tackled this problem is perhaps typical of many other institutions. Within the NHM we have a small team of people that are very good at building carefully managed websites. These people undertand the technology required to do this, and though a carefully managed process of permissions, various website get built - you can see some of them here. This process has a fer advanatages. The sites are often very high quality…However, there are alo many problems. The authors of the content are quite removed from the process of building the website. Consequently they have very limited control of what actually appears and how it apprears….With out doubt the biggest problem is that the process of building these sites is enormously protracted and slow. New site appear at the rate of about three of four a year, sometime less.
  • This is for various reasons but principally it is because the way the project is managed. Every year the NHM’s web development team produce a huge list of all the projects on the waiting list which they planed to do in the coming year, and I was always struck by the fact that the lists would never grow in length - the number of science projects was always more or less static. And if you examined this list close up you reaslise it was because a number of projects were no longer relavenbt because the authors had either died on the waiting list, or had subsequently left the museum. To my eyes there are two basic bottlenecks when it comes to building these sites. The first is the fact that the authors had to go to a group of people that had the technical proficiency to build the contect. The second is that the processes is so tightly managed that getting permission to add or alter content on the web was exceedingly difficult.
  • Anyone outside the NHM cannot have failed to notice the explosion of the content that has appreard on the web in the past few years. This is largly because these tools have blown away the technical barriers associted with publishing contenbt on the web. And the management barriers associted with getting permission to publish content. Most of these tools are very specific - allowing users to do one particular thing (perhaps publish a snipplet of text like twitter, or say an image on a photosharing site like flickr). But some tools are highly generic.
  • These tools - often called content management systems, are highly generic and enable user to very rapidly build customised websites with a very generic set of functions tailed to the needs of a particular audience. Four of the most popular sites are listed here, and they all provide basic web publishing functionality. Probably the most popular of these, and the one that we selected to use for the Scratchpads is Drupal.
  • Drupal is a very popular CMS syetm. It’s the foundation for many the websites of many universities, colleages, libraries, societies and even goivernment institutions. It has a very active developer community, som businesess entiely rely on Drupal, and arguably provides a much more sustainabily platform that some of the other lesser know CMS systems. One of the principle reasons why its so popular is that it is particlarly focused on community management of content. The social process needed to mediate the publication of content on the web are built in to Drupal very effectively.
  • This has been indipendentlally recognised in a number of studes comparing the functions of CMS systems. For example this report by IdealWare was published early this year, and rates Drupal very highly. Especially in the area of community functionality.
  • So Drupal was selected to be the foundation for the Scratchpad project. It supports the central management of multiple sites, providing us with a mechanism to supply customised community based websites to anyone that requests a site. THis is the home page of the Scratchpad project. Users can access a variety of resources from here. In parctiuclar they can also apply for a new site.
  • Fill in form…
  • Get a template empty site that they can populate and add new users. The Scratchpads have been built to support a wide range of activities required by taxonomists. These functions are easy to navigate, helping people to construct their sites quickly and easily. Crucially users and theme and focus their site on one or more of a thange of activities. This is crucial to building a sense of community amongst the userbase.
  • The project has been running since March 2007 and in that time we have built up…These form an ecosystem of different communities working toward their own goals in a self managed environment. A high proportion of the sites fail - Probably about 80% have relativly little content. The system is construicted to accomdate failure.
  • Some use cases for the Scratchpads are illustrated on this rather elaborate slide here….
  • From a technical perspective Scratchpad sit on a single Virtual Machine in the NHM server room. Currently they occypy about 100 GB of space. The server architecture is based on a LAMP configuration. Its all backed up using Tivoli storage manager. Eventually these go to tape as part of the NHM’s storage management and are picked up by a courrier and stored off site. The marginal costs of creating a new site are exceptionally low - probably about £3 per site, taking into account storage and staff time.
  • Developers site with instructions on how to set up your own Scratchpad server. You can also access individual modules from here. Functionality is partitions into modules of code which with with the Drupal core. Moduels that are not

Scratchpads: A standard implementation using Drupal Scratchpads: A standard implementation using Drupal Presentation Transcript

  • Vincent S. Smith & Simon Rycroft Scratchpads A standard implementation using Drupal
  • Macro taxonomy The foundation for biodiversity research
    • Inventory the Earth’s species
    • Document their relationships
    • “ Publish” & apply these data
    • 1.8 M described spp. (10M names)
    • 300M pages (over last 250 years)
    • 1.5-3B specimens
    Data set… People…
    • 4-6,000 scientists
    • 30-40,000 “pro-amateurs”
    • Many more citizen scientists?
  • Micro taxonomy The process of taxonomic research
    • Parochial
    • Specialised experts
    • Fragmented & distributed
    • Different (domain specific)
    • Communities of practice
    • Non transferable skills
    Methodology… Output…
    • Heterogeneous & scattered
    • High volume, low impact
    • Hard to find (use)
  • Managed Web Support
    • Pros
    • High quality
    • Bespoke
    • Branded*
    • Cons
    • Limited control
    • Limited updates
    • NHM centric
    • No services
    • No ownership
    • No connectivity
    • Protracted & slow
    The NHM experience Lepidoptera Solanaceae Chalcid Wasps Spanish Sandflies British Fleas Martian Meteorites Loch Ness Worms Bumble bees
  • Managed Web Support The NHM experience
    • Web Bottlenecks
    • Technology
    • Management
  • Explosion of Web 2.0 Solutions to technical and management bottlenecks
  • Content Management Systems Provide the basic web publishing functions Joomla WordPress Drupal Plone
  • Drupal’s strength - “community” Support the social processes of web publishing
  • Drupal - social networking Comparing CMS systems http://www.idealware.org/comparing_os_cms/
  • Scratchpads A multi-site implementation of Drupal http://scratchpads.eu
  • Scratchpad Application Form Free to any individual or community
  • Flexible Template Site With easy navigation
  • An ecosystem of communities Sites: 130+ Users: 1500+ Pages: 170k Circa 2 new sites/wk. March 2007
  • Scratchpad Technical Overview
  • Scratchpad management Scalable & sustainable technology One virtual machine, open-source, self-archiving, backed-up, multi-site configuration ( easy to move & upgrade, secure & reliable, screencasts, low admin., low marginal costs ) Hardware, software & user support
  • Scratchpad Developers Site http://home.scratchpads.eu/
  • Scratchpad SVN http://svn.scratchpads.eu/viewvc
  • Scratchpad Object Model http://scratchpads.eu/sites/scratchpads.eu/files/Scratchpad_Object_Model.pdf
  • Scratchpad Functionality Summary
  • Taxonomy Taxonomy import, management and navigation
  • Bibliographic data Reference manager / Endnote support for bibliographies
  • Images Image galleries, image upload & annotation
  • Phylogeny Nexus / Newick import for visualizing phylogenies
  • Character Matricies Molecular & morphological character matricies (discrete, morphometric and text characters)
  • Distribution Maps Presence absence country maps
  • Specimens & locations Specimen & location records (DwC)
  • Pages, Forums, Blogs, Newsletters Web fora with e-mail integration User blogs Static web pages Newsletters with e-mail integration
  • Mass Import Import from CSV text file to any content type
  • Create & switch between content in any language Multilingual Support
  • Administration Simple & advanced admin. AutoTag content (automatically adds structure) Private Groups Customized user profiles
  • Dynamically built species pages Integrating data & “publishing” in a Scratchpad Taxon Pages
  • Browsed through a taxonomy Integrating data & “publishing” in a Scratchpad Taxon Pages
  • Including 3 rd party content Integrating data & “publishing” in a Scratchpad Taxon Pages
  • With data curation tools Integrating data & “publishing” in a Scratchpad Taxon Pages
  • Listing all “authors” Integrating data & “publishing” in a Scratchpad Taxon Pages
  • Dated, permanent & citable Integrating data & “publishing” in a Scratchpad Taxon Pages
  • Control what to display Adjusting the layout Taxon Pages n.b. Taxon page tools have since been updated
  • User Support
  • Sandbox Site
  • Uservoice
  • Screencasts
  • Future Directions
  • BMC Bioinformatics BMC Bioinformatics 2009, 10 (Suppl 14) S6doi:10.1186/1471-2105-10-S14-S6 Published: 10 November 2009
  • ViBRANT INFRA-2010-1.2.3 (VRC) 17 Partners Approx €4M
    • Distribute hosting infrastructure
    • Web services on major data types
    • Registry of sites and data services
    • Common data portal
    • New services (keys / phylog.)
    • Vocab. / ontology site (GBIF)
    • User support
    • Sociological research
  • Questions?