Thank you for inviting me to speak here today. I am going to talk about changing roles for librarians and users, drawing to a large extent on my own personal experiences, with particular reference to Trove and Australian Newspapers.
Lets step back 100 years ago and look at the qualities a librarian needed to work in Australia. I shall read you extracts of an article from the Sydney Morning Herald dated 1907. “ A good all-round education is essential, and also a knowledge of two languages besides English. Practically it is necessary that intending librarians should hold a University degree, and understand French. A knowledge of German is desirable also, or any other living language, while Latin is distinctly useful. The average woman is not capable of administering a department properly, say the authorities. It is not likely that many women will become the heads of libraries; they are handicapped by various limitations; limitations perhaps of physical strength, perhaps of temperament. &quot;Nine out of every ten women are unfitted to be at the head of a library,&quot; remarked Mr. Anderson, the Government Librarian, &quot;but sometimes a tenth-is discovered, and she is beyond price.&quot; There are only 11 women at present employed in the Government libraries. Today 12,000 prof q libs in Australia and 84% are women. Thank goodness times have changed!!
Lets move forward to 25 years ago. This is me in my first job as a Public Reference Librarian after gaining my degree in Librarianship. I was working in a state of the art public library in Surrey, UK spending 90% of my time on the reference desk. Lets take a closer look at what is happening in this photograph. Control of Info – micofiche on desk Please wait here – took time, queue Access – in person or telephone Community information – posters on wall The key points being that 25 years ago information was controlled by librarians and was centralised, users found it difficult to easily find or get information themselves, and none of this was immediate, it all took time.
The most radical change I have experienced in my working life to date is the arrival of the internet. The internet changed everything for libraries . Before the internet information was Produced by a relatively few large and powerful publishers Discovered by metadata hand-crafted by librarians Expensive and centralised Post web, information is Produced by anyone Discovered by full text and bottom-up linking effects Cheap and distributed By December 2009 according to the ABS 2/3 of the population now has internet broadband access. Obviously in Morgan, South Australia this is not taken for granted –hence the signpost!
With the arrival of the internet and digital technologies libraries began to digitise resources so they could be accessed online. This was a very laborious and cost intensive process since libraries have always been concerned about long term preservation, not just digital access so standards in libraries for digitisation were high. Between 2001 and 2006 I managed the digitisation of over 100,000 items in New Zealand including maps, photos, architectural drawings, artworks, journals, archives and documents and around the world other librarians were doing the same. Then libraries decided to ramp it up and go into ‘mass digitisation’ for example digitising millions of pages of newspapers and books. To date millions of items have been digitised by libraries internationally. This photo of me is in 2004, I can’t describe the excitement we had when I received 50 CD’s containing thousands of digitised archival images back from our digitisation contractor. These were exciting times!
Initially most libraries built their own websites to give users access to their resources, but we came to realise that in the digital world users do not see institution walls and it was more helpful for them to discover resources in a large pool. Collaborative digital discovery services like Picture Australia and its New Zealand equivalent Matapihi were created. We could do this because libraries have been working to the same standards for many years such as Marc and Dublin Core. However we still tended to group like formats together in services for example digital pictures only and we didn’t mix the non-digitised records up with the digitised ones, so generally our library catalogues were still totally separate to our digital discovery services.
Slide used in presentation 2002. Utopia
Personally I do not see the current conditions threatening libraries, I see huge opportunities for libraries in the year 2010, and a chance for them to demonstrate their relevance in society, now more than ever before. However I know some of you may strongly disagree with me here. So I’m going to talk a bit more about the 2 projects I manage which are Australian Newspapers and Trove.
At this point since I’ve now mentioned the ‘G’ word I want to remind you all why we need libraries and why we are different to Google or Amazon or Wikipedia. It’s because we have made some promises about our content…. Long term preservation and access We have no commercial motives Universal access “ Free for all” ALWAYS AND FOREVER
Whilst we focused on digitisation and delivering things online we did not seem to realise that the public lost valuable social interactions that they had before in the physical environment. Web 2.0 is really about building those social engagements back into the digital world. To give you an example when I worked in the reference library users often used to underline words in books and write their own notes by the side. If they did this they would get a stiff fine, and if they did it repeatedly they would be banned from the library, but people still continued to do it. Now on a digital book it is possible to add a virtual post it note, or an annotation or comment and we encourage this!! It has taken us a long time to realise that what the users want to do, has not actually changed, and it is good to let people do these things.
Users rate finding information EQUALLY as important as doing stuff with it. ‘Doing stuff with it’ is not the icing on the cake it’s core.
Access services remotely. Access services on mobile devices Digital Commons NLB Singapore ‘Library in my pocket’
Innovator. Boulders and pebbles. (to develop services, to prioritise digitisation…).
And now we are in 2010. We have firmly acknowledged that in order to give users the best service, collaboration and data sharing are key. But more than this. The two direction statements the NLA is working towards are as follows: We will explore new models for creating and sharing information and for collecting materials, including supporting the creation of knowledge by our users . “ “ The changing expectations of users that they will not be passive receivers of information, but rather contributors and participants in information services.”
ANDP started 2007. Website gives all project information
The program is national and collaborative. Every state and territory library in Australia is involved. Each state and territory has made their selections of titles and provided microfilm to the NNLA and by 2011, 40 million articles will be available. After this it is hoped that state, territory and public libraries will begin to contribute regional titles.
The purpose of the ANDP is to provide access to historic Australian newspapers in an online environment Key Features of this service will be that: It is accessible online Access is free The service is full text searchable. Up until now people wanting to research historic Australian newspapers needed to go to libraries across Australia and scroll through reels of microfilm. This program aimed to provide an online service that will let people anywhere, anytime access these newspapers via the internet. The service is now available. It is free. You can full text search across every page of every newspaper in the service, including advertising, cartoons, letters to the editor as well as the news and sports articles.
State titles 4 million pages
A great enabler in this program was the existence of ANPlan – the Australian Newspaper Preservation Plan. This is also a national collaboration between the state and territory libraries. For a number of years ANPlan had been working to ensure that significant newspapers were preserved via microfilming. This meant that the microfilms could be sourced and used for digitisation. Digitising newspapers from microfilm costs about a third of that from hard copy, so this has enabled much more content to be digitised.
In addition to the titles selected for the program, the National Library has received a $1 million grant from the Vincent Fairfax Family Foundation to digitise The Sydney Morning Herald through to 1954. This project will be running concurrently with the ANDP, and the pages will be included in the delivery system being developed for the ANDP
A complete microfilm of the Sydney Morning Herald did not exist but due to a very generous donation of $1 million by the Vincent Fairfax Family Foundation we have been able to source the hard copies of the SMH and include these in the service. This title is now complete except for 3500 missing issues - now found in stacks which will be digitised and made available soon.
Of all the items you can digitise newspapers are one of the hardest to digitise and deliver for a number of reasons. The NLA decided that setting up a national infrastructure and aiming to have all Australian Newspapers accessible from one place would be the best thing for the nation. This was a big challenge and one that no other country had attempted to take. The national infrastructure consisted of 5 things: Online and offline storage for digital files Development of a NCM to manage the digitisation workflows Development of a public delivery system to provide access to the newspapers Quality assurance processes and team Establishing a panel of newspaper digitisation contractors
Beta service. Quickly found, no publicity. Agile development, new content new features added.
One of the innovative features that was in the first release was the ability for members of the public to correct or enhance the OCR text. When digitising old newspapers the process is to convert a digital image into full-text by use of Optical Character Recognition software (OCR). This works well on new clear documents but on old newspapers where the font and paper is of poor quality and microfilms may be out of focus the translation often goes into gibberish. After investigating every possible way technically of being able to improve this we came to the conclusion that the best way was by hand and human eye. We could not possibly afford to pay contractors to do this ‘re-keying’ so the lead programmer Kent Fitch suggested we open it up for the public to do. If text was made accurate the searching would be instantly improved for everyone since the search works over the OCR text.
This was very controversial since no such thing had ever been done by a library or archive before and was considered high risk. The risks were identified as: No one will do it OR People will deliberately vandalise the text. The likelihood that people would just do lots of it well was considered extremely unlikely. Because it was unknown if people would do it, it was decided not to put valuable time into developing a moderation module which may be unnecessary but to just see how it went without moderation. Also to lesson barriers it was decided registration would not be necessary. To reduce risk the added data would be kept in layers and not integrated into the original metadata (although it would appear to users as if they had changed the actual metadata). All layers of data would be searched on. While we were doing this we also decided to add in the ability for people to add comments and tags to articles as well. NEXT – quick demo.
This is the article view. Users can zoom in or out and choose to view the article in the context of the entire page. They can also navigate to any other page within the newspaper issue. The electronically generated text created through the OCR process is displayed on the left hand side. This is also where the users can use the 3 enhancement features. They can drag the viewing pane to see more of the or less. Users can tag the article with keywords and they can write comments and notes about the article. If users login they will be able to choose to make their tags and comments public or private. So they can share their comments with all users or they can add their own private research notes that only they can access. One feature that we believe is innovative and not available in any other online newspaper service, is the ability for the user to correct the electronically generated text. There are a number of reasons why the electronically created text is not always 100% accurate, mainly due to the quality of the original newspaper that the image was created from. Users can correct the text by clicking on the ‘Help fix this text’ button. We will now use these features on this article. The article we are looking at is the first report in an Australian paper of the sinking of the titantic.It’s in the Northern Territory Times on 19 April 1912.
I want to tag the article with ‘titantic sinking’. If a user does not login when they first enter the service then the first time they want to enhance an article they will be offered the option to login. At this point they can either login or enter the captcha to verify they are human (and not a robot – attempting to do something undesirable).
Once logged in or verified with captcha a user can enter their tags.
Now I want to add a comment. Those of you who read this article may have noticed that it was reported that all passengers were safely rescued from the titanic and the weather was calm. I’ll just add a comment to say this was unfortunately not the case.
Now I have zoomed in on the image and if the OCR text was inaccurate I would edit it in the box on the left. This is what we call the power edit mode. In this article the text is actually very accurate so has either OCR’d very well, or already been corrected by someone else.
Now we can review the article with all the enhancements we have made showing on the left. Tags, comments and corrections. We can view the history of all the enhancements (both ours and other peoples history).
This graph shows the rising rate of text correction. Text correction peaks over the christmas period and long weekends. There has been no time since release of service when text correction is not taking place. It goes on 24/7.
In 2010 – huge increase in text correction. 2 million lines per month
We did not prompt the public to give their opinion on text correction, though many did anyway. Although no-one had encountered anything like this before they quickly understood the aim and thought it was a good idea. Once people understood they likened it to Wikipedia, though it was not quite the same since the original image is always there for verification.
But also people gave these reasons for doing so much – in some cases up to 40 hrs a week. Because after all if you don’t enjoy it you wouldn’t keep at it, so loving it and finding it interesting and fun were really important.
30+ volunteers, 32 mill lines corrected in 1.3 mill articles
Single search. Integrates existing services into it. Fully migrate services 2 needs – efficiency in maintenance of single service behind scenes Satisfy public need to have single google style search
I am now going to talk about Trove – the search engine for Australian resources. Australian Newspapers service was in fact a test bed for the idea to transform service delivery and our internal IT infrastructure in the future. Because Australian Newspaper worked so well the beta model of software development, the underlying IT infrastructure and the application of user engagement has been applied to all the other discovery services the library manages which are rolled into Trove. Content of zones The key features of Trove are that 1. Firstly, and most importantly it is a single search. In one click you can simultaneously search across several groups of information- books, journals, magazines and articles: images: australian digitised newspapers: diaries, letters, archives: maps: music, sound, video: archived websites, about people and organisations. 2. Secondly you can browse through these groups or zones one at a time if you prefer to only seek one type of content for example newspapers. 3. Thirdly you are able to restrict your searches to – online content only, and/or content held in locations near to you. This is very useful feature for the large majority of users.
Results are unbiased – best and most relevant info possible – relevancy ranking. Similar to values of a good reference librarian (subject to initial choices made by user eg location, immediacy). Results are returned in the same zones that we saw on the home page. You can see in each zone how many results are found. Most searches retrieve vast numbers of results because of the wealth and richness of the repository that is being searched. It is likely that you will want to refine or limit your search results and you can do this by using the facets on the left hand side of the screen. The facets change depending what content you are looking at, so for example the book, journal, magazine and article zone has a facet to refine by braille book or audio book. We recognise that many people just want items that are immediately accessible ie digitised or online, as fast as possible, so the links to online content appear immediately at this stage although we haven’t yet drilled down to a detailed results screen. The check boxes to restrict the content to online or Australian are always visible so that they can be checked or unchecked at any point in the search. Here is an example – Merino sheep originating from Spain. The first twenty six merino type sheep were introduced in Australia in 1797 from Cape Hope, South Africa. Merinos are originally the typical Spanish sheep; bred five thousand years ago by the Tartessos at the valley of the river Guadalquivir, southern Spain. Thank you Spain! Numbers have grown in last 200 years and now have over 100 million in Australia.
Trove is an aggregation of 120 million items from over 1100 libraries and other organisations
Agile, every 2-3 weeks, very small team, ongoing usability.
New homepage, gives greater prominence to ‘contribute’ + have added in 400 million subsription resources
The recent interactions users have made are also displayed on the homepage for everyone to see. You can see the number of searches in the last hour, newspaper article corrections so far today, works merged or split this week, items tagged this week, and comments this month.
Add objects of personal or national significance
Old photos – especially for identifying people
Lists – adaptable, different purposes, by user demand.
Personal- private or public
Inst – virtual exhib
Educators, replicate trails, slideshows except can do themselves, teaching resources for classroom
Peoples names in comments. Searchable separately. For example users can comment on items as well as newspaper articles now eg images, books and archives and share valuable information, and rate items.
About 50,000 new digitised articles are added each week to the Newspaper zone. Many users have requested alerts for new content based on keyword search terms for the whole of Trove. This is still in progress. In the meantime alerting for individual articles has been implemented. Because these items in digitisation progress appear as ‘coming soon’ in the results list an user can choose to be notified when the item arrives. This radically improves the ‘get process’ for newspaper articles. Also subject keyword feeds
Monthly hall of fame – requested by users
Profile enhanced with activity – total rankings
This is a newspaper title information page – showing the date range digitised, title pi and information being pulled from wikipedia. Rich info rather than just bib data
This is what you would see for the Argus. We’ve added the link to the digitised newspapers as an external link at the bottom. For most newspapers there is already a page existing, but if there wasn’t I created one and it quickly got populated.
3000 comments and feedback
Forum so users can discuss and contact each other
Interconnected to forum and service, purpose to promote service and content within it. Build virtual environment.
Linked to tweets and social enagement, Raised money in few weeks
Physical group – met during floods
Leading climate research project, harnesses power of service and users
Re-purpose trove data
Increasing 3 million users – aiming for 11 (half population = current lib users)
National news and SMH
Good – expect via Google. Site map and content harvested
Not possible without Longstanding history of collab in australia Wanting to share data Making this easy via OAI, api Shift in thinking control to freedom New ideas – how can technology help us to do what we always wanted to? Old ideas revisited
Thank you for listening to me today. I am happy to take questions.
Manager - Trove
National Library of Australia
1 st International Seminar of the Library of Galicia:
“ Self service, satisfaction and seamlessness are definitive of information seekers expectations. Ease of use, convenience and availability are equally as important to information seekers as information quality and trustworthiness.”
http://climatehistory.com.au This landmark project, spanning the sciences and the humanities, draws together a team of leading climate scientists, water managers and historians to better understand south-eastern Australian climate history over the past 200–500 years. It is the first study of its kind in Australia.
Re-purposing information and sharing Blog using newspaper articles http://lynnwalsh.wordpress.com