Crowdsourcing transcription of handwritten archives

One of the big differences between libraries and archives is that libraries tend to have more of ‘the printed word’ whilst archives have vast amounts of ‘the handwritten record’.  While some libraries are getting up to speed with mass digitisation of books and journals and then being able to offer users full text searchable digitised items, this is still a distant dream for most archives.  Some archives are undertaking mass digitisation, but the second step – making handwritten records full-text searchable is a massive challenge.  The reason for this is in the technology and processing steps.

After scanning a ‘printed word’ page into an image file a piece of software called Optical CharacterRecognition (OCR) converts the image into searchable text.  The OCR works best with clean, clear, black and white typeface such as a word document or a book, not quite so well on old books and journals, and very poorly on old newspapers.  When it comes to converting handwriting it fails miserably.  It just can’t distinguish and convert handwriting to text in the way the human eye can.  Therefore archives can’t easily automate the second part of the digitisation process using OCR software like libraries can for the printed word.

If you at least get some OCR text from print that is readable and therefore searchable you can offer a service to users to full-text search the books or journals such as Google does. If the OCR text is poor there are some things you can do to improve it. You can encourage users of your service to correct the OCR text with a text correction tool so that the searching is improved, such as Trove does with the Australian Newspapers.

Unfortunately the only viable option open to archives to convert digital images into full-text searchable text is to use a manuscript transcription tool, in combination with harnessing the power of a crowd to do the transcription work.  The transcription work for handwritten records is much harder than for example text correcting old newspapers because the handwriting is often difficult to read, old fashioned, barely legible and not necessarily structured in lines or columns. There is often nothing to go on.

I recently stumbled across a blog all about manuscript transcription tools that is written by a software developer Ben W Brumfield in Texas. Ben developed his own software to transcribe his great-great grandmother’s journal. ‘FromthePage’ is now being used by archives because Ben has made it available open source.

A year ago he wrote an in-depthblog post that covered manuscript transcription tools under development, manuscript transcription projects in archives, and made some predications for future directions of manuscript transcription.  I am not going to repeat what he said here, I suggest you read the post in full.  He notes that software development in this area is still fragmented and young with no particular tools taking dominance. Most developed applications are being made available open source. A standout is ‘Scribe’ from the Zooniverse team, currently being used by both the ‘Old Weather’ project to transcribe maritime weather records and by ‘What’s the score’ project to transcribe music scores at the Bodleian Library, Oxford.

Before an archive implements a manuscript tool it needs to find out what it’s users  would most like to be easily full-text searchable from the vast vaults of all the content it has. It is important to find this out, because the crowd will only be motivated and swell in numbers if they really feel what they are doing is very important to a broad group of people and really matters either right now, or in the long-term and is also interesting.  They have to feel this before they will join in.  Once they have joined in there are other motivational tips you can do to keep them going.  Just implementing a manuscript tool is simply not enough.  You need to engage, watch, understand and learn from your crowd, for they hold the passion and power in their hands to make your project successful or not.

Photo by Rose Holley, outside Canberra Bus Station

Crowdsourcing Australian Climate Change

In my last blog post I described how knitters and yarn enthusiasts were crowdsourcing knitting patterns from digitised Australian newspapers in Trove for use in a crowdsourcing site called Ravelry http://www.ravelry.com.  This week I wanted to give another example of the Australian Newspapers in Trove giving leverage to yet another crowdsourcing site. This time it’s for research into climate change and the site is called OzDocs. http://ozdocs.climatehistory.com.au/

Australian newspapers hold unique content, for example convict records and climate records. In Australia official weather records only began in 1908 when the Bureau of Meteorology was established. However there are weather tables and forecasts appearing in Australian newspapers from 1803 onwards.  The newspapers therefore provide 200 years of weather records.  Newspapers not only give tables with statistics of temperature, rainfall, winds etc, but also eye witness accounts of weather conditions such as floods, droughts and fire.

The citizens and politicians of Australia have a high and ongoing level of interest in climate change and how it is affecting our nation. A project investigating climate change is SEARCH: SouthEastern Australian Recent Climate History. It spans the sciences and the humanities, drawing together a team of leading climate scientists, water managers and historians in Australia to better understand south-eastern Australian climate history over the past 200–500 years. The digital newspapers in Trove are a fundamental part of SEARCH’s research process. However even though the newspapers are full-text searchable it is still a challenge to find and bring together in context the eye-witness accounts and the weather tables so that the temperatures, rainfall and other statistics can be transcribed into a research database. This is why the SEARCH project has established this month the OzDocs citizen science project, with a $10,000 grant from the University of Melbourne so that the public can help them. Basically the public are asked to find and tag historic newspaper articles on weather conditions, and transcribe useful weather statistics from historic newspapers into a database.

In 2010 Joelle Gergis, the lead SEARCH investigator spoke to me and said:

“Having all this information online and being able to quickly access it has been amazing. Being able to find weather tables from 1803 onwards in the Sydney Gazette is crucial to our research. Official records from the Bureau of Meteorology only began 100 years ago so being able to access the newspaper records which are earlier than this is really useful. The sources in Trove also show how weather events have affected society, with eye witness accounts of floods and bushfires. For example we have been researching the 1851 Black Thursday bushfires in Victoria.”

In 2010 as a locust plague swept across the south-eastern side of Australia the pilot volunteers for the project working at the State Library of New South Wales noted that weather conditions in 1825 were very similar (heavy rainfall followed by nice weather then a terrible locust plague) and found eye-witness accounts in The Sydney Gazette and New South Wales Advertiser, 24 March 1825 of a similar locust plague.

“Prior to the late rains the caterpillar, that old enemy to the agriculturist interest of the Colony, made its appearance ; but, upon the visitation of the heavy showers, their ranks were consider ably thinned. However, since the present enchanting fine weather has again set in, the number of these destructive insects has increased to an unparalleled extent, covering whole fields in their course, which in some spots seemed to be towards the South, in a line from East to West. Wherever they make their appearance, the most complete destruction immediately follows. Upon Captain Campbell’s estate, in the district of Cooke, they were supposed to be at least two inches in height.”

This month I caught up with Joelle again to see how the newly released crowdsourcing part of the project is going. She said:

“We now have over 100 volunteers who have contributed over 4000 articles. The database will be searchable in our next release. It will be the country’s first publicly searchable database of climate information using a diverse collection of pre-20th century historical records. The database will give easy access to information for researchers, organisations, government departments and the public.”

The scope of OzDocs work has now been expanded to include not only the digitised newspapers but other resources that are held in the State Library of Victoria, State Library of New South Wales and the National Library of Australia.  These pre-date the newspapers by another 100 years and go back to the 1700’s. Joelle said:

“Our OzDocs volunteers will be working their way through logbooks of the first European explorers, governors’ correspondence, early settlers’ diaries, newspapers and the works of 18th and 19th century scholars.”

On of the questions the SEARCH project hopes to answer is what the South East region of Australia’s ‘natural’ climate has been like since 1788. This may ultimately help to refine current climate models, allowing more accurate climate change estimates to be developed for the future.  The lack of records in a consistent accessible format before 1900 is currently making this difficult.

For the project to be successful the volunteer numbers really need to increase a lot.  At the moment this is still quite a small scale effort compared to the knitting project Ravelry that I reported on in my last blog.  Volunteers can join by accessing the OzDocs site. http://ozdocs.climatehistory.com.au/