In my last post I outlined the
critical state of the National Archives of Australia. The NAA are now under a 'functional and efficiency review' led by Dave Tune. The NAA Advisory Council has declared the inability of the NAA to fully meet its mandated functions. Details about the
Tune Review are on the NAA website. The appeal to clients and stakeholders of the NAA to express their viewpoints on what the future role of the NAA should be be, the current critical issues and recommendations for going forward has resulted in 65 responses. To read the public responses to the NAA Tune review, which are in the form of formal 2-4 page letters go to this page:
http://www.naa.gov.au/about-us/tune-review_submissions.aspx
There is resounding negative feedback from clients about the difficulties in accessing archives; resounding affirmation from
researchers, historians and archivists that an open, effective democracy is reliant upon and should be supported by citizen's right of access to government records of significance; and resounding feedback from information and IT
professionals that the NAA are not currently up to the digital challenges of collecting, preserving and making accessible digital archives, do to long-standing under-funding which has led to a critical situation for staffing and ICT infrastructure.
I publish below my official response to the Tune Functional and Efficiency Review. It is also worth reading the response of the Australian Society of Archivists, and Professor Peter Stanley.
Response
to the Tune ‘Functional and Efficiency Review’ of the National Archives of
Australia (NAA).
To: David Tune,
Independent Reviewer OAM PSM.
Date: 28 June 2019
From: Response written by Rose Holley in
consultation with Dr Warwick Cathro,
and Eric Wainwright.
Together Warwick, Eric and I hold a great professional
interest in the effective operation of the National Archives of Australia. Over
the last ten years we have each successively committed more than two years of
our time working collaboratively with staff from NAA on high priority strategic
matters, ICT issues and projects. Together we have an in-depth strategic understanding
and working knowledge of the NAA physical and digital challenges and some ideas
for potential solutions to the issues. We met last week to pool our knowledge and
brainstorm the issues, in the spirit of helping and for the ‘common good’. We outline our professional opinions of the
key factors that need to be addressed, and how this can be done in terms of the
purpose, functions and role of the NAA. We thank you for this opportunity to
share our expertise. We acknowledge the difficult and stressful circumstances
that many of the NAA staff are now under and offer our support to them. We know
many of them to be truly dedicated and passionate about their roles. It is with
good intent and hopes for positive change that we respond to the call for help
from the NAA Advisory Council and offer our constructive recommendations on this
critical situation.
Relationships with the
NAA
Rose Holley
·
Currently working in a formal partnership with
the NAA. As Special Collections Curator at UNSW manages custody and access to
the NAA John Howard Collection.
·
An active member of the Australian Society of
Archivists networking with previous and current NAA archivists.
·
Previously NAA Director of Software
Implementation and Senior Project Manager 2012-2014. Managed a multi-million
dollar project to implement an audio-visual digital preservation and asset
management system at the NAA.
·
Is a ‘wannabe’ self-serve digital user of the
NAA collection with particular interest in finding and accessing the already
digitised ABC radio and TV Archives back to 1934. Would like to do this
remotely and digitally, but at present this is not possible.
Warwick Cathro
·
Assistant Director General at NLA 1989-2011.
Worked closely with senior executives at NAA in 2009 and 2010 to develop a $147
million joint bid with NAA and NFSA to tackle the challenges of digital
collecting, preservation and access.
Eric Wainwright
·
Deputy Director General at NLA 1988-1997. During
2011-2012 was an external consultant for NAA with his partner Dagmar Parer (NAA
Director 1989-2000, NAA consultant 2005-2014). Advised on strategies for
addressing key digital challenges of digital archiving and access.
Contents:
1. Summary
of Key Issues (Wainwright, Cathro, Holley perspective)
2. The
Tune Review Process and Communication
3. Core
Purpose of a National Archive
4. Vision,
Strategy, Leadership, Policy
5. Physical
Buildings
6. IT
Infrastructure
7. Organisational
Structure, Staffing and Project Management
8. Changes
and Collaborations (Digitisation, Preservation, Collections, Systems, Storage,
Facilities)
9. Resource
Estimates
Appendix 1: Precis of Joint NLA, NFSA,
NAA Digital Deluge Funding Bid 2009
“One of the
great ironies of the information age is that, while the late twentieth century
will undoubtedly have recorded more data than any other period in history, it
will also almost certainly have lost more information than any previous era.” Alexander Stille, Professor of Journalism
at Columbia University.
1.
Summary of Key Issues (Wainwright,
Cathro, Holley perspective)
After thoughtful and serious discussion Eric, Warwick and I are
unified in thinking that the major issues for the review team to consider are:
·
Will the Review itself be able to address the
complexity of issues currently faced by the NAA, given the shortness of the
Review timeframe, its limited promotion, the lack of input from other
collecting institutions and interested parties, lack of international
comparisons, limited records/archives expertise on the Review support team and
no leading edge technology expertise within the NAA itself? Why is there no
apparent involvement of the Government’s Digital Transformation Office?
·
The reduction in NAA funding over many years, the
associated decline in records experience/expertise in middle to senior staff,
and an inability of the NAA to provide adequate searchable records for a high
proportion of its collections.
·
The inability of the NAA to secure adequate
capital funds for digital developments to address the issues of the move of all
agencies to digital record creation, web sites and database approaches,
technological obsolescence, and higher expectations regarding local and network
access amongst potential records users.
·
Limited influence and risk-aversion of the NAA
(and weakness of the present Archives Act) in overcoming the unwillingness of
Agencies/Departments to allocate adequate resources to enable the NAA to
collect records of long-term national significance (particularly those in
digital formats), and to provide user access within a reasonable timeframe.
This problem has been further exacerbated by recent amendments to the Archives
Act (which were not subject to public advice and show no understanding of
serious research access requirements), increased user charges re digitised
copies, and a general move in Government away from a presumption of open public
access to records.
·
Lack of effective collaboration between the
NAA and other collecting institutions, which would allow sharing of
technologies, storage, expertise and more effective online user services.
·
Pressures (in part Government/political) on
the NAA to devote ever-increasing resources to family history tracing,
particularly related to military involvement, to the detriment of its general
support for Australian historical research and services to Commonwealth
government agencies.
2.
The Tune Review Process and
Communication
It has been alarming and discouraging that such a
significant and overdue review has gone ‘under the radar’ of many of the NAA key
stakeholders and clients. The Open Letter of appeal from the NAA Advisory
Council to give feedback for the review was placed on the NAA Facebook page on
14 June 2019, which is where we noticed it, and 2 weeks were then given for
people to provide feedback. The letter cannot be found on the NAA website and does
not appear to have been sent to all key stakeholders. There has been a brief
article in the Canberra Times about the review, which downplays it, but no
other media notice as far as we are aware. The very formal method of response
has perhaps alienated many average clients from being able to easily give a
viewpoint. It has not taken into account varying communication styles and
formats.
Recommendations:
Ø Before making
any decisions pro-actively seek input from the two major client groups: public users of records, and donor agencies.
This could be done by structured focus groups, workshops, and interviews. Have
an emphasis on questions that elicit responses about both the current and 20-30
year future expectations of clients. Consider that future requestors of records
may not be existing users now, either because they are currently unable to find
and get the records they want digitally, or their needs have not yet emerged.
In the future there may be a greater number of requests for born digital
records e.g. people requesting access to information held on themselves under
Freedom of Information laws. The recent
legal changes that allow government collection of individual’s metadata from
phone, email, and social media accounts, including personal messages, GIS
locations etc. should be considered in this broader picture. Every Australian is potentially a client of
the NAA, from all walks of life and their viewpoints matter. How would a 20
year old expect to be accessing information about themselves in the year 2060
from NAA, and how much information about them would be held by then?
Ø Pro-actively identify
the NAA’s key stakeholders and clients.
If any of them have not provided feedback in this review, invite them
to. Have reasonable timelines and appropriate methods e.g. focus groups,
interviews. Include the 26 National Collecting Institutions (NLA, NFSA, AWM,
AIATSIS, etc.), State and Territory Libraries (NSLA), State and Territory
Archives, Australian Society of Archivists, Australian Library and Information
Association, the Council of Australian
University Libraries (CAUL), the ABC, Defence, and ASIO.
Ø As the review
progresses keep the process transparent. Include names of the review team, the
stakeholders consulted, and timelines and progress on a website.
Ø
Provide key collecting institutions and major clients with
a draft of the final review for consideration and comment before public
release, including NLA, NFSA, AIATSIS, AWM, ABC and Defence.
The core purpose of the National Archives of Australia is to
collect, preserve and make accessible the significant government records of our
nation (Records of National Significance).
This is achieved by having an ongoing relationship with government
agencies during their active use of internal records (records management),
which then flows into eventual deposit of some records of national significance
(archives). Records encompass ‘information’, ‘data’ and ‘metadata’, and can be
physical, digitised or ‘born digital’. Traditionally many were paper, but there
is also a large amount of audio-visual e.g. film and radio and old data formats
e.g. floppy discs. Much of the ‘born digital’ will be emails, or social media
formats. The NAA are mandated to collect and preserve our nation’s records with
a far larger scope than that of any other institution, and at a time in history
when more records are being created than ever before. However are the NAA
actually collecting the significant records/databases/datasets, and how can the
public tell? Does the Archives Act adequately set out the core purposes for a
modern archive? Should there be an external advisory committee dealing with
collecting policies/matters that is accountable or representative of the two
client user groups: government agencies (200+); and public users for example
university researchers and family historians?
When budget is tight there is not an option to reduce the
scope of collecting, or prevent access to the collections, without breaking the
mandate. There will be significant gaps and loss of irreplaceable government
information if this happens. The NAA should hold a vast historical corpus of
data, which if mined could be of great economic, scientific, creative and
political value to the country. This includes the ABC Archives back to 1934,
war and service records of all Australians, weather and climate records, Prime
Minister’s and cabinet records. The power of information cannot be
under-estimated. For example records recently used for matters of national
importance include: cyber-security, sexual abuse, rights of aboriginals, and climate
change. Our constitution and the Archives Act supports and enforces citizens’
rights of access to information and data about ourselves, our society, our
culture, our politics and our social history.
Recommendations:
Ø Do not compromise,
lessen or reduce in scope the core principles and powers that enable unfettered
independent collection of government records of national significance; long
term preservation; and citizens’ rights of free access to records because of
the challenges faced. Archives are the backbone of a democratic and open
society. The Government must take whatever action necessary to restore public
confidence and resource these functions.
Ø Clarify if the
NAA is actually collecting the nationally significant
records/databases/datasets and how the public would know this.
Ø Review if the
1983 Archives Act sets out the core purposes for a modern archive.
Ø Consider if
there should be an independent external advisory body which separately advises the relevant Minister(s) on
all matters relating to records and archives, including FOI/public interest
issues, collecting policies and matters? For example like the UK Advisory
Council on National Records and Archives.
The NAA has a Corporate
Plan 2018/19-2021/22, an Archives
2020 Strategic Plan, a Digital
Continuity 2020 Policy, and is the lead agency for the Government’s Digital
Transition Policy. Recently it has just released an IT
Strategic Direction 2019-2022. However, these are very short term plans and
seem often devoid of detail for accountability, and for enabling an assessment
of current collecting and access strategies and their achievement. The contrast
with the UK National Archives current Digital Strategy is noteworthy. While the
NAA has produced many useful guides for Agency records policies and practices,
the results of the most recent Check-up
Plus survey suggests a very limited success in implementation across agencies
as a whole. There appears to be a lack of forward visioning for where
the NAA needs to be in the medium to long-term (5-30 years’ time, i.e. 2050)
and what it needs to get there. Clearly effort has been focused on successive
‘annual checkup surveys’ of recordkeeping activities of 160 agencies for the
last few years, and producing the digital transition, digital continuity and ‘
digitise
and disposal’ policies. Beyond offering general advice that information
should be valued and managed digitally, these policies seem largely
ineffectual. There is a lack of public visibility about planned programs of
digitisation and how these are selected and prioritised.
Warwick, Eric, myself, other collecting agencies, professional
peers and consultants have given significant time, ideas and advice to the NAA
over the last ten years as they transitioned into a critical state. It seems
there were long running internal systemic issues which still have not been
addressed, for reasons not known to us, despite our close connections with the
NAA. Since about 2005 there has been a concerning lack of benchmarking or
attempt to lead or follow international best archiving practice or services
(with the exception of
the Hive project). The NAA is no longer on
a par with the UK or US national archives.
Recommendations:
Ø Find out how
other International Archives are addressing the digital challenges of
collecting, preserving and access, and what are their budgets, legislative and
operational frameworks? Consult with exemplar International Archives e.g. Norway,
Finland, Sweden, the Netherlands, UK and the USA.
Ø Review the
role, makeup and operation of the NAA Advisory Council and executive leaders.
Ø Devote
resources to developing a 30 year vision and strategy, building on previous
work done, so that realistic resources and cost for achieving the vision can be
budgeted for.
Ø Undertake an
audit of all NAA Policies and Procedures (similar to that carried out in 2011
Wainwright Report Appendix 2). Review their effectiveness, relevance, overlap,
public and agency visibility, and consistency of style and writing.
Ø Pro-actively
seek feedback from ex and current staff, in a confidential, no blame
environment to help the review team discover why things have happened in the
way they have, so this can be prevented from re-occurring in the future.
Ø Review the
strategy for digitisation, and public visibility of the selection and
prioritisation of records being planned for digitisation.
Ø Members of the
review panel or any taskforce subsequently appointed must read relevant internal
and external documents previously written about developing strategies to
overcome the NAA digital challenges and improve services. Some example reports
are listed below where most of the recommendations were never actioned and
would still be relevant today.
·
ICT Business
Case: Dealing with the Digital Deluge. Exposure Draft 2009 led by Warwick
Cathro, NLA. Joint proposal to Government by NAA, NFSA and NLA for $147 million
over 5 years to address shared critical issues of digital collecting,
preservation, and access. If granted aim to complete by 2020. (Not approved). Executive Summary. Precis at Appendix 1.
·
Strategic
Directions for Digital Archiving at the NAA. 2011. Eric Wainwright. Options for
a 20 year strategy to 2031.
·
External Review of RecordSearch (NAA online users
experience and expectations vs actual experience) by APIS Group? 2011?
Recommendations for major improvements. (not actioned).
·
External Review of RecordSearch Usability. By Usability
One. 2012?
·
Access to Digital
Records Scoping Study for NAA. Eric Wainwright and Dagmar Parer. 2012. Recommendations
for addressing issues, including RecordSearch and developing forward strategy.
·
Access Clearance
Taskforce 2013 to tackle issues and backlog. (Not publicly available).
·
External Review of
NAA Access Clearance Issues, with recommendations. 2016 by Paul O’Sullivan OA
CNZM (Not publicly available).
·
Dealing
with the Digital Deluge- Ten Challenges for GLAM’s 2012, Holley,
R.
·
The proposed
Australian National Cultural Policy 2012- an overview for GLAM’s 2012,
Holley,R.(Includes summarised responses from 200 agencies on what a National
Cultural Policy should enable and resource).
·
The
Australian National Cultural Policy released 2013: an overview of ‘Creative
Australia’ for GLAM’s, 2013, Holley,R.
·
An Innovation Study: Challenges and Opportunities for Australia’s
Galleries, Libraries, Archives and Museums. CSIRO and Australian Centre for Broadband Innovation: 2014.
Mansfield,
T., Winter, C., Griffith, C., Dockerty, A., Brown, T. Sets out a vision to
2035.
5.
Physical Buildings
The NAA has physical locations in every state, with two
locations in Canberra. The NAA does not own any buildings but rents them. There
are specialised requirements for reading rooms, repositories, conservation
labs, server rooms, digital archives, exhibition spaces and community rooms
which are not easily achieved in standard rented office space. Ideally
locations should be easily accessible to the public with parking. Currently
there are 350 linear kilometres of physical records to be stored, with between
100 -1000 km waiting to be deposited. For more than ten years considerable
staff time and budget has been spent planning for office moves, hiring new and
often unsuitable premises, re-fitting rented premises, and re-locating archives
and premises. This is not efficient or effective use of public money. It
diverts energies from core archival business, into worrying about where and
when the next move will be, with little control of this. There have been unexpected notices to vacate rented
premises which have caused considerable stress to archives staff and extreme
inconvenience to clients. For example most recently the move of the Canberra
reading room from the building in Parkes to MOAD. The reading room in MOAD is too
small for demand and does not have enough space to store records for client
repeat visits. The new archives building at Mitchell which opened two years
ago, is not owned by the archives and does not have a reading room, exhibition
space, or public venue facility. It is smaller than the previous 3 rented
premises which meant all the records could not be re-located into it, and there
is no space for future expansion. Despite this the NAA have spent more than $50
million fitting it out to their specific requirements. Logic dictates that the
NAA should own suitable premises, that meet current and future needs. This
would give stability and assurance in the future. This is now a great
opportunity for Australia to set an international example of creating an iconic
Archives buildings fit for the 21st century that meet the needs and
expectations of clients as well as staff. Regardless of how many records are available
online there is a need for ambient spaces that inspire research and reflection,
and are digitally connected and equipped. There is lots of existing research
into expectations of clients for library/archives buildings and spaces and
fantastic international examples of wonderful architecturally designed
buildings to inspire.
Recommendations:
Ø Enable
government ownership of NAA buildings in each state.
Ø Ensure that NAA
buildings are fit for purpose, big enough, and designed to meet the current and
future expectations of clients.
Ø Consider the
opportunity to build a state of the art, architecturally designed archives that
will be internationally recognised.
The NAA has an inefficient network infrastructure, barely
fit for purpose. The digital expectations of staff and clients cannot be met, for
example the network is chronically slow with varying connectivity between
offices for basic tasks such as email. With no high speed network and limited server
and storage capacity, workflows in the state of the art enterprise audio-visual
digital preservation system are not fully used. Even though the software has
the functionality, it is not currently possible for the ABC to digitally
deposit online their born digital archives, or for the public to digitally
stream from home the retrospectively digitised ABC Archives. For example a
client will have to make contact with a member of staff, ask them to search for
items, if digitised staff download them
from a server, copy to a DVD, then ask the client to physically visit the
Sydney office to listen/watch from a DVD in the reading room. There is limited
resource and inadequate IT infrastructure to address the audio-visual
obsolescence challenges and migration pathways of digital media. No government
department is actively and regularly depositing born digital files to NAA as
far as we can tell. In the ideal future the NAA would be operating like a vast
computer super data centre, with high speed networks, to ingest, store,
reformat and deliver giant amounts of data to potentially millions of people. Data
analysis and mining will be key. It is questionable whether the NAA has the IT
expertise to develop or run an advanced IT infrastructure. It is noteworthy
that the NAA first set up a detailed examination of its future IT
infrastructure/digital requirements in relation to its Digital Preservation
Strategy at least as early as 2000 and so was a forerunner in this and in
establishing a Digital Archive prototype. But not much seems to have advanced
since then.
The view that IT are just a regular service section, who
support the email system and desktops needs to be changed. In order for the NAA to go forward into the
world of born digital archives, great technical knowledge and high level
digital dexterity must be spread throughout all sections of the organisation,
with a significant amount of IT experts who also understand the core business
of digital archiving, big data centres, and data analytics. Current efforts of existing staff to scope
digital system requirements both for digital archiving solutions and client
find and get interfaces may not lead to the best outcomes. Attempts to buy an
‘off the shelf’ ‘best in breed’ system are somewhat unrealistic, when they are
not known to exist. Significant internal development resource is likely needed.
RecordSearch, the public catalogue has not met client expectations for many
years and multiple internal and external reviews have suggested an overhaul,
upgrade and replacement. Its metadata is not discoverable by Google or Trove,
and no records can be bookmarked or persistently linked to by clients. It is
unclear what software is used by other national archives. Equivalent relevant data businesses like
Google, FindMyPast, Ancestry.com, the Church of Latter Day Saints and Facebook
can be learnt from in how they run their large scale IT operations, data ingest
and analytics, digitisation facilities, and search interfaces.
Recommendations:
Ø Learn from
exemplars that have efficient and effective large scale digitisation centres,
IT infrastructures, data centres, service delivery models and search portals
e.g. Google, Ancestry.Com, FindmyPast, Church of Latter Day Saints. Evaluate and consider applying similar IT
solutions to the NAA IT issues.
Ø Develop plans
and resourcing for how to address the inadequacies of the NAA IT
infrastructure, network, servers and storage, and what expertise and staffing
is needed to build and maintain an advanced IT infrastructure. Ideally a super
high speed data centre style infrastructure is needed now and into the future.
Ø Develop a
strategy to address the audio-visual obsolescence challenge and digital
preservation data migration requirements.
Ø Research what
software is used for an online public client search portal at leading
international archives, Ancestry.Com and FindmyPast. Make recommendations for how to replace or
upgrade RecordSearch with a state of the art online public search portal that
meets client expectations:
o
able to find and get records online regardless of the
records format or physical location
o
be intuitive, fast, free, easy to use
Ø Take a
‘user-centric’ approach to online discovery of records. Directly and
pro-actively involve public clients and government agencies in the development,
implementation and testing of an online public search portal.
The NAA has a diffused organisational structure, with some
responsibilities being managed locally in offices, and some centralised
nationally or a combination of both. Due to network and staff deficiencies
often what happens in practice does not match the organisational structure.
Digitisation is a good example. This is being done locally and nationally, and
also managed in two different divisions, with a multitude of different
standards. It is often difficult
therefore for staff, external stakeholders and clients to make contact with the
right person, and get clear answers to digitisation questions e.g. to what
standard should I digitise this? There have been at least three minor/major
changes to organisation functional areas in the last few years. Combined with a
slow but constant reduction in total number of staff due to government cuts,
loss of expert staff through voluntary redundancies or resignation, and new senior
appointments that lack archival and records management knowledge, there is a
significant skill gap at the NAA (digital archivists, IT professionals, project
managers, business analysts and conservators), with future skills needs being
unknown. The level of staffing appears
to be significantly under that required for scope of work. The extent of
records to be deposited far exceeds the collecting scope of the National
Library. The amount of digital records are
exponentially increasing, and are in addition to the ongoing management of paper
records and retrospective digitisation.
The NAA has been attempting to tackle some significant
projects over the last few years, without additional funding or staff, and
generally has moved existing staff onto large projects rather than recruiting
project managers or specialist staff. This has compromised service operations
and also led to a high rate of project failure. The projects needed to
re-establish the NAA as a leader in the field are highly complex,
inter-related, co-dependant, span across multiple locations and are likely to
take some time e.g. the procurement/development of an archives management
system, a digital preservation archive, upgrade of network, storage and
servers, and potentially moving buildings and records again. It is questionable
if the current multi-million dollar procurement project for a digital archive
is/will be effective in this environment, despite being much needed and overdue.
There is no public information on the website about the status or progress of
the Digital Archives Program/Taskforce and how or if this relates to the
Digital Archive prototype project from 2000. Have they all stopped or failed?
Recommendations:
Ø Review the
existing organisational structure. Ensure that staff, stakeholders and clients
have good understanding of and visibility about roles and responsibilities, and
that the structure meets needs through a period of change.
Ø Undertake a
workforce planning exercise to identify number of staff, roles required and
current and future gaps over the next ten years.
Ø Recruit
internationally so that the most skilled and innovative people in the world can
be employed to tackle the challenges.
Ø Review
appropriate salary levels, especially for IT roles in order to attract and
retain high calibre staff.
Ø Establish a
program management office with experienced project managers.
Ø Establish a position
responsible for co-ordinating, reviewing, writing and managing policies and
procedures to ensure a quality level of governance, consistency, effectiveness
and style and monitoring the provision of documents to agencies and the public.
Ø Ensure current
and future staff are trained and recruited for their leadership skills, and can
demonstrate leadership at all levels, so that united they can help lead the
organisation out of critical state and into effective operation.
8.
Changes and Collaborations (Digitisation,
Preservation, Collections, Systems, Storage, Facilities)
There are significant issues to be addressed which may
require significant changes to legislation and agency responsibilities. Currently
64% of the NAA collection is not discoverable by clients because it is not
described at item (file) level. Although
2 million items have been digitised over the last 15 years this only amounts to
5% of the collection and there are 39 million items not digitised. Other
institutions internationally are aiming for 100% digitisation e.g. Norway,
Finland. Although the NAA has made considerable investment in developing
audio-visual digitisation labs and software, there is extremely limited staff
expertise in this field with experts moving between ABC, NFSA, and NAA and many
now retired. It doesn’t seem viable for each institution to set up audio-visual
centres of excellence. It is not clear to many clients why there are multiple
government institutions that hold big collections of film and audio records:
the NFSA, NAA, AIATSIS, and NLA and they have to visit each in turn. It is also perhaps frustrating for some
researchers that they have to visit both the AWM and the NAA to get
commonwealth war records. A more recent arrangement to give custody of a Prime
Minister’s records to UNSW is interesting, invoking for the first time ‘section
64’ of the Archives Act, which allows the NAA to give custody of its records to
other people.
The physical size of the NAA collection is vast at 350
linear kilometres, but small compared to the amount of active records still
being held by agencies, which is about 1000 kilometres at
last
NAA survey. Another difficulty is the NAA being unsure how many of these
will become records of national significance. They conservatively estimate
6-10%. The same survey indicates that there are 128,000 TB of digital data held
by agencies with perhaps 3% being records of national significance. The
unresolved issues of how to digitally transfer, archive, access clear and make
public this data seem complex, costly and overwhelming. The NAA has embedded
workflows geared towards paper archival processes which are largely irrelevant
for digital. With the perceived lack of digitally experienced and IT focused staff,
combined with lack of resource over many years NAA is now backed into a corner.
If this was a business it would have gone bankrupt due to its lack of
transition to client expectations and needs. We provide some ideas to resolve
issues by institutional collaborations, innovation and technology.
Recommendations:
Ø Review ‘shared
custody’ of national archives. Should this now be extended to allow agencies
other than AWM and UNSW to manage NAA records? If so how will the shared
agencies be funded e.g. cost per linear metre of records?
Ø Give custody of
NAA Commonwealth personal war records to AWM so that both the official and
personal war records are held in one institution.
Ø Transfer the
responsibility of web archiving from NAA (who are mandated to collect but not
able to do so) to NLA who currently are web archiving. Investigate partnerships
to mirror/backup with the Internet
Archive.
Ø Give custody of
NAA audio-visual records to the NFSA (and also those of NLA and AIATSIS?). This
would enable the expertise, equipment and staff to be centralised so that
challenges of digitisation and audio-visual obsolescence can be more efficiently
and effectively tackled, with competition for and duplication of AV experts,
specific and costly historic and state of the art equipment ending.
Ø The ABC Archive
is a precious national historical resource. Other agencies have a stake in it
(for example, its metadata should be in Trove and it is vital that the content
be preserved). Organise a conference of all stakeholders, led by the ABC itself
to discuss how these outcomes can be assured.
Ø Review the
digitisation and rights issues with ABC Archives and find a path to enabling
free public online access to the digitised archives from 1934-1990 (in a
similar way that the UK BBC Archive is available).
Ø Develop state
of the art, highly automized, robotic, centralised ‘super centres’ that are
shared facilities between agencies for cultural heritage items including:
o
Digitisation centre for audio-visual (For NFSA, NAA, AWM,
NLA)
o
Digitisation centre for paper
o
Physical storage for paper/archives/records. (160 agencies
are currently outsourcing storage of 1700km records + NAA, NLA, AWM have
approx. 500km of content in storage).
o
data storage centre for current born digital data that will
become archives.
o
data archive and long-term preservation for digitised and
sentenced data.
Ø Assess
feasibility and cost of digitising 100% of NAA records and making them full-text
searchable (currently digitised records are not full-text searchable).
Ø Provide funding
to describe and make publicly available the 64% of the hidden physical NAA
archive.
Ø Access
clearance issues:
o
Address the backlog and provide a fixed compensation fee to
clients who have been waiting over the agreed times of the Archives Act prior
to the 25 April 2019 amendment.
o
Reverse the recent 25 April 2019 amendments to the Archives
Act which significantly increase the waiting times of clients from months to
years; discourage use of archives for research assignments and group study; and
reduce the ability of the client to question response times.
o
Have a risk managed approach to access clearance, rather
than a risk averse/conservative approach.
o
Place new requirements on agencies to access clear records
within a clearly specified and short timeline for example three months. If any
agency fails to respond within the timeframe then the records will automatically
deemed to be cleared and provided to the client (risk managed approach).
o
Ensure there is ‘how to’ training, procedures and guidance
in place for agencies on access clearance, and a NAA helpline to call for
advice.
o
Make completely transparent through a client portal the
progress through the workflow (like a parcel delivery).
o
Place responsibility of keeping the client informed on the NAA,
rather than the client having to follow up.
o
Change the manual process of inspection i.e. a person has
to read every page of every record into a fully or semi-automated digital
process.
o
Consider Rose Holley’s ‘Operation
Bumble Bee’ idea of Artificial Intelligence
(AI) Access Clearance. In summary: For paper records that have clearly typed
text first digitise them and apply optical character recognition (OCR) to make
them full text searchable. Ideally this method of digitisation should be used
anyway, but is not currently being done by NAA. Run the OCR output files
through a customised Artificial Intelligence (AI) Access Clearance Programme. This
would look for specific ‘buzz’ words
within certain sequences or within proximity to other words and if buzz words
are not found the records
would be automatically cleared for public access and made available in a public
search portal. If the buzz words were found then the records would be
automatically uploaded into an enhanced version of the existing NAA Crowdsourcing platform ‘TheHive’, available only to agencies
and NAA staff. Agencies would be prompted to login to the Hive to process
records for access clearance. This would enable a combination approach of
automation and human checking by eye, with transcription of hand written notes.
Agency staff would see the highlighted buzz words in the records, redact them
if appropriate, and make access decisions. After this the records would then
either automatically load to a public search portal if access cleared, or be
held in a retention period for review again by the agency in x years.
The artificial intelligence access
clearance ‘Bumble Bee’ program could also be applied to all the ‘born digital’
records, enabling fast, effective, automatic access clearance in the future.
Combining OCR, AI and human transcription would also make historic NAA records
full text searchable for clients and enable data mining of a huge corpus. This
would enable researchers to make new discoveries that bring social, scientific
and economic benefits to the nation.
Note: Australia is unique in the
international archive world for accepting all records, before they have been
selected as records of national significance or de-classified, and then only
trying to answer these questions when the record is requested by a client. This
is seen either as an unworkable inefficient model, or potentially very
progressive. It should therefore be reviewed.
To prevent future access clearance
issues:
o
Require all agencies to de-classify records before
depositing them with NAA (digital and physical).
o
Identify and select records of national significance at
time of deposit, rather than accepting everything and trying to decide which
records are significant and should be retained later.
Ø Review all
current business process workflows for paper and digital records with a view to
identifying and resolving the issues upstream, that cause the downstream issues
e.g. access clearance wait times
Ø Take a user
centric approach and significantly enhance both the public client and the donor
agency user experience.
Ø Train all NAA
staff in brainstorming and then run brainstorming sessions for all staff in the
NAA at all locations to gather ideas about how the issues can be resolved. Some
of the issues and workflows are very complex and best brainstormed by staff who
fully understand them.
9.
Resource Estimates
It is very clear that the NAA requires both a significant
injection of capital budget and ongoing recurrent investments if it is to have
any viable chance of meeting its currently legislated responsibilities. The substantial
investment required needs to be considered in terms of: the major economic
benefits and business advantages a digital archive would bring to the nation;
the aspirations of the government to rise in world rankings for freedom and
openness (reliant upon open data); and on the innovation and creativity that
will likely occur if 100% of the collection is digitised. It is noted that it
was an election commitment of the Liberal government in May 2019 to provide
funding
to digitise all WW2 records. It is our expectation that the NAA would have
consistently provided recommendations to government for capital funding bids
over the last fifteen years, particularly for the upgrade/replacement of
RecordSearch, and a Digital Archive, and that the NAA will be making its own response
to the Tune Review, outlining the operational resources it needs. We understand
the NAA currently has approximately 350 staff across 8 offices and an
operating
budget of
$95 million. We provide some very general estimates and
examples of funding granted for similar purposes to those now needed, that we
hope will help inform members of the review panel, and support NAA requests.
Estimates for NAA Collection:
·
Critical injection to address immediate NAA IT
infrastructure and systems gap (2009 Digital Deluge Bid) $26 million
·
Digitisation of AV and addressing obsolescence
challenge $50 million
·
Description of whole collection at item level
(26 million physical items) $52 million
·
Digitisation of whole collection (39 million
paper items, using NLA $2 per page costings) $2.3 billion
Funding Granted to similar projects:
·
2009: US National Archives (NARA) $795 million
to ‘go digital’
·
2009: Netherlands film and sound archives $252
for audio-visual obsolescence challenge
·
2006-2010 New Zealand Archives and Library $40
million for digital preservation system
·
2016: State Library of New South Wales $40 million
to replace digital infrastructure and digitise
·
2018: CSIRO $43 million for new building to
house national collections e.g. insects
·
2019: AWM $500 million for extension to public
galleries
Recommendations:
Ø Review the
operating budget of the NAA against its expected functions and take into
account the lack of ownership of buildings (budget required to rent, move and
fit out premises, and staff time to plan and resource), physical location in
each state, mandated scope of collecting and activities, proportion of IT staff
required.
Ø Provide
stimulus funding to the NAA to move them from critical into stable state,
particularly for IT infrastructure, public search portal and digital archive.
Appendix 1:
PRECIS OF JOINT
NLA, NAA, NFSA ‘DIGITAL DELUGE’ FUNDING BID 2009
In
2009/10 the National Archives of Australia, in collaboration with the National
Library of Australia (NLA) and the National Film and Sound Archive (NFSA),
submitted a funding bid under the title “Dealing with the Digital Deluge”. The bid was supported by a cost-benefit
analysis.
The
bid sought funding to allow the three agencies to fulfil their legislative
mandates in an environment where records are now in digital form, and to
respond to these four challenges:
·
the digital collecting challenge:
providing robust infrastructure to collect and store “born digital” content
·
the digital preservation challenge: preserving digital content for long term access in the face of
technical obsolescence
·
the audiovisual obsolescence challenge: migrating very large audiovisual collections to digital format
to rescue them from obsolescence which will render them inaccessible
·
the digital access challenge:
converting traditional content into digital form, and delivering digital
content to make it easily accessible to the Australian people.
The funding bid identified that the NAA lacked:
·
an IT infrastructure capable of managing high
volumes of digital records of archival value
·
the capacity to ingest, preserve
and store digital records up to Top Secret classification
·
the ability to process records
created in a variety of native file formats, including formats that are now
obsolete
·
off-site digital backup facilities
to ensure fast recovery and business continuity in the event of a disaster.
·
the capacity to preserve at-risk
analogue video collections.
·
a digital asset management system
which integrates with RecordSearch
·
the capability to digitise paper
records at volumes sufficient to meet user needs, including the staff numbers
required to process access clearances.
Many of these IT capability gaps
are likely to still exist, a decade after that funding bid.
The funding bid identified the
modules of work that need to be undertaken in order to address these gaps. It estimated the funding needed to meet the
NAA’s most critical challenges of preserving the collections amounted to $26M
over five years, and $5.3M per year thereafter.
Substantially higher funding of $147 million was needed to meet all four
of the challenges set out above.
The funding bid identified the
following opportunities for the NAA to collaborate with the NLA and the NFSA:
·
Establish a
joint, distributed disaster management and backup storage arrangement
·
Establish
joint contracts for video and audio preservation, led by the NFSA
·
Establish a
joint Legacy Formats Facility, led by the NLA, to support the ingest and
management of digital collections which are received as files stored on
physical carriers that are now obsolete.
·
Establish a
joint web harvesting activity, led by the NLA.
