Books can survive for centuries, but what about a CD Rom, a floppy disk, email or that website you looked at today? The trouble with these media forms is they could be effectively gone when the delete button is pressed, websites are updated, or the current format is no longer supported by manufacturers.
This is one of the greatest challenges facing the National Library and Archives New Zealand as they grapple with what to preserve for the future, or keep as a national record.
When it comes to technology, the National Library of New Zealand is taking a global lead as it keeps a record of what is happening today. However, putting materials online is not just for future historians but to also educate people today. New Zealand Archives, on the other hand, receives its material years after the government or some other appropriate body has finished with it.
But, a whole range of schemes are underway for New Zealand to preserve its digital “memory”.
The centrepiece of the project is the $24 million National Digital Heritage Archive, whose contents will eventually be available to researchers, with most of these available online.
The National Library is also developing a National Content Strategy, to help it decide what should be stored, how it should be managed and who should gain access.
It has also created EPIC, which through their libraries, offers New Zealanders access to more than 16,000 full text journals from New Zealand and overseas.
Matapihi, created in partnership with Archives New Zealand, is an image database, containing some 50,000 images of New Zealand places, events and people.
AnyQuestions was created last year as an online reference for New Zealand school students.
Archives New Zealand also created Archway, an $8 million search engine facil-ity that replaces paper-based systems that allow the documentation of government functions, agencies and records from 1830 to the present day. It went online in 2005.
It is currently creating a National Register of Archives and Manuscripts (NRAM)?–?a centralised web register of archives held in many New Zealand repositories.
Due for completion in 2007, the project is said to be particularly effective for smaller, community-based institutions which can use the NRAM system instead of investing in developing expensive systems of their own.
More than 8000 separate collections are documented on the system, which is used by researchers throughout New Zealand and internationally. Archives New Zealand plans to develop the potential for NRAM to provide a portal to a wider range of archival resources.
Taking the lead
Graham Coe believes that because of its size, New Zealand is the perfect place for trailblazing ideas to come to fruition.
Coe is director of digital innovation services at the National Library, spearheading its digital developments, with a central role of developing and maintaining relationships with both the private and government sectors in New Zealand and overseas.
Wellington-based Coe says New Zealand is one of the first countries globally to look at how to preserve its “digital heritage” in the same way his department collects print materials.
The National Digital Heritage Archive began a few years ago and is currently in its design phase. It will be open for public access about 18 months from now.
The project involves what Coe calls “genuine partnerships” with vendors. Prime amongst them are Sun Micro-systems, which recently declared the Library “a centre of excellence” for the
use and testing of Sun hardware and software. The National Library is also working closely with Elsevier, which produces specialist archiving software.
“We want to avoid developing bespoke software which the department is totally responsible for its maintenance. The digital age is changing rapidly in format and those charged with preserving the digital heritage of their nation will need to change their software. The vision is we will be a major player in an international market of institutions and collectively the institutions will move forward,” Coe explains.
The software will be written by the vendor with Library staff “tweaking it at the edges” to help it integrate with its systems.
The arrangement involves the National Library getting “a good price” in return for the vendors receiving help in getting a commercial product.
“Our partnership model is unique and requires constant massaging. A partnership depends on relationships. Neither partner is trying to commercially exploit the other. Each has strengths and weaknesses and you have to help each other,” say Coe.
“Problems are usually communication issues. People need to move away from defending their position and think collectively about creating a product that is attractive to a much broader audience than one’s own institution so you don’t get into a cul-de-sac.”
There are regular conference calls between the Library’s design team in Wellington and Sun’s software team in Chicago. Videoconferencing over the web and Skype phone links are used, with calls often lasting more than an hour.
“Free phone calls internationally is changing the way we work. We work in true partnership,” says Coe.
The partnership also involves creating a user community, with members coming from countries that are expressing interest or signing up to use the software.
The library is working on how and what material to receive, similar to the legal deposit scheme for print publishers.
Specialist robot tools are being developed to allow the archiving community to customise their website harvesting, say on a particular topic.
“We are working on more refined tools to pick up things of national significance, general events, sporting events, political parties. The idea is to give a snapshot of the 21st century in a similar way to newspaper archives of the past. We have an archive of 19th century provincial newspapers,” he says.
The National Library still contains much paper-based material, including a million pages of 19th century newspapers. Magazines such as Women’s Weekly are also kept on microfilm format.
“Women’s Weekly and the NZ Listener are prime candidates for online resourcing, but we will want to work with their publishers,” he says.
Coe says the record of the 21st century will be much richer, with oral histories and video to create a very vibrant and interesting view. This will fuelled by a research community and general public who will have greater expectations, especially as the National Library moves from just serving libraries to include the general population.
“Our role is becoming broader in the digital age. I hope that we become more noticeable and more people aware of our existence.”
Naturally, much storage will be needed, some five to seven terabytes a year. The National Library will have server farms storing this material, with “sophisticated” back-up. The basement of the National Library building in Wellington is earthquake- proof and discussions are underway for offsite back-up with institutions in Auckland and elsewhere.
Such changes may even see changes in the role of librarians, who may not be called librarians in the future. “The fundamental reason for facilitating access will remain but it may require people with a computer science degree in future. Their role has changed already and they are challenged by new delivery methods like text messaging. AnyQuestions is aimed at teenagers so the delivery method reflects what they are comfortable with?–?email and text messages,” Coe continues.
Coe says his role is 70 to 80 per cent trying to look ahead or engaging in negotiations with partners. The remainder is spent on operational issues, like people management, administration and budgeting for his IT team of 40.
Coe says he gets “a big buzz” from his ability to influence the direction the library takes with technology. He is currently promoting a scheme for a People’s Network of Records similar to what exists in Britain. “The National Library is pioneering and has the attention of lots of other nations.”
However, as a state agency, it faces constraints such as funding and a conservative approach to risk taking.
“Some might say innovation in the public sector is an oxymoron. Certainly innovation in the public sector is harder because the state sector tends to be risk averse and innovation and risk tend to go hand in hand.
“I see myself as an innovator, even though I am old bugger,” says the 62-year-old. “Sam Morgan says he needs 20-year-olds because they have the ideas. I think some of us old grey-haired people can give them a run for it occasionally!”
Context is king
National Archives chief information officer Ken Spagnolo says while Archives and the library often works together, there are differences between the two.
Libraries take more widespread general material, whereas Archives focuses more on government documents. Libraries tend to think about the subject matter, while Archives will think more of the collection.
For example, if the Library received a large donation, they would split it into sections, whereas Archives would keep it as a whole, never splitting records because it believes, “Context is king.”
The Public Records Act 2000 puts the onus on government departments to manage records well but it does not say how this should be done. Thus, Archives New Zealand has yet to start the process of digitising paper-based records and as yet, has no servers and scanners for this purpose.
Spagnolo says digitising records is the biggest issue currently facing archivists. Digitisation is very attractive because by letting people see items online, they are more accessible to the public than being in a physical office.
Digitisation also helps preserve a fragile document as frequent manual handling of something would damage it.
However, as Archives New Zealand works out how to store items, say in “born-digital format”, how does it ensure the data will be stored forever?.
Archives New Zealand is just starting to receive records from the 1980s and the formats they were stored in will have changed.
One way is translating such material to new formats and media, another is preserving the technology, so Archives becomes a technology museum.
Victoria State in Australia takes a middle way in taking a document in its original format. They make a PDF of it and keep both together. These form the digital archive that is stored, allowing someone to come back and look at the original.
“The idea is to preserve as much information as possible. You never get rid of the original as that is interesting in its own right,” says Spagnolo.
In July, Archives New Zealand formed a Digital Preservation Team to produce a pilot to test out the various schemes
available globally. From this, the depart-ment will produce a business case which it will then take to the government.
Archives New Zealand is working with the State Services Commission on issues such as Digital Rights Management and the protection of intellectual property. In time, various government agencies will produce a policy and action plan for an “expected large mass of digital data”.
Spagnolo says Archives New Zealand has transferred its own enterprise administration and business systems to technology like any other business.
But preparing for a digital age has led to a greater focus on technology, leading to Spagnolo becoming the department’s first chief information officer in 2004. This replaced a corporate services approach, with IT forming part of that. Further restructuring followed in April 2006, creating an IS group, giving Spagnolo responsibility for ICT, the archives library, its own internal records management functions, plus internal and external communications. He also develops strategy for the department’s own information management and runs the IS group.
Archives New Zealand uses an electronic document management scheme from Objective in Australia. Its records management uses a business classification scheme and organises records by function, rather than the old method of subject.
Much was driven by the Public Records Act 2005 which places obligations on public and private sector organisation on storing information.
“CIOs need to know how they go about documenting their business activities, whether you are a government department or private enterprise,” explains Spagnolo.
“In a government department, you have public activities and you need to provide a certain level of access. Citizens need to have trust in government which is why we keep these records.
“In the private sector, it is more a legal context. You can end up in court if you don’t have good records systems and you cannot prove your point.”
Thus, organisations, including Archives New Zealand, need to look at enterprise content management systems (ECMS) to manage their data.
A Google of its own
Spagnolo says Archway is the biggest ever project Archives has ever done, one he brands as “a Google for New Zealand Archives”.
“It’s more than just a finding aid. It has defined our work. It has two million items and is a listing of the archives,” he says.
Genealogists use the system a lot, so for a ship that arrived in New Zealand in 1840, people can archive the customs records, their ancestor name or boat and then they can request the records.
Spagnolo, who is originally from Baltimore, Maryland, says helping Archives and other government agencies with the use of technology and the issues around digital preservation is his main personal challenge. “I’m very much involved in these discussions. It is always changing but we are not there yet because there’s new information every week. People are producing papers every week.”
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.