Digital libraries (DLs) have emerged as one of the essential scholarly information systems in support of research and teaching. Many libraries have digitized their collections, such as pictures, books, or audios, to make them available on the web. Digital libraries are relatively new phenomena and, like many new and emergent information systems, face challenges of acceptance, utilization, and evaluation. The concept of a digital library has been perceived in different ways by different groups of people. To the community of librarians and LIS researchers, a digital library is an extension and augmentation of library services combined with a remote access to digitized resources. To computer scientists, a digital library is a distributed information system or networked multimedia information system (Fox et al. 1995). In this study, our focus is on the users’ perspective. A digital library is defined as the collection of digitized or digitally born items that are stored, managed, serviced, and preserved by digital library professionals.

The exponential growth of DLs has created a need for the evaluation of these emergent information systems. Digital library evaluation has become a critical issue for both professionals and researchers. DLs have their own unique characteristics and features compared to traditional library services. Also, as an information retrieval system, digital libraries are fairly different from other types of online information retrieval systems (e.g., search engines, online databases, OPACs). DLs have well-organized metadata, browsing categories, as well as digitized items in different formats. Therefore, previous evaluation frameworks used in traditional libraries or other information retrieval systems proved to be insufficient in assessing different aspects of DLs. The evaluation of DLs is conceptually complex and pragmatically challenging (Saracevic and Covi 2000). Borgman et al. (2000) also pointed out technical complexity, variety of content, and the lack of evaluation methods posed the key challenges of digital library evaluation.

Researchers have exerted efforts to develop new frameworks and methods of digital library evaluation. A number of evaluation criteria were suggested covering different dimensions of digital libraries. The early DL projects, funded by the National Science Foundation (NSF) as part of Digital Libraries Initiatives I and II, laid groundwork in evaluation research by producing DL prototypes and frameworks (Borgman et al. 2000; Buttenfield 1999; Hill et al. 2000; Van House et al. 1996). In particular, Hill et al. (1997) identified several criteria for digital library evaluation, such as ease of use, overall appeal, usefulness and overall performance. Saracevic (2004) identified six classes of criteria: content, technology, interface, process/service, user, and context. This evaluation framework covers multiple aspects of digital libraries comprehensively, and is one of the first attempts to assess the context aspect in digital libraries. Xie’s (2006/2008) evaluation framework shifted a focus to the users and posited five types of criteria: usability, collection quality, service quality, system performance efficiency, and user feedback solicitation. In particular, when developing the evaluation framework, she analyzed users’ perceptions based on diaries, questionnaires, and interviews. Zhang (2010) validated the evaluation criteria for digital libraries. Based on Saracevic's (2004) framework, she investigated the importance of evaluation criteria using the empirical survey data from heterogeneous stakeholders. Noh (2010) identified multiple dimensions and corresponding evaluation indices for electronic resources.

In Europe, DELOS Network of Excellence has conducted a series of projects regarding the evaluation of digital libraries. DELOS is a comprehensive and large scale DL project, which represents joint activities aimed at integrating and coordinating the ongoing research efforts of the major European teams working in the digital library area. Candela et al. (2007) established DELOS Manifesto which presents a three-tier DL framework incorporating six core components such as content, functionality, quality, policy, architecture, and user. Fuhr et al. (2001) proposed an evaluation scheme for digital libraries which covers four dimensions including data/collection, system/technology, users, and usage. Based on the examination of the interactions amongst digital library components, Tsakonas et al. (2004) proposed the major evaluation foci, such as usability, usefulness, and system performance. Fuhr et al. (2007) developed a DL evaluation framework based on a DELOS model and conducted a large-scale survey of DL evaluation activities.

Among different aspects of digital libraries, usability has been one of the major concerns in evaluation. Usability consists of multiple attributes from various perspectives such as learnability, efficiency, effectiveness, memorability, errors, and satisfaction (Nielson 1993). Several researchers tried to suggest a usability evaluation model tailored to digital libraries. For example, Jeng (2005) suggested an evaluation framework on usability of academic DLs focusing on four attributes: effectiveness, efficiency, satisfaction, and learnability. Ward and Hiller (2005) suggested usability evaluation criteria specific to library services, such as completion of the task, time and effort, and reaction to the product or service. Joo and Lee (2011) developed an instrument tool to measure the usability of digital libraries. They further tested the validity and reliability of the tool. Matusiak (2012) examined the relationship between usability and usefulness, and found that user perceptions of usefulness and usability, especially perceived ease of use, play an important role in user intentions to adopt and use digital collections.

Researchers strived to identify a set of evaluation criteria for digital libraries. However, there has been relatively less effort devoted to the investigation of users’ perceptions in selecting evaluation criteria. Ultimately, digital libraries are developed to provide information and services to users, and users’ opinions should be considered in the evaluation of digital libraries. It is important that all efforts in the evaluation of digital libraries should be rooted in users’ information needs and characteristics as well as contexts involving the users of those libraries (Marchionini et al. 1998).

This study is one of the few attempts to survey users’ perceptions of evaluation criteria for digital libraries. The investigation of user perceptions is a fundamental step in devising an evaluation framework that focuses on user needs and characteristics. In this study, the authors suggested a wide range of evaluation criteria in eight dimensions of digital libraries based on the document analysis. For the suggested evaluation criteria, this study examines to what extent users perceive the importance of each criterion in the evaluation of digital libraries.

Thus far, digital library evaluation criteria have been suggested mainly by librarians or researchers. To design a user-centered digital library, the evaluation needs to reflect users’ perspective in its evaluation criteria. This study is one of a few studies that investigated users’ perceptions of evaluation criteria for digital libraries.

This study intends to examine the following research question:

What are users’ perceptions of the importance of digital library evaluation criteria?

Two-round surveys were conducted to identify the importance of evaluation criteria and appropriateness of measures from different stakeholders of digital libraries including scholars, digital librarians, and users. This paper focuses on the identification of the importance of evaluation criteria from users’ perspectives.

The authors partnered with five academic libraries across the United States to collect data. Subjects of this study were recruited from these partner libraries: (1) University of Denver, (2) University of Florida, (3) University of Nevada Las Vegas, (4) Drake University, and (5) University of Wisconsin-Milwaukee. Each institution recruited six digital library users to participate in the study. The study employed purposeful sampling strategy. The sample included academic users with prior experience interacting with digital collections. To ensure the maximum variation sampling, participants were recruited from different groups of academic users with different gender and different majors, such as Linguistics, English, Psychology, and Computer Science. The user subjects included 10 faculty, 12 graduate students, and 8 undergraduate students. A $30 gift card was given to each subject as an incentive for his/her participation of the study. Table 1 presents the demographic data of the subjects.

A comprehensive survey was administered to investigate users’ perceptions of the importance of evaluation criteria in digital library evaluation. To suggest an initial set of evaluation criteria, a comprehensive document analysis was conducted. Using keywords of different combinations of “digital library”, “evaluation”, “criteria”, and other terms, relevant research papers were collected through Google Scholar and EBSCOhost databases. In addition, five websites related to digital libraries were also analyzed, such as DigiQUAL and DELOS, to identify evaluation criteria.

Digital library evaluation criteria were extracted from the retrieved pool of documents. Based on the document analysis, ten essential dimensions of digital libraries have been identified, including collection, information organization, interface design, system and technology, effects on users, services, preservation, administration, user engagement, and context. The administration and preservation dimensions were excluded from the user survey because users don’t have enough knowledge of these dimensions. For each dimension, the authors proposed a set of evaluation criteria. To help subjects understand the meaning of evaluation criteria, the definition associated with each criterion was provided in the survey. Subjects were instructed to rate the importance of evaluation criteria using a seven point scale. Since the subjects were selected from different locations with different digital library uses, the subjects were not asked to evaluate a specific digital library, but rather based their survey responses on their past interactions with digital library systems.

Descriptive statistics was used, including mean and standard deviation, to investigate the importance of evaluation criteria. Based on average ratings, the authors ranked the evaluation criteria from the most important to the least for each dimension.

The results section is organized in eight dimensions of digital libraries: collection, information organization, interface design, system and technology, effects on users, services, user engagement, and context. In the dimension of collections, quality related evaluation criteria are the ones that users considered the most important. “Authority (6.53)”, “item quality (6.27)”, and “digitization standards (6.20)” turned out to be the top three evaluation criteria. Following these criteria, “cost (6.10)”, “format compatibility (6.10)” and “contextual information (6.10)” were ranked fourth. In contrast, “size (5.57)”, “diversity (5.77)” and “completeness (5.77)” were considered to be the least important. It seems that users cared more about the quality and less about the comprehensiveness and variety of the collections. Table 2 presents the importance of evaluation criteria in the dimension of collections.