Home | Education & Distance Learning Articles | Article

Evaluating reference services in the electronic age

Library Trends - September 22, 2001

ABSTRACT

IN AN ELECTRONIC ERA, THE EVALUATION OF REFERENCE and related information services should still be based on the same principles used to evaluate traditional face-to-face reference services and printed reference tools. Traditional research methods--which are surveys and questionnaires, observation, individual and focus group interviews, and case studies--can be utilized very effectively in an electronic environment. However, electronic technologies offer interesting research opportunities not present in the traditional reference environment.

INTRODUCTION

At conferences and workshops on evaluating reference services, the most frequent recurring question librarians ask is, "How can the material on evaluating reference services be applied to assessing electronic reference services?" The best answer is, "Take existing methods, determine which will best meet the study goals, and then adapt those methods to the electronic environment."

In any environment, evaluating reference services still requires starting by assessing why reference services are being evaluated and what the organization plans to do with the study results. Before trying to decide how to evaluate electronic services, performance standards that set the level of achievement expected for the service should be explicitly stated. In determining the performance standards to be adopted, the organization must decide what values are crucial. Are members of the organization concerned primarily with

1. Economics--the cost or productivity of services;

2. The process--aspects of librarian/reference system and user interaction;

3. Resources--books, indexes, databases, staffing levels, equipment, design of physical or electronic environment; or

4. Products/outcomes--information or knowledge that the users obtain.

In an electronic environment the interactions between librarians and users often will no longer be truly face to face. Thus, process standards are the measures that most need to be reviewed in a digital reference environment. Librarian behaviors that are crucial in the reference-desk environment will need to be redefined for remote reference services. Work on redefining process standards has already begun. The Virtual Reference Desk (VRD) project has developed a list of User Transaction Standards to address aspects of librarian/system and user interaction. The standards address several "facets" related to quality: accessible, prompt turnaround, clear response policy, interactive and instructive (Kasowitz, Bennett & Lankes, 2000). Most of these facets address the process standards, rather than standards related to economics, resources, or products/outcomes.

In a remote electronic reference environment, accessibility and prompt turnaround could become dominant in user evaluations. Miwa (2000) used digital reference services features of acknowledgment, responsiveness, and tone of message to represent the process aspects of the reference interaction in a digital environment. She also looks at user situations as part of the process--for example, wording of the request by the user and user's ability to comprehend the message.

Broad goals for the study should be prepared in writing once a reasonable degree of consensus has been achieved on the particular set of standards that an organization wishes to emphasize. After broad goals have been developed, written objectives should be developed for each study goal. The objectives should be measurable so that, at the conclusion of the evaluation, one can identify any gaps between the present level and the desired level of reference service performance.

This present paper discusses how to apply traditional evaluation methods in an electronic reference environment once the study goals and objectives have been determined. Readers desiring additional information on setting performance standards and developing goals and objectives for reference service evaluation may wish to consult Evaluating Reference Services: A Practical Guide (Whitlatch, 2000).

All methods have strengths and weaknesses. Depending on the goals and objectives of the study, some methods will be more effective than others. As a general rule, utilizing more than one method is recommended in a single study, because the strengths of one method often compensate for the weaknesses of another. The advantages and disadvantages of the various methods may also change somewhat in an electronic environment. This paper considers how applying surveys, observation, interviews, and case studies--all traditional evaluation methods used in assessing face-to-face services--presents new opportunities and challenges in assessing electronic reference services.

SURVEYS AND QUESTIONNAIRES

Surveys or questionnaires are methods of directly collecting information on individuals' thoughts, beliefs, attitudes, and opinions, plus objective data, such as education, gender, and income. The survey method has been the most frequent way of assessing traditional reference services. In the past, surveys have been relied upon too heavily because they are the most efficient method of assessing a large group of representative users. Also. for the inexperienced researcher, surveys appear easy to design. The disadvantages, such as obtaining meaningless information from poorly designed questions and the lack of depth of information from standardized responses, are often not appreciated until too late. Another significant problem in using surveys is low response rates, particularly from surveys distributed through the mail. A substantial number of nonrespondents can bias the results; those who choose not to complete the survey might hold very different views from those who do.

Internet questionnaires can be used effectively to survey attitudes and opinions on the quality of reference service related to process (the interaction with the virtual reference service) and products/outcomes (the value of the information obtained). An Internet survey asking for an evaluation of service provided can be sent out within days after the user has received an answer. In contrast to surveys distributed in person at the reference desk or in the library, emailing the questionnaire can also be calculated to allow most users some time to use and further evaluate the information obtained through a reference interaction.

As Zhang (1999) points out, the Web provides new opportunities to conduct survey research more efficiently. Research costs for sending out Internet surveys are relatively low and the turn-around time short compared to conventional mail-in surveys. Also, email can be used effectively to follow up on paper-based surveys (Roselle & Neufeld, 1998). Most responses received in electronic format have been precoded, eliminating transcription errors and saving time and expense. McCullough (1998) notes that Web-based surveys are faster, generate more accurate information, and cost less. He has found that a respondent will typically complete a Web-based survey in about half the time it would take an interviewer to conduct that survey by telephone or in person.

Resolving the technical problems with Internet surveys requires greater technical expertise on the part of the researcher than does research conducted with traditional survey methods. However, services that provide Web survey forms and guidance to assist researchers in designing and developing Internet surveys are becoming common. Names and Web addresses for some of the services that have been positively discussed on the Academy of Management Research Listserv, rmnet@listserv.unc.edu, are provided in the Appendix.

Zhang (1999) also reviews potential problems and concerns related to Internet-based surveys. One of the greatest strengths of survey research is the ability to randomly select respondents in a manner that ensures a sample representative of the target population. In telephone surveys, respondents are randomly selected, but most online poll respondents are self-selected (Pew Research Center, 1999). The greatest difficulties with Internet surveys occur when the survey does not reach certain types of respondents who need to be included in the survey population. Biased samples and returns can be a major problem because certain social groups are underrepresented among Internet users.

1 2 3 4 Next »

If you would like to discuss any of the issues raised in this article with hundreds of other Education & Distance Learning enthusiasts from around the world, please feel free to visit the discussion forums & post a message.

Discuss this article in the discussion forums now.

Popular Education & Distance Learning Discussions From The Past

Medical from US Virgin Islands. **no pre-med**?? (6 posts)
by danpres2k - Last post on: 03-18-04 08:21
Hi, My friend is a computer science major and since the IT market has been dwindling for a while in the turmoil of enconomy, he is having second opinion on his career. He wants to go to a medical school now! but since he has already wasted nearly four years in getting the CS degree, there is a s... (Read More)

The public needs my service (1 posts)
by Roy Tumak - Last post on: 11-18-03 21:48
Look, I know it's morally injust to: 1)hide certain information or personal particulars from a potential employer, or to people who may rely on such information to make an informed decision such as buying a book, OR 2)use an unaccredited degree, OR 3)misrepresent an unaccredited degree as... (Read More)

Southern Eastern University (4 posts)
by John MacKenzie - Last post on: 01-01-04 13:11
Is anyone able to comment on the SAS Law School and The National Association of Paralegals'(both of the UK)joint law degree by correspondence awarded by a US university called Southern Eastern which is apparantly registered in Arkansas. Check out the website at www.saslawschool.org.uk The Unive... (Read More)

The "Expert" in Massage Parlors (1 posts)
by Minneapolis Pussy Hunter - Last post on: 08-01-04 19:34
The following question is for Gregg DesElms, a member of Degreecrapinfo: With your charming personality you could have met your real love thru the internet, why go the low route and pay $40000 for a hooker in a massage parlor??? Please, don't tell me you did it out of kindness and no return... (Read More)

You must register before posting in the Education & Distance Learning discussion forums. It's free & only takes a few seconds. Please also remember that no advertising is allowed...
Enter The Forums Here