Already a member?
Sign in
| Version | User | Scope of changes |
|---|---|---|
| Apr 4 2008, 9:31 AM EDT (current) | steve.frye | 2157 words added |
| Apr 4 2008, 9:30 AM EDT | steve.frye |
Changes
Key: Additions Deletions
MetaLib Assessment Literature Review
MetaLib Assessment Working Group
Usability Study Protocols
Surveys
Belliston, C. J., Howland, J. L., & Roberts, B. C. (2007). Undergraduate use of federated searching: A
survey of preferences and perceptions of value-added functionality. College & Research
Libraries, 68, 472-486.
The Consortium of Church Libraries and Archives (CCLA) in Utah conducted a usability study of their federated search engine, WebFeat, shortly before the system was up for license renewal. Three of their member academic libraries—Brigham Young University, Brigham Young University-Idaho, and Brigham Young University-Hawaii—participated. Undergraduates in the study (95) were given two hypothetical research assignments and conducted research using two methods: federated searching and non-federated searching. They then completed a questionnaire about their experiences. The study found that most students (70%) preferred federated searching over non-federated searching. Also, although on average 11% saved time using federated searching, only Brigham Young University students showed “a statistically significant difference [4 minutes, 11 seconds] between time required for research and the search method used” (477). Finally, the study concluded that on average, 17% of the students were “more satisfied with the results found through federated searching” (477).
Lambert, L. D., & Dabbour, K. S. (2007). Librarian perspectives on teaching metasearch and federated
search technologies. Internet Reference Services Quarterly, 12(3/4), 253-278.
This article reported on the assessment of both librarians’ and students’ reactions to metasearching. Two surveys of librarians were done in 2005: an online survey was sent to several listservs in the fall, and a small group discussion exercise was done at a conference held by California Academic and Research Libraries South in December 2005. There were only 33 responses to the online survey, and over 30% of those did not have a federated search system in their libraries. The group exercise had more participants—95—but only 10 of those had a federated search system in their libraries. The student assessment was an online survey of students at California State University Northridge. The survey was posted for 5 months from August 2005-January 2006; 88 students responded. More than half of these were graduate students, due to the heavy promotion of Metasearch and the survey by one librarian to graduate students in one of the colleges. Among the survey’s findings, most of the students expected Metasearch to be easier to use than native databases, and most felt it met that expectation. However, despite this, more than half of the students thought Metasearch needed instruction from a librarian (271).
Tang, R. (2007). User perceptions of MetaLib combined search: An investigation of how users make sense of federated searching. Internet Reference Services Quarterly, 12(1), 211-236.
This study by the Washington Research Library Consortium (WRLC) was conducted to measure the users’ understanding of federated searching. The investigation was conducted after the implementation of MetaLib and in response to low use of the system. The participants in the survey included librarians and students. After answering questions about their background and experience with federated searching, the participants were presented with screen shots of a search simulation designed to probe participants understanding of the search process and outcome. In addition, participants were asked to describe their understanding of the relationship between MetaLib, SFX and three sources of information—databases, library catalogs, and full text articles—and demonstrate this understanding by drawing a sketch. The student participants were library and information science students. These students viewed MetaLib's metasearch primarily as a tool for locating full text while librarians viewed MetaLib's metasearch as a secondary search tool with disappointing performance
Observations/task-based assessment
Haya, G., Nygren, E., & Widmark, W. (2007). Metalib and Google Scholar: A user study. Online
Information Review, 31, 365-375.
This study was done using 32 students from Uppsala University in Sweden, accessing resources available at Stockholm University. All of the students searched for academic articles for their theses using both Google Scholar and MetaLib, but one group received training before beginning their search. Each student searched with both systems for 20 minutes each, and in between the two sessions the students completed a questionnaire. Students saved articles they felt were relevant to their papers, and these were later analyzed as well. The study was conducted in a usability lab, which included audio and video recording of the session. The study found that students found more documents using Google Scholar, and those receiving instruction also found more articles from peer-reviewed journals. Student responses on the questionnaires found more positive reactions to Google Scholar over MetaLib; however, overall their responses to the two systems were fairly neutral. The most common problem with MetaLib given by students was the complexity of the system and problems with the interface (e.g., the browser’s back button did not work well with MetaLib). The study concluded that instruction “has impact” (373) and that both Google Scholar and MetaLib have a place in the search environment.
Mestre, L. S., Turner, C., Lang, B., & Morgan, B. (2007). Do we step together, in the same direction, at
the same time? How a consortium approached a federated search implementation. Internet
Reference Services Quarterly, 12(1/2), 111-132.
The Five College Libraries of Western Massachusetts consortium implemented MetaLib beginning in Spring 2004. This article described the planning and decision making process, the committees involved, designing the interface, databases to include, etc. The consortium named their implementation of MetaLib as CrossSearch. However, of the libraries, only the University of Massachusetts-Amherst decided to introduce CrossSearch to their classes; the other libraries postponed their implementation until ExLibris could improve the interface to allow for more customization. Three librarians at UMass included CrossSearch in some library instruction sessions to evaluate student response. The librarians used Categories and Sub Categories within CrossSearch to help students identify databases useful to their topic areas. The article discussed one librarian’s introduction in particular. She showed her students two databases in the native interfaces and had them locate one article in each. The students were then shown CrossSearch, and asked to search CrossSearch using the same topics they had searched earlier in the two databases. Students’ reaction to CrossSearch was positive. Several noticed the reduction in limiting options, but felt “the trade off might be worth it in many situations” (129). The students were also sent a follow-up survey, although the article did not present the findings of those surveys.
Ochoa, M., Jesano, R., Nemmers, J. R., Newsom, C., O’Brien, M., & Victor, P. (2007). Testing the
federated searching waters: A usability study of MetaLib. Journal of Web Librarianship, 1(3),
47-66.
The University of Florida conducted usability testing of MetaLib prior to implementing the system. Fifteen faculty, graduate, and undergraduate students participated in the testing and discussion of the product. The participants were given a pre-test questionnaire asking about their use of library resources and their computer and searching skills. They then had the opportunity to explore MetaLib for 3-5 minutes, and answered a questionnaire about their initial reactions. The second stage of the usability test involved a “scenario-based testing session” (53). Participants were given tasks on index cards and asked to think out loud as they completed those tasks. Responses were recording using Camtasia. One task force member took notes, while another answered participant questions. After the session, the participants answer a post-test questionnaire. They were also invited to attend a focus group session to discuss MetaLib; these sessions were held two weeks after the testing. The usability task force found that MetaLib is not intuitive, users did not retrieve results they anticipated, navigability of the system was poor, and most of MetaLib’s features were not easily understandable (55). However, the task force also concluded that their methodology was limited due to a short timeframe. They also felt that increasing the test group size and finding a test group more representative of the university population would have been beneficial.
Ponsford, B. C. (2007). User expectations in the time of Google: Usability testing of federated searching. Internet Reference Services Quarterly, 12(1), 159-178.
User testing at Texas A&M as part of MetaLib implementation used the Rubin usability factors: usefulness, effectiveness or ease of use, learnability, and attitude or likeability. The protocol used undergraduate and graduate students and faculty volunteers in two stages of usability testing, an assessment stage and a validation stage to confirm the success of changes made after the initial testing. Participants completed pre and post test surveys to gauge previous experience, then completed a set of typical library tasks (simple and complex topic searches, author & title searches, finding a specific journal, and finding a specific database). The sessions were recorded using Camtasia studio and a facilitator encouraged comments and took notes. The study detailed interesting results related to use of Tabs in design, preferred brief view for results display, effectiveness of subject based Quick Sets, interaction with browser commands and interaction with SFX.
Wrubel, L., & Schmidt, K. (2007). Usability testing of a metasearch interface: A case study. College & Research Libraries, 68(4), 292-311.
Researchers conducted observation-based formal usability testing on students at four campuses in the University System of Maryland and Affiliated Institutions (USMAI). Eighteen undergraduate students participated in the study. The students were presented with a list of topics and asked to locate two articles or books using MetaLib’s Quick Search and Cross Search functions. One of the researchers acted as a facilitator during the process, encouraging interaction and explanation of search strategies. The sessions were recorded using Morae software and a web camera. The study concluded that students do perceive metasearch to be useful, but found aspects of the interface design and search functionality frustrating, especially in comparison to internet search engines.
Interviews
Reeb, B., D'Ignazio, J., & Law, J. (2006). Federated search observed in the context of student writing: Taking steps towards improving user experience. College & Research Libraries News, 67(6), 352-355.
This study reported on the usability testing of federated search tools conducted at the University of Rochester. The protocol used contextual inquiry interviews with nine undergraduate students describing the methods that they used to gather research materials for a recently completed assignment. The students basically taught the interviewer how they selected an index, their search strategies, evaluation of results and how they accessed the full text. The sessions were recorded and throughout the interviews the interviewer maintained a relaxed, conversational rapport and sought additional information from the test subject in an attempt to reveal the student’s selection and evaluation processes. The study included very interesting findings related to design, context, and interpretation of bibliographic instruction.
Other
Avery, S., Ward, D., & Hinchliffe, L. J. (2007). Planning and implementing a federated searching system: An examination of the crucial roles of technical, functional, and usability testing. Internet Reference Services Quarterly, 12(1/2), 179-194.
A detailed description of three levels of testing recommended as a part of the planning and implementation of a federated search system. The first level is an internal technical testing, including authentication, interaction with native databases and standards compliance. The authors also recommend internal functionality testing by an implementation team, including a thorough examination of the search interface, default search settings, search options, results displays, handling of duplicate records, relevancy ranking and any other issues that might prove to be problematic when assisting users. The third recommendation is to conduct usability testing utilizing a pre-test, task oriented test, and post-test format. The authors make general suggestions based on their review of other organizations’ usability testing protocols, including performing the tests early and often, using simple means of recording observations and limiting the number of participants, but they do not provide details on any one protocol used by their institution.
Cervone, F. (2005). What we’ve learned from doing usability testing on open URL resolvers and
federated search engines. Computers in Libraries, 25(9), 10-14.
This article provides a review of what Northwestern University has found on how users interact with federated searching, both through their own usability testing and review of the literature of other studies. Research has found that dividing databases into subject areas, while seemingly a good idea, is not always what patrons prefer. Many subject areas are interdisciplinary and do not lend themselves into neat categorization of databases. Also, by using the Quick Search broad topic areas, patrons may use databases outside of their specific fields, which could lead to citations that otherwise might not be found. Another problem with categorizing databases is that this often results in long lists of database names, which students find both intimidating and confusing. Although librarians want users to be aware of all of the databases in their field of interest, even those not searchable via federated searching, research has determined that patrons “almost universally” do not use databases within a federated search system unless it can be searched via that system (13). Cervone makes a very good concluding point: “it is critical for [librarians] to make sure that [openURL and federated searching] work—from the patrons’ perspective” (14).
MetaLib Assessment Working Group
Usability Study Protocols
Surveys
Belliston, C. J., Howland, J. L., & Roberts, B. C. (2007). Undergraduate use of federated searching: A
survey of preferences and perceptions of value-added functionality. College & Research
Libraries, 68, 472-486.
The Consortium of Church Libraries and Archives (CCLA) in Utah conducted a usability study of their federated search engine, WebFeat, shortly before the system was up for license renewal. Three of their member academic libraries—Brigham Young University, Brigham Young University-Idaho, and Brigham Young University-Hawaii—participated. Undergraduates in the study (95) were given two hypothetical research assignments and conducted research using two methods: federated searching and non-federated searching. They then completed a questionnaire about their experiences. The study found that most students (70%) preferred federated searching over non-federated searching. Also, although on average 11% saved time using federated searching, only Brigham Young University students showed “a statistically significant difference [4 minutes, 11 seconds] between time required for research and the search method used” (477). Finally, the study concluded that on average, 17% of the students were “more satisfied with the results found through federated searching” (477).
Lambert, L. D., & Dabbour, K. S. (2007). Librarian perspectives on teaching metasearch and federated
search technologies. Internet Reference Services Quarterly, 12(3/4), 253-278.
This article reported on the assessment of both librarians’ and students’ reactions to metasearching. Two surveys of librarians were done in 2005: an online survey was sent to several listservs in the fall, and a small group discussion exercise was done at a conference held by California Academic and Research Libraries South in December 2005. There were only 33 responses to the online survey, and over 30% of those did not have a federated search system in their libraries. The group exercise had more participants—95—but only 10 of those had a federated search system in their libraries. The student assessment was an online survey of students at California State University Northridge. The survey was posted for 5 months from August 2005-January 2006; 88 students responded. More than half of these were graduate students, due to the heavy promotion of Metasearch and the survey by one librarian to graduate students in one of the colleges. Among the survey’s findings, most of the students expected Metasearch to be easier to use than native databases, and most felt it met that expectation. However, despite this, more than half of the students thought Metasearch needed instruction from a librarian (271).
Tang, R. (2007). User perceptions of MetaLib combined search: An investigation of how users make sense of federated searching. Internet Reference Services Quarterly, 12(1), 211-236.
This study by the Washington Research Library Consortium (WRLC) was conducted to measure the users’ understanding of federated searching. The investigation was conducted after the implementation of MetaLib and in response to low use of the system. The participants in the survey included librarians and students. After answering questions about their background and experience with federated searching, the participants were presented with screen shots of a search simulation designed to probe participants understanding of the search process and outcome. In addition, participants were asked to describe their understanding of the relationship between MetaLib, SFX and three sources of information—databases, library catalogs, and full text articles—and demonstrate this understanding by drawing a sketch. The student participants were library and information science students. These students viewed MetaLib's metasearch primarily as a tool for locating full text while librarians viewed MetaLib's metasearch as a secondary search tool with disappointing performance
Observations/task-based assessment
Haya, G., Nygren, E., & Widmark, W. (2007). Metalib and Google Scholar: A user study. Online
Information Review, 31, 365-375.
This study was done using 32 students from Uppsala University in Sweden, accessing resources available at Stockholm University. All of the students searched for academic articles for their theses using both Google Scholar and MetaLib, but one group received training before beginning their search. Each student searched with both systems for 20 minutes each, and in between the two sessions the students completed a questionnaire. Students saved articles they felt were relevant to their papers, and these were later analyzed as well. The study was conducted in a usability lab, which included audio and video recording of the session. The study found that students found more documents using Google Scholar, and those receiving instruction also found more articles from peer-reviewed journals. Student responses on the questionnaires found more positive reactions to Google Scholar over MetaLib; however, overall their responses to the two systems were fairly neutral. The most common problem with MetaLib given by students was the complexity of the system and problems with the interface (e.g., the browser’s back button did not work well with MetaLib). The study concluded that instruction “has impact” (373) and that both Google Scholar and MetaLib have a place in the search environment.
Mestre, L. S., Turner, C., Lang, B., & Morgan, B. (2007). Do we step together, in the same direction, at
the same time? How a consortium approached a federated search implementation. Internet
Reference Services Quarterly, 12(1/2), 111-132.
The Five College Libraries of Western Massachusetts consortium implemented MetaLib beginning in Spring 2004. This article described the planning and decision making process, the committees involved, designing the interface, databases to include, etc. The consortium named their implementation of MetaLib as CrossSearch. However, of the libraries, only the University of Massachusetts-Amherst decided to introduce CrossSearch to their classes; the other libraries postponed their implementation until ExLibris could improve the interface to allow for more customization. Three librarians at UMass included CrossSearch in some library instruction sessions to evaluate student response. The librarians used Categories and Sub Categories within CrossSearch to help students identify databases useful to their topic areas. The article discussed one librarian’s introduction in particular. She showed her students two databases in the native interfaces and had them locate one article in each. The students were then shown CrossSearch, and asked to search CrossSearch using the same topics they had searched earlier in the two databases. Students’ reaction to CrossSearch was positive. Several noticed the reduction in limiting options, but felt “the trade off might be worth it in many situations” (129). The students were also sent a follow-up survey, although the article did not present the findings of those surveys.
Ochoa, M., Jesano, R., Nemmers, J. R., Newsom, C., O’Brien, M., & Victor, P. (2007). Testing the
federated searching waters: A usability study of MetaLib. Journal of Web Librarianship, 1(3),
47-66.
The University of Florida conducted usability testing of MetaLib prior to implementing the system. Fifteen faculty, graduate, and undergraduate students participated in the testing and discussion of the product. The participants were given a pre-test questionnaire asking about their use of library resources and their computer and searching skills. They then had the opportunity to explore MetaLib for 3-5 minutes, and answered a questionnaire about their initial reactions. The second stage of the usability test involved a “scenario-based testing session” (53). Participants were given tasks on index cards and asked to think out loud as they completed those tasks. Responses were recording using Camtasia. One task force member took notes, while another answered participant questions. After the session, the participants answer a post-test questionnaire. They were also invited to attend a focus group session to discuss MetaLib; these sessions were held two weeks after the testing. The usability task force found that MetaLib is not intuitive, users did not retrieve results they anticipated, navigability of the system was poor, and most of MetaLib’s features were not easily understandable (55). However, the task force also concluded that their methodology was limited due to a short timeframe. They also felt that increasing the test group size and finding a test group more representative of the university population would have been beneficial.
Ponsford, B. C. (2007). User expectations in the time of Google: Usability testing of federated searching. Internet Reference Services Quarterly, 12(1), 159-178.
User testing at Texas A&M as part of MetaLib implementation used the Rubin usability factors: usefulness, effectiveness or ease of use, learnability, and attitude or likeability. The protocol used undergraduate and graduate students and faculty volunteers in two stages of usability testing, an assessment stage and a validation stage to confirm the success of changes made after the initial testing. Participants completed pre and post test surveys to gauge previous experience, then completed a set of typical library tasks (simple and complex topic searches, author & title searches, finding a specific journal, and finding a specific database). The sessions were recorded using Camtasia studio and a facilitator encouraged comments and took notes. The study detailed interesting results related to use of Tabs in design, preferred brief view for results display, effectiveness of subject based Quick Sets, interaction with browser commands and interaction with SFX.
Wrubel, L., & Schmidt, K. (2007). Usability testing of a metasearch interface: A case study. College & Research Libraries, 68(4), 292-311.
Researchers conducted observation-based formal usability testing on students at four campuses in the University System of Maryland and Affiliated Institutions (USMAI). Eighteen undergraduate students participated in the study. The students were presented with a list of topics and asked to locate two articles or books using MetaLib’s Quick Search and Cross Search functions. One of the researchers acted as a facilitator during the process, encouraging interaction and explanation of search strategies. The sessions were recorded using Morae software and a web camera. The study concluded that students do perceive metasearch to be useful, but found aspects of the interface design and search functionality frustrating, especially in comparison to internet search engines.
Interviews
Reeb, B., D'Ignazio, J., & Law, J. (2006). Federated search observed in the context of student writing: Taking steps towards improving user experience. College & Research Libraries News, 67(6), 352-355.
This study reported on the usability testing of federated search tools conducted at the University of Rochester. The protocol used contextual inquiry interviews with nine undergraduate students describing the methods that they used to gather research materials for a recently completed assignment. The students basically taught the interviewer how they selected an index, their search strategies, evaluation of results and how they accessed the full text. The sessions were recorded and throughout the interviews the interviewer maintained a relaxed, conversational rapport and sought additional information from the test subject in an attempt to reveal the student’s selection and evaluation processes. The study included very interesting findings related to design, context, and interpretation of bibliographic instruction.
Other
Avery, S., Ward, D., & Hinchliffe, L. J. (2007). Planning and implementing a federated searching system: An examination of the crucial roles of technical, functional, and usability testing. Internet Reference Services Quarterly, 12(1/2), 179-194.
A detailed description of three levels of testing recommended as a part of the planning and implementation of a federated search system. The first level is an internal technical testing, including authentication, interaction with native databases and standards compliance. The authors also recommend internal functionality testing by an implementation team, including a thorough examination of the search interface, default search settings, search options, results displays, handling of duplicate records, relevancy ranking and any other issues that might prove to be problematic when assisting users. The third recommendation is to conduct usability testing utilizing a pre-test, task oriented test, and post-test format. The authors make general suggestions based on their review of other organizations’ usability testing protocols, including performing the tests early and often, using simple means of recording observations and limiting the number of participants, but they do not provide details on any one protocol used by their institution.
Cervone, F. (2005). What we’ve learned from doing usability testing on open URL resolvers and
federated search engines. Computers in Libraries, 25(9), 10-14.
This article provides a review of what Northwestern University has found on how users interact with federated searching, both through their own usability testing and review of the literature of other studies. Research has found that dividing databases into subject areas, while seemingly a good idea, is not always what patrons prefer. Many subject areas are interdisciplinary and do not lend themselves into neat categorization of databases. Also, by using the Quick Search broad topic areas, patrons may use databases outside of their specific fields, which could lead to citations that otherwise might not be found. Another problem with categorizing databases is that this often results in long lists of database names, which students find both intimidating and confusing. Although librarians want users to be aware of all of the databases in their field of interest, even those not searchable via federated searching, research has determined that patrons “almost universally” do not use databases within a federated search system unless it can be searched via that system (13). Cervone makes a very good concluding point: “it is critical for [librarians] to make sure that [openURL and federated searching] work—from the patrons’ perspective” (14).
