MetaLib Assessment Literature ReviewMetaLib Assessment Working Group

--------------------------------------------------------------------------------------------------------
PART 1: Selected Articles Contained in Final Report

Usability Study Protocols

Surveys

Belliston, C. J., Howland, J. L., & Roberts, B. C. (2007). Undergraduate use of federated searching: A
survey of preferences and perceptions of value-added functionality. College & Research
Libraries, 68, 472-486.

The Consortium of Church Libraries and Archives (CCLA) in Utah conducted a usability study of their federated search engine, WebFeat, shortly before the system was up for license renewal. Three of their member academic libraries—Brigham Young University, Brigham Young University-Idaho, and Brigham Young University-Hawaii—participated. Undergraduates in the study (95) were given two hypothetical research assignments and conducted research using two methods: federated searching and non-federated searching. They then completed a questionnaire about their experiences. The study found that most students (70%) preferred federated searching over non-federated searching. Also, although on average 11% saved time using federated searching, only Brigham Young University students showed “a statistically significant difference [4 minutes, 11 seconds] between time required for research and the search method used” (477). Finally, the study concluded that on average, 17% of the students were “more satisfied with the results found through federated searching” (477).

Lambert, L. D., & Dabbour, K. S. (2007). Librarian perspectives on teaching metasearch and federated
search technologies. Internet Reference Services Quarterly, 12(3/4), 253-278.

This article reported on the assessment of both librarians’ and students’ reactions to metasearching. Two surveys of librarians were done in 2005: an online survey was sent to several listservs in the fall, and a small group discussion exercise was done at a conference held by California Academic and Research Libraries South in December 2005. There were only 33 responses to the online survey, and over 30% of those did not have a federated search system in their libraries. The group exercise had more participants—95—but only 10 of those had a federated search system in their libraries. The student assessment was an online survey of students at California State University Northridge. The survey was posted for 5 months from August 2005-January 2006; 88 students responded. More than half of these were graduate students, due to the heavy promotion of Metasearch and the survey by one librarian to graduate students in one of the colleges. Among the survey’s findings, most of the students expected Metasearch to be easier to use than native databases, and most felt it met that expectation. However, despite this, more than half of the students thought Metasearch needed instruction from a librarian (271).


Tang, R. (2007). User perceptions of MetaLib combined search: An investigation of how users make sense of federated searching. Internet Reference Services Quarterly, 12(1), 211-236.

This study by the Washington Research Library Consortium (WRLC) was conducted to measure the users’ understanding of federated searching. The investigation was conducted after the implementation of MetaLib and in response to low use of the system. The participants in the survey included librarians and students. After answering questions about their background and experience with federated searching, the participants were presented with screen shots of a search simulation designed to probe participants understanding of the search process and outcome. In addition, participants were asked to describe their understanding of the relationship between MetaLib, SFX and three sources of information—databases, library catalogs, and full text articles—and demonstrate this understanding by drawing a sketch. The student participants were library and information science students. These students viewed MetaLib's metasearch primarily as a tool for locating full text while librarians viewed MetaLib's metasearch as a secondary search tool with disappointing performance

Observations/task-based assessment

Haya, G., Nygren, E., & Widmark, W. (2007). Metalib and Google Scholar: A user study. Online
Information Review, 31, 365-375.

This study was done using 32 students from Uppsala University in Sweden, accessing resources available at Stockholm University. All of the students searched for academic articles for their theses using both Google Scholar and MetaLib, but one group received training before beginning their search. Each student searched with both systems for 20 minutes each, and in between the two sessions the students completed a questionnaire. Students saved articles they felt were relevant to their papers, and these were later analyzed as well. The study was conducted in a usability lab, which included audio and video recording of the session. The study found that students found more documents using Google Scholar, and those receiving instruction also found more articles from peer-reviewed journals. Student responses on the questionnaires found more positive reactions to Google Scholar over MetaLib; however, overall their responses to the two systems were fairly neutral. The most common problem with MetaLib given by students was the complexity of the system and problems with the interface (e.g., the browser’s back button did not work well with MetaLib). The study concluded that instruction “has impact” (373) and that both Google Scholar and MetaLib have a place in the search environment.

Mestre, L. S., Turner, C., Lang, B., & Morgan, B. (2007). Do we step together, in the same direction, at
the same time? How a consortium approached a federated search implementation. Internet
Reference Services Quarterly, 12(1/2), 111-132.

The Five College Libraries of Western Massachusetts consortium implemented MetaLib beginning in Spring 2004. This article described the planning and decision making process, the committees involved, designing the interface, databases to include, etc. The consortium named their implementation of MetaLib as CrossSearch. However, of the libraries, only the University of Massachusetts-Amherst decided to introduce CrossSearch to their classes; the other libraries postponed their implementation until ExLibris could improve the interface to allow for more customization. Three librarians at UMass included CrossSearch in some library instruction sessions to evaluate student response. The librarians used Categories and Sub Categories within CrossSearch to help students identify databases useful to their topic areas. The article discussed one librarian’s introduction in particular. She showed her students two databases in the native interfaces and had them locate one article in each. The students were then shown CrossSearch, and asked to search CrossSearch using the same topics they had searched earlier in the two databases. Students’ reaction to CrossSearch was positive. Several noticed the reduction in limiting options, but felt “the trade off might be worth it in many situations” (129). The students were also sent a follow-up survey, although the article did not present the findings of those surveys.

Ochoa, M., Jesano, R., Nemmers, J. R., Newsom, C., O’Brien, M., & Victor, P. (2007). Testing the
federated searching waters: A usability study of MetaLib. Journal of Web Librarianship, 1(3),
47-66.

The University of Florida conducted usability testing of MetaLib prior to implementing the system. Fifteen faculty, graduate, and undergraduate students participated in the testing and discussion of the product. The participants were given a pre-test questionnaire asking about their use of library resources and their computer and searching skills. They then had the opportunity to explore MetaLib for 3-5 minutes, and answered a questionnaire about their initial reactions. The second stage of the usability test involved a “scenario-based testing session” (53). Participants were given tasks on index cards and asked to think out loud as they completed those tasks. Responses were recording using Camtasia. One task force member took notes, while another answered participant questions. After the session, the participants answer a post-test questionnaire. They were also invited to attend a focus group session to discuss MetaLib; these sessions were held two weeks after the testing. The usability task force found that MetaLib is not intuitive, users did not retrieve results they anticipated, navigability of the system was poor, and most of MetaLib’s features were not easily understandable (55). However, the task force also concluded that their methodology was limited due to a short timeframe. They also felt that increasing the test group size and finding a test group more representative of the university population would have been beneficial.

Ponsford, B. C. (2007). User expectations in the time of Google: Usability testing of federated searching. Internet Reference Services Quarterly, 12(1), 159-178.

User testing at Texas A&M as part of MetaLib implementation used the Rubin usability factors: usefulness, effectiveness or ease of use, learnability, and attitude or likeability. The protocol used undergraduate and graduate students and faculty volunteers in two stages of usability testing, an assessment stage and a validation stage to confirm the success of changes made after the initial testing. Participants completed pre and post test surveys to gauge previous experience, then completed a set of typical library tasks (simple and complex topic searches, author & title searches, finding a specific journal, and finding a specific database). The sessions were recorded using Camtasia studio and a facilitator encouraged comments and took notes. The study detailed interesting results related to use of Tabs in design, preferred brief view for results display, effectiveness of subject based Quick Sets, interaction with browser commands and interaction with SFX.

Wrubel, L., & Schmidt, K. (2007). Usability testing of a metasearch interface: A case study. College & Research Libraries, 68(4), 292-311.

Researchers conducted observation-based formal usability testing on students at four campuses in the University System of Maryland and Affiliated Institutions (USMAI). Eighteen undergraduate students participated in the study. The students were presented with a list of topics and asked to locate two articles or books using MetaLib’s Quick Search and Cross Search functions. One of the researchers acted as a facilitator during the process, encouraging interaction and explanation of search strategies. The sessions were recorded using Morae software and a web camera. The study concluded that students do perceive metasearch to be useful, but found aspects of the interface design and search functionality frustrating, especially in comparison to internet search engines.

Interviews

Reeb, B., D'Ignazio, J., & Law, J. (2006). Federated search observed in the context of student writing: Taking steps towards improving user experience. College & Research Libraries News, 67(6), 352-355.

This study reported on the usability testing of federated search tools conducted at the University of Rochester. The protocol used contextual inquiry interviews with nine undergraduate students describing the methods that they used to gather research materials for a recently completed assignment. The students basically taught the interviewer how they selected an index, their search strategies, evaluation of results and how they accessed the full text. The sessions were recorded and throughout the interviews the interviewer maintained a relaxed, conversational rapport and sought additional information from the test subject in an attempt to reveal the student’s selection and evaluation processes. The study included very interesting findings related to design, context, and interpretation of bibliographic instruction.

Other

Avery, S., Ward, D., & Hinchliffe, L. J. (2007). Planning and implementing a federated searching system: An examination of the crucial roles of technical, functional, and usability testing. Internet Reference Services Quarterly, 12(1/2), 179-194.

A detailed description of three levels of testing recommended as a part of the planning and implementation of a federated search system. The first level is an internal technical testing, including authentication, interaction with native databases and standards compliance. The authors also recommend internal functionality testing by an implementation team, including a thorough examination of the search interface, default search settings, search options, results displays, handling of duplicate records, relevancy ranking and any other issues that might prove to be problematic when assisting users. The third recommendation is to conduct usability testing utilizing a pre-test, task oriented test, and post-test format. The authors make general suggestions based on their review of other organizations’ usability testing protocols, including performing the tests early and often, using simple means of recording observations and limiting the number of participants, but they do not provide details on any one protocol used by their institution.

Breeding, M. (2005). Plotting a new course for metasearch. Computers in Libraries, 25 (2), 27-29. Description

: An opinion piece that presents the inherent limitations in the federated search process and the advantages of a centralized search process like the one used by Google. Breeding also emphasizes the need for and importance of metasearching for libraries. Not a research study, no protocol used.

Cervone, F. (2005). What we’ve learned from doing usability testing on open URL resolvers and
federated search engines. Computers in Libraries, 25(9), 10-14.

This article provides a review of what Northwestern University has found on how users interact with federated searching, both through their own usability testing and review of the literature of other studies. Research has found that dividing databases into subject areas, while seemingly a good idea, is not always what patrons prefer. Many subject areas are interdisciplinary and do not lend themselves into neat categorization of databases. Also, by using the Quick Search broad topic areas, patrons may use databases outside of their specific fields, which could lead to citations that otherwise might not be found. Another problem with categorizing databases is that this often results in long lists of database names, which students find both intimidating and confusing. Although librarians want users to be aware of all of the databases in their field of interest, even those not searchable via federated searching, research has determined that patrons “almost universally” do not use databases within a federated search system unless it can be searched via that system (13). Cervone makes a very good concluding point: “it is critical for [librarians] to make sure that [openURL and federated searching] work—from the patrons’ perspective” (14).

--------------------------------------------------------------------------------------------------------
Part 2: Complete List of References Below

Directory Containing Full Text of Articles Below (Password Needed)

1. Aslam, J. A., & Montague, M. (2000). Bayes optimal metasearch: A probabilistic model for combining the results of multiple retrieval systems. Bayes Optimal Metasearch: A Probabilistic Model for Combining the Results of Multiple Retrieval Systems, , 379-381. Description: The authors introduce a a hightly technical critique of their own metasearch system for searching the 'free web'. This article has nothing to contribute to our assessment of Metalib.
2. Baer, W. (2004). Federated searching: Friend or foe? College & Research Libraries News, 65(9), 518. Description: Opinion piece that points out the positives and negatives of Federated Search.
3. Balas, J. L. (2006). Does one-stop searching really serve all? Computers in Libraries, 26(9), 42. Description: Opinion piece that barely touches on federated searching - the article is really about the two directions in retail, self-helping superstores and highly personalized niche stores... and what these trends might mean to libraries.
4. Belliston, C. J., Howland, J.L., Roberts, B.C. (2007). Undergraduate use of federated searching: A survey of preferences and perceptions of value-added functionality. College & Research Libraries, 68(6), 472-486. Description: Undergraduates undertake two hypothetical research assignments using different search method for each - one using federated searching and the other performed with nonfederated searching. They were then asked to complete a questionnaire about their experience. As they searched, they compiled a running bibliography - the resulting citations were then analyzed by librarians and faculty using two different rubrics, one created by librarians and the other created by faculty. Ninety-five undergraduates participated in this study. 70% of participants preferred federated searching. The librarian-created rubric revealed that citation sets gathered by non-federated searches were superior. The faculty-created rubric revealed no significant difference between federated and non-federated search sets.
5. Blecic, D. D., Fiscella, J. B., & Wiberley, S. E.,Jr. (2007). Measurement of use of electronic resources: Advances in use statistics and innovations in resource functionality. College & Research Libraries, 68(1), 26. Description: Discusses federated searching in the context of session and search statistics per resource. This article contains nothing of use for assessing metalib.
6. Boock, M., Nicholas, J., & Kristick, L. (2006). Continuing the quest for the quick search holy grail: Oregon state university libraries' federated search implementation. Internet Reference Services Quarterly, 11(4), 139-153. Description: This article describes the implementation of a federated search tool at Oregon State. Towards the end of the article, the author addresses the question of usability through user assessment. Apparently, a focus group was held along with the gathering of informal data from students - however, methodology of either of these methods is not included in the article.
   
7. Boss, S. C., & Nelson, M. L. (2005). Federated search tools: The next step in the quest for one-stop-shopping. Reference Librarian, (91), 139-160. Description: The authors provide a good overview of the federated search tools that existed in 2005. There is no mention of usability testing.
8. Cervone, F. (2005). What we've learned from doing usability testing on OpenURL resolvers and federated search engines. Computers in Libraries, 25(9), 10-14. Description: Excellent article that provides a review of what various usability studies and testing on the Northwestern campus have revealed about how users interact with federated search, native search, etc. Protocols are not described in detail.
9. Chen Xiaotian. (2006). MetaLib, WebFeat, and google: The strengths and weaknesses of federated search engines compared with google. Online Information Review, 30(4), 413-427. Description: The author compares MetaLib and WebFeat. Chen explains the differences between these two federated search engines and also compares them to Google. She characterizes MetaLib as offering “one-stop window-shopping – users can go to different subjects or categories to view different databases listed there but cannot search all databases with one search”. She compares conducting searches using either MetaLib or WebFeat to searches performed using Google. A good article in that it discusses many aspects of federated search, including interface, search results, speed, and convenience. -- No assessment protocol is given.
10. Cox, C. (2006). An analysis of the impact of federated search products on library instruction using the ACRL standards. Portal: Libraries & the Academy, 6(3), 253-267. Description: Author evaluates the impact that federated search *may* have on information literacy instruction through the lense of the ACRL information literacy competencies. Usability assessment is not the focus of this article.
11. Drake, M. A. (2006). Collaboration, competition, and controversy. Information Today, 23(3), 1. Description: Summary of current issues/initiatives in the information field, including the Google Library Project, peer review, born digital materials, collaboration between librarians and publishers, and GPO updates. Interesting, but not directly relevant to Metalib assessment.
12. Fahey, S. (2007). F*****ED searchers? the debate about federated search engines. Feliciter, 53(2), 62-63. Description: A brief thought piece that rehashes the familiar debate surrounding federated searching: dumbing down the search process, "good enough" search results, etc. Author contends that federated searching will improve the search experience for users. Opinion piece--no assessment protocols.
13. Fiehn, B. (2004). Federated searching: A viable alternative to web surfing! MultiMedia & Internet@Schools, 11(2), 29. Description: A basic review of federated searching for school media specialists, including a fair analysis of both strengths and weaknesses. Opinion piece--no assessment protocols.
14. Friend, F. J. (2006). Google scholar: Potentially good for users of academic information. JEP: The Journal of Electronic Publishing, 9(1), [n]. http://hdl.handle.net/2027/spo.3336451.0009.105 Description: In-depth review of the strengths and weaknesses of Google scholar as an academic research tool. Very interesting, but includes no assessment protocol.
15. Furlan, W. (2006). Library users expect link resolvers to provide full text while librarians expect accurate results. Evidence Based Library and Information Practice, 1(4), 60-63. Description: Describes and online survey and librarian focus groups conducted at two California State University campuses to assess SFX. Results of the study indicated that user and librarian expectations of SFX were slightly higer than their actual experiences.
16. Hane, P. J. (2003). The truth about federated searching. Information Today, 20(9), 24. Description: Five common misconceptions concerning federated searching are de-mystified by a representative of WebFeat. Interesting point of view from a federated search provider. No assesment protocol.
17. Hanna, K. A., Minick, M. B., & Eiszner, J. W. (2007). A nontraditional application of MetaSearch: Using ex libris' MetaLib to enhance delivery, of full-text articles in a university campus course management system. Internet Reference Services Quarterly, 12(3), 297-308. Description: Describes how IUPUI incorprated the MetaLib eShelf into the course management system to present online reading lists to students. The project is viewed as very successful.
18. Haya, G., Nygren, E., & Widmark, W. (2007). Metalib and google scholar: A user study. Online Information Review, 31(3), 365-375. A Swedish study that compared the search experience and search results of undergraduate students using MetaLib and Google Scholar. "A total of 32 undergraduate students searched academic articles for their thesis work. Searches were recorded using Morae software and were analysed along with the number of articles saved and responses to a questionnaire. All searched with both tools. Half of the students received training before searching." Google Scholar exhibited higher usability than Metalib based on the success rate in finding articles. Open ended responses to questions about the tools indicate a higher rate of satisfaction with Google Scholar, although the overall reaction to Google Scholar was only mildly positive. Overall, students were neutral to both tools. Instruction is valuable for both tools. This study was carried out in a usability lab, and relied on sophisticated tools to conduct an analsis of both the search experience and search results. This study could be difficult to replicate without these tools.
19. Helfer, D. S., & Wakimoto, J. C. (2005). Metasearching: The good, the bad, and the ugly of making it work in your library. Searcher, 13(2), 40. Description: A brief summary of the MetaLib implementation at CSU-Northridge, followed by a summary of MetaLib's strengths and weaknesses. While not a panacea, the author contends that MetaLib holds promise for the future. Opinion piece--no assessment protocol.
20. Jacsó, P. (2005). Google scholar: The pros and the cons. Online Information Review, 29(2), 208. Description: The author identified pros and cons of Google Scholar (beta version) searching shortly after its launch in Nov. 2004. Pro: serves as a federated search engine function for libraries that don't have one. Con: when results from Google Scholar were compared to results from the native database, GS usually found far fewer results than were actually available. Interesting article, but data is too outdated to be of much use; GS has likely improved in last 3 years. No relevant protocol given.
21. Jacsó, P. (2004). Thoughts about federated searching. Information Today, 21(9), 17. Description: Compares differences in result list appearance between 3 different federated search engines: MetaLib, MetaFind, and WebFeat. MetaLib's list is viewed as "the most succinct and most uniform result list." Sort options are "on par with MetaFind," but "its de-duplication feature was not activated in the tested release 3.10." Again, an interesting article, but too dated to be of much use (2004).
22. Lampert, L. D., & Dabbour, K. S. (2007). Librarian perspectives on teaching metasearch and federated search technologies. Internet Reference Services Quarterly, 12(3), 253-278. Description: 2 surveys of librarians were done in 2005: an online survey sent to several listservs (only 33 responses, and over 30% of those did not have a federated search system in their libraries), and a small group discussion exercise at a California librarians' conference (95 librarians divided into 3 groups; only 10 had a federated search system in their libraries). Also did online user survey of students in 2005 at Cal State Northridge. Copy of survey included in appendix; 88 surveys returned. Results are given (survey: Aug '05-Jan '06; federated searching started there in Aug '04). Opinion: We could possibly duplicate their online survey, but our timeframe is much shorter.
23. Lewis, N. (2002). Talking about a revolution? first impressions of ex Libris’s MetaLib. Ariadne, 32. Retrieved 12/3/2007, 2007, from http://www.ariadne.ac.uk/issue32/metalib/ Description: Discusses the experiences of the Univ. of East Anglia (UK) in their implementation of MetaLib. Written in 2002; last updated in 2004. Talks about the functionality of MetaLib (link to native databases or cross-search them). They encountered configuration issues due to differences in databases' Z39.50 servers, and had to get permission from database vendors to access these servers and get information on how to do that. Challenges: setting up access to various databases, ongoing maintenance of MetaLib resources, and the unreliability of Z39.50 servers. No assessment protocol given.
24. Lindahl, D. (2007). Metasearch in the users' context. Serials Librarian, 51(3), 215-234. Description: Discusses student usability issues with metasearch, but no information on the usability studies conducted. Advocates user centered design (UCD) when implementing metasearch and its search interface. Four categories of usability issues with metasearch "out of the box" interfaces: specifying queries, selecting databases, identifying and selecting results, and acquiring items. Gives examples on how U of Rochester libraries have addressed these issues. Also advocates automatic database selection based on user queries.
25. Luther, J. (2003). Trumping Google? Metasearching's Promise. Library Journal, 128(16), 36+. Description: Nothing new here. Discusses the pros and cons of metasearching, differences between Google searching and metasearching, reasons for librarian resistance, etc. Brief chart with descriptions of several metasearch vendors: WebFeat, Muse Global, ExLibris (MetaLib), Auto Graphics, Fretwell-Downing, GIS, Biblio-Mondo, and VTLS. No assessment protocols given. I particularly liked this fragment: "Metasearch is not a tool that allows librarians--or other expert users--to search with greater precision. It's not for us, it's for the average user."
26. Mestre, L. S. (2007). Do we step together, in the same direction, at the same time? how a consortium approached a federated search implementation. Internet Reference Services Quarterly, 12(1), 111-132. Description: Talks about the implementation of MetaLib at the Five College Libraries consortium in Western Massachusetts. Useful in that they are also in a consortial arrangement; discusses the work they did to configure MetaLib for the five libraries, and the charge of the different committees. Mentions usability testing several times, but gives little information about it, other than on pp. 128-130, where it discusses how a librarian taught MetaLib to a class and had them compare the results to the native interfaces. She also did a followup email survey with that class.
27. MetaLib working group. Retrieved 2/27/2008, 2008, from http://hul.harvard.edu/ois/projects/metalib/teams.html Description: A listing of the MetaLib committee members and its implementation sub-team members at Harvard. Sounds like they were very organized in their rollout of the system: sub-teams included content, rationalization, metadata, user interface, resources workflow and operation, usability testing/accessibility, and rollout and training. No other useful information included.
28. Miller, T. (2004). Federated searching: Put it in its place. Library Journal, , 32. Description: An opinion column on whether the online catalog should be the place to start a federated search. Since much of the library's information is no longer in the catalog, he feels the answer is no. He presents it in the following context: "If users want to find a book, they go to Amazon. If they need information, they go to Google." Nothing of use in this article.
29. Mullen, L. B., & Hartman, K. A. (2006). Google scholar and the library web site: The early response by ARL libraries. College & Research Libraries, 67(2), 106-122.
30. Needleman, M. H. (2006). An update on the NISO metasearch activity. Serials Review, 32(2), 143-145.
31. Neuhaus, C., Neuhaus, E., Asher, A., & Wrede, C. (2006). The depth and breadth of google scholar: An empirical study. Portal: Libraries and the Academy, Vol.6, no.2, Pp.127-141, 6(2), 127-141.
32. Newton, V. W., & Silberger, K. (2007). Simplifying complexity through a single federated search box. Online, 31(4), 19.
33. Nichols, D., Huntington, P., & Jamali, H. R. (2007). The use, users, and role of abstracts in the digital scholarly environment. The Journal of Academic Librarianship, 33(4), 446-453.
34. Nygren, E., Haya, G., & Widmark, W. (2006). Students' experience of metalib and google scholar. [Students' Experience of Metalib and Google Scholar] (11/6/2007)
35. Ochoa, M. (2007). Testing the federated searching waters: A usability study of MetaLib. Journal of Web Librarianship, 1(3), 47-66.
36. Parry, F. (2006). MetaGoogle! using MetaLib to deliver user-centered information services at loughborough. ALISS Quarterly, Vol.2, no.1, Pp.38-40.
37. Planning and implementing a federated searching system: An examination of the crucial roles of technical, functional, and usability testing.(2007). Internet Reference Services Quarterly, 12(1/2), 179-194.
38. Ponsford, B. C. (2007). User expectations in the time of google: Usability testing of federated searching. Internet Reference Services Quarterly, 12(1), 159-178. Description: User testing at Texas A&M as part of Metalib implementation. Interesting results related to use of Tabs in design, preferred brief view for results display, effectiveness of subject based Quick Sets, interaction with browser commands and interaction with SFX. The protocol used volunteers (ug, grad, fac). Participants completed pre and post test surveys to gauge previous experience, then completed a set of typical library tasks (simple and complex topic searches, author & title searches, find specific journal, find specific databse). The sessions were recorded using Camtasia studio and a facilitator encouraged comments and took notes.
39. Randall, S. (2006). Federated searching and usability testing: Building the perfect beast. Serials Review, 32(3), 181-182. Description: Summary of usability testing results conducted by Endeavor on their Discovery: Finder product. The article provides an overview of common themes - sorting and personalization issues; but does not describe the methodology used in the testing process in any detail.
40. Reeb, B., D'Ignazio, J., & Law, J. (2006). Federated search observed in the context of student writing: Taking steps towards improving user experience. College & Research Libraries News, 67(6), 352-355. Description: Testing of next generation federated search tools conducted at the University of Rochester. The protocol used contextual inquiry interviews with nine undergraduate students describing the methods that they used to gather research materials for a recently completed assignment. The students basically taught the interviewer how they selected an index, their search strategies, evaluation of results and how they accessed the full text. Very interesting findings related to design, context, and interpretation of bibliographic instruction.
41. Robb, D. (2006). In google's shadow. Computerworld, 40(12), 23-24. Description: Short article describing Google's enterprise search products. The article discusses the use of enterprise search systems for internal documents, intelligence products, and product catalogs; states that Google is strong in information retrieval, but not for more in-depth analysis (much like the Sadeh comparison of Google Scholar and metasearch products). No assessment protocols included.
42. Robinson, M. L., & Wusteman, J. (2007). Putting google scholar to the test: A preliminary study. Program: Electronic Library and Information Systems, Vol.41, no.1, Pp.71-80, 41(1), 71-80. Description: Compared search results from Google Scholar with Google, Ask.com, and Yahoo!. The protocol used complex queries submitted by 7 postgraduate students in a variety of disciplines. The queries were run in the four search engines and the students were asked to rank the results. Results were tested for precision, relative recall, Vaughn's abilitiy to rank results, and Vaughn's ability to retrieve top ranked pages. Significant differences were found in the results for scientific queries when comparing Google Scholar and Ask.com. Otherwise no significant differences in results.
43. Rochkind, J. (2007). (Meta)search like google. Library Journal, 132(3), 28. Description: Contrasts the local indexing technology used by the Google Scholar search engine with the cross searching technologies of metasearching. Then discusses the problems involved with loacl indexing in the current metasearch environment. No assessment protocols used.
44. Sadeh, T. (2006). Google scholar versus metasearch systems. Retrieved 11/6/2007, from http://library.cern.ch/HEPLW/12/papers/1 Description: A comparison of metasearch tools and critique of Google Scholar, focusing on the distinctions between the just-in-time processing used by metasearch engines and the just-in-case processing used by Google Scholar. Reviews the pros and cons of each type of search tool, especially noting the lack of details on the scope, coverage and accuracy of content included in Google Scholar. There is also discussion of ways that libraries are adopting Google Scholar and the future direction of the product, as well as a few other federated search repositories, CrossRef and Vivisimo. No assessment protocols included.
45. Solomon, M. (2004). A confederacy of databases: Federated search flying under many flags. Searcher, 12(7), 24-29. Description: This article provides a description of federated searching including discussion of the four main problem areas: relevance ranking, de-duplication, discipline specific taxonomy models and limited advanced searching capabilities. There is also a discussion of the need for standards, the OpenURL framework and a list of current federated search products with brief descriptions. No assessment protocols.
46. Stearn, D., & Nixon, C. From seamless to limitless access. Way Beyond Cool: Information for the Real World (Proceedings of the InfoToday 2003 Conference), , 309-312. Description: Powerpoint slides outlining types of access and technologies for broadcast and federated searching. No assessment protocols.
47. Tallent, E. (2004). Metasearching in boston college libraries - a case study of user reactions. New Library World, 105(1/2), 69. Description: A study by Boston College Libraries that assessed MetaLib usability via a combination of conservations and observations with 4 undergraduates and 2 graduate students. Interesting observations but there is not much detail in the methodology. Quite a few of the observations have been addressed by MetaLib in subsequent releases.
48. Tang, R. (2007). User perceptions of MetaLib combined search: An investigation of how users make sense of federated searching. Internet Reference Services Quarterly, 12(1), 211-236. Description: This study by the Washington Research Library Consortium (WRLC) was conducted to understand how professional librarians and students users made sense of MetaLib's metasearch functionality and the explore the implications of their perceptions of federated searching for information literacy training and interface design. The WRLC conducted a survey that contained a search simulation designed to probe participants understanding of the search process and outcome, and a specific task for participants to describe their understanding and illustrate by drawing a sketch. The student participants wre library and information science students. These students viewed MetaLib's metasearch primarily as a tool for locating full text while librarians viewed MetaLib's metasearch as a secondary search tool with disappointing performance. This article looks to be quite useful.
49. Taylor, M. (2005). Using the google search appliance for federated searching: A case study. Internet Reference Services Quarterly, 10(3/4), 45-55. Description: A case study of the University of Nevada, Reno Libraries experience testing the Google Search Appliance. This experiment raised technical and financial issues. Article is not very useful.
50. Webster, P. (2007). Challenges for federated searching. Internet Reference Services Quarterly, 12(3/4), 357-368. Description: Good background information. More of a literature review. No assessment in this article.
51. Webster, P. (2004). Metasearching in an academic environment. Online, 28(2), 20-23. Description: Background information, opinion piece. Not very useful.
52. Wilson, V. (2007). A content analysis of google scholar: Coverage varies by discipline and by database. Evidence Based Library and Information Practice, 2(1), 134-136. Description: A review of article #36, a content analysis of Google Scholar.
53. Wisniewski, J. (2007). Build it (and customize and market it) and they will come. Internet Reference Services Quarterly, 12(3), 341-355. Description: Article is about implementation. No assessment portion. Not very useful.
54. Wrubel, L., & Schmidt, K. (2007). Usability testing of a metasearch interface: A case study. College & Research Libraries, 68(4), 292-311. Description: Researchers conducted observation-based formal usability testing on students at four campuses in the University System of Maryland and Affiliated Institutions (USMAI). Eighteen undergraduate students participated in the study. Useful article.
55. Yahe, P. (2005). Libraries online: Competing with search engines. Media & Methods, 41(4), 4-5. Description: An article discussing the advantages of federated searching. No assessment component. Not very useful.
56. York, M. C. (2005). Calling the scholars home: Google scholar as a tool for rediscovering the academic library. Internet Reference Services Quarterly, Vol.10, no.3/4, Pp.117-133, 10(3/4), 117-133. Description: This article discusses how libraries and librarians present Google Scholar to their constituents. There is no usability assessment in this article. Not very useful.
    
    ---------------------------------------------------------------------------
    
    Articles that came to us after our initial literature search:
    
    
57. Breeding, M. (2005). Plotting a new course for metasearch. Computers in Libraries, 25 (2), 27-29. Description: An opinion piece that presents the inherent limitations in the federated search process and the advantages of a centralized search process like the one used by Google. Breeding also emphasizes the need for and importance of metasearching for libraries. Not a research study, no protocol used.