close
Welcome! Wikis are websites that everyone can build together. It's easy!

Metalib Assessment Bibliography

Breakdown of who is reading what...

1-10: Steve Frye
11-19: Valerie Malzacher
20-28: Ronna Hoeper
29-37: Scott Silet
38-46: Terri Muraski
47-56: Maureen Olle-LaJoie

Directory Containing Full Text of Articles Below (Password Needed)

  1. Aslam, J. A., & Montague, M. (2000). Bayes optimal metasearch: A probabilistic model for combining the results of multiple retrieval systems. Bayes Optimal Metasearch: A Probabilistic Model for Combining the Results of Multiple Retrieval Systems, , 379-381. Description: The authors introduce a a hightly technical critique of their own metasearch system for searching the 'free web'. This article has nothing to contribute to our assessment of Metalib.
  2. Baer, W. (2004). Federated searching: Friend or foe? College & Research Libraries News, 65(9), 518. Description: Opinion piece that points out the positives and negatives of Federated Search.
  3. Balas, J. L. (2006). Does one-stop searching really serve all? Computers in Libraries, 26(9), 42. Description: Opinion piece that barely touches on federated searching - the article is really about the two directions in retail, self-helping superstores and highly personalized niche stores... and what these trends might mean to libraries.
  4. Belliston, C. J., Howland, J.L., Roberts, B.C. (2007). Undergraduate use of federated searching: A survey of preferences and perceptions of value-added functionality. College & Research Libraries, 68(6), 472-486. Description: Undergraduates undertake two hypothetical research assignments using different search method for each - one using federated searching and the other performed with nonfederated searching. They were then asked to complete a questionnaire about their experience. As they searched, they compiled a running bibliography - the resulting citations were then analyzed by librarians and faculty using two different rubrics, one created by librarians and the other created by faculty. Ninety-five undergraduates participated in this study. 70% of participants preferred federated searching. The librarian-created rubric revealed that citation sets gathered by non-federated searches were superior. The faculty-created rubric revealed no significant difference between federated and non-federated search sets.
  5. Blecic, D. D., Fiscella, J. B., & Wiberley, S. E.,Jr. (2007). Measurement of use of electronic resources: Advances in use statistics and innovations in resource functionality. College & Research Libraries, 68(1), 26. Description: Discusses federated searching in the context of session and search statistics per resource. This article contains nothing of use for assessing metalib.
  6. Boock, M., Nicholas, J., & Kristick, L. (2006). Continuing the quest for the quick search holy grail: Oregon state university libraries' federated search implementation. Internet Reference Services Quarterly, 11(4), 139-153. Description: This article describes the implementation of a federated search tool at Oregon State. Towards the end of the article, the author addresses the question of usability through user assessment. Apparently, a focus group was held along with the gathering of informal data from students - however, methodology of either of these methods is not included in the article.
  7. Boss, S. C., & Nelson, M. L. (2005). Federated search tools: The next step in the quest for one-stop-shopping. Reference Librarian, (91), 139-160. Description: The authors provide a good overview of the federated search tools that existed in 2005. There is no mention of usability testing.
  8. Cervone, F. (2005). What we've learned from doing usability testing on OpenURL resolvers and federated search engines. Computers in Libraries, 25(9), 10-14. Description: Excellent article that provides a review of what various usability studies and testing on the Northwestern campus have revealed about how users interact with federated search, native search, etc. Protocols are not described in detail.
  9. Chen Xiaotian. (2006). MetaLib, WebFeat, and google: The strengths and weaknesses of federated search engines compared with google. Online Information Review, 30(4), 413-427. Description: The author compares MetaLib and WebFeat. Chen explains the differences between these two federated search engines and also compares them to Google. She characterizes MetaLib as offering “one-stop window-shopping – users can go to different subjects or categories to view different databases listed there but cannot search all databases with one search”. She compares conducting searches using either MetaLib or WebFeat to searches performed using Google. A good article in that it discusses many aspects of federated search, including interface, search results, speed, and convenience. -- No assessment protocol is given.
  10. Cox, C. (2006). An analysis of the impact of federated search products on library instruction using the ACRL standards. Portal: Libraries & the Academy, 6(3), 253-267. Description: Author evaluates the impact that federated search *may* have on information literacy instruction through the lense of the ACRL information literacy competencies. Usability assessment is not the focus of this article.
  11. Drake, M. A. (2006). Collaboration, competition, and controversy. Information Today, 23(3), 1. Description: Summary of current issues/initiatives in the information field, including the Google Library Project, peer review, born digital materials, collaboration between librarians and publishers, and GPO updates. Interesting, but not directly relevant to Metalib assessment.
  12. Fahey, S. (2007). F*****ED searchers? the debate about federated search engines. Feliciter, 53(2), 62-63. Description: A brief thought piece that rehashes the familiar debate surrounding federated searching: dumbing down the search process, "good enough" search results, etc. Author contends that federated searching will improve the search experience for users. Opinion piece--no assessment protocols.
  13. Fiehn, B. (2004). Federated searching: A viable alternative to web surfing! MultiMedia & Internet@Schools, 11(2), 29. Description: A basic review of federated searching for school media specialists, including a fair analysis of both strengths and weaknesses. Opinion piece--no assessment protocols.
  14. Friend, F. J. (2006). Google scholar: Potentially good for users of academic information. JEP: The Journal of Electronic Publishing, 9(1), [n]. http://hdl.handle.net/2027/spo.3336451.0009.105 Description: In-depth review of the strengths and weaknesses of Google scholar as an academic research tool. Very interesting, but includes no assessment protocol.
  15. Furlan, W. (2006). Library users expect link resolvers to provide full text while librarians expect accurate results. Evidence Based Library and Information Practice, 1(4), 60-63. Description: Describes and online survey and librarian focus groups conducted at two California State University campuses to assess SFX. Results of the study indicated that user and librarian expectations of SFX were slightly higer than their actual experiences.
  16. Hane, P. J. (2003). The truth about federated searching. Information Today, 20(9), 24. Description: Five common misconceptions concerning federated searching are de-mystified by a representative of WebFeat. Interesting point of view from a federated search provider. No assesment protocol.
  17. Hanna, K. A., Minick, M. B., & Eiszner, J. W. (2007). A nontraditional application of MetaSearch: Using ex libris' MetaLib to enhance delivery, of full-text articles in a university campus course management system. Internet Reference Services Quarterly, 12(3), 297-308. Description: Describes how IUPUI incorprated the MetaLib eShelf into the course management system to present online reading lists to students. The project is viewed as very successful.
  18. Haya, G., Nygren, E., & Widmark, W. (2007). Metalib and google scholar: A user study. Online Information Review, 31(3), 365-375. A Swedish study that compared the search experience and search results of undergraduate students using MetaLib and Google Scholar. "A total of 32 undergraduate students searched academic articles for their thesis work. Searches were recorded using Morae software and were analysed along with the number of articles saved and responses to a questionnaire. All searched with both tools. Half of the students received training before searching." Google Scholar exhibited higher usability than Metalib based on the success rate in finding articles. Open ended responses to questions about the tools indicate a higher rate of satisfaction with Google Scholar, although the overall reaction to Google Scholar was only mildly positive. Overall, students were neutral to both tools. Instruction is valuable for both tools. This study was carried out in a usability lab, and relied on sophisticated tools to conduct an analsis of both the search experience and search results. This study could be difficult to replicate without these tools.
  19. Helfer, D. S., & Wakimoto, J. C. (2005). Metasearching: The good, the bad, and the ugly of making it work in your library. Searcher, 13(2), 40. Description: A brief summary of the MetaLib implementation at CSU-Northridge, followed by a summary of MetaLib's strengths and weaknesses. While not a panacea, the author contends that MetaLib holds promise for the future. Opinion piece--no assessment protocol.
  20. Jacsó, P. (2005). Google scholar: The pros and the cons. Online Information Review, 29(2), 208. Description: The author identified pros and cons of Google Scholar (beta version) searching shortly after its launch in Nov. 2004. Pro: serves as a federated search engine function for libraries that don't have one. Con: when results from Google Scholar were compared to results from the native database, GS usually found far fewer results than were actually available. Interesting article, but data is too outdated to be of much use; GS has likely improved in last 3 years. No relevant protocol given.
  21. Jacsó, P. (2004). Thoughts about federated searching. Information Today, 21(9), 17. Description: Compares differences in result list appearance between 3 different federated search engines: MetaLib, MetaFind, and WebFeat. MetaLib's list is viewed as "the most succinct and most uniform result list." Sort options are "on par with MetaFind," but "its de-duplication feature was not activated in the tested release 3.10." Again, an interesting article, but too dated to be of much use (2004).
  22. Lampert, L. D., & Dabbour, K. S. (2007). Librarian perspectives on teaching metasearch and federated search technologies. Internet Reference Services Quarterly, 12(3), 253-278. Description: 2 surveys of librarians were done in 2005: an online survey sent to several listservs (only 33 responses, and over 30% of those did not have a federated search system in their libraries), and a small group discussion exercise at a California librarians' conference (95 librarians divided into 3 groups; only 10 had a federated search system in their libraries). Also did online user survey of students in 2005 at Cal State Northridge. Copy of survey included in appendix; 88 surveys returned. Results are given (survey: Aug '05-Jan '06; federated searching started there in Aug '04). Opinion: We could possibly duplicate their online survey, but our timeframe is much shorter.
  23. Lewis, N. (2002). Talking about a revolution? first impressions of ex Libris’s MetaLib. Ariadne, 32. Retrieved 12/3/2007, 2007, from http://www.ariadne.ac.uk/issue32/metalib/ Description: Discusses the experiences of the Univ. of East Anglia (UK) in their implementation of MetaLib. Written in 2002; last updated in 2004. Talks about the functionality of MetaLib (link to native databases or cross-search them). They encountered configuration issues due to differences in databases' Z39.50 servers, and had to get permission from database vendors to access these servers and get information on how to do that. Challenges: setting up access to various databases, ongoing maintenance of MetaLib resources, and the unreliability of Z39.50 servers. No assessment protocol given.
  24. Lindahl, D. (2007). Metasearch in the users' context. Serials Librarian, 51(3), 215-234. Description: Discusses student usability issues with metasearch, but no information on the usability studies conducted. Advocates user centered design (UCD) when implementing metasearch and its search interface. Four categories of usability issues with metasearch "out of the box" interfaces: specifying queries, selecting databases, identifying and selecting results, and acquiring items. Gives examples on how U of Rochester libraries have addressed these issues. Also advocates automatic database selection based on user queries.
  25. Luther, J. (2003). Trumping Google? Metasearching's Promise. Library Journal, 128(16), 36+. Description: Nothing new here. Discusses the pros and cons of metasearching, differences between Google searching and metasearching, reasons for librarian resistance, etc. Brief chart with descriptions of several metasearch vendors: WebFeat, Muse Global, ExLibris (MetaLib), Auto Graphics, Fretwell-Downing, GIS, Biblio-Mondo, and VTLS. No assessment protocols given. I particularly liked this fragment: "Metasearch is not a tool that allows librarians--or other expert users--to search with greater precision. It's not for us, it's for the average user."
  26. Mestre, L. S. (2007). Do we step together, in the same direction, at the same time? how a consortium approached a federated search implementation. Internet Reference Services Quarterly, 12(1), 111-132. Description: Talks about the implementation of MetaLib at the Five College Libraries consortium in Western Massachusetts. Useful in that they are also in a consortial arrangement; discusses the work they did to configure MetaLib for the five libraries, and the charge of the different committees. Mentions usability testing several times, but gives little information about it, other than on pp. 128-130, where it discusses how a librarian taught MetaLib to a class and had them compare the results to the native interfaces. She also did a followup email survey with that class.
  27. MetaLib working group. Retrieved 2/27/2008, 2008, from http://hul.harvard.edu/ois/projects/metalib/teams.html Description: A listing of the MetaLib committee members and its implementation sub-team members at Harvard. Sounds like they were very organized in their rollout of the system: sub-teams included content, rationalization, metadata, user interface, resources workflow and operation, usability testing/accessibility, and rollout and training. No other useful information included.
  28. Miller, T. (2004). Federated searching: Put it in its place. Library Journal, , 32. Description: An opinion column on whether the online catalog should be the place to start a federated search. Since much of the library's information is no longer in the catalog, he feels the answer is no. He presents it in the following context: "If users want to find a book, they go to Amazon. If they need information, they go to Google." Nothing of use in this article.
  29. Mullen, L. B., & Hartman, K. A. (2006). Google scholar and the library web site: The early response by ARL libraries. College & Research Libraries, 67(2), 106-122.
  30. Needleman, M. H. (2006). An update on the NISO metasearch activity. Serials Review, 32(2), 143-145.
  31. Neuhaus, C., Neuhaus, E., Asher, A., & Wrede, C. (2006). The depth and breadth of google scholar: An empirical study. Portal: Libraries and the Academy, Vol.6, no.2, Pp.127-141, 6(2), 127-141.
  32. Newton, V. W., & Silberger, K. (2007). Simplifying complexity through a single federated search box. Online, 31(4), 19.
  33. Nichols, D., Huntington, P., & Jamali, H. R. (2007). The use, users, and role of abstracts in the digital scholarly environment. The Journal of Academic Librarianship, 33(4), 446-453.
  34. Nygren, E., Haya, G., & Widmark, W. (2006). Students' experience of metalib and google scholar. [Students' Experience of Metalib and Google Scholar] (11/6/2007)
  35. Ochoa, M. (2007). Testing the federated searching waters: A usability study of MetaLib. Journal of Web Librarianship, 1(3), 47-66.
  36. Parry, F. (2006). MetaGoogle! using MetaLib to deliver user-centered information services at loughborough. ALISS Quarterly, Vol.2, no.1, Pp.38-40.
  37. Planning and implementing a federated searching system: An examination of the crucial roles of technical, functional, and usability testing.(2007). Internet Reference Services Quarterly, 12(1/2), 179-194.
  38. Ponsford, B. C. (2007). User expectations in the time of google: Usability testing of federated searching. Internet Reference Services Quarterly, 12(1), 159-178. Description: User testing at Texas A&M as part of Metalib implementation. Interesting results related to use of Tabs in design, preferred brief view for results display, effectiveness of subject based Quick Sets, interaction with browser commands and interaction with SFX. The protocol used volunteers (ug, grad, fac). Participants completed pre and post test surveys to gauge previous experience, then completed a set of typical library tasks (simple and complex topic searches, author & title searches, find specific journal, find specific databse). The sessions were recorded using Camtasia studio and a facilitator encouraged comments and took notes.
  39. Randall, S. (2006). Federated searching and usability testing: Building the perfect beast. Serials Review, 32(3), 181-182. Description: Summary of usability testing results conducted by Endeavor on their Discovery: Finder product. The article provides an overview of common themes - sorting and personalization issues; but does not describe the methodology used in the testing process in any detail.
  40. Reeb, B., D'Ignazio, J., & Law, J. (2006). Federated search observed in the context of student writing: Taking steps towards improving user experience. College & Research Libraries News, 67(6), 352-355. Description: Testing of next generation federated search tools conducted at the University of Rochester. The protocol used contextual inquiry interviews with nine undergraduate students describing the methods that they used to gather research materials for a recently completed assignment. The students basically taught the interviewer how they selected an index, their search strategies, evaluation of results and how they accessed the full text. Very interesting findings related to design, context, and interpretation of bibliographic instruction.
  41. Robb, D. (2006). In google's shadow. Computerworld, 40(12), 23-24. Description: Short article describing Google's enterprise search products. The article discusses the use of enterprise search systems for internal documents, intelligence products, and product catalogs; states that Google is strong in information retrieval, but not for more in-depth analysis (much like the Sadeh comparison of Google Scholar and metasearch products). No assessment protocols included.
  42. Robinson, M. L., & Wusteman, J. (2007). Putting google scholar to the test: A preliminary study. Program: Electronic Library and Information Systems, Vol.41, no.1, Pp.71-80, 41(1), 71-80. Description: Compared search results from Google Scholar with Google, Ask.com, and Yahoo!. The protocol used complex queries submitted by 7 postgraduate students in a variety of disciplines. The queries were run in the four search engines and the students were asked to rank the results. Results were tested for precision, relative recall, Vaughn's abilitiy to rank results, and Vaughn's ability to retrieve top ranked pages. Significant differences were found in the results for scientific queries when comparing Google Scholar and Ask.com. Otherwise no significant differences in results.
  43. Rochkind, J. (2007). (Meta)search like google. Library Journal, 132(3), 28. Description: Contrasts the local indexing technology used by the Google Scholar search engine with the cross searching technologies of metasearching. Then discusses the problems involved with loacl indexing in the current metasearch environment. No assessment protocols used.
  44. Sadeh, T. (2006). Google scholar versus metasearch systems. Retrieved 11/6/2007, from http://library.cern.ch/HEPLW/12/papers/1 Description: A comparison of metasearch tools and critique of Google Scholar, focusing on the distinctions between the just-in-time processing used by metasearch engines and the just-in-case processing used by Google Scholar. Reviews the pros and cons of each type of search tool, especially noting the lack of details on the scope, coverage and accuracy of content included in Google Scholar. There is also discussion of ways that libraries are adopting Google Scholar and the future direction of the product, as well as a few other federated search repositories, CrossRef and Vivisimo. No assessment protocols included.
  45. Solomon, M. (2004). A confederacy of databases: Federated search flying under many flags. Searcher, 12(7), 24-29. Description: This article provides a description of federated searching including discussion of the four main problem areas: relevance ranking, de-duplication, discipline specific taxonomy models and limited advanced searching capabilities. There is also a discussion of the need for standards, the OpenURL framework and a list of current federated search products with brief descriptions. No assessment protocols.
  46. Stearn, D., & Nixon, C. From seamless to limitless access. Way Beyond Cool: Information for the Real World (Proceedings of the InfoToday 2003 Conference), , 309-312. Description: Powerpoint slides outlining types of access and technologies for broadcast and federated searching. No assessment protocols.
  47. Tallent, E. (2004). Metasearching in boston college libraries - a case study of user reactions. New Library World, 105(1/2), 69. Description: A study by Boston College Libraries that assessed MetaLib usability via a combination of conservations and observations with 4 undergraduates and 2 graduate students. Interesting observations but there is not much detail in the methodology. Quite a few of the observations have been addressed by MetaLib in subsequent releases.
  48. Tang, R. (2007). User perceptions of MetaLib combined search: An investigation of how users make sense of federated searching. Internet Reference Services Quarterly, 12(1), 211-236. Description: This study by the Washington Research Library Consortium (WRLC) was conducted to understand how professional librarians and students users made sense of MetaLib's metasearch functionality and the explore the implications of their perceptions of federated searching for information literacy training and interface design. The WRLC conducted a survey that contained a search simulation designed to probe participants understanding of the search process and outcome, and a specific task for participants to describe their understanding and illustrate by drawing a sketch. The student participants wre library and information science students. These students viewed MetaLib's metasearch primarily as a tool for locating full text while librarians viewed MetaLib's metasearch as a secondary search tool with disappointing performance. This article looks to be quite useful.
  49. Taylor, M. (2005). Using the google search appliance for federated searching: A case study. Internet Reference Services Quarterly, 10(3/4), 45-55. Description: A case study of the University of Nevada, Reno Libraries experience testing the Google Search Appliance. This experiment raised technical and financial issues. Article is not very useful.
  50. Webster, P. (2007). Challenges for federated searching. Internet Reference Services Quarterly, 12(3/4), 357-368. Description: Good background information. More of a literature review. No assessment in this article.
  51. Webster, P. (2004). Metasearching in an academic environment. Online, 28(2), 20-23. Description: Background information, opinion piece. Not very useful.
  52. Wilson, V. (2007). A content analysis of google scholar: Coverage varies by discipline and by database. Evidence Based Library and Information Practice, 2(1), 134-136. Description: A review of article #36, a content analysis of Google Scholar.
  53. Wisniewski, J. (2007). Build it (and customize and market it) and they will come. Internet Reference Services Quarterly, 12(3), 341-355. Description: Article is about implementation. No assessment portion. Not very useful.
  54. Wrubel, L., & Schmidt, K. (2007). Usability testing of a metasearch interface: A case study. College & Research Libraries, 68(4), 292-311. Description: Researchers conducted observation-based formal usability testing on students at four campuses in the University System of Maryland and Affiliated Institutions (USMAI). Eighteen undergraduate students participated in the study. Useful article.
  55. Yahe, P. (2005). Libraries online: Competing with search engines. Media & Methods, 41(4), 4-5. Description: An article discussing the advantages of federated searching. No assessment component. Not very useful.
  56. York, M. C. (2005). Calling the scholars home: Google scholar as a tool for rediscovering the academic library. Internet Reference Services Quarterly, Vol.10, no.3/4, Pp.117-133, 10(3/4), 117-133. Description: This article discusses how libraries and librarians present Google Scholar to their constituents. There is no usability assessment in this article. Not very useful.

    ---------------------------------------------------------------------------

    Articles that came to us after our initial literature search:
  57. Breeding, M. (2005). Plotting a new course for metasearch. Computers in Libraries, 25 (2), 27-29. Description: An opinion piece that presents the inherent limitations in the federated search process and the advantages of a centralized search process like the one used by Google. Breeding also emphasizes the need for and importance of metasearching for libraries. Not a research study, no protocol used.



Latest page update: made by tmuraski , Apr 1 2008, 9:56 PM EDT (about this update About This Update tmuraski Edited by tmuraski

50 words added
1 word deleted

view changes
- complete history)
Keyword tags: None
More Info: links to this page

Threads

Anonymous  (Get credit for your thread)
Started By Thread Subject Replies Last Post
mollelajoie MetaLib Statistics 0 Apr 3 2008, 5:02 PM EDT by mollelajoie
mollelajoie
Thread started: Apr 3 2008, 5:02 PM EDT  Watch
When you click on the "Directory Containing Full Text of Articles Below" link on this page, you'll find a new folder called Statistics. That has a sampling of each of the type of statistics we can get out of MetaLib (the data is from October 2007 from UW-River Falls' instance of MetaLib). I also included a Word document that has a brief description of each of the statistics. Enjoy :)
Do you find this valuable?    
Keyword tags: MetaLib statistics
steve.frye Possible metalib assessment protocols that we can use 6 Apr 3 2008, 1:01 PM EDT by tmuraski
steve.frye
Thread started: Mar 13 2008, 10:53 AM EDT  Watch
Please post your protocols before we meet from 12-1:30pm on 3/24
Do you find this valuable?    
Keyword tags: None
Show Last Reply

Attachments

Word Document MetaLib Assessment Literature Review combined.doc (Word Document - 41k)
posted by tmuraski   Apr 3 2008, 1:04 PM EDT
Lit Review/Annotated bibliography for 11 selected articles
Top Contributors