Sol Lederman wrote:

I recently discovered an article, 5 Reasons Not to Use Google First, that sings my song. The article addresses this question:

Google is fast, clean and returns more results than any other search engine, but does it really find the information students need for quality academic research? The answer is often ‘no’. “While simply typing words into Google will work for many tasks, academic research demands more.” (Searching for and finding new information – tools, strategies and techniques)

The next paragraph gave me a chuckle.

As far back as 2004, James Morris, Dean of the School of Computer Science at Carnegie Mellon University, coined the term “infobesity,” to describe “the outcome of Google-izing research: a junk-information diet, consisting of overwhelming amounts of low-quality material that is hard to digest and leads to research papers of equally low quality.” (Is Google enough? Comparison of an internet search engine with academic library resources.)

The article continues with its list of five good reasons to not use Google first.

Note that the recommendation isn’t to skip Google altogether. There’s a balance that’s needed to get the best value when performing research. The findings in the “Is Google enough?” article summarizes this point really well:

Google is superior for coverage and accessibility. Library systems are superior for quality of results. Precision is similar for both systems. Good coverage requires use of both, as both have many unique items. Improving the skills of the searcher is likely to give better results from the library systems, but not from Google.

Filtering results is as much an art as a science.

Refining or clarifying an initial set of search results is a fairly common practice among search engine users that don’t know exactly what they are looking for. Yet in federated searching, refining a group of results is often not the best strategy for a user to find the best results they are looking for. Let’s do a quick walk-through of how a user might use a federated search application. Let’s say a user is interested in nanotechnology and is using our Explorit federated search application. When the user inputs the query, the search brings back roughly 2500 results (this is a rough number, and it can vary depending on the sources). After sorting through some of the results, the user decides that their interest really lies in medical nanotechnology, so they limit the search to just “medical nanotechnology”.

To get the best possible results from a federated search application, it makes sense to re-run the search with those two terms. The initial group of results was run with the term “nanotechnology” so it may contain some results with the term “medical” in it but since the original search strategy was to search for the nanotechnology industry as a whole, the odds of the best results for “medical technology” are slim to none. It’s far better for a user of federated search to type in the new query to get results from the various sources that match the new search strategy.

Now that the user has the information, how do they sort through it? Most search engines have their own relevancy ranking tuned to their own content. This means a two-term search strategy might be handled differently than a broader search and likely more efficiently.

The real goal, of course, to get the maximum number of results that match the users search strategy, and give them the tools for sorting through them. This strategy would also hold for true for discovery services as well, since many discovery services (such as Summon) only return 1,000 results in response to the query. Those 1,000 results are tuned to the initial query and refining such a small result set further, wouldn’t serve the needs of their users. The user would be better off, as we mentioned before, re-initiating the search.

Just as a side note, this 1,000 result limit is pretty common in large-scale indexes. Google for example, will only bring back 1,000 results for any query even though Google indicates that thousands or millions of results are available. Google brings back the 1,000 most relevant to the initial query, in an index that is filled with a huge amount of spam. So it’s a better idea to re-initiate your query with the refined search strategy to get access to more information.  This 1,000 limitation makes less sense in a product like Summon since presumably the index doesn’t have any spam in it.

Now that databases are getting larger and more numerous, Deep Web Technologies’ federated search services have opened up parts of  some industries that have been untouched since their beginning, (think libraries without books).

The digital book has been in the background for some time now. Though the Kindle was the first e-book reader to become widely popular, it was not the first e-book reader. The Sony Reader was launched earlier in 2007 (the year the Kindle was introduced) and Motricity sold e-book reader software for Windows Mobile and Palm devices. Source

Librarians are still largely unfazed by this technology, seated comfortably in the midst of  massive bookshelves, quietly, but audibly turning pages. So it’s no wonder that Cushing Academy received a lot of attention when they announced that they will be spending more money on databases and less on physical books; in fact they got rid of nearly all of them. The library has been completely remodeled. Instead of rows and rows of book shelves, patrons are greeted with bigger lounging areas, computer space, and a brand new coffee bar, making for a very welcoming educational environment.

The databases are linked with Deep Web Technologies’ next-generation federated search engine. E-books are included. The students are trained in the search mechanics, introduced to the databases they will need for class, then left to explore on their own.

Deep Web Tech provides a service that will be changing the way we receive information as well as the environment in which we receive it in.

While this trend of less books, but not book-less, has not picked up much momentum (the short list includes, but not limited to, University of California at Merced and Stanford’s engineering library). It certainly has, in the case of Cushing, led to increased library attendance and use of organizational resources. We believe that this industry trend is the beginning of  a world wide standard, although we may find ourselves missing the little things about the look, smell, feel, and the  physical connection we have with our books.

Something to remember: Cushing is a private high school and, as far as we have searched, the first of its kind to make this drastic shift.

Something to think about: Where do you think the future of the library is?  Or even better, do you think you will tell your children or grandchildren about a time that you had to physically search thousands of books by hand to get the one you needed? Would you say you missed it?

Maybe the only question that remains is: What will happen when the power goes out?

Deep Web Technologies is offering a unique chance to replace  existing WebFeat installations. The catch? You must be willing to save over $3,800 on next-generation federated search for your library!  Now, with Deep Web Technologies next-generation federated search, you can go deeper and faster than you have ever before!

Explorit, Deep Web Technologies’ federated search product, enables libraries to search 50+ academic sources simultaneously– fast, and includes value-added features such as alerts, reporting statistics and a customized interface. We are offering Explorit at $5995 for existing WebFeat customers – a savings of $3800.

With Explorit there aren’t any additional hidden fees. Just one low fixed price for the length of your contract, and free lifetime upgrades to customers. We don’t abandon our customers.

Click here for more information.

ScienceResearch.com received some positive attention from TCCLibraryblog.blogspot.com (TCCL) and Choice: Current Reviews for Academic Libraries (Choice) this week, in the form of an informative review on the features and functionality of ScienceResearch.com and an award, respectively. For those individuals that have not seen for themselves, ScienceResearch.com uses our federated search technology to further their vision to be  “the leading science research portal for students, librarians, researchers and scientists.” ScienceResearch.com is the product of combining hundreds of science-related websites and scholarly journals into one user-friendly resource, with the capability to customize search needs.

As I have mentioned in my previous post, specialized search engines are the future of information retrieval. The idea of one search engine provider for all information is as far-fetched as the idea of one restaurant serving all the world’s food choices e.g Irish, Mediterranean, Cajun, etc. The quality of food, in this example, would have a hard time comparing to another restaurant that specializes in one particular style. This can relate to information retrieval as well. The more  information we have access to, the greater the need for specialized  tools that are focused on filtering and categorizing said information.

I came across another post that recognizes ScienceResearch.com as an excellent information source. ScienceResearch.com was selected by Choice as an Outstanding Academic Website of 2009. October 2010 Choice: Current Reviews for Academic Libraries . Choice is the premier source for reviews of academic books, electronic media, and Internet resources of interest to those in higher education. Choice is also a publishing unit of the Association of College & Research Libraries, a division of the American Library Association.

I am always delighted to read about 3rd party application reviews. These sites manage a very respectable reputation as opinion leaders in their industry.  S0 it is fantastic that they recognize the value our technology has.

Software engineers very often want to give users the maximum degree of control over the software they have built. I like to call them “knobs and twiddles”. This is an admirable impulse and it’s often shared by the sales team (and let’s face it – it’s one of the rare times those two teams agree!). That can lead, though, to feature over-build. Each UI administrative setting represents a feature that a user might touch and use. Statistically, 45% of features on software applications are never used and only 20% are always used. But overloading the application with configuration settings and giving users plenty of knobs to twiddle carries with it the danger that users will be overwhelmed with choices and will lack the information to properly configure the application to it’s optimal settings. Or perhaps users will spend time twiddling the knobs without understanding the nuances of interaction, then get frustrated when knob A doesn’t have the expected effect on feature Y.

Federated search applications can be quite complex (collections, proxy servers, ILS’s etc.) and require orchestrating all of this together with a number of outside providers. This leads to a great number of parameters that can be tweaked to change performance, emphasize results form different vendors, change the layout, change the number results, etc. Clearly there is a benefit in offering users the ability to configure the application to their hearts content with the recognition that most users aren’t going to take full advantage of the configuration features. Software UI complexity is a common problem with any number of applications.  The approach we have taken to address the issue is with preset defaults based on their market and thus their business needs. For example, a search application for a library in an academic setting is different than a corporate research library. We have different templates for different markets and then let users know we have tested and configured their template based on their markets needs. Thus when a partner or customer administrator sets up their federated search, they select the profile that best matches their industry and needs. If they then chose to configure the application further, they will be starting from a point where the application is already configured to what is common in the industry.

Good UI design stems from understanding the ultimate question about your application – What are users trying to achieve by using the application? What business need are they trying to serve? Many software developers fail to understand the business needs for their application. By configuring the most common settings based on the customer’s needs you can speed adaption of your product and please your users by giving them an easy-to-administer application. Give the power users the tools they need but focus on the needs of the average users.

More Reading

• A case study in administrative user interface design using a CMS at Schlumberger.
  A good post on balancing complexity and flexibility in the UI.
• 12 Useful Techniques For Good User Interface Design
• Deep Web Technologies
• Effective UI: The Art of Building Great User Experience in Software
• User Interface Design and Evaluation
• Best User Interface Design Resources
• UI Fundamentals for programmers

However, many search engines return merely relevant results, and don’t expend any effort in attempting to rank the results. This means the list of results are probably relevant, but you have no idea which item in the list is the most relevant or important to your research. In other words, page 20 of your results list could contain the proverbial needle-in-the-haystack, but you’d never find it, unless you click and evaluate each and every article in the results list from page 1 to page 19, before your find what you really wanted. PubMed is an example of such a search engine, and it happens to be very important and popular in the medical community.

In federated search, you get both: Ranked results and relevant (but unranked) results. The reason for this, is because our federated search technology uses the meta data returned from each search engine queried to establish ranking. Check out my previous article, Ranking: The Secret Sauce for Searching the Deep Web, on how we specifically conduct our ranking. The problem is, the meta data returned from each search engine federated may not contain the search terms that were used in the search. When this happens, our federated search technology has a result, but is unable to establish its rank in relation to the other results. A result becomes an “unranked result,” in that it’s relevant but its rank is unknown.

Perhaps it’s best to give an example: Suppose you search for “blue laser semiconductor.” A result that contains only “blue laser semiconductor” in the title will be ranked very high. However, a similar article entitled “high-wavelength electronic media” will be unranked, unless “blue”, “laser”, and/or “semiconductor” appear in other metadata, such as the abstract. We know the article is relevant, because it was referenced by a search engine, but we don’t know how relevant because we’re not able to find the search term(s) in any of the metadata returned by the search engine being federated.

When you perform a search using our federated search technology, any results that are returned by the search engine which does not contain any of the search terms, or doesn’t contain the exact phrase searched for, in the title or abstract, are considered unranked by Explorit.

There is no particular order that unranked results are displayed, except for perhaps in the order that they arrived in, as they all have a ranking score of 0.

In some cases, bad results are returned by collections because they have a low-quality search engine that doesn’t handle multiple Boolean terms, exact-phrase searches, or large-phrase searches, well or at all.

The more complex the query you issue, the fewer number of ranked results you will get. Our recommendation, when using federated search, is to try simpler queries first.

Now I’d like to talk (briefly, at least for me) about another area where federated search represents a key strategic advantage: Intellectual property research and litigation. I wrote a 2-page sponsored article for Computers in Libraries Magazine, entitled “Next-Generation” Federated Search: Critical for Intellectual Property Research. It’s a quick read.

The crux of the article is: Federated search represents perhaps the single-most powerful way to ensure you leave no stone unturned, in researching or litigating intellectual property matters. This, of course, can be generalized for any business or organization where the cost for missing information is high. Aside from intellectual property, one could imagine that scientific study, medical research, national intelligence, journalism and other areas where effective research is key to success, are excellent candidates for utilizing federated search to create a strategic advantage.

Interestingly, federated search is an underused and misunderstood concept in legal and intellectual property circles. As we (and our competitors) better target our offerings to intellectual property attorneys and firms, I expect federated search will quickly become a must-have, versus a nice-to-have. Read the article to find out why.

In 2008, Thomson Reuters sold Dialog to ProQuest Information and Learning, a provider of information to researchers and libraries. As far as I can tell, there is very little press on the impact of the Dialog sale to Westlaw and its subscribers.

However, I’ve begun to notice an interesting development occurring for Deep Web Technologies, in that we have received an inordinate number of inquires and expressions of interest from the legal world over the past few months. Larger law firms and practices (especially those involving intellectual property and legal research), are beginning to take notice of federated search technology, and many in the legal world are beginning to see how federated search can provide a strategic advantage over mere access to Westlaw or similar organizations.

It makes me wonder whether there is a connection to the timing of this increased interest, with the loss of Dialog for Westlaw’s users.

Given that I am an intellectual property attorney, I can pick on intellectual property attorneys. We’re a diverse group — more diverse than appears at first blush. First, there are four main branches of intellectual property: Copyrights, Trademarks, Patents and Trade Secrets. There are specialty areas within each of them, such as trade dress under Trademarks and international under all branches. Let’s look at Patents for a minute. Two patent attorneys care about very different information if one patent attorney specializes in biomedical devices, and the other specializes in electronics. That’s an easy one to understand.

There can be strong differences, however, for patent attorneys in the same field. Consider two patent attorneys specializing in electronics, if one specializes in computer hardware and the other specializes in semiconductors. Interestingly, even two patent attorneys in same field of semiconductors could differ in their research needs if one specializes in patent drafting and the other specializes in litigation.

I could go on.

The point is, each and every attorney has very unique and differing research needs. No one information source has everything that every attorney needs. Not only that, many attorneys in large corporations, law firms or government offices have internal databases that also become a necessary part of their research.

Only federated search technology has the breadth to serve the particular needs of every attorney, because federated search conducts searches against all sources an attorney feels is important to their careers: Internal sources, subscriber-based sources (i.e. Westlaw) and favorite — subject matter specific — sources (i.e. IEEE, Wall Street Journal, etc).

In today’s world, where attorneys are pressured to keep fees low, yet expected to be on top of their game (both legally and in their respective subject matter), simple, affordable and comprehensive access to information is critical to success.

If you’re a Westlaw subscriber, I’d love to hear your thoughts on the loss of the Dialog database.