The annual Charleston Conference hosts a variety of appealing activities but before you do anything, please be sure to dig into your conference bag for your free copy of the Charleston Advisor.  Inside you will find a spectacular review of Deep Web Technologies’ Federated Search (xSearch) at the Stanford Library.  This review neatly summarizes the strengths of the application in four different categories: Content, User Interface/Searchability, Pricing and Contract Options.  We can’t tell you the score here – you’ll need to read the article – but we were very pleased to see our composite score (although not surprised!).

We are also pleased to announce that our Partner, Swets, will present Swetswise Searcher, powered by Explorit, at the Vendor Showcase, Table #26.  Abe will be around and will happily demonstrate the power of next-generation federated search.

We hope to see you there!

I use Google a lot when I want to search for the answer to a simple question, a tasty recipe, or I want a good laugh (it gets me every time). I also find that using a wide variety of search tools is essential if you’re serious about getting the best possible search results. For all of the individuals out there that understand the need for specialized search engines, this next part will be a gem.

WorldWideScience.org, a premier scientific search engine has received some recent attention.   Walter L. Warnick, Ph.D., Director of the Office of Scientific and Technical Information (OSTI) for the U.S. Department of Energy, demonstrated  WorldWideScience.org in front of multiple observers including those from the scientific community. According to openbiomed.info, “WorldWideScience.org is federated full-text, public, database of scientific and technical research information published from at least 70 cooperative countries”.  With only about 4% duplication with general search engines (that means Google), WorldWideScience.org   provides access to millions of deep web documents using a single, multi-language search query. One user wrote:

I decided to assess the biomedical research coverage by trying to find information on Methicillin-resistant Staphylococcus aureus (MRSA) and procedures to isolate patients with this infection.  I searched the keyword phrase “MRSA isolation protocol“: 892 documents were located, and when the result was sorted by date, open access resources such as the Directory of Open Access Journals (DOAJ), UK PubMed Central, and LENUS (Irish Health Repository) present very recent research.   In comparison, the SCIRUS database searched for MRSA isolation protocol finds quantitatively more research, but when sorted by date, all of the recent literature identified is available in Science Direct via subscription-only access or for purchase US $ 31.50.

If you haven’t tried out the WorldWideScience, and you like having a mind-blowing amount of relevant information from all over the world at your finger tips, I suggest you give it a spin; you won’t be sorry you did. We understand the need for collecting information that’s not for answering the simple questions, or for cute cats (he is really cute though, check the laugh link above).

Still need convincing? Here is a simple demonstration and a shout-out to whomever made a pretty good video about a search engine, even with the 80′s cartoon-like sounds and “rad” guitar background music, I find it quite entertaining, there’s a special place in my heart for you.

**(12/2/2010) Updated section “Full-text or metadata only”

This blog article is written by Abe Lederman, President and CTO of Deep Web Technologies:

A couple of weeks ago I had the pleasure of attending the Charleston Conference. It was my first time attending this conference and also my first time in Charleston, South Carolina, a city deservedly named last week the friendliest city in America by Travel & Leisure magazine.

On Thursday afternoon, Nov 4, 2010, I attended the 45 minute face-off between Serials Solutions’ Summon and EBSCO’s EDS (aka “EBSCO Discovery Service”) gladiators (see Carl Grant’s pre-face-off blog post).  This was a well attended session attracting several hundred other conference attendees.

The face-off was disappointing and did not live up to the hype that had been generated for the face-off and for Discovery Services over the last year. The format of the face-off, much like a presidential debate, was negotiated between the conference organizers, Serial Solutions and EBSCO, and left much to be desired. Representatives of Summon and EDS were given the questions in advance and, in the case of EDS, the prepared answers were read out loud to the audience. There was no opportunity for the audience to ask questions or to engage the Summon and EDS representatives.

Don Hawkins of Information Today has summarized the Moderator questions and the Summon and EDS responses in this blog post.

In the brief demonstration part of the face-off both Summon and EDS were each given 2 searches to conduct as a way to show off their respective products. As opposed to the Q&A portion of the face-off, these searches were not given to Summon and EDS in advance as evidenced by the fumbling of both sides. From my perspective, these demonstrations highlighted the fact that Discovery Services are not really any better when compared to Deep Web Technologies’ Explorit in terms of the quality of results that they return to the user.

There are a number of issues with Discovery Services, particularly with Summon and EDS, that are not being addressed in the press, although I am now starting to see them discussed by some of the librarians that I met in Charleston and elsewhere. In this blog article I will focus on 4 issues:

One size fits all  — Discovery Services provide uniform access to their index. Whether you are a student doing research for an astronomy paper, a psychology paper, or a history paper, the index being searched is the same. Discovery Services do not have the ability to allow the student to search a selected subset (chosen by a librarian or by the professor teaching the course) of the available content through the particular Discovery Service.

Not everything is going to be in the index – Discovery Services index content that is of general enough interest such that it makes business sense for these products to index. Discovery Services also need to establish business relationships with the owners of content in order to index it and may also require the owner of the content to expend effort in making their content available to the Discovery Service. This means that not all content is going to be in their indices, particularly niche (long-tail) content.

So, if a library desires to provide their patrons with comprehensive, one-stop access to all the content that they subscribe to, the use of Federated Search is still required. EBSCO acknowledges this while Serials Solutions does not.

Full-text or metadata only — As I point out later in this article, Discovery Services does not tell us when they are indexing the full-text of articles and when they are only indexing metadata. If the full-text of an article is not indexed by the Discovery’s service then a user’s search won’t find results that only mention terms of interest in the body of the article, regardless if the article is highly relevant! Federated search, on the other hand, searches the index created by the content holder, the full-text, and in many cases returns some gems that may otherwise remain undiscovered.

Vendor neutral — Both Summon and EDS are Discovery Services provided by multi-billion dollar publisher/aggregator companies whose main business is making sure that libraries purchase their content. Shouldn’t librarians worry whether results returned by Summon might be biased towards higher ranking of ProQuest results? Or even worse, shouldn’t librarians worry whether content from a competitor to the Discovery Service they are considering subscribing to is going to be missing. Carl Grant’s blog post addresses this issue in more detail.

I want to use this post to challenge all providers of Discovery Services, not just Summon and EDS, to become completely transparent by clearly listing on their web sites the content that has been indexed for use by your Discovery Service. For each publisher’s content that you index please indicate what journals / databases you are indexing, the period of coverage, whether you have indexed only an item’s meta-data or also its full-text, and how much your index typically lags behind current content available from each publisher. It would also be great if someone would create a table of the top 100 or 200 publishers and list the above information cross-indexing each publisher and each Discovery Service.

It might be eye opening to discover (pun intended) what major publishers’ content are missing from the various Discovery Services. I have been told, but have not had a chance to confirm, that EBSCO content is not in Summon and ProQuest content is not in EDS.  Current lack of transparency makes this extremely difficult.

Only when all of this information becomes readily available can librarians make informed decisions whether to purchase a Discovery Service or purchase a Federated Search solution such as Deep Web Technologies’ next generation Explorit product.

Let’s hope that the next face-off is more conducive to a fair and open evaluation of research solutions and includes not only Discovery Services solutions but also leading Federated Search solutions, as well.

-Abe Lederman

President and CTO

Deep Web Technologies

Sol’s article was prescient as just yesterday the New York Times published the article, E.U. Launches Formal Antitrust Investigation of Google, in which the E.U. Commission informed the world:

“… that it was also looking into whether Google may have given its own services ‘preferential placement’ in search results.”

So as Sol, Carl Grant and I have pointed out in recent blog articles, librarians who are evaluating whether to subscribe to a discovery service such as Summon or EDS need to be really concerned about “vendor neutrality.” Might Summon or EDS, today or in the future, favor results from some publishers to increase usage of some sources? If Google is being accused of such bias, might not EBSCO or ProQuest also have a bias?

Beyond the question of whether or not I would even subscribe to a discovery service, assuming that I did, I would much prefer to select such a service from an independent 3rd party vendor whose main business was not selling you content.

ScienceResearch.com received some positive attention from TCCLibraryblog.blogspot.com (TCCL) and Choice: Current Reviews for Academic Libraries (Choice) this week, in the form of an informative review on the features and functionality of ScienceResearch.com and an award, respectively. For those individuals that have not seen for themselves, ScienceResearch.com uses our federated search technology to further their vision to be  “the leading science research portal for students, librarians, researchers and scientists.” ScienceResearch.com is the product of combining hundreds of science-related websites and scholarly journals into one user-friendly resource, with the capability to customize search needs.

As I have mentioned in my previous post, specialized search engines are the future of information retrieval. The idea of one search engine provider for all information is as far-fetched as the idea of one restaurant serving all the world’s food choices e.g Irish, Mediterranean, Cajun, etc. The quality of food, in this example, would have a hard time comparing to another restaurant that specializes in one particular style. This can relate to information retrieval as well. The more  information we have access to, the greater the need for specialized  tools that are focused on filtering and categorizing said information.

I came across another post that recognizes ScienceResearch.com as an excellent information source. ScienceResearch.com was selected by Choice as an Outstanding Academic Website of 2009. October 2010 Choice: Current Reviews for Academic Libraries . Choice is the premier source for reviews of academic books, electronic media, and Internet resources of interest to those in higher education. Choice is also a publishing unit of the Association of College & Research Libraries, a division of the American Library Association.

I am always delighted to read about 3rd party application reviews. These sites manage a very respectable reputation as opinion leaders in their industry.  S0 it is fantastic that they recognize the value our technology has.

Let’s be honest, who doesn’t like top 10 lists? VH1 was practically built on this idea.

Top10tag.com is a website committed to taking the most bizarre, high-quality products and service reviews, links to most amazing websites, innovative and useful tools, interesting and fun stuff, and putting it all together in an easy-to-read top 10 list.

So when top10tag.com compiled a list of of the best specialized search engines, it was no surprise that Google was nowhere to be found.

Top10tag.com named Mednar Medical Search Engine (Mednar)#5 in the Top 10 Best Specialized Search Engines. Mednar uses Deep Web Technologies‘ federated search engine to segment multiple medical databases where Google, Google Health, or many other general search engines do not have access.

The list appears to be comprised of specialized technologies that create a unique user experience which may prove difficult for a single company to maintain. Getting back to my previous example, Google has many specialized searches. However, at this time,  they do not satisfy some of the grasp and depth users require out of their search engine information. Since Mednar and the technology behind it  provides access to a large, high-quality collection of medical research content not available anywhere else, it was a “shoo-in” to say the least.

Google is great for a variety of reasons that we may all agree with: calendar, document storage, VOIP services, data, etc. However, when it comes to outside-the-box search needs such as images without keywords, sounds, and databases in the deep web to name a few, we may also attest that Google is heavily restricted.

In order to stand out from the crowd, Deep Web Technologies is committed to bringing the highest quality technology, and relevant data to the user by the fastest methods possible.

What kind of specialized search engines do you believe will exist in the near future?

In the article, he very briefly discusses federated search and the value to newspapers.  He says, “A near relative of enterprise search is federated search. It is an effort to collect all forms of content –articles, archives, advertising and classifieds — and organize them so they are all displayed in response to a user’s query (i.e. searching for “pizza” yields results that include reviews, ratings advertising for pizza restaurants and even job listings for pizza delivery. Seattle Times.com, Boston.com and StarTribune.com are all using federated search.)”

He goes on to say, “Federated search is especially helpful for newspapers that have multiple websites.”  With all due respect to Mr. Krasilovsk, he is promoting a common misconception about federated search.  Federated search makes it possible to search other search engines, aggregating all the results, de-duplicating them and producing a ranked list from multiple search engines.

This is an important distinction, because by adding true federated search to their websites, online newspapers have a tremendous opportunity to increase the value of the search results to their readers.  It’s no longer about just the information located within the news websites themselves, but about the topics of interest to their readers.  For example, a news story about breast cancer can turn up articles about the topic from the NIH, governmental sources, etc.

This makes for a richer experience for the user, giving them incentives to come back and stay longer, ultimately increasing ad revenues for the online newspaper.

Thank you, Mr. Krasilovsk. for bringing up the concept of federated search.  If you truly harness the power of federated search for newspapers, the opportunities are boundless.

I can’t believe that a whole year has gone by since my first Year in Review post. A lot has happened in 2008. In this post I’d like to highlight a few of the most significant events of the past year.

First of all I would be remiss if I didn’t mention that in early December the Federated Search blog turned a year old. The blog is going strong and has become the most authoritative, comprehensive resource on federated search on the Web. Sol has written a wonderful article detailing his experiences authoring the blog. Sol’s article is being published by FUMSI this month.

One of the more significant events of 2008 was the acquisition in February of WebFeat by ProQuest, parent company of Serials Solutions federated search vendor. I remember arriving at my hotel in Boston on Thursday evening, February 14 (yes, I was traveling on business on Valentine’s Day) and starting up my email and seeing a flurry of messages about the acquisition of WebFeat. I quickly got on the phone with Sol and we brainstormed on the significance of the announcement. A couple of hours later Sol published his assessment in the federated search blog.

Today, the WebFeat and Serials Solutions products are yet to be fully integrated although I hear that’s coming later this year. This event was significant because it decreased the number of low-cost options for federated search to be deployed in lots of libraries to mostly just one vendor. The acquisition of WebFeat by ProQuest also meant that a major federated search vendor (i.e. unaffiliated with a content publisher) was no longer independent. Sol’s post on the acquisition speculated that we might see federated search costs rise following the acquisition. This hasn’t been the case as far as I can tell.

Microsoft recently introduced federated search in Windows Vista and in Windows Server 2008. Before that, Microsoft announced federated search features in Microsoft Search Server 2008. I wrote about this last year. I believe that Microsoft is trying to further position itself as having the federated search of choice (in the enterprise), in direct competition with Google’s Search Appliance. For the industry, I think it’s a good thing that Microsoft aims to make “federated search” a household term. A potential downside, however, is that those not familiar with the technology may view federated search as something that should be much cheaper than it is. Neither Microsoft nor Google’s federated search capabilities are much of a threat in the library.

This year saw renewed interest in the “deep web”. In April Google announced that they were unlocking the “deep web” by filling out forms and harvesting retrieved content into their index. Since the announcement came from Google my inbox was inundated for the next several weeks with copies of the Google press release and/or articles commenting on this Google effort. The reality as explained by Sol’s blog post is that there is little chance of Google being able to systematically and comprehensively add significant portions of the “deep web” to their index. Later in the year Infovell, which soon afterwards became DeepDyve, announced that they were unleashing the “deep web”. For some reason DeepDyve marketing folks claimed that they were the first to make deep web content available to researchers. Neither Google nor DeepDyve are using federated search technology to access the deep web. However, federated search is the best technology available to unlock the treasure troves of content in the deep web, so this renewed interest in the deep web is really positive.

This fall we saw an unprecedented meltdown of the U.S. economy. It is not clear yet what the impact of the economy will be to the federated search industry in 2009. We have seen public libraries being shut down (or attempted to be shutdown). It is unfortunate that federated search is viewed by some as a nice to have, and by others as not even necessary. Carl Grant, in one of my favorite blog posts of 2008, argues against the fallacy that federated search (or metasearch as referred by Carl) is not really needed and that users would be better served by learning to use the native interfaces of the information sources they are searching.

I believe that we in the federated search industry, particularly in this economic climate, must make a stronger case for the value of federated search. We must show that federated search provides a strong ROI to organizations that acquire federated search solutions. We must show that federated search significantly reduces the amount of time that researchers spend looking for information. Not only will researchers spend less time looking for information but they are much less likely to miss finding important results, either because they didn’t search the source containing the critical item of information, or because this critical item was buried in one of the databases being searched.

2008 was a good year for Deep Web Technologies. Software engineering was kept busy making significant enhancements to our technology, including developing a new Ajax-based User Interface, adding a clustering capability to Explorit, and releasing a major upgrade to our Alerts product. Major upgrades were made to a number of the portals that we’ve developed including Science.gov, WorldWideScience.org, Science Accelerator, Scitopia.org and DTIC MultiSearch. In June, Sol and I attended a ceremony in Seoul, Korea where WorldWideScience.org was turned over to ICSTI. Late in the year we launched the first two of our vertical portals, Biznar and Mednar. I was excited to learn while on vacation, just last week, that Mednar was chosen as the best new health search engine by AltSearchEngines.com.

I wish you all a happy, healthy and prosperous 2009.