Net Neutrality and Federating Searching

Jake, a librarian in the D.C. area and beer aficionado (he’s the beerbrarian), writes on how the neutrality of federated search solutions is often overlooked and that it is most disconcerting that librarians and users of federated search solutions are not even aware of the bias in federated search results. This bias is the result of the ulterior motives of some federated search / discovery services vendors whose primary business is selling content.

Jake writes:

I am credentialed at an institution that uses EHIS. I searched for dozens of terms, and the results weren’t pleasant for EHIS. It’s a crude test, but EHIS failed it.

EHIS consistently promoted EBSCO resources, favoring Academic Search Premier, an interdisciplinary EBSCO database, over product from other vendors that are more specialized.

This subject is one that I have addressed in several blog posts before including this post last December, If Google might be Doing it …, but I welcome reinforcement of this concern. In the last 6-12 months following the initial craze with Discovery Services I have seen significant more questioning in the library community of Discovery Services such as ProQuest’s Summon and EBSCO’s EDS.

Here at Deep Web Technologies we have put lots of emphasis on our relevance ranking algorithms, assigning a rank to each result returned (we bring back hundreds to several thousand results for some of the broader queries) based on how closely the title, author and snippets match the user’s query and not what the source that returned the result is.

On Sunday, June 26th at the ALA Summer National Conference in New Orleans I’ll be speaking on a panel on The Age of Discovery: Understanding Discovery Services, Federated Search, and Web scale. You can be assured that this is one topic that I will be discussing in my presentation.

In the article I noted that:

A 2006 study by Outsell reported a 31.9 percent failure rate among business users when researching topics using the major search engines. A separate study from Convera shows that professionals in virtually every industry are having trouble finding important work-related information on the major search engines.

The two studies make a strong case for federated search not because the big search engines don’t have value – they do – but because federated search fills in the holes in content coverage. If nearly one-third of my searches didn’t find the business information I needed I’d be a bit antsy. Deep Web Technologies builds search applications that scour the quality content sources in the deep Web. The big public search engines can’t search the deep Web particularly well. And, Deep Web Technologies’ federated search applications don’t have the issues that the big search engines have of producing a high noise to value ratio. This is because, by design, federated search applications only search the quality content sources. Google and the others search a multitude of sources. Basically, anything that has a link to it can be indexed by the search engines. So, the user is left to separate the wheat from the chaff. Another problem with diving into the search engines to look for quality research material is that what the search engine thinks is most relevant to you may not be so. Search engines generally rank content highly if it is popular — i.e. if it has many links to it. Popular is not the same as valuable.

The best strategy for professional researchers is to build search portals of high quality content searches in their areas of specialization. This includes public, in house, and subscription sources. Deep Web Technologies has many years of experience building these kinds of applications and in handling the authentication issues that inevitably arise with accessing subscription content. Plus, Deep Web Technologies applications allow for easy sorting and clustering of search results into subtopics. And, Deep Web Technologies provides an alerts service where a user creates a profile of search terms he or she is interested in and the system performs regularly scheduled searches for the terms. The system sends an email to the researcher when new and relevant documents appear among their content sources.

No, don’t give up on Google, just know that it’s not enough.

In March, 2008 I explored the “incremental results” feature which Deep Web Technologies makes available in all its federated search applications. As a consultant to Deep Web Technologies I may be somewhat biased but I do believe that this feature is a huge differentiator for the company.

What are incremental results?

The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results.

Why is it a big deal to provide incremental results? It’s because we live in the age of Google speed. Users don’t want to wait the 30 seconds it could take a content source to provide its results. The achilles heel of federated search is the fact that we have no control over how quickly sources respond with their results. If a federated search application is searching 30 sources at once and 29 of them return results quickly but one is slow to respond then the traditional approach to displaying search results has users wait until the last source returns its results. This is bad news for the impatient user.

Deep Web Technologies’ approach is to wait just a few seconds, long enough to get a variety of documents from a number of sources. It then relevance ranks those documents and displays those results quickly to users. While users are inspecting those first results, Explorit (Deep Web Technologies’ federated search engine) is gathering results from the other sources to display when the user is ready.

Explorit is polite to users. It doesn’t simply overwrite the first set of search results with a later batch. It instead informs the users that a newer set is available and asks the user if he wants that set. The user can take the offer, turn it down or defer it (waiting until later to refresh the results.)

Incremental results are a nice way to balance the federated search speed issue with the user demand for speed. We think the feature works well. You can judge for yourself at Science.gov, WorldWideScience.org, and Scitopia.org.

In this first “Best of” article, Diagnosing federated search source problems: it’s harder than you think, I introduced the complex nature of isolating and debugging source access problems. The gist of the challenge is that there are a number of potential points of failure and it’s not always obvious what is failing. Without knowing what is failing it’s not possible to correct the problem or to know who to notify to get the problem corrected. Plus, in order to be able to take action on a source access problem one needs to be aware of the problem. This requires a monitoring system that regularly probes all sources and alerts the appropriate persons of a problem.

Many users of federated search don’t realize that maintaining connectors (the software component accesses sources) is a substantial amount of work that requires a substantial investment. Prospective customers of federated search often focus too much on the bells and whistles of a particular implementation but don’t ask enough questions about whether their important sources can be searched and about what happens when a connector stops working.

Deep Web Technologies appreciates the difficulty of connector development and management. The company has a large catalog of connectors it has developed so its dedicated connector team has extensive experience with connector issues and is quick to identify and correct problems within its control. For those problems beyond its control, Deep Web Technologies has a publisher relations staff that can work with content providers to get those corrected. And, to ensure that problems are quickly discovered, Deep Web Technologies has developed custom software that frequently probes every single sources (except, of course, for those behind firewalls). If a source is intermittently down then an alarm is raised and the publisher, if appropriate is notified. If a source is “down hard” then the connector team swings into action and determines who owns the problem. If the problem is one that will result in a significant outage for a particular source then, upon a customer’s request, the source can be taken offline so that it is not searched at all during the outage period.

Connectors are the foundation of federated search. If the content you want isn’t available then nothing else matters. That’s why Deep Web Technologies works diligently to minimize the amount of time that a source isn’t available.

Senior Software Developer: Nov 1^st

Paul graces Deep Web Technologies with a B.S. in Mathematics from The University of Vermont, a M.S. in Computer Science from John Hopkin’s University in Baltimore, and 17 years of software development experience. The team he works with is constantly predicting needs and trends in the marketplace to actively contribute to the design and architecture of new solutions, so you can see his talents on the back-end of our new products. While writing software that creates business value, Paul has the experience and technical know-how to engineer high-quality solutions, and collaborate with his team to translate our business requirements into a robust and scalable technology. In short— Paul values teamwork over heroics, collaboration over technical silos and pragmatism over technology dogma. “I am a Sr. Software Engineer and own my products because ‘the business’ is my business too.”

Like most remote access jobs Paul works from home. This allows him to enjoy more time with his three children (one that’s five, and a set of boy/girl twins that are two and a half), which I’m sure is a full-time job in it of itself. The picture above illustrates Paul’s dedication to the joy of his children at any cost; can you imagine what pulling 130 pounds on a bike feels like!? As a software development veteran that has worked in both the public and private sector, Paul enjoys the mechanics of the private industry much more because of the opportunity for swift  innovation, flexible policy changes, and the risks of failure that drives companies to succeed.

Welcome Paul! We hope your time here at Deep Web Tech allows you to achieve the personal goals that you have set forth.

We strive, every day, to challenge the talented minds here at Deep Web Technologies by giving them the flexibility to be comfortable, driven, and creative.

“Hire giants, so that you may become a company of giants.”
-Ogilvy
We hire giants.

Deep Web Technologies has a soft spot for fuzzy creatures.  This is the newest member of our team, Arthur the mighty black bear. We received the letter, below, last week commemorating Arthur’s release into the wild. Of course, this was after plumping himself on porridge in preparation for the winter. We have not found any real work for Arthur to do around our office; he seems to have a blast going through our trash as well as the fish tank. He does, however, keep one intern, who has dreams to join the circus, very busy while they work on their act.

Dear Abe,

On behalf of the board, staff and animals of The Wildlife Center, thank you for your recent donation of $250 that we received on December 28, 2010. As we understand it, this gift matches your employees’ donations of $265. This gift will be applied to the rehabilitation of a black bear. Enclosed is a sponsorship certificate for you to hang in your office.

Over the course of twenty-five years we have probably cared for 25,000 wild animals and returned 15,000 back to the wild. We have introduced thousands of children and adults to wildlife through our education and outreach programs, taught a hundred interns about wildlife care and natural history, worked for the preservation of New Mexico habitats, informed wildlife management decisions in the state and improved practice in wildlife rehabilitation through experimentation, research and collaboration.

Your support has made this progress possible. We have more to do, especially in terms of education and wildlife advocacy. We expect to be on the job for another twenty five years, keeping wildlife part of the New Mexico experience.

Come by and talk to us about our plans. Share with us your ideas for being more effective in preserving wildlife and their habitats. And most important, keep your interest in wildlife alive.

Sincerely,

Katherine Eagleson

Executive Director

If anyone has questions on how to get into contact with the Wildlife Center, or would like to add on to our contribution, please let us know. Every little bit helps. Arthur can certainly attest to that.

We wanted to let all of our readers know that we will be attending the ALA 2011 Midwinter Meeting, January 7-11. The Midwinter Meeting is in sunny San Diego, CA at the San Diego Convention Center. Our President Abe Lederman said, “We will be meeting with some of our current and potential partners to have an opportunity to  integrate  our organizational mobility, the ALA Midwinter convention is a great way see new and old faces as well as project us into the new year.”

The line-up of benefits include: 200+ discussion groups covering a variety of hot topics, 2000+ committee meetings and events, entrance to the exhibits, including the opening reception, the ALA President’s Program, ALA/ERT/Booklist Author Forum, Arthur Curley Memorial Lecture**, and the Sunrise Speaker Series. Needless to say, we will be quite busy for those days. The partners that we plan to meet with include Basch, Sweats, and two other recent partnerships we will be announcing after the New Year.

If anyone would like a face-to-face meeting, please contact us, we would love to arrange an engagement.

See you there!

**Arthur Curley served as president of the American Library Association in 1994-1995 and was director of the Boston Public Library. He was a champion of the arts and of the library’s role as a center that can transform the community.  The lecture series commemorates his lifelong dedication to the principles of intellectual freedom and free public access to information. Mr. Curley was an ALA member for more than 30 years.

Deep Web Technologies just made a few more connections! We would like to welcome Exiquio (E zee ke o) Cooper-Anderson, Connector Developer, one of the newest members of our connector team since October 25th!

Our connector development team is at the core of our business model. The team builds, maintains, monitors, and fixes the connections between our application and the multiple database sources.

Exiquio’s home base, while not actually born there, is in Georgia, where he started his professional career. His education is highly sought after and non-traditional; Exiquio has been largely self-taught and mentored in multiple programming languages, making him a perfect fit here at Deep Web Technologies. He is training on our specialized C2 connector development software, which is an experience that he can’t find anywhere else in the world. Our VP of Professional Services, Brian DeSpain wrote a very compelling blog outlining our approach to connector development.

Exiquio feels that his time here at Deep Web Tech will allow him to establish a solid foundation where he can raise his 3-month-old daughter Aliyah (pictured above).  Also, being surrounded by the professionals here is enriching his programming skills– taking them to the next level.

When Exiquio is not building connectors to the world’s prime information sources, he moonlights as a professor. His subject? Distribution Of Pain– Exiquio is an avid gamer. He is specialized in turn-based strategy games (think: Civilization), RPGs (Final Fantasy), FPS (Call of Duty), and simulation racing (Gran Turismo 5) . That roughly translates into many opponents shaking at the very thought of being matched up one-on-one.

Fun Fact: Exiquio was inspired to live here in Santa Fe by the multiple western movies set in and around our local area.

Welcome Exiquio!  May your experience here allow you to reach your dreams.

We strive, everyday, to challenge the talented minds here at Deep Web Technologies by giving them the flexibility to be comfortable, driven, and creative.

“Hire giants, so that you may become a company of giants.”

-Ogilvy

We hire giants.

**(12/2/2010) Updated section “Full-text or metadata only”

This blog article is written by Abe Lederman, President and CTO of Deep Web Technologies:

A couple of weeks ago I had the pleasure of attending the Charleston Conference. It was my first time attending this conference and also my first time in Charleston, South Carolina, a city deservedly named last week the friendliest city in America by Travel & Leisure magazine.

On Thursday afternoon, Nov 4, 2010, I attended the 45 minute face-off between Serials Solutions’ Summon and EBSCO’s EDS (aka “EBSCO Discovery Service”) gladiators (see Carl Grant’s pre-face-off blog post).  This was a well attended session attracting several hundred other conference attendees.

The face-off was disappointing and did not live up to the hype that had been generated for the face-off and for Discovery Services over the last year. The format of the face-off, much like a presidential debate, was negotiated between the conference organizers, Serial Solutions and EBSCO, and left much to be desired. Representatives of Summon and EDS were given the questions in advance and, in the case of EDS, the prepared answers were read out loud to the audience. There was no opportunity for the audience to ask questions or to engage the Summon and EDS representatives.

Don Hawkins of Information Today has summarized the Moderator questions and the Summon and EDS responses in this blog post.

In the brief demonstration part of the face-off both Summon and EDS were each given 2 searches to conduct as a way to show off their respective products. As opposed to the Q&A portion of the face-off, these searches were not given to Summon and EDS in advance as evidenced by the fumbling of both sides. From my perspective, these demonstrations highlighted the fact that Discovery Services are not really any better when compared to Deep Web Technologies’ Explorit in terms of the quality of results that they return to the user.

There are a number of issues with Discovery Services, particularly with Summon and EDS, that are not being addressed in the press, although I am now starting to see them discussed by some of the librarians that I met in Charleston and elsewhere. In this blog article I will focus on 4 issues:

One size fits all  — Discovery Services provide uniform access to their index. Whether you are a student doing research for an astronomy paper, a psychology paper, or a history paper, the index being searched is the same. Discovery Services do not have the ability to allow the student to search a selected subset (chosen by a librarian or by the professor teaching the course) of the available content through the particular Discovery Service.

Not everything is going to be in the index – Discovery Services index content that is of general enough interest such that it makes business sense for these products to index. Discovery Services also need to establish business relationships with the owners of content in order to index it and may also require the owner of the content to expend effort in making their content available to the Discovery Service. This means that not all content is going to be in their indices, particularly niche (long-tail) content.

So, if a library desires to provide their patrons with comprehensive, one-stop access to all the content that they subscribe to, the use of Federated Search is still required. EBSCO acknowledges this while Serials Solutions does not.

Full-text or metadata only — As I point out later in this article, Discovery Services does not tell us when they are indexing the full-text of articles and when they are only indexing metadata. If the full-text of an article is not indexed by the Discovery’s service then a user’s search won’t find results that only mention terms of interest in the body of the article, regardless if the article is highly relevant! Federated search, on the other hand, searches the index created by the content holder, the full-text, and in many cases returns some gems that may otherwise remain undiscovered.

Vendor neutral — Both Summon and EDS are Discovery Services provided by multi-billion dollar publisher/aggregator companies whose main business is making sure that libraries purchase their content. Shouldn’t librarians worry whether results returned by Summon might be biased towards higher ranking of ProQuest results? Or even worse, shouldn’t librarians worry whether content from a competitor to the Discovery Service they are considering subscribing to is going to be missing. Carl Grant’s blog post addresses this issue in more detail.

I want to use this post to challenge all providers of Discovery Services, not just Summon and EDS, to become completely transparent by clearly listing on their web sites the content that has been indexed for use by your Discovery Service. For each publisher’s content that you index please indicate what journals / databases you are indexing, the period of coverage, whether you have indexed only an item’s meta-data or also its full-text, and how much your index typically lags behind current content available from each publisher. It would also be great if someone would create a table of the top 100 or 200 publishers and list the above information cross-indexing each publisher and each Discovery Service.

It might be eye opening to discover (pun intended) what major publishers’ content are missing from the various Discovery Services. I have been told, but have not had a chance to confirm, that EBSCO content is not in Summon and ProQuest content is not in EDS.  Current lack of transparency makes this extremely difficult.

Only when all of this information becomes readily available can librarians make informed decisions whether to purchase a Discovery Service or purchase a Federated Search solution such as Deep Web Technologies’ next generation Explorit product.

Let’s hope that the next face-off is more conducive to a fair and open evaluation of research solutions and includes not only Discovery Services solutions but also leading Federated Search solutions, as well.

-Abe Lederman

President and CTO

Deep Web Technologies