The National Federation of Advanced Information Services (NFAIS™) is releasing a draft Discovery Service Code of Practice for review and comment by March 16, 2012. NFAIS believes that discovery services have the potential to provide ease of information discovery, access, and use, benefiting not only its member organizations, but also the global community of information seekers. However, the relative newness of these services has generated questions and concerns among information providers and librarians as to how these services meet expectations with regard to issues related to traditional search and retrieval services; e.g. usage reports, ranking algorithms, content coverage, updates, product identification, etc. Accordingly, the NFAIS Code Development Task Force has developed this draft document to assist those who choose to use this new distribution channel through the provision of guidelines that will help avoid the disruption of the delicate balance of interests involved.

I recently got an email raising concerns about the draft paper and why services such as my company’s Explorit Federated Search were being excluded by NFAIS.  I don’t want to go further into the specific concerns of the email but I do want to comment on some things related to Discovery Services.

First, I want to say that, despite all the marketing, the concept of a Discovery Service is not new. Two or three years ago OCLC coined the term Web Scale Discovery to refer to their WorldCat index. Soon thereafter ProQuest started referring to Summon as a Discovery Service and put a lot of marketing muscle behind that. As a consequence of that the term Discovery Service has now become synonymous with a large centralized index even though federated search application such as my company’s Explorit can certainly be considered to be a Discovery Service, and in fact I have been talking about Explorit being a Discovery tool much longer than OCLC, Summon and EBSCO have adopted the Discovery Service label for just what they do.

Large centralized indexes, mostly of metadata have been around for decades now. Consider Web of Science, Scopus, Infotrieve, Ingenta and many others. So OCLC and ProQuest didn’t invent something new, they just developed an improved version of something that’s been around for a long time, stuck a new label on it, and through substantial marketing efforts have gotten the term – Discovery Service – to have become associated exclusively with large centralized indexes.

On the very positive side, for what we do and the issues I have raised about Discovery Services in the past, this NFAIS effort at developing a Code of Practice for Discovery Services (i.e. WorldCat, Summon, EDS, and Primo) raises a fairly substantial set of issues with these Services that Deep Web Technologies can address. I can see this paper as useful in highlighting issues with these Discovery Services. Three big issues that come to mind (I can think of others) are:

1. Muddiness about who owns content and inadvertent access to content (including meta data). “Lack of user authentication/verfiication and/or the inability to reliably identify an institution’s holdings” is cited by the NFAIS paper as a concern.
2. The ranking algorithm. Whose content is shown first in Discovery Service search results? There are issues with conflicts of interest among Discovery Service providers that makes me uneasy.
3. Coverage. What sources are available to users of Discovery Services? Who gets to decide that? What about access to the very specialized sources that Discovery Services are less likely to have access to?

Take a look at the NFAIS  draft Code of Practice and raise your concerns.

Net Neutrality and Federating Searching

Jake, a librarian in the D.C. area and beer aficionado (he’s the beerbrarian), writes on how the neutrality of federated search solutions is often overlooked and that it is most disconcerting that librarians and users of federated search solutions are not even aware of the bias in federated search results. This bias is the result of the ulterior motives of some federated search / discovery services vendors whose primary business is selling content.

Jake writes:

I am credentialed at an institution that uses EHIS. I searched for dozens of terms, and the results weren’t pleasant for EHIS. It’s a crude test, but EHIS failed it.

EHIS consistently promoted EBSCO resources, favoring Academic Search Premier, an interdisciplinary EBSCO database, over product from other vendors that are more specialized.

This subject is one that I have addressed in several blog posts before including this post last December, If Google might be Doing it …, but I welcome reinforcement of this concern. In the last 6-12 months following the initial craze with Discovery Services I have seen significant more questioning in the library community of Discovery Services such as ProQuest’s Summon and EBSCO’s EDS.

Here at Deep Web Technologies we have put lots of emphasis on our relevance ranking algorithms, assigning a rank to each result returned (we bring back hundreds to several thousand results for some of the broader queries) based on how closely the title, author and snippets match the user’s query and not what the source that returned the result is.

On Sunday, June 26th at the ALA Summer National Conference in New Orleans I’ll be speaking on a panel on The Age of Discovery: Understanding Discovery Services, Federated Search, and Web scale. You can be assured that this is one topic that I will be discussing in my presentation.

In the article I noted that:

A 2006 study by Outsell reported a 31.9 percent failure rate among business users when researching topics using the major search engines. A separate study from Convera shows that professionals in virtually every industry are having trouble finding important work-related information on the major search engines.

The two studies make a strong case for federated search not because the big search engines don’t have value – they do – but because federated search fills in the holes in content coverage. If nearly one-third of my searches didn’t find the business information I needed I’d be a bit antsy. Deep Web Technologies builds search applications that scour the quality content sources in the deep Web. The big public search engines can’t search the deep Web particularly well. And, Deep Web Technologies’ federated search applications don’t have the issues that the big search engines have of producing a high noise to value ratio. This is because, by design, federated search applications only search the quality content sources. Google and the others search a multitude of sources. Basically, anything that has a link to it can be indexed by the search engines. So, the user is left to separate the wheat from the chaff. Another problem with diving into the search engines to look for quality research material is that what the search engine thinks is most relevant to you may not be so. Search engines generally rank content highly if it is popular — i.e. if it has many links to it. Popular is not the same as valuable.

The best strategy for professional researchers is to build search portals of high quality content searches in their areas of specialization. This includes public, in house, and subscription sources. Deep Web Technologies has many years of experience building these kinds of applications and in handling the authentication issues that inevitably arise with accessing subscription content. Plus, Deep Web Technologies applications allow for easy sorting and clustering of search results into subtopics. And, Deep Web Technologies provides an alerts service where a user creates a profile of search terms he or she is interested in and the system performs regularly scheduled searches for the terms. The system sends an email to the researcher when new and relevant documents appear among their content sources.

No, don’t give up on Google, just know that it’s not enough.

In March, 2008 I explored the “incremental results” feature which Deep Web Technologies makes available in all its federated search applications. As a consultant to Deep Web Technologies I may be somewhat biased but I do believe that this feature is a huge differentiator for the company.

What are incremental results?

The idea is simple: display results in chunks as they are received from the sources being searched. Science.gov, WorldWideScience.org, and Scitopia.org are three applications that display incremental results.

Why is it a big deal to provide incremental results? It’s because we live in the age of Google speed. Users don’t want to wait the 30 seconds it could take a content source to provide its results. The achilles heel of federated search is the fact that we have no control over how quickly sources respond with their results. If a federated search application is searching 30 sources at once and 29 of them return results quickly but one is slow to respond then the traditional approach to displaying search results has users wait until the last source returns its results. This is bad news for the impatient user.

Deep Web Technologies’ approach is to wait just a few seconds, long enough to get a variety of documents from a number of sources. It then relevance ranks those documents and displays those results quickly to users. While users are inspecting those first results, Explorit (Deep Web Technologies’ federated search engine) is gathering results from the other sources to display when the user is ready.

Explorit is polite to users. It doesn’t simply overwrite the first set of search results with a later batch. It instead informs the users that a newer set is available and asks the user if he wants that set. The user can take the offer, turn it down or defer it (waiting until later to refresh the results.)

Incremental results are a nice way to balance the federated search speed issue with the user demand for speed. We think the feature works well. You can judge for yourself at Science.gov, WorldWideScience.org, and Scitopia.org.

In this first “Best of” article, Diagnosing federated search source problems: it’s harder than you think, I introduced the complex nature of isolating and debugging source access problems. The gist of the challenge is that there are a number of potential points of failure and it’s not always obvious what is failing. Without knowing what is failing it’s not possible to correct the problem or to know who to notify to get the problem corrected. Plus, in order to be able to take action on a source access problem one needs to be aware of the problem. This requires a monitoring system that regularly probes all sources and alerts the appropriate persons of a problem.

Many users of federated search don’t realize that maintaining connectors (the software component accesses sources) is a substantial amount of work that requires a substantial investment. Prospective customers of federated search often focus too much on the bells and whistles of a particular implementation but don’t ask enough questions about whether their important sources can be searched and about what happens when a connector stops working.

Deep Web Technologies appreciates the difficulty of connector development and management. The company has a large catalog of connectors it has developed so its dedicated connector team has extensive experience with connector issues and is quick to identify and correct problems within its control. For those problems beyond its control, Deep Web Technologies has a publisher relations staff that can work with content providers to get those corrected. And, to ensure that problems are quickly discovered, Deep Web Technologies has developed custom software that frequently probes every single sources (except, of course, for those behind firewalls). If a source is intermittently down then an alarm is raised and the publisher, if appropriate is notified. If a source is “down hard” then the connector team swings into action and determines who owns the problem. If the problem is one that will result in a significant outage for a particular source then, upon a customer’s request, the source can be taken offline so that it is not searched at all during the outage period.

Connectors are the foundation of federated search. If the content you want isn’t available then nothing else matters. That’s why Deep Web Technologies works diligently to minimize the amount of time that a source isn’t available.

Senior Software Developer: Nov 1^st

Paul graces Deep Web Technologies with a B.S. in Mathematics from The University of Vermont, a M.S. in Computer Science from John Hopkin’s University in Baltimore, and 17 years of software development experience. The team he works with is constantly predicting needs and trends in the marketplace to actively contribute to the design and architecture of new solutions, so you can see his talents on the back-end of our new products. While writing software that creates business value, Paul has the experience and technical know-how to engineer high-quality solutions, and collaborate with his team to translate our business requirements into a robust and scalable technology. In short— Paul values teamwork over heroics, collaboration over technical silos and pragmatism over technology dogma. “I am a Sr. Software Engineer and own my products because ‘the business’ is my business too.”

Like most remote access jobs Paul works from home. This allows him to enjoy more time with his three children (one that’s five, and a set of boy/girl twins that are two and a half), which I’m sure is a full-time job in it of itself. The picture above illustrates Paul’s dedication to the joy of his children at any cost; can you imagine what pulling 130 pounds on a bike feels like!? As a software development veteran that has worked in both the public and private sector, Paul enjoys the mechanics of the private industry much more because of the opportunity for swift  innovation, flexible policy changes, and the risks of failure that drives companies to succeed.

Welcome Paul! We hope your time here at Deep Web Tech allows you to achieve the personal goals that you have set forth.

We strive, every day, to challenge the talented minds here at Deep Web Technologies by giving them the flexibility to be comfortable, driven, and creative.

“Hire giants, so that you may become a company of giants.”
-Ogilvy
We hire giants.

Deep Web Technologies has a soft spot for fuzzy creatures.  This is the newest member of our team, Arthur the mighty black bear. We received the letter, below, last week commemorating Arthur’s release into the wild. Of course, this was after plumping himself on porridge in preparation for the winter. We have not found any real work for Arthur to do around our office; he seems to have a blast going through our trash as well as the fish tank. He does, however, keep one intern, who has dreams to join the circus, very busy while they work on their act.

Dear Abe,

On behalf of the board, staff and animals of The Wildlife Center, thank you for your recent donation of $250 that we received on December 28, 2010. As we understand it, this gift matches your employees’ donations of $265. This gift will be applied to the rehabilitation of a black bear. Enclosed is a sponsorship certificate for you to hang in your office.

Over the course of twenty-five years we have probably cared for 25,000 wild animals and returned 15,000 back to the wild. We have introduced thousands of children and adults to wildlife through our education and outreach programs, taught a hundred interns about wildlife care and natural history, worked for the preservation of New Mexico habitats, informed wildlife management decisions in the state and improved practice in wildlife rehabilitation through experimentation, research and collaboration.

Your support has made this progress possible. We have more to do, especially in terms of education and wildlife advocacy. We expect to be on the job for another twenty five years, keeping wildlife part of the New Mexico experience.

Come by and talk to us about our plans. Share with us your ideas for being more effective in preserving wildlife and their habitats. And most important, keep your interest in wildlife alive.

Sincerely,

Katherine Eagleson

Executive Director

If anyone has questions on how to get into contact with the Wildlife Center, or would like to add on to our contribution, please let us know. Every little bit helps. Arthur can certainly attest to that.

We wanted to let all of our readers know that we will be attending the ALA 2011 Midwinter Meeting, January 7-11. The Midwinter Meeting is in sunny San Diego, CA at the San Diego Convention Center. Our President Abe Lederman said, “We will be meeting with some of our current and potential partners to have an opportunity to  integrate  our organizational mobility, the ALA Midwinter convention is a great way see new and old faces as well as project us into the new year.”

The line-up of benefits include: 200+ discussion groups covering a variety of hot topics, 2000+ committee meetings and events, entrance to the exhibits, including the opening reception, the ALA President’s Program, ALA/ERT/Booklist Author Forum, Arthur Curley Memorial Lecture**, and the Sunrise Speaker Series. Needless to say, we will be quite busy for those days. The partners that we plan to meet with include Basch, Sweats, and two other recent partnerships we will be announcing after the New Year.

If anyone would like a face-to-face meeting, please contact us, we would love to arrange an engagement.

See you there!

**Arthur Curley served as president of the American Library Association in 1994-1995 and was director of the Boston Public Library. He was a champion of the arts and of the library’s role as a center that can transform the community.  The lecture series commemorates his lifelong dedication to the principles of intellectual freedom and free public access to information. Mr. Curley was an ALA member for more than 30 years.

Deep Web Technologies just made a few more connections! We would like to welcome Exiquio (E zee ke o) Cooper-Anderson, Connector Developer, one of the newest members of our connector team since October 25th!

Our connector development team is at the core of our business model. The team builds, maintains, monitors, and fixes the connections between our application and the multiple database sources.

Exiquio’s home base, while not actually born there, is in Georgia, where he started his professional career. His education is highly sought after and non-traditional; Exiquio has been largely self-taught and mentored in multiple programming languages, making him a perfect fit here at Deep Web Technologies. He is training on our specialized C2 connector development software, which is an experience that he can’t find anywhere else in the world. Our VP of Professional Services, Brian DeSpain wrote a very compelling blog outlining our approach to connector development.

Exiquio feels that his time here at Deep Web Tech will allow him to establish a solid foundation where he can raise his 3-month-old daughter Aliyah (pictured above).  Also, being surrounded by the professionals here is enriching his programming skills– taking them to the next level.

When Exiquio is not building connectors to the world’s prime information sources, he moonlights as a professor. His subject? Distribution Of Pain– Exiquio is an avid gamer. He is specialized in turn-based strategy games (think: Civilization), RPGs (Final Fantasy), FPS (Call of Duty), and simulation racing (Gran Turismo 5) . That roughly translates into many opponents shaking at the very thought of being matched up one-on-one.

Fun Fact: Exiquio was inspired to live here in Santa Fe by the multiple western movies set in and around our local area.

Welcome Exiquio!  May your experience here allow you to reach your dreams.

We strive, everyday, to challenge the talented minds here at Deep Web Technologies by giving them the flexibility to be comfortable, driven, and creative.

“Hire giants, so that you may become a company of giants.”

-Ogilvy

We hire giants.