This afternoon, I put aside an hour from yet another hectic day to read Dr. Walter Warnick’s article, “Federated Search as a Transformational Technology Enabling Knowledge Discovery: the Role of WorldWideScience.org.” This article by Dr. Warnick–or Walt to me–presents a wonderful overview of OSTI’s mission dating all the way back to 1947. OSTI (Department of Energy Office of Scientific and Technical Information), originally known as the Technical Information Division, was tasked with collecting and disseminating the wealth of non-classified research from the Manhattan Project.  Having lived in Los Alamos the past 15 years, where development of the atomic bomb took place, I’m very familiar with the history of the Manhattan Project and the reasons behind the creation of OSTI. Nevertheless, I found Walt’s article to be an informative and insightful read that provided a unique insider’s perspective.

Dr. Warnick talks quite a bit about the OSTI corollary, which asserts that accelerating the diffusion of scientific knowledge will accelerate the advancement of science.  In the 12 years that I have known him, it has been Dr. Warnick’s singular goal to do everything in his power to increase the speed of scientific discovery.  I know Walt to be a trail-blazer, highly respected among federal government employees in his dedication and leadership at OSTI.  He has made major strides towards making science more accessible to “science-attentive” citizens, researchers and students.

The article focuses on the major role played by OSTI in championing, supporting and adopting federated search, which is the enabling technology for WorldWideScience.org, Science.gov, DOE Science Accelerator and other sites developed and maintained by OSTI. Deep Web Technologies has benefitted greatly from our 12-year partnership with OSTI, who has supported the development of the Explorit federated search technology, motivated us to keep pushing the boundaries of federated search capabilities and been an eager early adopter of our products.

In my next blog article,  I will be highlighting a few of the many accomplishments achieve through our partnership with OSTI, so please stay tuned.

If you really think about it, these two facts appear to be mutually exclusive. How can the moon sit motionless above me, yet at the same time travel as fast as a speeding bullet??!? Anyone trained in basic physics, science or mathematics has the answer, of course (See An Empirical Explanation of the Speed-Distance Effect). However, it helps to illustrate a fundamental truism that applies in every facet of our lives: Everything, and I do mean everything, is relative. Seemingly contradictory facts, concepts or ideas, can actually coexist or mean the same thing, and is influenced or observed through the lenses of our respective points of view, perspective, context or situation. And likewise, seemingly identical facts, concepts or ideas, can be different, depending on our respective points of view, perspective, context or situation.

We really came to grips with this notion under Einstein’s Theory of Special Relativity, which taught us that time itself could be relative (i.e. time dilation), depending on one’s velocity or proximity to gravitational bodies. Objectively, actual measurements of speed, time or size vary, depending on the speed, distance or proximity to a gravitational body of the observer. And, therefore, one person’s view of the world can be very different from another person’s view of the world, yet both be people can be factually and scientifically correct.

Subjectively, perceived measurements of speed, time and size can also vary, even when they really are the same.

Subjective measurements usually vary when compared to actual measurements, depending on an individual’s point of view, perspective, context or situation. Even assuming a measure of speed, time or size is objectively the same (which, as explained above, isn’t always the case), our individual perceptions are subjective. A retiree may feel they are “going fast,” when behind the wheel of a car, but the teenager behind them thinks they are “going slow.” An interesting article was published several year ago, about the concept of Subjective Time, and how our perceptions of time vary depending on how engaged we are, whether we’re doing something we’re interested in, and other factors. It’s a quick, but thought provoking article.

Subjective time, in the context of user experience on the Internet, is about reducing “boredom points” in user interaction. More fun, less yawn. It’s a powerful concept that is often overlooked, or marginalized, by focusing purely on the speed of a web-based application. In the context of federated search, the latest fad is on discovery services (see Discovering Discovery Services in the Federated Search Blog, which we sponsor). The primary motivator behind discovery services is speed, without an evaluation of individual context and subjective time. As an attorney, I could never rely on a search mechanism that only searched the meta data of articles, as I need to be assured I will find articles containing the specific search terms I desire, not just articles that happen to contain the search term within the title or abstract.

Interestingly, because of my background in federated search (i.e. my context and situation), I understand and appreciate the limitations of discovery services. Most students and professionals do not understand the nuances of true federated search versus a discovery service, and therefore their reliance on either google or a discovery service occurs at their peril unbeknown to them. Inadvertently, the ongoing pursuit to deliver google-like speeds has introduced hidden risks for users.

The incremental results feature of our product (see Scitopia.org for an example), represents a concerted effort to reduce subjective time, while providing access to the original sources for true full-text searching.

This brings me back to my hot tub, contemplating the contrasts of the actual versus the perceived speed of the moon, and how we as individuals make measurements based on our respective points of view, perspective, context or situation. The objective measures are the same, but:

• 30 minutes of watching the moon feels like an instant to me, but is an intolerable and insufferable bore to my children (subjective time differences).
• That driver who cut me off the other day was totally unjustified, yet was speeding home to comfort their dying relative (different perspectives).
• The car the retiree thinks is going fast, is slow by the teenager’s standards (subjective speed differences).
• Librarians may like discovery services because of their perceived speed, but they can’t guarantee a comprehensive search for the professional researcher, such as an attorney (different contexts).

Federated search is renowned by serious researchers as a way to cut through the garden gnomes and other spurious results by searching only select information sources and by targeting the deep web. This means that an electrical engineer who wants to read up on solar cell fabrication can search for “gallium arsenic” on a federated search site like ScienceResearch.com and quickly uncover the highest-quality information because only science-specific sources are searched, instead of the entire spectrum of the Internet. (See? Serious researchers.) Additionally, at a library that subscribes to electronic resources and uses federated search to access those subscriptions, our electrical engineer would have one-stop access to full text articles that could never be located through a popular search engine.

For example, here’s what happens when I search Google for “gallium arsenic.”

First, Google corrects me. I couldn’t possibly know precisely what I’m searching for, so it changes my search term to “gallium arsenide” without my permission. This is annoying to someone who’s just pretending to be a serious researcher, so I can only imagine what our electrical engineer would be thinking if he needed to find that really great article he stumbled across not too long ago with “gallium arsenic” in the title. The next problem is that the top two results are the Wikipedia entry on gallium arsenide, and a sponsored link from a company that sells manufacturing quantities of gallium arsenide. Well, I already know what it is, and I don’t have room in my garage. There are links to scientific journals lower in the results, but they’re scattered among more commercial sites, a sustainable energy wiki, and a page advertising a conference in Oregon. Sensing that a gnome might pop up at any moment, I click “Search instead for gallium arsenic.” (At least Google offers to search for what I wanted after it’s made up my mind for me.) The same sort of results come up – not bad, but not what I need if I’m going to learn about the different substrates that are used in the fabrication of gallium arsenic cells.

Contrast this with what happens when I search ScienceResearch.com for “gallium arsenic” – or “gallium arsenide,” as the case may be. First, the search is run on the search term I asked for. Calling this a bonus of federated search might seem too much like I’m hard-selling you that used car by saying, “Plus, when you turn the key, the engine starts!” but in comparison to Google’s behavior, it’s an important feature. Now for the results: the top results are from National Institute of Standards and Technology, which just might be a more reliable source than Wikipedia, and following that is a result about “Gallium-Arsenic Substrate Fixture and Substrate Fixing Methods.”

Our electrical engineer is pleased. And with the time he saved by using federated search, maybe now he can have a little fun with Google.

Abe Lederman, Deep Web Technologies’ president, will be visiting the D.C. area next week from Monday the 8th through Thursday the 11th.  Along with a few client meetings, he’ll also be demonstrating an Alpha version of our new federated search application. We’re looking forward to getting feedback. (Note:  If you’re in the D.C. area and would like to meet with Abe, please let us know A.S.A.P. – info at deepwebtech.com.)

We’ll also be looking for additional feedback on our new application in the near future through beta testers…interested?

“How long for results? Katherine asked.

“A few minutes to write the spider and launch it. After that, maybe fifteen for the spider to exhaust itself.”

“So fast?” Katherine looked encouraged.

Trish nodded. Traditional search engines often required a full day to crawl across the entire online universe, find new documents, digest their content, and add it to their searchable database. But this was not the kind of search spider Trish would write.

“I’ll write a program called a delegator,” Trish explained. “It’s not entirely Kosher, but it’s fast. Essentially, it’s a program that orders other people’s search engines to do our work. Most databases have a search function built in – libraries, museums, universities, governments. So I write a spider that finds their search engines, inputs your keywords, and asks them to search. This way, we harness the power of thousands of engines, working in unison.”

Katherine looked impressed. “Parallel processing.”

A kind of metasystem. “I’ll call you if I get anything.”

Clearly, if Dan Brown is penning the concept of “metasystems” in his fictional books, and popular search vendors such as Yahoo! are playing with federation in their labs, large-scale, real-time single search has arrived to the masses.

Well, so far Sol hasn’t taken me up on my challenge, so I’m going to take a stab at it.

One of the earliest mentions of federated search that I know of is in a D-Lib article dating back to September 1998 entitled:

Multilingual Federated Searching Across Heterogeneous Collections

Doing some research for this article I just found an earlier mention of federated search in a May 1996 article in IEEE’s Computer Magazine entitled:

Federating Diverse Collections of Scientific Literature

Back in December 1997 I started implementation of my first federated search engine, Distributed Explorer, and for reasons that I don’t recall now I referred to the technology as Distributed Search and not Federated Search. A first presentation on Distributed Explorer that I gave in May 1998 is available here:

ESTN: Environmental Sciences Technology Network

Does anyone know of another early use of the term “federated search”? Please let me know!

As a consequence, I am (from time-to-time) invited to speak and consult for small to medium businesses, and have managed to developed an expertise in web hosting, ISP and SaaS. It is that latter category, that I provide some level of strategic input and guidance for Deep Web Technologies.

Well, this August 10th to the 14th, you can catch me speaking at the Hosting Con 2009 conference in Washington, DC. I will be talking about “planning for an eventual acquisition,” which is applicable to just about any business whose owner(s) envision themselves relaxing on a tropical island in return for their years of hard labor starting and growing a successful business.

I also have the honor of moderating a keynote with two experts in data protection policies in US and EU, one hailing from the US Department of Commerce, the other from the European Commission.

It’s a jammed-pack conference, full of outstanding speakers and sessions, involving 4 tracks: Marketing & Sales, Emerging Trends, Technology & Operations, and Business Development.

Wish me luck, and I hope to identify new partners for Deep Web Technologies!

Before I started my internship at Deep Web, I had no idea what quality assurance (more commonly QA) was. I guessed it was one of those corporate phrases that sound impressive but have little substance. In that vein, perhaps the job entailed actualizing paradigms or some such. Sadly, the truth was nothing so glamorous.

QA turned out to be testing done systematically. My first day on the job, I was sat down in a room full of Deep Web engineers and told to go explore the Scitopia.org testbed and look for things wrong with it. I was later given a few pages of standards every Deep Web Technologies’ implementation should meet. This is where the systematic part comes in: I would use the product and make sure it met every one of those standards. That’s what QA at Deep Web Technologies is all about.

Usually intern work is the work no professional would touch with a ten-and-a-half foot pole. Why is QA relegated to that low level? Well, it’s simple work with few skills required, compared to, say, software engineering, which a degree is normally required for.  All that’s required is basic computer skills and the kind of mindset that delights in making things break (which I have). In fact, as I recall, I didn’t even get much training, that’s how simple and easy QA work is. Of course, it could be that my mind just works that way…

Regardless of how easy the job is, interning at Deep Web Technologies turned out well. Even two months here is sure to look good on my resume. I’ve gotten experience with the tech industry. I’ve laid the foundations for a professional network. I’ve decided whether I’m interested in further pursuing QA positions later in life (I am – maybe even at Deep Web Technologies again).   All things considered, this internship has been a real positive experience for me. Just peachy, in fact.

I’d recommend Deep Web Technologies to anybody looking for a tech internship, as they’ll probably get a similar experience. Well, actually, I’ll recommend it only if they have that QA kind of mind. Otherwise, the only negative part of working at Deep Web Technologies might well be actually working. And that kind of attitude… well, it certainly won’t enable you to meet your synergy benchmarks.