Shifting Search from Static to Real-time

I've been mulling something that keeps tugging at my mind as a Big Idea for some time now, and I may as well Think Out Loud about it and see what comes up.

To summarize, I think Search is about to undergo an important evolution. It remains to be seen if this is punctuated equilibrium or a slow, constant process (it sort of feels like both), but the end result strikes me as extremely important: Very soon, we will be able to ask Search a very basic and extraordinarily important question that I can best summarize as this: What are people saying about (my query) right now?

When it first hit critical mass, it seemed Google answered this question. For the first time, you could ask a question in your native tongue, and get an answer. It felt immediate, but save for the speed with which the search results were rendered, it was not. Instead, it was archival - Google was the ultimate interface for stuff that had already been said - a while ago. When you queried Google, you got the popular wisdom - but only after it was uttered, edited into HTML format, published on the web, and then crawled and stored by Google's technology. True, that has sped up - Google indexes a lot of sites more than once a day now - but as it nears the event horizon, this approach to search won't scale.

In short, Google represents a remarkable achievement: the ability to query the static web. But it remains to be seen if it can shift into a new phase: querying the realtime web.

It's inarguable that the web is shifting into a new time axis. Blogging was the first real indication of this, but blogging, while much faster than the traditional HTML-driven web, is, in the end, still the HTML-driven web. To its credit, Technorati saw blogging as the vanguard of a shift to real time, and tried to become the first search engine for "the live web". It failed to gain critical mass, but I think the main reason was that the web was not yet "alive".

That is changing, rapidly. Yes, I'm thinking about Twitter, of course, which is quickly gaining critical mass as a conversation hub answering the question "what are you doing?" But I'm also thinking about ambient data more broadly, in particular as described by John Markoff's article (posted here). All of us are creating fountains of ambient data, from our phones, our web surfing, our offline purchasing, our interactions with tollbooths, you name it. Combine that ambient data (the imprint we leave on the digital world from our actions) with declarative data (what we proactively say we are doing right now) and you've got a major, delicious, wonderful, massive search problem, er, opportunity.

And with that search challenge comes an equally exciting monetization opportunity.

Imagine AdSense, Live. Here's my scenario:

Let's say you are in the market to buy something - anything. Just to keep things simple, I'll use the age old digital camera example. Say you are considering a Canon EOS. Before the shift to the live web (for most mortals, that'd be now), chances are good you'd start by Googling it.

Given the way we humans work, the results are far from helpful, to my mind. You get a list of top pages for "Canon EOS", and you are off on a major research project, trying to make sense of what folks have written about Canon, or what Canon has to say about its products, comparison shopping engines on price...it's not a very good experience. Wouldn't you feel better if you could just start by asking a trusted friend? But one that has the scale of Google?

So imagine a service that feels just like Google, but instead of gathering static web results, it gathers liveweb results - what people are saying, right now (or some approximation of now - say the past few hours or so), about the Canon EOS? And/or, you could post your query to that engine, and you could get realtime results that were created - by other humans - directly in response to you? Well, you can get a taste of what such an engine might look like on search.twitter.com, but that's just a taste. Add in your social graph (what your friends, and your friend's friends are saying), far more sophisticated algorithms and - most importantly - a critical mass of real time data - and those results could be truly game changing.

Now, imagine what AdSense might look like next to those results? Of course Canon will want to be there, pitching why its EOS line is the best, and of course, so will all of its competitors. Just like AdSense now. But instead of static text ads, these ads would be the beginning of true conversations between those brands and yourself.

And that's what's got me so excited. It's coming, quickly, and the game is truly afoot.