16 April 2007 - 21:20Yahoo Pipes - the revolution took place

I took a look at Yahoo Pipes and decided to write a small post about it. Don’t expect too much from it, as it is the case lately I don’t have the time to take a deep look at it.
I like the theme that Tim O’Reilly put on it in this post:
Jon expressed a vision of web sites as data sources that could be re-used, and of a new programming paradigm that took the whole internet as its platform… Using the Pipes editor, you can fetch any data source via its RSS, Atom or other XML feed, extract the data you want, combine it with data from another source, apply various built-in filters (sort, unique (with the “ue” this time:-), count, truncate, union, join, as well as user-defined filters), and apply simple programming tools like for loops. In short, it’s a good start on the Unix shell for mashups.

I concur with it, it is true: thru the use of RSS data is being exposed in a standard way and this standard is used for putting together different data sources. You could argue that RSS broke down the barriers where were separating content in different sites and that Pipes is the software which lets you combine these datasources in very imaginative ways. Pretty good. Except that I don’t think it will achieve the penetration touted by most people.
The biggest obstacle to making come true this vision of web sites as a series of datasources is the fact that Yahoo PIpes serves niche users and not breaking the barriers between various sites/datasources them. Mind you, breaking these barriers was not an easy task and keeping these barriers down relies heavily on unpaid abor (imagine what would happen to most of the Yahoo Pipes if people would stop pushing stories onto digg, book-marking items on del.icio.us, blogging about Lindsay Lohan, etc…). In a sense Yahoo Pipes lets you surf people’s past-times, but this is another story.

As I was saying, the problem is that there are way the needs to surf these datasources are very narrow. Taking a look at the numbers of times each pipe has been run would indicate that the pipes created so far are not that popular (low 5 digits are nothing today), which would indicate that they are not easily consumed. I assume that they are not easily consumed because these pipes are servicing very pointed needs, needs which are not shared by most people.
Looking at Yahoo Pipes as if it is a market you end up with the impression that it is an illiquid market: you cannot map pipe consumers to pipe producers easily. The main way of mapping pipe consumers to pipe producers is to make each pipe consumer create its own pipes by using an editor (which I must confess I have not used), the pipe consumer becomes a pipe producers and the market becomes liquid. Well, this is not happening because your average Joe will not be able to construct a meaningful Pipe, partly because it doesn’t really know where to look for content, how to filter that content efficiently and partly because it has no desire to learn how to chop-up XML (even if Yahoo claims to have made it braindead-easy).

Your average Joe needs an agent that would do that for him, that would bridge the gap between a pipe consumer and a pipe producer. Well, that agent would very likely not do it for free. So how would you put together average Joe and the agent that would service its very narrow needs? In an old post I was musing about how to service niche communities by creating mashups that would use other sites (Amazon, eBay, Google Maps and now Yahoo Pipes). An example of a Yahoo Pipe that could target a niche efficiently could be putting up a this Pipe that lets you find an apartment near parks, public libraries, produce markets, etc… You could take this pipe and put it up on a real-estate agency’s site and provide that real estate agency’s customer with something worth-while. The customer could do exactly the same thing on Yahoo Pipes, but it doesn’t know how to. Well, the real estate agency could provide this on its site and provide a better experience. The real estate agency coupled with the web-site developer that puts up pipe on the real estate agency’s site is the agent that bridges the gap between the pipe consumer and the pipe producers. I see this as a way to make the Yahoo Pipes market liquid.
Another way to make this market liquid would be create a market-place where pipe consumers would make requests for pipes along with prices and pipe producers would fulfill these requests and get paid, similar to Amazon’s MTurk. To be honest, I’m not sure if this is feasible, but it is worth noting.

Yahoo Pipes would be an interesting service. The web has been turned into a huge database where anyone with a browser can find what it needs. The revolution happened. Nobody cares.

Later Edit: In a certain way an Yahoo Pipe is a piece of code, a piece of intelectual property if you wish. Yahoo Pipes forces pipe producers to give their IP away for free (as far as I see). I wonder if this is not a barrier to adoption as well. If I make a good pipe I’d like to see some benefit coming out of it.
Another point concerning pipe creation is the large number of websites/datasources. Putting together an efficient pipe requires the pipe creator to know the data in these datasources pretty well, the barrier to pipe creation is not just chopping up data coming from different sources, but also picking up the appropriate sources for data. This is a far greater barrier to adoption as far as I see it because it shifts the effort of pipe creation from data transformation (trivial) to knowledge about data (not so trivial). My .02$