Yahoo! Pipes takes aggregation and integration to the masses
In recent years, widespread adoption of RSS technology really pushed the concept of aggregating content/data from multiple sources into the mainstream. The problem is that as the number of sources increases, so does the amount of junk and duplicate content. Similarly, there are a number of web-based tools and services out there that are great on their own – but would be even better if they were used in conjunction with one another (i.e. the struggle for social bookmarking supremacy amongst Digg, del.icio.us, and countless others).In recent years, widespread adoption of RSS technology really pushed the concept of aggregating content/data from multiple sources into the mainstream.
The problem is that as the number of sources increases, so does the amount of junk and duplicate content.
Similarly, there are a number of web-based tools and services out there that are great on their own – but would be even better if they were used in conjunction with one another (i.e. the struggle for social bookmarking supremacy amongst Digg, del.icio.us, and countless others).
Yahoo’s Pipes service (which has been around in beta for about a year and a half now), gives users the power to aggregate and mash together all sorts of data from around the web in more ways than ever. Users can create a Pipe by pulling in a number of RSS feeds / web searches / APIs, aggregating that content into one “pool”, and applying logic to strip out irrelevant or duplicate content from the final output.
The drag-and-drop UI is slick and intuitive, allowing everyone from the layman to the developer to build algorithms of varying complexity without needing to create a single line of code. Here’s a glimpse into the look and feel of the UI:
CNN, MSNBC, and NYT.
I would drag a block for each feed into the Pipes workspace, and enter the URL for each site’s “Financial News” feed. Then, I would drag in a Union module that pulls all of that content together, a Filter module that only includes content whose description contains “Paulson bailout”, then another Filter that removes content with duplicate descriptions such as repurposed AP reports, and a Sort module that sorts your output by publication date.
Then you can start looking through the output of your Pipe to see how your content looks. If you’re noticing that you still have a couple strange things slipping through the cracks, you can tweak your logic accordingly over time. Perhaps most importantly, the open and modular nature of Pipes allows you to include (or be inspired by) Pipes that have been created and shared by other users, since the underlying architecture is openly accessible and easy to interpret.
Many people also add search controls to their pipe, so you can search across some of the most reliable sources. Here are some Pipes that I think are interesting:
- Amazon, Craigslist, and Ebay Price Watcher: http://pipes.yahoo.com/idopipes/watcher
- Favorite Artists News (pulled from your Favorite Artists on Last.fm): http://pipes.yahoo.com/associativetrails/artistnews
- YouTunes Top 100 (list of YouTube music videos ordered by ranking in iTunes Top 100): http://pipes.yahoo.com/ejcampbell/MM_0K18W3BG9N1OnBx2yXQ
- Free iTunes Downloads – Videos (includes tv shows and music videos): http://pipes.yahoo.com/pipes/pipe.info?_id=wqbPSrrB2xG53oZR0kqv4w
What are your favorite Pipes?
Comments
Post new comment