Clash of the APIs: Social Web Traffic Jam

Something I've noticed lately is that the social web has been having more than is fair share of traffic jams. As each application launches its own API (application programming interface), inviting other Web 2.0 apps to play in each sandbox, the communications seem to get bogged down. Who hasn't experienced this phenomenon lately?

One application plagued by API time out errors and traffic jams seems to be Twitter. Often, when using a Twitter client like Twhirl, the user will get booted for five minutes for too many calls to the Twitter API. Why does this happen, when the user has no other Twitter-related applications open? Because other Twitter-based clients, like FriendFeed, are pulling from the Twitter stream as well, refreshing consistently at varying intervals.

Just having multiple applications, services and clients wading in the Twitter stream isn't enough to truly clog the river of data. No, the river of data gets clogged when a popular tweeter starts to drive up the popularity of various clients, promoting the ones they use most or like best. FriendFeed is one example of growing popularity causing problems. Ever since Twitter users like Scoble have declared it their choice for life streaming, FriendFeed use has skyrocketed.

FriendFeed (and many other Twitter clients, services and applications) pulls data on your Tweets, tweet buddies, blog entries, social bookmarks, social network updates and more, putting it all in a simple to use web site. It's very useful in that you can not only follow people, you can interact. Twitter is an easy example of the recently bogged down internet, but nearly any application with an open API and a variety of calls against it from third party developers can fall victim to these traffic jams.

What is the solution? The answer is that there is no good solution yet. Twitter continues to upgrade its servers to combat the downtime. Twitter clients and other online applications that draw on open APIs of other sites have begun to streamline their code and restrict the number of calls being made on an API. Users of conflicting sites like Twhirl and FriendFeed remove the Twitter stream call from the online applications to reduce the load.

What do you do if you don't want to give up one client feature for the ease of use of another? What happens if you don't want to turn off one of your Twitter calls so another client can work better, for example. The answer is that your online world starts to fragment. Applications and clients falling victim to the traffic jam problem see immediate drops in their user base. People who use the applications start to move on to other clients that are more consistently active. Until a better solution than home remedies and less usage are found, these traffic jams will continue to clog the internet, making it impossible to use it to its full potential.