therning.org/ magnus

Ah, someone else is feeling the pain with the del.icio.us API.

Interesting information about throttling on usage of posts/all. It all adds up to del.icio.us API being nearly useless for experimentation and interesting new uses of bookmarks. I’m just so happy there are other social bookmark sites that don’t put whimsical limitations on API usage.

I was getting too many reports of problems on using epilicious together with del.icio.us. What finally did it was Gnome bug #491977. The patch provided was applied in the 2.20 tree. In the current development tree I simply removed support for del.icio.us alltogether. It’s just not worth the hassle!

It’s just somewhat unfortunate that the name of the plugin doesn’t fit that well anymore.

As I’ve said before del.icio.us is not easy to program against. Here’s a brief recount of my latest explorations into how to program against del.icio.us.

I received a report about some problems with epilicious, to which I responded with a pointer to my post about my past experiences. In response I got a suggestion to pick apart the “official” Firefox plugin for del.icio.us. It turned up some interesting details. AFAICS they are using the API in an undocumented way!

First of all every single request is a POST rather than a GET. The data they send is _user=<cookie>. That cookie is retrieved from Firefoxes cookie jar; it’s set after the user logs in at https://secure.del.icio.us/login. Apparently this allows them to use cookie:cookie as the username and password for Basic Authentication. Also, every request has src=ffbmext1.4.27 as an argument in the URI (1.4.27 is the current version of their plugin). There’s also an undocumented “function” for getting only the hashes of the bookmarks at https://api.del.icio.us/v1/posts/hashes?. The maybe most interesting detail is that they don’t seem to bother with the threat of throttling at all!

Hopeful that this might offer a way around the increasingly erratic behaviour of the API I played around with python. No luck though! A few hours of investigation, some quick hacking and long testing (once you’re throttled it takes a LONG time for del.icio.us to let you back in, sometimes more than 10 minutes!!) later I found myself not being able to hold back on profanities. The API usage description is a joke! Programming against something like that is little better than programming Windows based on MSDN

It seem the official plugin can get away with disregarding throttling because all interaction with del.icio.us happens with user interaction between each call. It takes a very fast user to trigger the throttling.

I can understand that del.icio.us wasn’t designed to be used a base for synchronising bookmarks, the throttling clearly shows that. However, one second between requests is not enough to avoid throttling and there is no indication on how long one remains in a throttled state.

This exercise has pushed me to put more energy into looking for alternative backends for epilicious. So far ma.gnolia seems good and I’d like to add support for Google bookmarks as well.

As I’ve told before del.icio.us can be a very fickle partner to interact with. Lately I’ve again started seeing obstinate behaviour from del.icio.us and yesterday I finally got around to adding support for ma.gnolia.com to epilicious. They support an API that is “del.icio.us compatible” (I believe there are some things missing, but everything epilicious needs is there) so adding the feature was extremely simple.

So far it’s not quite complete, there is no GUI to change the backend store you’ll have to interact with GConf directly, other parts of the GUI lie by still referring to del.icio.us even though ma.gnolia.com is being used. However, the backend is working and the only problem so far seems to be that ma.gnolia.com returns badly formatted XML at times (still haven’t investigated it thoroughly to say anything definite).

I haven’t pushed the changes into any of my repos yet. The only way to get it is by using the epiphany-extension-epilicious-pre package in my APT repo. Debian Sid users can simply install the package, others will have to use the source.

I haven’t had much trouble with del.icio.us before. My little epiphany plugin has worked fine. The two utility programs I wrote to test it (one makes a backup of all your bookmarks on del.icio.us the other restores them) worked fine too. Now all of a sudden it seems it’s all going downhill. A few days ago I received an email from a user saying that syncing failed for him, he got error code 500. I had never received a 500 and the API documentation doesn’t mention it either.

Short interlude here. del.icio.us asks of developers that programs using the del.icio.us API wait at least 1 second between requests. Otherwise requests will be throttled by responding with error code 503.

So, guessing a little I made my use of the API a little less aggressive. I waited a second for every request, not only after having been throttled. That solved the problem. All’s well again, right? No!!!

Yesterday I was playing around a bit with epilicious again. Moving to darcs and some refactoring. To test my changes I cleaned out my del.icio.us account using delbackup. It failed with a code 500. So, I modified the code, treating 500 in the same way as 503—wait a second. Then I received a 999 and I was locked out from del.icio.us and asked to “hold off for a few minutes and try again later, in a gentler fashion”. WTF?

I modified the code again, waiting 2 seconds for 503, 10 seconds for 500, and 15 seconds for 999ⁱ. Then I waited for the lock-out to end and I tried again. Locked out again! Tiring!

So, now I’ve modified the code again, now waiting 2 seconds before each request, 2 seconds if I ever get a 503, 10 seconds for a 500, and 15 seconds if I get a 999.

Now I’m instead waiting an awfully long time to get something done, and what happens then? I get locked out since I’m obviously not waiting long enough after receiving a 999.

I am clearly waiting long enough between requests! Has del.icio.us introduced some new arbitrary, undocumented limitation on using the API?

If anyone knows anything that I could stick as an option next to using del.icio.us I would love to hear about itⁱⁱ. Two requirements:

1. Web API, preferably as easy as del.icio.us API (even better, a compatible API)
2. Use https for connections

Dear lazyweb, any suggestions?

1. It’s not as silly as it sounds to wait when receiving a 999. From some testing I concluded that receiving a 999 doesn’t result in being locked out immidiately.[back]
2. I know of scuttle but I’m having problem finding basic information about it, e.g. what URI is the API on? I’m also not sure it provides API access over https[back]

I noticed yesterday that synching was failing due to some bug in pydelicious. That was the last straw and now pydelicious is being chucked out of epilicious. It is a module written by Frank Timmerman, and it can be found here (when it isn’t totally infested with Wiki spam anyway). I was happy to be able to use it for as long as I have since it allowed me to concentrate on other parts of epilicious. Now however epilicious is working fine but pydelicious isn’t. I am not interested in fixing pydelicious since I think it’s a bit of mess and it’d be difficult for me not to do a clean-up at the same time That would most likely take more time than writing something custom made for epilicious.

I’ve written a minimal Python module that offers exactly as much of the del.icio.us API as epilicious needs. If someone’s interested in extending it into a more full-fledged del.icio.us module—be my guest. You can find it in the development BZR repo (libepilicious/delicious.py).

I’ll be making another release of epilicious ASAP.