[maemo-developers] explanations of the recent download problems

[From: Ferenc Szekely]

Hello,

I owe you an explanation about the problems we faced with the firmware
downloads, repository.maemo.org and the windows update wizard server
backend since Xmas till the beginning of Jan.

The infrastructure is built so that the content from
repository.maemo.org is served thru a huge caching network. We were
supposed to use the same network for the firmware downloads as well,
but due to a misconfiguration and _my negligence_ all the requests
ended up at our origin server (stage.maemo.org).

Soon after the N810 launch the traffic dramatically increased and the
server could not handle it anymore. It was actually never meant to
handle such load.

We first realized the problem with tablets-dev.nokia.com where it took
ages to get any firmware. This was the time when somebody started a
torrent for the new OS 2008 image which helped a lot. We could talk
about why Nokia did not offer this, but I am not the person who will
tell you the reasons.
So the firmware images from tablets-dev could have never been served
from the caching network properly, because our backend scripts did not
send the correct HTTP headers back to the caches and the caches were
not configured either to fulfill our authentication requirements. I
take all the blame for this, so this is why I spoke about negligence
above. I must admit that fixing this was on my todo list for ages.

Almost at the same when tablets-dev "broke down" we started to receive
complaints about the update wizard on windows failing to fetch images
from our server. Well, what a coincidence... Making this backend work
properly with the caching network required a one liner change in a
script. This was done pretty quickly by the developers and the patch
was applied a week ago. Right now I am confident that the requests
coming form windows users end up at the caching network.

Beside that I am pretty sure we also got some DOS attacks at the same
time, which we clearly see at garage.maemo.org. I never went and
checked logs at stage.maemo.org, but I suspect that we were hit on all
fronts. We have taken the necessary measures to avoid these things in
the future.

I must thank my fellow Nokia colleagues (the file delivery platform
and our team here), our ISP crew and our cache providers to help and
find solutions for the problems. I believe if it was not a holiday
season we could have solved these much faster.

Never the less the problems should be over by now. If you still
experience long delays, or weird HTTP responses then please mail me
with the exact URL you're trying to access.

Thanks for the patience, and apologies for the inconvenience we caused.

Best regards,
ferenc