[maemo-developers] How to destroy your community

[From: Jeff Moe]

On Tuesday 19 January 2010 06:13:00 tero.kojo at nokia.com wrote:
> > Some gems:
> > 1) "It's also important to set up an official web site which is down as
> > often as it's up. It's not enough to have no web site at all; in such
> > situations, the community has an irritating habit of creating sites of
> > its own. But a flaky site can forestall the creation of those sites,
> > ensuring that information is hard to find. "
> 
> Jeff, you have been hanging around here for what, 2 months?
> And those two months have been the time in which this place has seen it's biggest growth of all time. Sure that made the system break in the corners, but it's being fixed.

I've had my device for 1200 hours so far (50 days). For how many of those hours have the various services been down? I'm not sure everything has ever been working for a full complete day. "Break in the corners"? That's quite a gloss.

We've heard for quite some time (months before I came) about the big "server move". Is this done? AFAICT, the servers have been moved and the new system is set up with "tweaks" being done (such as akamai serving gunzipped files!). Correct me if I'm wrong, and I hope I am, but it appears the servers have been moved to an ISP that cannot fix a broken NFS mount on the weekend. Is this really the situation? How on earth did maemo get stuck with this ISP? Who are they, in fact?

It also looks like maemo.org DNS depends upon Nokia's nameservers. Apart from the NFS problems over the weekend, DNS resolution was also awry. Is there no one to fix DNS issues at Nokia on weekends? Is this really why DNS was down for so long? That can hardly be blamed on growth. A small pentium could keep up with maemo DNS requests. This is a problem of design and procedure, not of growth. DNS nowadays is often as simple as filling in a web form. This isn't rocket science.

Perhaps it's time to move maemo.org DNS out of Nokia's hands and to a separate provider. There are likely hundreds that will provide better service than Nokia. I recommend DirectNIC since they were able to withstand, with zero downtime, hurricane Katrina from downtown New Orleans, but there are many many providers that could handle it.

> Hang around for a couple of more years and then come back with the statistics.

Do you have any statistics for 2009? Would you be willing to share them? Do you realize even if you run things perfectly 100% for the rest of 2010 you have zero chance of 99.9% uptime (a reasonable baseline)? So I'll have to wait until 2011 statistics til the servers are more in line with industry expectations.

A few admins commented in threads about the outage along the line of "if something is down, we work non-stop until it is back up" or even "we'd be laid off if this happend at work". It seems that attitude is lacking with respect to maemo. During the most recent outage, I was able to provision an entirely new server and have a fresh OS install on a second one while on "vacation" in the mountains with a seriously crap wifi connection, yet no one in all of maemo/nokia was able to do anything to alleviate the outages?

I don't have to be here for years to see how things are managed and where the problems are.

-Jeff Moe
http://wiki.maemo.org/User:Jebba