[maemo-developers] Continued outages and lame updates

[From: Valerio Valerio]

Hi,

this email was sent to the council private list, but since there's nothing
private here, and the email has some valuable points/suggestions, I'm
replying here in the public mailing list.
I did some censorship in the email in order to avoid flame wars, since
there's some rants including names, if the author wants he can paste these
parts here, IMO these parts are not valuable to the discussion, so should be
avoided :).


On Sun, Jan 3, 2010 at 6:46 PM, Jeff Moe <moe at blagblagblag.org> wrote:

> I don't need to tell you that the *.maemo.org infrastructure is very
> broken.
> Surely you have experienced frequent outages yourselves. I haven't
> experienced
> such poor service from any other distro that I can remember, big or small.
>

True, but I've told you several times that there's a ongoing servers move,
this is more complicated that it seems (+ holidays).
Those older in the community know that before the users influx brought by
the N900, things were very different.

>
> I am writing to ask that some sort of Standard Operating Procedure be
> developed for handling outages and server updates. I have brought up some
> of
> these points on the maemo-devel mailing list (
> http://lists.maemo.org/pipermail/maemo-developers/2010-January/023329.html):
>
> ==========================================
> I think we should look to Fedora since they have a similar arrangement:
> "community" distribution with corporate overlord.
>
> This is how they do it:
>
> * IRC channel of admin issues: #fedora-admin and #fedora-noc where you can
> watch things "live".
>
> * Standard Operating Procedure for outages:
> https://fedoraproject.org/wiki/Outage_Infrastructure_SOP
>
> * *PAGER* access, available to the public, where you can page one of 9
> admins
> (a bit unbelievable, actually):
> https://admin.fedoraproject.org/pager
>
> * More people would know how the whole *.maemo.org infrastructure actually
> worked if information about it was public. The joke is that it runs on a
> N700.
> But people can make this joke because the actual server set up is known by
> only a few. Compare that to this dream:
> https://fedoraproject.org/wiki/Category:Infrastructure_SOPs
>

It's a 770 ;)

I agree, we need more communication from the team that deals with the
servers.

>
> Anyway, they are doing things far better and I don't see people griping
> about
> outages over there much at all.
>
> What's the procedure for Maemo? Dive into #maemo-devel and hope someone
> knows
> WTF is up? Their answer is usually "wait for x-fade". Post to talk.m.o.?
> Hit
> reload on qaiku? Post a comment there? Add more here?
> https://bugs.maemo.org/show_bug.cgi?id=5818
>
> Surprisingly I was told by an @nokian that reporting to that bug *was* the
> correct place to report outages (!).
>
>
> Anyway, there are organizations all around the world that run servers
> 24/7/365
> with minimal outages that have more than 2 admins with access. That is
> obvious. The maemo infrastructure is no where near approaching 99% uptime
> (let
> alone .999s). A mere "40" submitted builds is also a loss of time and
> momentum
> of many developers...
>
> ==========================================
>
> After further discussion (not in the thread) with Nigel Jone's from Fedora,
> there are *18* people that can fix Fedora buildserver issues. They use
> "git"
> and "puppet" to track configuration changes. None of this is majick or
> unique.
> These are standard system administration practices which should be followed
> by
> Maemo.
>
> Also, I have suggested using mirrors of content, which is what every other
> distro that I know of, big or small, community or corporate does. For
> instance
> see:
>
> http://www.debian.org/mirror/list
> http://www.novell.com/products/opensuse/downloads/ftp/int_mirrors.html
> http://mirrors.fedoraproject.org/publiclist
> http://api.mandriva.com/mirrors/list.php
> http://www.ubuntu.com/getubuntu/downloadmirrors
> http://www.gentoo.org/main/en/mirrors2.xml
> Even tiny distros like puppy linux have mirrors:
> http://www.puppylinux.com/download/
>
> Here is a thread I started on maemo-devel about mirrors:
> http://lists.maemo.org/pipermail/maemo-developers/2010-January/023363.html
>
> I think the sysadmins should provide more info about what they are doing,
> what
> is planned, when the outages are coming, etc. This is in line with common
> practices. For instance, they could:
>
> * Send emails to maemo-devel (or whatever list), advising of forthcoming
> outages.
>
> * Send updates when they are aware of an outage (e.g. "the builder is known
> down, we're working on it")
>
> * Send updates when work has been complete ("the new garage server is up")
>
> Right now there are occasional "tweets" to qaiku, but this falls far short.
> If
> you need examples of this done correctly, let me know and I'll send you
> some.
>
> STFU and just sit and wait is not a good answer, but I've been hearing that
> for a month now. I have wasted much of the last month just waiting for
> things to work correctly. These
> failures are killing the momentum, initiative, and goodwill of your
> developer
> community. Not only that, it is dividing them....
>
> -Jeff
> http://wiki.maemo.org/User:Jebba
> http://maemo.org/profile/view/jebba/
>
> P.S. http://maemo.org/community/council/ should probably have the
> council at maemo.org address on it and a list of current members.
>

That's the council blog, but yes we should link to this page:
http://wiki.maemo.org/Community_Council

Thanks for the suggestions.


Best regards,

-- 
Valério Valério
Maemo Community Council Chair

http://www.valeriovalerio.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.maemo.org/pipermail/maemo-developers/attachments/20100104/98a99d0e/attachment.htm