Hi,<br><br>this email was sent to the council private list, but since there's nothing private here, and the email has some valuable points/suggestions, I'm replying here in the public mailing list.<br>I did some censorship in the email in order to avoid flame wars, since there's some rants including names, if the author wants he can paste these parts here, IMO these parts are not valuable to the discussion, so should be avoided :).<br>
<br><br><div class="gmail_quote">On Sun, Jan 3, 2010 at 6:46 PM, Jeff Moe <span dir="ltr"><<a href="mailto:moe@blagblagblag.org">moe@blagblagblag.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
I don't need to tell you that the *.<a href="http://maemo.org" target="_blank">maemo.org</a> infrastructure is very broken.<br>
Surely you have experienced frequent outages yourselves. I haven't experienced<br>
such poor service from any other distro that I can remember, big or small.<br></blockquote><div><br>True, but I've told you several times that there's a ongoing servers move, this is more complicated that it seems (+ holidays).<br>
Those older in the community know that before the users influx brought by the N900, things were very different. <br></div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
I am writing to ask that some sort of Standard Operating Procedure be<br>
developed for handling outages and server updates. I have brought up some of<br>
these points on the maemo-devel mailing list (<br>
<a href="http://lists.maemo.org/pipermail/maemo-developers/2010-January/023329.html" target="_blank">http://lists.maemo.org/pipermail/maemo-developers/2010-January/023329.html</a> ):<br>
<br>
==========================================<br>
I think we should look to Fedora since they have a similar arrangement:<br>
"community" distribution with corporate overlord.<br>
<br>
This is how they do it:<br>
<br>
* IRC channel of admin issues: #fedora-admin and #fedora-noc where you can<br>
watch things "live".<br>
<br>
* Standard Operating Procedure for outages:<br>
<a href="https://fedoraproject.org/wiki/Outage_Infrastructure_SOP" target="_blank">https://fedoraproject.org/wiki/Outage_Infrastructure_SOP</a><br>
<br>
* *PAGER* access, available to the public, where you can page one of 9 admins<br>
(a bit unbelievable, actually):<br>
<a href="https://admin.fedoraproject.org/pager" target="_blank">https://admin.fedoraproject.org/pager</a><br>
<br>
* More people would know how the whole *.<a href="http://maemo.org" target="_blank">maemo.org</a> infrastructure actually<br>
worked if information about it was public. The joke is that it runs on a N700.<br>
But people can make this joke because the actual server set up is known by<br>
only a few. Compare that to this dream:<br>
<a href="https://fedoraproject.org/wiki/Category:Infrastructure_SOPs" target="_blank">https://fedoraproject.org/wiki/Category:Infrastructure_SOPs</a><br></blockquote><div><br>It's a 770 ;)<br><br>I agree, we need more communication from the team that deals with the servers. <br>
</div><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<br>
Anyway, they are doing things far better and I don't see people griping about<br>
outages over there much at all.<br>
<br>
What's the procedure for Maemo? Dive into #maemo-devel and hope someone knows<br>
WTF is up? Their answer is usually "wait for x-fade". Post to talk.m.o.? Hit<br>
reload on qaiku? Post a comment there? Add more here?<br>
<a href="https://bugs.maemo.org/show_bug.cgi?id=5818" target="_blank">https://bugs.maemo.org/show_bug.cgi?id=5818</a><br>
<br>
Surprisingly I was told by an @nokian that reporting to that bug *was* the<br>
correct place to report outages (!).<br>
<br>
<br>
Anyway, there are organizations all around the world that run servers 24/7/365<br>
with minimal outages that have more than 2 admins with access. That is<br>
obvious. The maemo infrastructure is no where near approaching 99% uptime (let<br>
alone .999s). A mere "40" submitted builds is also a loss of time and momentum<br>
of many developers...<br>
<br>
==========================================<br>
<br>
After further discussion (not in the thread) with Nigel Jone's from Fedora,<br>
there are *18* people that can fix Fedora buildserver issues. They use "git"<br>
and "puppet" to track configuration changes. None of this is majick or unique.<br>
These are standard system administration practices which should be followed by<br>
Maemo.<br>
<br>
Also, I have suggested using mirrors of content, which is what every other<br>
distro that I know of, big or small, community or corporate does. For instance<br>
see:<br>
<br>
<a href="http://www.debian.org/mirror/list" target="_blank">http://www.debian.org/mirror/list</a><br>
<a href="http://www.novell.com/products/opensuse/downloads/ftp/int_mirrors.html" target="_blank">http://www.novell.com/products/opensuse/downloads/ftp/int_mirrors.html</a><br>
<a href="http://mirrors.fedoraproject.org/publiclist" target="_blank">http://mirrors.fedoraproject.org/publiclist</a><br>
<a href="http://api.mandriva.com/mirrors/list.php" target="_blank">http://api.mandriva.com/mirrors/list.php</a><br>
<a href="http://www.ubuntu.com/getubuntu/downloadmirrors" target="_blank">http://www.ubuntu.com/getubuntu/downloadmirrors</a><br>
<a href="http://www.gentoo.org/main/en/mirrors2.xml" target="_blank">http://www.gentoo.org/main/en/mirrors2.xml</a><br>
Even tiny distros like puppy linux have mirrors:<br>
<a href="http://www.puppylinux.com/download/" target="_blank">http://www.puppylinux.com/download/</a><br>
<br>
Here is a thread I started on maemo-devel about mirrors:<br>
<a href="http://lists.maemo.org/pipermail/maemo-developers/2010-January/023363.html" target="_blank">http://lists.maemo.org/pipermail/maemo-developers/2010-January/023363.html</a><br>
<br>
I think the sysadmins should provide more info about what they are doing, what<br>
is planned, when the outages are coming, etc. This is in line with common<br>
practices. For instance, they could:<br>
<br>
* Send emails to maemo-devel (or whatever list), advising of forthcoming<br>
outages.<br>
<br>
* Send updates when they are aware of an outage (e.g. "the builder is known<br>
down, we're working on it")<br>
<br>
* Send updates when work has been complete ("the new garage server is up")<br>
<br>
Right now there are occasional "tweets" to qaiku, but this falls far short. If<br>
you need examples of this done correctly, let me know and I'll send you some.<br>
<br>
STFU and just sit and wait is not a good answer, but I've been hearing that<br>
for a month now. I have wasted much of the last month just waiting for things to work correctly. These<br>
failures are killing the momentum, initiative, and goodwill of your developer<br>
community. Not only that, it is dividing them....<br>
<br>
-Jeff<br>
<a href="http://wiki.maemo.org/User:Jebba" target="_blank">http://wiki.maemo.org/User:Jebba</a><br>
<a href="http://maemo.org/profile/view/jebba/" target="_blank">http://maemo.org/profile/view/jebba/</a><br>
<br>
P.S. <a href="http://maemo.org/community/council/" target="_blank">http://maemo.org/community/council/</a> should probably have the<br>
<a href="mailto:council@maemo.org">council@maemo.org</a> address on it and a list of current members.<br></blockquote><div><br>That's the council blog, but yes we should link to this page: <a href="http://wiki.maemo.org/Community_Council">http://wiki.maemo.org/Community_Council</a><br>
<br>Thanks for the suggestions. <br></div></div><br><br>Best regards,<br><br>-- <br>Valério Valério<br>Maemo Community Council Chair<br><br><a href="http://www.valeriovalerio.org">http://www.valeriovalerio.org</a><br>