[maemo-developers] Autobuilder apt-get problems => failing builds

[From: Jeremiah Foster]

On Jan 1, 2010, at 16:00, Jeff Moe wrote:

> On Friday 01 January 2010 11:37:24 you wrote:
>> 2010/1/1 Jeff Moe <moe at blagblagblag.org>:
>>> On Friday 01 January 2010 11:12:47 Ed Bartosh wrote:
>>>> 2010/1/1 Jeremiah Foster <jeremiah at jeremiahfoster.com>:
>>>>> On Jan 1, 2010, at 15:00, Ed Bartosh wrote:
>>>>>> 2010/1/1 Andrew Flegg <andrew at bleb.org>:
>>>>>>> Hi,
>>>>>>> 
>>>>>>> Attempting to upload a new version of vim to the Fremantle
>>>>>>> auto-builder, I get the following failure:
>>>>>>> 
>>>>>>> 
>>>>>>> https://garage.maemo.org/builder/fremantle/vim_7.2-0maemo6/armel.roo
>>>>>>> t.l og.FAILED.txt
>>>>>> 
>>> 
> 
> Well, you'd have a better case if things worked more reliably.
> 
>> Anyway it was not working for a long time and people became confused.
>> And now I'm restarting all those 40 builds manually.
> 
> I think we should look to Fedora since they have a similar arrangement: 
> "community" distribution with corporate overlord.
> 
> This is how they do it:
> 
> * IRC channel of admin issues: #fedora-admin and #fedora-noc where you can 
> watch things "live".

Good idea.
> 
> * Standard Operating Procedure for outages:
> https://fedoraproject.org/wiki/Outage_Infrastructure_SOP

Really good idea, we should have an SOP for that.
> 
> * *PAGER* access, available to the public, where you can page one of 9 admins 
> (a bit unbelievable, actually):
> https://admin.fedoraproject.org/pager

I don't want to wear a pager, and I don't think it is necessary, but still.

> * More people would know how the whole *.maemo.org infrastructure actually 
> worked if information about it was public. The joke is that it runs on a N700. 

It's no joke. </sarcasm> Nokia is doing a lot to address the infrastructure problem. Part of it comes from the open / not open split and how a International Corporation deals with integrating Free Software developers into a corporate workflow. Nokia is not perfect, but they are willing to learn and work hard, so I thin being patient here is worth your while. Red Hat has not been prefect here either.

> But people can make this joke because the actual server set up is known by 
> only a few. Compare that to this dream:
> https://fedoraproject.org/wiki/Category:Infrastructure_SOPs

This is awesome, I think this is a good idea. Part of the problem has been critical mass - we haven't had enough developers willing to take on these kind of issues, Nokia has had to provide it or recruit people. We have grown a lot in 2009 so maybe we can get teams from the community to do this sort of work.
> 
> Anyway, they are doing things far better and I don't see people griping about 
> outages over there much at all.

Different project, different goals, they've been around a lot longer (they were once Red Hat remember) and they have a lot of experience. You have to give credit to Nokia and the people who work hard on the infrastructure - they do a lot of really hard work and this whole experiment is fairly new for Nokia. I am confident the whole process will improve.
> 
> What's the procedure for Maemo? Dive into #maemo-devel and hope someone knows 
> WTF is up? Their answer is usually "wait for x-fade". Post to talk.m.o.? Hit 
> reload on qaiku? Post a comment there? Add more here?
> https://bugs.maemo.org/show_bug.cgi?id=5818

> Surprisingly I was told by an @nokian that reporting to that bug *was* the 
> correct place to report outages (!).

I think it would be less than honest to not say this is suboptimal.

> Anyway, there are organizations all around the world that run servers 24/7/365 
> with minimal outages that have more than 2 admins with access. That is 
> obvious. The maemo infrastructure is no where near approaching 99% uptime (let 
> alone .999s). A mere "40" submitted builds is also a loss of time and momentum 
> of many developers...

Nokia is a handset maker. They are not a company that does a lot of public facing internet server work. That is changing. Constructive criticism from the community will help, like the good examples you have linked to. We as a community need to continue to show a good example and demonstrate to Nokia that we are reliable and reasonable, then we can build out maemo into something we are proud of.

Jeremiah