[maemo-community] Extras and Fremantle

[From: Jeremiah Foster]

On Feb 27, 2009, at 12:55 PM, Graham Cobb wrote:

> On Friday 27 February 2009 08:47:08 Jeremiah Foster wrote:
>> I feel the only way to combat this type of rogue repository creation
>> is to assure that the maemo.org repos are;
>>
>> 	1. of significantly higher quality
>> 	2. provide added value, like the way they are highly available
>> 	3. are more secure
>
> Hi Jeremiah
>
> Great to have you on board!  Now that you have had a chance to get  
> your feet
> under the desk I think it would be great if you could lead the  
> process of
> defining what our repository structure should look like for the next  
> year or
> so (i.e. Freemantle).  Of course, it is a community process but  
> having a
> Debmaster on board means we have someone with expertise to help lead  
> us and
> with some time to implement things!

This is an interesting project, hopefully working closely with Niels  
and the community we all can come up with an efficient way to  
implement new and existing repos.

> I think most people would agree that when we set up the current  
> structure
> (Extras and extras-devel) it was primarily to address the  
> "repository hell"
> we had previously: every developer had their own repository so users  
> didn't
> know where to find things, some packages (mainly ported from Debian)  
> were in
> multiple repositories (with different versions and different build  
> options),
> repositories would die and their contents would be lost, etc.

This is a nightmare, not just for the developer but for a user who  
wants to install cool apps on their tablet.

> The extras/extras-devel structure has definitely significantly  
> improved
> things: there is one place for users to go for production versions,  
> there is
> one place for developers to put test versions, there is only one  
> copy of each
> package, it is possible to automatically rebuild against new SDKs,  
> etc.
>
> Of course there are some problems as well.  One of the main problems  
> is
> quality.  It is a really hard problem not only because assuring and  
> testing
> quality is hard but also because the right policy is unclear.   
> Before we put
> in place tools and processes to enforce quality rules we have to  
> make sure we
> understand what the policy is that we are trying to implement.

There is some work being done to address QA and testing. Debian has a  
tool called lintian which is debian's policy checker. This tool can be  
adopted for maemo and run automatically over new packages. Developers  
can run it on their local machines to assure that they are in  
compliance with policy before they upload, and fix anything if they  
aren't. Of course, not everything from lintian is relevant for maemo,  
so it will be a while before we have a working tool, but hopefully not  
too long.
>
> At first sight it appears obvious: we should ensure that packages  
> cannot get
> into the repositories unless they have sufficient quality (for some  
> value
> of "sufficient").  But, on the other hand, if we set the bar too  
> high we are
> in danger of recreating the repository hell: developers who cannot  
> meet the
> bar (early versions, not enough time to improve quality, not enough  
> time to
> prove the quality, not enough users to test and evaluate it) will  
> end up
> having to create their own repositories again.  Personally, I  
> consider that a
> worse problem than the (current) quality problem (others may  
> disagree).
>
> To illustrate the problem let me use some examples of my own packages:
>
> GPE is a reasonably major set of applications for the tablet; fairly  
> widely
> installed; several developers; reasonable quality; active bug  
> tracker and
> some bugs even get fixed!

Time to start a company - when you fix bugs you are no longer 'open  
source'. :)

> I currently have a stable version in Extras, a beta
> version in extras-devel (which I intend to move to extras soon, to  
> replace
> the current stable version) plus I have two other private  
> repositories: one
> for the GPE development team which just builds from SVN every night,  
> with no
> testing; the other for my own use, used primarily for installation  
> testing.
> I do not put new versions even in extras-devel unless I have had a  
> few people
> from the GPE team test that it installs and does not crash. This  
> seems to
> work reasonably well except several end-users wish to use the Beta  
> version
> (and I want them to, to get testing before promoting it to extras)  
> and they
> find that if they enable extras-devel they are in danger of  
> installing all
> sorts of other packages of very variable status and possibly  
> breaking their
> tablet.
>
> Xsisusb, on the other hand, is a very small project.  It has at most  
> 4-5 users
> and it is really a proof-of-concept which requires more work before  
> it is
> usable.  I do have packages in extras-devel but none in extras.  It is
> announced in ITT but does not have a download page: only hard-core  
> hackers
> are likely to want this so that works fine.  On the other hand, if  
> there was
> a quality bar for entry to extras-devel this package would certainly  
> fail it
> and I would need to create my own repository.

I envision a policy checking tool that would check policy, but would  
not prevent packages that failed these checks. So you could still have  
your package in the repos while you worked on policy compliance.

>
> OpenSync is a "real" project: several developers; related to major  
> projects
> like KDE and packaged in Debian; I released previous versions for  
> Gregale and
> Bora. On the other hand, OpenSync is currently being re-written and  
> the
> current version is still very broken; definitely pre-alpha.  I have  
> not even
> put it into extras-devel.  For this I have a personal daily build  
> repository
> (for use mainly by me but also a couple of other people who are very  
> keen to
> help with getting it working).  Once we get something which at least  
> "works"
> at some basic level I will want to put an "alpha release" somewhere  
> for the
> small team to work from.  I intend that to be extras-devel (just  
> like for
> Xsisusb).  Again, it would definitely fail any quality metrics and  
> is not
> intended for anyone to install unless they are actively in  
> discussions with
> me or other members of the OpenSync team.
>
> These are just three examples of projects in different stages and of  
> different
> sizes (in terms of code and of users).  There are many other examples,
> different again.  So, I would be really interested to hear proposals  
> from you
> (and anyone else) on how we should evolve our repository structure  
> to support
> all these and minimise the need to attach packages to ITT posts or  
> create
> personal repositories (although I believe they do have a legitimate  
> role for
> personal use and for development team use).  Is it time for us to  
> move to a
> Debian style stable/testing/unstable/experimental structure?  Can we  
> do
> something clever with control file information to have the user  
> select more
> or less safe versions of things to install?

I feel that if we increase the quality of packages in Extras-devel,  
that will also increase the quality of packages in Extras. I think the  
tried-and-true method shown by debian, i.e. a policy checking tool, is  
the right way to go. It warns those managing the repos about the more  
common build problems, it warns developers what they need to fix, and  
it lets users know that the packages in the Extras* repos have had  
some quality control - so they are more likely to go there first. If  
users go there first, developers will follow since developers want  
users. Add to this scenario increased security, better tools, and more  
and better packages.

I think this is a recipe for an excellent ecosystem.

Looking forward to feedback and thanks again for your email,

Jeremiah