[maemo-developers] 3.2007.10-7 - Detailed change log?

[From: Eero Tamminen]

Hi,

First I should mention that I was discussing only bugfixes,
I agree that features should be listed, but I've understood
from Quim's mail that this should be already improving.

(Like all my other mails on this list, these are personal opinions.)

ext Neil MacLeod wrote:
>> We certainly know what we've fixed, but the issue is that there's
>> no reasonable way to know what part of that is relevant to you.
>>
> At the very least publish those bugs that are aliased to bugs in
> the Public bugzilla.  Any half-decent bug tracking system should allow
> this type of query.

Agree, and the fixed bugs in public bugzilla could be marked fixed at
the same time when the changelog is released, so that people can
verify the fixes immediately.


> You have the public bugs aliased to internal bug references and I assume
> (bad I know) that a similar relationship exists internally in which case
> simply pull out those aliased bugs that are fixed in the most recent release.

I think there might be few bugs in the public bugzilla that are not
reported internally.  At least several bugs in the public bugzilla don't
have internal bugzilla numbers in them.  For some this might be because
there's some problem with the bug report, for example the steps are not
specific enough or the bug was not reproduceble.

Anyway, I think we should have more strict rules about that. Besides
the internal bug always referring the corresponding public one, IMHO
Maemo QA should add internal bug numbers to external bugzilla once the
bugs have been reported internally.  It kind of tells the community
when and that the message has been forwarded.

Quim, could you comment this?


>> I don't think anyone here is willing to take the cost of e.g. testing
>> all the major bugs fixed from the intermediate development releases and
>> test whether they can be also replicated in the previous public release,
>> especially as <1% of that work would actually produce anything to the
>> release notes (and even less to end-user release notes) and this large
>> testing cost (for example some tests need quite elaborate setups)
>> wouldn't do anything to make the new release any better!
>>
> Is there any scope here to introduce automated testing? It sounds like
> that's what you need.

We (of course) have a huge amount of automated tests, for the
(almost as) huge number of requirements, but:
- Automated UI testing is less reliable than unit testing
- Different testing tools and methods (which are used internally)
  have different pros and cons
All this means that although test execution and most of test setup
is automated, a lot of the automated tests requires somebody to manually
verify that the test results are correct and sometimes even to verify
that test was executed correctly from the intermediate results
(screenshots etc) or just by monitoring it being run.

To give some scope on this, go to www.w3.org and browse through
the specifications required to implement a standards compliant
HTML widget for a browser application.  Then think about how many
test-cases and different tests you would need for the different
aspects in the specs.  And this is just one part of one application,
then there are rest of the apps, rest of the system, non-functional
requirements etc.

On top of that we have ad-hoc manual testing done by everybody in
the product organization.


If you were asking whether we automate tests for each bug that
has been reported and/or fixed, in general tests for bugs from ad-hoc
testing are not automated.


> However I fail to understand what your point has to do with the 
> generation of a changelog.

It's about the amount of work needed from reducing the "raw"
list of all internal bugfixes which:
- is way too large for anybody to digest as such
- 99% of it would be fixes to bugs that are not present in
  _any_ of the public releases
- has items which don't tell anything without the test-case
  description (which increase the size significantly)
- might have information that our legal wants to be cleaned out

To something that is actually:
- relevant (*tested* to happen also in previous public release)
- readable (subjects & test-cases "translated" to more legible form)
to outsiders.

Certainly it's not impossible, but I guess currently this money
is spent better improving the software.


Your request about listing public bugzilla bugs on the other hand was
something that should be fairly easy and straightforward to do.
I support it, these are the issues that users themselves have bumbed
into and seen the effort to report to us.


	- Eero