Mailingliste - Einträge 2005

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [qftestJUI] Option for regoldening checks

  • Subject: Re: [qftestJUI] Option for regoldening checks
  • From: Gregor Schmid <Gregor.Schmid@?.de>
  • Date: 15 Dec 2005 16:35:34 +0100

Hi Rob,

I think you're missing something. There should be no manual searching
and node unfolding if you use "Jump to next error (Ctrl-N). The idea
is that after a failed test-run you:

1 open the run-log

2 use Ctrl-N to jump to the next error

3 analyze the error

4 if it's a check that should be regoldened, use the new functionality
  to do that (i.e. a menu entry with a new shortcut, e.g. Ctrl-A).
  This'll pop up the test-suite with the updated check node

- repeat 2-4 until all errors are resolved

Save the test-suite and you're done.

Best regards,

"Robert Slifka" <rslifka@?.com> writes:

>    Hi Gregor,
>    In our environment, we keep our tests small and we don't have "known
>    failure" tests, so this wouldn't be an issue.  I.e. the tests will
>    fully pass prior to enabling a regoldening option.
>    Let's go through a sample scenario, using screenshots as an example:
>    1 - Through some design decision, we realize we could have chosen a
>    better background color.
>    2 - We change the background color.
>    3 - Our passing tests now fail.  Open each in QFS.
>    4 - Review of those passing tests show that the color change was the
>    only cause of a dozen screenshot failures.
>    5 - Enabling screenshot regoldening, re-run each test manually in QFS.
>    6 - Voila!
>    Reviewing each regoldened check is acceptable, so long as the tool
>    provides support for it.  O/w it's a lot of manual searching and
>    node-unfolding.
>    If you automatically expanded the path to regoldened nodes after
>    running in this node, that would make it much easier.
>    Or opened a new window containing only those nodes that were
>    regoldened.  However in this approach you lose context which will make
>    verification difficult.
>    Rob
>    -----Original Message-----
>    From: Gregor Schmid [[1]mailto:Gregor.Schmid@?.de]
>    Sent: Tue 12/13/2005 11:44 PM
>    To: qftestJUI-list@?.de
>    Cc: Robert Slifka
>    Subject: Re: [qftestJUI] Option for regoldening checks
>    Hi Rob,
>    there are plans for regoldening checks for version 2 of qftest, but
>    according to these you'll still need to look at each error
>    individually and say "yes, this was correct, use that result as the
>    new value for the check". If I understand you correctly you're
>    proposing to automate that even further but how should qftestJUI know
>    which check was a real error and which was actually OK?
>    Best regards,
>        Greg
>    "Robert Slifka" <rslifka@?.com> writes:
>    >    Hi QFS team,
>    >    Frequently we'll want to regolden a type of check when running
>    tests.
>    >    For example, if we have a test with several image checks of the
>    same
>    >    component and that component has say, changed color.  I
>    understand you
>    >    can go to them individually and use the current screenshot, but
>    it
>    >    would be much easier to have QFS regolden certain checks on its
>    own.
>    >    Thanks,
>    >    Rob

Gregor Schmid                                Gregor.Schmid@?.de
Quality First Software GmbH           
Tulpenstr. 41                                Tel: +49 8171 919870
DE-82538 Geretsried                          Fax: +49 8171 919876