[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
RE: [qftestJUI] Option for regoldening checks
- Subject: RE: [qftestJUI] Option for regoldening checks
- From: "Robert Slifka" <rslifka@?.COM>
- Date: Wed, 14 Dec 2005 08:16:04 -0800
Title: RE: [qftestJUI] Option for regoldening checks
In our environment, we keep our tests small and we don't have "known failure" tests, so this wouldn't be an issue. I.e. the tests will fully pass prior to enabling a regoldening option.
Let's go through a sample scenario, using screenshots as an example:
1 - Through some design decision, we realize we could have chosen a better background color.
2 - We change the background color.
3 - Our passing tests now fail. Open each in QFS.
4 - Review of those passing tests show that the color change was the only cause of a dozen screenshot failures.
5 - Enabling screenshot regoldening, re-run each test manually in QFS.
6 - Voila!
Reviewing each regoldened check is acceptable, so long as the tool provides support for it. O/w it's a lot of manual searching and node-unfolding.
If you automatically expanded the path to regoldened nodes after running in this node, that would make it much easier.
Or opened a new window containing only those nodes that were regoldened. However in this approach you lose context which will make verification difficult.
From: Gregor Schmid [mailto:Gregor.Schmid@?.de]
Sent: Tue 12/13/2005 11:44 PM
Cc: Robert Slifka
Subject: Re: [qftestJUI] Option for regoldening checks
there are plans for regoldening checks for version 2 of qftest, but
according to these you'll still need to look at each error
individually and say "yes, this was correct, use that result as the
new value for the check". If I understand you correctly you're
proposing to automate that even further but how should qftestJUI know
which check was a real error and which was actually OK?
"Robert Slifka" <rslifka@?.com> writes:
> Hi QFS team,
> Frequently we'll want to regolden a type of check when running tests.
> For example, if we have a test with several image checks of the same
> component and that component has say, changed color. I understand you
> can go to them individually and use the current screenshot, but it
> would be much easier to have QFS regolden certain checks on its own.
Gregor Schmid Gregor.Schmid@?.de
Quality First Software GmbH http://www.qfs.de
Tulpenstr. 41 Tel: +49 8171 919870
DE-82538 Geretsried Fax: +49 8171 919876