2017 up to now  | 2016 | 2015 | 2014 | 2013 | 2012 | 2011 | 2010 | 2009 | 2008 | 2007

(older archive entries before 2007 are not shown here, but included in the onsite-search)

Mailing List - Entries of 2011


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [QF-Test] Timed Test-Case / Test-Set / Test-Suite


  • Subject: Re: [QF-Test] Timed Test-Case / Test-Set / Test-Suite
  • From: Gregor Schmid <Gregor.Schmid@?.de>
  • Date: Mon, 11 Jul 2011 22:18:38 +0200

Hi Ivan,

at this stage the evaluation part is not QF-Test's job, which ends
when creating run-log and report(s). Tracking test-suite execution
time over a longer period and detecting significant changes must be
done externally by whatever tool you use to setup and start the tests.

However, having a time-limit after which execution of a test-suite is
aborted could be quite handy. Every now and then we have a rouge
suite that fails on a basic dependency setup so it tries time and
again to connect to the SUT for each test-case. It still runs through
with many exceptions, but can take a long time. Of course the
preferred way of handling this is to look out for un-resolvable
dependencies and stop the whole test via a script with

rc.stopTest()

in such a case. But that will only catch the foreseeable problems, not
the occasional really weird ones. For those, a limit for the total
execution time of a test-suite could be quite handy. I'll take a note.
If this feature gets some votes or the pain grows beyond the
occasional time lost it may even get implemented ;-).

Best regards,
    Greg

Ivan Boelle <ivan.boelle@?.com> writes:

> Gregor,
>
> Yes I Will try to get some dumps before freeing the test workstation.
>
> About the general performance, I was thinking about a "per Suite"
> timer rather than a per node.
> For example:
> As I know the average Time for one of my Suites is about 45mn. I would
> like to be warned if it changes more than 20% (< 35mn or > 55mn)
>
> Best regards
>
>
>
> Le 8 juil. 2011 à 21:32, Gregor Schmid <Gregor.Schmid@?.de> a écrit :
>
>> Hi Ivan,
>>
>> I've been thinking about this, but can't even come up with an
>> inofficial solution. Currently it is not possible to implement
>> a general timeout, but it is an interesting idea.
>>
>> If you run into such a problem with a hanging test again, please try
>> to find out more information about why this might be the case. Useful
>> would be full thread dumps of both QF-Test and the SUT as well as a
>> look at memory consumption. All of this information is available via
>> the jvisualvm tool that comes with a current JDK.
>>
>> The other part of the question, tracking general performance, is
>> easier. This can be done by implementing a TestRunListener. However,
>> I'm not sure what you would look for. It's hard to define general
>> expectations for performance per node.
>>
>> Best regards,
>>    Greg
>>
>> Ivan Boelle <ivan.boelle@?.com> writes:
>>
>>> Hello,
>>>
>>> I would like to set a global maximum time on a Test-case or Test-Set.
>>> I found the "timed sequence" node but I would like to set it globally instead of on a very
>>> particular sequence.
>>>
>>> My goals are:
>>> - Avoid an issue with our workstations/Jenkins/SUT/? which leads to a (random) test step that
>>> never ends. No ending means, no runlog, and test-environment busy until someone kill the thing
>>> manually.
>>> - Check the "global-average" performance and detect major performance regressions.
>>>
>>> Any idea ?
>>>
>>> Best regards,
>>>
>>> ---
>>> Ivan Boelle
>>> INT, Interactive Network Technologies

-- 
Gregor Schmid                                Gregor.Schmid@?.de
Quality First Software GmbH                     http://www.qfs.de
Tulpenstr. 41                               Tel: +49 8171 38648-0
DE-82538 Geretsried                         Fax: +49 8171 3864816
GF: Gregor Schmid, Karlheinz Kellerer          HRB München 140833