Re: About repeating exactly the same tests

On Sat, 23 Nov 2002, Ko Tensei_at_KIT wrote:

> By reading the "repeating exactly the same tests" in User Manual,
> I'v learned to set the command-line option as below.
> --------------------------------------------------------
> srv1> polysrv --unique_world no --local_rng_seed 201
> --verb_lvl 10 --delete_old_addrs 0 --log ./tmp/srv.log
> (one polysrv process only in my case)
> ..
> clt1> polyclt --unique_world no --local_rng_seed 101
> --verb_lvl 1 --dump errs --console ./tmp/runclt.10.101.0.log
> --delete_old_addrs 0 --ports 3000:40000 --proxy 127.0.0.1:8080
> clt2> polyclt --unique_world no --local_rng_seed 102
> --verb_lvl 1 --dump errs --console ./tmp/runclt.10.101.8.log
> --delete_old_addrs 0 --ports 3000:40000 --proxy 127.0.0.1:8080
> --------------------------------------------------------

For the record, the above is based on
        http://www.web-polygraph.org/docs/userman/repeat.html

> It works quite well, for normal comparison test. However, I still noticed
> there are some little differences between 2 tests (though there's no change
> in Polygraph setting). e.g. the Offered DHR/BHR is different,
> 59.62%/39.47% in one test, 59.91%/40.92% in another. Is it normal?

Whether the differences are normal depends on the measurement and
workload, at least. Any measurement that depends on device under test
(including network gear and Polygraph drones) may differ since devices
may behave differently under similar conditions. The example you are
giving (offered hit ratios) does not depend directly on the device
under test as long as the device does not abort transactions or
corrupt data.

Not all workloads can be repeated without differences. For example, if
your workload uses best-effort robots, then the offered request rate
and the number of submitted requests will probably differ (because
response time of the device under test will differ at least a little
bit). The same is true for workloads that use time-based phase
schedule -- it may be impossible for Polygraph to observe exactly the
same phase timings and so different number of requests may be
submitted per phase (possibly affecting offered rates and ratios).
Differences due to imperfect timings should be minor. You can compare
request counts and other low-level stats to try to find the cause for
the difference.

> There're several parameter in .pg file, such as abort_prob and
> HitIfRepeat, which seemingly would bring the difference.

Bugs notwithstanding, random distributions in Polygraph should repeat
the same sequence given the same random seed and identical PGL
configuration (which is what you are doing in your tests).

Alex.

-- 
                            | HTTP performance - Web Polygraph benchmark
www.measurement-factory.com | HTTP compliance+ - Co-Advisor test suite
                            | all of the above - PolyBox appliance