On Wed, 19 May 2004, Ittai Gilat wrote:
> I am running a PolyMix3-like workload with Polygraph 2.6.4 for
> measuring performance regressions on a web proxy.
>
> I do this by comparing the CPU utilization on the proxy in two
> different runs, assuming to get the same measurement in both.
>
> The problem I am having is that the mean response size I got in two
> runs differs by about 7%, causing CPU utilization to differ as well.
>
> Is there a way to make actual throughput the same in both runs?
There are at least two possibilities here. First, the mean may be
affected by a very large object randomly selected during one of the
runs. Second, the mean difference may reflect a deeper difference
between the tests (in offered load, proxy behavior, or both).
Have you compared response size distributions to see whether the
difference is caused by a few large objects or is more fundamental
(i.e., the shape of the distributions is different)? Reporter tool
(available in Polygraph 2.8.0) builds size histograms (CDFs) or you
can use raw data from the lx tool output.
To completely eliminate the first cause, you would probably have to
use a custom size distribution with a known "finite" maximum (not a
heavy tailed distribution). You can also use content database or patch
Polygraph to prevent large objects from being generated.
To virtually eliminate the first cause, you can force Polygraph to use
the same random seed for both tests. See
http://www.web-polygraph.org/docs/userman/repeat.html
This still allows for some limited randomness due, for example, to
timing of test phases and such.
If the difference is deeper, you would have to find the real cause and
eliminate it (if it is not the proxy itself). The obvious things to
start looking for is differences in the kinds of responses (e.g., one
test may have a lot more 200 responses to IMS requests due to clocks
being out of sync). I am sure you checked for these things already.
BTW, we are working on a "comparator" tool that compares results from
two tests to find out what measurements have changed. Using this tool
requires using development versions of Polygraph (or, at least, 2.8.0,
not sure). If you can upgrade, please let me know whether you want to
give comparator a try.
Thanks,
Alex.
-- Protocol performance, functionality, and reliability testing. Tools, services, and know-how. http://www.measurement-factory.com/
This archive was generated by hypermail 2b29 : Mon Feb 06 2006 - 12:00:27 MST