Re: Web Polygraph testing

From: Alex Rousskov (rousskov@ircache.net)
Date: Tue Dec 07 1999 - 10:14:56 MST


On Tue, 7 Dec 1999, Lanchester, Paul wrote:

> I am a technical consultant with an Internet transit carrier based in
> Europe. We are investigating the applicability of caching to our
> trans-european transit network, and the Internet connectivity services
> which we sell. We have read the report from your last bake-off and are
> looking forward to the next one in January. It is of particular interest
> to us that Inktomi declined to take part in the last one, as they are
> one of the organisations applying considerable sales pressure to our
> organisation.

Inktomi has not yet registered for the second bake-off. We would
certainly love to see them on the list of participants, but there is
little we can do to persuade Inktomi to change company policy. Our
weapons are reason and competitive attitude; for good or bad, we cannot
use "sales pressure". Your best bet is to increase your pressure on them
to come to the second bake-off and show off their product.
 
> We have looked at cache evaluation software, and have come to two
> conclusions,
>
> a) most cache manufacturers / vendors have some sort of benchmarking
> software which shows their cache to be better than everyone elses.

True. However, I think there is a trend of migrating to standard
Polygraph testing/results. Vendors do know that buyers do not trust
custom, vendor-specific workloads/tests much. Our lab has already
published Polygraph results for many caching vendors, and more are to
come.

> b) Most published Cache testing involves Ethernet networks, and focuses
> on maximum cache performance.

Agree, but that is changing as well (see below).
 
> What AUCS (my employer) intends to do is to produce a specification for
> testing which will be more typical of the Internet, that is using serial
> WAN connections at high percentage utilisations, and to look for
> reductions in utilisation and improved response time to the user upon
> implementing caching. We would then use this to test a small number of
> cache products, to determine the most suitable for our network, and use
> the resulting performance figures to justify the costs involved in the
> implementation of caching.

You may have noticed that the second bake-off does include WAN emulation
component. I believe it is now feasible to simulate a wide variety of
environments with Polygraph, including "serial WAN connections at high
percentage utilisations" that you are describing. [ This was a yet
another shameless plug for Polygraph and our approach, of course. ]

> As the effort (and costs) involved in setting up a lab such as yours are
> not inconsiderable, I would like to ask if you would consider becoming
> involved in our project, both in design, and execution of the testing in
> your lab. We would be able to obtain any necessary additional hardware
> such as Cisco routers and would of course be prepared to pay for your
> services either directly, or by means of a donation.

Our lab and expertise are available for these kind of tests, and we are
always excited about developing new interesting workloads. We will
contact you off the list for specifics.

Thanks,

Alex.



This archive was generated by hypermail 2b29 : Tue Jul 10 2001 - 12:00:10 MDT