language agnostic - Statistical method to know when enough performance test iterations have been performed -

i'm doing performance/load testing of service. imagine test function this:

bytespersecond = test(filesize: 10mb, concurrency: 5) 

using this, i'll populate table of results different sizes , levels of concurrency. there other variables too, idea.

the test function spins concurrency requests , tracks throughput. rate starts off @ zero, spikes , dips until stabilises on 'true' value.

however can take while stability occur, , there lot of combinations of input evaluate.

how can test function decide when it's performed enough samples? enough, suppose mean result isn't going change beyond margin if testing continues.

i remember reading article while ago (from 1 of jsperf authors) discussed robust method, cannot find article more.

one simple method compute standard deviation on sliding window of values. there better approach?

iiuc, you're describing classic problem of estimating confidence interval of mean unknown variance. is, suppose have n results, x₁, ..., x_n, each of x_i sample process of don't know much: not mean, not variance, , not distribution's shape. required confidence interval, you'd whether n large enough that, high probability true mean within interval of mean.

(note relatively-weak conditions, central limit theorem guarantees sample mean converge normal distribution, apply directly need variance.)

so, in case, classic solution determine if n large enough, follows:

• start calculating sample mean μ = ∑_i [x_i] / n. calculate normalized sample variance s² = ∑_i [(x_i - μ)²] / (n - 1)

• depending on size of n:

  • if n > 30, confidence interval approximated μ ± z_{α / 2}(s / √(n)), where, if necessary, can find here explanation on z , α.

  • if n < 30, confidence interval approximated μ ± t_{α / 2}(s / √(n)); see again here explanation of t value, table.

• if confidence enough, stop. otherwise, increase n.