Re: [PATCH v9 0/5] rust: adds Bitmap API, ID pool and bindings

[Burak Emir]

On Tue, May 27, 2025 at 4:27 PM Yury Norov <yury.norov@xxxxxxxxx> wrote:

> So, 95% CI means z=1.96, isn't? And to me it should be, for example for
> the first line: 5.18 +- 1.96*0.32/sqrt(32) = 5.18 +- 0.11 = [5.07, 5.29].
> Can you check your math please?

Facepalm... I used scipy to get the CI, and passed alpha (5%) instead
of 1-alpha (95%) for confidence level.
I am very sorry...

> > Results for sparse bitmap:
> > +------------+------+-----------+--------------+-----------+-----------+
> > | Benchmark  | Code | Mean (ns) | Std Dev (ns) | 95% CI Lo | 95% CI Hi |
> > +------------+------+-----------+--------------+-----------+-----------+
> > | find_bit/  | C    |     22.51 |        12.34 |     22.38 |     22.65 |
> > | next_bit   | Rust |     30.53 |        20.44 |     30.30 |     30.75 |
> > +------------+------+-----------+--------------+-----------+-----------+
> > | find_zero/ | C    |      5.69 |         0.22 |      5.68 |      5.69 |
> > | next_zero  | Rust |      5.68 |         0.29 |      5.68 |      5.68 |
> > +------------+------+-----------+--------------+-----------+-----------+
>
> Your numbers look pretty weird. I wrote the test such that on a typical
> x86 system it takes milliseconds for each subtest to pass.  Here you
> report nanoseconds-scaled times. Did you divide by the number of
> iterations? If so, please mention it.

Yes, I had divided by number of iterations.
Now that you are questioning this, I have second thoughts.
I will repeat the benchmark, without dividing by iterations.

> Please print raw output of your test in patch #4 which adds the test.
> Because the test is tightly coupled to it's C version, we need to make
> sure it hast the same format - fields alignment, etc.

Will do. In v10, I have removed the module name from the output and
the lines match exactly now.

> I would prefer to have detailed performance discussion in the
> corresponding patch (#4), and here in cover letter you'd just mention
> the overall performance difference - 2%, as I can see.
>
ok.

cheers,
Burak