[PATCH v12 09/16] iio: afe: rescale: fix accuracy for small fractional scales

From: Liam Beguin
Date: Sat Jan 08 2022 - 15:54:04 EST


The approximation caused by integer divisions can be costly on smaller
scale values since the decimal part is significant compared to the
integer part. Switch to an IIO_VAL_INT_PLUS_NANO scale type in such
cases to maintain accuracy.

Signed-off-by: Liam Beguin <liambeguin@xxxxxxxxx>
Reviewed-by: Peter Rosin <peda@xxxxxxxxxx>
---
drivers/iio/afe/iio-rescale.c | 20 +++++++++++++++++---
1 file changed, 17 insertions(+), 3 deletions(-)

diff --git a/drivers/iio/afe/iio-rescale.c b/drivers/iio/afe/iio-rescale.c
index 68eb3e341939..ed3ef994997f 100644
--- a/drivers/iio/afe/iio-rescale.c
+++ b/drivers/iio/afe/iio-rescale.c
@@ -24,7 +24,7 @@ int rescale_process_scale(struct rescale *rescale, int scale_type,
int *val, int *val2)
{
s64 tmp;
- s32 rem;
+ s32 rem, rem2;
u32 mult;
u32 neg;

@@ -43,9 +43,23 @@ int rescale_process_scale(struct rescale *rescale, int scale_type,
tmp = (s64)*val * 1000000000LL;
tmp = div_s64(tmp, rescale->denominator);
tmp *= rescale->numerator;
- tmp = div_s64(tmp, 1000000000LL);
+
+ tmp = div_s64_rem(tmp, 1000000000LL, &rem);
*val = tmp;
- return scale_type;
+
+ if (!rem)
+ return scale_type;
+
+ tmp = 1 << *val2;
+
+ rem2 = *val % (int)tmp;
+ *val = *val / (int)tmp;
+
+ *val2 = rem / (int)tmp;
+ if (rem2)
+ *val2 += div_s64((s64)rem2 * 1000000000LL, tmp);
+
+ return IIO_VAL_INT_PLUS_NANO;
case IIO_VAL_INT_PLUS_NANO:
case IIO_VAL_INT_PLUS_MICRO:
mult = scale_type == IIO_VAL_INT_PLUS_NANO ? GIGA : MEGA;
--
2.34.0