Precision

[James115]

Hello since the yihi sx 350 j has a precision in tc of 0.01 ohm, i would like to know what is the precision of our chip. Thanks

[Karsten]

Readings in the monitor show 3 significant digits on my DNA 200

[VapingBad]

The data capture from device monitor uses three decimal places in the CSV produced (0.001) but it will be higher internally, IDK the limit.  The precision show through EScribe or the mod screen will be far lower than the internal precision. EG: in Escribe you can set material profile data points to in 0.0001 steps, but internally it is 16 bits below the decimal point or increments of 0.0000152587890625. 

[Karsten]

VapingBad said:

The data capture from device monitor uses three decimal places in the CSV produced (0.001) but it will be higher internally, IDK the limit.  The precision show through EScribe or the mod screen will be far lower than the internal precision. EG: in Escribe you can set material profile data points to in 0.0001 steps, but internally it is 16 bits below the decimal point or increments of 0.0000152587890625. 

That's insanely precise. Begs the question: just how accurate is it at that precision?