James115 Posted September 11, 2016 Author Report Share Posted September 11, 2016 Hello since the yihi sx 350 j has a precision in tc of 0.01 ohm, i would like to know what is the precision of our chip. Thanks Link to comment Share on other sites More sharing options...
Karsten Posted September 11, 2016 Report Share Posted September 11, 2016 Readings in the monitor show 3 significant digits on my DNA 200 Link to comment Share on other sites More sharing options...
VapingBad Posted September 11, 2016 Report Share Posted September 11, 2016 The data capture from device monitor uses three decimal places in the CSV produced (0.001) but it will be higher internally, IDK the limit. The precision show through EScribe or the mod screen will be far lower than the internal precision. EG: in Escribe you can set material profile data points to in 0.0001 steps, but internally it is 16 bits below the decimal point or increments of 0.0000152587890625. Link to comment Share on other sites More sharing options...
Karsten Posted September 11, 2016 Report Share Posted September 11, 2016 VapingBad said:The data capture from device monitor uses three decimal places in the CSV produced (0.001) but it will be higher internally, IDK the limit. The precision show through EScribe or the mod screen will be far lower than the internal precision. EG: in Escribe you can set material profile data points to in 0.0001 steps, but internally it is 16 bits below the decimal point or increments of 0.0000152587890625. That's insanely precise. Begs the question: just how accurate is it at that precision? Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now