Jump to content

Precision


James115
 Share

Recommended Posts

The data capture from device monitor uses three decimal places in the CSV produced (0.001) but it will be higher internally, IDK the limit.  The precision show through EScribe or the mod screen will be far lower than the internal precision. EG: in Escribe you can set material profile data points to in 0.0001 steps, but internally it is 16 bits below the decimal point or increments of 0.0000152587890625. 
Link to comment
Share on other sites

VapingBad said:

The data capture from device monitor uses three decimal places in the CSV produced (0.001) but it will be higher internally, IDK the limit.  The precision show through EScribe or the mod screen will be far lower than the internal precision. EG: in Escribe you can set material profile data points to in 0.0001 steps, but internally it is 16 bits below the decimal point or increments of 0.0000152587890625. 

That's insanely precise. Begs the question: just how accurate is it at that precision?
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...