It depends upon your lat and lon, but figure about 10 feet. Basically earth circumference is 24,000 miles * 5280 feet per mile divided by (2**24) (bits in the sign+significand of a 32 bit float). The math works out to 7.5 feet, but one always has to pay some imprecise math tax... One can see the quantization effect with Heather V5. Heather can calculate and display a lat/lon scattergram of the receiver fixes as they come in. The default is a 10x10 grid at 3 meters per division. The lat/lon coordinates are "plotted" in an internal bit map. The coordinates are also stored as 32-bit floats in the plot queue. If one tells Heather to change the scale factor (meters/division) of the scattergram, it is re-created from the low-res plot queue data. gpsd1.gif is from the full-res double precision data the receiver sends, gpsd2.gif is from the single precision floating point plot queue data. The center points of two scattergrams are slightly different because it was changed to the current GPS coordinate when the plot was re-generated. The pixel colors change every hour. -------------------- > What does the chop-off translate to in terms of distance?

Hi At the often quoted 1.5 ns / meter error level, that would be almost 3.5 ns. At the “worst case” 3 ns / meter you would almost get to 7 ns. Bob > On Dec 7, 2016, at 7:17 PM, Mark Sims <holrum@hotmail.com> wrote: > > It depends upon your lat and lon, but figure about 10 feet. Basically earth circumference is 24,000 miles * 5280 feet per mile divided by (2**24) (bits in the sign+significand of a 32 bit float). The math works out to 7.5 feet, but one always has to pay some imprecise math tax... > > One can see the quantization effect with Heather V5. Heather can calculate and display a lat/lon scattergram of the receiver fixes as they come in. The default is a 10x10 grid at 3 meters per division. The lat/lon coordinates are "plotted" in an internal bit map. The coordinates are also stored as 32-bit floats in the plot queue. If one tells Heather to change the scale factor (meters/division) of the scattergram, it is re-created from the low-res plot queue data. > > gpsd1.gif is from the full-res double precision data the receiver sends, gpsd2.gif is from the single precision floating point plot queue data. The center points of two scattergrams are slightly different because it was changed to the current GPS coordinate when the plot was re-generated. The pixel colors change every hour. > > -------------------- > >> What does the chop-off translate to in terms of distance? > <gpsd1.gif><gpsd2.gif>_______________________________________________ > time-nuts mailing list -- time-nuts@febo.com > To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts > and follow the instructions there.