time-nuts@lists.febo.com

Discussion of precise time and frequency measurement

View all threads

Outlier detection and removal

OP
Ole Petter Ronningen
Sat, Aug 19, 2017 6:49 PM

Hi, all

A reasonably well known method of outlier detection in phase data is to
convert to frequency and look for outliers more than k multiples of the
Median Absolute Deviation. I believe this is how outlier detection is done
in Stable32[1].

I have implemented an approximation to this method - I do not convert to
"frequency" as such, instead I simply take differences of subsequent phase
points. Then calculate the (absolute) median of the result, remove outliers
bigger than k multiples of the median, and integrate back to phase.

The results agree with Stable32 (bar a factor of 1.48-something on k) - the
same number of outliers are identified, so I have a reasonable confidence
in my approach. I have a gut feeling that my approach is equivalent to
converting to "proper frequency" - but I thought I would ask the more (than
me) mathematically gifted members of this lists if I am committing some
grave sin in my simplistic approach?

Thanks!
Ole

[1]
http://www.stable32.com/Outliers%20in%20Time%20and%20Frequency%20Measurements.pdf

Hi, all A reasonably well known method of outlier detection in phase data is to convert to frequency and look for outliers more than k multiples of the Median Absolute Deviation. I believe this is how outlier detection is done in Stable32[1]. I have implemented an approximation to this method - I do not convert to "frequency" as such, instead I simply take differences of subsequent phase points. Then calculate the (absolute) median of the result, remove outliers bigger than k multiples of the median, and integrate back to phase. The results agree with Stable32 (bar a factor of 1.48-something on k) - the same number of outliers are identified, so I have a reasonable confidence in my approach. I have a gut feeling that my approach is equivalent to converting to "proper frequency" - but I thought I would ask the more (than me) mathematically gifted members of this lists if I am committing some grave sin in my simplistic approach? Thanks! Ole [1] http://www.stable32.com/Outliers%20in%20Time%20and%20Frequency%20Measurements.pdf