calculating stats with gaps in the data
I'm looking at the python AllanTools package.. does it deal with gaps in
the data series (e.g. I've got a series of phase and/or frequency
measurements, 1 per second, but there's gaps of a few seconds every 30
minutes or so)
Only Stable32 handles data gaps seamlessly. Give it a try (read the manual for details).
But also ask yourself how much gaps matter. Yes, they affect the accuracy of your y-axis sigma scale and your x-axis tau scale. A few seconds every 30 minutes is, what, a 0.1% error? That's like one pixel in a ADEV plot; not significant.
What I've done when I need a perfectly seamless data set is just interpolate for rare and obviously missing phase data points. That keeps the timescale intact. This is especially important if you plan to Fourier transform the data: under no circumstances do you want to slip a sample in that case.
/tvb
----- Original Message -----
From: "jimlux" jimlux@earthlink.net
To: "Discussion of precise time and frequency measurement" time-nuts@febo.com
Sent: Thursday, May 25, 2017 6:11 AM
Subject: [time-nuts] calculating stats with gaps in the data
I'm looking at the python AllanTools package.. does it deal with gaps in
the data series (e.g. I've got a series of phase and/or frequency
measurements, 1 per second, but there's gaps of a few seconds every 30
minutes or so)
There are 'better' ways of handling gaps when calculating ADEV and
siblings. Patrizia Tavela has a nice method: you pad out the time series,
tagging missing points with NaNs say, and then if a difference contains a
missing data point, you drop it. It works very well. I expect this is in
Stable32. I think it's implemented in allantools. It's definitely
implemented in the Matlab functions I wrote (tftools on GitHub).
Cheers
Michael
On Fri, 26 May 2017 at 12:00 am, Tom Van Baak tvb@leapsecond.com wrote:
Only Stable32 handles data gaps seamlessly. Give it a try (read the manual
for details).
But also ask yourself how much gaps matter. Yes, they affect the accuracy
of your y-axis sigma scale and your x-axis tau scale. A few seconds every
30 minutes is, what, a 0.1% error? That's like one pixel in a ADEV plot;
not significant.
What I've done when I need a perfectly seamless data set is just
interpolate for rare and obviously missing phase data points. That keeps
the timescale intact. This is especially important if you plan to Fourier
transform the data: under no circumstances do you want to slip a sample in
that case.
/tvb
----- Original Message -----
From: "jimlux" jimlux@earthlink.net
To: "Discussion of precise time and frequency measurement" <
time-nuts@febo.com>
Sent: Thursday, May 25, 2017 6:11 AM
Subject: [time-nuts] calculating stats with gaps in the data
I'm looking at the python AllanTools package.. does it deal with gaps in
the data series (e.g. I've got a series of phase and/or frequency
measurements, 1 per second, but there's gaps of a few seconds every 30
minutes or so)
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
If you do want to Fourier transform your data and you do have missing
data points, I can highly recommend the Lomb-Scargle periodogram. I put it
through its paces a while back. I took 3 sin waves of different periods and
amplitude of 1 and added them together. I used 10000 data points. Then I
added Gaussian noise of various standard deviations. And I also removed
various amounts of data up to 90%.
The LSP still found the periods with noise of sd=5 and 90% of the data
gone. With sd=10 it could still find signal with 50% of the points removed.
I have some nice plots of all this if anyone is interested.
The main disadvantage of LSP is speed. It performs as O(n^2). This was a
huge disadvantage back in the 70s when it was published, but with today's
computing power it's not a problem. Unless you have 1000000 data points,
then patience is required.
Jim Palfreyman
On 26 May 2017 at 07:12, Michael Wouters michaeljwouters@gmail.com wrote:
There are 'better' ways of handling gaps when calculating ADEV and
siblings. Patrizia Tavela has a nice method: you pad out the time series,
tagging missing points with NaNs say, and then if a difference contains a
missing data point, you drop it. It works very well. I expect this is in
Stable32. I think it's implemented in allantools. It's definitely
implemented in the Matlab functions I wrote (tftools on GitHub).
Cheers
Michael
On Fri, 26 May 2017 at 12:00 am, Tom Van Baak tvb@leapsecond.com wrote:
Only Stable32 handles data gaps seamlessly. Give it a try (read the
manual
for details).
But also ask yourself how much gaps matter. Yes, they affect the accuracy
of your y-axis sigma scale and your x-axis tau scale. A few seconds every
30 minutes is, what, a 0.1% error? That's like one pixel in a ADEV plot;
not significant.
What I've done when I need a perfectly seamless data set is just
interpolate for rare and obviously missing phase data points. That keeps
the timescale intact. This is especially important if you plan to Fourier
transform the data: under no circumstances do you want to slip a sample
in
that case.
/tvb
----- Original Message -----
From: "jimlux" jimlux@earthlink.net
To: "Discussion of precise time and frequency measurement" <
time-nuts@febo.com>
Sent: Thursday, May 25, 2017 6:11 AM
Subject: [time-nuts] calculating stats with gaps in the data
I'm looking at the python AllanTools package.. does it deal with gaps
in
the data series (e.g. I've got a series of phase and/or frequency
measurements, 1 per second, but there's gaps of a few seconds every 30
minutes or so)
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to
https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/
mailman/listinfo/time-nuts
and follow the instructions there.