Re: [time-nuts] Trimble Thunderbolt no longer determines the correct date
Brad,
Strictly speaking, there's no problem with GPS, or Trimble, or the Thunderbolt GPSDO: each did exactly as designed and documented. Anyone working with timing systems (from astronomy to calendars to watch making to operating systems) knows there are many subtle details. Just wait until 2038 for MOAR (the unix Mother of all Rollovers)!
Each GPS receiver manufacturer deals with dates and times and rollovers differently. Still, it sounds like the 3rd party who wrote software for the E911 or paging system forgot to read the manual on the GPS receiver they chose.
The Trimble Thunderbolts are used in many Radio Paging Systems to
synchronize their transmitters in simulcast mode.
Systems that are using the models affected by the 1024 week epoch
are currently off the air or functioning poorly.
If you are able, can you explain to us what exactly in the transmitters is the problem? All of our TBolts continue to give valid 1 PPS and 10 MHz outputs. What exactly do you mean by "synchronizing transmitters"?
Trimble's only distributor, Novotech, did not know about it and had no
inventory of the new replacement E series Thunderbolt GPS Receivers.
Trimble says they are shipping new units from their overseas factory in
about 2 weeks -- that's the best they can offer!
Is there no way for E911 people to manually set the clock on their system?
I read here on the Time Nuts messages that some are considering: "some in-line
device that re-writes the serial data as it comes out of the Thunderbolt
Right, that was an idea I mentioned. Easy to do but I don't think anyone bothered, because:
-
Mark had already fixed Heather.exe for his users, some NTP people added code to their TBolt drivers, Didier updated his LCD Monitor board -- these are all solutions that keep the same TBolt and just tweak the interpretation of the received TSIP date. Hence no need to update the TBolt or create an inline date-fixer gizmo.
-
Almost all of us use TBolts for their precise time & frequency outputs, not their TSIP packets, so rollovers don't matter.
Does anyone plan to do this? Or does anyone have any ideas for a short-term solution.
Any suggestions would be sincerely appreciated.
Presumably the E911 system runs some kind of software or operating system? Surely there's a way to have the guys who wrote the s/w put a fix in? That should be much faster and cheaper than waiting 2 weeks for new h/w, no?
Bob,
Next up is the comment that it took two weeks and $27,000 to fix.
Wow! At that price, I bet quite a few time-nuts will have a R-Pi or Arduino or PIC solution ready by the weekend. Me, I'm busy with family and eclipse and so will have to pass on the offer.
You'll want to continuously read serial, unpack TSIP (the DLE stuff), fix the 0x8F-AB message, repack into TSIP and serial output. You may even get away with a single UART since the Rx/Tx pins are both 9600 baud and full duplex. The 0x8F-AB date fix involves converting UTC ymd to #days, adding 1024, converting #days to UTC ymd, where #days is any linear day counting system that works from 1980 to 2100. Both Mark and I posted samples a while ago. It may take some work to make the inline translator code asynchronous enough to avoid data loss or excessive packet latency, though. And it's impossible to do real-time byte-for-byte translation because the DLE escapes in TSIP can slightly alter the byte count.
Adrian,
While it wouldn't be difficult to build such a device, manufacturing a decent
quantity in less than 2 weeks to beat Trimble would be a tall order.
Agreed. You know as there's a customer, a weekend project can easily turn into a 2 week project.
/tvb
Tom, et.al.
The E911 installation, in the news, is just one of several. Others are hospitals, fire stations, etc. using different dispatch systems.
In a wide-area simulcast-overlap paging system, the transmitters in the same coverage area are carefully set to all transmit at exactly the same time. Each transmitter frequency is set to be slightly different in order to control signal overlap/signal cancellation (read: a few Hertz). This has allowed simulcast paging to achieve better coverage that any other technology that I am aware of — especially in a one-to-many transmission. Some of the fine points of this system engineering are discussed here: http://braddye.com/angus_flex_at_6400.html http://braddye.com/angus_flex_at_6400.html
A simple explanation of simulcasting is here: http://braddye.com/simulcasting.html http://braddye.com/simulcasting.html
So to me "synchronizing transmitters” means the control system sends the traffic out to all the transmitters (over satellite) and tells them all to hold the messages in a buffer until “the big hand points straight up” or whatever data command the system uses. (excuse the vernacular)
The problems being experienced right now appear to be the interface of the ThunderBolt with the Zetron Model 620 simulcast controller over TSIP. The Zetron box is also called a “wireless data encoder.” https://www.zetron.com/Portals/0/PDFs/products/Spec%20Sheet%20PDFs/04-Paging/005-1232%20Model%20600-620%20Spec.pdf https://www.zetron.com/Portals/0/PDFs/products/Spec%20Sheet%20PDFs/04-Paging/005-1232%20Model%20600-620%20Spec.pdf This is beyond my pay grade since I am 75 years old and semi-retired.
It is not our goal to blame a particular piece of equipment for this problem. The facts are the 1024 roll over happened and just about nobody in the paging business knew it was coming. A solution does not need to be found before replacement hardware arrives from wherever it is manufactured—in two weeks. I am sure the operators would rather solve it with a translator box, or something, than buy a new GPSDO for every transmitter in a system — because there are many.
When I worked in the paging industry and there was a public safety emergency because a system was off the air, no one ate or slept until it was fixed. Well that may be a slight exaggeration because I have never missed many meals.
Best Regards,
Brad Dye, K9IQY (licensed 60 years)
Editor, The Wireless Messaging News
P.O. Box 266
Fairfield, IL 62837 USA
Telephone: 618-599-7869
Skype: braddye
http://www.braddye.com http://www.braddye.com/
On Aug 10, 2017, at 7:59 PM, Tom Van Baak <tvb@LeapSecond.com mailto:tvb@LeapSecond.com> wrote:
Brad,
Strictly speaking, there's no problem with GPS, or Trimble, or the Thunderbolt GPSDO: each did exactly as designed and documented. Anyone working with timing systems (from astronomy to calendars to watch making to operating systems) knows there are many subtle details. Just wait until 2038 for MOAR (the unix Mother of all Rollovers)!
Each GPS receiver manufacturer deals with dates and times and rollovers differently. Still, it sounds like the 3rd party who wrote software for the E911 or paging system forgot to read the manual on the GPS receiver they chose.
The Trimble Thunderbolts are used in many Radio Paging Systems to
synchronize their transmitters in simulcast mode.
Systems that are using the models affected by the 1024 week epoch
are currently off the air or functioning poorly.
If you are able, can you explain to us what exactly in the transmitters is the problem? All of our TBolts continue to give valid 1 PPS and 10 MHz outputs. What exactly do you mean by "synchronizing transmitters"?
Trimble's only distributor, Novotech, did not know about it and had no
inventory of the new replacement E series Thunderbolt GPS Receivers.
Trimble says they are shipping new units from their overseas factory in
about 2 weeks -- that's the best they can offer!
Is there no way for E911 people to manually set the clock on their system?
I read here on the Time Nuts messages that some are considering: "some in-line
device that re-writes the serial data as it comes out of the Thunderbolt
Right, that was an idea I mentioned. Easy to do but I don't think anyone bothered, because:
-
Mark had already fixed Heather.exe for his users, some NTP people added code to their TBolt drivers, Didier updated his LCD Monitor board -- these are all solutions that keep the same TBolt and just tweak the interpretation of the received TSIP date. Hence no need to update the TBolt or create an inline date-fixer gizmo.
-
Almost all of us use TBolts for their precise time & frequency outputs, not their TSIP packets, so rollovers don't matter.
Does anyone plan to do this? Or does anyone have any ideas for a short-term solution.
Any suggestions would be sincerely appreciated.
Presumably the E911 system runs some kind of software or operating system? Surely there's a way to have the guys who wrote the s/w put a fix in? That should be much faster and cheaper than waiting 2 weeks for new h/w, no?
Bob,
Next up is the comment that it took two weeks and $27,000 to fix.
Wow! At that price, I bet quite a few time-nuts will have a R-Pi or Arduino or PIC solution ready by the weekend. Me, I'm busy with family and eclipse and so will have to pass on the offer.
You'll want to continuously read serial, unpack TSIP (the DLE stuff), fix the 0x8F-AB message, repack into TSIP and serial output. You may even get away with a single UART since the Rx/Tx pins are both 9600 baud and full duplex. The 0x8F-AB date fix involves converting UTC ymd to #days, adding 1024, converting #days to UTC ymd, where #days is any linear day counting system that works from 1980 to 2100. Both Mark and I posted samples a while ago. It may take some work to make the inline translator code asynchronous enough to avoid data loss or excessive packet latency, though. And it's impossible to do real-time byte-for-byte translation because the DLE escapes in TSIP can slightly alter the byte count.
Adrian,
While it wouldn't be difficult to build such a device, manufacturing a decent
quantity in less than 2 weeks to beat Trimble would be a tall order.
Agreed. You know as there's a customer, a weekend project can easily turn into a 2 week project.
/tvb
time-nuts mailing list -- time-nuts@febo.com mailto:time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
The E911 installation, in the news, is just one of several. Others are hospitals,
fire stations, etc. using different dispatch systems.
Hey, at least important things like mobile phones, ISP's, Google, Amazon, FedEx and Starbucks aren't affected ;-)
In a wide-area simulcast-overlap paging system, the transmitters in the same
coverage area are carefully set to all transmit at exactly the same time.
That's fine. And very clever. But why is this "life safety" system tied to GPS, to a particular vendor, to a particular model of receiver (that clearly states in the documentation that it has a 1024 week / 19.6 year window of valid UTC times)?
So to me "synchronizing transmitters” means the control system sends the
traffic out to all the transmitters (over satellite) and tells them all to hold the
messages in a buffer until “the big hand points straight up” or whatever data
command the system uses. (excuse the vernacular)
Exactly. In most of the precise timing world the "big hand" is the "top of the second", or the so-called 1 PPS pulse. The idea is that all 1PPS agree with each other, whether from a cesium clock, or WWVB receiver, or NTP, or GPS (or any other GNSS system).
Since the paging system failed it sounds like it was synchronized to some "hand" other than 1PPS. The rare GPS rollover events tend not to disrupt the 1PPS output -- it is still perfectly aligned with UTC -- which is why almost no one else worries about the recent TBolt episode, or any other GPS receiver for that matter.
The problems being experienced right now appear to be the interface of the ThunderBolt
with the Zetron Model 620 simulcast controller over TSIP. The Zetron box is also called
a “wireless data encoder.”
Ah, ok. So do you or anyone have contact within Zetron? The easy fix would be for them to upgrade their firmware and send out a patch. Probably cheaper than supplying new receivers from Trimble. I don't know; for us, a s/w fix is easy compared to a h/w fix or a h/w swap-out. But in the real world, once technicians have driven to a remote installation, maybe there's no real difference between a s/w fix and a h/w swap.
It is not our goal to blame a particular piece of equipment for this problem.
Right, no need to blame. I think many of us would just want to pinpoint the root cause of the problem, out of engineering curiosity. By root cause I mean actual schematics or lines of source code. It's always been my hope, after every one of these widespread infrastructure events, that the actual source code or design decisions be published eventually so that we can all learn from it.
The facts are the 1024 roll over happened and just about nobody in the paging
business knew it was coming.
Ok, now you know about GPS rollovers! Fun, isn't it? Leap seconds are fun too.
When the dust settles, you may want to look into the more general topic of life safety infrastructure vs. free-from-the-sky time & frequency. These days nanosecond precise time is cheaper than water -- but it's also fragile. A lot has been written about this. It's both a wake-up call for naive vendors of products based on GPS alone and also an opportunity for vendors who know how to design and market resilient timing products.
/tvb
Nice post Tom. I was also wondering about the replacement hardware vs software patch issue. Just speculation on my part but perhaps changing the software involves an extensive QA / test process, vs swapping out a piece of hardware ? Again this is just pure speculation on my part.
Mark Spencer
On Aug 11, 2017, at 9:26 AM, Tom Van Baak tvb@LeapSecond.com wrote:
The E911 installation, in the news, is just one of several. Others are hospitals,
fire stations, etc. using different dispatch systems.
Hey, at least important things like mobile phones, ISP's, Google, Amazon, FedEx and Starbucks aren't affected ;-)
In a wide-area simulcast-overlap paging system, the transmitters in the same
coverage area are carefully set to all transmit at exactly the same time.
That's fine. And very clever. But why is this "life safety" system tied to GPS, to a particular vendor, to a particular model of receiver (that clearly states in the documentation that it has a 1024 week / 19.6 year window of valid UTC times)?
So to me "synchronizing transmitters” means the control system sends the
traffic out to all the transmitters (over satellite) and tells them all to hold the
messages in a buffer until “the big hand points straight up” or whatever data
command the system uses. (excuse the vernacular)
Exactly. In most of the precise timing world the "big hand" is the "top of the second", or the so-called 1 PPS pulse. The idea is that all 1PPS agree with each other, whether from a cesium clock, or WWVB receiver, or NTP, or GPS (or any other GNSS system).
Since the paging system failed it sounds like it was synchronized to some "hand" other than 1PPS. The rare GPS rollover events tend not to disrupt the 1PPS output -- it is still perfectly aligned with UTC -- which is why almost no one else worries about the recent TBolt episode, or any other GPS receiver for that matter.
The problems being experienced right now appear to be the interface of the ThunderBolt
with the Zetron Model 620 simulcast controller over TSIP. The Zetron box is also called
a “wireless data encoder.”
Ah, ok. So do you or anyone have contact within Zetron? The easy fix would be for them to upgrade their firmware and send out a patch. Probably cheaper than supplying new receivers from Trimble. I don't know; for us, a s/w fix is easy compared to a h/w fix or a h/w swap-out. But in the real world, once technicians have driven to a remote installation, maybe there's no real difference between a s/w fix and a h/w swap.
It is not our goal to blame a particular piece of equipment for this problem.
Right, no need to blame. I think many of us would just want to pinpoint the root cause of the problem, out of engineering curiosity. By root cause I mean actual schematics or lines of source code. It's always been my hope, after every one of these widespread infrastructure events, that the actual source code or design decisions be published eventually so that we can all learn from it.
The facts are the 1024 roll over happened and just about nobody in the paging
business knew it was coming.
Ok, now you know about GPS rollovers! Fun, isn't it? Leap seconds are fun too.
When the dust settles, you may want to look into the more general topic of life safety infrastructure vs. free-from-the-sky time & frequency. These days nanosecond precise time is cheaper than water -- but it's also fragile. A lot has been written about this. It's both a wake-up call for naive vendors of products based on GPS alone and also an opportunity for vendors who know how to design and market resilient timing products.
/tvb
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Hi
The whole “offset frequency” simulcast thing is pretty old. It most certainly pre-dates
GPS. It’s actually old enough that the first OCXO I ever designed at Motorola went into that kind
of system. The “time sync” thing came along a while after that.
There’s always been a lot of infrastructure gear that goes on chugging for a long time.
Budgets are pretty skinny for a lot of very important stuff. That doesn’t just go for the
gear, it also goes for the payroll to support them. Maybe it should not be so, but in most
of the places I’m aware of, it is.
Bob
On Aug 11, 2017, at 12:26 PM, Tom Van Baak tvb@leapsecond.com wrote:
The E911 installation, in the news, is just one of several. Others are hospitals,
fire stations, etc. using different dispatch systems.
Hey, at least important things like mobile phones, ISP's, Google, Amazon, FedEx and Starbucks aren't affected ;-)
In a wide-area simulcast-overlap paging system, the transmitters in the same
coverage area are carefully set to all transmit at exactly the same time.
That's fine. And very clever. But why is this "life safety" system tied to GPS, to a particular vendor, to a particular model of receiver (that clearly states in the
Hi
Best guess is that the “real work” on the firmware took place …errr… a bit over 19.6 years ago.
That’s a massively long time ago in terms of development tools and hardware. Simply getting
a tool suite back up and going on the “old code” would be a big task in most organizations. Ask
me about stuff I did 20 years ago and you’ll get a bit of a blank stare. That assumes you still have
me on the payroll. Dig into it from scratch with nobody having direct experience … yikes. Yes, I’ve
been involved in those sort of projects, they rarely end well ….
Bob
On Aug 11, 2017, at 12:51 PM, Mark Spencer mark@alignedsolutions.com wrote:
Nice post Tom. I was also wondering about the replacement hardware vs software patch issue. Just speculation on my part but perhaps changing the software involves an extensive QA / test process, vs swapping out a piece of hardware ? Again this is just pure speculation on my part.
Mark Spencer
On Aug 11, 2017, at 9:26 AM, Tom Van Baak tvb@LeapSecond.com wrote:
The E911 installation, in the news, is just one of several. Others are hospitals,
fire stations, etc. using different dispatch systems.
Hey, at least important things like mobile phones, ISP's, Google, Amazon, FedEx and Starbucks aren't affected ;-)
In a wide-area simulcast-overlap paging system, the transmitters in the same
coverage area are carefully set to all transmit at exactly the same time.
That's fine. And very clever. But why is this "life safety" system tied to GPS, to a particular vendor, to a particular model of receiver (that clearly states in the documentation that it has a 1024 week / 19.6 year window of valid UTC times)?
So to me "synchronizing transmitters” means the control system sends the
traffic out to all the transmitters (over satellite) and tells them all to hold the
messages in a buffer until “the big hand points straight up” or whatever data
command the system uses. (excuse the vernacular)
Exactly. In most of the precise timing world the "big hand" is the "top of the second", or the so-called 1 PPS pulse. The idea is that all 1PPS agree with each other, whether from a cesium clock, or WWVB receiver, or NTP, or GPS (or any other GNSS system).
Since the paging system failed it sounds like it was synchronized to some "hand" other than 1PPS. The rare GPS rollover events tend not to disrupt the 1PPS output -- it is still perfectly aligned with UTC -- which is why almost no one else worries about the recent TBolt episode, or any other GPS receiver for that matter.
The problems being experienced right now appear to be the interface of the ThunderBolt
with the Zetron Model 620 simulcast controller over TSIP. The Zetron box is also called
a “wireless data encoder.”
Ah, ok. So do you or anyone have contact within Zetron? The easy fix would be for them to upgrade their firmware and send out a patch. Probably cheaper than supplying new receivers from Trimble. I don't know; for us, a s/w fix is easy compared to a h/w fix or a h/w swap-out. But in the real world, once technicians have driven to a remote installation, maybe there's no real difference between a s/w fix and a h/w swap.
It is not our goal to blame a particular piece of equipment for this problem.
Right, no need to blame. I think many of us would just want to pinpoint the root cause of the problem, out of engineering curiosity. By root cause I mean actual schematics or lines of source code. It's always been my hope, after every one of these widespread infrastructure events, that the actual source code or design decisions be published eventually so that we can all learn from it.
The facts are the 1024 roll over happened and just about nobody in the paging
business knew it was coming.
Ok, now you know about GPS rollovers! Fun, isn't it? Leap seconds are fun too.
When the dust settles, you may want to look into the more general topic of life safety infrastructure vs. free-from-the-sky time & frequency. These days nanosecond precise time is cheaper than water -- but it's also fragile. A lot has been written about this. It's both a wake-up call for naive vendors of products based on GPS alone and also an opportunity for vendors who know how to design and market resilient timing products.
/tvb
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/mailman/listinfo/time-nuts
and follow the instructions there.
Agreed. Re-inventing stuff from twenty years ago was uneconomic and
possibly impossible when I was in Silicon Valley, twenty years ago.
Jeremy
On Fri, Aug 11, 2017 at 9:57 AM, Bob kb8tq kb8tq@n1k.org wrote:
Hi
Best guess is that the “real work” on the firmware took place …errr… a bit
over 19.6 years ago.
That’s a massively long time ago in terms of development tools and
hardware. Simply getting
a tool suite back up and going on the “old code” would be a big task in
most organizations. Ask
me about stuff I did 20 years ago and you’ll get a bit of a blank stare.
That assumes you still have
me on the payroll. Dig into it from scratch with nobody having direct
experience … yikes. Yes, I’ve
been involved in those sort of projects, they rarely end well ….
Bob
On Aug 11, 2017, at 12:51 PM, Mark Spencer mark@alignedsolutions.com
wrote:
Nice post Tom. I was also wondering about the replacement hardware vs
software patch issue. Just speculation on my part but perhaps changing the
software involves an extensive QA / test process, vs swapping out a piece
of hardware ? Again this is just pure speculation on my part.
Mark Spencer
On Aug 11, 2017, at 9:26 AM, Tom Van Baak tvb@LeapSecond.com wrote:
The E911 installation, in the news, is just one of several. Others are
hospitals,
fire stations, etc. using different dispatch systems.
Hey, at least important things like mobile phones, ISP's, Google,
Amazon, FedEx and Starbucks aren't affected ;-)
In a wide-area simulcast-overlap paging system, the transmitters in
the same
coverage area are carefully set to all transmit at exactly the same
time.
That's fine. And very clever. But why is this "life safety" system tied
to GPS, to a particular vendor, to a particular model of receiver (that
clearly states in the documentation that it has a 1024 week / 19.6 year
window of valid UTC times)?
So to me "synchronizing transmitters” means the control system sends
the
traffic out to all the transmitters (over satellite) and tells them
all to hold the
messages in a buffer until “the big hand points straight up” or
whatever data
command the system uses. (excuse the vernacular)
Exactly. In most of the precise timing world the "big hand" is the "top
of the second", or the so-called 1 PPS pulse. The idea is that all 1PPS
agree with each other, whether from a cesium clock, or WWVB receiver, or
NTP, or GPS (or any other GNSS system).
Since the paging system failed it sounds like it was synchronized to
some "hand" other than 1PPS. The rare GPS rollover events tend not to
disrupt the 1PPS output -- it is still perfectly aligned with UTC -- which
is why almost no one else worries about the recent TBolt episode, or any
other GPS receiver for that matter.
The problems being experienced right now appear to be the interface of
the ThunderBolt
with the Zetron Model 620 simulcast controller over TSIP. The Zetron
box is also called
a “wireless data encoder.”
Ah, ok. So do you or anyone have contact within Zetron? The easy fix
would be for them to upgrade their firmware and send out a patch. Probably
cheaper than supplying new receivers from Trimble. I don't know; for us, a
s/w fix is easy compared to a h/w fix or a h/w swap-out. But in the real
world, once technicians have driven to a remote installation, maybe there's
no real difference between a s/w fix and a h/w swap.
It is not our goal to blame a particular piece of equipment for this
problem.
Right, no need to blame. I think many of us would just want to pinpoint
the root cause of the problem, out of engineering curiosity. By root cause
I mean actual schematics or lines of source code. It's always been my hope,
after every one of these widespread infrastructure events, that the actual
source code or design decisions be published eventually so that we can all
learn from it.
The facts are the 1024 roll over happened and just about nobody in the
paging
business knew it was coming.
Ok, now you know about GPS rollovers! Fun, isn't it? Leap seconds are
fun too.
When the dust settles, you may want to look into the more general topic
of life safety infrastructure vs. free-from-the-sky time & frequency. These
days nanosecond precise time is cheaper than water -- but it's also
fragile. A lot has been written about this. It's both a wake-up call for
naive vendors of products based on GPS alone and also an opportunity for
vendors who know how to design and market resilient timing products.
/tvb
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/
mailman/listinfo/time-nuts
and follow the instructions there.
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/
mailman/listinfo/time-nuts
and follow the instructions there.
time-nuts mailing list -- time-nuts@febo.com
To unsubscribe, go to https://www.febo.com/cgi-bin/
mailman/listinfo/time-nuts
and follow the instructions there.
On 08/11/2017 06:26 PM, Tom Van Baak wrote:
The E911 installation, in the news, is just one of several. Others are hospitals,
fire stations, etc. using different dispatch systems.
Hey, at least important things like mobile phones, ISP's, Google, Amazon, FedEx and Starbucks aren't affected ;-)
In a wide-area simulcast-overlap paging system, the transmitters in the same
coverage area are carefully set to all transmit at exactly the same time.
That's fine. And very clever. But why is this "life safety" system tied to GPS, to a particular vendor, to a particular model of receiver (that clearly states in the documentation that it has a 1024 week / 19.6 year window of valid UTC times)?
Due to ignorance. This is why I have been working for over a decade to
spread the knowledge. What is "generally known" at time-nuts and PTTI
does not reach these folks. There is no common knowledge here.
So to me "synchronizing transmitters” means the control system sends the
traffic out to all the transmitters (over satellite) and tells them all to hold the
messages in a buffer until “the big hand points straight up” or whatever data
command the system uses. (excuse the vernacular)
Exactly. In most of the precise timing world the "big hand" is the "top of the second", or the so-called 1 PPS pulse. The idea is that all 1PPS agree with each other, whether from a cesium clock, or WWVB receiver, or NTP, or GPS (or any other GNSS system).
Since the paging system failed it sounds like it was synchronized to some "hand" other than 1PPS. The rare GPS rollover events tend not to disrupt the 1PPS output -- it is still perfectly aligned with UTC -- which is why almost no one else worries about the recent TBolt episode, or any other GPS receiver for that matter.
If they also extract the time, and several systems do that, then you are
out of luck thought.
There is great benefits in synchrocasting, something I have worked with
for over a decade too, but then delivering alternative to GPS dependence
on each station.
The problems being experienced right now appear to be the interface of the ThunderBolt
with the Zetron Model 620 simulcast controller over TSIP. The Zetron box is also called
a “wireless data encoder.”
Ah, ok. So do you or anyone have contact within Zetron? The easy fix would be for them to upgrade their firmware and send out a patch. Probably cheaper than supplying new receivers from Trimble. I don't know; for us, a s/w fix is easy compared to a h/w fix or a h/w swap-out. But in the real world, once technicians have driven to a remote installation, maybe there's no real difference between a s/w fix and a h/w swap.
Might work out, but to be honest, don't hold your breath. Transmitter
vendors have not the best record of understanding the system aspects and
how they can contribute. Worth a try thought. Any user of GPS should
have a workaround for the 1024 weeks wrap-around if they use the time.
Oh, there is a lack of common specification of GPS user equipment
covering stuff like this, every little market have their "unique"
requirements. It ends up that they are not comparing notes, learning
from each other and find common strategies for common problems. Most of
the uniqueness is in interfacing to their environment.
It is not our goal to blame a particular piece of equipment for this problem.
Right, no need to blame. I think many of us would just want to pinpoint the root cause of the problem, out of engineering curiosity. By root cause I mean actual schematics or lines of source code. It's always been my hope, after every one of these widespread infrastructure events, that the actual source code or design decisions be published eventually so that we can all learn from it.
No, it is treated as company secrets. We are far from the point where it
is open source and can be inspected by many eyes. You are expected to
buy a "black box" and trust the vendor to do the right thing. When the
vendor do not do the right thing, throw the box and buy a new one. Some
vendors have been black-listed in the process.
I just wished the vendors was acting better.
I had to go to Washington for this mess.
Cheers,
Magnus