Hi Peter,
I also have a T-Bolt monitored by LH5 but am not seeing any glitches like yours. No clear idea what may be the root cause, but from your screen dump all the metrics are affected so it is not likely to be just a failing temp sensor. Maybe something global such as power cleanliness.

"The power of accurate observation is commonly called cynicism by those who have not got it. »
George Bernard Shaw

On 12/15/2016 8:35 AM, Mike Cook wrote:

Hi Mike,

Indeed, that may be an issue. It uses some power supply from the Chinese
eBay vendor; the power supply looks to have been taken from some sort of
Cisco hardware and repurposed for the Thunderbolt.

I'll check it out later to see if anything weird is going on, but
finding odd transient issues is always fun.

Cheers!
-Pete

Hi

If you do need to troubleshoot a weird temperature issue, there are dirt cheap USB
thermometers out there. It’s a bit of a gamble on finding the one(s?) out of the batch that
works right. When you get one that does work properly, they are quite adequate for
tracking something like a 0.5 degree “bump” down.

Bob

Pete kindly sent me the raw data. Well, it's not quite "raw" in the TBolt binary TSIP sense, but a Heather-processed ascii log format. There's something very odd going on here, either with Pete's TBolt, and/or with Mark's Heather v5. I'm happy Pete has a keen eye and spotted this.

1. Attached are two plots of field 6 of the dump file, which is temperature. It shows some classic DS1620 temperature sensor glitches during the first day. This has been discussed at length in years past. The clue are readings that look like xx.75 + ~1/140 degrees. Not a problem in itself.

2. It also shows some truly erratic behavior the last day and a half, with multiple, massive, sudden temperature drops going down several degrees. I've never seen this. It's hard to know for sure, because LH is not storing the actual TBolt raw data in the log file.

3. I would have expected the lines in the log file to be paced exactly once a second. Instead there are some 50,000 cases where the gap is 0 or 2 seconds instead of 1. This is likely a LH problem and not a TBolt/TSIP problem.

4. In addition, there are many, and much larger, data gaps in the log file. The first is a 2 minute gap in data from 16:21:34 1165681294 to 16:23:33 1165681413. In fact there are over a hundred of these larger gaps, as large as 314, 414, and 525 seconds. This is very odd and I've not seen this before with TBolt or LH.

So Pete, Mark and I will talk. Something is very wrong here. We'll take this off-list and report back with the results.

Thanks,
/tvb

----- Original Message -----
From: "Pete Stephenson" pete@heypete.com
To: "Discussion of precise time and frequency measurement" time-nuts@febo.com
Sent: Wednesday, December 14, 2016 6:05 PM
Subject: [time-nuts] Temperature weirdness with Thunderbolt & Lady Heather 5

Tom wrote:

I think the more revealing trace is the DAC voltage.  There are ~70mV
plunges, to a dead quiet (at this scale), more negative value.  70mV is
huge, corresponding to a -35e-9 frequency shift (350mHz).  If the DAC
voltage actually changed that much, it would pull the OCXO so far off
frequency during these events that it would take much, much longer than
shown (indeed, much longer than the width of these events) to
re-stabilize.  Yet we see clean jumps within seconds, and no settling.
(Unfortunately, the Tbolt's estimation of frequency is not plotted on
the posted screen shot.)

I find it hard to believe that LH does much processing of the reported
DAC voltage, so I think it's safe to say (1) the LH plot shows
accurately what the Tbolt is reporting (at least WRT the DAC voltage),
and (2) the actual DAC voltage is not doing what the Tbolt is reporting.

Looks like a sick Tbolt to me.

Best regards,

Charles

On 12/15/2016 7:45 PM, Charles Steinmetz wrote:

Apologies, I did have the frequency plot turned off. I've taken some
more screenshots with the frequency plot turned on.

screenshot.png is a close-up of one of the odd spikes, while
screenshot2.png took place ~10 minutes later (I just scrolled to the right).

The small frequency jumps in screenshot2.png are due to satellites
entering and leaving the field of view. Due to the setup of my
apartment, the antenna location is decidedly sub-optimal and has a clear
view only to the northwest.

Any idea what might be the issue? I can do SMT rework to replace a bad
temperature sensor or other faulty chips, within reason (super-fine
pitch stuff is a pain).

Cheers!
-Pete

Hi

With multiple things going nuts at the same time, power would be the first
thing on my list. Second would be the serial i/o stuff. In both cases things
like connectors and cables are very much on the suspect list.

Bob

On 12/15/2016 3:05 AM, Pete Stephenson wrote:

[snip]

Hi all,

I'm pleased -- but a little frustrated -- to report that I've been
unable to reproduce the issue after a few days of trying. A screenshot
of the happy Thunderbolt as shown by Lady Heather is attached.

At first, I thought the issue might have been a dodgy USB-to-serial
dongle, so I connected the Tbolt directly to the computer's hardware
serial port (which normally is connected to another device). The issue
immediately cleared up.

However, when I later reconnected the Tbolt to the same USB-to-serial
adapter (which uses a genuine FTDI chip) the problem did not recur.

The power supply is a used, Cisco power supply provided by the Chinese
eBay vendor that sold me the Thunderbolt. My digital multimeter showed
that the voltages provided were, while not dead-on -12V, +12V, and +5V,
were within the specs required by the Thunderbolt's manual.

Additionally, I probed the power pins with my oscilloscope both while
the Thunderbolt was powered off and also when I disconnected the power
connector. Visually, the traces appeared smooth, with no visible ripple
or spikes. The FFT function on the scope showed no spikes or anything
unusual between DC and 5 MHz (I didn't bother checking beyond that).

Since the Thunderbolt had been turned off for a few months to save
electricity, I thought that perhaps the issue only occurred when the
unit was started after being cold. I unplugged it and let it cool
overnight, then started it again. Again, nothing unusual. It's working fine.

The only other thing I can think that might have caused an issue is I
had recently connected another USB device (an Arduino Pro Micro
programmed as an EDtracker: http://www.edtracker.org.uk/) that had been
previously setup to use the same COM port number as the FTDI adapter.
Although Windows should prevent (or at least warn about) such conflicts,
it did not. It's possible that the output of the EDtracker was getting
mixed in with the serial data from the Thunderbolt and causing
corruption that Lady Heather interpreted as temperature and DAC excursions.

Unfortunately, I can't seem to replicate the issue, so I'm out of ideas.

In short: everything I can think to check seems to be ok.

Thanks to all for the help and suggestions, but I think this issue is
resolved for the time being.

Cheers!
-Pete

Hi

Do you (or your assistant) have a cell phone? Where was it when the original issue occurred?

Yes, it’s a bit of a long shot, but I have seen test data messed up by the phone
in a tech’s pocket. We finally worked it out when the speaker on the test computer started chirping
as the cell phone did it’s “check into the mothership” thing.

Bob