I believe the issue is arising from the understanding of "what" is being
interrupted.

When a UART or GPIO is triggering an interrupt - for the UART it's a FIFO
notice that data is available to be pulled out, for the GPIO it's
notification of a state change (and optionally, a specific state change:
high-low/low-high).

When a USB PHY trips an interrupt it's to indicate that carrier data is
ready, or for a USB MAC that burst blocks over the link are being decoded.
There is an intermediate step between the USB carrier data being
received/decoded, and being interpreted by the driver as a specific type of
data - and the time that it is made available to the host OS.  Because that
is handled in the driver, it is subject to all of the other host OS
interrupts - so timing is not deterministic for the USB driver.

The differences here are key.

A 1PPS wired to a GPIO interrupt for the specific pulse edge (rising) will
be associated with a very precise correlation to an interrupt vs system
tick value.

A UART time edge (watching the seconds value increment) will be subject to
the rate at which bytes are transferred out of the UART FIFO, then math is
done to detect the second transition edge.

A USB-to-Serial converter will first encode the bytes and status pin states
into a payload compliant with USB trafic standards, transmit that data to a
USB Host (PHY/MAC) device, which  triggers and interrupt for the driver
to decode the data back into meaningful data for the host, then forwards
virtual interrupt triggers to the tasks which are watching the states and
bytes.  Everything on the host-side is subject to system overhead, USB
driver optimization, etc. and is thus variable.  The host-side backend of
the USB transfer is where your variability is coming from IMHO.

This is why the recommendation is to use direct (not bridged/adapted)
UART+GPIO from most RPi NTP server projects.  The timing is most critical
to the 1PPS edge, so the most direct correlation possible to a tick value
in the OS is the goal.  There is nothing saying that the NMEA messages must
have such low latency however - in general they merely must arrive in time
before the next 1PPS, so USB serial is fine for that traffic.

-Tim

On Thu, Jul 9, 2020 at 9:00 AM time-nuts-request@lists.febo.com wrote:

Using GPIO with an RPi is a good direction, of course.  That wasn't my
question.  Some data may help explain.
Configuration =
RPi4 (raspbian buster)
Uputronics RPi GPS board (includes PPS) connected to GPIO pins.  This
is the time of day source for the RPi4.  (via GPSD+chrony).
Navisys USB GR701 (includes PPS and  Integrated serial->USB
conversion). Contains integrated Prolific Technology, Inc. PL2303

The observed PPS variation on the Uputronics PPS is a few microseconds.
(ppstest was used for measurements).  Using a PPS to drive NTP's computer
clock disciplining, which is in turn used to measure the same PPS makes for
a dubious circular measurement.    It is comforting though to see that the
variance is in the ~1-3 microsecond range.  One must also add Trent's
interrupt latency measurement (3 microseconds).  With the GPIO connection
the RPi time base should be within say 10 microseconds of UTC.

The USB-connected Navisys fares worse.
[image: image.png]
By the time the PPS reaches the OS there is about 1 msec of variance with
an average offset of a bit over 0.6 msec.
I suspect the 1 msec USB polling is the largest latency contributor, though
there are other sources as mentioned by Tim.
I'd like to reduce the USB polling contribution by polling at 125
microseconds as the Linux PPS folks suggest
(http://linuxpps.org/doku.php/technical_information)    Would an FTDI-based
USB convertor do the trick?

Why bother with GPS/USB?  Sometimes I use laptops.  Few laptops today
directly support PPS/serial.
I just checked with Dell and found zero laptops with native RS232.

On Fri, Jul 10, 2020 at 2:17 PM Tim S tim.strommen@gmail.com wrote:

On 7/11/20 1:30 PM, Steven Sommars wrote:

I would not expect another kind of USB to serial converter to do better.
The problem is higher up in the way that Linux handles USB devices. The
USB hardware can certainly handle higher rates (and does), but the
"software interrupts" as the event travels up the stack limits the
timing resolution.

You might want to look into one of the "real time" linux kernels or
other similar implementations - they might have "turned some of the
knobs" to improve the handling of device data.

USB device handling in Linux (and Windows, and Mac OSX) is quite
complex, if only because USB itself is quite complex in that it has to
support multiple "kinds" of devices with wildly varying properties (HID,
Mass Storage, Isochronous data, etc.) - Not to mention all the
complexities associated with hot plugging and unplugging and
"enumeration" and "power control".

You might also want to delve into the handling of USB request Blocks
(URBs) which is how Linux handles USB related events.

https://www.oreilly.com/library/view/linux-device-drivers/0596005903/ch13.html

https://www.kernel.org/doc/html/v4.12/driver-api/usb/index.html

The above document describes a variety of ways to get at USB devices in
non-standard ways, through the USB API.

Another interesting alternative might be to modify the low level
interrupt handler for your Prolific chip (or whatever you're using) -
you could probably snapshot the system timer in the IRQ. But then you're
faced with the challenge of propagating that time to where you need it,
and hopefully in a way that doesn't break Linux.

In any case, your problem is not that it's USB. Your problem is that
Linux has some compromises in how it handles USB devices that
essentially imposes a 1kHz quantization on it.

There is a reason why USB support was late in coming to Linux compared
to other devices. And there's a reason why everyone curses at serial
interfaces, particularly over USB. Their character at a time behavior
fits real well for read() and write() in most OSes, but the ioctl() call
tends to be very, very complex. And getting high speed or deterministic
behavior is always a challenge.

I feel your pain. All of my serial interfaces stopped working when I
went from Mavericks to Mojave... I finally got a SiLabs interface
working, and one instance of a FTDI device.  The Prolific PL2303, not
yet.  I was seriously contemplating making Ethernet to USB interfaces on
an Arduino, where there's no OS involved.

I have so many pieces of equipment with USB interfaces, all a bit
different, all sort of using a serial port model.

On Fri, Jul 10, 2020 at 12:17 PM Tim S tim.strommen@gmail.com wrote:

On a processor like the Ti Sitara AM3358 found in the beaglebone or
the Broadcom processors in the various RPis, it's a little more
complex than that.  Both at the hardware level and at the software
level.

The UART IP block in the SoC has a module clock supplied from the
processor clock tree.  Likely two, one to drive the logic in the UART
and another to clock the external signals.  For the AM3358, there is a
100 MHz clock for the UART and an up to 48 MHz clock for the external
signals.

If the UART FIFO has reached a trigger point or the RTS line has
changed state, the UART will only raise its interrupt line on the edge
of the module clock, e.g. 100 MHz.  A GPIO controller would be the
same way.

That interrupt line does not go straight to the processor.  It goes to
an interrupt controller, such as the ARM GIC or a proprietary
controller.  The AM3358 uses a TI controller.  This controller deals
with priority of interrupts, masking, level/edge triggering (this is
about how the interrupt line from the UART or GPIO controller to the
INTC acts, not about how a GPIO will trigger an interrupt), steering
of interrupts to multiple CPU cores, and finally raising the interrupt
line in a CPU core.  Which might not happen immediately if another
interrupt is already executing.  The interrupt controller has its own
clock domain too, running at maybe 1/2 the processor's frequency.  It
takes a non-zero and non-constant amount of time for the INTC to pass
on the interrupt.

Once that happens, the CPU automatically does some stuff to respond to
the interrupt line.  It needs to finish executing the current CPU
instruction.  They don't all take one cycle, so this might be more or
less time.  The CPU also needs to jump to the main interrupt handler,
saving the current address.

We haven't even run one single software instruction yet, but we are
finally at the interrupt handler and have some control.  But it has
some stuff it must do first.  Save registers, switch stacks, etc.
That might access memory, which might not be cached, and so there's
more delay here.

It's got to figure out what interrupt occurred from the INTC.  This is
just a read from the memory mapped register.  But that means an AXI
bus transaction.  The CPU sends a request over the SoC interconnect
addressed to the interrupt controller.  The controller sends a
response back with the data.  Maybe there is contention on the AXI bus
due to data from other AXI masters, e.g. bus mastering DMA from the
USB controller.  More non-constant delays.

At this point the kernel interrupt handler will have figured out the
interrupt is from the UART and will call the handler registered by the
UART driver, GPIO driver or maybe a PPS driver.  What I've seen
discussed seems to treat this as the starting point of an interrupt.
But it's hardly the start.  We've gone through several IP blocks,
crossed a half dozen clock domains, sent and received messages via an
on chip network, and executed hundreds of instructions.

To figure out there was an interrupt from the UART, that it was due to
RTS changing, and do something about it, it takes quite a few
instructions.  And a number of register accesses that needs to do to
the UART, usually that means from over AXI to a AXI to APB interface
than then APB to a UART (I've never seen a UART as an AXI master, they
are always on a simpler peripheral bus like APB or OCP).  This takes
time.

Figuring out a USB interrupt was due to the completion of an interrupt
transfer, and that an interrupt is present in the end point, also
takes that same stuff.  Except the USB controller might be on AXI
directly.  It's quite a bit more complex.  But there isn't really a
fundamental difference on the software level.

On Sat, Jul 11, 2020 at 7:37 PM jimlux jimlux@earthlink.net wrote:

The real time kernel primarily is about trying to get closer to making
hard real time guarantees.  To put a hard maximum on latencies.  You
can't really do that on Linux, but the real time patches made it
better.  Mostly this is about finding a place where X could not happen
in time because the CPU was doing Y instead, and Y couldn't be
interrupted.  And find a way to interrupt Y so X doesn't have to wait.

One of the biggest changes was to thread all interrupts.  So that
interrupts can be interrupted.  If you are after low interrupt
latency, this actually makes things worse.  But it makes the worst
case much better than before, at the expense of the average case.

I thought Linux had USB support quite early.  I remember making a
cable so I could test the USB 1.0 in my 430HX based computer, the
first Intel chipset to have USB.  I believe it didn't work properly,
some kind of chipset problem.  Unless it was the 430FX, it's been so
long.  Google tells me the 430FX didn't have USB.  But I recall that
Intel's first chipset with USB had a broken controller and the
Ministry of Truth changed the advertising to "it never had USB."

A check shows there was Linux USB support in 1996, while Windows
didn't get it until 1997.  There was a big redesign of the Linux USB
stack by Linus in 1999.  I remember things working quite a bit better
after that was done.

On Sonntag, 12. Juli 2020 12:35:54 CEST Trent Piepho wrote:

If you think about using an AM3358, there's zero reason to use a GPIO for PPS
input. There are much better options, like the gptimer inputs or the eCAP
engine, which runs on a 200MHz clock and is therefore able to create much more
accurate timestamps.

GPIO is especially unsuited for fast response, since they typically have
multiple sources of jitter and latency. One reason is that there's only one
interrupt line per GPIO bank, each bank grouping 32 individual i/o lines. That
means a lot of code to run to figure out the actual GPIO, plus dealing with
level vs. edge triggering on the OS level etc.

You definitely want a timestamping mechanism that is hardware supported.

BR,
Matthias

On Sun, Jul 12, 2020 at 5:08 AM Matthias Welwarsky
time-nuts@welwarsky.de wrote:

A much better way to go for sure. There would be just the latency of
the PPS vs the capture block's clock, plus anything in the logic of
that block.  All the delays that happen afterward, both just in
getting the interrupt to the CPU and also in handling the interrupt in
software before a PPS timestamp can be generated, can be subtracted
out by using the timer.

How much difference does it make?  I've actually got data on that, see
attached graph.  This is for a Cyclone V, but it's very comparable to
an AM3358, other than being dual core.  The uncorrected jitter
histogram is similar to what using a GPIO would give.  Though it uses
a custom PPS driver, which I wrote to try to do a good job at
timestamping the interrupt consistently.  The time measured here is
the difference of the PPS timestamp vs 1 second on the system clock.
This means we have:

Jitter in the Telit JN3 GPS PPS signal
Delay for PPS hardware in CycloneV to detect an edge.  I think it had
about a 200 MHz clock.
Jitter in the system clock for τ=1 sec
All the delays getting that interrupt to the CPU over AXI,
interrupting the ARM, querying the INTC, etc. that I've already gone
on about for some length.
Software time spent in kernel code (Interrupt handler, GIC driver, PPS
driver), before we get to create a timestamp.

Of course this is just the jitter in this latency.  The constant part
we don't actually know.  And this is usually about 1.5 µs, but can be
up to 100 μs.

The Corrected part of the histogram is what you'd get with a capture
mode timer.  In this case the clock was 50 MHz for that timer.  This
leaves the jitter from the first three sources: the GPS, the PPS
capture hardware, and the system clock.  And this puts us down in the
100 ns range, with a worse case of up to about 1 µs.

So that's what using a timer to capture a PPS can provide.  Two orders
of magnitude decrease in jitter.  We can also see that if one isn't
using a timer, the jitter in how long an interrupt handler takes to
generate a timestamp totally dominates the jitter from the GPS, system
clock, and a timer module.

On 12.07.2020 3:57, jimlux wrote:

I beg to differ. With correct USB convertor I achieve sub millisecond
variance (see attached screenshot fro FTDI232RL chip). I get similar
results on other computers too.

All Prolific  chips I saw claimed to be USB 2.0 Full-speed. That means
they are polled only once in 1ms and there is no way how to change it
(poll rate is selected at hardware level).

Petr Titera

Petr,

Is the variance plot based on PPS timestamps, or on NTP's smoothing of the
timestamps?

Have you measured the offset?

On Mon, Jul 13, 2020 at 10:54 AM Petr Titěra petr@titera.eu wrote: