Duplicate dev_nonce's and data loss

[danieroux]

Duplicate dev_nonce's, for the same dev_addr and created_at leads to data being lost.

We’ve been seeing devices successfully re-joining and the data not reaching the application-server.

In these cases, there are no reported errors in device_error. Particularly: “validate dev-nonce error” is not present in these cases.

We traced it to this. Whenever we see this situation, we lose the data. This happens on the latest Chirpstack version as of the 15th of November:

dev_eui	dev_addr	created_at	dev_nonce
ffffffffffffffe	015e5a93	2021-11-18T20:58:08.489875Z	4777
ffffffffffffffe	009ed597	2021-11-18T20:58:08.493642Z	4777

This happens only to devices that can see more than one gateway, so my guess is that the JOIN does not de-duplicated?

This query found a bunch of these:

with prev_nonces as (
 select encode(dev_eui, 'hex') as dev_eui, encode(dev_addr, 'hex') as dev_addr, created_at, dev_nonce, lag(dev_nonce) over (order by dev_eui,  created_at) as previous_dev_nonce
 from device_activation
)
select *
from prev_nonces
where dev_nonce = previous_dev_nonce and dev_eui = 'ffffffffffffffe' order by created_at

[csanso-limit]

This may be a stretch but I think #557 is related to this.
I think I've experienced the same errors as you but with a single gateway, although the problem makes more sense with multiple gateways.

Duplicate dev_nonce's, for the same dev_addr and created_at leads to data being lost.

If you notice closely the created_at is not exactly the same, it's off by 3767 microseconds.

so my guess is that the JOIN does not de-duplicated?

I'm pretty sure that is exactly the problem, they are not getting de-duplicated.
The de-duplication logic probably needs a rethink, at least in the case of JOINs.
Because it does seem like JOINs are absent from being de-duplicated.
It could also be that de-duplication does not take different timestamps or frequencies into consideration.

[csanso-limit]

I am curious to know what dev_addr does the device endup receiving from the network server?
The first JOIN dev_addr, the second JOIN dev_addr or none?

[brocaar]

This has been fixed by 1b50594.