How to scan and catch all advertisements from a beacon without missing any

[Zimbu98]

I'm trying to scan for a beacon (using whitelist) and capture each of the advertisements from the beacon. The beacon is advertising every 500ms, and the ESP is scanning continuously. But no matter how I adjust the window/interval settings, I still get frequent missed broadcasts, i.e. the beacon broadcasts but the ESP misses it.

My sketch is:

#include <Arduino.h>
#include <NimBLEDevice.h>

NimBLEScan *pBLEScan;
NimBLEAddress a("dd:34:02:06:34:17", 1);
uint lastAdv = millis();

class MyAdvertisedDeviceCallbacks : public BLEAdvertisedDeviceCallbacks
{
    void onResult(BLEAdvertisedDevice *advertisedDevice)
    {
        Serial.printf("%d: Found %s with RSSI %d \n", millis() -lastAdv, advertisedDevice->getAddress().toString().c_str(), advertisedDevice->getRSSI());
        lastAdv = millis();
    }
};

void setup()
{
    Serial.begin(115200);
    NimBLEDevice::init("");
    NimBLEDevice::whiteListAdd(a);
    Serial.println("Scanning...");
    pBLEScan = NimBLEDevice::getScan();
    pBLEScan->setAdvertisedDeviceCallbacks(new MyAdvertisedDeviceCallbacks(), true);
    pBLEScan->setFilterPolicy(BLE_HCI_SCAN_FILT_USE_WL);
    pBLEScan->setMaxResults(0); // do not store the scan results, use callback only.
//    pBLEScan->setInterval(100); 
//    pBLEScan->setWindow(99);  
    pBLEScan->setActiveScan(false); // Set to false to stop sending scan requests
}

void loop()
{
    if(pBLEScan->isScanning() == false)
    {
        pBLEScan->start(0, nullptr, false);
    }
}

Results in the serial from the above sketch are as follows (note the bold items where I have added comments). The first number is the number of milliseconds since the last broadcast scan result:

502: Found dd:34:02:06:34:17 with RSSI -77
501: Found dd:34:02:06:34:17 with RSSI -76
502: Found dd:34:02:06:34:17 with RSSI -71
508: Found dd:34:02:06:34:17 with RSSI -77
1515: Found dd:34:02:06:34:17 with RSSI -77 (missed two broadcasts)
503: Found dd:34:02:06:34:17 with RSSI -77
1012: Found dd:34:02:06:34:17 with RSSI -78 (missed one broadcast)
1000: Found dd:34:02:06:34:17 with RSSI -76 (missed one broadcast)
508: Found dd:34:02:06:34:17 with RSSI -70
503: Found dd:34:02:06:34:17 with RSSI -77
502: Found dd:34:02:06:34:17 with RSSI -76
502: Found dd:34:02:06:34:17 with RSSI -69
501: Found dd:34:02:06:34:17 with RSSI -77
505: Found dd:34:02:06:34:17 with RSSI -76
502: Found dd:34:02:06:34:17 with RSSI -69
501: Found dd:34:02:06:34:17 with RSSI -78
500: Found dd:34:02:06:34:17 with RSSI -76
503: Found dd:34:02:06:34:17 with RSSI -70
506: Found dd:34:02:06:34:17 with RSSI -78
505: Found dd:34:02:06:34:17 with RSSI -76
502: Found dd:34:02:06:34:17 with RSSI -70
504: Found dd:34:02:06:34:17 with RSSI -79
1001: Found dd:34:02:06:34:17 with RSSI -75 (missed one broadcast)
509: Found dd:34:02:06:34:17 with RSSI -69
505: Found dd:34:02:06:34:17 with RSSI -78
502: Found dd:34:02:06:34:17 with RSSI -75
509: Found dd:34:02:06:34:17 with RSSI -70
503: Found dd:34:02:06:34:17 with RSSI -79
1515: Found dd:34:02:06:34:17 with RSSI -79 (missed two broadcasts)
504: Found dd:34:02:06:34:17 with RSSI -75
1522: Found dd:34:02:06:34:17 with RSSI -71 (missed two broadcasts)
505: Found dd:34:02:06:34:17 with RSSI -78
507: Found dd:34:02:06:34:17 with RSSI -75
508: Found dd:34:02:06:34:17 with RSSI -70
506: Found dd:34:02:06:34:17 with RSSI -79
502: Found dd:34:02:06:34:17 with RSSI -75
506: Found dd:34:02:06:34:17 with RSSI -69
503: Found dd:34:02:06:34:17 with RSSI -72
503: Found dd:34:02:06:34:17 with RSSI -76
509: Found dd:34:02:06:34:17 with RSSI -78
502: Found dd:34:02:06:34:17 with RSSI -77
506: Found dd:34:02:06:34:17 with RSSI -79
506: Found dd:34:02:06:34:17 with RSSI -76
504: Found dd:34:02:06:34:17 with RSSI -78
1010: Found dd:34:02:06:34:17 with RSSI -75 (missed one broadcast)
503: Found dd:34:02:06:34:17 with RSSI -78
1011: Found dd:34:02:06:34:17 with RSSI -75 (missed one broadcast)
505: Found dd:34:02:06:34:17 with RSSI -78
503: Found dd:34:02:06:34:17 with RSSI -78

I have played around with all different variations of interval and window in the pBLEscan settings: both commented out, 100/99, 500/490, 1000/900, 10000/9500, 50/49, 200/180, 600/560. No matter what the setting, I still get missed broadcasts (sometimes two!) as shown above.

Any idea why and how to fix this?

[h2zero]

How long does the device advertise for? It could be that the device is advertising on a different channel than the scanner at these moments, the solution to that is to make the interval and window the same and short, something like 50/50.

[h2zero]

Do you control this beacon? Perhaps there is some changes we could make to the advertising that would improve this.

[Zimbu98]

Sure, I can change the interval or tx of the beacon. I have previously tried changing to really short intervals like 200ms. It still has the same problems, just smaller numbers.

For example, when I set beacon interval at 500ms (see earlier serial examples I posted) I get scans every 500 or so ms on the esp, but then get some slow ones that take around 1000, 1500, or even 2000ms.

If I change the beacon interval to 200, I get scans every 200 or so ms on the esp, but then get some slow ones that take around 400, 600, or even 800ms.

Maybe there’s a clue in that behavior, but I can’t figure it out.

[h2zero]

Ok that's good to know. So if I'm understanding correctly you want to advertise in short bursts every 500ms. Can you share how you are using the advertisement interval?

If the device is only advertising once every 500ms it would explain the misses. It would be best to set the advertisement interval to something like 20ms for 60-100ms duration every 500ms so that more advertisements are sent each time.

[Zimbu98]

I'm using a standard iBeacon which broadcasts for only a very short period every 500ms (if the interval is set to 500ms). The length of time of the broadcast is extremely short. I don't remember the actual number but it is something like 10ms or less.

So, first broadcast starts at 0ms, ends at 10ms, then second broadcast starts at 500ms, ends at 510ms, then third broadcast starts at 1500ms, ends at 1510ms, etc. etc. That's pretty much the standard for all iBeacons. The 10ms part is not configurable.

[h2zero]

I'm not familiar with that limitation, do you have a link?

If that is the case then missing advertisements every now and again is to be expected.

[Zimbu98]

That limitation is basically the definition of iBeacons. All iBeacons function this exact same way. You can adjust the TX power higher for a stronger signal, and you can adjust the interval, but you can NOT adjust the "window" of a beacon since it is only sending its little data packet once per interval.

Like you said, the scanner has to be listening on the right channel during the tiny little burst transmission from the beacon. So you might think this is the source of the problem, but...

If you remember back a few messages earlier, I noted that when I adjust the interval of the beacon, the problem still remains that same:

"For example, when I set beacon interval at 500ms (see earlier serial examples I posted) I get scans every 500 or so ms on the esp, but then get some slow ones that take around 1000, 1500, or even 2000ms."

"If I change the beacon interval to 200, I get scans every 200 or so ms on the esp, but then get some slow ones that take around 400, 600, or even 800ms."

If the scanner is missing the little burst from the beacon, you would think that sending multiple little bursts would solve the problem. Imagine the scanner is set at 800/800 (interval/window). During the scanning window (800 x 0.625 = 500ms), the scanner should only have to see the beacon burst one time, then it would be able to report that during that 500ms window, it saw beacon xyz. No missed scans at all. (I am not sure but I think the scanner does not report the same beacon twice, so even if the beacon broadcasts 5 times during that 500ms window, the scanner would only report it once)

But that is not happening. At 500ms beacon interval, I get periodic gaps in the scan results lasting 1-2 seconds. When I adjust the beacon interval to 200ms, I get gaps in the scan results lasting 400-800ms. It's interesting that the scan gaps are a simple multiple of the beacon interval, but I can't figure out why.

By the way, the timing of the periodic gaps in scan results don't seem to show any pattern. Sometimes the gaps pop up after 40 successful scans in a row, then sometimes they pop up after 3 scans, or 8, or 28, or 1.

Side issue: Setting the scanner at 800/800 interval/window works for a while, but then the ESP freezes up after some random period of time (1 hour to maybe 4 hours). Literally stops everything, even a simple heartbeat serial print in the void loop stops. Changing it to 800/750 works, and the scanner never freezes up the ESP. Lots of discussion about this in the ESP github for a couple years now.

[h2zero]

Thanks for the info! I understand better now, I was looking at this from the iBeacon example that is in this repo that will advertise for 100ms with a short interval which would send multiple advertisements.

The fact that it stops completely for 1-2 seconds suggests that the host semaphore is possibly not being released. This would likely be due to a controller bug but would require hardware debugging to prove. It could also be a freertos issue as well, which could also explain the complete stoppage of your application.

I'll try to setup a test for this and see if I can reproduce it.

[Zimbu98]

I can post a simple sketch that I am using for this testing if you like.

I can also send a you an iBeacon that you can use for testing too. Just send me your mailing address in some confidential way. (Is there DMing on GitHub?)