memcached error on process startup

[meezaan]

I'm running a PHP in container built on unit:php8.2. Here is the Dockerfile I'm using: https://github.com/islamic-network/php/blob/master/8/8.2/Dockerfile.nunit.

Every few seconds I'm seeing the following in my logs:

./src/libmemcached/hosts.cc:268 Assertion "memcached_server_count(ptr) * MEMCACHED_POINTS_PER_SERVER <= MEMCACHED_CONTINUUM_SIZE" failed for function "update_continuum" likely for "invalid size information bein │
│ ?0  0x7f28a903e9f6 in /lib/x86_64-linux-gnu/libmemcached.so.11(+0xb9f6) [0x7f28a903e9f6]                                                                                                                           │
│ ?1  0x7f28a9046264 in /lib/x86_64-linux-gnu/libmemcached.so.11(+0x13264) [0x7f28a9046264]                                                                                                                          │
│ #2  0x7f28a906e8da in zim_Memcached_addServers at /usr/local/lib/php/extensions/no-debug-non-zts-20220829/memcached.so                                                                                             │
│ #3  0x7f28ad296fa2 in execute_ex at /usr/local/lib/libphp.so                                                                                                                                                       │
│ #4  0x7f28ad2197ed in zend_call_function at /usr/local/lib/libphp.so                                                                                                                                               │
│ ?5  0x7f28ad130b9f in /usr/local/lib/libphp.so(+0x384b9f) [0x7f28ad130b9f]                                                                                                                                         │
│ #6  0x7f28ad2952cf in execute_ex at /usr/local/lib/libphp.so                                                                                                                                                       │
│ #7  0x7f28ad298213 in zend_execute at /usr/local/lib/libphp.so                                                                                                                                                     │
│ #8  0x7f28ad2281a8 in zend_execute_scripts at /usr/local/lib/libphp.so                                                                                                                                             │
│ #9  0x7f28ad1c27ae in php_execute_script at /usr/local/lib/libphp.so                                                                                                                                               │
│ ?10  0x7f28adcfeb73 in /usr/lib/unit/modules/php.unit.so(+0xcb73) [0x7f28adcfeb73]                                                                                                                                 │
│ ?11  0x7f28adcff501 in /usr/lib/unit/modules/php.unit.so(+0xd501) [0x7f28adcff501]                                                                                                                                 │
│ ?12  0x7f28adcfd5a2 in /usr/lib/unit/modules/php.unit.so(+0xb5a2) [0x7f28adcfd5a2]                                                                                                                                 │
│ ?13  0x7f28adcfddda in /usr/lib/unit/modules/php.unit.so(+0xbdda) [0x7f28adcfddda]                                                                                                                                 │
│ ?14  0x7f28adcfe0a8 in /usr/lib/unit/modules/php.unit.so(+0xc0a8) [0x7f28adcfe0a8]                                                                                                                                 │
│ ?15  0x7f28adcffd54 in /usr/lib/unit/modules/php.unit.so(+0xdd54) [0x7f28adcffd54]                                                                                                                                 │
│ ?16  0x55ab7749c34f in unit: "xxxxxxxxxx" application(+0x2634f) [0x55ab7749c34f]                                                                                                                              │
│ ?17  0x55ab7749d617 in unit: "xxxxxxxxxx" application(+0x27617) [0x55ab7749d617]                                                                                                                              │
│ ?18  0x55ab774a140c in unit: "xxxxxxxxxx" application(+0x2b40c) [0x55ab774a140c]                                                                                                                              │
│ ?19  0x55ab7749dcfc in unit: "xxxxxxxxxx" application(+0x27cfc) [0x55ab7749dcfc]                                                                                                                              │
│ ?20  0x55ab7749e349 in unit: "xxxxxxxxxx" application(+0x28349) [0x55ab7749e349]                                                                                                                              │
│ #21  0x55ab774a8792 in nxt_event_engine_start at unit: "xxxxxxxxxx" application                                                                                                                               │
│ ?22  0x55ab7749abad in unit: "xxxxxxxxxx" application(+0x24bad) [0x55ab7749abad]                                                                                                                              │
│ ?23  0x7f28add3324a in /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f28add3324a]                                                                                                                                  │
│ #24  0x7f28add33305 in __libc_start_main at /lib/x86_64-linux-gnu/libc.so.6                                                                                                                                        │
│ #25  0x55ab7749abf1 in _start at unit: "xxxxxxxxxx" application

Unit limits are set, so each process is killed and a new one comes up after serving 10,000 requests. I suspect this is happening each time 10,000 requests are served because I'm serving between 200 and 400 a second which is why I'm seeing it so often.

There's no connectivity problem between the PHP and memcached containers.

Still investigating, but just thought I'd report this here.

[meezaan]

Okay, I have not been able to discover the cause of this, except that it appears to be leading to higher CPU usage, and it only appears on the busy services.

And I can confirm this happens each time unit recycles a process after processing the number of requests assigned to applications.<<appname>>.limits.requests in my .unit.conf.json.

[ac000]

In the version of libmemcached I'm looking at

MEMCACHED_POINTS_PER_SERVER is 100
MEMCACHED_CONTINUUM_SIZE is MEMCACHED_POINTS_PER_SERVER * 100

./src/libmemcached/hosts.cc:268 Assertion "memcached_server_count(ptr) * MEMCACHED_POINTS_PER_SERVER <= MEMCACHED_CONTINUUM_SIZE" failed for function "update_continuum" likely for "invalid size information bein

So, this seems to indicate that the number of memcache servers that libmemcache thinks there is, is > 100.

OK, here's some docs

memcached_server_count() provides you a count of the current number of servers being used by a memcached_st structure.

memcached_server_add() pushes a single server into the memcached_st structure. This server will be placed at the end. Duplicate servers are allowed, so duplication is not checked.

I wonder if this is what you are hitting?

Each time Unit restarts the app it re-adds the same servers.

I'm not entirely sure how that would be happening (it would mean some state is being maintained between restarts).

Something you could try is adding a call to resetServerList() before you add the servers...

[meezaan]

Thanks @ac000 that appears to have helped. Will monitor the logs and keep you posted.

[meezaan]

Yeah this does solve the problem, but it does mean that Unit processes have an element of state between each restart?

[ac000]

Thanks @ac000 that appears to have helped. Will monitor the logs and keep you posted.

Cool...

Yeah this does solve the problem, but it does mean that Unit processes have an element of state between each restart?

That's the bit I'm curious about.

New application processes get fork(2)'d off of the 'prototype' process and that shouldn't have executed any PHP scripts. My current suspicion is that PHP/libmemcached is keeping some state some place...

[meezaan]

Maybe it's related to opcache. Could try and see if this happens with opcache disabled, but not sure why that would make a difference. Because if no php is executed as you say, then there is nothing cached at the prototype process level.

[ac000]

Maybe you have this enabled...

memcached.sess_persistent bool
Whether or not to re-use the memcached connections corresponding to the value(s) of session.save_path after the execution of the script ends. Don't use this if certain settings (e.g. SASL settings, sess_binary_protocol) would be overridden between requests. Default is Off.

Not that I know for sure if this could cause the issue...

[meezaan]

No, this is what I have enabled. https://github.com/islamic-network/api.aladhan.com/blob/bc8fef29420a11a4c8cd570eed2f304f2377c753/config/kipchak.memcached.php#L17.

There's no state or and there are no sessions anywhere.

But what is strange is that it does not happen the 100th time the same process executes the php code. It happens whenever a new process starts. Or maybe it does happen the first time a new unit process hits the 100th request - that's not easy to tell with the volume because we would probably hit the 100 count in a couple of hundred ms - but then I would think it would happen on every subsequent request too. But this can be tested offline quite easily, I'll see if I can do that next week.

Not sure it warrants leaving the issue open, though.