timestamping sended messages using msg_control

[GreenFatGuy]

Hello!

I have a server with Solarflare Communications XtremeScale SFC9250 NIC. I am also using Onload 8.1.2.26.
My application is based on your example src/tests/onload/hwtimestamping/tx_timestamping.c. It opens a socket, sends packets, and tries to receive timestamps of each packet sent via reading MSG_ERRQUEUE.
I run this application like this: sudo EF_TX_TIMESTAMPING=1 onload --profile=latency ./sendmsg_app <app_options>

I've noticed that I manage to retrieve timestamps when flags are set via setsockopt:

int optval = SOF_TIMESTAMPING_TX_HARDWARE  |
                  SOF_TIMESTAMPING_RAW_HARDWARE |
                  SOF_TIMESTAMPING_SYS_HARDWARE |
                  SOF_TIMESTAMPING_OPT_TSONLY |
                  SOF_TIMESTAMPING_OPT_ID;

setsockopt(sockfd, SOL_SOCKET, SO_TIMESTAMPING, &optval, sizeof(optval));

However, I don't get any timestamps when I set the flag SOF_TIMESTAMPING_TX_HARDWARE per message via control_msg:

int optval = SOF_TIMESTAMPING_RAW_HARDWARE |
                  SOF_TIMESTAMPING_SYS_HARDWARE |
                  SOF_TIMESTAMPING_OPT_TSONLY |
                  SOF_TIMESTAMPING_OPT_ID;

setsockopt(sockfd, SOL_SOCKET, SO_TIMESTAMPING, &optval, sizeof(optval));
...
struct msghdr msg;
memset(&msg, 0, sizeof(msg));
...
char control[CMSG_SPACE(unsigned int)];
msg.msg_control = control;
msg.msg_controllen = sizeof(control);

struct cmsghdr* msghdr = CMSG_FIRSTHDR(&msg);
msghdr->cmsg_len = CMSG_LEN(sizeof(uint32_t));
msghdr->cmsg_level = SOL_SOCKET;
msghdr->cmsg_type = SO_TIMESTAMPING;
*(unsigned int*)(CMSG_DATA(msghdr)) = SOF_TIMESTAMPING_TX_HARDWARE; 
...
sendmsg(sockfd, &msg, 0);

If I try recvmsg(sockfd, &some_msg, MSG_ERRQUEUE) afterward, I constantly get EAGAIN.

This setup works without onload: I've tested it on my local machine. It is not capable of reporting hardware timestamps, but I get software ones just fine. Also, the possibility of timestamping per msg is written in Linux documentation (section 1.3.4)

I must say that I did not find anywhere that onload can do so. However, I didn't see anything saying the opposite. I am sorry if I missed this information somewhere.

The reason why I am interested in setting up a timestamping option per message rather than per socket is that I don't want to experience any overhead for every message. As far as I know, if we talking about a standard Linux network stack, there will be some overhead if we timestamp packets. By sampling messages that are being timestamped, I am trying to lower it.
So here are two main questions from me:

1. Can you tell if there is some significant overhead of timestamping every packet I send? If yes, can you provide me with any approximate number what slowdown I can expect?
2. If there is an option of timestamping only some messages using onload, could you please tell me how to do it?

[jfeather-amd]

Hi @GreenFatGuy,

Thanks for raising this issue!

1. Can you tell if there is some significant overhead of timestamping every packet I send? If yes, can you provide me with any approximate number what slowdown I can expect?

Unfortunately, I don't have any performance metrics to hand for this specific case, comparing timestamped sends to non-timestamped sends. However, I expect the answer to this question would depend greatly on the design of your application. Particularly, I would think that the main overhead of timestamping is likely to come from polling the error queue of the socket you're sending over, rather than during the actual sending of the packet itself. It might be worth having a go at testing this with your application, both timestamped and not, to see what the difference is for your use case. To better understand any overhead, it might be interesting to measure:

• the total time the application runs for;
• the time taken to configure timestamping via sockopts;
• the average time for (bursts of) sends; and
• the average time to poll the error queue.

2. If there is an option of timestamping only some messages using onload, could you please tell me how to do it?

I've had a look through the code and, indeed, it appears that onload does not support requesting timestamping via control messages (see ci_ip_cmsg_send for the currently understood control messages on transmit, linked version is master). I have raised this internally under ON-15846 as an enhancement, and expect that it's a good candidate to be included.

Hope this helps!

[GreenFatGuy]

Hi @jfeather-amd !

Thank you for your answers, that brings some light! Awesome that this feature might appear in future releases!

Considering measurements on my side: I understand that the main part of the overhead is caused due to error queue polling. I was speaking mainly about overhead inside the onload stack, which I am not in control of. I would expect that there is some because with timestamping on there is some additional work happening: writing timestamps, queuing messages with info back to the error queue, etc.

I will try to measure this aspect as far as I can. If I get some robust results that I will be sure about - I will try to bring them back as I think this might be useful for some people to know.