Break submit trace into two spans and add tracing to wrapper #592
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The first (main()) does basic arg parsing, and creates an overarching tracing span for the submission. The second (submit()) does the rest. The primary motivation is to create a container tracing span for the submit steps, so they do not clutter the trace graph and provide a clear separation between submit and execution stages.
|
Discussion points: Very good, but we need the following:
|
retzkek
commented
Dec 18, 2024
| export JSB_PARENT_SPAN_ID=`echo $TRACEPARENT | cut -d'-' -f 3` | ||
| sample=`echo $TRACEPARENT | cut -d'-' -f 4` | ||
| # 8-byte random span ID e.g. adce7acee6441ec74 | ||
| export JSB_SPAN_ID=`head -c8 /dev/urandom | hexdump -e '"%02x"'` |
There was a problem hiding this comment.
Any better portable ideas to generate a span ID?
uuidgen was suggested, which appears to be available alongside hexdump, but the output would need to be munged a bit - we need 8 bytes/16 chars, so could be the last two parts* which should be unique even if uuidgen falls back to time-based (which is probably unlikely). Is that better? ¯\_(ツ)_/¯
Apptainer> for x in {1..5}; do uuidgen | tr -d '-' | cut -c '17-'; done
98ea0d9203e32de4
a670a63510d929e5
a362b70ee7c45a2c
adbdcecf9ff8fe00
a003fc48358eb47b
Apptainer> for x in {1..5}; do uuidgen -t | tr -d '-' | cut -c '17-'; done
9604001a4af11e5f
9d58001a4af11e5f
a312001a4af11e5f
a2ed001a4af11e5f
b76a001a4af11e5f
- the first 8 bytes will include the UUID version, but then maybe that's not bad really.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This breaks the initial trace into two spans: a
jobsuboverarching span and ajobsub_submitspan that contains the submit steps. This way any subsequent spans are not shown as being part of the submit, and the submit span can be collapsed without hiding them. There's still a bit of discontinuity where thejobsubspan ends with the submit and does not encapsulate downstream spans, but that would be tough to avoid and I don't think it causes any major issues, the trace otherwise appears OK in Jaeger.Further, this roughly adds a tracing span to the job wrapper (only
simple.shfor now). I anticipate some iteration here before merging and including in DAGs. Some key items: