loki.source.journal - logs not preserving order when sent to loki with duplicate timestamps

[Amoolaa]

What's wrong?

Hello 👋

I am regularly experiencing cases where the microsecond precision of journal is not high enough and there are multiple log lines with the same REALTIME_TIMESTAMP appearing. When querying journal via journalctl, the log order is preserved correctly. However, receiving logs via loki.source.journal and sending to loki does not preserve this order (presumably because it's just ordering based on timestamp in loki, unlike journal). This is an issue as we are attempting to replace rsyslog with Alloy's journal component in Amazon Linux 2023.

With journal's microsecond precision vs loki's nanosecond precision, I'm thinking it may be possible to increment the REALTIME_TIMESTAMP by one nanosecond for each entry with the same timestamp to at least try and prevent collisions (in the same way that loki does it in limits_config.increment_duplicate_timestamp).

Steps to reproduce

Run a basic systemd unit to run this python script:

import time
from datetime import datetime
import sys

def collect_timestamps():
    timestamps = []
    for _ in range(5):
        current_time = datetime.now().strftime('%Y-%m-%d %H:%M:%S.%f')
        timestamps.append(current_time)
    return timestamps

def main():
    while True:
        timestamps = collect_timestamps()
        for timestamp in timestamps:
            print(timestamp, end='\n', flush=False)
        sys.stdout.flush()
        time.sleep(10) 

if __name__ == "__main__":
    main()

that flushes a bunch of timestamps to stdout every 10 seconds.

System information

Amazon Linux 2023

Software version

Grafana Alloy v1.5.1

Configuration

Logs