Graphs going backward in time... sometimes

[dmitrig01]

I'm not exactly sure how to reproduce this – so feel free to close as irreproducible. But: when Riemann is experiencing issues (this usually happens right before some kind of failure – which happens when the rest of the system is also experiencing failure, so way too much data is getting sent to riemann), the graphs start going backward and forward... if nothing else, here is a beautiful screenshot:

[dmitrig01]

actually – it's not only under failure scenarios that it goes backward in time (I have a feeling that screenshot is more due to riemann issues – filed riemann/riemann#554). Here is a screenshot during normal operation:

The configuration behind this graph is just

(where (service #"redis (command|multi|script) ms")
            (by :service (percentiles 1 [0.5 0.95 0.99] index)))

along with the query service =~ "redis % ms 0.99"