Reading logs backwards

2026-02-11

When something breaks, the instinct is to search the log for the first line that says ERROR and start there. I have wasted a lot of hours that way. The first error is usually a symptom that has already travelled a long distance from where the problem started.

Start at the end

Open the log at the last line and walk upward. You are looking for the last moment the system was doing something ordinary. Everything between that line and the bottom is the failure; everything above it is noise you do not need yet. This narrows most incidents to a window of a few dozen lines instead of a few thousand.

Timestamps are the real signal

Gaps matter more than messages. A quiet three seconds in a service that normally logs forty lines a second is louder than any stack trace. I now grep for the shape of time rather than the text:

awk '{ print $1, $2 }' app.log | uniq -c | sort -rn | head

Which is crude, and works far more often than it should.

Write logs for the person reading them backwards

This changes how I log. A line that says retrying is nearly useless; retry 3/5 for user-sync after 502 tells the backwards reader everything in one line. Include the thing being acted on, the attempt number, and the reason. The person reading it will not have the surrounding context, because the surrounding context is the part they are still looking for.