Why do some web responses not end with a newline and what's the best way to handle them?

Question

It seems like when I curl an HTTP request, sometimes I get a newline and other times I do not.

In my shell, I use the -w "\n" option to append a newline, but using that also makes me wonder if there's a way to add the newline only when the response does not end with one.

In my shell, it sometimes breaks cycling back through history and the reverse search if the response does not have a newline.

Answer 1

Messages consist of headers, an empty line, and optional body (4.1 Message Types). Some responses such as 304 Not Modified do not have a body, and they end at the first empty line. If there is a body, the message ends where Content-Length says so (but there are other cases: 4.4 Message Length). If there is a body and the message ends with a newline, the newline is part of the body and not of the request/response. If the request/response itself is prefaced or followed by additional newlines, it's a buggy implementation.