Description
I did this
It appears that whenever an upload is stalling (e.g. uploading from a pipe and waiting for more data) and curl isn't expecting a 100 Continue response, any response (data) being sent by the server ahead of request completion gets buffered. I have observed the buffering in the following scenarios:
- Server responds with a
100 Continue, then starts sending a response: response data is buffered. - Server ignores
Expect: 100-continueheader and starts sending a response: until the--expect100-timeoutis reached, data is written to the output immediately; after the timeout, it is buffered. - Disabling the
Expectbehaviour on the client side using--header Expect:(as documented here): response data is buffered.
In all cases, whenever curl receives input data to send, it also writes the accumulated response data to the output. Once the upload is complete, e.g. EOF on the input pipe, buffering returns to normal as well. The behaviour is identical with a terminal on stdout (as below) and with --output file --no-buffer.
To reproduce scenario 1, here's a simple HTTP/1.1 server that blindly sends a 100 Continue and then a response:
import socketserver
import time
class Handler(socketserver.BaseRequestHandler):
def handle(self):
self.request.sendall(b'HTTP/1.1 100 Continue\r\n\r\n')
time.sleep(1)
self.request.sendall(b'HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n')
for i in range(100):
d = f'{i}\n'.encode()
self.request.sendall(f'{len(d):x}\r\n'.encode() + d + b'\r\n')
time.sleep(1)
with socketserver.TCPServer(('127.0.0.1', 8000), Handler) as server:
server.serve_forever()Then,
curl -sv --upload-file - http://127.0.0.1:8000/
from a terminal will buffer until receiving an LF.
To reproduce scenarios 2 and 3, here's an even simpler HTTP/0.9 server:
import socketserver
import time
class Handler(socketserver.BaseRequestHandler):
def handle(self):
for i in range(100):
self.request.sendall(f'{i}\n'.encode())
time.sleep(1)
with socketserver.TCPServer(('127.0.0.1', 8001), Handler) as server:
server.serve_forever()For scenario 2,
curl -sv --upload-file - --http0.9 --expect100-timeout 10 http://127.0.0.1:8001/
will output the first 10 numbers immediately, then declare the end of the expect phase via * Done waiting for 100-continue on stderr, then buffer until LF on stdin.
For scenario 3,
curl -sv --upload-file - --http0.9 --header Expect: http://127.0.0.1:8001/
will buffer from the beginning. Also of note, it doesn't even output the request > PUT / HTTP/1.1 + headers either.
I expected the following
Regular buffering behaviour, i.e. line buffering when stdout is a terminal, and no buffering with --no-buffer.
curl/libcurl version
curl 8.12.1 (x86_64-pc-linux-gnu) libcurl/8.12.1 GnuTLS/3.8.9 zlib/1.3.1 brotli/1.1.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.2 libssh2/1.11.1 nghttp2/1.64.0 ngtcp2/1.11.0 nghttp3/1.8.0 librtmp/2.3 OpenLDAP/2.6.9
Release-Date: 2025-02-13, security patched: 8.12.1-3
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd
operating system
Debian sid, Linux $hostname 6.12.17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.17-1 (2025-03-01) x86_64 GNU/Linux
Why I found this
This was discovered in the process of trying to turn curl into an IRC client.