8000 HTTP response during a stalled upload gets buffered when curl isn't expecting 100 Continue · Issue #16788 · curl/curl · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

HTTP response during a stalled upload gets buffered when curl isn't expecting 100 Continue #16788

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
JustAnotherArchivist opened this issue Mar 21, 2025 · 7 comments
Open

HTTP response during a stalled upload gets buffered when curl isn't expecting 100 Continue #16788

JustAnotherArchivist opened this issue Mar 21, 2025 · 7 comments
Assignees
@icing
Labels
HTTP

Comments

@JustAnotherArchivist
Copy link
Contributor
JustAnotherArchivist commented Mar 21, 2025 •
edited
Loading

I did this

It appears that whenever an upload is stalling (e.g. uploading from a pipe and waiting for more data) and curl isn't expecting a 100 Continue response, any response (data) being sent by the server ahead of request completion gets buffered. I have observed the buffering in the following scenarios:

  1. Server responds with a 100 Continue, then starts sending a response: response data is buffered.
  2. Server ignores Expect: 100-continue header and starts sending a response: until the --expect100-timeout is reached, data is written to the output immediately; after the timeout, it is buffered.
  3. Disabling the Expect behaviour on the client side using --header Expect: (as documented here): response data is buffered.

In all cases, whenever curl receives input data to send, it also writes the accumulated response data to the output. Once the upload is complete, e.g. EOF on the input pipe, buffering returns to normal as well. The behaviour is identical with a terminal on stdout (as below) and with --output file --no-buffer.

To reproduce scenario 1, here's a simple HTTP/1.1 server that blindly sends a 100 Continue and then a response:

import socketserver
import time

class Handler(socketserver.BaseRequestHandler):
	def handle(self):
		self.request.sendall(b'HTTP/1.1 100 Continue\r\n\r\n')
		time.sleep(1)
		self.request.sendall(b'HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n')
		for i in range(100):
			d = f'{i}\n'.encode()
			self.request.sendall(f'{len(d):x}\r\n'.encode() + d + b'\r\n')
			time.sleep(1)

with socketserver.TCPServer(('127.0.0.1', 8000), Handler) as server:
	server.serve_forever()

Then,

curl -sv --upload-file - http://127.0.0.1:8000/

from a terminal will buffer until receiving an LF.

To reproduce scenarios 2 and 3, here's an even simpler HTTP/0.9 server:

import socketserver
import time

class Handler(socketserver.BaseRequestHandler):
	def handle(self):
		for i in range(100):
			self.request.sendall(f'{i}\n'.encode())
			time.sleep(1)

with socketserver.TCPServer(('127.0.0.1', 8001), Handler) as server:
	server.serve_forever()

For scenario 2,

curl -sv --upload-file - --http0.9 --expect100-timeout 10 http://127.0.0.1:8001/

will output the first 10 numbers immediately, then declare the end of the expect phase via * Done waiting for 100-continue on stderr, then buffer until LF on stdin.

For scenario 3,

curl -sv --upload-file - --http0.9 --header Expect: http://127.0.0.1:8001/

will buffer from the beginning. Also of note, it doesn't even output the request > PUT / HTTP/1.1 + headers either.

I expected the following

Regular buffering behaviour, i.e. line buffering when stdout is a terminal, and no buffering with --no-buffer.

curl/libcurl version

curl 8.12.1 (x86_64-pc-linux-gnu) libcurl/8.12.1 GnuTLS/3.8.9 zlib/1.3.1 brotli/1.1.0 zstd/1.5.7 libidn2/2.3.8 libpsl/0.21.2 libssh2/1.11.1 nghttp2/1.64.0 ngtcp2/1.11.0 nghttp3/1.8.0 librtmp/2.3 OpenLDAP/2.6.9
Release-Date: 2025-02-13, security patched: 8.12.1-3
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli GSS-API HSTS HTTP2 HTTP3 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM PSL SPNEGO SSL threadsafe TLS-SRP UnixSockets zstd

operating system

Debian sid, Linux $hostname 6.12.17-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.17-1 (2025-03-01) x86_64 GNU/Linux

Why I found this

This was discovered in the process of trying to turn curl into an IRC client.
@bagder bagder added the HTTP label Mar 23, 2025
@icing
Copy link
Contributor
icing commented Mar 25, 2025

I am not sure I understand the problem you are stating. Curl does an upload from stdin, the server sends - before the upload is done - a 200 response. Curl continues the upload.

What do you think is wrong with curl's behaviour here?

@icing icing self-assigned this Mar 25, 2025
@JustAnotherArchivist
Copy link 8000
Contributor Author

I think the received parts of the response body should be written to the output as they're being received, i.e. while the upload is still ongoing. Especially so when --no-buffer is explicitly specified, but also when output buffering is disabled automatically by curl (which appears to be what's happening when stdout is a terminal).

@icing
Copy link
Contributor
icing commented Apr 2, 2025

@JustAnotherArchivist in my tests with uploading a file and a server that echoes that back right away, curl writes out the response while it is still uploading the file.

The difference to your example is that the upload is from a file and not stdin. When curl reads from stdin, it will do so blocking. That prevents any response from the server being processed when stdin has no data available.

Does that explain what you are seeing?

@JustAnotherArchivist
Copy link
Contributor Author
JustAnotherArchivist commented Apr 6, 2025 •
edited
Loading

@icing Hmm, I see. It at least shows that there are more cases to consider. But I don't think it's the full explanation; curl does write out the response while waiting for the 100 response, even when stdin has no data available, like in scenario 2 above...

Edit: Actually, I was probably half-asleep when I wrote this; as I mentioned in the original writeup, file uploads where data is available continuously do indeed not cause this issue. It might be interesting to test this with a setup where the reads can be delayed, perhaps with a networked file system.

@bagder
Copy link
Member
bagder commented Apr 6, 2025 •
edited
Loading

When curl reads from stdin, it will do so blocking

Only for -d@- not for PUT (-T -).

This "buffering" you mention, do you mean the fact that the response headers are not shown immediately?

@bagder
Copy link
Member
bagder commented Apr 6, 2025

curl -svvv --upload-file - http://127.0.0.1:8000/

on the scenario 1 shows this:

13:30:36.605176 [0-x] == Info: [READ] client_reset, clear readers
13:30:36.605208 [0-0] == Info: [SETUP] added
13:30:36.605226 [0-0] == Info:   Trying 127.0.0.1:8000...
13:30:36.605280 [0-0] == Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=0
13:30:36.605300 [0-0] == Info: [SETUP] Curl_conn_connect(block=0) -> 0, done=1
13:30:36.605317 [0-0] == Info: Connected to 127.0.0.1 (127.0.0.1) port 8000
13:30:36.605334 [0-0] == Info: using HTTP/1.x
13:30:36.605346 [0-0] == Info: [READ] add fread reader, len=-1 -> 0
13:30:36.605387 [0-0] == Info: [READ] client_read(len=65408) -> 0, nread=0, eos=0
13:30:36.605415 [0-0] => Send header, 128 bytes (0x80)
0000: PUT / HTTP/1.1
0010: Host: 127.0.0.1:8000
0026: User-Agent: curl/8.13.0
003f: Accept: */*
004c: Transfer-Encoding: chunked
0068: Expect: 100-continue
007e: 
13:30:36.605481 [0-0] <= Recv header, 23 bytes (0x17)
0000: HTTP/1.1 100 Continue
13:30:36.605515 [0-0] == Info: [WRITE] [OUT] wrote 23 header bytes -> 23
13:30:36.605543 [0-0] == Info: [WRITE] [PAUSE] writing 23/23 bytes of type 2c -> 0
13:30:36.605578 [0-0] == Info: [WRITE] download_write header(type=2c, blen=23) -> 0
13:30:36.605624 [0-0] == Info: [WRITE] client_write(type=2c, len=23) -> 0
13:30:36.605662 [0-0] <= Recv header, 2 bytes (0x2)
0000: 
13:30:36.605694 [0-0] == Info: [WRITE] header_collect pushed(type=8, len=2) -> 0
13:30:36.605740 [0-0] == Info: [WRITE] [OUT] wrote 2 header bytes -> 2
13:30:36.605779 [0-0] == Info: [WRITE] [PAUSE] writing 2/2 bytes of type 24 -> 0
13:30:36.605825 [0-0] == Info: [WRITE] download_write header(type=24, blen=2) -> 0
13:30:36.605868 [0-0] == Info: [WRITE] client_write(type=24, len=2) -> 0
13:30:36.605909 [0-0] == Info: [WRITE] xfer_write_resp(len=25, eos=0) -> 0

Until I press return, as then it unsticks:

0000: 2
0003: 0.
0007: 2
000a: 1.
000e: 2
0011: 2.
0015: 2
0018: 3.
001c: 2
001f: 4.
0023: 2
...
13:31:08.029392 [0-0] == Info: [WRITE] http_chunked, chunk start of 2 bytes
0
13:31:08.029429 [0-0] == Info: [WRITE] [OUT] wrote 2 body bytes -> 2
13:31:08.029464 [0-0] == Info: [WRITE] [PAUSE] writing 2/2 bytes of type 1 -> 0
13:31:08.029505 [0-0] == Info: [WRITE] download_write body(type=1, blen=2) -> 0
13:31:08.029544 [0-0] == Info: [WRITE] http_chunked, write 2 body bytes, 0 bytes in chunk remain
13:31:08.029595 [0-0] == Info: [WRITE] http_chunked, chunk start of 2 bytes
1
13:31:08.029638 [0-0] == Info: [WRITE] [OUT] wrote 2 body bytes -> 2
13:31:08.029671 [0-0] == Info: [WRITE] [PAUSE] writing 2/2 bytes of type 1 -> 0
13:31:08.029711 [0-0] == Info: [WRITE] download_write body(type=1, blen=2) -> 0

@bagder
Copy link
Member
bagder commented Apr 6, 2025

Something is weird here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@icing icing

Labels
HTTP
Milestone
No milestone
Development

No branches or pull requests
3 participants
@icing @bagder @JustAnotherArchivist
0