ESP8266 HTTPClient cannot be reused to connect to a second, different host

[cdzombak]

Basic Infos

• This issue complies with the issue POLICY doc.
• I have read the documentation at readthedocs and the issue is not addressed there.
• I have tested that the issue is present in current master branch (aka latest git).
• I have searched the issue tracker for a similar issue.
• If there is a stack dump, I have decoded it. n/a
• I have filled out all fields below.

Platform

• Hardware: ESP8266
• Core Version: 9d024d1
• Development Env: PlatformIO
• Operating System: macOS 11.6

Settings in IDE

• Module: Wemos D1 mini r2
• Flash Mode: unknown
• Flash Size: 4MB/1MB
• lwip Variant: unknown
• Reset Method: unknown
• Flash Frequency: unknown
• CPU Frequency: 160MHz
• Upload Using: SERIAL
• Upload Speed: 115200

Problem Description

In my program, I am trying to reuse an HTTPClient to make a second connection to a different server. If the first server sends Connection: keep-alive, the HTTPClient attempts to reuse the connection when connecting to the new server, even though it's connecting to a new host.

It seems to me like begin or beginInternal should check if the new connection will be to the same host/port, and clear _canReuse otherwise. setURL does something like that, but it is only called when handling redirects.

You can work around this by calling setReuse(false), but I don't think that should be necessary to handle this use case.

The code below demonstrates this:

MCVE Sketch

// nb. This is a minimal bug reproduction and does not reflect best practices.

#include <Arduino.h>
#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>

#define WIFI_ESSID "xxx"
#define WIFI_PASSWORD "xxx"
#define EXT_IP_URL "https://ip.dzdz.cz"
#define POST_URL "https://hookb.in/nPpMqzWewdtZ7Qrr7gYB"

WiFiClientSecure wifiClient;

void setup() {
    Serial.begin(115200);

    Serial.printf_P(PSTR("%lu: Connecting to WiFi (%s)\r\n"), millis(), WIFI_ESSID);
    WiFi.mode(WIFI_STA);
    WiFi.begin(WIFI_ESSID, WIFI_PASSWORD);
    yield();

    Serial.printf_P(PSTR("%lu: Waiting for initial WiFi connection\r\n\t"), millis());
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    Serial.printf_P(PSTR("\n%lu: Connected. My IP: %s\r\n"), millis(), WiFi.localIP().toString().c_str());

    // This is used to make this a minimal bug reproduction.
    // This is not example code.
    wifiClient.setInsecure(); // Do not copy this into your project.
}

void loop() {
    HTTPClient httpClient;
    httpClient.begin(wifiClient, EXT_IP_URL);
    Serial.printf_P(PSTR("%lu: Starting GET request to %s\r\n"), millis(), EXT_IP_URL);
    int respCode = httpClient.GET();
    String getResult = "";
    if (respCode >= 400) {
        Serial.printf_P(PSTR("%lu: HTTP Error %d\r\n"), millis(), respCode);
    } else if (respCode > 0) {
        Serial.printf_P(PSTR("%lu: HTTP %d\r\n"), millis(), respCode);
        getResult = httpClient.getString();
        Serial.printf_P(PSTR("\t%s\r\n"), getResult.c_str());
    } else {
        Serial.printf_P(PSTR("%lu: error: %s\r\n"), millis(), HTTPClient::errorToString(respCode).c_str());
    }
    httpClient.end();

//    The first request, above, succeeds; the second request, below, will fail with HTTP 404.
//    httpClient.setReuse(false); // Uncommenting this line works around the bug

    httpClient.begin(wifiClient, POST_URL);
    Serial.printf_P(PSTR("%lu: Starting POST request to %s\r\n"), millis(), POST_URL);
    respCode = httpClient.POST(getResult);
    if (respCode >= 400) {
        Serial.printf_P(PSTR("%lu: HTTP Error %d\r\n"), millis(), respCode);
    } else if (respCode > 0) {
        Serial.printf_P(PSTR("%lu: HTTP %d\r\n"), millis(), respCode);
    } else {
        Serial.printf_P(PSTR("%lu: error: %s\r\n"), millis(), HTTPClient::errorToString(respCode).c_str());
    }
    httpClient.end();
}

Debug Messages

SDK:2.2.2-dev(38a443e)/Core:3.0.2=30002000/lwIP:STABLE-2_1_2_RELEASE/glue:1.2-48-g7421258/BearSSL:6105635
70: Connecting to WiFi (dzombak)
fpm close 1
mode : sta(ec:fa:bc:04:2c:9d)
add if0
76: Waiting for initial WiFi connection
	.....scandone
state: 0 -> 2 (b0)
.state: 2 -> 3 (0)
state: 3 -> 5 (10)
add 0
aid 1
cnt

connected with dzombak, channel 1
dhcp client start...
ip:192.168.1.191,mask:255.255.255.0,gw:192.168.1.1
.
4302: Connected. My IP: 192.168.1.191
[HTTP-Client][begin] url: https://ip.dzdz.cz
[HTTP-Client][begin] host: ip.dzdz.cz port: 443 url:
4306: Starting GET request to https://ip.dzdz.cz
[HTTP-Client][sendRequest] type: 'GET' redirCount: 0
[HTTP-Client] connected to ip.dzdz.cz:443
[HTTP-Client] sending request header
-----
GET / HTTP/1.1
Host: ip.dzdz.cz
User-Agent: ESP8266HTTPClient
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Connection: keep-alive
Content-Length: 0

-----
'HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 200 OK
'HTTP-Client][handleHeaderResponse] RX: 'Server: nginx
'HTTP-Client][handleHeaderResponse] RX: 'Date: Fri, 08 Oct 2021 17:44:08 GMT
'HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/plain
'HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 13
'HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive
'HTTP-Client][handleHeaderResponse] RX: 'Strict-Transport-Security: max-age=15768000;
'HTTP-Client][handleHeaderResponse] RX: 'X-Frame-Options: DENY
'HTTP-Client][handleHeaderResponse] RX: 'X-Content-Type-Options: nosniff
'HTTP-Client][handleHeaderResponse] RX: '
[HTTP-Client][handleHeaderResponse] code: 200
[HTTP-Client][handleHeaderResponse] size: 13
5095: HTTP 200
[HTTP-Client][end] tcp keep open for reuse
	73.145.176.71
[HTTP-Client][end] tcp keep open for reuse
[HTTP-Client][begin] url: https://hookb.in/nPpMqzWewdtZ7Qrr7gYB
[HTTP-Client][begin] host: hookb.in port: 443 url: /nPpMqzWewdtZ7Qrr7gYB
5117: Starting POST request to https://hookb.in/nPpMqzWewdtZ7Qrr7gYB
[HTTP-Client][sendRequest] type: 'POST' redirCount: 0
[HTTP-Client] connect: already connected, reusing connection
[HTTP-Client] sending request header
-----
POST /nPpMqzWewdtZ7Qrr7gYB HTTP/1.1
Host: hookb.in
User-Agent: ESP8266HTTPClient
Accept-Encoding: identity;q=1,chunked;q=0.1,*;q=0
Connection: keep-alive
Content-Length: 13

-----
'HTTP-Client][handleHeaderResponse] RX: 'HTTP/1.1 404 Not Found
'HTTP-Client][handleHeaderResponse] RX: 'Server: nginx
'HTTP-Client][handleHeaderResponse] RX: 'Date: Fri, 08 Oct 2021 17:44:08 GMT
'HTTP-Client][handleHeaderResponse] RX: 'Content-Type: text/html; charset=utf-8
'HTTP-Client][handleHeaderResponse] RX: 'Content-Length: 2833
'HTTP-Client][handleHeaderResponse] RX: 'Connection: keep-alive
'HTTP-Client][handleHeaderResponse] RX: 'Vary: Accept-Encoding
'HTTP-Client][handleHeaderResponse] RX: 'ETag: "61606e2a-b11"
'HTTP-Client][handleHeaderResponse] RX: '
[HTTP-Client][handleHeaderResponse] code: 404
[HTTP-Client][handleHeaderResponse] size: 2833
5330: HTTP Error 404
[HTTP-Client][end] still data in buffer (2833), clean up.
[HTTP-Client][end] tcp keep open for reuse

[Ectalite]

Same error here, thanks for the workaround.

I tried with two HTTPClients but it didn't worked better, it worked one time but no more luck...

void sendLedInstruction(String path, String message) {
  http.begin(clientwifi, "http://192.168.1.21" + path); //Specify request destination
  int httpCode = http.POST(message);                    //Send the request
  http.end();                                            //Close connection
  http2.begin(clientwifi, "http://192.168.1.22" + path); //Specify request destination
  httpCode = http2.POST(message);                                    //Send the request
  http2.end();                                            //Close connection
}

[paulocsanz]

Does it work if you use HttpClient::setURL instead of HttpClient::begin?

HttpClient::begin could probably be fixed to ensure _canReuse is set to false to show that it's the first request to that url.

Although I'm not sure if begin could be called twice to the same URL with the expectation of connection reuse. I think it's unlikely.

[Ectalite]

Does it work if you use HttpClient::setURL instead of HttpClient::begin?

HttpClient::begin could probably be fixed to ensure _canReuse is set to false to show that it's the first request to that url.

Although I'm not sure if begin could be called twice to the same URL with the expectation of connection reuse. I think it's unlikely.

I tested it with HttpClient::setURL, but it seems to work half of the time.

Here is the function I use to to send a message to two esp8266 slaves from a master :

void sendLedInstruction(String path, String message) {
  http.begin(clientwifi, "http://192.168.1.21/" + path); //Specify request destination
  http.POST(message);                      //Send the request
  http.setURL("http://192.168.1.22/" + path);
  http.POST(message);
  http.end();                                                                     //Close connection
}

I previously used this method and it worked all the time:

void sendLedInstruction(String path, String message) {
  http.begin(clientwifi, "http://192.168.1.21" + path); //Specify request destination
  http.POST(message);                    //Send the request
  http.end();                                            //Close connection
  http.begin(clientwifi, "http://192.168.1.22" + path); //Specify request destination
  http.POST(message);                                    //Send the request
  http.end();                                            //Close connection
}

[Ectalite]

Thank you !