WebSocket disconnects after hours of uptime, then enters reconnect loop with "Invalid frame header"

[Yago-Tavares]

I have a Socket.IO setup that works perfectly at first, but after several hours of uptime (sometimes ~3 hours, other times 1–2 days), the WebSocket connection suddenly drops. After that, the client tries to reconnect, but every attempt fails in a loop until I restart the server.

The browser console shows:

WebSocket connection to 'wss://test.com/socket.io/?userId=1&companyId=1&EIO=4&transport=websocket' failed: Invalid frame header

On the client, I log connection errors:

socket.io.on("reconnect_error", err => console.error("Error on trying to reconnect:", err));
socket.on("connect_error", err => console.error("Connect error:", err.message, err));
socket.on("disconnect", reason => console.warn("Socket disconnected:", reason));

But the only message I consistently get is:
Connect error: websocket error F: websocket error

After this point, the only way to recover is restarting the server.

socket.io: 4.8.1
socket.io-client: 4.8.1
@socket.io/redis-adapter: 8.3.0
redis: 5.8.2
Node (client): 16.20.2
Node (server): 20.12.2
Deployment: multiple pods behind Redis pub/sub

Server

io = new SocketIO(httpServer, {
  adapter: createAdapter(pubClient, subClient),
  cors: { origin: process.env.FRONTEND_URL },
  pingInterval: 30000,
  pingTimeout: 30000
});

Client

const socket = io(BACKEND_URL, {
  autoConnect: false,
  transports: ["websocket"],
  reconnection: true,
  query: { userId, companyId }
});

Expected behavior

Socket.IO should reconnect normally, not enter a permanent failure state until the entire server is restarted.

[darrachequesne]

Hi! Thanks for reporting the issue.

• does it fail for all clients? Or just one?
• how many clients are connected in parallel?
• is the server overloaded? Or out of memory?

Possibly related:

• WebSocket connection to failed: Invalid frame header #2557
• Invalid frame header #2690
• WebSocket connection to failed: Invalid frame header #4039

[Yago-Tavares]

• It fails for all clients.
• I initially tested with only one client, but I can connect multiple clients and the same error occurs for all of them.
• The server is running properly and is neither overloaded nor running out of memory.

Regarding the other related issues, I saw that the cause was often a mismatch between the client and server versions, but in my case both sides are using the same version: 4.8.1.

[Yago-Tavares]

Here are some new logs from the server side:

io.on("connection", async socket => {
      logger.info("Client Connected");
      logger.info("Handshake headers:", socket.request.headers);
      const { userId, companyId } = socket.handshake.query;

      if (userId && userId !== "undefined" && userId !== "null") {
        const user = await User.findByPk(`${userId}`);
        if (user) {
          user.online = true;
          await user.save();
        }
      }

      if (companyId) {
        socket.join(`notification_${companyId}`);
      }

      socket.on("disconnect", reason => {
        logger.warn(`Client disconnected. Reason: ${reason}`);
      });

      socket.on("connect_error", err => {
        logger.info(`Connect error: ${JSON.stringify(err)}`);
        console.log(`Connect error: ${err.message}`);
        console.log(err instanceof Error);
        console.log(err.message);
        console.log(err.data);
      });

      socket.on("error", err => {
        logger.error(`Socket error: ${err.message || err}`);
      });

      socket.on("reconnect_attempt", () => {
        logger.info("Client is attempting to reconnect...");
      });

      socket.on("joinChatBox", (ticketId: string) => {
        logger.info("A client joined a ticket channel");
        socket.join(ticketId);
      });

      socket.on("joinNotification", (companyId: string) => {
        logger.info(
          `A client joined notification channel for company ${companyId}`
        );
        socket.join(`notification_${companyId}`);
      });

      socket.on("joinNotificationPending", (companyId: string) => {
        logger.info(
          `A client joined notification pending channel for company ${companyId}`
        );
        socket.join(`notification_${companyId}`);
      });

      socket.on("joinTickets", (status: string) => {
        logger.info(`A client joined to ${status} tickets channel.`);
        socket.join(status);
      });

      socket.on("ping", callback => {
        callback();
      });
    });

    io.engine.on("connection_error", err => {
      console.log("connection_error");
      logger.error("connection_error")
      logger.error("Engine Socket.IO error:", {
        code: err.code,
        message: err.message,
        req: {
          headers: err.req.headers,
          url: err.req.url
        }
      });
    });

    io.engine.on("headers", (headers, req) => {
      logger.info(`Connection headers request: ${req.url}`);
    });

    setInterval(() => {
      if (!io || !io.sockets) {
        logger.error("Restart socket");
        initIO(httpServer);
        return;
      }

      logger.debug(
        `Healthcheck Socket.IO: ${io.sockets.sockets.size} clientes conectados`
      );
    }, 60000);```

When the error starts happening, I only receive the headers logs, like this:
`Connection headers request: /socket.io/?userId=1&companyId=1&EIO=4&transport=websocket`

But the client never connects — it keeps trying again and again.

[darrachequesne]

Thanks, that's really useful 👍

This means that the client actually reaches the server, but the WebSocket connection is denied, either at the Engine level (engine.io package) or at the WebSocket level underneath (ws package).

Could you please log the headers, when the error starts happening? Notably the Upgrade and Connection headers.

Also, in the "connection" event handler, please register the listeners before doing some async work (await User.findByPk(....)), else you might miss some events.

[Yago-Tavares]

I added a console log on the server side to display all headers when the error starts:

io.engine.on("headers", (headers, req) => {
      logger.info(`Tentativa de conexão recebida: ${req.url}`);
      logger.info(JSON.stringify(req.headers, null, 2));
    });

And here’s the output:

{
  "host": "dev-api.xama.pro",
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
  "accept-encoding": "gzip, deflate, br, zstd",
  "accept-language": "pt-BR,pt;q=0.9,en-US;q=0.8,en;q=0.7",
  "cache-control": "no-cache",
  "connection": "Upgrade",
  "cookie": "jrt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpZCI6MSwidG9rZW5WZXJzaW9uIjowLCJjb21wYW55SWQiOjEsImlhdCI6MTc2MTc1ODIwNSwiZXhwIjoxNzYyMzYzMDA1fQ.vaEvf4eVPf45ahd1JXfEdpqmrJFFqU3O65QjyHlqSH0",
  "origin": "https://dev.xama.pro",
  "pragma": "no-cache",
  "sec-websocket-extensions": "permessage-deflate; client_max_window_bits",
  "sec-websocket-key": "/Q+KmcotNL1LdNvZxNtC9g==",
  "sec-websocket-version": "13",
  "upgrade": "websocket",
  "x-forwarded-for": "177.84.220.66",
  "x-forwarded-host": "dev-api.xama.pro",
  "x-forwarded-port": "443",
  "x-forwarded-proto": "wss",
  "x-forwarded-server": "traefik-secondary-5f7896bbcd-xx565",
  "x-real-ip": "177.84.220.66"
}

On the client side, I added the following logs to provide more information:

socket.on("connect_error", err => {
      console.log("Socket ativo:", socket.active);
      console.error("Erro no connect:", err.message, err);
      console.log("Description error:", err.description);
      console.log("Context error:", err.context);
      console.log(JSON.stringify(err));
    });

And this is the result: