Skill 07 · Clickhouse Js Node Troubleshooting
Subchapter 7.8
reference/socket-hangup.mdMarkdown10 KBView on GitHub
Symptom: socket hang up or ECONNRESET errors, often intermittent.
Root cause: The server or load balancer closes the Keep-Alive connection before the client detects it and stops reusing the socket.
Quick triage:
Requires:
>= 0.2.0(logging support withlog.levelconfig option). In>= 1.18.1, the default log level changed fromOFFtoWARN, so this step may already be active. In>= 1.18.2, the client auto-emits a WARN log with Keep-Alive troubleshooting hints when anECONNRESETis detected. In>= 1.12.0, a warning is logged when a socket is closed without fully consuming the stream.
import { createClient, ClickHouseLogLevel } from "@clickhouse/client";
const client = createClient({
log: { level: ClickHouseLogLevel.WARN },
});Look for log lines about unconsumed or dangling streams — these are a common hidden cause. A dangling stream is a query response stream that was never fully consumed or explicitly closed with ResultSet.close(). Because the Node.js client reuses sockets (Keep-Alive), leaving a stream open corrupts the socket and causes the next request to fail with ECONNRESET. Errors on every request strongly suggest dangling streams rather than a Keep-Alive timeout mismatch.
Common dangling stream patterns:
// ❌ Wrong — result stream never consumed; socket is left open
const resultSet = await client.query({ query: "SELECT ..." });
// result is abandoned without calling .json(), .text(), .stream(), or .close()
// ❌ Wrong — stream created but not fully piped/iterated
const resultSet = await client.query({
query: "SELECT ...",
format: "JSONEachRow",
});
const stream = resultSet.stream();
// stream is never iterated and resultSet is never closed
// ✓ Correct — consume via .json()
const resultSet = await client.query({ query: "SELECT ..." });
const data = await resultSet.json();
// ✓ Correct — consume via async iteration
const resultSet = await client.query({
query: "SELECT ...",
format: "JSONEachRow",
});
for await (const rows of resultSet.stream()) {
// process rows
}
// ✓ Correct — explicitly close; this destroys the underlying socket immediately
const resultSet = await client.query({ query: "SELECT ..." });
resultSet.close();Add the no-floating-promises (opens in a new tab) ESLint rule. Unhandled promises leave streams dangling, which can cause the server to close the socket.
Even with await, if the returned ResultSet is not consumed (no .json(), .text(), .close(), or full stream iteration), the socket is left open. The ESLint rule catches the promise case; code review is needed for the “awaited but unconsumed result” case.
curl -v --data-binary "SELECT 1" <your_clickhouse_url>Check the response headers:
< Connection: Keep-Alive
< Keep-Alive: timeout=10Requires:
>= 0.3.0(keep_alive.idle_socket_ttlwas introduced in 0.3.0 with a default of 2500 ms, replacing the olderkeep_alive.socket_ttlfrom 0.1.1 which was removed in 0.3.0).
The default idle_socket_ttl in the client is 2500 ms, which is safe for servers with a 3 s timeout (common in ClickHouse < 23.11). If your server has a higher timeout (e.g., 10 s), you can safely increase:
const client = createClient({
keep_alive: {
idle_socket_ttl: 9000, // stay ~500ms below the server's timeout
},
});⚠️ If you still get errors after increasing, lower the value, not raise it.
Tip (
>= 1.18.3): Enablekeep_alive.eagerly_destroy_stale_sockets: trueto proactively destroy sockets that have been idle longer thanidle_socket_ttlbefore each request. This helps when event loop delays prevent the idle timeout callback from firing on time.
Requires:
>= 1.0.0(request_timeoutdefault was fixed to 30 000 ms in 0.3.0;url-based configuration includingrequest_timeoutvia URL params available since 1.0.0).
Load balancers may close idle connections mid-query. Force periodic progress headers:
const client = createClient({
request_timeout: 400_000, // e.g. 400s for long queries
clickhouse_settings: {
send_progress_in_http_headers: 1,
http_headers_progress_interval_ms: "110000", // string — UInt64 type; set ~10s below LB idle timeout
},
});Node.js defaults to a total received HTTP header limit of approximately 16 KB (this can be increased via the --max-http-header-size CLI flag1Note 1Since >= 1.18.5, the ClickHouse JS Node client forwards a per-client header limit via the max_response_headers_size (bytes) option on createClient (it maps to Node’s http(s).request({ maxHeaderSize })). On older versions, the practical workarounds are the --max-http-header-size CLI flag / NODE_OPTIONS (process-wide) or supplying a custom http.Agent configured with maxHeaderSize. Back to reference 1). ClickHouse sends a new progress header with each interval (~200 bytes), and after ~75 progress headers accumulate, Node.js will throw an exception and terminate the request unless that limit is raised.
Maximum safe query duration formula:
Max duration (seconds) ≈ http_headers_progress_interval_ms × 75 ÷ 1000Examples:
http_headers_progress_interval_ms: '10000' (10s) → ~12.5 minutes max safe durationhttp_headers_progress_interval_ms: '60000' (60s) → ~75 minutes max safe durationhttp_headers_progress_interval_ms: '120000' (120s) → ~2.5 hours max safe durationNote:
http_headers_progress_interval_msis aUInt64ClickHouse setting, so it must be passed as a string (e.g.,'10000').
Raising the Node.js header limit (e.g., to 64 KB):
If you need a longer max safe duration without lengthening the progress interval, raise Node’s HTTP header limit. For example, increasing it from the default 16 KB to 64 KB quadruples the max safe duration (≈300 progress headers instead of ≈75).
// Option 1 (recommended, since `>= 1.18.5`) — per-client, no process-wide flag needed
const client = createClient({
request_timeout: 400_000,
max_response_headers_size: 65536, // 64 KB; lifts the per-request header cap
clickhouse_settings: {
send_progress_in_http_headers: 1,
http_headers_progress_interval_ms: "110000",
},
});# Option 2 — CLI flag when launching your app (process-wide; older client versions)
node --max-http-header-size=65536 app.js
# Option 3 — environment variable (works with any Node entry point, including npm/ts-node)
NODE_OPTIONS="--max-http-header-size=65536" node app.jsWith maxHeaderSize = 65536 (64 KB), the formula becomes:
Max duration (seconds) ≈ http_headers_progress_interval_ms × 300 ÷ 1000
Max duration ≈ http_headers_progress_interval_ms ÷ 1000 × 300Examples at 64 KB:
http_headers_progress_interval_ms: '10000' (10s) → ~50 minutes max safe durationhttp_headers_progress_interval_ms: '60000' (60s) → ~5 hours max safe durationhttp_headers_progress_interval_ms: '120000' (120s) → ~10 hours max safe durationGuidelines for choosing the interval (subject to your load balancer’s idle timeout — see trade-offs below):
'10000' ms (10s) intervals, if your LB idle timeout allows'60000' ms (60s) intervals, if your LB idle timeout allows'120000' ms (120s) intervals, if your LB idle timeout allowshttp_headers_progress_interval_ms to extend the safe duration, while keeping it below your LB idle timeout and within Node.js header-limit constraintsUse this command to experiment and debug:
curl -v "http://localhost:8123/?function_sleep_max_microseconds_per_block=10000000&wait_end_of_query=1&send_progress_in_http_headers=1&max_block_size=1&query=select+sum(sleepEachRow(1))+from+numbers(10)+FORMAT+JSONEachRow"Experimenting with the exact load balancer stack might be required.
Important trade-offs:
As a rule of thumb, set the interval slightly below your load balancer’s idle timeout—typically by a few seconds (for example, often around 5–20 seconds), depending on your load balancer, proxies, and network behavior—while staying under the header limit for your expected query duration.
Alternatively — fire-and-forget (mutations only): Mutations (INSERT ... SELECT, OPTIMIZE, ALTER) are not cancelled on the server when the client connection is lost. You can send the mutation and immediately close the connection, then poll system.query_log or system.mutations for status. This bypasses both the load balancer idle timeout and the Node.js header limit. See the client repo examples (opens in a new tab) for a concrete implementation.
Requires:
>= 0.1.1(Keep-Alive disable option introduced in 0.1.1).
Adds overhead (new TCP connection per request) but eliminates all Keep-Alive issues:
const client = createClient({
keep_alive: { enabled: false },
});