Streaming Responses¶

Most APIs return a small JSON blob you can read in one gulp. Some don't — live feeds, Server-Sent Events, newline-delimited logs. For those, load the body chunk by chunk with responseType: "stream".

Streaming is opt-in only. It is never auto-detected. If you didn't ask for it, you get the whole body at once.

How to stream¶

let res = await HTTP.get("https://api.example.com/events", {
  responseType: "stream"
})

When responseType is "stream":

1. Response headers arrive first (status, Content-Type, cookies, etc.)
2. The call returns immediately with a stream object — body is not pre-loaded
3. You read chunks via stream.read(), for await, or stream.collect()
4. Two internal timers guard the connection (lifetime + idle)
5. Cumulative bytes read are capped at the response size limit (2 MB default)

HTTP.get({ responseType: "stream" })
    → headers received, cookies parsed
    → res.stream created, timers start
    → you read chunks one at a time
    → stream ends (done), cancel(), or timer/limit hit

For non-stream requests, the full body is read first, Content-Length is checked, then data and content are returned. Streams skip the upfront Content-Length check — the size limit is enforced as you read.

What streaming is good for¶

For typical small JSON APIs, use responseType: "auto" — simpler and returns res.data directly. Don't stream what you can swallow whole.

What is NOT possible¶

These are hard limits from the implementation:

Stream response object¶

Fields present when responseType: "stream":

Fields not present on stream responses:

Non-2xx responses still return a stream object — you can read the error body as chunks if the server sends one.

Stream interface¶

res.stream exposes exactly three methods plus async iteration:

stream.read() → { value, done }¶

Reads one chunk at a time:

while (true) {
  let result = await res.stream.read()

  if (result.done) break          // stream finished normally

  let chunk = result.value        // UTF-8 string
  Bot.inspect(chunk)
}

Throws if the stream was aborted, timed out, or hit a size limit.

for await (chunk of res.stream)¶

Same as repeated read() calls — each chunk is a UTF-8 string:

for await (let chunk of res.stream) {
  Bot.inspect(chunk)
}

stream.collect(maxSize?) → string¶

Reads all remaining chunks and joins them into one UTF-8 string.

let fullBody = await res.stream.collect()           // default max: 2 MB
let partial  = await res.stream.collect(512 * 1024) // custom max: 512 KB

• Uses the same guarded reader as read() — respects chunk and cumulative limits
• Throws if total collected size exceeds maxSize
• Clears timers on successful completion

stream.cancel() → void¶

Stops the stream immediately. Synchronous — does not return a promise.

for await (let chunk of res.stream) {
  if (chunk.includes("END")) {
    res.stream.cancel()
    break
  }
}

Destroys the underlying connection and clears both timers. Any subsequent read() throws "Stream cancelled by user" or "Stream has been closed or aborted".

Timers¶

Two independent timers start the moment the response is parsed (when await HTTP.get(...) resolves or just before a callback command runs):

Lifetime timer (plan-based)¶

Counts from response parse time — not from your first read(). If you do other work after await before reading, that time counts. Procrastination has consequences.

Error message: Stream lifetime limit of Xs exceeded

Idle timer (fixed)¶

15 seconds without receiving any chunk → stream aborted.

Resets every time a chunk arrives. Error message: Stream idle timeout of 15s exceeded (no data received)

Size limits during read¶

Request timeout vs stream timers¶

For direct requests (no cfProxy):

For cfProxy requests:

So for long-lived SSE feeds, a direct request gives you up to the plan lifetime (30–120s). Through cfProxy, the worker must respond within timeout.

Try it — copy-paste examples¶

SSE (text event stream)¶

let res = await HTTP.get("https://api.example.com/live", {
  responseType: "stream",
  headers: { Accept: "text/event-stream" }
})

if (!res.ok || !res.isStream) {
  Bot.sendMessage("Feed unavailable.")
  return
}

try {
  for await (let chunk of res.stream) {
    // Chunks may split mid-line — buffer incomplete lines yourself
    for (let line of chunk.split("\n")) {
      if (line.startsWith("data: ")) {
        Bot.inspect(line.slice(6))
      }
    }
  }
} catch (err) {
  Bot.sendMessage("Stream ended: " + err.message)
}

NDJSON — collect then parse¶

Only works if the full body is under 2 MB:

let res = await HTTP.get("https://api.example.com/export.ndjson", {
  responseType: "stream"
})

let raw = await res.stream.collect()
let records = raw.split("\n").filter(Boolean).map(JSON.parse)

Bot.sendMessage("Loaded " + records.length + " records.")

Read until enough, then cancel¶

let res = await HTTP.get(url, { responseType: "stream" })
let found = false

for await (let chunk of res.stream) {
  if (chunk.includes("target_value")) {
    found = true
    res.stream.cancel()
    break
  }
}

Non-2xx error body¶

let res = await HTTP.get("https://api.example.com/missing", {
  responseType: "stream"
})

if (!res.ok) {
  let errorBody = await res.stream.collect()
  Bot.sendMessage("Error " + res.status + ": " + errorBody)
}

With fallback commands¶

Streaming works with success / error, but timers start when the HTTP response is parsed — before your callback command executes. Start reading immediately inside the callback.

HTTP.get({
  url: "https://api.example.com/feed",
  responseType: "stream",
  success: "/processFeed"
})

// /processFeed — read immediately, don't do slow work first
if (!response.isStream) return

try {
  for await (let chunk of response.stream) {
    Bot.inspect(chunk)
  }
} catch (err) {
  Bot.inspect("Stream error: " + err.message)
}

In callbacks: response.stream, response.isStream, response.contentType via the response global. See Fallback Commands.

Error reference¶

All stream errors are thrown during read(), for await, or collect() — wrap in try/catch.

Streaming vs regular response¶

See Limits & Timeouts for the full caps table.