Genace
Sign inGet started

SSE chunk parser: decode generation events across arbitrary boundaries

A network chunk is a transport detail, not an SSE message boundary. One event may span several chunks, and one chunk may contain several events. UTF-8 characters can also span byte reads. A generation client needs a streaming decoder followed by a line and event parser; calling JSON.parse on every chunk will eventually fail under ordinary network conditions.

Parse lines before parsing JSON

Genace currently emits named events with JSON data terminated by a blank line. The following narrow parser supports its event and data fields, comments, multiple data lines, and CR, LF, or CRLF line endings. It is application code, not a full replacement for every EventSource feature. It deliberately does not implement event IDs, retry directives, or browser reconnection policy.

export function generationParser(emit) {
  const decoder = new TextDecoder();
  let line = '', skipLF = false, name = 'message', data = [];
  function consumeLine() {
    if (!line) {
      if (data.length) emit({ event: name || 'message', data: data.join('\n') });
      name = 'message'; data = [];
    } else if (!line.startsWith(':')) {
      const colon = line.indexOf(':');
      const field = colon < 0 ? line : line.slice(0, colon);
      let value = colon < 0 ? '' : line.slice(colon + 1);
      if (value.startsWith(' ')) value = value.slice(1);
      if (field === 'event') name = value;
      if (field === 'data') data.push(value);
    }
    line = '';
  }
  return (bytes, final = false) => {
    const text = decoder.decode(bytes, { stream: !final });
    for (const character of text) {
      if (skipLF) { skipLF = false; if (character === '\n') continue; }
      if (character === '\r') { consumeLine(); skipLF = true; }
      else if (character === '\n') consumeLine();
      else line += character;
    }
  };
}

Call the returned function for each Uint8Array and flush the decoder once at the end. Incomplete data without a terminating blank line is not a completed event. Add explicit line and record size caps in a production client so an unexpected peer cannot grow buffers without limit.

Validate event payloads separately

After an event is assembled, parse its data as JSON and validate the fields your application consumes. Genace events include state, progress, outputs, and error; unavailable values can be null. Do not confuse the event name error with an HTTP failure or a transport exception. A received terminal error event is data describing a failed or canceled job.

Save the remote job ID outside the stream because these event payloads do not carry a durable replay cursor. A terminal progress record followed by done can repeat the same result. Make downstream collection tolerant of repeated terminal information rather than publishing twice.

Construct adversarial boundaries offline

Encode a valid fixture once, then split it at every byte position and verify the reconstructed events are identical. Include a multibyte character, a CRLF pair split across chunks, two events in one chunk, comment lines, and multiple data lines. A truncated final event should remain uncommitted.

Also test malformed JSON inside a complete event. Report a protocol payload problem with the saved job ID, then recover current state through GET if appropriate. Do not launch another generation because an observer’s parser failed.

Acceptance criteria

Verify equivalent results for whole-buffer and one-byte-at-a-time delivery. Confirm terminal state is checked before outputs are used and null progress is not reported as measured zero percent. Ensure buffer limits produce a controlled disconnect and recovery path rather than an unhandled memory error.

Read the SSE overview, reconnect without IDs, and disconnect tests for client orchestration beyond parsing.

Continue with code and integration

Where these facts come from