Stream replies as server-sent events

Set stream to true and the reply arrives as server-sent events while the model writes it, instead of in one piece at the end. You can show the first words at once.

curl -N https://sven.swedishembedded.com/v1/chat/completions \
  -H "Authorization: Bearer $SVEN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "MODEL_ID",
    "messages": [{"role": "user", "content": "Write a haiku about the sea"}],
    "stream": true,
    "stream_options": {"include_usage": true}
  }'

The events#

The response is text/event-stream. Each event is a line data: followed by a JSON chunk and a blank line, in the shape OpenAI's clients read:

1. A first chunk whose delta names the role.
2. Chunks whose delta carries text (content), reasoning where the model writes it, or pieces of a tool call.
3. A chunk with a finish_reason.
4. With stream_options.include_usage, a last chunk with an empty choices list and the usage of the whole call.
5. The line data: [DONE].

A client library does all of this for you. If you read the stream yourself, stop at [DONE].

Errors before and after the first token#

A request that is refused, for a bad key, a missing model or not enough credit, is answered with an ordinary error status and JSON body before any stream starts. See Errors.

Once the stream has started its status is already 200. A failure after that arrives as an event, data: with an error object holding a message and a type, and the stream then ends. Treat a stream that ends without [DONE] as incomplete. If it broke after the model had reported its usage you are charged for that usage; a retry with the same Idempotency-Key replays the output delivered so far and ends with the same error, and does not charge again. See Safe retries.

If a server fails before your answer starts, another one takes the request and you do not notice.

Long answers and slow readers#

A streamed answer has no length limit of its own beyond the request's max_tokens. If you read more slowly than the model writes, we wait for you rather than cut the stream. A client that reads nothing at all for 30 seconds is treated as gone, as if the connection had dropped.

If your connection drops#

The reply still runs to its end on our side, and you are charged for what it produced. To receive the whole answer without being charged again, send the same request again with the same Idempotency-Key. While the first call is still running, the retry is answered 409: wait a moment and try again. See Safe retries.

Streaming with Anthropic's client#

POST /v1/messages streams the same way, in the events Anthropic's client reads: the Anthropic tabs above stream it with the client's own helper. See Using Anthropic's client.