What Happens After You Send a Message in an LLM Chat UI

A chat interface makes an interaction with a large language model look simple: type a message, press Enter, and watch an answer appear. Behind that interaction is a distributed request pipeline involving the browser, application servers, databases, policy systems, model gateways, inference workers, and often tools or retrieval services.

In an existing conversation, the new text is only one part of the model’s input. The application must reconstruct enough of the conversation, combine it with hidden instructions and other context, fit everything into a finite context window, run inference, stream the result, and persist the new state.

A representative path is:

Specific products combine, reorder, or omit some of these stages. The model API may also be local to the application rather than a separate service. The general principle remains the same: the web application owns the conversation, while an ordinary language-model invocation transforms a bounded input token sequence into an output token sequence.

Before the Message Is Sent #

As the user types, the browser normally updates a text area and keeps the draft in component state. It may also:

• resize the composer;
• save the draft to local storage;
• upload and scan attachments before the message is sent;
• recognize slash commands or mentions;
• estimate whether the input exceeds a product limit; and
• enable or disable the send button.

This work does not usually involve the language model. Autocomplete, spell-checking, and input-method editors can generate their own browser or operating-system events, but the chat application still receives final text.

Pressing Enter, clicking Send, or tapping the mobile equivalent triggers a send handler. The handler captures an immutable version of the current draft, because the composer may immediately be cleared for the next message. It also creates identifiers used to correlate the optimistic UI, network request, and eventual database records.

Common identifiers include:

The browser usually adds the user message to the transcript immediately. This is an optimistic update: the UI displays the expected state before the server confirms it. If submission fails, the message can be marked as failed and offered for retry instead of disappearing.

Submitting the Turn #

The browser sends an authenticated HTTPS request containing the conversation identifier, the new message, attachment references, selected model, and product-specific options. It may use a conventional POST request:

POST /api/conversations/conv_123/messages HTTP/1.1
Authorization: Bearer ******
Content-Type: application/json
Idempotency-Key: turn_789

{
  "message_id": "msg_456",
  "parent_id": "msg_455",
  "content": [
    {
      "type": "text",
      "text": "Compare that approach with optimistic locking."
    }
  ],
  "model": "example-chat-model"
}

This is the application’s own API, not necessarily the model provider’s API. The backend may later translate the turn into an OpenAI-compatible request, a provider-specific protocol, or a direct call into an inference runtime.

The request passes through ordinary web infrastructure such as a content delivery network, web application firewall, load balancer, and API gateway. These layers can terminate TLS, reject an oversized body, enforce a request rate, or route the request to a chat service.

The chat service authenticates the user and authorizes access to the conversation. Knowing a conversation ID must not be sufficient to read or extend another user’s conversation. It also validates the parent message, model selection, feature flags, attachment ownership, and account quota.

Recording the User Message #

The application normally persists the user message before starting expensive generation. A simplified data model might contain:

Persisting the message first makes the turn recoverable if the browser disconnects or an inference worker fails. The message or generation status can move through states such as pending, running, completed, failed, and cancelled.

An existing conversation is not always a simple list. Editing an old message, regenerating an answer, or choosing among alternative answers creates a tree:

parent_id tells the backend which path is active. When the user sends F, the context should normally contain A, B, C, D, F, not the sibling answer E or its descendant G.

Some systems instead copy branches into separate linear conversations. Either representation must resolve to one ordered history before model invocation.

Reconstructing the Conversation #

The model does not inherently know that the browser is displaying an existing conversation. Most chat inference endpoints are stateless: every invocation must supply the context needed for that invocation. A conversation ID is useful to the application, but it has no meaning to the model unless a stateful model service explicitly implements one.

The backend loads the active message path and builds a logical input resembling:

system: You are a helpful assistant.
developer: Follow the application's formatting and tool-use rules.
user: How does optimistic concurrency control work?
assistant: It detects conflicts rather than holding a lock...
user: Compare that approach with optimistic locking.

The visible transcript is not necessarily the exact model input. The backend may add:

• system and developer instructions;
• account, workspace, or custom instructions;
• the current date and locale;
• model-specific control tokens;
• tool definitions and tool results;
• retrieved documents or web results;
• attachment contents or image references;
• summaries of older messages;
• output-format constraints; and
• safety or product-policy instructions.

Conversely, some visible data may not be sent. UI-only status messages, error banners, citations, reaction buttons, and hidden branches are application state rather than conversational context.

Fitting the Context Window #

Every model has a maximum context measured in tokens. Input tokens, generated output tokens, and sometimes model-specific reasoning tokens compete for that budget. The complete database history cannot simply be appended forever.

A context manager first estimates or computes the token count using the tokenizer associated with the selected model. It then reserves room for the answer:

If the input is too large, the application can:

• drop the oldest conversational turns;
• retain the first instructions and most recent turns;
• summarize older turns and include the summary;
• retrieve only history relevant to the new message;
• reduce retrieved documents or tool descriptions;
• reject the request and ask the user to shorten it; or
• route the turn to a model with a larger context window.

Naively deleting old messages can break references such as “compare it with the first design.” Summarization saves tokens but can lose exact wording, code, numbers, uncertainty, or attribution. Context management is therefore an application policy, not merely a tokenizer operation.

Attachments need their own processing. Text files may be parsed, divided into chunks, indexed, or inserted directly. Images may become encoded image inputs for a multimodal model. Large document collections are often searched first so that only relevant excerpts enter the model context.

Moderation, Limits, and Scheduling #

Before inference, one or more services may evaluate the request. Depending on the product, these checks can include:

• abuse and content classification;
• prompt-injection defenses around external content;
• malware scanning for attachments;
• per-user and per-organization quotas;
• token and spending limits;
• regional data-processing restrictions;
• model availability and entitlement; and
• concurrency or rate limits.

Policy can be enforced before prompt construction, after prompt construction, while output is streaming, or at all three points. Rejecting early avoids wasting inference capacity, while output checks can catch content that did not exist until generation.

Accepted requests enter a scheduler or model gateway. It chooses a provider, region, model deployment, and inference worker according to availability, latency, cost, hardware capacity, and tenant policy. A hosted product may route different turns in the same conversation to different physical machines.

The scheduler can batch unrelated users’ requests so that a GPU processes them efficiently. Continuous batching admits and removes sequences as other sequences finish, rather than waiting for an entire fixed batch to complete. Isolation must still prevent one request from observing another request’s tokens or data.

Converting Messages into Model Input #

The chat service commonly represents a turn as structured data:

[
  {
    "role": "system",
    "content": "You are a helpful assistant."
  },
  {
    "role": "user",
    "content": "Why is the sky blue?"
  }
]

The transformer cannot consume JSON objects, role names, or JavaScript strings directly. Before inference, the service must turn this structure into the integer token sequence expected by the particular model:

Chat Templates #

A chat template defines how roles, content parts, tool calls, and turn boundaries are serialized. A simplified Jinja-like template might be:

{{ bos_token }}
{% for message in messages %}
<|{{ message.role }}|>
{{ message.content }}
<|end|>
{% endfor %}
{% if add_generation_prompt %}
<|assistant|>
{% endif %}

Applying it to the JSON above could produce:

<|begin|><|system|>
You are a helpful assistant.
<|end|><|user|>
Why is the sky blue?
<|end|><|assistant|>

This is illustrative rather than the template of a specific commercial model. Real templates differ in their special tokens, whitespace, role ordering, multimodal parts, tool-call encoding, and placement of the generation prompt. Some use explicit role tokens; others use headers or instruction delimiters. The tokenizer must recognize any special tokens used by the template.

The final assistant marker tells the model that the next tokens should belong to the assistant. The end marker emitted during decoding can tell the runtime that the turn is complete. Using the wrong template can significantly degrade behavior even if all the visible text is present: the model was trained to associate its own delimiters with particular roles and boundaries.

Chat Templates and Vendor Response Schemas #

A chat template does not by itself explain why every response from one vendor has the same JSON shape. These structures exist at different layers:

The vendor’s API contract defines stable fields such as message role, content, finish reason, usage, citations, or tool calls. A gateway can expose that same contract while routing requests to several model versions, each with a different internal template and tokenizer. It translates the common request into the selected model’s format and packages the result back into the common response format.

Several different kinds of schema are easy to conflate:

These schemas are commonly described using JSON Schema or a vendor-specific subset resembling it. For example, an application might request assistant content matching:

{
  "type": "object",
  "properties": {
    "answer": {
      "type": "string"
    },
    "confidence": {
      "type": "number",
      "minimum": 0,
      "maximum": 1
    }
  },
  "required": ["answer", "confidence"],
  "additionalProperties": false
}

Support is not necessarily identical to a complete published JSON Schema draft. A provider may accept only selected keywords, require every property to be listed in required, limit recursion or schema size, or reject unsupported combinations. The application’s SDK may validate the request before sending it, but the server remains authoritative.

An API response envelope being valid JSON does not mean the assistant’s text is JSON. A normal response can have a schema-defined outer object while its content field contains arbitrary Markdown. Conversely, structured-output mode constrains that content to an object or other JSON value. Tool calls are another structured channel: the model proposes arguments conforming to a tool schema, but the host must still parse and validate them before execution.

The web UI can add another proprietary protocol on top. ChatGPT, Gemini, and Claude need not use the same internal request format as their public APIs, and their browser event streams can include UI-specific data. Responses from a single product nevertheless look consistent because its frontend and backend agree on one application schema. The raw transformer produces token probabilities, not a vendor response object.

Structured output also has several possible implementations. A model may emit ordinary text that the gateway parses, generate special tool-call tokens, or be constrained during decoding to match a JSON grammar. In every case, the gateway validates and wraps the result before returning it to the client.

Why a Vendor’s Answers Have a Recognizable Shape #

The JSON response schema explains the machine-readable envelope, but it does not explain recurring prose patterns inside the assistant message. Answers from one product may repeatedly use a summary followed by bullets, bold labels, caveats, or an ending such as “If you want, I can also …”. That style can come from several layers:

• behavior learned during pretraining, instruction tuning, and preference optimization;
• hidden system or developer instructions setting tone and format;
• product-specific custom instructions or conversation memory;
• examples supplied in the prompt;
• model defaults such as verbosity and Markdown use; and
• post-processing that adds citations, notices, or other UI elements.

The chat template contributes role and boundary markers, but usually does not contain the prose blueprint for every answer. A template can say, in effect, “assistant output begins here”; a system instruction can say “end by offering a useful next step”; and training can make that behavior likely even when no such sentence appears in the current prompt.

Recurring phrasing is not a protocol guarantee. ChatGPT does not have to end every answer with an offer to summarize, nor do Gemini or Claude have one mandatory textual ending. The wording can change with the selected model, system instructions, user request, conversation history, product experiments, and sampling. An application that needs a guaranteed structure should request schema-constrained output or validate an explicit format rather than parse a vendor’s customary prose.

How the Tokenizer Is Implemented #

A tokenizer consists of an algorithm plus model-specific data. Common algorithms include byte-pair encoding (BPE), WordPiece, Unigram/SentencePiece, and byte-level variants. The data normally defines:

• a vocabulary mapping token byte strings or pieces to integer IDs;
• merge rules, scores, or a segmentation model;
• normalization and pre-tokenization rules;
• special tokens and their IDs; and
• post-processing rules for adding model control tokens.

It is not necessarily a single executable binary. In a browser or SDK it can be a JavaScript, Python, Rust, Java, or native library. An inference server often links a fast Rust or C++ implementation into the server process. Another architecture runs tokenization in an API worker or dedicated service and sends token IDs to the GPU worker. A command-line tokenizer executable is possible, but starting a process for every request would add avoidable overhead.

Open model repositories commonly store tokenizer data beside the model using files such as:

Names and layouts vary by framework. A hosted vendor may instead keep these artifacts in an internal model registry, object store, container image, or local worker cache. The deployment pins a tokenizer revision to a model revision. This pairing is part of the model contract: changing the mapping while retaining the same weights would cause integer IDs to refer to the wrong learned embeddings.

The application and model service must also agree when they count tokens for limits or billing. A client-side estimate made with a similar but different tokenizer can disagree with the tokens actually processed by the provider.

A Concrete Token-ID Example #

For the public GPT-2 byte-level BPE tokenizer, the exact text:

Hello, world!

is encoded as:

text pieces: ["Hello", ",", " world", "!"]
token IDs:   [15496, 11, 995, 0]

The leading space belongs to the " world" token. This is one reason token boundaries do not match word boundaries. Another tokenizer can assign a completely different sequence to the same text. Capitalization and spacing can also change the result:

"Hello"  != "hello"
"world"  != " world"

Those inequalities mean “can tokenize to different IDs,” not that the words must always be single tokens. Code, uncommon names, typos, and some writing systems often split into more pieces. Modern byte-aware tokenizers can represent arbitrary input, even when no convenient whole-word token exists.

A real chat request would have additional IDs for the serialized system message, role delimiters, turn delimiters, and assistant generation marker. Token IDs are meaningful only with the named tokenizer and revision, so an article or log should never present an ID sequence without identifying that mapping.

Images and audio do not usually pass through the text tokenizer. Separate encoders or tokenizers turn them into image patches, audio features, or modality-specific tokens that are combined with the text representation.

What Happens to Typos #

The tokenizer does not normally correct spelling. It faithfully encodes the received bytes or characters, and a typo may become an uncommon token or several smaller tokens. For example, a misspelled word can still be represented from subword or byte pieces. The language model may infer the intended word from context, but that is learned prediction rather than tokenizer correction, and it is not guaranteed.

A typo can be changed at other stages:

• the browser’s spell-checker can alter the composer before submission;
• the application can run an explicit spelling or query-rewriting step;
• a retrieval system can search spelling variants or use semantic embeddings;
• an ingestion pipeline can apply OCR correction or normalization to documents; and
• the model can be instructed to silently interpret or visibly correct likely mistakes.

Applications should preserve the original user message and source document even when they create a normalized copy for search. Silent rewriting can change names, source code, identifiers, quoted text, medical terms, or the user’s actual meaning. Retrieval systems commonly keep both raw and normalized fields: the normalized form improves matching, while the raw form is shown to the user and supplied for accurate quotation. Any generated correction should retain provenance and, when ambiguity matters, ask for clarification rather than inventing a replacement.

Prefill: Reading the Input #

For a typical decoder-only chat model, prefill begins after chat templating and tokenization. Its primary input is the complete sequence of input token IDs:

token IDs: [1, 32001, 887, 526, 263, 8444, 20255, 13, 32002, ...]
positions: [0,     1,   2,   3,   4,    5,     6,  7,     8, ...]

The runtime also supplies batch metadata, sequence lengths, position information, and a causal attention mask or equivalent optimized metadata. In a batch, shorter sequences may be padded or packed together. The exact token IDs above are illustrative because IDs have meaning only for one tokenizer revision.

The model first looks up each token ID in its learned embedding table, producing a vector rather than a word:

Here:

• S is the sequence length: the number of input token positions;
• V is the vocabulary size: the number of token IDs that can index the table; and
• H is the model’s hidden dimension: the number of learned scalar components in each token embedding.

The token-ID input is therefore a one-dimensional integer vector of shape [S]. The embedding table is a learned floating-point matrix with one row for each vocabulary entry. Looking up all S IDs selects S rows and produces a floating-point matrix of shape [S, H]. With a batch dimension B, the corresponding shapes are [B, S], [V, H], and [B, S, H]; the same embedding table is shared across the batch. The dtypes shown are common examples rather than requirements. A quantized embedding table is normally dequantized or converted to the model’s compute dtype when its selected rows are used.

Position information is added or applied so that identical tokens at different locations can behave differently. In the additive design shown above, E and P have the same shape and are added component by component. P is not appended as extra feature columns: concatenation would produce [S, 2H], whereas numerical addition preserves [S, H].

For example, suppose token ID 995 appears at positions 2 and 9. The embedding lookup initially returns the same learned vector, E[995], at both positions. A model using learned absolute position embeddings can instead pass these distinct vectors into its first layer:

position 2: E[995] + P[2]
position 9: E[995] + P[9]

This also distinguishes "dog bites man" from "man bites dog". The two sequences contain the same three token embeddings, but those embeddings occupy different positions. Without position information, self-attention would not have a direct representation of that order.

Not every model literally adds a position vector:

• With rotary position embeddings, or RoPE, the runtime rotates components of each query and key according to its position. The base token embedding can remain the same, while the attention calculation becomes sensitive to absolute and relative positions.
• Other architectures use relative-position biases or related mechanisms.

The position-aware vectors pass through every transformer layer. In a simplified decoder layer:

The causal rule allows the representation at position i to attend only to positions up to i, not to later input positions.

• Although that dependency is causal, GPUs can compute the prefill representations for all input positions in large parallel matrix operations.
• This differs from decoding, where each newly selected token must exist before the next step can begin.

For every layer and input position, attention computes a key vector and a value vector. The runtime retains these vectors in the KV cache:

It does not generally retain every temporary activation from prefill after inference. The KV cache is the important persistent result because each later decoding step can attend to the existing sequence without recomputing keys and values for all earlier tokens.

At the final layer, the hidden state for the last input position is projected through the language-model output head into one score, or logit, per vocabulary token. After any penalties, constraints, and sampling transform those logits, they define the candidates for the first generated token. Prefill therefore culminates in two results:

1. a populated per-layer KV cache for the input prefix; and
2. logits from which the first output token can be selected.

After selecting that first token, the runtime enters decoding. Each decoding step adds one position’s keys and values to the existing cache and produces logits for the following token. Encoder-decoder models divide this work differently, but most modern text chat models use the decoder-only pattern described here.

Prefill work grows with the input length and is often compute-intensive. This is why a long existing conversation can increase the delay before any answer appears even when the answer itself is short.

A Prefix-Caching Example #

Suppose an application sends a 10,000-token manual followed by a question:

During request 1, the server computes KV entries for the entire sequence. A prefix cache can retain the KV blocks for the stable system instructions and manual. A later request might be:

If the serialized bytes produce the same leading token IDs and the deployment uses the same model, tokenizer, template, and relevant inference settings, the runtime can attach the cached KV blocks and prefill only the new suffix. The model still attends to the manual; the server is reusing its already computed keys and values, not omitting the document.

Production caches often divide prefixes into fixed-size token blocks and hash each block together with model and tenant metadata. The reusable length may therefore end at the last matching block boundary rather than the last individual matching token.

Caching does not generally help when a token near the beginning changes:

All tokens after the first difference have different attention context and their cached state cannot simply be spliced in. Likewise, changing whitespace, tool definitions, hidden instructions, attachment ordering, or the chat template can alter the token prefix even if the visible question is unchanged.

Prefix caching is distinct from caching a final answer. It reuses internal attention state but still performs decoding, so a sampled response can differ. It also differs from the per-request KV cache used during ordinary decoding: prefix caching preserves reusable KV state across requests, while a normal KV cache only needs to live for one active generation.

Providers impose capacity, expiration, minimum-prefix-length, isolation, and pricing rules. Some populate caches automatically; others require an explicit cache object or marker. Merely reusing a conversation ID does not guarantee a hit, and cache keys must prevent one tenant from acquiring another tenant’s private prefix state.

Decoding: Producing the Answer #

After prefill, the model predicts a probability distribution for the next token. A decoding policy selects one token, appends it to the sequence, and runs another model step:

This loop continues until the model:

• emits a stop token or stop sequence;
• reaches an output-token limit;
• requests a tool;
• is stopped by a policy system;
• encounters an error; or
• receives a cancellation signal.

Selection can be greedy or use sampling controls such as temperature and top-p. Products may also apply repetition penalties, constrained decoding, grammar enforcement, speculative decoding, or provider-specific reasoning mechanisms.

The model does not normally generate an entire paragraph in one operation. It generates tokens incrementally. A token can be a whole word, part of a word, punctuation, or whitespace. The server must decode token IDs back into text carefully because a character can span token-byte boundaries.

Streaming Back to the Browser #

The inference worker emits token or text deltas to the model gateway. The gateway translates them into application events and forwards them through the chat service to the browser.

Common transports are:

• a streamed fetch() response;
• Server-Sent Events (SSE);
• WebSocket messages; or
• a provider stream consumed by the backend and relayed over another protocol.

An SSE-like stream might contain:

event: message_start
data: {"message_id":"msg_457"}

event: content_delta
data: {"text":"Optimistic locking"}

event: content_delta
data: {"text":" is a common form"}

event: content_delta
data: {"text":" of optimistic concurrency control."}

event: message_end
data: {"finish_reason":"stop"}

The transport may carry much more than text: reasoning summaries, citations, tool calls, usage counters, warnings, or heartbeat events. Sequence numbers help detect duplicate or missing events after reconnects.

Streaming improves perceived latency because the user can read before generation ends. It does not necessarily reduce total generation time. Important latency measurements include:

Several buffers can delay visible output even after decoding starts: the inference server, reverse proxy, compression layer, content filter, and browser stream parser. Streaming endpoints usually disable proxy buffering and flush small events deliberately.

Rendering the Partial Answer #

The browser reads events and updates the pending assistant message. Updating the document object model once for every tiny token can be expensive, so the UI often accumulates deltas and renders them at animation-frame or short timer intervals.

Markdown introduces another complication: a partial stream is frequently invalid Markdown. A code fence, link, table, or emphasis marker may not yet be closed. The renderer must tolerate incomplete input and rerender as more text arrives. Syntax highlighting and mathematical typesetting may be deferred or throttled because repeatedly processing the whole answer causes visible jank.

The browser should treat generated HTML as untrusted. A Markdown renderer must escape or sanitize dangerous output rather than letting model-generated script, event-handler attributes, or unsafe URLs execute in the page.

Auto-scrolling is conditional. The view can follow the answer while the user is already near the bottom, but it should not repeatedly pull the user away from an older message they are reading.

Tool Calls and Multi-Step Turns #

A response is not always user-visible prose. If tools were supplied in the prompt, the model may emit a structured request:

{
  "name": "get_order",
  "arguments": {
    "order_id": "42"
  }
}

The application, not the model, executes the tool. A typical agent loop is:

One visible user turn can therefore contain several model invocations and several calls to databases, search engines, code interpreters, or remote APIs. The UI may show intermediate states such as “Searching,” “Reading document,” or “Waiting for approval.”

Tool results consume context tokens and can themselves be untrusted. The host must enforce permissions, timeouts, result-size limits, and side-effect controls. Destructive actions should not become safe merely because the model requested them.

Retrieval follows a related pattern. The application may rewrite the user’s question into a search query, retrieve candidate documents, rerank them, insert selected excerpts into the context, and only then invoke the answering model. Some systems retrieve before the first model invocation; others let the model choose a search tool during the turn.

Completing and Persisting the Answer #

When decoding finishes, the model service emits a finish reason and final metadata. The application assembles all deltas into the canonical assistant message and records:

• generated content and structured content parts;
• model and model-version identifiers;
• input, output, cached, or reasoning token usage;
• tool calls and tool results;
• citations or source references;
• finish reason;
• latency and trace identifiers; and
• policy decisions or error details.

The final database write should be idempotent. Retries and duplicate stream events must not create multiple assistant messages. Some systems persist partial checkpoints during generation so that an interrupted answer can still be displayed; others save only the final assembled message.

The server sends a terminal event only after it knows the outcome. The browser then marks the assistant message as complete, stops the activity indicator, enables actions such as copy or regenerate, and reconciles optimistic IDs with server-issued IDs.

Background work can continue after the answer appears. The application might generate a conversation title, update a search index, compute safety analytics, extract long-term memory, record billing, or emit quality metrics. These tasks should not extend user-visible latency unless their result is required for the turn.

Cancellation, Retry, and Regeneration #

Pressing Stop usually cancels generation rather than reversing already generated text. The browser sends a cancellation request or closes the stream; the backend propagates a cancellation signal to queued work, tool calls, and the inference runtime. The provider may already have generated additional tokens in transit, and billing semantics vary.

The partial answer can be retained with a cancelled finish reason. If it is kept in later context, the application should distinguish it from an answer that ended normally.

A network disconnect is not the same as cancellation. The browser may vanish while server-side generation continues. Applications choose among several policies:

• cancel work as soon as the client disconnects;
• let generation finish and persist the answer;
• keep a resumable event stream for reconnection; or
• continue only until a short disconnect grace period expires.

Retrying submission should reuse an idempotency key so that an uncertain network outcome does not append the user message twice.

Regenerate normally creates a new assistant sibling under the same user message. It can reuse the same prompt while changing the random sample, model, or options. It does not have to overwrite the previous answer. Edit and resubmit creates a branch from the edited message’s parent, because all later messages were conditioned on the original wording.

Failures at Different Layers #

The UI should not reduce every failure to “the model failed.” Different stages have different recovery behavior:

Automatic retries are safe only when the operation is idempotent or its effects are known. Retrying text generation can produce a different answer. Retrying a tool that sends an email, places an order, or deletes a file can duplicate a real-world side effect.

Errors should be represented separately from assistant prose. A generated sentence saying “an error occurred” is not equivalent to a transport or tool failure and should not control application state.

Privacy, Security, and Observability #

The request can contain more than the text the user just typed. It may include earlier messages, uploaded files, retrieved private documents, tool outputs, and hidden instructions. Each service in the path should receive only the data it needs.

Important controls include:

• encryption in transit and at rest;
• authorization on every conversation and attachment lookup;
• tenant isolation in storage, caches, queues, logs, and inference;
• redaction or exclusion of sensitive prompt data from telemetry;
• retention and deletion policies;
• audited tool execution and user approval;
• output sanitization in the browser; and
• protection against instructions embedded in retrieved content.

Observability still matters. Distributed tracing can connect a browser turn to gateway, retrieval, tool, and inference spans. Useful metrics include queue time, time to first token, tokens per second, context size, cache-hit rate, finish reason, cancellation rate, and failure stage. Logging complete prompts is convenient for debugging but risky; metadata, hashes, sampled redacted content, and access-controlled diagnostic stores are safer defaults.

The Essential Mental Model #

An existing LLM chat is best understood as a stateful application wrapped around one or more bounded model invocations:

The apparent continuity of the conversation is produced by the application reconstructing and resending context. The apparent typing effect is produced by streaming incremental decoding. The apparent ability to search, calculate, or take action is produced by an application-controlled loop around the model.

That separation explains many otherwise surprising behaviors: old details disappear when they fall outside the selected context, long conversations take longer to start, regeneration produces a different branch, a stopped answer can remain partially visible, and the model cannot access a conversation, database, or website unless the surrounding application explicitly gives it that information or capability.