AI API Calls for SDEs: From HTTP to Generated Tokens
Trace one request through tokenization, prefill, decode, KV cache, streaming and tool execution—with failure boundaries, budgets and runnable local code.
Sources & version details
In this article
An AI call is a distributed program, not a remote brain
Consider an SRE assistant asked why a deployment is Pending. Your application obtains a Kubernetes snapshot, selects a runbook, sends relevant evidence to a model and displays a diagnosis. If a repair is requested, a separate executor must authorize it. We follow this one request from application code to generated text. The running example is a reference design, not a reconstruction of any vendor’s private infrastructure.
The prerequisite is ordinary backend engineering: HTTP transports a request; TLS protects it in transit; authentication identifies a caller; a queue separates admission from execution; a timeout is an observation by one participant. Add three ML concepts: tokens are vocabulary IDs, weights are learned parameters, and inference computes a conditional output distribution using those parameters. Training changes weights; ordinary inference does not.
A successful HTTP status proves neither factual correctness nor completed business work. Define the contract as three independent outcomes: transport completed, model response completed, task accepted. A fluent diagnosis without evidence can pass the first two and fail the third. Use a model for fuzzy language and synthesis; use deterministic code for authorization, arithmetic, state transitions and invariants.
At the model boundary the operation is pθ(y₁…yₙ | x), factorized into next-token conditional probabilities. θ denotes fixed weights; x is the actual assembled context, not everything your application knows. The generated answer is a sample or decoding decision from this distribution. It is not a database transaction that looked up an authoritative incident record. To make “insufficient GPU” defensible, attach snapshot ID, capture time and the exact scheduler event to the answer.
Separate state ownership explicitly. The application owns identity, conversation and task status; the engine owns temporary sequence/KV state; the executor owns durable action outcomes. A model restart may discard KV without deleting conversation history. A process crash after an action can lose the response while the action remains committed. This is why recovery must consult the owner of the state instead of asking the model to remember what happened.
From SDK to admitted request
An SDK serializes your messages and settings, adds authentication, sends HTTPS and maps response events into objects. It does not usually run the large model on your laptop. A reference serving path includes an ingress, authentication and quota checks, a model router, admission control and an engine queue. Their exact order and implementation vary. Keep the provider key on your backend; a browser calls your own authenticated endpoint.
Give each logical diagnosis a request ID, deadline, output limit and model configuration. Give each network attempt its own attempt ID. A bounded concurrency semaphore prevents a traffic spike from becoming unlimited in-flight work; admission should reject work that cannot meet its deadline. Requests per minute and tokens per minute constrain different resources: a few long prompts can exhaust token quota before request quota.
A provider-side conversation object may store history, but the transformer still operates on the context assembled for that inference. Do not assume the server remembers omitted facts, that a prior upload is automatically included, or that changing a model alias preserves behavior. Persist the application’s canonical conversation and evidence versions; record the resolved model/version when the API exposes it.
A connection pool reuses established connections, reducing repeated DNS/TCP/TLS setup where applicable. HTTP/2 can multiplex streams, but multiplexing does not remove server queueing or model quota. Configure connection, response-idle and overall operation deadlines separately. Use monotonic time for elapsed deadlines; derive each downstream timeout from the remaining budget rather than resetting a full timeout at every hop.
Retries can multiply invisibly: if the SDK permits three attempts and your outer worker also permits three, one logical request can become nine upstream attempts. Set one retry owner or coordinate both layers, and include every attempt in the budget. A 20-second deadline with 8 seconds already spent leaves at most 12 seconds for backoff plus the next attempt, not another 20. Retrying a POST is not inherently safe merely because the payload is identical. [12]
| Layer | Input → output | Failure ownership |
|---|---|---|
| Application | Question + evidence → request | Identity, privacy, budget |
| Serving | Token IDs → scores → IDs | Queue, memory, generation |
| Transport | Event bytes → complete items | Framing, partial output, cancellation |
| Executor | Authorized action → receipt | Permissions, idempotency, audit |
Messages become tokens, embeddings and tensors
A chat template inserts role delimiters and special tokens around message content. The tokenizer maps the resulting sequence to integer IDs using a model-specific vocabulary and encoding algorithm. A token is not a word, a character or a network packet. Chinese text, source code and whitespace tokenize differently; count the exact templated input with the correct tokenizer where available. Hidden service instructions and media processing can add usage beyond your visible text. [1]
For a teaching decoder, input IDs have shape [B,T]. Looking them up in E of shape [V,d] produces X of shape [B,T,d]: B is batch size, T sequence length, V vocabulary size and d hidden width. The lookup is not a search of web pages; each row is a learned vector. Position information lets the network distinguish order. Inference applies layers of attention, normalization, residual addition and feed-forward computation to these vectors.
Pictures and audio are not generally interpreted by reading their Base64 characters as ordinary prose. A serving stack decodes the transport payload and uses model-specific image patches, audio features or token sequences. Base64 expands bytes to roughly 4/3 of their original size before JSON overhead, but that is not the model’s visual-token billing rule. Validate media type, dimensions, duration and payload limits separately from text-token budgets.
Consider a toy byte-pair encoder with initial symbols a,b,a,b. If its highest-priority merge is (a,b)→ab, the sequence becomes [ab,ab]; another vocabulary might retain four IDs. Real encoders use trained merge priorities or different algorithms and additional rules. Therefore a numeric token ID has meaning only with its tokenizer vocabulary/version. Never compare token IDs from two models as if they were universal character codes.
For
Prefill: read the prompt before generating
A causal attention head forms Q, K and V with shape [T,d_h]. QKᵀ has shape [T,T]; a causal mask forbids reading future positions. Scaling by √d_h prevents dot-product variance from growing unchecked under a simplified independent-component assumption. Softmax is applied across key positions, so the result multiplied by V again has shape [T,d_h]. Multiple heads are concatenated and projected; modern architectures may use grouped KV heads and different normalization. [2]
Prefill computes prompt positions in parallel within each layer because the entire prompt is known, while the mask preserves causality. The final prompt position supplies logits for the first generated token. Attention arithmetic for dense full-context prefill scales quadratically in T, while projections and MLPs add other costs. Efficient kernels can avoid materializing the complete score matrix without removing all quadratic attention arithmetic. A long prompt can therefore delay the first output even when the answer is short.
In the incident example, sending 100,000 log lines is usually inferior to selecting the relevant time interval, scheduler events and cited runbook paragraphs. Retrieval does not expand the model’s actual context limit. Reserve room for output and tool observations; if your budget is 16,000 tokens, 2,000 instructions, 4,000 dialogue and 2,000 reserved output leave 8,000 evidence tokens. These are synthetic numbers, not a named model’s limits.
A minimal numeric head makes the mechanism tangible. Let
For a dense layer, Q/K/V and output projections scale roughly with T d², and the two attention products with T² d; the feed-forward block adds terms involving its intermediate width. Thus “attention is quadratic” is not a complete latency model. Matrix tiling reuses chunks in fast on-chip memory, reducing costly high-bandwidth-memory traffic. Kernel fusion can avoid writing intermediate tensors. These optimizations change data movement and constants; actual speed depends on shapes, precision, hardware and batch.
Decode, sampling and KV cache
Decode repeatedly computes the next-token distribution, chooses a token and appends it. With KV caching, past key/value tensors are retained, so a new token computes its own Q/K/V and attends to the prior keys and values rather than recomputing every prior position. The cache is numerical execution state, not a database of facts or the application’s long-term memory. Its validity depends on the model, prefix, positions and relevant configuration. [3]
For a conventional full-attention cache, bytes ≈ 2 L B T H_kv d_h s, where L is layers, B active sequences of equal length, T cached positions, H_kv KV heads and s bytes per element. With
Logits are unnormalized scores. Temperature divides scores before softmax; top-p retains a high-probability nucleus and renormalizes. Neither is a truth knob. For logits [2,1,0] at temperature1, stable softmax gives approximately [0.6652,0.2447,0.0900]. Greedy selection takes the largest score but does not guarantee correctness or cross-hardware reproducibility. Stop tokens, output limits and tool-call outcomes may end generation; a length-limited JSON fragment is not a successful structured result.
At cached decode step t, a full-attention head has q_t [1,d_h], K_cache [t,d_h] and V_cache [t,d_h]. The scores have shape[1,t]. Appending one token grows cached state; it does not require recalculating old keys, but the new query still reads the visible historical keys/values. Consequently per-step attention cost increases with visible context. Caching removes repeated computation, not all dependence on history. [13]
A block table translates sequence position to storage: with16-token blocks, position35 lies in logical block2 at offset3. A table[7,2,9] places that entry in physical block9. This is an illustrative address map, not a fixed vLLM kernel layout. Physical tensors can be reordered for vectorized/coalesced reads. Shared prefix blocks require reference accounting and safe handling of later writes; a cache key must not allow one tenant to retrieve another tenant’s private prompt state.
GPU scheduling: why throughput and latency disagree
An engine batches work to amortize weight reads and improve GPU utilization. Continuous batching can admit and retire sequences between iterations instead of waiting for a whole static batch. Long prompts, long outputs and uneven request arrivals compete for compute and KV capacity. Faster aggregate tokens/s can coexist with worse per-user latency. Admission limits should account for token and memory budgets, not just CPU or request count.
PagedAttention maps logical KV blocks onto physical memory blocks, reducing fragmentation and enabling efficient sharing; it does not make GPU memory infinite. Prefix caching reuses compatible already-computed prompt state and primarily saves repeated prefill work. It is not response caching, and it does not remove the decode work for a new answer. Exact cache keys, eviction, tenant isolation and model compatibility remain correctness requirements. [4] [5]
At low decode batch sizes, reading weights and KV can dominate arithmetic, making memory bandwidth important. Larger batches may move the bottleneck toward compute. Tensor parallelism splits work but adds communication; prefill/decode disaggregation separates resource profiles but transfers KV state. Quantization reduces representation size with quality and kernel tradeoffs. Speculative decoding validates proposed tokens in groups; its gain depends on acceptance and verification cost. None is a universal speedup without a workload definition.
Use a roofline-style lower bound: kernel time cannot be smaller than both FLOPs/achievable-compute-rate and bytes-moved/achievable-bandwidth. A synthetic8-billion-parameter model at2 bytes per weight is16GB. Reading all those weights once at1TB/s already takes16ms before KV, arithmetic or communication. This is a simplified bandwidth floor, not a measured model speed; reuse, caching, quantization and partitioning change the traffic.
A scheduler iteration allocates token work and KV capacity, launches kernels, collects sampled IDs and retires finished sequences. A large prefill can monopolize a step and delay ongoing decode; chunked prefill subdivides that work to improve interleaving but introduces scheduling tradeoffs. For an autoscaler, queue wait, active/cached tokens and decode latency can be more informative than GPU utilization alone. Scaling up also requires loading weights and warming execution paths, so reactive scaling is not instantaneous.
Streaming is an application protocol over bytes
A stream carries events, often using text/event-stream, over a response body. TCP reads, HTTP chunks, SSE events, JSON deltas, model tokens and rendered characters are different boundaries. One read may end halfway through a multibyte Chinese character, or contain several events. A robust client incrementally decodes UTF-8, buffers incomplete lines/events and only parses complete payloads. Native browser EventSource is not a general replacement for an authenticated JSON POST; fetch streaming or the provider SDK is often the appropriate interface. [6]
Treat generation as a state machine: waiting → streaming → completed, failed or cancelled. HTTP 200 can precede an error event. A clean socket close without the provider’s required completion indication can still mean an incomplete result. Preserve partial text as partial, never silently promote it to success. Tool arguments may arrive as partial JSON; execute only after the tool-call item is complete, parsed and authorized. Provider event names and stop semantics differ; the local lab below uses its own clearly labeled protocol. [7]
A reverse proxy may buffer output and make a healthy model look non-streaming. A slow browser can accumulate an unbounded queue unless the backend applies backpressure or bounded buffering. Cancellation should close upstream work when supported, but disconnecting a client does not prove billing or GPU work has stopped. Define cancellation propagation, maximum buffered bytes, idle timeout and a total deadline as distinct controls.
The parser has three nested buffers: incomplete UTF-8 code points, incomplete lines, and an event’s data fields. CRLF must count as one line break even if CR and LF arrive in different reads. A blank line dispatches an SSE event; multiple data fields are joined with newline. Only then should the JSON decoder run. The application must separately accumulate deltas by response/item ID—one stream may contain text and several tool items.
Transport flow control does not bound every application queue. If producer output is copied into an unbounded Python list before browser delivery, TCP backpressure may arrive too late to protect memory. Cap queued bytes and total response size, apply a slow-consumer policy, and stop reading or cancel upstream where supported. The teaching parser bounds individual events only; production code also needs whole-response and lifetime limits, plus provider-specific terminal handling.
Tool calls turn inference into an agent loop
For client-executed tools, the model returns a proposed name and arguments; your host validates and executes it, then sends the observation back for another inference. Provider-hosted tools follow a different execution boundary. A model saying “I restarted the service” is not proof of a restart. Link a business action to an executor receipt, resource revision and audit record. Anthropic’s workflow/agent distinction is useful here: code can choose the sequence, or the model can propose which tool comes next. [8]
In our SRE case, read_snapshot and search_runbook are allowlisted read operations with tenant scope. restart_deployment is a separate capability requiring resource-specific authorization and, when configured, approval. Treat runbook contents as data, not permission-granting instructions. Structured-output validation ensures shape, not safety or truth. Reject unknown tools and additional fields rather than running model-generated shell commands as a convenient fallback.
Bound the number of model rounds, tool calls, elapsed time and spend. If a write times out, inspect its durable operation state before retrying: a lost response may hide a completed mutation. Use application-level idempotency with a request fingerprint and conflict detection, not just a reused UUID. Neither an HTTP retry nor a provider request ID establishes exactly-once execution of your Kubernetes change.
Use an explicit operation lifecycle: proposed, validated, authorized, executing, succeeded/failed/unknown. For a restart, authorization binds principal, namespace, deployment, action and resource revision. Recheck relevant state immediately before the write to reduce time-of-check/time-of-use races. If the resource changed since diagnosis, require a fresh decision rather than applying an old approval to a new object.
Persist an operation intent before the side effect, but do not call that an atomic transaction with Kubernetes. A crash can occur between the external mutation and recording its receipt. Recovery needs an operation identifier and a way to inspect the external resource or safely recognize duplicate work. A transactional outbox solves reliable delivery from a local database; it does not by itself guarantee exactly-once effects in an arbitrary remote system.
Latency, cost and failure ownership
Measure time to first token (TTFT), time between output tokens, total completion time and accepted-task success separately. TTFT includes connection overhead, queueing, prefill and delivery; it is not pure GPU time. For N output tokens and a simplified constant inter-token time Δ, total time is approximately TTFT+(N−1)Δ. At TTFT0.6s, N101 and Δ0.02s, total is2.6s. Streaming improves perceived responsiveness, not necessarily total completion time.
Use a cost ledger, not a character-count guess. With hypothetical input/output prices of $2/$8 per million tokens, a 2,000-input/500-output request costs $0.008 before caching, tool fees and retries. This is an arithmetic example, not a provider price quote. Internal reasoning or media tokens may be accounted differently. The product metric is cost per accepted task: total paid attempts divided by accepted tasks, including unsuccessful attempts and review labor where relevant.
For 429 responses honor documented retry guidance and use bounded exponential backoff with jitter; never retry forever. For invalid schema, fix the request. For 5xx or disconnects, retry only within the logical deadline and budget, preserving partial-result state. Track provider request ID, attempt ID, model, usage, timestamps, completion reason and redacted tool receipts. Do not log raw customer prompts or credentials by default. High TTFT suggests queue/prefill investigation; smooth generation delivered in bursts suggests buffering; correct transport with wrong facts requires evidence/model evaluation.
Little’s law gives a capacity sanity check under stable conditions: average in-flight requests
Build traces with parent task→model attempt→tool operation, not a single generic “AI latency” span. Record queue wait when observable; otherwise label client-observed TTFT rather than inventing server timings. Compare p50/p95/p99 within matched prompt-length, output-length, model and cache cohorts. A fall in average latency caused by shorter answers or more rejected tasks is not automatically an infrastructure improvement.
What this model explains—and where it stops
This account targets autoregressive, decoder-style text generation behind an API. Embedding endpoints return vectors without an autoregressive text loop; rerankers return relevance scores; diffusion models and realtime speech systems may have different execution and streaming semantics. An API that looks chat-compatible need not expose the same tokenizer, tool behavior, cache isolation or stop reasons. Test the contract you actually deploy.
The frontier is joint optimization: schedule mixed prefill/decode work fairly, move or compress KV without erasing gains, validate speculative output efficiently and maintain quality while reducing precision. At the application layer, the harder unresolved questions include long-horizon task reliability, evidence freshness, prompt-injection resilience and reproducible evaluation under model changes. Lower per-token cost does not solve authorization, and longer context does not establish factual grounding.
A managed API buys operational delegation but hides scheduler, kernel and cache details; self-hosting exposes these knobs and transfers capacity, patching and security responsibility to your team. Choose on data constraints, workload predictability, operational skill and total accepted-task cost, not token price alone. Preserve a narrow provider adapter that normalizes events without erasing refusal, incomplete or tool-call semantics.
The endpoint of this tutorial is a debugging discipline: identify which state owner can answer the question, which measurement distinguishes competing explanations, and which operation is safe to repeat. When evidence is missing, preserve uncertainty. The remaining practical section checks small, falsifiable invariants; it cannot establish production reliability without real workload traces and controlled deployment evidence.
Practice and verification
Run the complete local lab
Save lab.py and run python3 lab.py with Python3.11+; only the standard library is needed, with no random seed because all inputs are deterministic. This is a protocol-and-arithmetic lab, not a neural network or full production SSE library. It feeds a synthetic UTF-8 response at every byte-fragment size, verifies three newline forms, and rejects truncated, malformed and failed streams. The delta/done/error event schema is ours, not an API compatibility claim.
Download Python lab"""Original teaching protocol, not a provider SDK. Python 3.11+, stdlib only.
No network, keys, GPU, or randomness. Feed arbitrary byte fragments.
Supports LF, CRLF and CR lines, UTF-8 BOM, comments and multiline data.
Does not implement EventSource reconnect, retry or Last-Event-ID semantics.
"""
import codecs
import json
import math
class Stream:
def __init__(self, max_event=65536):
self.decoder = codecs.getincrementaldecoder('utf-8-sig')('strict')
self.line = ''
self.data = []
self.event = ''
self.cr = False
self.size = 0
self.max_event = max_event
self.text = ''
self.state = 'waiting'
def feed(self, chunk):
self.consume(self.decoder.decode(chunk))
def consume(self, text):
for c in text:
if c == '\n' and self.cr:
self.cr = False
continue
self.cr = False
if c in '\r\n':
self.on_line(self.line)
self.line = ''
self.cr = c == '\r'
else:
self.line += c
if len(self.line.encode('utf-8')) > self.max_event:
raise ValueError('line too large')
def on_line(self, line):
if not line:
self.dispatch()
self.data, self.event, self.size = [], '', 0
return
if line.startswith(':'):
return
field, _, value = line.partition(':')
if value.startswith(' '):
value = value[1:]
if field == 'event':
self.event = value
elif field == 'data':
self.size += len(value.encode('utf-8')) + 1
if self.size > self.max_event:
raise ValueError('event too large')
self.data.append(value)
def dispatch(self):
if not self.data:
return
if self.state in {'completed', 'failed'}:
raise ValueError('data after terminal state')
payload = json.loads('\n'.join(self.data))
if not isinstance(payload, dict):
raise ValueError('payload must be object')
if self.event == 'delta':
if not isinstance(payload.get('text'), str):
raise ValueError('delta.text must be string')
self.text += payload['text']
self.state = 'streaming'
elif self.event == 'done':
if payload.get('reason') != 'stop':
raise ValueError('not an accepted completion')
self.state = 'completed'
elif self.event == 'error':
self.state = 'failed'
else:
raise ValueError('unknown teaching event')
def finish(self):
self.consume(self.decoder.decode(b'', final=True))
if self.line or self.data or self.event:
raise ValueError('unfinished event')
if self.state != 'completed':
raise ValueError('no successful completion')
return self.text
def softmax(logits):
if not logits or not all(math.isfinite(x) for x in logits):
raise ValueError('finite nonempty logits required')
m = max(logits)
values = [math.exp(x - m) for x in logits]
return [v / sum(values) for v in values]
def decode(raw, width):
s = Stream()
for start in range(0, len(raw), width):
s.feed(raw[start:start+width])
return s.finish()
def must_fail(raw):
try:
decode(raw, 1)
except (ValueError, UnicodeDecodeError):
return
raise AssertionError('invalid stream accepted')
if __name__ == '__main__':
text = 'Pending:GPU 不足'
wire = (': heartbeat\n\nevent: delta\ndata: ' +
json.dumps({'text': text}, ensure_ascii=False) +
'\n\nevent: done\ndata: {"reason":"stop"}\n\n')
raw = wire.encode()
for newline in ['\n', '\r\n', '\r']:
variant = b'\xef\xbb\xbf' + wire.replace('\n', newline).encode()
for width in range(1, len(variant)+1):
assert decode(variant, width) == text
must_fail(raw.split(b'event: done')[0])
must_fail(raw[:-1])
must_fail(b'event: error\ndata: {"error":"overloaded"}\n\n')
must_fail(b'event: delta\ndata: {bad}\n\n')
must_fail(b'event: delta\ndata: {"text":3}\n\n')
must_fail(b'\xff')
must_fail(raw + b'event: delta\ndata: {"text":"late"}\n\n')
try:
Stream(max_event=4).feed(b'data: oversized')
except ValueError:
pass
else:
raise AssertionError('size limit failed')
p = softmax([2,1,0])
assert abs(sum(p)-1) < 1e-12
assert all(abs(a-b)<1e-12 for a,b in zip(p,softmax([1002,1001,1000])))
kv = 2*32*1*8192*8*128*2
assert kv == 1024**3
cost = (2000*2+500*8)/1e6
assert abs(cost-0.008) < 1e-12
print(text)
print('softmax:', ', '.join(f'{x:.4f}' for x in p))
print(f'KV: {kv/1024**3:.1f} GiB; hypothetical cost: ${cost:.3f}')
print('PASS: byte fragmentation, newline variants, truncation, errors, bounds and arithmetic')
python3 lab.py
# Pending:GPU 不足
# softmax: 0.6652, 0.2447, 0.0900
# KV: 1.0 GiB; hypothetical cost: $0.008
# PASS: byte fragmentation, newline variants, truncation, errors, bounds and arithmeticFor a fair comparison, keep the exact response bytes fixed and vary only fragmentation. A naive JSON-per-read parser fails without any model change. Separately hold the logits fixed and add a constant: stable softmax must remain unchanged. Remove the done event and verify that partial text never becomes success. This isolates transport and numeric correctness; it does not evaluate factual answers or model quality.
Original interview questions and grading
1. Why can HTTP200 still produce a failed task?
Hint: three contracts. Transport admission is separate from stream completion and business acceptance. Detect terminal/error events, preserve partial state and validate the result against evidence.
2. Why does increasing context reduce serving capacity?
Hint: KV bytes and prefill. Calculate cache per active request, reserve weights/activations, then account for queue and compute. Doubling T doubles this simple KV estimate but not necessarily end-to-end latency.
3. Can you retry a timed-out restart safely?
Hint: uncertain side effects. Query the executor operation ledger; compare fingerprint and resource revision. Replay a recorded result, reject conflicting reuse, and reconcile unknown outcomes before another mutation.
4. Does prefix caching remove output generation cost?
Hint: distinguish three caches. KV holds execution state, prefix caching reuses compatible prefill, response caching returns an earlier answer. Only the last avoids generation for a hit, and it requires tenant, freshness and request compatibility checks.
Score each answer1–5: name the component; define its contract; explain the mechanism; handle a counterexample and failure; design an observable, fair validation with limits. These are original exercises, not reported company interview questions.
References and next tutorials
- Chat templates — Hugging Face Transformers (main documentation, checked 2026-11-11)https://huggingface.co/docs/transformers/main/en/chat_templating
- Attention Is All You Need — Vaswani et al., first submitted 2017-06-12; arXiv v7 2023-08-02https://arxiv.org/abs/1706.03762v7
- KV cache strategies — Hugging Face Transformers (main documentation)https://huggingface.co/docs/transformers/main/en/kv_cache
- PagedAttention — Kwon et al., arXiv v1 2023-09-12https://arxiv.org/abs/2309.06180v1
- Automatic prefix caching — vLLM documentationhttps://docs.vllm.ai/en/latest/features/automatic_prefix_caching/
- Server-sent events — WHATWG HTML standardhttps://html.spec.whatwg.org/multipage/server-sent-events.html
- Streaming messages — Claude Platform documentationhttps://platform.claude.com/docs/en/build-with-claude/streaming
- Building effective agents — Anthropic, 2024-12-19https://www.anthropic.com/research/building-effective-agents
- Attention Is All You Need — PDF v7https://arxiv.org/pdf/1706.03762v7
- PagedAttention — PDF v1https://arxiv.org/pdf/2309.06180v1
- vLLM — source codehttps://github.com/vllm-project/vllm
- HTTP Semantics — RFC 9110, June 2022https://www.rfc-editor.org/rfc/rfc9110.html
- Caching — Hugging Face Transformers (main documentation)https://huggingface.co/docs/transformers/main/en/cache_explanation