AI on Kubernetes: from raw corpus to an API that generates tokens
Build the data, training, checkpoint, release and inference lifecycle. Includes a tested CPU lab, optional GPT-2 training, Kubernetes manifests and GPU serving boundaries.
Sources & version details
In this article
Architecture and responsibility boundaries
| Stage | Input → output | Owner and failure boundary |
|---|---|---|
| Ingestion | Source records → immutable raw snapshot | Data owner: reject unlicensed or malformed records; quarantine rather than silently drop. |
| Preparation | Snapshot → split manifest + tokenizer | Data engineering: schema, duplicate families and leakage boundaries; retries must preserve the same split. |
| Training | Versioned examples + config → checkpoint | Training owner: all ranks recover together; incomplete checkpoint is never selected. |
| Release | Candidate + evidence → immutable approved revision | Model owner: reject candidate without replacing current serving revision; record gate version. |
| Serving | Approved weights/tokenizer → ready replica | Platform: verify digests before loading, then warm up; failure leaves the replica out of service. |
| Request | Authorized prompt + token budget → streamed output | Gateway/engine: bound queue and deadline, propagate cancellation; no transparent replay of observed output. |
Start with a service, not a pile of tools
We will build a private text-completion service whose model is trained from a versioned corpus. The offline path turns permitted source records into a dataset, tokenizer, training run, checkpoint and approved release. The online path authenticates a caller, checks a request budget, tokenizes the prompt, schedules model execution and returns generated tokens. Kubernetes schedules processes; it does not automatically supply data quality, model quality or safe authority. Each of those needs an explicit owner and artifact.
The downloadable kit closes that lifecycle with a deliberately small learned bigram model. It was run locally, including training, exact checkpoint continuation, release verification, HTTP and streaming responses. A separate two-layer GPT-2 training implementation and vLLM reference show the Transformer transition, but have not been executed here. Docker, a cluster and GPUs are unavailable in this environment. The article distinguishes a buildable CPU lab from the additional work required for a production GPU platform; neither small model is a useful assistant.
Follow one record through the system: document d17 enters raw snapshot R1; cleaning revision C2 creates dataset D3; tokenizer T1 maps its text to integers; run J4 produces checkpoint K400 and export M1. Release R7 binds M1, T1, the serving image digest and generation limits. Request q9 resolves the public model alias to R7 once and records that identity. The control plane changes desired versions and policies; the data plane moves bytes and runs computation. A model registry stores references and evidence, not a live GPU process. A Service routes to replicas, not to a training checkpoint. Keeping these distinctions prevents an alias update from being mistaken for a completed rollout.
The prerequisites explain the failure modes
A container image packages code and user-space libraries, not a compatible host GPU driver. A Pod is a scheduling and restart unit; a Job tracks completion; a Deployment maintains serving replicas; a Service selects endpoints. A PVC asks for persistent storage, whose access mode and topology affect placement. Learn these distinctions before treating a restart as recovery: a restarted process without its data cursor or optimizer state may repeat data or lose learning progress.
For the model, understand a token as an integer under a specific tokenizer, logits as unnormalized next-token scores, softmax as their probability distribution, and cross-entropy as a training loss. An optimizer updates weights; inference uses fixed weights. A checkpoint for resuming training contains more state than an inference export. Storage consistency, HTTP timeouts, TLS, identity and metrics matter as much as matrix multiplication because they determine whether the right model answers the right caller.
When kubectl submits a Job, the API server authenticates and authorizes the caller, admission checks the object, and accepted desired state is persisted. Controllers create Pods; the scheduler chooses a feasible node using resource requests, constraints and volume topology. Kubelet asks the container runtime to start the image and reports status. Device plugins advertise GPU resources; CSI drivers provision and attach volumes. None of these components runs the optimizer on behalf of the container. A Pod may be Running while the model is still loading: readiness is the separate signal used for ordinary Service endpoints. A startup probe protects slow initialization from premature liveness restarts; a readiness failure removes service eligibility without itself restarting the container.
Ingestion: immutable records before training windows
Give every source an ID, permission/license, acquisition time, content hash and retention rule. Preserve a raw immutable snapshot, then record the cleaning transformation and its code version. Parse tables, code and document boundaries intentionally; removing whitespace indiscriminately can change meaning. Deduplicate exact bytes and near duplicates as separate operations. Split by source, user or time before producing overlapping windows, otherwise nearly identical passages can leak into validation. A hash split is deterministic but does not guarantee that two related documents land together.
The kit accepts original CC0 synthetic JSONL, normalizes whitespace, removes exact duplicates and splits by content hash. Its manifest records source and prepared-data hashes. This is intentionally narrower than a production data platform. Production adds PII handling, consent, malicious-document quarantine, language/source balance and tombstones propagated into derived shards. Data versioning tools or an object-store catalog record lineage; they do not decide whether training is permitted. A deletion may require retiring datasets and retraining affected models; deleting one object does not unlearn weights.
A production ingestion worker should acknowledge a source offset only after the corresponding immutable object and manifest entry are durably committed under a defined protocol. A retry can write the same content-addressed object again; a different transformation version must create a different derived identity. Content hashes detect byte equality, not semantic similarity, consent or quality. Near-duplicate clustering must precede a group split when related documents must stay together. For a teaching split with validation when hash modulo 5 is zero, the expected validation fraction is 20%, not exactly 20% for a finite corpus. A 100 GB raw snapshot plus a 60 GB derived representation costs at least 160 GB before replicas, checkpoints and retained older versions. Streaming shards bound worker memory, but compression, shard size and access order jointly determine CPU cost and read amplification.
Tokenizer, packing and the training contract
A model release must include its exact vocabulary, normalization, special-token IDs and, for chat, message template. Training with one tokenizer and serving with another can silently change every input ID. In the Transformer path the tokenizer learns byte-level BPE from the training split only. Each record ends with EOS; padding is masked out of the loss. Packing multiple records into a sequence improves utilization but requires an explicit policy for whether attention may cross record boundaries.
For a batch of B sequences of length T, input IDs and labels have shape B×T; logits have shape B×T×V. Causal language modeling predicts the next token, not the current token. The supplied Hugging Face model shifts labels internally, so labels start as the same IDs and padding positions become −100. Do not also shift them in the data loader. Validation uses the same tokenizer and masking contract; report loss per valid token rather than averaging unequal batches without weights.
For an explicit arithmetic example, let a demonstration tokenizer encode “AI helps” as [17, 42] and EOS as 1. The input [17, 42, 1] creates two supervised transitions: 17→42 and [17,42]→1. These IDs are invented for explanation, not the tokenizer IDs shipped in the kit. With batch size 4, sequence length 128 and vocabulary 512, logits contain 262,144 scalars, or 1 MiB in FP32 before gradients. Embedding lookup turns integer IDs into B×T×d vectors; positional information distinguishes token order. Packing two documents into one row must explicitly decide whether the second may attend to the first. EOS is a learned boundary token, not an automatic attention firewall. Masking padding in the loss is also distinct from masking it in attention.
What training actually computes
The forward pass embeds tokens, applies causal layers and produces logits. The loss penalizes low probability on the observed next token. Backpropagation computes derivatives; AdamW maintains moving first and second moments and updates weights. Gradient accumulation spreads an effective batch over several microbatches, but the loss must be scaled consistently. For example, microbatch 2×accumulation 8×data-parallel 4 gives 64 sequences per update; that does not imply an equal number of tokens when lengths vary.
Our measured baseline uses a V×V trainable logit table: only the immediately previous character affects the next one. It is a genuine optimized autoregressive model, but lacks attention and long context. The optional GPT-2 path instead has 2 causal layers,4 heads and width 128. Its export demonstrates model/tokenizer packaging, not high-quality language learning. At scale, separate pretraining, supervised fine-tuning, preference optimization and RL rollouts: they use different data, reward or reference-model state and resource patterns. A rollout service is not the same workload as an optimizer Job.
In the loss, b indexes sequences, t indexes next-token predictions, m is 1 for a valid target and 0 for padding, and N_valid is the sum of m. In effective batch size, B_micro counts sequences per replica, A is accumulation steps and D is data-parallel replicas. Averaging each microbatch equally is only equivalent to token averaging when valid-token counts match.
One causal head forms
In these formulas, M is the causal mask (zero for allowed positions, −∞ otherwise), d_h is head width, z_j is logit j and y is the target ID. AdamW uses gradient g_t, moment coefficients β₁/β₂, learning rate η, weight-decay coefficient λ_w and numerical stabilizer ε. Moments and their squares are elementwise. The AdamW equation shows the common AdamW update with bias correction and no AMSGrad; implementation details and finite precision can change numerical results.
Scale training without confusing parallelism with scheduling
Data parallelism replicates a model and synchronizes gradients; tensor parallelism partitions operations within layers; pipeline parallelism partitions layers; state sharding partitions parameters, gradients or optimizer state. They solve different memory and communication problems. Mixed-precision Adam training can require roughly 16 bytes per parameter for weights, gradients, a master copy and two moments under one common implementation; activations, buffers and fragmentation are extra. A7 B example therefore starts near 112 GB before activations, not the 14 GB of BF16 inference weights. This is a budget illustration, not a universal framework allocation.
A multi-node run needs all ranks to agree on world size, rendezvous and collective order. GPU locality and interconnect determine whether synchronization dominates. Kueue-style admission controls quotas and queues; gang scheduling helps avoid a job occupying half its needed workers indefinitely; neither changes the gradient algorithm. A failed rank can stall collectives, so define timeouts, coordinated termination and restart from a completed checkpoint. Do not assume increasing replicas on a Deployment creates a distributed trainer.
Under equal local batch sizes, each of D replicas computes a gradient g_r; an all-reduce sum followed by division by D produces the gradient of the combined batch. With unequal valid-token counts, equal replica weighting is wrong: normalize by global valid tokens. Ring all-reduce moves approximately 2(D−1)G/D bytes per rank for a G-byte gradient under an idealized ring, excluding latency and overlap. For
The ring estimate uses decimal bytes consistently: G is gradient bytes per replica, D the number of ranks, and B_net effective bytes per second. It is a lower-bound model, not an end-to-end timing prediction.
Kubernetes resources must match the two lifecycles
Use batch Jobs for bounded preparation and training, Deployments for persistent inference, and Services for stable internal names. CPU preparation, GPU training and latency-sensitive inference should have distinct resource pools and priorities. Request CPU and RAM as well as GPUs; a GPU pod starved of CPU parsing or network bandwidth may waste its accelerator. Drivers, a container runtime integration and a device plugin make GPU resources allocatable; the scheduler cannot manufacture GPU memory by overcommitting a normal extended-resource limit.
The CPU kit uses one namespace, one dynamically provisioned RWO claim, one training Job and one serving Deployment. A read-only serving mount and artifact hash check stop partial or tampered releases from becoming ready. Recreate is intentional because this small shared-volume lab is not a zero-downtime architecture. At production scale, stage immutable approved artifacts from object storage into independent pod caches and use readiness only after complete loading. Use topology-aware placement and storage; a volume available in one zone does not follow an arbitrary GPU into another zone.
The manifest requests one CPU for training, so scheduler placement accounts for that reservation; a limit of two CPUs bounds runtime consumption through the runtime/cgroup implementation. Memory requests guide placement, while exceeding the memory limit can lead to an OOM kill. A GPU extended resource counts devices, not bytes of free VRAM. If a Pod remains Pending, distinguish insufficient devices from incompatible node affinity and an unbound or topologically incompatible volume. If the container starts but CUDA fails, investigate driver/runtime/device access, not only the scheduler. RWO permits a volume to be mounted read-write on one node; it does not mean one Pod, and it is not a cross-node shared filesystem. The lab uses one reader and one writer on a small volume, not a distributed dataset service.
Checkpoint, candidate and release are different objects
A resumable checkpoint includes weights, optimizer and scheduler state, step, random generators, data version and sampler position. Saving weights alone restarts optimization from a different state. Write into a staging location, complete all required shards and metadata, verify them, then publish a completion marker. An object store does not turn a multi-object write into an atomic transaction. Readers should select only completed manifests; failed uploads leave garbage to clean, not a half-valid checkpoint.
An inference export strips training-only state but retains architecture, tokenizer, adapters, precision, weight files, checksums and license lineage. The registry is a catalog of these immutable artifacts and their approval evidence; it need not initially be a separate commercial service. A release points to a specific candidate, serving image and configuration. Use a distinct pointer for each environment. Rollback selects a previous compatible tuple, rather than changing only a model name while leaving an incompatible tokenizer or quantization kernel deployed.
An object-store checkpoint writer first writes uniquely named shards, then hashes them, then writes a manifest listing every expected shard and its size/hash. Only after completeness checks does it update a small “latest complete” pointer with a conditional write or equivalent concurrency control. A reader fetches one manifest and verifies all referenced shards; it never discovers a release by listing whichever files happen to exist. A crash before the manifest leaves garbage to collect, not an approved checkpoint. For a local filesystem the kit uses a temporary file and rename, which avoids exposing a partially written file; durability under power loss additionally depends on filesystem and directory-sync semantics. Export and admission are separate: a valid file can still contain a poor model. Rollback changes routing to the old complete bundle; it cannot undo tokens already delivered.
From model files to the first and subsequent tokens
At startup, fetch the approved artifact, verify hashes, load compatible weights, allocate execution buffers, initialize kernels and warm representative shapes before readiness. Prefill processes prompt tokens and creates cached keys and values. Decode then appends tokens autoregressively, reusing the cache. Continuous batching admits new sequences as older ones finish, improving utilization but complicating fairness and cancellation. Tokenization, queue delay and network buffering contribute to time-to-first-token even when kernels are fast.
For ordinary attention, a useful KV estimate is 2×layers×KV-heads×head-dimension×bytes×resident-tokens. In a synthetic 32-layer,8-KV-head,128-dimension,BF16 example, KV costs 131,072 bytes/token:8,192 resident tokens cost 1 GiB, before metadata and allocation overhead. Grouped-query attention changes KV heads, not necessarily query heads. A long prompt can exhaust cache even when model weights fit. Weight quantization and KV quantization are separate choices with separate quality and kernel support.
In the memory equation, L is layer count, H_KV is KV heads per layer, d_h is head width, b is bytes per cached scalar, and T_i is the resident token count of sequence i. The factor 2 represents both keys and values. This dense-attention estimate excludes allocator overhead and does not directly describe latent-attention or compressed-cache architectures.
Prefill for a T-token prompt computes its representations and stores each layer’s K and V. Decode adds one new query and attends to cached keys; it avoids recomputing previous K/V, but still reads the growing cache. Dense attention has roughly quadratic prompt interaction work and linear-in-context work per decode step, excluding projections and feed-forward computation. Paged KV allocation maps logical token blocks to physical blocks instead of requiring one contiguous maximum-length allocation. With block size16 and sequence lengths17 and31, two sequences occupy64 token slots, of which16 are unused; contiguous reservations of128 each would allocate256 slots. This is allocation arithmetic, not a throughput benchmark. A shared prefix can reuse blocks only when model, tokenizer, positions and relevant execution settings match; reference counts must prevent freeing a block still used by another request.
In the allocation equation, P is tokens per physical block and T_i is sequence length; ceiling rounds each sequence up to a whole block. The example excludes cross-sequence prefix sharing and block metadata.
The API path is a controlled state machine
The production path is client→TLS gateway→identity and tenant policy→quota/admission→model router→private inference service→stream. Validate body size, model permission, tokenizer-specific input length and reserved output tokens before admitting GPU work. A request ID connects gateway and engine events. A model alias resolves to an immutable release for the duration of the request; changing the alias must not switch weights midway through an existing stream.
Streaming uses incremental events, not one JSON document per entire connection. Disable unwanted proxy buffering and set compatible idle deadlines. Once response bytes reach a caller, a transparent retry can duplicate text or bill work twice. Retry only before disclosure under a documented policy, or return an explicit resumability contract. Propagate disconnect and cancellation to the engine so KV and admission slots are released. A tool call in generated text is a proposal, not permission to execute a payment or mutation.
The gateway must validate a credential rather than trust a caller-supplied tenant field. Resolve its subject to an allowed model and tenant budget, tokenize with the selected model’s tokenizer, and require input tokens plus reserved output tokens to fit the context window. A reservation protects scarce serving capacity; final usage reconciles actual generated tokens. In SSE, each event is terminated by a blank line and may be split across TCP reads, so the client buffers bytes until complete events. HTTP success headers can precede a later generation failure; record both transport status and generation termination reason. Stop conditions include EOS, output limit, deadline and cancellation. Temperature divides logits before sampling; low temperature sharpens a distribution but does not prove factual accuracy.
Capacity, autoscaling and rollout economics
Scale on workload, not only GPU utilization. Track queue depth and age, running sequences, input/output tokens, cache occupancy, first-token latency and inter-token latency. A saturated GPU can be healthy batch work or an overloaded interactive service. For a stable system Little’s law relates in-flight requests to arrival rate times mean residence time; it does not predict tail latency near saturation. If 10 requests/second each remain 2 seconds, the mean is 20 in flight, but context length and output limits still determine memory demand.
Cold starts include scheduling, image pull, artifact download, weight loading and warmup. HPA cannot erase these delays; reserve headroom, prewarm popular models and bound queues. Canary a new release on a small fraction of traffic with comparable prompt/output distributions, then increase deliberately. A PDB constrains voluntary disruptions, not every GPU failure; graceful drain stops admission and lets bounded streams finish. A one-GPU Recreate lab has no spare capacity for zero-downtime updates. Record idle, failed and rollout capacity when calculating cost per accepted request.
Separate time to first token into ingress, authentication/tokenization, queueing, prefill and first-token transport. A synthetic budget of20+10+80+150+10 ms totals270 ms. Optimizing prefill by30 ms cannot fix a queue that grows by seconds under overload. Inter-token latency describes intervals after the first token; average generation throughput can improve while a particular user waits longer due to batching. For an illustrative M/M/1 queue with arrival rate λ and service rate μ, mean residence time is 1/(μ−λ);
Security and observability span every stage
Separate data-ingestion identity, training identity, registry promotion authority and serving identity. A serving pod should read an approved artifact, not overwrite training data or promote its own weights. Use workload identity and short-lived credentials, prevent secrets in images and logs, restrict egress, and audit release changes. NetworkPolicy limits network reachability when the CNI enforces it; it does not authenticate an API user. A service account token is unnecessary for a pod that never calls the Kubernetes API.
Connect dataset hash, run ID, checkpoint, model release, serving image and request ID in metadata. Measure data rejection, training throughput and loss, checkpoint duration, GPU errors, queue time, token timings, cancellation and rejection counts. Do not label metrics with arbitrary prompts or request IDs: that creates high cardinality and privacy risk. Keep detailed traces in a controlled store with sampling and retention. Separate observed failures from missing telemetry, and distinguish an availability SLO from correctness and safety assessments.
Follow authority separately from data: a trainer may read D3 and write only its run prefix; a release controller may approve a candidate; a serving identity may read only approved bundles. Namespace labels are not tenant authentication. NetworkPolicy restricts connections, while an IAM policy authorizes an object operation; both are needed when storage is external. Observability follows the same chain with spans for queue, prefill and decode, counters for accepted/rejected requests, histograms for latency and gauges for resident capacity. Trace IDs belong in sampled traces or exemplars, not unbounded metric labels. Store the model revision with each completed response so a latency or quality regression can be attributed. Missing spans can mean sampling, dropped telemetry or unfinished work; absence alone is not proof that an operation never happened.
Choose a stack by control point and unresolved constraint
A managed cloud can supply nodes, storage and identity integration, while the platform team still owns dataset contracts and model acceptance. NVIDIA’s GPU software sits near device provisioning and execution; PyTorch near training; Kubernetes near workload reconciliation; vLLM near serving and KV scheduling. MLflow-style registries catalog experiments and releases; object stores retain bytes; gateways own external identity and admission. These layers overlap but are not interchangeable. Avoid buying several dashboards while leaving release authority and rollback ownership undefined.
Frontier designs split prefill and decode, transfer KV across machines, route by prefix locality and offload cache. They may improve resource matching but add network, consistency and cancellation paths. Multi-tenant adapter serving adds adapter-version isolation; quantization adds calibration and kernel constraints; speculative decoding adds draft work and an acceptance algorithm. None is free throughput. Start with the single-node invariant-preserving service, measure its bottleneck, then change one boundary at a time. Persistent open problems include portable performance, fair multi-tenant scheduling, auditable training data deletion, reliable long-context quality and efficient recovery at cluster scale.
Choose optimizations by the resource they move. Quantization reduces weight storage or arithmetic precision but needs calibration and supported kernels; it may not shrink the unquantized KV cache. Prefix caching saves repeated prefill only on compatible shared prefixes, not arbitrary semantically similar text. Prefill/decode disaggregation moves compute to different pools but introduces KV transfer: transferring a1 GiB cache across an effective10 GiB/s link takes at least0.1 s before protocol and contention costs. Speculative decoding proposes several draft tokens and verifies them with the target; savings depend on acceptance and verification cost, not draft speed alone. Each technique adds a new state owner or compatibility constraint. A useful design review names the old bottleneck, the transferred work, the new failure path and the conditions under which the simpler design wins.
Practice and verification
Download the complete source kit
1. Run the complete CPU lifecycle
Use Python 3.11 and an isolated environment. Unpack the kit and work inside its directory. It contains 100 original CC0 synthetic records; this is a pipeline fixture, not a useful language corpus. Preparation deduplicates normalized records and assigns train/validation by stable hash before token pairs are created. The seed is 42. Training minimizes next-character cross-entropy with SGD; it does not implement attention. The release gate checks artifact hashes and validation NLL below 4 for this fixture only.
python3.11 -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python pipeline.py prepare --source corpus.jsonl --root ./run
python pipeline.py train --root ./run --steps 400
python pipeline.py promote --root ./run
python serve.py --root ./run --port 8000# In another terminal
curl -s http://127.0.0.1:8000/health
curl -s http://127.0.0.1:8000/v1/completions -H 'Content-Type: application/json' \
-d '{"prompt":"A model","max_tokens":16}'
curl -N http://127.0.0.1:8000/v1/completions -H 'Content-Type: application/json' \
-d '{"prompt":"A model","max_tokens":16,"stream":true}'2. Build and schedule the CPU version
Prerequisites: Docker, kubectl configured for your own cluster, a default StorageClass that supports fsGroup, and a NetworkPolicy-capable CNI. A kind cluster can load the local image; other clusters require pushing it to a registry and changing both image fields. The namespace is restricted, containers run as UID 10001, API tokens are not mounted, and the Service is private. The single RWO PVC and Recreate strategy deliberately trade availability for simplicity. Server restarts before the first release exists are expected; production should sequence deployment after approval. Do not publish this unauthenticated lab to the internet.
docker build -t tokenlab-ai:local .
# For kind only; otherwise push image to your registry and edit YAML
kind load docker-image tokenlab-ai:local
kubectl apply -f kubernetes.yaml
kubectl -n tokenlab-ai wait --for=condition=complete job/train-v1 --timeout=600s
kubectl -n tokenlab-ai rollout status deployment/model-v1 --timeout=300s
kubectl -n tokenlab-ai port-forward service/model-api 8000:8000
# Then run the curl commands above; Ctrl-C stops the local tunnel.3. Replace bigram training with a real Transformer
The optional path trains a fresh byte-BPE tokenizer on training records only and a random two-layer GPT-2: four heads, width 128, context 128, AdamW at 3e-4. Padding labels are -100; GPT-2 shifts labels internally. Do not shift twice. It saves optimizer and RNG states, exports safetensors plus tokenizer/config, and computes token-weighted held-out NLL before creating approved/. The fixed gate is only a synthetic acceptance example. It refuses to overwrite an existing approved release. These dependencies and this execution path have not been run here. Use the complete scripts below rather than fragments.
# Separate environment recommended; optional path not executed here
pip install -r requirements-transformer.txt
python transformer.py --root ./run --steps 200
python transformer.py --root ./run --steps 400 --resume
python approve_transformer.py --root ./run --max-nll 4.0
# Optional training Job (existing namespace/PVC required)
docker build -f Dockerfile.transformer -t tokenlab-transformer:lab .
kind load docker-image tokenlab-transformer:lab
kubectl -n tokenlab-ai scale deployment model-v1 --replicas=0
kubectl apply -f transformer-job.yaml
kubectl -n tokenlab-ai wait --for=condition=complete job/transformer-train --timeout=7200s4. GPU inference: a separately validated integration
The optional training Job uses the same artifacts PVC. Stop the CPU Deployment first to avoid cross-node RWO attachment conflicts. The vLLM reference mounts the approved export and exposes a private Service; GPT2LMHeadModel is listed in vLLM 0.14.0 support documentation, but this particular export has not been compatibility-tested. Before applying, adapt image UID, writable caches and security context to restricted admission; do not weaken the namespace policy. Provision a compatible NVIDIA driver/device plugin, GPU and attachable storage. The README names the exact remaining checks. This is not a turnkey production GPU distribution.
Measured results, failure cases and fair comparisons
On the supplied CPU fixture, initial training NLL was 3.73767 nats/token; after 400 steps it was 1.70074, with held-out NLL 1.70306. These are bigram losses, not Transformer benchmarks. The verifier compares uninterrupted training with 200+200 resumed steps byte-for-byte, checks HTTP and SSE termination, rejects invalid token budgets, and detects altered artifacts. Repeat on your machine; do not use these numbers to compare vendors.
python verify.py
python mechanism_math.py
# PASS: loss descent, exact resume, artifact integrity, HTTP generation, SSE, invalid inputFor cluster acceptance, kill the trainer after a checkpoint, corrupt a candidate without changing the current release, send an unknown character, exhaust concurrency, disconnect an SSE client, and make a new replica unready. Verify old service continuity where the chosen deployment strategy permits it. Compare equal datasets, tokenizers, precision, input/output lengths, arrival traces and hardware; report queue delay, TTFT, inter-token latency, failures, cost and quality together. A faster failed request is not an improvement. The single-replica teaching deployment cannot demonstrate zero-downtime failover.
Original interview exercises · not reported company questions
1. Why can a resumed run diverge even when weights were restored?
Hint 1
Think about Adam and the next batch.
Hint 2
RNG, optimizer moments, scheduler, data cursor and parallel topology influence the next update.
Answer and analysis
A model export is not a resumable checkpoint. Restore the dataset fingerprint, optimizer and RNG state, then define whether exact or statistically equivalent continuation is required. Exact cross-hardware GPU equality is not guaranteed. Compare a fixed uninterrupted run against an interrupted run; never infer equality merely from a successful load.
Scoring: 1 — name the component; 2 — explain its state; 3 — trace the failure; 4 — propose a correct recovery with limits; 5 — design a fair counterexample and justify the trade-off.
2. Why can a 14 GB model fail on a 24 GB GPU?
Hint 1
Weights are only one memory category.
Hint 2
Include KV cache, activation workspace, runtime reservations and fragmentation.
Answer and analysis
With 32 layers, 8 KV heads, head dimension 128 and BF16, KV uses 128 KiB per token. Eight sequences of 8192 tokens require 8 GiB just for KV; adding 14 decimal GB weights leaves little safe space on a nominal 24 GB device. Limit admitted token budgets and measure peak allocation. Quantizing weights alone does not necessarily quantize KV.
Scoring: 1 — name the component; 2 — explain its state; 3 — trace the failure; 4 — propose a correct recovery with limits; 5 — design a fair counterexample and justify the trade-off.
3. Should a gateway retry after streaming the first token?
Hint 1
Has the client observed an output already?
Hint 2
Restarting generation may duplicate or change text and billing.
Answer and analysis
Do not transparently replay an observed stream. Mark it interrupted and let the client explicitly start another request, unless a designed resumable protocol can replay the same buffered event sequence. Before output, retry only bounded eligible failures with an overall deadline. On disconnect, cancel engine work and release KV capacity. Request IDs correlate evidence; they do not automatically make generation idempotent.
Scoring: 1 — name the component; 2 — explain its state; 3 — trace the failure; 4 — propose a correct recovery with limits; 5 — design a fair counterexample and justify the trade-off.
Complete implementation files (shared English code comments)
mechanism_math.py
"""Deterministic checks for the article's synthetic arithmetic. Standard library only.
No cluster, model benchmark, GPU, network or stochastic experiment.
"""
import math
p0=0.5
loss0=-math.log(p0)
loss1=math.log(math.exp(-0.05)+math.exp(0.05))-0.05
assert abs(loss0-0.69314718056)<1e-10
assert abs(loss1-0.64439666007)<1e-10 and loss1<loss0
# First AdamW moment correction with a scalar gradient.
g=0.5;beta1=0.9;beta2=0.999
m=(1-beta1)*g;v=(1-beta2)*g*g
assert math.isclose(m/(1-beta1),g)
assert math.isclose(v/(1-beta2),g*g)
assert 4*128*512*4==1024**2
assert 2*(4-1)*8/4/25==0.48
slots=16*sum(math.ceil(t/16) for t in [17,31])
assert slots==64 and slots-17-31==16
assert 2*32*8*128*2*8192==1024**3
assert sum([20,10,80,150,10])==270
assert 1/(10-8)==0.5 and 1/(10-9.5)==2
print('PASS: gradient step, AdamW bias correction, logits bytes, ring budget, KV blocks, TTFT and queue arithmetic')
pipeline.py
"""Transparent language-model lifecycle. Python 3.11 + numpy 2.3.5.
The model is a learned bigram softmax, not a Transformer or general assistant.
"""
import argparse, hashlib, json, os, tempfile
from pathlib import Path
import numpy as np
def digest(b): return hashlib.sha256(b).hexdigest()
def atomic(path, data):
path=Path(path); path.parent.mkdir(parents=True,exist_ok=True)
fd,tmp=tempfile.mkstemp(dir=path.parent)
with os.fdopen(fd,'wb') as f: f.write(data); f.flush(); os.fsync(f.fileno())
os.replace(tmp,path)
def prepare(source,out):
records=[json.loads(x) for x in Path(source).read_text().splitlines() if x.strip()]
seen=set(); groups={'train':[],'validation':[]}
for r in records:
if r.get('license')!='CC0' or not isinstance(r.get('text'),str): raise ValueError('unsupported rights/schema')
text=' '.join(r['text'].split())
if not text: raise ValueError('empty text')
h=digest(text.encode())
if h in seen: continue
seen.add(h)
# Split before windows; one source record never crosses splits.
groups['validation' if int(h[:8],16)%5==0 else 'train'].append({'id':r['id'],'text':text,'sha256':h})
if not all(groups.values()): raise ValueError('both splits required')
payload=json.dumps(groups,sort_keys=True).encode()
atomic(Path(out)/'dataset.json',payload)
atomic(Path(out)/'data-manifest.json',json.dumps({'sha256':digest(payload),'source_sha256':digest(Path(source).read_bytes()),'split':'content-hash modulo 5','counts':{k:len(v) for k,v in groups.items()}},indent=2).encode())
return groups
def pairs(records,vocab):
lookup={c:i+2 for i,c in enumerate(vocab)}; xs=[];ys=[]
for r in records:
ids=[0]+[lookup[c] for c in r['text']]+[1]
xs.extend(ids[:-1]);ys.extend(ids[1:])
return np.array(xs),np.array(ys)
def loss(w,x,y):
z=w[x]; z=z-z.max(axis=1,keepdims=True)
return float(np.mean(np.log(np.exp(z).sum(axis=1))-z[np.arange(len(x)),y]))
def train(root,steps=400,resume=False):
root=Path(root); raw=(root/'dataset.json').read_bytes();ds=json.loads(raw)
# Vocabulary belongs to training data. Fixture is ASCII; production needs UNK policy.
vocab=sorted(set(''.join(r['text'] for r in ds['train'])))
if any(c not in vocab for r in ds['validation'] for c in r['text']): raise ValueError('validation has unknown characters')
x,y=pairs(ds['train'],vocab); vx,vy=pairs(ds['validation'],vocab)
rng=np.random.default_rng(42);w=np.zeros((len(vocab)+2,)*2);start=0
cp=root/'checkpoint.npz'
if resume and cp.exists():
with np.load(cp,allow_pickle=False) as c:
if str(c['data_hash'])!=digest(raw): raise ValueError('dataset changed during resume')
w=c['weights'];start=int(c['step']);rng.bit_generator.state=json.loads(str(c['rng']))
initial=loss(w,x,y)
for step in range(start,steps):
ix=rng.integers(0,len(x),size=128); xx=x[ix];yy=y[ix]
z=w[xx];z-=z.max(axis=1,keepdims=True);p=np.exp(z);p/=p.sum(axis=1,keepdims=True)
p[np.arange(len(xx)),yy]-=1;g=np.zeros_like(w);np.add.at(g,xx,p/len(xx));w-=3*g
if (step+1)%50==0 or step+1==steps:
import io
buf=io.BytesIO();np.savez(buf,weights=w,step=step+1,rng=json.dumps(rng.bit_generator.state),data_hash=digest(raw));atomic(cp,buf.getvalue())
model={'architecture':'bigram-softmax-v1','vocab':vocab,'weights':w.tolist(),'data_sha256':digest(raw),'steps':steps,'seed':42}
modelraw=json.dumps(model,separators=(',',':')).encode();atomic(root/'model.json',modelraw)
metrics={'initial_train_nll':initial,'train_nll':loss(w,x,y),'validation_nll':loss(w,vx,vy),'model_sha256':digest(modelraw),'steps':steps}
atomic(root/'metrics.json',json.dumps(metrics,indent=2).encode());print(json.dumps(metrics));return metrics
def promote(root):
root=Path(root);m=json.loads((root/'metrics.json').read_text()); raw=(root/'model.json').read_bytes()
if digest(raw)!=m['model_sha256'] or not np.isfinite(m['validation_nll']) or m['validation_nll']>=4: raise ValueError('release gate failed')
# This threshold only validates the declared teaching fixture, not language quality.
atomic(root/'release.json',json.dumps({'model_sha256':digest(raw),'data_sha256':digest((root/'dataset.json').read_bytes()),'artifact':'model.json','gate':'fixture validation NLL < 4; not a safety evaluation'},indent=2).encode())
if __name__=='__main__':
p=argparse.ArgumentParser();p.add_argument('action',choices=['prepare','train','promote']);p.add_argument('--root',default='/work');p.add_argument('--source',default='corpus.jsonl');p.add_argument('--steps',type=int,default=400);p.add_argument('--resume',action='store_true');a=p.parse_args()
if a.action=='prepare': prepare(a.source,a.root)
elif a.action=='train': train(a.root,a.steps,a.resume)
else: promote(a.root)
serve.py
"""Local/private teaching API; no public ingress, TLS or identity provider."""
import argparse, hashlib, json, threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
import numpy as np
def load(root):
root=Path(root);release=json.loads((root/'release.json').read_text());raw=(root/'model.json').read_bytes()
if hashlib.sha256(raw).hexdigest()!=release['model_sha256']: raise ValueError('artifact integrity mismatch')
return json.loads(raw),release['model_sha256']
def application(model,revision):
w=np.array(model['weights']);vocab=model['vocab'];lookup={c:i+2 for i,c in enumerate(vocab)};slots=threading.BoundedSemaphore(2)
class Handler(BaseHTTPRequestHandler):
def log_message(self,*args): pass # No prompts in request logs.
def respond(self,status,obj):
raw=json.dumps(obj).encode();self.send_response(status);self.send_header('Content-Type','application/json');self.send_header('Content-Length',str(len(raw)));self.end_headers();self.wfile.write(raw)
def do_GET(self):
self.respond(200 if self.path=='/health' else 404,{'model_revision':revision})
def do_POST(self):
if self.path!='/v1/completions': return self.respond(404,{'error':'not found'})
self.connection.settimeout(10)
try:
size=int(self.headers.get('Content-Length','0'))
if not 0<size<=8192: return self.respond(413,{'error':'body limit'})
body=json.loads(self.rfile.read(size))
if not isinstance(body,dict): raise ValueError('object required')
prompt=body.get('prompt','');limit=body.get('max_tokens',32)
if not isinstance(prompt,str) or len(prompt)>512 or type(limit)!=int or not 1<=limit<=128: raise ValueError('invalid prompt or token limit')
if any(c not in lookup for c in prompt): raise ValueError('unknown character')
if type(body.get('stream',False))!=bool: raise ValueError('stream must be boolean')
except (ValueError,TypeError,TimeoutError): return self.respond(400,{'error':'invalid request'})
if not slots.acquire(blocking=False): return self.respond(429,{'error':'capacity'})
try:
token=lookup[prompt[-1]] if prompt else 0;rng=np.random.default_rng(42);out=[]
if body.get('stream'):
self.send_response(200);self.send_header('Content-Type','text/event-stream');self.send_header('Cache-Control','no-cache');self.send_header('Connection','close');self.end_headers();self.close_connection=True
for _ in range(limit):
z=w[token].copy();z[0]=-np.inf;p=np.exp(z-np.max(z));p/=p.sum();token=int(rng.choice(len(p),p=p))
if token==1: break
out.append(vocab[token-2])
if body.get('stream'):
self.wfile.write(('data: '+json.dumps({'token':out[-1],'model_revision':revision})+'\n\n').encode());self.wfile.flush()
if body.get('stream'): self.wfile.write(b'data: [DONE]\n\n');self.wfile.flush()
else: self.respond(200,{'text':''.join(out),'completion_tokens':len(out),'model_revision':revision})
except (BrokenPipeError,ConnectionResetError,TimeoutError): pass
finally: slots.release()
return Handler
if __name__=='__main__':
p=argparse.ArgumentParser();p.add_argument('--root',default='/work');p.add_argument('--host',default='127.0.0.1');p.add_argument('--port',type=int,default=8000);a=p.parse_args()
model,rev=load(a.root);ThreadingHTTPServer((a.host,a.port),application(model,rev)).serve_forever()
transformer.py
"""Optional real GPT-2 architecture training path; not run by Token Lab.
Python 3.11, torch 2.6.0, transformers 4.51.3, tokenizers 0.21.1.
Reads the SAME prepared dataset.json; trains its own byte BPE tokenizer.
A tiny random model demonstrates mechanics, not useful assistant capability.
"""
import argparse,json,os,random
from pathlib import Path
import torch
from tokenizers import ByteLevelBPETokenizer
from transformers import GPT2Config,GPT2LMHeadModel,PreTrainedTokenizerFast
p=argparse.ArgumentParser();p.add_argument('--root',default='/work');p.add_argument('--steps',type=int,default=200);p.add_argument('--resume',action='store_true');a=p.parse_args()
root=Path(a.root);ds=json.loads((root/'dataset.json').read_text());torch.manual_seed(42);random.seed(42)
# CPU default for broad compatibility; set DEVICE=cuda for a provisioned GPU.
device=os.environ.get('DEVICE','cpu');tokenizer=ByteLevelBPETokenizer()
tokenizer.train_from_iterator((r['text'] for r in ds['train']),vocab_size=512,min_frequency=2,special_tokens=['<bos>','<eos>','<pad>'])
tok=PreTrainedTokenizerFast(tokenizer_object=tokenizer._tokenizer,bos_token='<bos>',eos_token='<eos>',pad_token='<pad>')
config=GPT2Config(vocab_size=len(tok),n_positions=128,n_ctx=128,n_embd=128,n_layer=2,n_head=4,bos_token_id=tok.bos_token_id,eos_token_id=tok.eos_token_id,pad_token_id=tok.pad_token_id)
model=GPT2LMHeadModel(config).to(device);optimizer=torch.optim.AdamW(model.parameters(),lr=3e-4)
rows=[tok(r['text'],max_length=126,truncation=True)['input_ids']+[tok.eos_token_id] for r in ds['train']]
import hashlib
fingerprint=hashlib.sha256((root/'dataset.json').read_bytes()).hexdigest();cp=root/'transformer-checkpoint.pt';start=0
if a.resume and cp.exists():
state=torch.load(cp,map_location='cpu',weights_only=True)
if state['data_hash']!=fingerprint: raise ValueError('dataset mismatch')
model.load_state_dict(state['model']);optimizer.load_state_dict(state['optimizer']);start=state['step'];torch.set_rng_state(state['cpu_rng']);random.setstate(state['python_rng'])
if device.startswith('cuda'): torch.cuda.set_rng_state_all(state['cuda_rng'])
model.train()
for step in range(start,a.steps):
chosen=random.choices(rows,k=4);width=max(map(len,chosen));ids=torch.full((4,width),tok.pad_token_id,dtype=torch.long,device=device);mask=torch.zeros_like(ids)
for i,row in enumerate(chosen): ids[i,:len(row)]=torch.tensor(row,device=device);mask[i,:len(row)]=1
labels=ids.clone();labels[mask==0]=-100
optimizer.zero_grad(set_to_none=True);loss=model(input_ids=ids,attention_mask=mask,labels=labels).loss
loss.backward();torch.nn.utils.clip_grad_norm_(model.parameters(),1);optimizer.step()
if (step+1)%50==0 or step+1==a.steps:
state={'model':model.state_dict(),'optimizer':optimizer.state_dict(),'step':step+1,'cpu_rng':torch.get_rng_state(),'python_rng':random.getstate(),'cuda_rng':torch.cuda.get_rng_state_all() if device.startswith('cuda') else [],'data_hash':fingerprint}
tmp=root/'checkpoint.tmp';torch.save(state,tmp);os.replace(tmp,cp)
print(json.dumps({'step':step+1,'training_loss':float(loss.detach())}))
export=root/'transformer-candidate';export.mkdir(exist_ok=True)
model.save_pretrained(export,safe_serialization=True);tok.save_pretrained(export)
# Candidate only: no release pointer is changed. Evaluate before promotion.
(export/'lineage.json').write_text(json.dumps({'data_sha256':fingerprint,'seed':42,'steps':a.steps,'status':'candidate-not-approved'}))
approve_transformer.py
"""Held-out next-token NLL gate for this synthetic fixture only. Not a safety gate.
Unexecuted optional Transformer path. Uses CPU; no paid APIs or uploaded data.
"""
import argparse,hashlib,json,math,os,shutil
from pathlib import Path
import torch
from transformers import AutoTokenizer, GPT2LMHeadModel
p=argparse.ArgumentParser();p.add_argument('--root',default='/work');p.add_argument('--max-nll',type=float,required=True);a=p.parse_args()
r=Path(a.root);src=r/'transformer-candidate';dst=r/'approved'
if dst.exists(): raise ValueError('use a fresh release directory; never overwrite a served artifact')
data=json.loads((r/'dataset.json').read_text());lineage=json.loads((src/'lineage.json').read_text())
assert lineage['data_sha256']==hashlib.sha256((r/'dataset.json').read_bytes()).hexdigest()
tok=AutoTokenizer.from_pretrained(src,local_files_only=True);model=GPT2LMHeadModel.from_pretrained(src,local_files_only=True).eval()
total=0.;n=0
with torch.no_grad():
for row in data['validation']:
ids=tok(row['text'],max_length=126,truncation=True)['input_ids']+[tok.eos_token_id]
x=torch.tensor([ids]);loss=model(input_ids=x,labels=x).loss.item()
total+=loss*(len(ids)-1);n+=len(ids)-1
nll=total/n
print(json.dumps({'validation_nll':nll,'tokens':n,'threshold':a.max_nll}))
if not math.isfinite(nll) or nll>=a.max_nll: raise ValueError('candidate rejected; approved version unchanged')
files={f.name:hashlib.sha256(f.read_bytes()).hexdigest() for f in src.iterdir() if f.is_file()}
staging=r/'approval-staging'
if staging.exists(): shutil.rmtree(staging)
shutil.copytree(src,staging)
(staging/'release-manifest.json').write_text(json.dumps({'files':files,'validation_nll':nll,'gate':'synthetic fixture only','data_sha256':lineage['data_sha256']},indent=2))
os.replace(staging,dst) # Atomic directory rename on this one filesystem, not object-store transaction.
verify.py
"""Actual local training/resume/integrity/API checks; no cluster required."""
import importlib.util,json,tempfile,threading,urllib.request,urllib.error
from pathlib import Path
from http.server import ThreadingHTTPServer
import numpy as np
from pipeline import prepare,train,promote
from serve import load,application
here=Path(__file__).parent
with tempfile.TemporaryDirectory() as t:
a=Path(t)/'a';b=Path(t)/'b'
for root in (a,b):prepare(here/'corpus.jsonl',root)
metrics=train(a);train(b,200);train(b,400,True)
assert (a/'model.json').read_bytes()==(b/'model.json').read_bytes(),'resume differs'
assert metrics['train_nll']<metrics['initial_train_nll'];promote(a)
model,revision=load(a);server=ThreadingHTTPServer(('127.0.0.1',0),application(model,revision))
thread=threading.Thread(target=server.serve_forever,daemon=True);thread.start()
url=f'http://127.0.0.1:{server.server_port}'
def post(body):
request=urllib.request.Request(url+'/v1/completions',data=json.dumps(body).encode(),headers={'Content-Type':'application/json'})
return urllib.request.urlopen(request,timeout=5).read()
assert json.loads(urllib.request.urlopen(url+'/health').read())['model_revision']==revision
result=json.loads(post({'prompt':'Data ','max_tokens':16}));assert result['completion_tokens']<=16 and result['model_revision']==revision
stream=post({'prompt':'Data ','max_tokens':16,'stream':True});assert stream.endswith(b'data: [DONE]\n\n')
try:post({'prompt':'Data ','max_tokens':0});raise AssertionError('invalid request accepted')
except urllib.error.HTTPError as e:assert e.code==400
server.shutdown();server.server_close()
(a/'model.json').write_text('{}')
try:load(a);raise AssertionError('tamper accepted')
except ValueError:pass
print('PASS: loss descent, exact resume, artifact integrity, HTTP generation, SSE, invalid input')
kubernetes.yaml
apiVersion: v1
kind: Namespace
metadata:
name: tokenlab-ai
labels:
pod-security.kubernetes.io/enforce: restricted
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: artifacts, namespace: tokenlab-ai}
spec:
accessModes: [ReadWriteOnce]
resources:
requests: {storage: 1Gi}
---
apiVersion: batch/v1
kind: Job
metadata: {name: train-v1, namespace: tokenlab-ai}
spec:
backoffLimit: 1
activeDeadlineSeconds: 600
template:
metadata:
labels: {app: trainer}
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext: {runAsNonRoot: true, runAsUser: 10001, runAsGroup: 10001, fsGroup: 10001, seccompProfile: {type: RuntimeDefault}}
containers:
- name: train
image: tokenlab-ai:local
imagePullPolicy: IfNotPresent
command: [sh, -ec]
args:
- python pipeline.py prepare --source corpus.jsonl --root /work && python pipeline.py train --root /work --resume && python pipeline.py promote --root /work
resources:
requests: {cpu: '1', memory: 256Mi}
limits: {cpu: '2', memory: 512Mi}
securityContext: {allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: {drop: [ALL]}}
volumeMounts:
- {name: artifacts, mountPath: /work}
volumes:
- name: artifacts
persistentVolumeClaim: {claimName: artifacts}
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: model-v1, namespace: tokenlab-ai}
spec:
replicas: 1
strategy: {type: Recreate}
selector:
matchLabels: {app: model-v1}
template:
metadata:
labels: {app: model-v1}
spec:
automountServiceAccountToken: false
securityContext: {runAsNonRoot: true, runAsUser: 10001, runAsGroup: 10001, fsGroup: 10001, seccompProfile: {type: RuntimeDefault}}
containers:
- name: server
image: tokenlab-ai:local
imagePullPolicy: IfNotPresent
resources:
requests: {cpu: '250m', memory: 128Mi}
limits: {cpu: '1', memory: 256Mi}
ports: [{containerPort: 8000}]
startupProbe:
httpGet: {path: /health, port: 8000}
failureThreshold: 30
periodSeconds: 2
readinessProbe:
httpGet: {path: /health, port: 8000}
periodSeconds: 5
livenessProbe:
httpGet: {path: /health, port: 8000}
periodSeconds: 15
securityContext: {allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, capabilities: {drop: [ALL]}}
volumeMounts:
- {name: artifacts, mountPath: /work, readOnly: true}
volumes:
- name: artifacts
persistentVolumeClaim: {claimName: artifacts}
---
apiVersion: v1
kind: Service
metadata: {name: model-api, namespace: tokenlab-ai}
spec:
selector: {app: model-v1}
ports: [{port: 8000, targetPort: 8000}]
type: ClusterIP
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: {name: private-model, namespace: tokenlab-ai}
spec:
podSelector: {}
policyTypes: [Ingress, Egress]
ingress:
- from:
- podSelector:
matchLabels: {role: authorized-client}
ports: [{protocol: TCP, port: 8000}]
egress: []
transformer-job.yaml
# Optional, not executed here. First create namespace/PVC from kubernetes.yaml.
apiVersion: batch/v1
kind: Job
metadata:
name: transformer-train
namespace: tokenlab-ai
spec:
backoffLimit: 1
activeDeadlineSeconds: 7200
template:
metadata:
labels:
app: trainer
spec:
restartPolicy: Never
automountServiceAccountToken: false
securityContext:
runAsNonRoot: true
runAsUser: 10001
runAsGroup: 10001
fsGroup: 10001
seccompProfile:
type: RuntimeDefault
containers:
- name: train
image: tokenlab-transformer:lab
imagePullPolicy: IfNotPresent
command:
- sh
- -c
args:
- python pipeline.py prepare --source corpus.jsonl --root /work && python transformer.py
--root /work --steps 200 --resume && python approve_transformer.py --root
/work --max-nll 4.0
resources:
requests:
cpu: '2'
memory: 2Gi
limits:
cpu: '4'
memory: 4Gi
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop:
- ALL
volumeMounts:
- name: artifacts
mountPath: /work
volumes:
- name: artifacts
persistentVolumeClaim:
claimName: artifacts
gpu-serving.yaml
# Optional unexecuted serving reference. Populate /models/approved first.
# Replace the image tag with a verified digest in your registry before production.
apiVersion: apps/v1
kind: Deployment
metadata: {name: transformer-v1, namespace: tokenlab-ai}
spec:
replicas: 1
strategy: {type: Recreate}
selector:
matchLabels: {app: transformer-v1}
template:
metadata:
labels: {app: transformer-v1}
spec:
automountServiceAccountToken: false
containers:
- name: vllm
image: vllm/vllm-openai:v0.14.0
args: [--model, /models/approved, --served-model-name, tokenlab-transformer, --max-model-len, '128', --dtype, float32, --gpu-memory-utilization, '0.5', --max-num-seqs, '8']
resources:
requests: {cpu: '2', memory: 4Gi}
limits: {cpu: '4', memory: 8Gi, nvidia.com/gpu: '1'}
ports: [{containerPort: 8000}]
startupProbe:
httpGet: {path: /health, port: 8000}
failureThreshold: 180
periodSeconds: 10
readinessProbe:
httpGet: {path: /health, port: 8000}
volumeMounts:
- {name: models, mountPath: /models, readOnly: true}
- {name: shm, mountPath: /dev/shm}
volumes:
- name: models
persistentVolumeClaim: {claimName: artifacts}
- name: shm
emptyDir: {medium: Memory, sizeLimit: 1Gi}
# Integrate into a separately hardened GPU namespace: the public image's UID,
# writable cache locations, device access and driver/CUDA matrix must be checked.
# Do not apply blindly to the restricted CPU namespace above. See README.
---
apiVersion: v1
kind: Service
metadata: {name: transformer-api, namespace: tokenlab-ai}
spec:
selector: {app: transformer-v1}
ports: [{port: 8000, targetPort: 8000}]
Dockerfile
FROM python:3.11.14-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY pipeline.py serve.py corpus.jsonl ./
USER 10001:10001
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1
CMD ["python", "serve.py", "--host", "0.0.0.0"]
Dockerfile.transformer
FROM python:3.11.14-slim
WORKDIR /app
COPY requirements-transformer.txt .
RUN pip install --no-cache-dir -r requirements-transformer.txt
COPY pipeline.py transformer.py approve_transformer.py corpus.jsonl ./
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 TOKENIZERS_PARALLELISM=false
USER 10001:10001
CMD ["python", "transformer.py", "--root", "/work"]