The GPU cluster is warm, the endpoint is live, and the team is calling it a ship. But the model artifact was pulled from a public registry without a hash check. The training corpus was never audited for poisoned samples. Every tool the agent can call is default-open. These are not edge cases — they are how most first deployments go, and they are why secure ML model deployment best practices have become a first-class engineering discipline rather than a compliance afterthought.
This post covers the four operational layers where production ML deployments typically fail on security: the supply chain before inference, the input/output controls at the endpoint, the privilege model governing what the system can do, and the monitoring that tells you when something has already gone wrong.
The Threat Surface Differs From a Web App
The OWASP Top 10 for LLM Applications (2025) maps ten risk classes for deployed LLM applications. At least four — supply chain vulnerabilities (LLM03), data and model poisoning (LLM04), system prompt leakage (LLM07), and excessive agency (LLM06) — are not primarily runtime issues. They are baked into the model or the deployment configuration before the first real request arrives.
The asymmetry is worth understanding: a backdoored model weight (LLM03) behaves identically to a clean model on standard benchmark inputs. The malicious behavior activates only on a specific trigger pattern the attacker controls. Standard evaluation pipelines will not catch this, because the trigger is by design out-of-distribution relative to any test set. This is qualitatively different from a SQL injection vulnerability, which at least shows up under a fuzzer pointed at the endpoint.
Supply Chain: Verify the Artifact Before It Lands
Before a model weight file reaches an inference server, three things should be true: you know where it came from, you can verify it has not been tampered with, and you have an inventory of what Python packages built and serve it. The attack paths this closes off — serialization flaws, namespace hijacking and hub poisoning — are catalogued in ML model supply chain attacks.
Hash verification. Every .safetensors or .pt file should have a SHA-256 digest recorded at training time and checked at deploy time. Note where that guarantee actually comes from on Hugging Face Hub: large weight files are stored and addressed by their LFS SHA-256 (Xet-backed repositories reconstruct a file from content-addressed chunks keyed on the same hash), and huggingface_hub resolves a download against a repository revision. A model card carries no integrity metadata, so nothing in the README is checked. The practical control is to pin revision= to a full commit SHA rather than a branch name, and to compare the downloaded file against a digest your own build recorded.
Model registry with lineage. MLflow’s model registry ties a registered model version to the run that produced it, including training data paths, git commit SHA, and evaluation metrics. This is the minimum audit trail a production deployment should carry, and it is also the promotion gate that keeps an unverified artifact from reaching an endpoint at all. Hardening that registry itself — default-deny access, isolated artifact storage and digest checks on promotion — is the subject of MLflow model registry security.
import mlflow
import mlflow.pytorch
with mlflow.start_run() as run:
mlflow.set_tags({
"training_dataset_sha256": dataset_hash,
"git_commit": git_rev,
"base_model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
})
mlflow.pytorch.log_model(
model,
artifact_path="model",
registered_model_name="inference-prod",
)SBOM for dependencies. The Python packages used at inference time — transformers, vllm, torch — are part of the supply chain. Generate a CycloneDX SBOM at build time with cyclonedx-py environment -o sbom.json and scan it against OSV or a commercial SCA tool before baking it into the container image. Sonatype’s analysis of OWASP LLM03 documents how poisoned open-source inference dependencies are a documented, not theoretical, attack vector.
Input and Output Controls at Inference
Oz and Keskin’s IntechOpen chapter on operationalizing the NIST AI RMF for LLMs proposes a layered filtering approach: rule-based heuristics first, then an auxiliary classifier for patterns the rules miss. Both layers run before the input reaches the main model. That layering is the authors’ architecture, not a NIST requirement.
A practical implementation in a FastAPI wrapper around a vLLM or Ray Serve backend:
from fastapi import FastAPI
from pydantic import BaseModel, field_validator
import re
app = FastAPI()
INJECTION_PATTERNS = [
re.compile(r"ignore (all |previous |above )?instructions", re.IGNORECASE),
re.compile(r"system prompt", re.IGNORECASE),
re.compile(r"<\|.*?\|>"), # control token formats
]
class InferenceRequest(BaseModel):
prompt: str
max_tokens: int = 512
@field_validator("prompt")
@classmethod
def block_injection_patterns(cls, v: str) -> str:
for pattern in INJECTION_PATTERNS:
if pattern.search(v):
raise ValueError("Request blocked by input policy")
if len(v) > 8192:
raise ValueError("Prompt exceeds max allowed length")
return v
class InferenceResponse(BaseModel):
text: str
finish_reason: str
@app.post("/generate", response_model=InferenceResponse)
async def generate(req: InferenceRequest) -> InferenceResponse:
# Route to backend only after validation passes
...On the output side, OWASP LLM05 (Improper Output Handling) covers the case where model output is passed to a downstream system — a shell, a database query builder, a code executor — without sanitization. If your pipeline does this, the output needs the same validation treatment as any external input. Enforce a schema (Pydantic, JSON Schema) before the output reaches any downstream consumer; never pass raw model text directly to a subprocess or SQL builder.
Least-Privilege Deployment
OWASP LLM06 (Excessive Agency) is the most operationally neglected risk in agentic deployments. An agent granted write access to a production database “because it might need it” has an unlimited blast radius if its behavior is ever compromised. The same principle that limits a web server’s filesystem permissions applies here: scope tool access to exactly what the task requires.
Concretely:
- RAG retrievers should have read-only access to the vector store, with no path to write or delete.
- Agents calling external APIs should do so through a proxy that enforces rate limits, allowlists endpoints, and logs every call with a correlation ID.
- The inference endpoint’s network egress should be restricted. A model serving API does not need outbound internet access to Hugging Face Hub in production.
How much of this you can enforce depends on the serving layer. Managed endpoints expose network and IAM controls but hide the runtime; self-hosted runtimes expose everything and make it your job. That trade-off is compared directly in model serving compared across SageMaker, Vertex AI and Databricks, and the governance evidence a regulated deployment has to emit on top is covered in best MLOps platform for regulated industries. The platform’s own isolation model matters as much as the endpoint config: a bare tracking server and a full Kubernetes stack sit at opposite ends, compared in MLflow vs Kubeflow security. On the managed clouds, the IAM privilege-escalation and network-isolation differences are reviewed in SageMaker vs Vertex AI security.
Monitoring for Adversarial Signals
Standard ML monitoring tracks data drift (PSI, KS test on input distributions) and label drift. Secure deployment monitoring adds a third dimension: adversarial success rate — what fraction of requests trigger policy blocks, and is that rate spiking?
# prometheus scrape config
scrape_configs:
- job_name: inference_security
static_configs:
- targets: ["inference-svc:8080"]
metrics_path: /metricsRegister these metrics in your application using prometheus_client:
from prometheus_client import Counter, Histogram
REQUESTS = Counter(
"inference_requests_total",
"Total inference requests",
["status"], # allowed | blocked | error
)
VIOLATIONS = Counter(
"inference_policy_violations_total",
"Input policy violations",
["rule"], # injection | length | schema
)
INPUT_LENGTH = Histogram(
"inference_input_length_bytes",
"Input prompt size distribution",
buckets=[256, 512, 1024, 2048, 4096, 8192],
)A spike in inference_policy_violations_total without a corresponding spike in inference_requests_total signals targeted probing, not organic traffic growth. A p99 latency spike that tracks with a specific input_length_bytes bucket can indicate token-stuffing attempts designed to exhaust the KV cache — OWASP LLM10 (Unbounded Consumption).
Set alert thresholds on both and wire them to the same on-call rotation as infrastructure alerts. The NIST AI RMF is voluntary guidance and mandates nothing, but its Manage function points at documented response procedures for identified AI risks, which for a deployed LLM means playbooks for prompt injection and knowledge-base poisoning. Those playbooks should exist before the first production request, not after the first incident.
Canary Before Full Rollout
Shadow deployment — routing 5-10% of live traffic to the new model, comparing outputs without serving them to users — is the safest rollout pattern. It surfaces behavioral regressions under real traffic distributions before any user is affected. Ray Serve and BentoML both support traffic-split policies natively. Keep the prior model version registered in MLflow and actively serving until the canary has accumulated at least 24 hours of traffic across all time-of-day segments. The rollback path should be a one-command operation before the canary is ever enabled.
Related across the network
- Hugging Face Security Incidents: Malicious Models and Token Theft — ai-alert.org
- Monitoring LLM Outputs in Production: Anomalies and Drift — aidefense.dev
- Best AI Monitoring Tools 2026: LLM Observability Compared — aiincidents.org
- How AI Model Evaluation Metrics Work: A Practitioner’s Guide — aisecbench.com
- MLOps observability: drift detection, degradation alerting and Evidently/WhyLabs integration — sentryml.com
