LLM10: Model Theft

Model Extraction Attacks

In a model extraction attack, an adversary makes thousands of queries to a model API and uses the input-output pairs to train a clone model. Research has shown that with sufficient queries, attackers can create a distilled version of a proprietary model that reproduces 80-95% of its behavior. The cost of extraction is a fraction of the cost of training, making this a significant IP theft vector.

Insecure Model Storage

Model weights are often stored insecurely: public S3 buckets, unencrypted model registries, or embedded in container images pushed to public registries. A single misconfigured IAM policy can expose months of training work and millions of dollars in compute investment. Model weights should be treated with the same security as source code — or more, given the investment they represent.

Value of Stolen Models

A stolen model is valuable in multiple ways: (1) Direct commercial use — deploying the stolen model as a competing service. (2) Fine-tuning — using the stolen model as a base for specialized downstream tasks. (3) Vulnerability research — studying the model offline to find prompt injection techniques, biases, and safety bypasses that can be exploited against the original service.

⚔️ Attack Examples & Code Patterns

# ❌ VULNERABLE — model weights baked into Docker image
FROM python:3.11
COPY ./model_weights/ /app/models/    # 15GB proprietary weights
COPY ./app/ /app/
CMD ["python", "serve.py"]
# Anyone who pulls this image gets the model weights

# ✅ SAFE — download at runtime with authentication
FROM python:3.11
COPY ./app/ /app/
# Model weights fetched securely at startup
ENV MODEL_REGISTRY="s3://private-models/production/"
CMD ["python", "serve.py"]
# serve.py downloads model with IAM role, not baked in

# Model extraction attack — querying API to collect training data
import openai

# Generate diverse input-output pairs
pairs = []
for category in CATEGORIES:
    for template in TEMPLATES:
        prompt = template.format(category=category)
        response = target_api.complete(prompt)
        pairs.append({"input": prompt, "output": response})

# Use collected pairs to fine-tune a clone model
# clone_model = train(base_model, pairs)  # 80%+ accuracy clone

# ✅ DETECTION: Monitor for unusual API patterns
# - Same API key making >10K requests/day
# - Systematic prompts across categories
# - Low user engagement (no follow-ups, just queries)

🔍 Detection Checklist

🛡️ Mitigation Strategy

Implement rate limiting and query monitoring on model APIs. Use watermarking techniques in model outputs. Encrypt model weights at rest and in transit. Apply access controls to model artifact storage. Monitor for unusual API query patterns indicative of extraction attempts.