CWE-1357

What is CWE-1357 (Use of a System Element with Insecure Security Configuration)?

In an AI/ML context, this weakness refers to integrating base foundation models, neural weights checkpoints, training dataset configurations, or ML pipeline elements without enforcing secure defaults, signature checks, or dependency lockdowns.

Vulnerability Insights

AI systems rely heavily on pre-trained components sourced from third-party hubs. If a foundation model or model configuration file (e.g. tokenizer configuration, chat template) is loaded without validating its integrity or source, an attacker who has poisoned the upstream repository can compromise the downstream AI system. This can lead to weights-level model backdooring, private data exfiltration through crafted tokenizer sequences, or prompt-execution escapes.

Impact on Systems

• Model Integrity Compromise: Attackers insert hidden backdoors that activate specific malicious responses under target trigger conditions.
• Information Leakage: Insecure model configurations or templates can be manipulated to leak sensitive training data or system prompt structures.
• Supply Chain Contamination: Poisoned dependencies propagate down the application chain, compromising all dependent services.

Real-World Attack Scenario

An attacker publishes a fine-tuned replica of a popular open-source model on a public model hub. This model contains modified weight tensors designed to trigger high-probability output bias when a specific cryptographic key is present in the prompt. Because the application imports the model directly by name without verifying a secure cryptographic hash (SHA-256) or checking the author signature, it deploys the backdoored model, allowing the attacker to bypass authentication checks during prompt processing.

Code Examples

Vulnerable Implementation

from transformers import AutoModelForCausalLM, AutoTokenizer

# VULNERABLE: Loading a model directly from a public hub without verifying hash, configuration integrity, or switching off remote code
def load_llm_model(model_name):
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
    model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True)
    return tokenizer, model

Secure Alternative

import hashlib
from transformers import AutoModelForCausalLM, AutoTokenizer

# SECURE: Enforcing strict hash validation, disabling remote code execution, and using pinned model revision digests
EXPECTED_SHA256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"

def load_llm_model_secure(model_dir, revision_id):
    # Verify local weights directory structure or use pinned repository commit hash (revision_id)
    tokenizer = AutoTokenizer.from_pretrained(
        model_dir, 
        revision=revision_id,
        trust_remote_code=False  # Disable arbitrary execution of model config scripts
    )
    model = AutoModelForCausalLM.from_pretrained(
        model_dir, 
        revision=revision_id,
        trust_remote_code=False
    )
    return tokenizer, model

Remediation

Always lock down AI dependencies using explicit commit hashes or local cryptographic checksums (e.g., AIBOM logs). Disable dynamic remote code execution (trust_remote_code=False) when loading public weights. Implement continuous model monitoring and behavior auditing to detect weight anomalies post-deployment.