How to Assess AI Models for Finding Security Vulnerabilities: A Step-by-Step Guide

Introduction

Security vulnerability detection is a critical task in software development and cybersecurity. Recent evaluations by the UK's AI Security Institute have shown that advanced language models like OpenAI's GPT-5.5 can match the performance of specialized models like Claude Mythos in identifying vulnerabilities. This guide walks you through the process of evaluating AI models for this purpose, using the institute's methodology as a blueprint. Whether you're a security researcher, developer, or AI enthusiast, you'll learn how to set up tests, compare models, and interpret results effectively.

How to Assess AI Models for Finding Security Vulnerabilities: A Step-by-Step Guide — Source: www.schneier.com

What You Need

Access to AI models: At least two models to compare (e.g., GPT-5.5, Claude Mythos, or smaller/open-source models). Ensure you have API keys or local deployments.
Evaluation dataset: A curated set of code snippets or software components with known security vulnerabilities (e.g., from CVE databases or synthetic benchmarks).
Ground truth labels: Verified list of vulnerabilities in the dataset to compare against model outputs.
Scaffolding tools: For cheaper/smaller models, you may need additional prompting frameworks or tool integration to enhance output quality.
Computing resources: Sufficient hardware or cloud credits for running model queries (especially for large models).
Metric definitions: Clear criteria for success: detection rate, false positive rate, and time per query.

Step-by-Step Evaluation Process

Step 1: Define Evaluation Objectives

Before you begin, clarify what you want to measure. In the UK's AI Security Institute study, the goal was to compare GPT-5.5's vulnerability detection ability against Claude Mythos. Decide whether you're interested in raw detection power, cost efficiency, or required scaffolding. Write down your key questions: "Which model finds more true vulnerabilities?" "How much manual guidance does each model need?"

Step 2: Select Models and Acquire Access

Choose at least one baseline model (like Mythos) and one candidate (like GPT-5.5). Ensure both are generally available—GPT-5.5 is widely accessible, while Mythos may require specific subscriptions. For comparison, also consider a smaller, cheaper model (e.g., GPT-3.5 or a fine-tuned BERT). Note: the UK institute found that cheaper models can be equally effective if given proper scaffolding—extra prompts or tool integration. Obtain API credentials or run local inference.

Step 3: Prepare the Dataset

Curate a test set of codebases with known vulnerabilities. Use public repositories like the CVE database or synthetic benchmarks such as OWASP Benchmark. For each sample, record:

The source code or binary
The exact vulnerability type (e.g., SQL injection, buffer overflow)
Location in code (file and line number)
Severity level

This ground truth will be used to score model outputs. Ensure the dataset is diverse to avoid overfitting.

Step 4: Design the Prompting Strategy

Create a consistent prompt for each model to ensure fair comparison. For example: "Analyze this code and list any security vulnerabilities you find. For each, provide the vulnerable line, type, and a recommended fix." For small models, you may need to add scaffolding (see Tips). The UK institute noted that cheaper models required more elaborate prompts—like including example vulnerabilities or step-by-step reasoning instructions. Document your exact prompt for reproducibility.

Step 5: Run the Evaluation

Submit each dataset sample to each model. Record:

Raw output: the list of vulnerabilities flagged
Response time per query
Any additional data like confidence scores or reasoning chain

Run multiple trials if possible to account for model randomness (temperature settings). Use a script to automate API calls and save outputs to a structured file (JSON or CSV).

Step 6: Score the Results

Compare model outputs against the ground truth. Calculate:

True Positives (TP): Vulnerabilities correctly identified
False Positives (FP): Non-existent vulnerabilities flagged
False Negatives (FN): Missed real vulnerabilities
True Negatives (TN): Correctly not flagged (if dataset includes clean code)

Derive metrics: Recall = TP/(TP+FN), Precision = TP/(TP+FP), F1 Score. The UK study found that GPT-5.5 and Mythos had comparable recall and precision, while the cheaper model matched them after proper scaffolding.

Step 7: Analyze Cost and Scaffolding Requirements

Evaluate the trade-offs. For each model, calculate total cost (API fees × number of queries). Note that GPT-5.5 might be more expensive per query than a smaller model. However, the smaller model may require manual prompt engineering or tool integration—this scaffolding effort adds time and expertise. Compare the total cost of ownership: pay-per-query for big models vs. fixed labor for small models. The UK institute highlighted that scaffolding could make smaller models equally effective, but at a development cost.

Step 8: Draw Conclusions and Document

Based on your data, decide which model best fits your use case. For example:

If you need out-of-the-box accuracy with minimal setup, GPT-5.5 or Mythos are suitable.
If you have budget constraints and skilled prompt engineers, a smaller model with scaffolding can be equivalent.

Document your methodology, prompt templates, and results in a report. This transparency allows others to replicate your evaluation and validates your findings.

Tips for Success

Always use a diverse dataset: Include real-world vulnerabilities from different languages (C++, Python, Java) to avoid model bias.
Optimize scaffolding iteratively: For small models, test multiple prompt structures—include chain-of-thought, code line numbers, or multi-turn dialogues. The UK institute's success with cheaper models relied on careful scaffolding.
Control for randomness: Set temperature=0 for deterministic outputs, or run 5-10 trials and average metrics.
Consider ethical implications: Vulnerability detection tools should not be used to find exploits without permission. Always test on your own or authorized systems.
Update your evaluation regularly: As models improve, repeat the evaluation to stay current. The landscape changes quickly—what's true today may not hold next month.
Share your findings: Contribute to community benchmarks to help others decide which model to use for security tasks.

By following these steps, you can rigorously assess any AI model's ability to find security vulnerabilities—just as the UK's AI Security Institute did with GPT-5.5 and Mythos. The key is a balanced approach: measure performance, cost, and human effort to make an informed choice.

Tags: