Introduction
As AI applications evolve from conversational interfaces into increasingly autonomous systems, the infrastructure required to operate them is becoming more complex.
Modern agentic applications must do more than generate accurate responses. They must evaluate incoming requests, identify potential safety and privacy concerns, select appropriate downstream capabilities, and coordinate execution across different models and tools. Many applications approach these responsibilities through a combination of static rules and generative language models. While flexible, this approach can introduce unnecessary computational overhead when large models are used to perform narrowly defined classification tasks.
At AISquared, we believe these responsibilities can often be handled more efficiently by specialized models designed for specific purposes.
This philosophy motivated the development of Bolt CLS, a family of lightweight classification models built to support AI safety, sensitive information detection, and intelligent model routing. Rather than generating textual judgments about incoming requests, Bolt CLS models use sequence-classification architectures to produce structured predictions. This allows applications to evaluate requests and make routing decisions without requiring autoregressive generation for each classification operation.
The Bolt CLS family consists of four models:
- Bolt CLS Content: Identifies potentially unsafe or policy-violating content.
- Bolt CLS Jailbreak: Detects attempts to circumvent downstream model safeguards.
- Bolt CLS PII: Identifies requests containing personally identifiable information.
- Bolt CLS Router: Selects appropriate downstream capabilities based on request requirements and dynamically supplied route descriptions.
Although these models address different tasks, they share a common architectural principle: use specialized classification models for decisions that do not require generative inference. In this post, we examine the architecture, training methodologies, and evaluation results behind Bolt CLS, including the tradeoffs involved in safety detection and the design of a flexible, capability-based model router.
Designing Bolt CLS for Specialized Classification
The Bolt CLS family was developed around a straightforward observation: many decisions made within an AI inference pipeline are classification problems rather than generation problems.
Consider an incoming user request. An application may need to determine whether it contains sensitive personal information, whether it represents a jailbreak attempt, or which downstream model is best equipped to handle it. These decisions require interpreting the semantic content of the request, but they do not necessarily require generating natural-language responses.
Traditional generative approaches may prompt a language model to inspect the request and return a textual classification. Bolt CLS instead uses sequence-classification heads that produce scores corresponding directly to the available classification outcomes. This distinction is particularly relevant for applications processing large volumes of requests, where repeatedly invoking generative inference for preliminary screening and routing can introduce unnecessary overhead.
The Bolt CLS Model Family
The four Bolt CLS models address complementary responsibilities within an enterprise inference pipeline.
| Model | Parameters | Primary Function |
|---|---|---|
| Bolt CLS Content | 0.1B | Detect potentially unsafe content |
| Bolt CLS Jailbreak | 0.1B | Detect attempts to circumvent model safeguards |
| Bolt CLS PII | 0.1B | Identify personally identifiable information |
| Bolt CLS Router | 0.4B | Select appropriate downstream models and capabilities |
The detection models are binary classifiers, each focused on identifying a particular condition within an incoming request. The Router solves a different problem: selecting one destination from a dynamically configured collection of candidate capabilities.
This separation is intentional.
A request containing sensitive information is not necessarily unsafe, and an unsafe request is not necessarily a jailbreak attempt. Similarly, determining whether a request requires additional safety evaluation is a different problem from determining which model should execute it. By training separate classifiers for these responsibilities, applications can incorporate specialized predictions into their own execution and policy decisions. In practice, these models can be deployed independently or combined within a larger inference pipeline.
Training the Bolt CLS Detection Models
Bolt CLS Content, Jailbreak, and PII were developed as binary sequence-classification models. Each training example consists of a candidate request and a corresponding label indicating whether the relevant condition is present. Although the models share a common classification architecture, their training datasets differ substantially in size and class distribution.
Training Datasets
We trained the detection models using datasets tailored to their respective objectives.
| Model | Training Samples | Positive Class |
|---|---|---|
| Bolt CLS Content | ~95,000 | 50.2% |
| Bolt CLS Jailbreak | ~15,000 | 14.0% |
| Bolt CLS PII | ~140,000 | 67.0% |
These class distributions introduce different optimization challenges. Content detection has a relatively balanced distribution of positive and negative examples. Jailbreak detection has a considerably smaller positive class, while the PII training dataset contains a majority of positive examples. Class imbalance matters because a classifier can achieve apparently strong overall accuracy while performing poorly on the examples that are most important to detect.
For safety-sensitive applications, false negatives can be particularly consequential. Missing a jailbreak attempt or failing to identify sensitive information may introduce risks that are not captured adequately by overall accuracy. Over-optimizing for detection, however, increases the risk of false positives; a classifier that flags too many legitimate requests may cause unnecessary processing, review, or interruptions to otherwise valid workflows. Taking these constraints into account, we therefore designed the training and evaluation process to account for both detection sensitivity and specificity.
Optimization with Class-Weighted Focal Loss
To address these constraints, we used class-weighted focal loss to address differences in training distributions and emphasize difficult classification examples. Focal loss reduces the relative contribution of examples that a model already classifies confidently, shifting more attention toward examples that remain difficult to distinguish. Class weighting provides an additional mechanism for adjusting the contribution of different classes during optimization. Together, these techniques allow the training objective to account for both class imbalance and classification difficulty.
For the three detection models, we calculated model-specific class weights based on their respective training distributions and applied an additional positive-class multiplier where appropriate. The resulting effective weights were:
| Model | Negative-Class Weight | Positive-Class Weight | Positive-to-Negative Ratio |
|---|---|---|---|
| Bolt CLS Content | 1.00 | 2.00 | 2.00× |
| Bolt CLS Jailbreak | 0.58 | 7.21 | 12.42× |
| Bolt CLS PII | 1.52 | 1.49 | 0.98× |
The Jailbreak model received the strongest relative emphasis on positive examples, reflecting the comparatively low prevalence of jailbreak attempts in its training dataset. The PII model, by contrast, used nearly balanced effective class weights after accounting for the higher prevalence of positive examples. These training decisions were intended to improve the models’ ability to distinguish the classes relevant to each detection objective.
Separating Training from Classification Thresholds
An important design decision was separating the process of learning classification scores from the process of converting those scores into operational decisions. During training, class weighting and focal loss influence how a model learns to distinguish between positive and negative examples. At inference time, a classification threshold determines when the model’s prediction score is interpreted as a positive detection. These are two separate concerns to optimize for.
For each binary classifier, approximately 10% of the supplied training split was reserved for stratified validation, while the existing test split remained separate for final evaluation. After restoring the best training checkpoint, we selected an initial decision threshold using validation predictions. The threshold-selection procedure maximized positive-class F1, breaking ties in favor of recall and then precision. This provided a consistent initial operating point for evaluation, but the threshold that maximizes F1 is not necessarily the most appropriate threshold for every deployment.
A system that screens requests for additional review may tolerate more false positives in exchange for stronger recall. An application that automatically blocks requests may require a substantially different balance. For this reason, Bolt CLS detection models support configurable classification thresholds. This allows applications to adjust detection sensitivity according to their own requirements rather than relying on a single operating point for every use case.
Bolt CLS Router: Capability-Based Model Selection
While the detection models address binary classification tasks, Bolt CLS Router was developed to solve a different problem: choosing an appropriate downstream model or capability for an incoming request.
As organizations deploy multiple language models and specialized agents, routing becomes an increasingly important part of operating AI infrastructure. Different models vary in their reasoning capabilities, supported tasks and modalities, inference costs, and expected response latency. A straightforward summarization request may be handled effectively by a smaller model, while complex analysis may require a more capable destination. Routing every request through the same powerful model can introduce unnecessary computational expense. Conversely, selecting an insufficiently capable destination can compromise the quality of execution.
Traditional approaches often address routing through fixed rules or predefined associations between task categories and particular models. Bolt CLS Router takes a more flexible approach: Instead of learning to associate requests exclusively with fixed model identities, the Router evaluates dynamically supplied descriptions of available capabilities.
Dynamic Candidate Routes
Bolt CLS Router is a multi-class classifier that selects from a configurable set of candidate destinations. The model supports between two and ten candidate routes per request, and each candidate can be represented using structured metadata describing its capabilities and operational characteristics.
Supported information includes:
- Capability description: The types of tasks the destination is designed to handle.
- Cost tier: The relative inference cost associated with the destination.
- Latency tier: The expected response latency.
- Reasoning depth: The complexity of reasoning the destination can support.
- Default status: Whether the destination should receive a slight preference when multiple candidates are similarly appropriate.
The Router evaluates the incoming request against these descriptions and selects a destination according to the configured routing objective. This approach separates routing decisions from the specific identities of the underlying models. For example, an application might define a destination for general conversational tasks and another for complex analytical reasoning. The model associated with either destination could change over time without requiring the routing interface to depend directly on a particular provider or model name. This flexibility is especially useful in environments where model availability, costs, and capabilities evolve frequently.
Router Training
The Router was trained using approximately 98,000 examples representing different requests and candidate configurations.
| Training Characteristic | Value |
|---|---|
| Training samples | ~98,000 |
| Candidate routes per request | 2–10 |
| Mean candidate routes per request | 5.98 |
| Distinct candidate route definitions | 76 |
One of the primary training objectives was encouraging the Router to evaluate candidates based on their capabilities rather than superficial characteristics such as their position within the input. Because candidate sets can vary in both size and ordering, a router that consistently favors particular candidate positions would be poorly suited to dynamic routing environments. To address this, we systematically shuffled candidate orderings, input configurations, and contextual descriptions during training, thus encouraging the model to focus on the semantic relationship between task requirements and candidate capabilities rather than learning fixed positional preferences. For optimization, we used inverse-frequency-weighted focal loss across candidate-position classes, without the additional positive-class multiplier used for binary detection.
Configurable Routing Objectives
Choosing the best destination is not always a question of selecting the most capable model. Instead, organizations may prioritize operational requirements depending on the application. For example, a high-volume customer support workflow may favor cost efficiency, while an interactive application may prioritize low latency. Complex analytical tasks may warrant selecting the strongest available reasoning capability regardless of operating cost.
Bolt CLS Router supports five configurable routing modes:
| Routing Mode | Selection Objective |
|---|---|
| balanced | Balance capability, cost, and latency |
| cost_optimized | Prioritize cost efficiency while maintaining sufficient capability |
| low_latency | Prioritize responsiveness while maintaining sufficient capability |
| quality | Favor the strongest available capabilities |
| description_based | Select based on semantic alignment with candidate descriptions |
The selected routing mode is incorporated directly into the Router’s input, and allows the model to evaluate identical requests and candidate sets under different optimization objectives.
For example, consider a request requiring complex legal and governance analysis.
Two downstream destinations may be capable of handling the task:
- A general-purpose reasoning model with strong analytical capabilities and a relatively high inference cost.
- A more advanced reasoning model with greater reasoning depth but substantially higher operating costs.
Under cost_optimized, the Router should favor the less expensive destination if it is sufficiently capable of completing the task. Under quality, the stronger reasoning destination may be preferred.
This example illustrates the intended behavior of the routing modes rather than a measured model prediction.
The broader advantage is that applications can change routing priorities without maintaining separate routing implementations or retraining a router for every optimization objective.
Evaluating Bolt CLS
We evaluated the Bolt CLS family using held-out datasets designed to assess the models’ classification and routing capabilities. For the Content, Jailbreak, and PII models, the evaluation datasets combined internal data with publicly available detection datasets. We measured several complementary metrics, including accuracy, precision, recall, F1, ROC AUC, and false-negative rates.
For the Router, we evaluated exact agreement with designated target routes and a more permissive measure that recognizes alternative acceptable destinations. These distinctions are important because classification and routing performance cannot be adequately described by accuracy alone.
Safety Detection Evaluation
Evaluation Datasets
The three detection models were evaluated using datasets with substantially different positive-class distributions.
| Model | Test Samples | Positive Class |
|---|---|---|
| Bolt CLS Content | 40,416 | 35.6% |
| Bolt CLS Jailbreak | 21,491 | 8.6% |
| Bolt CLS PII | 36,092 | 84.3% |
The differing distributions are important when interpreting evaluation results. For example, positive examples represent only 8.6% of the Jailbreak evaluation dataset. A classifier that frequently predicts the negative class could achieve relatively high overall accuracy while missing a meaningful proportion of actual jailbreak attempts. Similarly, the high prevalence of positive examples in the PII evaluation dataset affects how metrics such as precision and overall accuracy should be interpreted. For this reason, we evaluated the models using multiple complementary metrics rather than relying on overall accuracy alone.
Performance at Validation-Selected Thresholds
Our initial evaluations used thresholds selected during the training-validation process. These thresholds were chosen to maximize positive-class F1 on validation predictions.
| Model | Accuracy | F1 | ROC AUC | False-Negative Rate | Threshold |
|---|---|---|---|---|---|
| Bolt CLS Content | 76.47% | 0.7386 | 0.8942 | 6.75% | 0.5271 |
| Bolt CLS Jailbreak | 95.31% | 0.7068 | 0.9756 | 34.18% | 0.6343 |
| Bolt CLS PII | 93.77% | 0.9617 | 0.9995 | 7.34% | 0.5238 |
Based on these evaluations, Bolt CLS PII demonstrated particularly strong discrimination between positive and negative examples, achieving a ROC AUC of 0.9995. Bolt CLS Content achieved a ROC AUC of 0.8942, while Bolt CLS Jailbreak achieved 0.9756. However, the Jailbreak results illustrate why ROC AUC and overall accuracy should not be interpreted as sufficient evidence of operational detection performance. Despite achieving 95.31% accuracy and a ROC AUC of 0.9756, the model missed 34.18% of positive examples at its validation-selected threshold. For applications where missed jailbreak attempts are particularly consequential, this operating point may not provide sufficient detection sensitivity. This motivated additional evaluation of the models at lower classification thresholds.
High-Recall Detection Performance
We examined alternative thresholds designed to prioritize recall. At the selected operating points, all three models achieved recall exceeding 99% on their respective evaluation datasets.
| Model | Threshold | Accuracy | Precision | Recall | F1 | False-Positive Rate |
|---|---|---|---|---|---|---|
| Bolt CLS Content | 0.30 | 61.41% | 48.00% | 99.12% | 0.6467 | 59.47% |
| Bolt CLS Jailbreak | 0.20 | 78.51% | 28.47% | 99.30% | 0.4425 | 23.45% |
| Bolt CLS PII | 0.25 | 99.11% | 99.89% | 99.04% | 0.9947 | 0.56% |
These results demonstrate that detection thresholds can substantially change model behavior. For Content detection, lowering the threshold increased recall to 99.12%, but also produced a false-positive rate of 59.47%. For Jailbreak detection, recall increased to 99.30%, with a false-positive rate of 23.45%.
These tradeoffs suggest that high-recall configurations may be better suited to initial screening stages, where flagged requests undergo additional evaluation rather than being rejected automatically. Interestingly, Bolt CLS PII exhibited a different performance profile. At a threshold of 0.25, it achieved 99.04% recall while maintaining 99.89% precision and a false-positive rate of just 0.56%. This suggests that the PII classifier can achieve strong detection sensitivity while preserving high precision on the evaluated dataset.
Bolt CLS PII: Examining the Precision-Recall Tradeoff
Given the importance of identifying sensitive personal information, we conducted a more detailed analysis of Bolt CLS PII across multiple operating thresholds.
The model achieved a ROC AUC of 0.9995 and an average precision of 0.9999 on the evaluation dataset, indicating strong separation between positive and negative examples. We then examined how changes to the classification threshold affected detection sensitivity and precision.
| Threshold | Accuracy | Precision | Recall | F1 | False-Positive Rate |
|---|---|---|---|---|---|
| 0.5238 | 93.77% | 99.95% | 92.66% | 0.9617 | 0.25% |
| 0.25 | 99.11% | 99.89% | 99.04% | 0.9947 | 0.56% |
| 0.10 | 99.69% | 99.73% | 99.90% | 0.9982 | 1.44% |
| 0.05 | 99.58% | 99.53% | 99.97% | 0.9975 | 2.53% |
At a threshold of 0.10, Bolt CLS PII correctly identified 30,381 of the 30,410 positive examples in the evaluation dataset. This corresponds to 99.90% recall and 99.73% precision, with just 29 positive examples missed. Reducing the threshold further to 0.05 increased recall to approximately 99.97%, while precision remained at 99.53%. These findings demonstrate that the model maintained strong precision even at thresholds designed to minimize missed detections.
However, the analysis also highlights an important methodological consideration. The lower thresholds were examined using the evaluation dataset rather than selected independently before testing. As a result, these findings should be understood as retrospective threshold analysis. Further calibration using representative deployment data would be necessary before selecting a production operating point. The appropriate threshold will also depend on the prevalence of PII-containing requests and the consequences of false positives and false negatives within the application.
From Classification Scores to Operational Decisions
The safety detection results illustrate a broader architectural principle: Classification performance and operational policy are related, but they are not the same thing.
For example, identifying potentially unsafe content may trigger additional evaluation, require human review, or cause a request to be rejected, depending on organizational policies. Similarly, detecting personal information may trigger additional safeguards, data-handling procedures, or routing restrictions. A high-recall classifier with a substantial false-positive rate may be useful as an initial screening mechanism but inappropriate as the sole basis for automatically blocking requests.
Conversely, a classifier operating at a high-precision threshold may reduce false alarms while allowing more problematic inputs to pass undetected. Bolt CLS supports configurable decision thresholds so applications can determine how classification scores should influence downstream behavior. This separation between detection and enforcement is particularly important in enterprise environments, where operational requirements and risk tolerances vary substantially between applications.
Evaluating Bolt CLS Router
In addition to the three detection models, we evaluated Bolt CLS Router across 10,880 held-out routing tasks. The evaluation measured two complementary outcomes:
Top-1 routing accuracy measures whether the Router selected the designated target destination.
Effective routing accuracy measures whether the selected destination fell within a programmatically defined set of acceptable alternatives.
| Metric | Result |
|---|---|
| Top-1 routing accuracy | 89.81% |
| Effective routing accuracy | 94.19% |
Bolt CLS Router selected the designated target destination in approximately nine out of ten evaluated tasks. When accounting for acceptable alternative destinations, effective routing accuracy increased to 94.19%. This distinction is important because model routing does not always have a single correct answer.
Consider a straightforward conversational request where an application may have both a smaller conversational model and a larger general-purpose model available for routing. Although the smaller model might be preferred for cost efficiency, either destination may be capable of completing the task. If the router selects the larger model when the smaller model is designated as the target, exact top-1 accuracy counts this selection as incorrect. Effective routing accuracy instead counts that selection as acceptable. As a result, this metric provides a broader view of whether the Router selects a destination considered capable of handling the task.
Integrating Bolt CLS into Agentic AI Systems
Bolt CLS models are designed to operate as specialized components within larger AI applications. Although each model can be deployed independently, their complementary capabilities allow applications to separate request evaluation, policy enforcement, and model selection from downstream generative execution.
Consider an enterprise AI assistant that provides employees access to multiple models, tools, and internal data sources. An illustrative request-processing workflow might involve five stages.
- Request intake
- A user submits a request to the application.
- Safety and privacy classification
- Bolt CLS Content, Jailbreak, and PII evaluate the request and produce classification scores corresponding to their detection objectives.
- Policy evaluation
- The application interprets those scores according to organizational policies.
- Depending on the results, it may apply additional safeguards, request further evaluation, or allow the request to proceed.
- Capability-based routing
- For eligible requests, Bolt CLS Router evaluates the available destinations and selects an appropriate capability based on the request and configured routing objective.
- Downstream execution
- The selected model or agent processes the request according to the application’s execution policies and available permissions.
Applications may deploy only a subset of the Bolt CLS models, execute independent classification tasks in parallel, or incorporate additional evaluation and authorization stages. Rather than requiring a single large model to handle safety classification, privacy detection, routing, and execution, applications can distribute these tasks across specialized components. This creates opportunities to reduce unnecessary generative inference while allowing organizations to define how different classification signals influence application behavior. It also supports more flexible multi-model environments, where routing configurations can evolve as available capabilities change.
Key Considerations
Although the Bolt CLS evaluations demonstrate promising results, several considerations remain important for production deployment.
- Threshold calibration. The high-recall operating points examined in our evaluations were selected retrospectively. Production thresholds should be calibrated using representative data and validated independently.
- False-positive tradeoffs. The Content and Jailbreak classifiers achieved high recall at the cost of substantial false-positive rates. These operating points may be better suited to additional screening than automatic enforcement.
- Evaluation data distributions. The prevalence of positive examples differed considerably across the detection datasets and may not reflect production traffic. Metrics such as precision can change when the underlying class distribution changes.
- Routing objective compliance. Effective routing accuracy measures whether a selected destination is considered capable of completing a task. It does not fully evaluate whether the selection optimizes cost, latency, or quality under the configured routing mode.
- End-to-end performance. The reported evaluations measure individual classification and routing capabilities. Additional system-level evaluation is necessary to quantify the effects of integrating these models into complete inference pipelines.
These considerations provide useful directions for consideration within organizations considering these models for use within their environments.
Conclusion
As enterprise AI systems become more autonomous, the infrastructure surrounding generative models becomes increasingly important. Applications must determine which requests require additional safeguards, how sensitive information should be handled, and which downstream capabilities are appropriate for different tasks. Bolt CLS demonstrates how specialized sequence-classification models can address distinct responsibilities within an agentic AI system.
The Content, Jailbreak, and PII classifiers provide targeted detection signals that applications can incorporate into their safety and privacy workflows. Bolt CLS Router introduces a capability-based approach to model selection, allowing applications to evaluate dynamically configured destinations and adjust routing behavior according to operational priorities.
Our evaluations highlight both the strengths and practical considerations of this approach. Bolt CLS PII demonstrated strong precision-recall characteristics, achieving 99.90% recall and 99.73% precision at a retrospectively examined threshold. Bolt CLS Router achieved 89.81% top-1 accuracy and 94.19% effective routing accuracy across 10,880 held-out tasks. The Content and Jailbreak results further illustrate why classification thresholds and downstream policies must be considered alongside benchmark metrics. Together, these findings support our broader approach to building modular AI infrastructure, where specialized models contribute focused capabilities rather than requiring a single generative model to perform every responsibility.
As AI systems grow more capable, we believe the next step is making the infrastructure around them more intelligent, efficient, and adaptable.
Bolt CLS is an important part of that effort.
