Context-Conditioned Option Selection: What Black-Box Testing of Jev Teaches Us About AI Decision Systems

Across enterprise software, we are witnessing a quiet shift from general conversational assistants to domain-specialized decision models. These models are deployed inside automated workflows: determining medical clinical trial eligibility, routing insurance claims, approving credit tiers, and enforcing compliance rules.

Unlike an interactive chatbot, a decision model is usually invoked with a tightly bounded context, a specific question, and a discrete set of candidate options. The implicit operational assumption is straightforward: given factual evidence in the context, the model systematically evaluates each candidate option against logical criteria and selects the correct answer.

In our latest research technical report—Context-Conditioned Option Selection: A Black-Box Investigation of Jev and the Evolution of a Similarity-Based Decision Model (Zenodo DOI: 10.5281/zenodo.22864020)—we subjected Jev, a specialized decision model developed by Jevons Technologies, to extensive black-box empirical scrutiny.

The findings challenge the assumption that decision models perform true answer verification. Instead, they demonstrate that observed choices are governed by context-conditioned similarity scoring coupled with position-dependent probability allocation.

---

1. The Core Empirical Discovery: Contextual Ordering Bias

To isolate how Jev evaluates candidates, we constructed controlled test batteries across domain-specific scenarios (including clinical entity selection and operational eligibility). In each case, candidate entities were embedded into a descriptive context alongside clear decision criteria.

The most striking behavioral phenomenon we uncovered was a profound contextual primacy effect:

When two or more entities in the context possess identical qualifying attributes and equal factual eligibility, Jev consistently allocates dominant selection probability to whichever eligible entity was introduced first in the narrative text.

To verify that this was not an artifact of prompt syntax, we ran two diagnostic controls:

• The Context Inversion Test: We kept all factual statements, predicates, candidate options, and query phrasing completely identical, but reversed the order in which the two eligible entities were mentioned in the context passage. In 100% of symmetrical test cases, Jev's selection flipped to match the newly first-introduced entity.
• The Option List Invariance Test: We kept the context passage fixed and permuted the order in which candidate options were presented in the prompt's choice list (e.g., swapping Option A and Option B). This resulted in zero change in selection behavior. Jev was completely insensitive to option list ordering, but acutely sensitive to context narrative position.

---

2. The Fragility of Invalid Choice Sets

What happens when a decision model faces a query where none of the candidate options satisfy the criteria?

In human clinical triage or legal review, the only correct answer is an explicit rejection or an abstention. However, when presented with invalid choice sets, Jev did not trigger an abstention mechanism. Instead, it systematically selected whichever candidate possessed the highest surface-level lexical and semantic overlap with the context text.

This confirms that the model's underlying scoring function is not computing a boolean proof over evidence spans: it is computing a relative compatibility score across the candidate set and normalizing it over a softmax distribution. An invalid option that shares a high density of domain terms with the context will reliably outscore a completely valid but tersely phrased candidate.

---

3. Reconstructing the Engine: An Evolutionary Reverse-Engineering

Because Jev is a proprietary black-box system, we wanted to discover whether an explicit algorithmic pipeline could replicate its exact output choices across all test sets.

Through iterative hypothesis testing, we developed and evaluated five successive algorithmic models:

1. Baseline Cosine Similarity: Measuring bi-encoder embedding similarity between context-query pairs and option strings. While fast, this failed to capture directional relational dependencies.
2. Candidate-Specific Query Rewriting: Generating declarative hypothesis statements for each candidate and computing joint normalized cross-entropy. This improved candidate discrimination but missed position bias.
3. Neutral Context Enrichment: Pre-filtering narrative text to decouple entity declarations from stylistic decorators, isolating core subject-predicate-object claims.
4. Cross-Encoder Proxy Scoring: Passing the full sequence pair (context + rewritten candidate) through deep cross-attention layers, capturing intricate semantic interactions.
5. Sequential Probability Allocation: Factoring an exponential position decay into the scoring function to reflect context token distance.

The final multi-stage architecture—combining neutral context enrichment, cross-encoder proxy scoring, and sequential probability allocation—achieved 100% concordance with Jev's empirical selection decisions across all evaluated test cases.

---

4. Architectural Lessons for Enterprise AI Deployments

These findings have immediate, practical consequences for engineering teams deploying AI in high-stakes enterprise workflows:

A. Beware the "Illusion of Verification"

When an AI model produces a 92% confidence score for Option A, it is easy for product teams to assume that the model has verified Option A against ground truth. In reality, that score often indicates that Option A was encountered earlier in the narrative stream or shares higher lexical affinity with the prompt tokens.

B. Upstream Pipeline Changes Can Alter Downstream Decisions

Consider an enterprise ingestion pipeline that extracts text from PDFs or multi-source databases. If an upstream update changes the extraction order of two paragraphs, the underlying business facts remain identical—yet a context-conditioned decision model may flip its final recommendation simply because Entity B now precedes Entity A in the text stream.

C. Essential Guardrails for Decision Systems

To safely deploy decision models, enterprise architectures should implement three defensive design patterns:

• Bidirectional Permutation Testing: Automatically evaluate contexts under both forward and reversed entity ordering. If the model's decision inverts, the pipeline must flag the decision for human review rather than executing autonomously.
• Null-Hypothesis & Distractor Calibration: Regularly inject unanswerable queries and synthetic distractors into regression test suites to monitor whether the model appropriately abstains.
• Decoupled Verification: Never allow the similarity engine that nominates candidate options to act as the final authority on logical validity. Couple statistical candidate retrieval with deterministic, rule-based verification layers.

---

5. Full Report & Citation

The complete technical report contains the full dataset of test prompts, symmetrical inverted pairs, similarity matrix tables, and mathematical formulations for the sequential probability allocation algorithm.

• Zenodo Record: https://zenodo.org/records/22864020
• DOI: 10.5281/zenodo.22864020
• Direct PDF: Download Report PDF
• Research Deep-Dive Dossier: Read the Interactive Research Dossier