Context-Conditioned Option Selection: A Black-Box Investigation of Jev and the Evolution of a Similarity-Based Decision Model
An empirical investigation into specialized decision model behavior, revealing that observed option choices are governed by context-conditioned similarity scoring and sequential probability allocation rather than intrinsic factual verification.
1. Executive Summary & Research Problem
Specialized decision models are increasingly integrated into mission-critical pipelines—ranging from automated triage and medical record routing to credit decisioning and compliance checking. Unlike general-purpose conversational LLMs, decision models are typically prompted with a bounded context, a structured query, and an explicit set of candidate options, with the expectation that the selected option reflects rigorous logical verification against the provided evidence.
This technical report conducts a black-box empirical evaluation of Jev, a specialized decision model developed by Jevons Technologies. Through controlled experimental test suites spanning medical domain selection, structural context permutations, and invalid choice sets, we explore a foundational question:
"Does the decision model perform independent answer verification, or does it evaluate options through context-conditioned similarity scoring coupled with position-dependent probability allocation?"
2. Key Empirical Findings
A. Contextual Ordering Bias (The Primacy Effect)
The most significant behavioral discovery is a pronounced contextual ordering bias. In test cases where multiple candidate entities within the context possess identical qualifying attributes and criteria satisfaction:
- First-Entity Favoritism: Jev consistently and overwhelmingly selects the first eligible entity mentioned in the context passage.
- Context Inversion Test: When the narrative order of entity presentation in the context is flipped (while keeping all factual predicates, attributes, and options unchanged), Jev's selection inverts in 100% of the symmetrical test cases.
- Option List Invariance: Conversely, permuting the order of candidate choices in the prompt (e.g., swapping Option A and Option B in the candidate list) yields zero change in selection behavior. The bias is strictly anchored to narrative token positions in the context.
B. Invalid Choice Sets and Partial Identity Matches
When presented with invalid choice sets—where none of the provided candidate options logically satisfy the query predicates—the model does not trigger an abstain or "none of the above" behavior. Instead, it systematically falls back to the candidate with the highest lexical or semantic overlap with the prompt, confirming that selection is driven by relative proximity scoring rather than boolean constraint satisfaction.
3. Evolutionary Model Architecture
To understand and mathematically replicate Jev's internal decision mechanics without internal white-box weights, the paper follows an iterative hypothesis testing methodology. We developed a series of proxy algorithms that systematically converge on Jev's observed choice distributions:
Progression of the Algorithmic Models
- Stage 1: Cosine Similarity Baseline: Standard bi-encoder embedding comparison between query-context pairs and candidate options. Proved insufficient to capture subtle syntactic dependencies.
- Stage 2: Candidate-Specific Query Rewriting & Joint Normalization: Reformulating the context-query pair into candidate-specific declarative assertions, followed by joint softmax normalization across the active option set.
- Stage 3: Neutral Context Enrichment: Preprocessing contexts to decouple entity declarations from sentiment or stylistic modifiers, evaluating entity-predicate alignment directly.
- Stage 4: Cross-Encoder Proxy Scoring: Utilizing joint attention cross-encoders to score full sequence interaction between candidate statements and context spans.
- Stage 5: Sequential Probability Allocation: Introducing an exponential decay or primacy weight factor tied to context token offset. This final multi-stage formulation achieved 100% concordance with Jev's empirical selection choices across all test suites.
4. Implications for High-Stakes Enterprise AI
These results carry critical warnings for enterprise architects and ML practitioners deploying proprietary decision models in regulated domains:
- The Illusion of Verification: A model returning high probability for Option A does not imply that Option A has been verified as factually true. It often indicates that Option A was encountered earlier in the narrative stream.
- Vulnerability to Prompt Formatting: Upstream document layout changes, PDF-to-text extraction ordering, or multi-source retrieval merging can inadvertently alter decision outputs without changing underlying business facts.
- Mandatory Architectural Countermeasures:
- Bidirectional context permutation testing in automated evaluation pipelines.
- Synthetic distractor and null-hypothesis calibration tests.
- Explicit decoupling of candidate retrieval from independent logic constraint verifiers.
5. Citation & BibTeX
6. Companion Technical Post & Related Works
Read the detailed companion analysis discussing how these findings impact real-world enterprise architectures: