Papers

Research

My research spans transformer memory, representation geometry, and the reliability of systems serving language models.

Two of these papers come straight out of building Metriqual: what fails in a multi-provider gateway, and whether conversation state survives failover. The other two are about how transformers store and move information.

4 arXiv preprints, newest first. Bibliographic details follow the arXiv records. Bold marks my name; author order is as published.

  1. LLM infrastructure reliabilityarXiv:2607.17525 [cs.LG]

    FailureAtlas: A Taxonomy of Failure Modes in Multi-Provider LLM Serving Infrastructure

    Authors: Vishal Pandey, Gopal Singh

    arXiv preprint · Survey paper, 14 pages, 1 figure

    Question
    Gateways that route, load-balance and rate-limit requests across model APIs are now production infrastructure, but their failure modes are scattered across issue trackers and post-mortems. Which failures belong to this layer, and why do some go unnoticed?
    Approach
    A two-axis taxonomy that classifies each failure by origin layer (network and transport, streaming and protocol, state and session, model behaviour, governance and cost) and by detectability (loud or silent). It is populated with five verified entries from public bug reports and first-hand stress testing, each with a mechanistic root-cause analysis. Three entries include standalone reproduction scripts.
    Reported result
    The most operationally severe failures are silent: they return HTTP 200, pass standard health checks and corrupt application state. Two were found first-hand during evaluation: a concurrency race condition that loses conversation history, and a streaming index collision that corrupts tool-call payloads.
    Scope
    A survey with five catalogued entries. Detecting the silent cases requires semantic-level observability, not status codes.
    Read paperPDFCodeWrite-up
  2. Stateful failover and evaluationarXiv:2607.15899 [cs.LG]

    ContinuityBench: A Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing

    Authors: Vishal Pandey, Gopal Singh

    arXiv preprint · 16 pages, 2 figures

    Question
    High API availability does not mean a conversation survives. When a primary provider fails, stateless failover keeps the service up but silently drops the history. How often, and can a proxy prevent it?
    Approach
    Two metrics, Continuity Preservation Rate (CPR) and Continuity Latency Overhead (CLO), and a stateful multi-provider proxy that uses a history-forwarding strategy to rebuild conversational state on a fallback endpoint. The paper also releases continuity-bench, an open harness that injects provider failures under high concurrency.
    Reported result
    Across 750 failover events, the evaluated stateful proxy reached a CPR of 99.20% (95% CI: 98.27% to 99.63%), against near 0% for standard stateless failover. The latency analysis argues for asynchronous exponential backoff with jitter to avoid retry storms against fallback APIs with strict limits.
    Scope
    Results describe the proxy and the failure conditions evaluated in the paper, not a general guarantee for every deployment.
    Read paperPDFCodeWrite-up
  3. (revised )Mechanistic interpretabilityarXiv:2606.09287 [cs.LG]

    Trajectory Geometry of Transformer Representations Across Layers

    Authors: Vishal Pandey, Gopal Singh, Yacine Mahdid

    arXiv preprint · 18 pages, 9 figures

    Question
    Interpretability usually asks what a layer encodes. This paper asks how representations move from one layer to the next, and whether the shape of that path says something about the computation.
    Approach
    The forward pass is treated as a trajectory through representation space, using geometric tools from computational neuroscience. Five metrics are computed directly in the ambient space, with no probes: trajectory length, curvature, a semantic convergence index, layerwise cosine similarity and representational stability. Models: GPT-2, TinyLlama and Qwen2.5, with five controlled prompt families.
    Reported result
    Related prompts converge in middle-to-late layers. Reasoning tasks bend more than lexical variations (0.71 to 0.83 rad against 0.27 to 0.31 rad). Ambiguous tokens split into separate paths. A three-phase structure of encoding, elaboration and output preparation appears in all three architectures. All four effects vanish under shuffled-layer and random-embedding controls.
    Scope
    Three open model families and controlled prompt sets. The pipeline is released as open source and is model-agnostic.
    Read paperPDFCodeWrite-up
  4. Attention architectures and associative memoryarXiv:2605.11196 [cs.LG]

    Variational Linear Attention: Stable Associative Memory for Long-Context Transformers

    Authors: Vishal Pandey, Gopal Singh

    arXiv preprint · 20 pages

    Question
    Linear attention runs in linear time, but its memory state grows without bound and stored associations start to interfere. Can the update itself be made stable?
    Approach
    Variational Linear Attention (VLA) reframes the memory update as an online regularised least-squares problem, with an adaptive penalty matrix maintained through the Sherman-Morrison rank-1 formula. The paper proves that normalising the write direction to unit length gives a recurrence Jacobian with spectral norm exactly 1, and that the state norm is self-limiting under bounded inputs.
    Reported result
    At T = 1,000, VLA reduces the state norm by 109× relative to standard linear attention. It reaches near-perfect exact-match accuracy on multi-query associative recall within the per-head memory regime, holds up better than DeltaNet and standard linear attention as memory load rises, and keeps 62% accuracy at the capacity boundary. A Triton-fused kernel runs 14× faster than sequential Python and drops below softmax attention latency at around 43,000 tokens.
    Scope
    Recall results hold within the effective per-head memory regime. The latency crossover is specific to the fused kernel reported.
    Read paperPDFWrite-up