Turning model-behavior research into a reproducible engineering workbench

Problem beyond prompt examples

Prompt comparisons are useful, but they do not expose where a behavior is represented or whether an intervention is repeatable. The workbench turns steering experiments into explicit model, layer, vector, coefficient, seed, and evaluation records.

My role

I built the repository as an end-to-end research tool: model adapters, activation capture and intervention hooks, API contracts, experiment persistence, UI workflows, tests, and documentation.

Architecture

The backend isolates model loading, activation capture, vector construction, intervention, and generation behind typed API contracts. The UI makes baseline and steered runs comparable without hiding the experiment configuration.

Local execution keeps model weights and prompts under the operator’s control. Saved manifests make a result reproducible across sessions when the same model and environment are available.

What failed or was unreliable

Steering effects vary across models, layers, prompt families, and coefficient ranges. A visually strong single example can be misleading. Memory use and hook cleanup also need deliberate handling in repeated local runs.

Evaluation

The repository treats steering as an experiment, not a guarantee. Comparisons retain baseline output, steered output, configuration, and evaluation notes. Tests cover API and core workflow behavior; they do not establish broad behavioral validity.

Tradeoffs

Local execution improves control and inspectability but increases setup and hardware variability. A UI lowers the barrier to exploration but can make experimental results look more settled than they are, so limitations stay close to the output.

What I would improve next

Next steps include broader model adapters, dataset-level evaluations, uncertainty summaries across prompts, and stronger environment capture for cross-machine reproducibility.