Wire your eval framework.One snippet at a time.
Three copy-paste quickstarts for the eval frameworks your team is most likely already running — pytest-style, OpenAI Evals, and Braintrust. Every snippet reads your pl_… API key from the same env-var stack Plumbline AI hands out at registration, then posts a signed record against /api/eval-records.
Install + auth
Set the env vars, then call the SDK.
Every quickstart below reads the same five variables. Mint a pl_… key for a fresh model on the registration page — the plaintext is shown once and stored only as SHA-256. For an unauthenticated reader, placeholders below show the wire shape.
Set these in your shell, your CI secrets, or your framework's credentials store before invoking any of the quickstarts.
POLSIA_API_KEY—pl_…bearer token from registration.POLSIA_API_BASE— public ledger origin (matches the SDK'sAPI_ORIGIN).POLSIA_MODEL_ID— public id the ledger hashes into every record.POLSIA_MODEL_VERSION— semver the SDK attaches to each captured run.POLSIA_RUN_ID— optional run identifier; the Node SDK mints one if omitted.
First call via the Node SDK
POLSIA_API_KEY=pl_<polsia-api-key> \ POLSIA_MODEL_ID=<polsia-model-id> \ POLSIA_MODEL_VERSION=<polsia-model-version> \ POLSIA_API_BASE=https://plumbline-ai-9.polsia.app \ node -e "import('@plumbline/eval-sdk').then(m=>m.recordEval({input:'hello',output:'hi',version:'<polsia-model-version>'}))"Framework quickstarts
Three frameworks. One wire.
Every snippet below posts the same EvalRecordCreate body to /api/eval-records under your pl_… bearer key — so a captured run looks identical regardless of which framework minted it.
pytest is the lingua franca of Python test suites. Drop a plumbline_record(input, output) fixture into your existing eval test, attach it as a finalizer on your model-under-test call, and every passing or failing run produces a signed Plumbline record alongside your pytest-html output.
Install
pip install pytest requestsMinimal client
# test_my_model.py
#
# A minimal pytest-style eval that posts a Plumbline record alongside
# the test result. Run with: pytest test_my_model.py -v
import os
import requests
import pytest
API_BASE = os.environ["POLSIA_API_BASE"]
API_KEY = os.environ["POLSIA_API_KEY"]
MODEL_ID = os.environ["POLSIA_MODEL_ID"]
MODEL_VER = os.environ["POLSIA_MODEL_VERSION"]
def _post_record(input_, output_, run_id):
requests.post(
f"{API_BASE}/api/eval-records",
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
json={
"model_id": MODEL_ID,
"run_id": run_id,
"version": MODEL_VER,
"input": input_,
"output": output_,
},
timeout=10,
).raise_for_status()
def test_smoke():
prompt = {"role": "user", "content": "Hello"}
response = {"role": "assistant", "content": "Hi"}
yield _post_record(prompt, response, "pytest-smoke-001")
assert response["content"]Full reference
View pytest how-to guidesOpenAI Evals runs JSON-defined tasks against an OpenAI-compatible completion endpoint. Add a record_plumbline_final step to your eval template's final_summarize and every Eval invocation in CI pushes a signed (input, output, score) record to Plumbline alongside the oaieval summary output.
Install
git clone https://github.com/openai/evals
cd evals
pip install -e .Minimal client
# register_plumbline_eval.py — registered via oaieval
#
# Adds a final_summarize step that forwards every recorded run to
# Plumbline's /api/eval-records alongside the openai/evals summary output.
import os
import requests
from evals.eval import Eval
from evals.registry import registry
API_BASE = os.environ["POLSIA_API_BASE"]
API_KEY = os.environ["POLSIA_API_KEY"]
MODEL_ID = os.environ["POLSIA_MODEL_ID"]
class RecordPlumblineFinal(Eval):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self._run_id = kwargs.get("run_id", "oaieval-final")
def eval(self, input_obj, expected):
output = self.completion_fn(input_obj)
score = self.score(output, expected)
return {
"input": input_obj,
"output": output,
"expected": expected,
"score": score,
"plumbline": self._post_to_plumbline(input_obj, output),
}
def _post_to_plumbline(self, input_obj, output):
resp = requests.post(
f"{API_BASE}/api/eval-records",
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
json={
"model_id": MODEL_ID,
"run_id": self._run_id,
"version": os.environ["POLSIA_MODEL_VERSION"],
"input": input_obj,
"output": output,
},
timeout=10,
)
return resp.json()
registry.add("plumbline-record-final", RecordPlumblineFinal)Full reference
View openai/evals on GitHubBraintrust's Eval() / braintrust SDK runs end-to-end eval loops with built-in scoring. Wrap your scorer with braintrust_eval_with_plumbline(...), which calls your scorer and then forwards (input, output, score) to Plumbline's /api/eval-records so the same run shows up on /dashboard/runs and your Braintrust project simultaneously.
Install
pip install braintrustMinimal client
# braintrust_plumbline.py
#
# Thin wrapper around braintrust.Eval that forwards (input, output,
# score) to Plumbline's /api/eval-records so the run shows up on
# /dashboard/runs AND on your Braintrust project simultaneously.
import os
import requests
import braintrust
API_BASE = os.environ["POLSIA_API_BASE"]
API_KEY = os.environ["POLSIA_API_KEY"]
MODEL_ID = os.environ["POLSIA_MODEL_ID"]
def braintrust_eval_with_plumbline(name, data, scorer, run_id="bt-run-001"):
eval_result = braintrust.Eval(
name=name,
data=data,
task=lambda x: x["expected"],
scores=[scorer],
).run()
for sample in eval_result["results"]:
requests.post(
f"{API_BASE}/api/eval-records",
headers={
"Content-Type": "application/json",
"Authorization": f"Bearer {API_KEY}",
},
json={
"model_id": MODEL_ID,
"run_id": run_id,
"version": os.environ["POLSIA_MODEL_VERSION"],
"input": sample["input"],
"output": sample["output"],
},
timeout=10,
).raise_for_status()
return eval_resultFull reference
View Braintrust docsNext · Pick a direction
Picked a framework. Now register the model.
Mint a fresh pl_… key and grab your paste-runnable snippets, or read the full curl walkthrough end-to-end.