Add azd ai eval extension for Foundry evaluations - #9339
Draft
m7md7sien wants to merge 102 commits into
Draft
Conversation
|
Azure Pipelines: 7 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run. |
New azd extension exposing azd ai eval, registering the azure.ai.eval service-target provider. Scaffold only: manifest, entrypoint, root command.
Lifted eval_api (models, operations, poller, generation, portal_urls) and dataset_api from azure.ai.agents, de-agent-scoped. Added evalcore with IsTransientError and an EvaluatorList that accepts either a bare string or a mapping with a threshold. Skipped artifacts.go and eval_config.go since the config model differs.
… commands - EvalConfig and GenerateConfig types with validation covering cross-references, duplicate names, unsupported target types, and evaluation levels - ResolveGroup picks the only group or errors with the available names - ArtifactPath accepts a directory or an explicit file path for local_dir - evalContext resolves the project endpoint (flag, azd env, host env) and builds both clients against the azd developer CLI credential - dataset create/update/list/show/delete with -o json - Tier-0 tests for parsing, validation, group resolution, and path handling
- buildEvalGroupRequest maps evaluators to testing criteria, keeping the builtin prefix on evaluator_name while stripping it from name, and carries per-evaluator thresholds in initialization_parameters - run resolves the group from --eval-id, a pinned id, or the azd environment, creating it when absent, then binds the dataset to the run since the group has no dataset binding today - Local datasets are sent inline with optional truncation; registered datasets are referenced by id
- evaluator upload/update/list/show/builtins/delete; rubric evaluators only in M1, code evaluators deferred to M2 with the folder walk and RBAC they require - normalizeRubricBody accepts a bare definition or a full document - results show/export with per-criteria pass and fail counts, --failed-only, and JSON or CSV output, replacing the counts-only view - Added ListEvaluators, ListEvaluatorVersions, DeleteEvaluatorVersion, and CancelOpenAIEvalRun to the eval client
init scaffolds both YAML files and the artifact directories without any service call, so it works offline and unauthenticated. Built-ins are referenced from the group but never declared as custom evaluators. A dataset flag containing a path becomes a local source; a bare name references a registered dataset. Tests assert the scaffold loads and validates, and that paths are used verbatim rather than re-rooted.
…tion Registers the azure.ai.eval service target so azd up and azd deploy reach this extension; the extension ships no deploy command of its own. - Reads the eval config from the service entry's inline properties, the same AdditionalProperties channel the agents extension uses - Deploy reconciles datasets, then evaluators, then eval groups, since a group references the versions the first two resolve to; it fails fast and the next deploy resumes - Datasets are change-detected with a local SHA-256 digest kept in the azd environment, because the dataset API returns no content hash and comparing against the service would mean downloading the blob every deploy - Evaluator definitions come back inline, so those are compared directly - Package and Publish are no-ops; eval artifacts are plain files already on disk
- generate submits the rubric and dataset generation jobs, downloads the artifacts locally, and writes source references into the deployment spec - MergeArtifactRefs edits through the yaml Node API so comments, key order, and hand-edited sibling keys survive; matching is by name and merging is idempotent - Raised the client poll budget from 2s x 300 to 5s x 720. The old 10 minute limit gave up while the service was still working, which is the timeout that forced a second command - A supplied --evaluator or a local --dataset is honored and its generation is skipped - Tests cover comment preservation, section creation, idempotence, and fingerprinting
Adds the templated release pipeline for the new extension and lists it as a dependency of the microsoft.foundry meta-package. The registry.json artifact entries are generated by the release, so they are not hand-authored here.
… tests Live testing against a real project found two issues. 1. The dataset model only bound snake_case URIs (data_uri, blob_uri), but the project endpoint returns camelCase (dataUri). ResolvedBlobURI therefore returned empty, which would have failed the generate download much later with no useful error. Both spellings are now accepted. 2. Built-in evaluators do not share one input contract. builtin.ifeval requires instruction_id_list and is rejected under the agent-target data mapping with MissingRequiredDataMapping. The live tests now select an evaluator whose inputs match, and the helper documents why. Live tests are gated behind the live build tag and AZURE_AI_EVAL_E2E_LIVE, and clean up every resource they create. Verified: builtin listing, the full dataset pending-upload lifecycle with version auto-increment, and eval group creation returning 201.
…published contract The builder sent one fixed data mapping and one fixed set of initialization parameters to every evaluator. That only suited agent-target quality evaluators, and the service rejected the rest. The evaluator listing publishes a full contract per evaluator: definition.data_schema (accepted and required inputs), definition.init_parameters, and supported_evaluation_levels. The builder now reads it and shapes each testing criterion accordingly. This fixes four concrete defects. - Required inputs were never honoured, so builtin.ifeval (instruction_id_list), builtin.similarity (ground_truth) and builtin.retrieval (context) all failed with MissingRequiredDataMapping. Fields not supplied by the agent target are now bound to dataset columns, and the item schema declares them. - Inputs an evaluator does not accept were sent anyway. - initialization_parameters always carried model, deployment_name and threshold. No evaluator accepts 'model', and builtin.ifeval accepts nothing at all. Parameters are now filtered to the declared properties, and a required one that is missing is reported locally. - evaluation_level was sent as run metadata, where it has no effect. It is an initialization parameter on the evaluators that declare it. It also encodes an exclusivity rule the service enforces: 'messages' and 'query'/'response' cannot both appear in a mapping, so the evaluation level selects between the conversation and turn shapes. A missing dataset column is now caught before the request is sent and names the column, rather than surfacing as a 400 pointing at testing_criteria[0].data_mapping. An evaluator with no published contract keeps the previous agent-target shape, so custom evaluators are unaffected. Verified against a live project: all ten built-ins are accepted, where two previously failed. The live test exercises the shipping builder rather than a hand-rolled request, so a regression in this logic fails the suite. Adds DeleteOpenAIEval so those tests clean up after themselves.
…invokes Without build.ps1 and build.sh the dev kit reported a successful build in under a second and produced no binary, so azd x pack had nothing to package and azd x publish failed with 'Artifacts not found'. Copied from azure.ai.agents with the version package path retargeted, plus its golangci config. Verified end to end: build, pack, publish, install from the local registry, and 'azd ai eval --help' listing every command.
…ly returns The listing spells it evaluator_type, so the TYPE column in 'evaluator list' and 'evaluator builtins' was always blank. Both spellings are now accepted.
Running a real 'azd deploy' against the service-target provider surfaced two failures that no unit test covered. Evaluator references only decoded from YAML. azd hands the service entry to the extension as JSON, so a group written as '- builtin.task_adherence' -- the form the CLI's own init command writes -- failed with 'cannot unmarshal string into EvaluatorRef'. EvaluatorList now decodes and encodes the mixed string-or-mapping form through JSON as well, and a test asserts the two decoders agree. The dataset reconciler passed the declared version straight to UploadNewVersion, which derives the next version from it. A declaration without an explicit version passed empty, so every deploy retried 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset once that version existed. It now looks up the latest registered version first. Verified against a live project: first deploy publishes the dataset at 1.0 and creates the group; an unchanged redeploy reports 'unchanged at version 1.0' and uploads nothing; and editing the dataset publishes 2.0 and recreates the group, since groups are immutable.
azd core does not resolve $ref for extensions. It strips the ServiceConfig fields it owns and leaves $ref at the top of the map for the owning extension to resolve, so a service authored the way the spec documents it -- host: azure.ai.eval plus $ref: ./evals/azure.yaml -- parsed to an empty config. azd deploy then reported success in three seconds having created nothing, which is worse than failing. The provider now calls foundry.ResolveFileRefs with the project root from the azd project client. Relative source paths inside an included file are written against that file, but ResolveFileRefs inlines content without rebasing them, so the include's own directory is now the base for source resolution. Verified against a live project: the $ref form deploys, and the dataset fingerprint matches the one from the equivalent inline config, confirming both forms resolve to the same file.
…ation changes Change detection only covered upstream artifacts, so retargeting a group at a different agent, swapping an evaluator, or changing the judge model left the old group in place. Groups are immutable, so the edit silently had no effect and later runs kept evaluating the previous definition. The group's declaration is now fingerprinted alongside the dataset and evaluator artifacts. The id and description are excluded: one is server assigned and the other is cosmetic, so neither should force a recreate. The digest is recorded when an existing group is reused as well as when one is created. Recording it only on create meant a group deployed before this change never established a baseline, and the first edit after it would still go undetected. Verified against a live project: changing the target produced a new group id, and two further deploys with no change reused it.
The data-plane clients trace every request and response through log.Printf, which Go writes to stderr by default, so a plain command interleaved raw URLs and status lines with its own output. A long generate run was mostly HTTP traces. Ports the debug setup from the agents extension: the standard logger is discarded unless --debug or AZD_EXT_DEBUG is set, and debug output goes to a dated file rather than the terminal. The hook chains the SDK PersistentPreRunE instead of replacing it. Assigning PersistentPreRun has no effect once the E variant is set, and overwriting the E variant would drop the SDK own setup. Also reports jobs as submitted when generate is given --no-wait, which is a successful submission rather than an empty result.
…fails Data generation with an agent source is accepted and then fails within seconds with DataGenerationJobSystemError, whose message says only that something went wrong and to try again. It is not transient: it reproduces for every agent tried, while the identical request without the agent source runs normally. The CLI now names the agent, says a retry will not help, and points at the two workarounds, instead of relaying advice that cannot succeed.
The spec lists run start, list, show and cancel, and M1 requires every operation to be reachable atomically, but run was a single composite command with no subcommands. Listing runs, inspecting one, and cancelling an in-flight run were unreachable, even though the client already had the calls. Adds run list, run show and run cancel. Each takes the eval group id as an optional argument and otherwise falls back to the id recorded in the azd environment, matching results show. Cancelling a run that already reached a terminal state is refused locally, because the service reports success either way and the CLI would otherwise claim to have cancelled a finished run. Two related fixes. Passing --project-endpoint disabled the azd environment cache entirely: the environment name was only resolved when the endpoint came from azd, so every cached eval group and run id lookup returned empty. The name is now resolved independently of where the endpoint came from. The spec documents --wait and --no-wait, but cobra does not derive the negative form from a bool, so --no-wait was rejected as an unknown flag. Verified live: start with --no-wait, list, show, cancel, and the terminal-state guard on a second cancel. JSON output checked on the new subcommands.
The spec lists run start alongside list, show and cancel. The behaviour existed only as the composite `azd ai eval run`, so the atomic name in the spec did not resolve. Both forms are now built by one constructor, so their flags cannot drift apart, and a test asserts that.
… work Exercising the atomic write commands against a live project found three failures. None were covered by tests, because none of these paths had been run end to end. dataset update always collided. It passed the --version flag straight to UploadNewVersion, which derives the next version from what it is given, so an omitted flag restarted at 1.0 and the service returned 409 TemporaryDataReferencesForExistingAsset. The flag help promised the opposite, that omitting it would take the next version. This is the same defect that was fixed in the deploy reconciler earlier, so the discovery is now centralised in DatasetClient.UploadNextVersion and both callers use it, rather than being fixed twice and available to be missed a third time. evaluator upload rejected every hand-authored rubric. The service needs a type discriminator on the definition, and without it fails the whole request with "The request field is required", which names a field that is present. Generated rubrics carry the type, so only the hand-authored path documented in the spec was affected. The type is now filled in when absent and left alone when set. evaluator show returned 404. It omitted the version segment from the path, but the service has no route for an unversioned evaluator, despite the doc comment claiming the latest would be fetched. The latest version is now resolved first, comparing numerically because versions are integers as strings and a lexical compare ranks "9" above "15" -- the service already publishes evaluators at version 15 and 17. Verified live: dataset create, show, update to 2.0, list and delete; evaluator upload, show resolving the latest, update to version 2, list and delete. Both suites leave nothing behind.
…nfig Deploying a config that declares a custom evaluator, rather than only built-in ones, failed in two ways. Every earlier test used built-ins, so neither showed up. The evaluator was republished on every deploy. The service enriches a definition when it stores it, so a rubric consisting of nothing but type and dimensions comes back carrying data_schema, init_parameters and metrics it was never given. Comparing whole documents therefore never matched. Only the keys the author actually wrote are compared now, structurally, so key order and formatting are not changes either. This is what the spec means by repeated azd up creating no redundant versions. The eval group was then rejected with a request for a model that had been set. Evaluators disagree on what the judge model is called: built-ins declare deployment_name, and a custom rubric declares model. The builder sent only deployment_name, so the custom evaluator saw its required parameter missing. The judge model is now bound under whichever name the evaluator declares. Verified live: first deploy publishes the evaluator and creates the group, two redeploys report it unchanged and publish nothing, editing the rubric publishes the next version and recreates the group, and a further redeploy is a no-op again.
…d dataset The flag is documented as taking a path or the name of a registered dataset, and means use this one instead of generating. It only suppressed generation when the value looked like a local path, so passing the name of an existing dataset still submitted a generation job and, since agent-seeded generation is currently broken server-side, failed the whole command. --evaluator already skipped unconditionally, so the two flags disagreed. Both the skip and the default-spec synthesis now key off whether the flag was supplied at all. This was the last thing standing between a generated config and the documented end-to-end flow. Verified live: init scaffolds a group referencing its own rubric, generate writes that rubric and merges the reference into the same file while preserving comments and ordering, azd up registers the dataset and evaluator and creates the group, and the run completes and scores against the generated rubric.
…listing
GET /datasets/{name}/versions returns nothing for a second or two after a
version is created, even though the version itself reads back immediately.
Measured: empty at 0s, populated at 2s.
That undermines the version discovery added for dataset update, which reads the
listing to decide what to increment from. An empty listing is ambiguous -- it
means either a new dataset or a stale read -- so back-to-back create and update
could still restart at 1.0 and take a 409.
Rather than delaying every first upload to wait for the index, a conflict is
now treated as the stale read it is: re-read the listing, which by then
reflects reality, and retry once. The common path is unchanged.
The live test asserted on the first listing response and was failing for the
same reason. It now polls, and says why.
Verified: create immediately followed by update produces 2.0 rather than a
conflict, and the full live suite passes.
…requires The shared extension build template invokes ci-build.ps1 and ci-test.ps1 from the extension directory. Neither existed, so the release pipeline added alongside this extension would have failed on its first run. Both are modelled on the agents extension with two deliberate differences. ci-build.ps1 reads version.txt from the extension directory rather than its parent, where no such file exists, so the default works when the pipeline is not supplying -Version. It accepts -BuildRecordMode, which the template always passes, but builds nothing extra: this extension has no record/playback mode and no pipeline step consumes a record binary. ci-test.ps1 passes --junitfile explicitly. The pipeline publishes **/junitTestReport.xml from the extension directory, and the extension template does not set GOTESTSUM_JUNITFILE the way the CLI build does, so without this no test results would surface in the build. Verified locally with gotestsum installed: 97 tests across 7 suites reported. Also adds the README and CHANGELOG that 17 of the 21 extensions ship. The README documents the deployed shape, the command surface, the rubric weight constraint, and how to run the live tests.
The manifest declared two capabilities the extension did not back. metadata was declared but the command was never registered, so azd could not discover the command tree: azd ai eval metadata failed with unknown command while the same call against a peer extension returned its full tree. azd uses this for discovery, so the declaration was actively misleading. The command is now registered and reports nine commands. lifecycle-events was declared but no event handlers exist. The SDK only starts its event manager when handlers are registered, so the capability was an unused permission rather than a broken promise. It is removed; the listen command is still invoked because the service-target-provider capability triggers it, which a deploy after the change confirms. Adds tests over the manifest so neither can drift again: every declared capability must be backed by the command that implements it, the declared provider name must match the host the code registers, and version.txt must agree with the manifest version, which until now was only a comment asking for it.
…ction from a file Auditing every flag and API sequence the spec documents against the running extension turned up two gaps. The spec describes a drift check that was never implemented. It matters because of how change detection works: when local content is unchanged, the version recorded at the last deploy is reused, so a version published outside the repo would be silently ignored and the eval group pinned to older data. A deploy now fails when the service holds a newer version than the recorded one, naming both versions. An explicit version: on the declaration skips the check, because that is the author stating which version they want. This was added after testing the remedy the error message suggests and finding it did not work -- the message now describes something that does. --gen-instruction-file was documented but absent. A useful generation instruction is usually longer than fits on a command line, and putting it in a file makes it reviewable with the rest of the config. Verified live: publishing a version out-of-band fails the next deploy, and pinning that version lets it through.
… edits M1 exits on all the spec examples running end to end, so I ran them verbatim. Two did not. --eval-id could never work. It is meant to run an existing group ignoring the config, and appears in both the CI/CD example and the recovery advice, but a run needs a target and a dataset and an eval group carries neither: the group holds only its testing criteria, and the dataset travels on the run. Every --eval-id invocation failed asking for a target. The pairing survives in the group's previous run, so re-running a group now repeats what it last ran, and a group that has never run says so and points at the config-based path. The failure-and-recovery example promised an error that did not exist. A run sends a local dataset inline, so unregistered local edits were evaluated silently and the results could not be traced to any dataset version. That now fails with the message the spec documents, once a deploy has recorded a fingerprint to compare against. Before that there is nothing to have drifted from, and running is how a group first comes into existence. Verified live: the CI/CD example returns JSON with a run id, and with unregistered edits the config-based run fails while --eval-id succeeds, which is exactly the recovery the spec describes.
Trace-backed, stored-response, model-target and target-less runs are all deferred, so their data sources, the flags that reached them, and the config that declared them come out. What remains is an agent target over a dataset, which is what the four hero scenarios use. An eval that names no target is now an error rather than a target-less run. It used to mean the rows already held both sides of the exchange, and that is exactly the deferred case, so accepting it silently would ship the feature through the back door. The code evaluator settings leave EvaluatorDecl for the same reason: image_tag, metrics, data_schema and init_parameters only ever fed a python grader.
Name positionally, content through --from-file, and versions under their own subcommand so that list means the project's datasets rather than one dataset's history. create and update split the same upload the way they do for evaluators. Also drops a stray copy of eval_api/schedules.go from the working tree. It was left behind by the checkout that built the M2 branch, was never tracked here, and had been compiling into M1 ever since - dead code that M1 does not call and that its own history does not contain.
Evals gain list, show and delete at the root; creation stays with azd up so
that reconciliation has one owner. Jobs gain list, show and cancel, which is
the resume path for a generate started with --no-wait.
Both job routes turned out to carry more than the spec's table records: list
and cancel exist for datasets and evaluators alike, verified against the live
project. Cancel is the colon form - {id}:cancel reaches the action while
{id}/cancel is a 404 - and needs a body, because without a content type the
route answers 415.
A job id does not say which kind started it, so show and cancel try both
rather than making the caller remember.
The per-sample views move under the run they belong to: run output list, show and export. The positional argument is now the run rather than the eval, since the eval is what --eval names and the API path nests runs under it. run output show is new, and reads one row whole. The listing truncates the input and the reason to keep a table readable, so there was no way to see either in full. run stops being executable. Once run output exists, a bare run would make azd ai eval run list read as run the thing called list.
Generation becomes dataset generate and evaluator generate, because the service splits it the same way: two long-running resources, one per artifact. A single composite verb left partial failure undefined, could not regenerate one artifact after the other was hand-edited, and gave --no-wait nothing to reattach to. Neither command edits azure.yaml any more - init declares where the artifacts live, these fill them in, so a generation run produces a data-file-only diff. --fail-on gates a build on quality: any-failure, or pass-rate=<0..1>. It is opt-in, because failing samples are the expected output of a working evaluation and a default that returned non-zero would break a build the first time a grader disagreed. Errored rows count against the rate - they are in the total and did not pass - and a run that scored nothing breaches rather than dividing by zero, so a broken evaluation cannot hold a gate open. The exit code matters more than the flag: 2 says the evaluation regressed, 1 says it could not run. Those have different owners. It exits from applyGate because the extension SDK collapses every returned error to 1.
The pass rate reads as a percentage and the failure prints the two lines the CI scenario shows, so a pipeline log looks like the spec rather than close to it. Also records what the counts mean, which the contract left ambiguous: errored rows sit inside total. Verified on a run built to mix them - total=3 passed=2 errored=1. Had they been outside it, two passes and one error would have reported total=2 and scored a perfect rate, which is the broken evaluation a gate exists to catch.
create and update differ only in which starting state they accept, and the check was asking the version listing. That listing lags a publish by up to a second and a half, so an update issued straight after a create was told the evaluator it had just made did not exist - which the live suite caught, with show succeeding between the two because the direct read had already converged. The same lag is why reconciliation waits on both views before creating an eval that references a fresh evaluator. Here only one view is needed, and it is the faster one.
The spec's configuration model is one `azure.ai.eval` service per eval, with the eval's body in `evals/<eval-name>.yaml` and the service key supplying its name. The extension implemented an older shape: a single `evals/azure.yaml` carrying parallel `datasets:`, `evaluators:` and `evals:` lists. That made the service key meaningless, gave azd no way to order one eval after the agent it evaluates, and forced a name to be repeated in three places. - `EvalConfig` is now one eval: `description`, `dataset`, `evaluators`, `target`, `options`. `Eval(name)` resolves it against the service key. - Evaluators are declared once, in the list the eval references. A built-in needs nothing; one carrying a `source` is the config's to publish. `CustomEvaluators()` is that subset. - `ResolveEvalConfigPath` resolves `--eval <name>` to a file, and names the candidates rather than guessing when there is more than one. `options.eval_model` is gone. A judge deployment is a testing criterion's `initialization_parameters.deployment_name`, which differs per evaluator, so it is declared on the evaluator reference; `--judge-model` goes with it. Verified against the live schemas: every built-in that judges declares `deployment_name` required, so `init` stamps the detected deployment onto each evaluator it writes rather than leaving behind an eval that cannot run. `EvaluatorList.MarshalJSON` dropped `source` and `initialization_parameters`. The eval fingerprint is taken over that encoding, so a judge-model change was invisible to reconciliation and would not have recreated the eval. Caught by the fingerprint test. `evals/generate.yaml` takes its documented shape: `generationModel` plus `dataset:`/`evaluator:` maps keyed by artifact name, so `dataset generate <name>` reads exactly the entry it was asked for. The resolution order -- flags, spec, then the eval's own target -- is settled in `resolvePlan` before the client is built, so a missing model or an out-of-range sample count costs no authentication round trip. `MergeArtifactRefs` is deleted. The spec is explicit that `generate` writes artifacts only and never edits `azure.yaml`, which is what keeps a generation run a reviewable data-file-only diff; the code was already unreachable from any command. `init` follows: `--name`, `--output-dir` (was `--out-dir`), `--generation-model`, a `<target>-smoke` default eval name, the built-in plus generated rubric default evaluator set, and a service entry keyed by the eval name that `uses` both the Foundry project and the target agent.
`dataset generate d --agent-instruction-file <absent>` answered "a model deployment is required", naming a flag the caller had not touched and saying nothing about the one they had. Splitting the offline resolution out of `prepareGeneration` had moved the instruction-file read behind the model check, and both checks are local, so the only thing deciding which error the user sees is the order they run in. The instruction file is read in `resolvePlan` now. An input the caller named and got wrong is a typo they can act on; a missing generation model has a documented default path and is the weaker complaint. Caught by the live CLI suite. Pinned offline now too, since the ordering is exactly the kind of thing a refactor moves without noticing and a 500-second live run is a slow way to find out.
`init` is the first command in Scenario 1 and the one that produces the local diff every later step depends on, and nothing was testing it end to end. It resolves the project and edits azure.yaml over azd's gRPC channel, so the CLI suite -- which runs the extension binary directly -- cannot reach it: with no azd on the other end the command refuses before it does anything. What covered it instead was a unit test calling the scaffold function, which cannot see the service entry azd writes, the detection that reads the project, or the terminal output the spec pins line for line. tests/hero runs the extension the way a user installs it. Scenario 1's output is compared whole rather than by keyword: every line is a promise the spec makes to a reader deciding whether to adopt this, and a keyword assertion would pass while their terminal said something else. It found one: with `--dataset prod-golden` -- a dataset that already exists -- `init` still closed with "Next: azd ai eval dataset generate prod-golden", sending the reader to submit a billed generation job for an artifact they had just supplied. Next steps are now built from what was actually scheduled, and when everything is already in place they point at `azd up` instead. Two properties are asserted that only this harness can see: that `init` makes no service calls, by running it behind a proxy pointing nowhere; and that a second `init` refuses without having edited the project, then under --force leaves exactly one eval service rather than a duplicate that would deploy the same eval twice. TestMain fails rather than skips, and rejects a stale install. A suite that silently reports on a binary other than the one under test is worse than no suite, and `azd x pack` leaves the local registry's checksum alone when the version has not changed, so a stale install is the easy mistake to make. The guard builds the working tree and compares help output, which is the cheapest fingerprint that actually moves with the code. Verified by making it fail.
The flag that writes results to a file was `--out-file`. The spec, its Scenario 4, and `azd ai skill download` all say `--output-file`. It took reading the two documents side by side to notice, which is the wrong way to find something a user types from memory. The surface is a contract -- with the spec, and with the sibling Foundry extensions whose vocabulary this shares -- and nothing was checking it. So the tree and the flags are pinned: - Every command in the tree, so one that is renamed, dropped or quietly added has to be acknowledged against the spec's command table. - `init`'s flags whole, because they are a table in the spec: an extra flag there is a promise the spec does not make, a missing one is a promise it does. - The spellings this extension must not invent, including the ones the spec explicitly rules out (`--judge-model`) and the ones belonging to M2 (`--from-traces`, `--folder`, `--cron`), so a deferred surface cannot reappear by accident. - That `init` takes no `--project-endpoint` and every command that does reach the service takes one. Verified by renaming the flag back and watching both guards fail.
`run start --no-wait` closed with "Check progress with: azd ai eval results show <eval> --run-id <run>". `results` became `run output` when the surface was rebuilt, so the one instruction printed at the exact moment a user needs it was the one thing guaranteed to fail. The no-previous-run error had the same problem, sending the reader to a bare `azd ai eval run` that is now a group. Nothing catches this on its own: the string compiles, the command that prints it succeeds, and only somebody who follows the advice finds out. So every `azd ai eval ...` in a non-comment line is now resolved against the real command tree, with trailing prose trimmed so "run start and summarize" is read as the command it begins with. Verified by restoring the original message and watching the test name it. Also fixes an assertion that the live suite caught being wrong rather than the code: `--failed-only` counted the bare substring "FAIL" against `ResultCounts.Failed`. Two things were conflated. The per-criterion table's FAILED column header contains FAIL, and one rendered line is one evaluator's verdict on one sample -- a sample failing two evaluators is two lines, while `ResultCounts.Failed` counts samples. It matches verdict cells on a word boundary against the failing results in the payload now, so it holds whatever the judge decides.
build.sh cross-compiles six platforms. Every build and test in this extension's history has been windows/amd64, so five of them ship unexecuted. The test suites carry the same assumption rather than guarding against it: both harnesses derive the .exe suffix from the path separator, and the hero suite shells out to \�zd\ and to a proxy address, none of which has been exercised off Windows. Recorded next to the note that already says what CI does and does not cover, since that is what a release reads.
The previous note claimed only windows/amd64 had ever been executed. That was true of my desk, not of CI: release-ext-azure-ai-evaluations.yml has a pr: trigger on this directory, and every BuildMatrix job -- windows, linux and darwin amd64 -- runs this script. The untagged tests are covered on all three. What is genuinely uncovered is narrower and worse: ci-test.ps1 runs go test ./... untagged, so the live and hero suites are never compiled by any pipeline on any platform. Both have only ever run on Windows, by hand. Darwin needs no manual pass -- CI has it. Linux does, for the tagged suites only.
ci-test.ps1 runs go test ./... untagged, so nothing in any pipeline compiled the live or hero suites. A change that broke one reached main silently -- the surface tests added earlier would not catch it either, because they are untagged too. go vet -tags live,hero needs no credentials and no endpoint, so it costs a few seconds and runs on every platform the tests already do. It runs after the tests so a genuine test failure still reports first. Verified both ways: breaking a file under either tag fails the script with exit 1 and suppresses the success line; the clean tree exits 0.
For a few seconds after a publish the service can answer the next one with the version it just assigned, writing over that version's definition instead of adding one. So \evaluator create\ followed by \evaluator update\ - which is what a first authoring session looks like - reported success and left a single version holding the second rubric, with every eval bound to the first scoring against something nobody chose. It cannot be waited out by watching for state, because there is none to watch: a publish 258ms after the first came back as version 1 and one six seconds later advanced to 2, while the version listing was already answering with 1 - and immediately after a create that same listing 404s, so a guard trusting it would stand down in exactly the case it exists for. What is reliable is what the caller has already read. Both call sites read the evaluator to decide between creating and updating, so that document is passed into the publish: it says which version exists and when it was written. The publish waits until that version has had a few seconds, then checks that what came back is a new one, reissuing the same body if it is not. A collision that happens anyway cannot be undone - the request that collided has already written - so the recheck is bounded and ends in an error naming the version that was replaced, rather than in a reported success.
\--no-wait\ exists so the caller can walk away, and the line they walk away
with is the one they paste when they come back. It said
Submitted. \�zd ai eval job show <job-id>\ reports its progress.
The id was on the line above, so nothing was lost - but the instruction itself
was not usable, and the check that every command the CLI suggests is a real
command passed it, because \job show\ is one. A second check now refuses a
suggestion that still has a placeholder in it, and it was confirmed to fail on
the line it was written for.
Also drops checkEvaluatorDrift. It reads the version listing to catch a
version published outside the repo, and nothing calls it: the rubric path
always reads the latest version and republishes when the local definition
differs, so there is no window for it to guard. It belongs to the code
evaluator path, which fingerprints instead of comparing, and that is M2's.
Two gaps in the exit-code contract, both found by reading the spec's own promises back against the code. A run that finished with status failed or error exited 0. The reason was printed, so a person reading the terminal saw it, but a pipeline gating on the command saw success - and the spec makes exit 1 mean exactly this case. It is checked before the gate, because the gate's exit 2 means the evaluation regressed and a run that produced no results has not regressed; telling those apart is what the separate code is for. \ un show\ only reports it when --wait was asked: without it, this is an inspection command and answering the question is a success whatever the answer. And evaluator drift was never detected. checkEvaluatorDrift existed and nothing called it, so \�zd up\ would publish over a version somebody had deliberately published from the portal and report success. It is called now, and it is answered from the version recorded at the last deploy rather than from the version listing - the listing lags a publish, and would report an evaluator as un-drifted for the first seconds of its newest version's life. Evaluators need this comparison at all only because their definitions come back inline: local and remote disagreeing says nothing on its own about who moved.
\ un start\ waited for a run and then printed one line: its id and its status. The verdict per evaluator - which is the entire reason to run one - was returned by the service and thrown away, so the answer to the question the command had just been asked took a second command to see. It now prints a row per evaluator with pass, fail and pass rate, sorted by name so two runs of the same eval read the same way, followed by the overall rate over samples rather than over verdicts: a sample that failed two evaluators is one sample to go and look at. Errored rows are named rather than folded into the fail column. The evaluator never reached a verdict on them, and counting them as failures reports a service problem as a quality problem. For the same reason a criterion's rate is over what it scored, and a rate over nothing prints as a dash - 0.0% would read as a total failure rather than as no data.
Two things. \ un start --no-wait -o json\ printed the service's run object verbatim. A pipeline reading it to reattach later had to know that the run id is \id\, that the eval id echoed back is not necessarily the one the command resolved, and that the declaration name it wants to log is not there at all - while depending on every other field the API happens to return. It now emits the handoff the spec documents: run_id, eval_id, eval_name, status, created_at, and nothing this extension does not promise to keep. And the live suite's credential now retries. The token refresh shells out to azd and the SDK gives that subprocess ten seconds; with the whole suite running it sometimes does not finish in ten, which surfaced as \AzureDeveloperCLICredential: exit status 1\ on whichever test asked at the wrong moment - reproducibly at 10.1s, and never when that test ran alone. Nothing about the request was wrong, so it is retried rather than reported.
The handoff shape was covered by a unit test and nothing else, so nothing proved the id it hands back is one the next command can actually resolve - which is the only property a pipeline cares about. This starts a run without waiting, reads run_id out of the JSON, and reattaches with it. The plain-output half asserts the printed reattach line carries the real eval id and contains no angle bracket at all, because the placeholder bug this replaces was in exactly such a line and read as though it were a command.
Every other extension has a cspell.yaml importing the shared config and listing its own vocabulary. This one had none, so all 96 of its words were unknown - the module name, the package names, the built-in evaluator names, and a handful of ordinary English cspell does not carry. Added, along with US spellings for the four words that were genuinely British and a format string whose %s butted against the next word and read as one. And pkg/extensions has a guard requiring every extension that declares providers to prove the ones it registers match its manifest. The provider was declared and registered correctly; the test asserting so was simply absent, so nothing would have caught them drifting apart. Also: a dataset create followed immediately by an update was refused as though the dataset did not exist. The existence check read the version listing, which lags a publish - the same thing evaluators were already reading directly - so it now falls back to a point read of version 1, which goes consistent at once. And run start --no-wait now renders created_at as RFC 3339. The service sends epoch seconds on a run and a formatted string elsewhere, so passing it through handed a script a value whose type depended on which route produced it.
M1's exit criteria include \-o json\ and \--no-prompt\ throughout. Both come from the SDK's root command, so every command inherits them and nothing had to be checked - right up until a command declares a flag by one of those names, which shadows the global silently and leaves that one command unable to answer in JSON or to run unattended. Nothing was watching for that.
…te from the repo The reconciler publishes at a pinned version and treats a conflict as the signal to stop, which only holds if the service refuses the write. Nothing checked that. It does refuse - 409 on the pending upload - so the assumption was sound, but it was an assumption about someone else's service with an eval's scoring hanging off it. Now it fails loudly if that ever changes. The companion test covers the ordinary path: two uploads in a row must add two versions, issued back to back because the version listing lags a publish and that is the window where the second could be told the dataset is new and restart at the version the first just took. Also removes azd-eval-extension-implementation.md. It was a working note from building this, not documentation anyone should find in the extension - and it had gone stale, still calling M2's features M4.
…them Two shapes were the reviewer's, and ours had drifted from both. The run summary now leads with the run's identity - which eval, how many samples, how long - before the per-evaluator table, and the table carries the mean score. The service does not return a mean, so it is averaged over the rows the run scored; rows an evaluator never scored are left out rather than counted as zero, which would drag the average toward a number nothing produced. When the rows were not read the column is dropped rather than filled with dashes, so an absent average never reads as a bad one. And \ un output list\ is one row per evaluated sample instead of one per verdict. A sample that failed three evaluators was listed three times, which makes a run look three times as broken as it is - and the row now names every evaluator that failed it, which is what says whether the sample is wrong or one evaluator is.
A second upload issued moments after the first was refused with a 409 and the conflict was handed to the user, for a publish that should simply have added a version. The recovery re-read the version listing and gave up when it was still behind - but the listing lags a publish by a second or two, which is exactly the window the recovery exists for, so it gave up precisely when it was needed. The version the service just refused is proof that version exists, whatever the listing says, so the next one is tried. The listing is still consulted and still wins when it has caught up and moved further ahead, because then it knows about versions somebody else published. The walk is bounded, so a service refusing everything ends in the conflict rather than in a loop. Found by the full live suite, which is the only place the two uploads land close enough together; the same test passes alone because the listing has time to settle. The offline tests reproduce it deterministically instead.
A bundle dependency has to resolve from the extension registry at install time. azure.ai.evaluations is not published yet, so listing it would break 'azd extension install microsoft.foundry' for everyone the moment that bundle ships, whether or not they want the eval extension. The two release from separate pipelines and neither waits for the other, so the entry goes in only once azure.ai.evaluations is in the registry. This PR therefore leaves the shared bundle untouched; the release pipeline it does add is what publishes the extension in the first place.
m7md7sien
force-pushed
the
feat/azure-ai-evaluations-extension
branch
from
August 4, 2026 21:20
f432aad to
a244703
Compare
Nothing in the extension calls it. Endpoint resolution reads --project-endpoint, the azd environment and the host environment variable, and none of them touch another extension's config. The only mention of azure.ai.project is the service host, used to order after a Foundry project service when the repo already declares one — and that case already implies the user has the extension. A hard dependency would install azure.ai.projects for everyone using eval, including the endpoint-only repos this extension deliberately supports, and pin them to a version range they never asked for. Every sibling but azure.ai.agents declares none.
It recorded what existed at one commit and how much of it was proven, which the tests and the spec now each say better. A hand-maintained command list is wrong the moment a flag moves, and nothing linked to it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
azd ai eval, a new azd extension for Foundry evaluations, implementing the design spec.Status: draft. The spec is still in review (foundrysdk_specs#251) and open comments there may still move the command surface.
What it does
azd ai eval initscaffoldsevals/eval_generate.yamlandevals/azure.yaml, and declares the eval service in the project'sazure.yamlthrough azd's ownProject().AddService— the same call the agents extension uses, so azd owns that edit.azd ai eval generatesubmits the data- and rubric-generation jobs, downloads the artifacts locally, and writessource:references back intoevals/azure.yaml.azd up/azd deployreconcile datasets, evaluators and eval groups through aservice-target-providerregistered forhost: azure.ai.eval.azd ai eval runensures the eval group exists, starts a run, waits and summarizes.Atomic commands cover
dataset,evaluator,run,resultsandschedule. Every read command supports-o json; every command supports--no-prompt.Verification
-tags live.Notes for reviewers
$refis resolved by the extension: azd core strips the fields it owns and leaves$refinAdditionalPropertiesfor the owning extension.