Reimplement ToolDescriptionEvaluator VectorDB on Microsoft.Extensions.VectorData#3057
Conversation
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
|
@copilot Pull the latest changes from main into this branch. That'll take care of the failing pipelines |
|
@copilot Review and streamline the unit tests added for Make these changes:
Preserve tests covering custom behavior: sorted upsert/update/delete operations, batch upsert, top-K truncation, score thresholds, supported and unsupported search inputs, validation, unsupported Do not change production behavior or perform unrelated cleanup. Run the smallest applicable |
Addressed in ef0cbda. Removed the redundant metric/property/parallel/end-to-end tests, strengthened the store metadata assertion to verify |
|
Azure Pipelines: Successfully started running 1 pipeline(s). There may be pipelines that require an authorized user to comment /azp run to run. |
There was a problem hiding this comment.
Pull request overview
This PR refactors the ToolDescriptionEvaluator in-memory vector database to implement the Microsoft.Extensions.VectorData abstractions, enabling provider-swappability (e.g., Azure AI Search/Cosmos DB/Redis) while preserving the existing brute-force similarity search behavior and updating the CLI to use the new async collection APIs.
Changes:
- Reworked the in-memory vector store to derive from
VectorStoreand exposeVectorStoreCollection<string, Entry>-compatible collections. - Updated the evaluator (
Program.cs) to populate and query viaUpsertAsync/SearchAsyncand consumeVectorSearchResult<Entry>(Record/Score). - Added unit test project and coverage for
VectorDBandInMemoryVectorStoreCollection, and addedMicrosoft.Extensions.VectorData.Abstractionsvia central package management.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| Microsoft.Mcp.slnx | Adds the ToolDescriptionEvaluator unit test project to the main solution structure. |
| eng/tools/ToolDescriptionEvaluator/ToolDescriptionEvaluator.sln | Adds unit test project and expands solution configurations. |
| eng/tools/ToolDescriptionEvaluator/tests/ToolDescriptionEvaluator.UnitTests/VectorDBTests.cs | New unit tests validating VectorDB store behavior and metadata. |
| eng/tools/ToolDescriptionEvaluator/tests/ToolDescriptionEvaluator.UnitTests/InMemoryVectorStoreCollectionTests.cs | New unit tests validating collection upsert/get/delete/search semantics. |
| eng/tools/ToolDescriptionEvaluator/tests/ToolDescriptionEvaluator.UnitTests/ToolDescriptionEvaluator.UnitTests.csproj | Adds the new unit test project definition and references. |
| eng/tools/ToolDescriptionEvaluator/src/VectorDb/VectorDB.cs | Implements VectorStore + VectorStoreCollection in-memory store/collection, span-based metrics, and vector search. |
| eng/tools/ToolDescriptionEvaluator/src/ToolDescriptionEvaluator.csproj | Adds Microsoft.Extensions.VectorData.Abstractions dependency. |
| eng/tools/ToolDescriptionEvaluator/src/Program.cs | Migrates evaluator logic to async VectorData collection APIs and adjusts output/process encoding behavior. |
| Directory.Packages.props | Adds central package version for Microsoft.Extensions.VectorData.Abstractions. |
| public override VectorStoreCollection<TKey, TRecord> GetCollection<TKey, TRecord>(string name, VectorStoreCollectionDefinition? definition = null) | ||
| { | ||
| if (typeof(TKey) != typeof(string) || typeof(TRecord) != typeof(Entry)) | ||
| { | ||
| throw new NotSupportedException($"This store only supports collections of <{nameof(String)}, {nameof(Entry)}>."); | ||
| } | ||
|
|
||
| return (VectorStoreCollection<TKey, TRecord>)(object)GetOrCreateCollection(name); | ||
| } |
| private IComparer<VectorSearchResult<Entry>> ResultComparer => Comparer<VectorSearchResult<Entry>>.Create((a, b) => | ||
| { | ||
| int result = Nullable.Compare(a.Score, b.Score); | ||
| return _distanceMetric.BiggerIsCloser ? -result : result; | ||
| }); |
The
ToolDescriptionEvaluator'sVectorDBwas a bespoke in-memory store with a custom API. Building it on theMicrosoft.Extensions.VectorDataabstractions lets it be swapped for another provider with minimal consumer changes and unblocks future use ofMicrosoft.Extensions.DataIngestionfor chunking and processing.Changes
VectorDBfromVectorStoreand addedInMemoryVectorStoreCollection : VectorStoreCollection<string, Entry>. The existing brute-force cosine-similarity search, sorted-key storage, and parallel divide-and-conquer query path are preserved behind the standardUpsertAsync,SearchAsync,GetAsync, andDeleteAsyncsurface.EntrywithVectorStoreKey,VectorStoreData, andVectorStoreVectorattributes.IDistanceMetric,CosineSimilarity, andDotProduct, now using spans for allocation-freeTensorPrimitivescalls.Program.csto resolve the collection from the store and consumeVectorSearchResult<Entry>records and scores.azmcp tools listunder Windows code page 65001 and decodes redirected stdout and stderr as UTF-8. This prevents Unicode characters such as→from being emitted as the invalid JSON control byte0x1A; Linux and macOS retain direct process execution.Microsoft.Extensions.VectorData.Abstractionsthrough central package management and registered the new unit-test project in the relevant solutions.Notes
GetAsync(Expression<...>)intentionally throwsNotSupportedException; implementing it requiresExpression.Compile(), which is incompatible with AOT. The evaluator does not use this overload.top; ranking of the requested results remains unchanged.ProcessStartInfo.StandardOutputEncodingcontrols parent-side decoding only. The Windows wrapper is also required to make the child process emit UTF-8 rather than OEM code-page bytes.Validation
Azure.Mcp.Server.csprojandToolDescriptionEvaluator.slnbuild successfully.ToolDescriptionEvaluator.UnitTestspass..\scripts\Run-ToolDescriptionEvaluator.ps1 -Area "keyvault"dynamically loaded and saved all 340 tools without the previous JSON deserialization failure. Full scoring still requires valid Azure OpenAI credentials.Invoking Livetests
Copilot submitted PRs are not trustworthy by default. Users with
writeaccess to the repo need to validate the contents of this PR before leaving a comment with the text/azp run mcp - pullrequest - live. This will trigger the necessary livetest workflows to complete required validation.