Make search_issues semantic by default - #2964
Merged
iulia-b merged 4 commits intoAug 6, 2026
Merged
Conversation
Copy link
Copy Markdown
Contributor
Sorry, something went wrong.
kelsey-myers
force-pushed
the
km/search-issues-semantic-default
branch
from
2a8189e to
ac663c4
Compare
July 28, 2026 11:19
Copy link
Copy Markdown
Contributor
Author
|
Reasoning for why OSS as opposed to the remote server is detailed in github/plan-track-agentic-toolkit#729 (tradeoffs dropdown) — tl;dr: making a change to expose Client.BaseURL() to handle GHES cases was relatively cheap, we keep one source of truth for the tool description and avoid patching the tool with semantic-specific overrides in the remote repo, and make future changes easier to eval. I'm open to discussing this though! |
Sorry, something went wrong.
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Pull request overview
Makes issue search semantic by default on supported hosts, with lexical fallback for GHES.
Changes:
- Adds host-aware tool configuration and semantic search mode.
- Preserves qualifier quotes while removing free-text quotes.
- Updates tests, snapshots, and generated documentation.
| File | Description |
|---|---|
| README.md | Updates query guidance. |
| pkg/utils/api.go | Exposes host classification. |
| pkg/github/tools.go | Adds host-aware tool options. |
| pkg/github/search_utils.go | Implements semantic search preparation. |
| pkg/github/search_semantic_test.go | Tests quote handling and descriptions. |
| pkg/github/issues.go | Makes issue search host-aware. |
| pkg/github/issues_test.go | Updates semantic request expectations. |
| pkg/github/inventory.go | Propagates tool options. |
| pkg/github/__toolsnaps__/search_issues.snap | Updates default schema snapshot. |
| pkg/github/__toolsnaps__/search_issues_ff_fields_param.snap | Updates flagged schema snapshot. |
| internal/ghmcp/server.go | Configures host type for stdio. |
| docs/insiders-features.md | Updates generated query guidance. |
| docs/feature-flags.md | Updates generated query guidance. |
Review details
- Files reviewed: 13/13 changed files
- Comments generated: 2
- Review effort level: Medium
Sorry, something went wrong.
kelsey-myers
force-pushed
the
km/search-issues-semantic-default
branch
2 times, most recently
from
fd4f747 to
8fa8095
Compare
July 28, 2026 14:07
6 tasks
kelsey-myers
force-pushed
the
km/search-issues-semantic-default
branch
from
8fa8095 to
ccd71b0
Compare
July 29, 2026 08:09
kelsey-myers
marked this pull request as ready for review
July 29, 2026 08:13
kelsey-myers
force-pushed
the
km/search-issues-semantic-default
branch
from
bfb69a3 to
ae1f748
Compare
July 29, 2026 13:56
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
Review details
Suppressed comments (3)pkg/github/issues.go:1679
- Using one translation key for both engine-specific defaults breaks the host-aware description in HTTP mode. TranslationHelper caches the first value for a key (pkg/translations/translations.go:35-50), and HTTP startup first builds default-dotcom tools in initGlobalToolScopeMap (pkg/http/server.go:158-159) before constructing the GHES inventory. The semantic text is therefore cached and returned here even when mode is lexical. Use distinct semantic and lexical translation keys (while deciding how to preserve the existing override key) so the advertised engine matches the handler.
Description: t("TOOL_SEARCH_ISSUES_DESCRIPTION", toolDescription),
pkg/github/search_utils.go:163
- Blindly deleting phrase quotes can turn literal words into search operators. For example, "auth OR login failure" becomes auth OR login failure; the uppercase OR is then interpreted as boolean syntax and, as described in this PR, forces lexical fallback. That defeats the purpose of unquoting and changes the phrase's meaning. Parse free-text quoted segments so reserved operators inside them are neutralized (for example by lowercasing them) when the quotes are removed.
protected := qualifierQuotePattern.ReplaceAllString(query, "${1}"+sentinel+"${2}"+sentinel)
stripped := strings.ReplaceAll(protected, `"`, "")
return strings.ReplaceAll(stripped, sentinel, `"`)
pkg/github/issues.go:1694
- The request-level GHES fallback is not covered: the new description test checks only schema text, while every handler case constructs the zero-value semantic tool. Add a handler test using WithHost(utils.HostTypeGHES) that verifies search_type is omitted (and that semantic quote rewriting is not applied). This directly guards the fallback whose purpose is to prevent GHES 403 responses.
result, err := searchIssuesHandler(ctx, deps, args, mode, options...)
- Files reviewed: 12/12 changed files
- Comments generated: 0 new
- Review effort level: Balanced
Sorry, something went wrong.
RossTarrant
reviewed
Aug 6, 2026
Comment thread
pkg/github/issues.go
Comment on lines
1595
to
1968
| @@ -1647,7 +1676,7 @@ func SearchIssues(t translations.TranslationHelperFunc) inventory.ServerTool { | |||
| ToolsetMetadataIssues, | |||
| mcp.Tool{ | |||
| Name: "search_issues", | |||
| Description: t("TOOL_SEARCH_ISSUES_DESCRIPTION", "Search for issues in GitHub repositories using issues search syntax already scoped to is:issue"), | |||
| Description: t("TOOL_SEARCH_ISSUES_DESCRIPTION", toolDescription), | |||
| Annotations: &mcp.ToolAnnotations{ | |||
| Title: t("TOOL_SEARCH_ISSUES_USER_TITLE", "Search issues"), | |||
| ReadOnlyHint: true, | |||
| @@ -1662,7 +1691,7 @@ func SearchIssues(t translations.TranslationHelperFunc) inventory.ServerTool { | |||
| return utils.NewToolResultError(err.Error()), nil, nil | |||
| } | |||
| options = append(options, withFieldsFiltering(deps, "search_issues", fields)) | |||
| result, err := searchIssuesHandler(ctx, deps, args, options...) | |||
| result, err := searchIssuesHandler(ctx, deps, args, mode, options...) | |||
| return result, nil, err | |||
| }) | |||
| } | |||
| @@ -1930,10 +1959,10 @@ func fetchIssueReadEnrichment(ctx context.Context, gqlClient *githubv4.Client, n | |||
| // searchIssuesHandler runs the REST issues search, enriches each hit with custom field values | |||
| // fetched via a single follow-up GraphQL nodes() query, and applies any post-process options | |||
| // (e.g. IFC labelling). | |||
| func searchIssuesHandler(ctx context.Context, deps ToolDependencies, args map[string]any, options ...searchOption) (*mcp.CallToolResult, error) { | |||
| func searchIssuesHandler(ctx context.Context, deps ToolDependencies, args map[string]any, mode searchMode, options ...searchOption) (*mcp.CallToolResult, error) { | |||
| const errorPrefix = "failed to search issues" | |||
|
|
|||
| query, opts, err := prepareSearchArgs(args, "issue") | |||
| query, opts, err := prepareSearchArgs(args, "issue", mode) | |||
| if err != nil { | |||
| return utils.NewToolResultError(err.Error()), nil | |||
| } | |||
Copy link
Copy Markdown
Contributor
There was a problem hiding this comment.
We might need to account for the translation cache here. In HTTP mode initGlobalToolScopeMap builds the tools without WithHost before the host-aware inventory is created. That caches the semantic description, which means a GHES server can advertise semantic search even though it correctly executes lexical search
Instead could we pass the configured host into the scope-map construction which should keep the description and behavior aligned
Sorry, something went wrong.
RossTarrant
requested changes
Aug 6, 2026
RossTarrant
approved these changes
Aug 6, 2026
1 task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Make search_issues use semantic search by default on hosts that support it, falling back to lexical on GHES.
Why
Closes github/plan-track-agentic-toolkit#729
An A/B experiment on the remote server compared lexical, semantic, hybrid and agent-chosen variants of search_issues over a week. Semantic reduced the number of searches an agent needed per session to 1.7, against lexical's 2.4, with query length roughly flat. Fewer searches means fewer round-trips of results into the agent's context.
Semantic search is not available on GitHub Enterprise Server, where the API returns 403 for search_type=semantic, so those hosts keep lexical behaviour.
What changed
advanced_search is deliberately left alone. The semantic query class already extends the advanced one, so the parameter changes nothing when semantic runs, and on the lexical fallback it makes things worse: under advanced search an unparenthesised boolean binds repo: to only its own clause, so the search escapes the repository the caller asked about. Prod-verified over four query shapes; the existing field. conditional in prepareSearchArgs is unchanged, since field genuinely does require it.
Eval details — how the query description was chosenThe query parameter description was picked by A/B testing four phrasings against a set of benchmark cases that prompt with synonym lists ("auth, authentication, or login problems") to see whether the model reaches for boolean OR. OR isn't supported by semantic search, so the query silently degrades to lexical.
Measured as: of the queries emitted for those cases, how many contained OR.
"When the user gives alternative wordings, include them as plain words rather than joining them with OR."
"Semantic matching already finds related wording, so synonyms don't need listing or joining with OR."
"Semantic matching already covers related wording and synonyms." (no mention of OR)
Findings:
MCP impact
search_issues descriptions changed, and the underlying search engine changes on non-GHES hosts. Parameters are unchanged.
Prompts tested (tool changes only)
Security / limits
Semantic issue search sits in a separate API rate limit family: 10 requests/minute versus 30/minute for standard search. This applies per request based on search_type, so making semantic the default lowers the effective rate limit for issue search on supported hosts. The experiment recorded no rate limit errors across ~76k semantic calls.
Tool renaming
Lint & tests
Docs