-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Improve agent behavior: investigate before speculating #2960
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Improve agent behavior: investigate before speculating #2960
Conversation
- Add investigation-first approach to all prompt templates - Replace 'minimize questions' with 'investigate immediately' requirement - Never speculate without verification using available tools - Add action word analysis system for request categorization - Enhance completion criteria with root cause fixing & verification - Add semantic understanding for ambiguous terms
📬 CODENOTIFYThe following users are being notified based on files changed in this PR: @bhavyausMatched files:
@bryanchen-dMatched files:
|
|
@microsoft-github-policy-service agree |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This pull request aims to improve agent behavior by adding investigation-first prompting patterns and creating an action word analysis system. The changes add comprehensive guidelines across multiple AI model prompts to encourage root cause investigation, verification testing, and semantic understanding of user requests.
Changes:
- Added investigation and verification guidelines to all agent prompt templates (OpenAI GPT-4, GPT-5, GPT-5.1, Claude 4.5, and default prompts)
- Created ActionWordService and actionWordAnalysis module to categorize user requests by action words (fix, create, modify, etc.)
- Added semantic understanding guidance with specific domain examples to reduce clarifying questions
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| src/extension/prompts/node/agent/openai/hiddenModelBPrompt.tsx | Added 5 new guidelines emphasizing verification testing, root cause fixing, semantic understanding, and immediate investigation over speculation |
| src/extension/prompts/node/agent/openai/gpt5Prompt.tsx | Added identical 5 new guidelines for GPT-5 model prompts |
| src/extension/prompts/node/agent/openai/gpt51Prompt.tsx | Added extensive completion criteria, semantic understanding section, and quality standards (27 new lines) for GPT-5.1 |
| src/extension/prompts/node/agent/openai/defaultOpenAIPrompt.tsx | Added investigation and verification guidelines to the keep-going reminder section |
| src/extension/prompts/node/agent/anthropicPrompts.tsx | Added structured completion_criteria and semantic_understanding_and_quality tags with comprehensive guidelines for Claude models |
| src/extension/prompt/common/actionWordService.ts | Created new service interface and implementation for analyzing user requests based on action words |
| src/extension/prompt/common/actionWordAnalysis.ts | Implemented action word detection, categorization, and completion criteria logic with dictionaries of action/subject words |
| src/extension/extension/vscode/services.ts | Registered the new ActionWordService in the dependency injection container |
src/extension/prompts/node/agent/openai/defaultOpenAIPrompt.tsx
Outdated
Show resolved
Hide resolved
src/extension/prompts/node/agent/openai/defaultOpenAIPrompt.tsx
Outdated
Show resolved
Hide resolved
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
The ActionWordService was created but never used anywhere in the codebase. The prompt template changes are effective without this additional service layer. - Remove actionWordAnalysis.ts (unused analysis logic) - Remove actionWordService.ts (unused service wrapper) - Remove service registration from services.ts This addresses reviewer feedback about dead code. The investigation-first behavior is already implemented directly in the prompt templates.
Added action word dictionary to hopefully force copilot into a certain mode of thinking quicker and more efficiently then debating what the user is actually requesting.Additionally:Add action word analysis system for request categorizationIdeally this will assist in more complete requests that shouldn't require nearly as many follow up prompts. In testing I was able to generate a a simple 3d car driving game in ThreeJS in 6-10 requests.
EDIT: Didn't realize intention was already getting extracted, essentially doing what I was trying to do but better, it would be more productive to edit those files to extract better intentions rather than implementing this whole new "Action Service"