FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

feat: Add LLM token usage tracking to Application Insights by Prachig-Microsoft · Pull Request #586 · microsoft/content-processing-solution-accelerator · GitHub

/ content-processing-solution-accelerator Public

feat: Add LLM token usage tracking to Application Insights - #586

Open
Prachig-Microsoft wants to merge 27 commits into
devfrom
feature/token-usage-tracking
ConversationCommits (27)ChecksFiles changed
Open

feat: Add LLM token usage tracking to Application Insights#586
Prachig-Microsoft wants to merge 27 commits into
devfrom
feature/token-usage-tracking

Conversation

Prachig-Microsoft commented May 12, 2026

Copy link
Copy Markdown
Contributor

Implement comprehensive token usage tracking across all LLM call sites in ContentProcessor and ContentProcessorWorkflow, following the MACAE psl-token-usage branch pattern.

Changes:

  • Add token_usage_utils.py with extract/emit helpers for both projects
  • Instrument MapHandler (ContentProcessor) with per-call token events
  • Instrument RAI, Summarize, GapAnalysis executors (Workflow)
  • Add summary and per-model token events in save_handler
  • Add 18 KQL queries for Azure Workbook visualization
  • Add unit tests (18 tests) for token_usage_utils

Events emitted to Application Insights:

  • LLM_Agent_Token_Usage: per agent/step with model, process_id
  • LLM_Model_Token_Usage: per model deployment aggregation
  • LLM_Token_Usage_Summary: per document totals

Does this introduce a breaking change?

  • Yes
  • No

Golden Path Validation

  • [ x] I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

  • [ x] I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

  • ...

Other Information

github-actions Bot commented May 12, 2026 •
edited
Loading

Copy link
Copy Markdown

Coverage Report •
FileStmtsMissCoverMissing
TOTAL121716186% 
report-only-changed-files is enabled. No files were changed during this commit :)

Tests Skipped Failures Errors Time
244 0 💤 0 ❌ 0 🔥 4.955s ⏱️

Prachig-Microsoft marked this pull request as ready for review May 25, 2026 10:13
Copilot AI review requested due to automatic review settings May 25, 2026 10:13
Prachig-Microsoft force-pushed the feature/token-usage-tracking branch from 321eca7 to f4ee50e Compare June 2, 2026 10:28
fix: exclude telemetry module from Workflow coverage (tested in Conte…
efebd81 
…ntProcessor)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 2, 2026 10:33
Copilot AI reviewed Jun 2, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 15 out of 17 changed files in this pull request and generated 3 comments.

Comment thread src/ContentProcessorWorkflow/src/steps/summarize/executor/summarize_executor.py Outdated
Comment thread src/ContentProcessorWorkflow/src/steps/rai/executor/rai_executor.py Outdated
Comment thread src/ContentProcessorWorkflow/src/steps/gap_analysis/executor/gap_executor.py Outdated
chore: remove token usage workbook files from branch
ab495c8 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 11, 2026 09:29
fix: switch ContentProcessor base image to python:3.12-slim-bookworm
cf487d9 
The devcontainers/python image on MCR is returning 403 Forbidden.
Use the official Python slim image instead, which is lighter and
appropriate for production containers.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI reviewed Jun 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 5 comments.

Comment thread src/ContentProcessorWorkflow/src/steps/summarize/executor/summarize_executor.py Outdated
Comment thread src/ContentProcessorWorkflow/src/steps/rai/executor/rai_executor.py Outdated
Comment thread src/ContentProcessorWorkflow/src/steps/gap_analysis/executor/gap_executor.py Outdated
Comment thread infra/main.json
Comment thread src/ContentProcessor/src/libs/pipeline/handlers/map_handler.py
fix: address review comments - event_id collision and nondeterministi…
bdb51ba 
…c file_types

- Pass agent_name and model_deployment_name to emit_summary for unique event_id
- Sort file_types set before joining for deterministic telemetry dimensions

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings June 11, 2026 09:43
Copilot AI reviewed Jun 11, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Pull request overview

Copilot reviewed 16 out of 18 changed files in this pull request and generated 4 comments.

Comment thread infra/main.json
Comment thread infra/main.json
Comment thread src/ContentProcessor/src/libs/llm_token_telemetry.py
Comment thread src/ContentProcessorWorkflow/src/libs/llm_token_telemetry.py

This branch had an error being deployed

1 failed deployment
production — bdb51ba7 Deployed Jun 12, 2026 by Prachig-Microsoft via Run / deploy / Deploy on Linux / deploy-linux #386
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
Avijit-Microsoft Awaiting requested review from Avijit-Microsoft Avijit-Microsoft is a code owner
Roopan-Microsoft Awaiting requested review from Roopan-Microsoft Roopan-Microsoft is a code owner
Prajwal-Microsoft Awaiting requested review from Prajwal-Microsoft Prajwal-Microsoft is a code owner
Vinay Sharma (Vinay-Microsoft) Awaiting requested review from Vinay-Microsoft Vinay-Microsoft is a code owner
Anish Arora (aniaroramsft) Awaiting requested review from aniaroramsft aniaroramsft is a code owner
Todd Herman (toherman-msft) Awaiting requested review from toherman-msft toherman-msft is a code owner
nchandhi Awaiting requested review from nchandhi nchandhi is a code owner
dgp10801 Awaiting requested review from dgp10801 dgp10801 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

Back | FazBrowse Home | New Git URL