| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Add GaqSummaryRepository and GaqSummaryInvalidationRepository and export them from the repositories index. Update migration to replace the previous invalidated_at field with created_at and add updated_at to mirror default Sequelize tables.
Rename and change mcReproducible to be the coverage float not the boolean.
Forgot to remove.
Add GAQ summary invalidation whenever underlying data affecting GAQ changes. These changes ensure GAQ summary caches/records are marked for recomputation whenever detectors, QC flags, or run QC times that influence GAQ summaries are modified.
Introduce background processing for GAQ summary invalidations and wire it into the app scheduler. Changes include:
- Add gaq config to services config.
- Move GaqService to lib/server/services/gaq and update imports across controllers/use-cases/tests.
- Extend GaqService to pop invalidations and recalculate summaries within a transaction.
- Add GaqWorker that guards concurrent runs and calls GaqService to process a batch of invalid summaries.
- Schedule the GaqWorker in application startup when GAQ recalculation is enabled.
- Add a soft-delete filter (where: { deleted: false }) when querying QC flags to map ids to run numbers as otherwise summary invalidations occur even on already deleted flags.
These changes enable periodic recalculation of GAQ summaries when invalidations are queued in the table.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Validate GAQ summary invalidation behaviour on QC flag create/verify/delete, deleteAllForDataPass, explicit/default GAQ detector changes, and run QC time updates.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Converted gaqWorker to a singleton. This allows adding of pause() and resume() methods to prevent the worker from processing invalidated summaries during test execution. Reduced the default GAQ recalculation period from 1 minute to 10 seconds to improve test suite performance.
Adds tests that verify the worker removes an invalidation and adds a summary, upserts for an already present summary, batch processes correctly, and doesn't run concurrent recalculations. Update resetDatabaseContent to pause/resume the GAQ worker to avoid worker failures when the invalidation table is dropped.
GaqService summary recalculation and invalidation processing functions tested directly circumventing worker. Tests verify correct computed fields, summary upsert, behaviour when no coverage exists, and batched invalidation processing.
Add a calculation_failed boolean column to gaq_summaries. This allows us to know whether a summary has been attempted to be calculated but unsuccessful due to limited data etc.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Remove not-null constraints from several GAQ summary columns in the migration to allow NULL when values are unavailable.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Stop returning early when summary is missing; always upsert a GaqSummary record and include a calculationFailed boolean. This ensures failed calculations are persisted for monitoring instead of being silently skipped.
Update test to expect that when there is no coverage data for a run, a GAQ summary is stored with calculationFailed set to true.
Store invalidation timestamp directly on gaq_summaries instead of a separate gaq_summary_invalidations table.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
More details logged to help debugging the state of the GAQ worker.
…consistently exceeds batch size Also move any GAQ worker related logging to within the GAQ worker itself (separation of concerns).
Sequelize model omitted the primaryKey that migration file defines.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
Extract invalidation into helpers. Guard GAQ updateRun reload so don't run reload unnecessarily. Add to notComputable field in model allowNull default params. Add more test coverage for invalidation on run patch.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
Protects invalidatedAt from being set to null if the datapass and runNumber becomes invalid for whatever reason in the middle of computeSummary. ComputeSummary would overwrite the invalidatedAt column with null at the end. Now it performs a check to prevent this.
Add a helper that logs a warning when the env var is set to an invalid value instead of silently falling back to the default. If batchSize is not enough log the first warning after 5 consecutive overflow ticks, then every 30 ticks while it persists and recovery line when the streak ends so operators can grep for "backlog recovered". Log at debug level when the worker is paused or resumed.
Codecov Report❌ Patch coverage is 92.90323% with 11 lines in your changes missing coverage. Please review. @@ Coverage Diff @@
## main #2175 +/- ##
==========================================
+ Coverage 45.82% 46.49% +0.66%
==========================================
Files 1039 1043 +4
Lines 17189 17318 +129
Branches 3138 3168 +30
==========================================
+ Hits 7877 8052 +175
+ Misses 9312 9266 -46 ☔ View full report in Codecov by Harness.
|
Sorry, something went wrong.
resetDatabaseContent unconditionally called gaqWorker.resume(), which silently undid any pause set up by a calling test suite. Capture the worker's paused state before pausing and only resume it if it wasn't already paused on entry. Adds an isPaused getter on GaqWorker to expose the state without reaching into the private field.
…e/O2B-1563/Create-GAQ-summary-invalidation-mechanism
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
…-invalidation-mechanism" This reverts commit 23f6e04.
…ism' into feature/O2B-1564/Create-background-GAQ-summary-worker
| Back | FazBrowse Home | New Git URL |
I have a JIRA ticket
Notable changes for users:
Notable changes for developers:
Changes made to the database:
A configurable background worker that runs a function that batch recalculates GAQ summaries that have an invalidatedAt column that is not null.
Dynamically adjusts batch size between a min and a max based on backlog.
Worker can be paused (useful for testing but maybe in prod for whatever reason).