refactor(testing): fix flaky terminal test - #3230
Conversation
Sorry, something went wrong.
Codecov Report
@@ Coverage Diff @@
## main #3230 +/- ##
=======================================
Coverage 46.90% 46.90%
=======================================
Files 23 23
Lines 1196 1196
Branches 237 237
=======================================
Hits 561 561
Misses 451 451
Partials 184 184 Continue to review full report at Codecov.
|
Sorry, something went wrong.
| // Read more: https://thisthat.dev/dom-content-loaded-vs-load/ | ||
| await this.page.waitForLoadState("load") | ||
| // Give it an extra second just in case it's feeling extra slow | ||
| await this.page.waitForTimeout(1000) |
There was a problem hiding this comment.
ahah these things make me sad, totally fair though
Sorry, something went wrong.
There was a problem hiding this comment.
lol i know. sadly, this timeout was the only thing that made it consistently pass.
Sorry, something went wrong.
| // See Playwright docs: https://playwright.dev/docs/pom/ | ||
| export class CodeServer { | ||
| page: Page | ||
| editorSelector = "div.monaco-workbench" |
There was a problem hiding this comment.
🛠️ I realized how many places I was using this in the model so refactored it into a property of the class
Sorry, something went wrong.
| // Read more: https://thisthat.dev/dom-content-loaded-vs-load/ | ||
| await this.page.waitForLoadState("load") | ||
| // Give it an extra second just in case it's feeling extra slow | ||
| await this.page.waitForTimeout(1000) |
There was a problem hiding this comment.
lol i know. sadly, this timeout was the only thing that made it consistently pass.
Sorry, something went wrong.
| // Click text=Command Palette | ||
| await this.page.hover("text=Command Palette") | ||
| await this.page.click("text=Command Palette") | ||
|
|
||
| // Type Terminal: Focus Terminal | ||
| await this.page.keyboard.type("Terminal: Focus Terminal") | ||
|
|
||
| // Click Terminal: Focus Terminal | ||
| await this.page.hover("text=Terminal: Focus Terminal") | ||
| await this.page.click("text=Terminal: Focus Terminal") |
There was a problem hiding this comment.
This approach produces consistent results — no matter if the terminal is already focused, visible/not visible.
Sorry, something went wrong.
| // which is why we wait for it twice | ||
| await this.page.waitForSelector("div.terminal.xterm.focus") | ||
| // Wait for terminal textarea to show up | ||
| await this.page.waitForSelector("textarea.xterm-helper-textarea") |
There was a problem hiding this comment.
This also seems to be better than looking for the div.terminal.xterm.focus selector, which may or may not appear.
H/T to @kylecarbs for the suggestion 🎉
xterm source
Sorry, something went wrong.
| // It may take a second to process | ||
| await page.waitForTimeout(1000) |
There was a problem hiding this comment.
There is no great way of knowing that the terminal received both the input and the enter and processed it (because it's a <canvas> element so we can't assert the text on screen) so having this timeout accounts for any slowness that might occur.
Sorry, something went wrong.
This PR fixes a flaky implementation of the focusTerminal method on the CodeServer model.
This was caught after merging into #3169 main and the e2e tests failed here 😅
Changes:
Before
The should show the Integrated Terminal only passed intermittently in Firefox:
✓ CodeServer should show the Integrated Terminal (20s) x 3) CodeServer should show the Integrated Terminal (47s) x 1) CodeServer should show the Integrated Terminal (47s) x 2) CodeServer should show the Integrated Terminal (47s) x 4) CodeServer should show the Integrated Terminal (48s) 1 passed (52s) 4 failed codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============= 1) codeServer.test.ts:41:8 › [firefox] CodeServer should show the Integrated Terminal ============What's happening: it's waiting for the selector div.terminal.xterm.focus to be visible, which doesn't happen for some reason and the test times-out.
After