Run Playwright tests on Firefox & "Safari" nightly (#28757)

* Run Playwright tests on Firefox Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Update playwright.config.ts * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Update end-to-end-tests.yaml * Iterate Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Finalise Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * Documentation Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> * typo Signed-off-by: Michael Telatynski <7t3chguy@gmail.com> --------- Signed-off-by: Michael Telatynski <7t3chguy@gmail.com>
2024-12-19 23:42:09 +00:00
parent be181d2c79
commit 5448de5dd6
23 changed files with 266 additions and 161 deletions
--- a/docs/playwright.md
+++ b/docs/playwright.md
@@ -53,15 +53,11 @@ yarn run test:playwright:open --headed --debug

 See more command line options at <https://playwright.dev/docs/test-cli>.

-### Running with Rust cryptography
+## Projects

-`matrix-js-sdk` is currently in the
-[process](https://github.com/vector-im/element-web/issues/21972) of being
-updated to replace its end-to-end encryption implementation to use the [Matrix
-Rust SDK](https://github.com/matrix-org/matrix-rust-sdk). This is not currently
-enabled by default, but it is possible to have Playwright configure Element to use
-the Rust crypto implementation by passing `--project="Rust Crypto"` or using
-the top left options in open mode.
+By default, Playwright will run all "Projects", this means tests will run against Chrome, Firefox and "Safari" (Webkit).
+We only run tests against Chrome in pull request CI, but all projects in the merge queue.
+Some tests are excluded from running on certain browsers due to incompatibilities in the test harness.

 ## How the Tests Work

@@ -224,3 +220,14 @@ We use test tags to categorise tests for running subsets more efficiently.

 - `@mergequeue`: Tests that are slow or flaky and cover areas of the app we update seldom, should not be run on every PR commit but will be run in the Merge Queue.
 - `@screenshot`: Tests that use `toMatchScreenshot` to speed up a run of `test:playwright:screenshots`. A test with this tag must not also have the `@mergequeue` tag as this would cause false positives in the stale screenshot detection.
+- `@no-$project`: Tests which are unsupported in $Project. These tests will be skipped when running in $Project.
+
+Anything testing Matrix media will need to have `@no-firefox` and `@no-webkit` as those rely on the service worker which
+has to be disabled in Playwright on Firefox & Webkit to retain routing functionality.
+Anything testing VoIP/microphone will need to have `@no-webkit` as fake microphone functionality is not available
+there at this time.
+
+## Colima
+
+If you are running under Colima, you may need to set the environment variable `TMPDIR` to `/tmp/colima` or a path
+within `$HOME` to allow bind mounting temporary directories into the Docker containers.