I will be honest, you lost me at "takes days or weeks to resolve end to end tests". E2E tests are just one more tool in your arsenal. Certainly only essential aspects of an app should be covered. But if it's taking you that long to resolve any tests, some test runner plumbing is missing.
1. Are you running all tests multiple times daily on main branch to catch flakey tests? When a test fails do you have a trace id to track down what service calls were made during the calls as part of the test?
2. Do you run all tests for every PR so you know when regressions are introduced?
3. For tests that rely on A/B tests, do you gate them so they don't run outside of the A/B test?
4. For cases where tests are broken upstream ( a PR for a transitive dependency get merged anyway) - do tests continue to run, but not fail the build while the ticket to resolve them is still open?
Having domain level tests is great, but it still won't tell you the system can correctly perform its essential functions. If e2e tests are constantly failing, how can manual testing work either?
There are weekly releases, and these releases get delayed when E2E tests fail.
Yes, E2E tests are flaky.
The simple solution is to fix those tests. But tests will only increase as more logic is built. What this means is fixing flakiness becomes a moving target.
The idea here was to decouple the dependencies so much so that the E2E tests are no longer required. This is an extreme idea and the idea of the post was to find logical flaws by proposing a new framework to address the problem.
To your 1 & 2 points: Yes, we run tests for each build i.e. for every commit including the PRs. Those tests are unit and integration tests. But, E2E tests are not run as they are heavy in terms of resources and dependencies to get them working.
PS: I searched on the "takes days or weeks to resolve end-to-end tests" as I thought I had made a mistake in my post. But it only returned 1 result which was your comment.
Perhaps, my writing was not clear enough, which may have created the confusion.
I will be honest, you lost me at "takes days or weeks to resolve end to end tests". E2E tests are just one more tool in your arsenal. Certainly only essential aspects of an app should be covered. But if it's taking you that long to resolve any tests, some test runner plumbing is missing.
1. Are you running all tests multiple times daily on main branch to catch flakey tests? When a test fails do you have a trace id to track down what service calls were made during the calls as part of the test?
2. Do you run all tests for every PR so you know when regressions are introduced?
3. For tests that rely on A/B tests, do you gate them so they don't run outside of the A/B test?
4. For cases where tests are broken upstream ( a PR for a transitive dependency get merged anyway) - do tests continue to run, but not fail the build while the ticket to resolve them is still open?
Having domain level tests is great, but it still won't tell you the system can correctly perform its essential functions. If e2e tests are constantly failing, how can manual testing work either?
Thanks for reading the post.
There are weekly releases, and these releases get delayed when E2E tests fail.
Yes, E2E tests are flaky.
The simple solution is to fix those tests. But tests will only increase as more logic is built. What this means is fixing flakiness becomes a moving target.
The idea here was to decouple the dependencies so much so that the E2E tests are no longer required. This is an extreme idea and the idea of the post was to find logical flaws by proposing a new framework to address the problem.
To your 1 & 2 points: Yes, we run tests for each build i.e. for every commit including the PRs. Those tests are unit and integration tests. But, E2E tests are not run as they are heavy in terms of resources and dependencies to get them working.
PS: I searched on the "takes days or weeks to resolve end-to-end tests" as I thought I had made a mistake in my post. But it only returned 1 result which was your comment.
Perhaps, my writing was not clear enough, which may have created the confusion.