Flaky test detector for GitHub Actions

The idea

Flaky tests get handled the same way everywhere: someone hits re-run, it goes green, everyone moves on, and the team quietly stops trusting red builds. On GitHub Actions this also burns paid minutes, so the failure mode has a literal invoice attached. The data needed to fix it already exists in workflow history and JUnit artifacts; nobody on a small team ever assembles it.

BuildPulse proves companies pay for flake detection, but it is aimed and priced at larger orgs. The wedge is zero-config: a GitHub App that ingests the JUnit artifacts teams already produce, with no SDK, no test-runner wrapper, and one honest trick, pass-after-retry on an identical SHA is flakiness by definition. Detection alone is a report; opening the quarantine PR is the product.

Who pays

The buyer is the tech lead of a 3 to 20 engineer team on GitHub Actions whose CI bill and merge-queue latency are both visibly worsened by reruns. At 30 to 100 dollars a month it undercuts the incumbent and sells self-serve through the GitHub Marketplace, which is also the distribution channel, alongside free analysis for public repos as a growth loop (badge in the README).

MVP scope

• GitHub App install with webhook-driven JUnit artifact ingestion, zero config
• Pass-after-retry detection on identical SHAs with a cost-ranked flaky-test table
• Per-test history view and repo-level wasted-minutes estimate
• One-click quarantine PR with skip annotation plus auto-filed tracking issue
• Stripe billing, free for public repos
• Skip for v1: non-GitHub CI providers, test runners without JUnit output, auto-root-cause suggestions, retry-budget enforcement, Slack digests

Build FlakeRadar, a GitHub App that detects and quarantines flaky tests for teams on GitHub Actions. Stack: Next.js dashboard, Postgres (on Neon or RDS), a GitHub App for installation and webhooks, Stripe for billing.

Core flow: team installs the GitHub App on a repo. No SDK, no config: the app listens to workflow_run webhooks, downloads JUnit XML artifacts from completed runs (document a three-line upload-artifact snippet for repos not yet uploading them), and ingests every test result into Postgres keyed by (repo, test full name, commit SHA, run attempt). Flake detection: a test that both failed and passed on the identical SHA (retry of the same run, or re-run of the same commit) is flagged flaky. Score each flaky test by failures-per-week times median rerun duration to rank by minutes burned.

Dashboard pages: repo overview with a flakiness trend chart and estimated CI minutes wasted per week, ranked flaky-test table with per-test history (sparkline of pass/fail over time, first seen, last seen), and settings. Per test, a Quarantine button opens a PR that tags the test with the framework-appropriate skip annotation (@pytest.mark.skip, test.skip, @Disabled) including a comment linking to an auto-filed GitHub issue so the skip is never silent.

Implementation notes: dedupe artifact ingestion by artifact ID; JUnit parsing must tolerate the nonstandard XML that pytest, Jest and Gradle each emit.

Pricing: free for public repos, 30 GBP/month for 5 private repos, 100 GBP/month unlimited.