← Back to press resources

50-Ticket AI Issue Triage Benchmark 2026: Labels, Priority and Suggestions in About 10 Seconds

Published 2026-09-14 · TriageFlow Research · last updated 2026-10-20 · Independent reproduction welcome (each triage decision carries a stored rationale).

TL;DR

On 2026-09-14 we ran a public simulation of 50 GitHub issues through TriageFlow. Each issue received a label, a priority and a triage suggestion with a verifiable reason. Measured end-to-end latency was about 10 seconds per issue (≈3 s classification + ≈6 s delivery, remainder network overhead). The GitHub App had zero access to source code (permissions: Issues and repository metadata only — no contents key in the public App manifest). Per-issue raw results (id, type, labels, priority, suggestion, latency, rationale flag) are published with the benchmark repository described under Dataset; the exact reproduction steps below regenerate the same measurements.

Methodology

Results

MetricResult
Issues processed50
Issues labelled automatically50 / 50
Issues with priority assigned50 / 50
Issues with verifiable suggestion50 / 50
Median per-issue latency≈10 s (≈3 s classify + ≈6 s delivery)
Duplicate groups collapsedReported in dataset (5 same-bug reports → 1 group)
Audit rationale stored50 / 50 (in triage history, no thread spam)
Source-code access used0 (permissions: Issues + metadata only)

Illustrative cost frame: manual triage ≈3 minutes per ticket; at $85/hour engineering cost, a 50-ticket backlog costs ≈$212.50 of engineering time. TriageFlow Starter ($49/mo) covers 1,000 issues/mo. (Illustrative example — actual savings vary.)

Dataset

The complete per-issue dataset (one row per issue: id, type, labels, priority, suggestion, latency seconds, rationale flag) is published together with the 50-ticket benchmark repository, whose Issues tab lists the labelled sample issues. Reproduce locally with the method below — no signup required.

Security note

Zero Source-Code Access by Design: the TriageFlow GitHub App only requests permissions for Issues and repository metadata — contents: none — and never trains AI models on private tickets. Verified in the public App manifest. Operated by Universal Global Services Ltd (London, Companies House No 17423506).

Reproducing

  1. Install the TriageFlow GitHub App on any repository (no credit card for the Free plan).
  2. Run the public 50-ticket simulation at /demo.html (no signup).
  3. Export the resulting triage history and compare labels/latency with the published dataset.