Release Scorecard Definitions¶
Release Output¶
| Metric | How to Measure |
|---|---|
| Ratio of story-to-bug tickets | Total of feature tickets over total of bug tickets. Include current quality release and previous feature release. |
| Mean time from P1 bug report to delivery | With a new or existing Jira filter, with:
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”. |
| Mean time from P2 bug report to delivery | With a new or existing Jira filter, with:
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”. |
| Mean time from P3 bug report to delivery | With a new or existing Jira filter, with:
Copy a list of Jira tickets with the above information and paste them into a spreadsheet. Calculate the average by using a formula “=Release Day-Created”. |
| Number of P1 bugs reported in production | Bug severity guidelines are in progress, after which a Jira query will be built |
| Number of P2 bugs reported in production | Bug severity guidelines are in progress, after which a Jira query will be built |
| Number of P3 bugs reported in production | Bug severity guidelines are in progress, after which a Jira query will be built |
Release Management¶
| Metric | How to Measure |
|---|---|
| Number of misses in each release cycle | Same as number of bugs reported in production, + count of issues reported in documentation or marketing by customers, CSMs, Support or Product. E.g. dot releases from customer reports, undocumented breaking changes. |
| Number of setbacks to users/staff resulting in a negative reaction | Manual count of negative reactions reported by CSMs, Support or Product. E.g. features that got pushed. |
| Number of measurable actions from each release retrospective driven from root causes & completed and reported to R&D | Count of completed and reported items from Release Retrospective document. |
| Number of deadlines missed in release checklist | Manual count of deadlines not met from items posted to Release Checklist. |
Release Heartbeat¶
| Metric | How to Measure |
|---|---|
| Time updated to RC1 (PST) | Check Release Discussion channel history for date/time RC1 was cut. |
| How many RCs cut | Check Release Discussion channel history for how many RCs were cut for that release. |
| Number of days between when final RC is cut and the release date | Check Release Discussion channel for post with official release build. Oxygen = 16th - Day Final RC is cut |
| Community + customer bugs reported during release timeframe (17th to 16th) | With a new or existing Jira filter, with:
|
| Number of customer bugs fixed during release | With a new or existing Jira filter, with:
|
| Total valid bugs in fix version | After closing current release: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, “Cannot Reproduce”, “Won’t Fix”) AND fixVersion = latestReleasedVersion() |
| Total valid bugs in fix version found by test automation | Check “Se”, “Selenium-found, “Rainforest-found” Jira labels. |
| Total valid bugs found after RC1 is cut | After closing current release, adjust dates as per above, and use this Jira query:
project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, “Cannot Reproduce”, “Won’t fix”) AND created > “START” AND created < “END” |
| Valid bugs found after RC1 fixed in release | After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, “Cannot Reproduce”, “Won’t Fix”) AND created > “START” AND created < “END” AND fixVersion = latestReleasedVersion() |
| Valid bugs found after RC1 pushed to next release | After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND resolution not in (Duplicate, “Cannot Reproduce”, “Won’t Fix”) AND created > “START” AND created < “END” AND fixVersion = earliestUnreleasedVersion() |
| Valid bugs found after RC1 fix version = other (eg unscheduled, not set) | After closing current release, adjust dates as per above, and use this Jira query: project = Mattermost AND issuetype = Bug AND created > “START” AND created < “END” AND resolution not in (Duplicate, “Cannot Reproduce”, “Won’t Fix”) AND (fixVersion not in (latestReleasedVersion(), earliestUnreleasedVersion()) OR fixVersion is EMPTY) |
| (Non-security) Bugs requiring patch release | After any patch release goes out (after the normal release date): Check Changelog for total number of non-security patch releases. |
| Total features/improvements in fix version | With a new or existing Jira filter, with:
|
| Critical security issues found during release timeframe | With a new or existing Jira filter, check for Security Vulnerability tickets:
|
| Moderate security issues found during release timeframe | With a new or existing Jira filter, check for Security Vulnerability tickets:
|
| Minor security issues found during release timeframe | With a new or existing Jira filter, check for Security Vulnerability tickets:
|
