Issues Metrics:
New Issues Logged
140 Issues were identified during this release cycle. Of the 140 issues identified during this release cycle we have closed 41 of them.
For a 29% reported issue closure rate.
Only 17% of these issues were identified as regression items.
A total of 84 issues were closed during this release cycle, 59 of which are in this update. The remaining 25 items will be included in the next update.
82% of new defects were identified by our QA team.
Only 18% of issue logged during this release cycle were externally identified issues. Only 9% of the issues were associated with the most recent release.
136 new issues were reported during this release cycle, 112 were identified by our testing team and 24 were reported by customers.
Due to the wide range of versions that our customers are using, the reported issues may have been versions 1.64.x, 2.0.x, or 2.1.x which means that the 18% reported by customers is not necessarily indicative of issues within the most recent release.
Automation Metrics:
Due to improvements within the Manitou Web Client we've seen a 5% average automation response time decrease (1,226ms to 1,165ms)
15% response time decrease in the Reports automation tests response time.
6% response time decrease in the Customer Wizard automation tests response time.
A total of 81,081 automated tests were executed during this release cycle.
Dashboard: 7,644 individual tests
New Customer: 1,183 individual tests
Reports: 72,254 individual tests
Dashboard = 91 iterations of 84 assertions for a total of 7,644 individual automated test executions.
New Customer = 91 iterations of 14 assertions for a total of 1,183 individual automated test executions.
Reports = 91 iterations of 794 assertions for a total of 72,254 individual automated test executions.
Average overall automation response time is 61ms faster with 2.1.13:
| Version 2.1.12 | Version 2.1.13 | |
| Dashboard Response Time | 194ms | 218ms (+12%) |
New Customer Response Time | 3,300ms | 3,100ms (-6%) |
| Reports Response Time | 112ms | 95ms (-15%) |
| Overall Average Response Time | 1,226ms | 1,165ms (-5%) |
Builds Tested Metrics
Between 2.1.12 and 2.1.13:
142 builds tested
191 code modifications verified.
The Quality Assurance team tested an average of 11 builds per day.
Manual Test Metrics:
QA has conducted at 655 manual tests (est. test steps 1965) against 2.1.13. This is after completing 78% of 841 test items in the coverage spreadsheet.
