Completely ignore too-many-failures builds in TestFailures in most circumstances

Because a semi-arbitrary subset of tests are run in a too-many-failures build, we can't
really use them to perform regression analysis. The only time we want to pay attention to
too-many-failures builds is when we're trying to explain when the current bout of
too-many-failures started.

Fixes <http://webkit.org/b/64106> TestFailures page sometimes claims a test started failing
in a build that didn't even run it (because it exited early due to too many failues)

Patch by Adam Roben <aroben@apple.com> on 2011-07-07
Reviewed by David Kilzer.

* BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:
(LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Removed old, broken
too-many-failures handling that would cause us to blame builds that didn't even run a given
test for breaking it. Instead, skip over all too-many-failures builds unless the most recent
build was itself a too-many-failures build.

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@90571 268f45cc-cd09-0410-ab3c-d52691b4dbfc
parent b36a3fe4
......@@ -118,6 +118,20 @@ LayoutTestHistoryAnalyzer.prototype = {
var self = this;
self._loader.start(nextBuildName, function(tests, tooManyFailures) {
if (tooManyFailures) {
var firstBuildName = Object.keys(self._history)[0];
// If the first (i.e., current or most recent) build exited early due to too many
// failures, we want to process other too-many-failures builds normally to try to
// figure out when the too-many-failures started occurring. If the first/current
// build did not exit due to too many failures, then too-many-failures builds will
// only confuse our analysis (since they run a semi-arbitrary subset of tests), so
// we should just skip them entirely.
if (firstBuildName && !self._history[firstBuildName].tooManyFailures) {
callback(true);
return;
}
}
++self._testRunsSinceLastInterestingChange;
var historyItem = {
......@@ -150,16 +164,6 @@ LayoutTestHistoryAnalyzer.prototype = {
historyItem.tests[testName] = tests[testName];
}
if (tooManyFailures && previousHistoryItem) {
// Not all tests were run due to too many failures. Just assume that all the tests
// that failed in the last build would still have failed in this build had they been
// run.
for (var testName in previousHistoryItem.tests) {
historyItem.tests[testName] = previousHistoryItem.tests[testName];
delete previousHistoryItem.tests[testName];
}
}
var previousUnexplainedFailuresCount = previousBuildName ? Object.keys(self._history[previousBuildName].tests).length : 0;
var unexplainedFailuresCount = Object.keys(self._history[nextBuildName].tests).length;
......
2011-07-07 Adam Roben <aroben@apple.com>
Completely ignore too-many-failures builds in TestFailures in most circumstances
Because a semi-arbitrary subset of tests are run in a too-many-failures build, we can't
really use them to perform regression analysis. The only time we want to pay attention to
too-many-failures builds is when we're trying to explain when the current bout of
too-many-failures started.
Fixes <http://webkit.org/b/64106> TestFailures page sometimes claims a test started failing
in a build that didn't even run it (because it exited early due to too many failues)
Reviewed by David Kilzer.
* BuildSlaveSupport/build.webkit.org-config/public_html/TestFailures/LayoutTestHistoryAnalyzer.js:
(LayoutTestHistoryAnalyzer.prototype._incorporateBuildHistory): Removed old, broken
too-many-failures handling that would cause us to blame builds that didn't even run a given
test for breaking it. Instead, skip over all too-many-failures builds unless the most recent
build was itself a too-many-failures build.
2011-07-07 Adam Roben <aroben@apple.com>
Teach webkitpy's Checkout class to use commit-log-editor to create commit messages
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment