Commit af00a02f authored by dpranke@chromium.org's avatar dpranke@chromium.org
Browse files

2010-08-24 Dirk Pranke <dpranke@chromium.org>

        Reviewed by Eric Seidel.

        new-run-webkit-tests: revise unit testing strategy for test failures

        This change revamps the way we store test data for new-run-webkit-tests.
        Previously we had a few copies of test files from the main test tree.
        Now we have a bunch of completely fake test data and use the test
        data to generate different kinds of test failures, getting
        much better coverage.

        https://bugs.webkit.org/show_bug.cgi?id=44556

        * Scripts/webkitpy/layout_tests/data/failures/expected/crash.html: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image-expected.checksum: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image.html: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image_checksum-expected.checksum: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image_checksum-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image_checksum-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/image_checksum.html: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_image.html: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/missing_text.html: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/text-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/failures/expected/text.html: Added.
        * Scripts/webkitpy/layout_tests/data/failures/unexpected/text-image-checksum-expected.checksum: Added.
        * Scripts/webkitpy/layout_tests/data/failures/unexpected/text-image-checksum-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/failures/unexpected/text-image-checksum-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/failures/unexpected/text-image-checksum.html: Added.
        * Scripts/webkitpy/layout_tests/data/image/canvas-bg.html: Removed.
        * Scripts/webkitpy/layout_tests/data/image/canvas-zoom-expected.checksum: Removed.
        * Scripts/webkitpy/layout_tests/data/image/canvas-zoom-expected.png: Removed.
        * Scripts/webkitpy/layout_tests/data/image/canvas-zoom-expected.txt: Removed.
        * Scripts/webkitpy/layout_tests/data/image/canvas-zoom.html: Removed.
        * Scripts/webkitpy/layout_tests/data/misc/crash-expected.txt: Removed.
        * Scripts/webkitpy/layout_tests/data/misc/crash.html: Removed.
        * Scripts/webkitpy/layout_tests/data/misc/missing-expectation.html: Removed.
        * Scripts/webkitpy/layout_tests/data/misc/passing-expected.txt: Removed.
        * Scripts/webkitpy/layout_tests/data/misc/passing.html: Removed.
        * Scripts/webkitpy/layout_tests/data/passes/image-expected.checksum: Added.
        * Scripts/webkitpy/layout_tests/data/passes/image-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/passes/image-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/passes/image.html: Added.
        * Scripts/webkitpy/layout_tests/data/passes/platform_image-expected.checksum: Added.
        * Scripts/webkitpy/layout_tests/data/passes/platform_image-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/passes/platform_image-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/passes/platform_image.html: Added.
        * Scripts/webkitpy/layout_tests/data/passes/text-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/passes/text.html: Added.
        * Scripts/webkitpy/layout_tests/data/platform/test/image/canvas-bg-expected.checksum: Removed.
        * Scripts/webkitpy/layout_tests/data/platform/test/image/canvas-bg-expected.png: Removed.
        * Scripts/webkitpy/layout_tests/data/platform/test/image/canvas-bg-expected.txt: Removed.
        * Scripts/webkitpy/layout_tests/data/platform/test/passes/platform_image-expected.checksum: Added.
        * Scripts/webkitpy/layout_tests/data/platform/test/passes/platform_image-expected.png: Added.
        * Scripts/webkitpy/layout_tests/data/platform/test/passes/platform_image-expected.txt: Added.
        * Scripts/webkitpy/layout_tests/data/platform/test/test_expectations.txt:
        * Scripts/webkitpy/layout_tests/data/text/article-element-expected.txt: Removed.
        * Scripts/webkitpy/layout_tests/data/text/article-element.html: Removed.
        * Scripts/webkitpy/layout_tests/layout_package/test_expectations_unittest.py:
        * Scripts/webkitpy/layout_tests/port/test.py:
        * Scripts/webkitpy/layout_tests/run_webkit_tests_unittest.py:
        * Scripts/webkitpy/style/checkers/test_expectations_unittest.py:

git-svn-id: http://svn.webkit.org/repository/webkit/trunk@65951 268f45cc-cd09-0410-ab3c-d52691b4dbfc
parent 367f437e
layer at (0,0) size 785x620
RenderView at (0,0) size 785x600
layer at (0,0) size 785x620
RenderBlock {HTML} at (0,0) size 785x620
RenderBody {BODY} at (8,8) size 769x604
RenderBlock {DIV} at (0,0) size 604x604 [border: (2px solid #000000)]
WONTFIX : misc/missing-expectation.html = MISSING PASS
WONTFIX : failures/expected/text.html = TEXT
WONTFIX SKIP : failures/expected/crash.html = CRASH
WONTFIX : failures/expected/image_checksum.html = IMAGE
WONTFIX : failures/expected/missing_text.html = MISSING PASS
WONTFIX : failures/expected/missing_image.html = MISSING PASS
Various tests for the article element.
On success, you will see a series of "PASS" messages, followed by "TEST COMPLETE".
<article> closes <p>:
PASS article1.parentNode.nodeName == "p" is false
<p> does not close <article>:
PASS p1.parentNode.nodeName is "ARTICLE"
<article> can be nested inside <article>:
PASS article3.parentNode.id is "article2"
Residual style:
PASS getWeight("article4") is "bold"
PASS getWeight("span1") is "bold"
FormatBlock:
PASS document.getElementById("span2").parentNode.nodeName is "ARTICLE"
PASS successfullyParsed is true
TEST COMPLETE
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML//EN">
<html>
<head>
<link rel="stylesheet" href="../../fast/js/resources/js-test-style.css">
<script src="../../fast/js/resources/js-test-pre.js"></script>
</head>
<body>
<p id="description"></p>
<div id="console"></div>
<script src="script-tests/article-element.js"></script>
<script src="../../fast/js/resources/js-test-post.js"></script>
</body>
</html>
......@@ -33,16 +33,8 @@ import os
import sys
import unittest
try:
d = os.path.dirname(__file__)
except NameError:
d = os.path.dirname(sys.argv[0])
sys.path.append(os.path.abspath(os.path.join(d, '..')))
sys.path.append(os.path.abspath(os.path.join(d, '../../thirdparty')))
import port
from test_expectations import *
from webkitpy.layout_tests import port
from webkitpy.layout_tests.layout_package.test_expectations import *
class FunctionsTest(unittest.TestCase):
def test_result_was_expected(self):
......@@ -98,18 +90,18 @@ class TestExpectationsTest(unittest.TestCase):
return os.path.join(self._port.layout_tests_dir(), test_name)
def get_basic_tests(self):
return [self.get_test('text/article-element.html'),
self.get_test('image/canvas-bg.html'),
self.get_test('image/canvas-zoom.html'),
self.get_test('misc/crash.html'),
self.get_test('misc/passing.html')]
return [self.get_test('failures/expected/text.html'),
self.get_test('failures/expected/image_checksum.html'),
self.get_test('failures/expected/crash.html'),
self.get_test('failures/expected/missing_text.html'),
self.get_test('passes/text.html')]
def get_basic_expectations(self):
return """
BUG_TEST : text/article-element.html = TEXT
BUG_TEST SKIP : misc/crash.html = CRASH
BUG_TEST REBASELINE : misc/missing-expectation.html = MISSING
BUG_TEST : image = IMAGE
BUG_TEST : failures/expected/text.html = TEXT
BUG_TEST SKIP : failures/expected/crash.html = CRASH
BUG_TEST REBASELINE : failure/expected/missing_image.html = MISSING
BUG_TEST : failures/expected/image_checksum.html = IMAGE
"""
def parse_exp(self, expectations, overrides=None):
......@@ -128,23 +120,23 @@ BUG_TEST : image = IMAGE
def test_basic(self):
self.parse_exp(self.get_basic_expectations())
self.assert_exp('text/article-element.html', TEXT)
self.assert_exp('image/canvas-zoom.html', IMAGE)
self.assert_exp('misc/passing.html', PASS)
self.assert_exp('failures/expected/text.html', TEXT)
self.assert_exp('failures/expected/image_checksum.html', IMAGE)
self.assert_exp('passes/text.html', PASS)
def test_duplicates(self):
self.assertRaises(SyntaxError, self.parse_exp, """
BUG_TEST : text/article-element.html = TEXT
BUG_TEST : text/article-element.html = IMAGE""")
BUG_TEST : failures/expected/text.html = TEXT
BUG_TEST : failures/expected/text.html = IMAGE""")
self.assertRaises(SyntaxError, self.parse_exp,
self.get_basic_expectations(), """
BUG_TEST : text/article-element.html = TEXT
BUG_TEST : text/article-element.html = IMAGE""")
BUG_TEST : failures/expected/text.html = TEXT
BUG_TEST : failures/expected/text.html = IMAGE""")
def test_overrides(self):
self.parse_exp(self.get_basic_expectations(), """
BUG_OVERRIDE : text/article-element.html = IMAGE""")
self.assert_exp('text/article-element.html', IMAGE)
BUG_OVERRIDE : failures/expected/text.html = IMAGE""")
self.assert_exp('failures/expected/text.html', IMAGE)
def test_matches_an_expected_result(self):
......@@ -153,16 +145,14 @@ BUG_OVERRIDE : text/article-element.html = IMAGE""")
self.get_test(test), result, pixel_tests_enabled)
self.parse_exp(self.get_basic_expectations())
self.assertTrue(match('text/article-element.html', TEXT, True))
self.assertTrue(match('text/article-element.html', TEXT, False))
self.assertFalse(match('text/article-element.html', CRASH, True))
self.assertFalse(match('text/article-element.html', CRASH, False))
self.assertTrue(match('image/canvas-bg.html', IMAGE, True))
self.assertTrue(match('image/canvas-bg.html', PASS, False))
self.assertTrue(match('misc/crash.html', SKIP, False))
self.assertTrue(match('misc/passing.html', PASS, False))
self.assertTrue(match('failures/expected/text.html', TEXT, True))
self.assertTrue(match('failures/expected/text.html', TEXT, False))
self.assertFalse(match('failures/expected/text.html', CRASH, True))
self.assertFalse(match('failures/expected/text.html', CRASH, False))
self.assertTrue(match('failures/expected/image_checksum.html', IMAGE, True))
self.assertTrue(match('failures/expected/image_checksum.html', PASS, False))
self.assertTrue(match('failures/expected/crash.html', SKIP, False))
self.assertTrue(match('passes/text.html', PASS, False))
if __name__ == '__main__':
unittest.main()
......@@ -48,7 +48,7 @@ class TestPort(base.Port):
def baseline_path(self):
return os.path.join(self.layout_tests_dir(), 'platform',
self.name())
self.name() + self.version())
def baseline_search_path(self):
return [self.baseline_path()]
......@@ -56,16 +56,18 @@ class TestPort(base.Port):
def check_build(self, needs_http):
return True
def compare_text(self, expected_text, actual_text):
return False
def diff_image(self, expected_filename, actual_filename,
diff_filename=None, tolerance=0):
return False
def diff_text(self, expected_text, actual_text,
expected_filename, actual_filename):
return ''
with codecs.open(actual_filename, "r", "utf-8") as actual_fh:
actual_contents = actual_fh.read()
with codecs.open(expected_filename, "r", "utf-8") as expected_fh:
expected_contents = expected_fh.read()
diffed = actual_contents != expected_contents
if diffed and diff_filename:
with codecs.open(diff_filename, "w", "utf-8") as diff_fh:
diff_fh.write("< %s\n---\n> %s\n" %
(expected_contents, actual_contents))
return diffed
def layout_tests_dir(self):
return self.path_from_webkit_base('WebKitTools', 'Scripts',
......@@ -82,6 +84,9 @@ class TestPort(base.Port):
'webkitpy', 'layout_tests', 'data', 'platform', 'test',
'test_expectations.txt')
def _path_to_wdiff(self):
return None
def results_directory(self):
return '/tmp/' + self._options.results_directory
......@@ -127,10 +132,7 @@ class TestPort(base.Port):
def test_platform_name_to_name(self, test_platform_name):
return test_platform_name
def version():
return ''
def wdiff_text(self, expected_filename, actual_filename):
def version(self):
return ''
......@@ -150,16 +152,58 @@ class TestDriver(base.Driver):
return 0
def run_test(self, uri, timeoutms, image_hash):
if not self._image_written and self._port._options.pixel_tests:
with open(self._image_path, "w") as f:
f.write("bad png file from TestDriver")
self._image_written = True
# We special-case this because we can't fake an image hash for a
# missing expectation.
if uri.find('misc/missing-expectation') != -1:
return (False, False, 'deadbeefdeadbeefdeadbeefdeadbeef', '', None)
return (False, False, image_hash, '', None)
basename = uri[(uri.rfind("/") + 1):uri.rfind(".html")]
error = ''
checksum = None
# There are four currently supported types of tests: text, image,
# image hash (checksum), and stderr output. The fake output
# is the basename of the file + "-" plus the type of test output
# (or a blank string for stderr).
#
# If 'image' or 'check' appears in the basename, we assume this is
# simulating a pixel test.
#
# If 'failures' appears in the URI, then we assume this test should
# fail. Which type of failures are determined by which strings appear
# in the basename of the test. For failures that produce outputs,
# we change the fake output to basename + "_failed-".
#
# The fact that each test produces (more or less) unique output data
# will allow us to see if any results get crossed by the rest of the
# program.
if 'failures' in uri:
crash = 'crash' in basename
timeout = 'timeout' in basename
if 'error' in basename:
error = basename + "_error\n"
if 'text' in basename:
output = basename + '_failed-txt\n'
else:
output = basename + '-txt\n'
if self._port.options().pixel_tests:
if ('image' in basename or 'check' in basename):
checksum = basename + "-checksum\n"
if 'image' in basename:
with open(self._image_path, "w") as f:
f.write(basename + "_failed-png\n")
elif 'check' in basename:
with open(self._image_path, "w") as f:
f.write(basename + "-png\n")
if 'checksum' in basename:
checksum = basename + "_failed-checksum\n"
else:
crash = False
timeout = False
output = basename + '-txt\n'
if self._port.options().pixel_tests and (
'image' in basename or 'check' in basename):
checksum = basename + '-checksum\n'
with open(self._image_path, "w") as f:
f.write(basename + "-png")
return (crash, timeout, checksum, output, error)
def start(self):
pass
......
......@@ -48,8 +48,11 @@ from webkitpy.layout_tests.layout_package import dump_render_tree_thread
from webkitpy.thirdparty.mock import Mock
def passing_run(args, port_obj=None, record_results=False):
def passing_run(args, port_obj=None, record_results=False,
tests_included=False):
args.extend(['--print', 'nothing'])
if not tests_included:
args.extend(['passes', 'failures/expected'])
if not record_results:
args.append('--no-record-results')
options, args = run_webkit_tests.parse_args(args)
......@@ -75,7 +78,7 @@ class MainTest(unittest.TestCase):
self.assertTrue(passing_run(['--platform', 'test']))
self.assertTrue(passing_run(['--platform', 'test', '--run-singly']))
self.assertTrue(passing_run(['--platform', 'test',
'text/article-element.html']))
'passes/text.html'], tests_included=True))
def test_one_child_process(self):
(res, buildbot_output, regular_output) = logging_run(
......@@ -122,21 +125,19 @@ class RebaselineTest(unittest.TestCase):
original_open = codecs.open
try:
# Test that we update expectations in place. If the expectation
# is mssing, update the expected generic location.
# is missing, update the expected generic location.
file_list = []
codecs.open = _mocked_open(original_open, file_list)
passing_run(['--platform', 'test', '--pixel-tests',
'--reset-results',
'image/canvas-bg.html',
'image/canvas-zoom.html',
'misc/missing-expectation.html'])
self.assertEqual(len(file_list), 9)
'passes/image.html',
'failures/expected/missing_image.html'],
tests_included=True)
self.assertEqual(len(file_list), 6)
self.assertBaselines(file_list,
"data/image/canvas-zoom")
"data/passes/image")
self.assertBaselines(file_list,
"data/platform/test/image/canvas-bg")
self.assertBaselines(file_list,
"data/misc/missing-expectation")
"data/failures/expected/missing_image")
finally:
codecs.open = original_open
......@@ -150,16 +151,14 @@ class RebaselineTest(unittest.TestCase):
codecs.open = _mocked_open(original_open, file_list)
passing_run(['--platform', 'test', '--pixel-tests',
'--new-baseline',
'image/canvas-zoom.html',
'image/canvas-bg.html',
'misc/missing-expectation.html'])
self.assertEqual(len(file_list), 9)
self.assertBaselines(file_list,
"data/platform/test/image/canvas-zoom")
'passes/image.html',
'failures/expected/missing_image.html'],
tests_included=True)
self.assertEqual(len(file_list), 6)
self.assertBaselines(file_list,
"data/platform/test/image/canvas-bg")
"data/platform/test/passes/image")
self.assertBaselines(file_list,
"data/platform/test/misc/missing-expectation")
"data/platform/test/failures/expected/missing_image")
finally:
codecs.open = original_open
......
......@@ -67,7 +67,7 @@ class TestExpectationsTestCase(unittest.TestCase):
def setUp(self):
self._error_collector = ErrorCollector()
port_obj = port.get('test')
self._test_file = os.path.join(port_obj.layout_tests_dir(), 'misc/passing.html')
self._test_file = os.path.join(port_obj.layout_tests_dir(), 'passes/text.html')
def process_expectations(self, expectations, overrides=None):
self._checker = TestExpectationsChecker()
......@@ -84,88 +84,88 @@ class TestExpectationsTestCase(unittest.TestCase):
def test_valid_expectations(self):
self.assert_lines_lint(
["misc/passing.html = PASS"],
["passes/text.html = PASS"],
"")
self.assert_lines_lint(
["misc/passing.html = FAIL PASS"],
["passes/text.html = FAIL PASS"],
"")
self.assert_lines_lint(
["misc/passing.html = CRASH TIMEOUT FAIL PASS"],
["passes/text.html = CRASH TIMEOUT FAIL PASS"],
"")
self.assert_lines_lint(
["BUG1234 TEST : misc/passing.html = PASS FAIL"],
["BUG1234 TEST : passes/text.html = PASS FAIL"],
"")
self.assert_lines_lint(
["SKIP BUG1234 : misc/passing.html = TIMEOUT PASS"],
["SKIP BUG1234 : passes/text.html = TIMEOUT PASS"],
"")
self.assert_lines_lint(
["BUG1234 DEBUG : misc/passing.html = TIMEOUT PASS"],
["BUG1234 DEBUG : passes/text.html = TIMEOUT PASS"],
"")
self.assert_lines_lint(
["BUG1234 DEBUG SKIP : misc/passing.html = TIMEOUT PASS"],
["BUG1234 DEBUG SKIP : passes/text.html = TIMEOUT PASS"],
"")
self.assert_lines_lint(
["BUG1234 TEST DEBUG SKIP : misc/passing.html = TIMEOUT PASS"],
["BUG1234 TEST DEBUG SKIP : passes/text.html = TIMEOUT PASS"],
"")
self.assert_lines_lint(
["BUG1234 DEBUG TEST : misc/passing.html = TIMEOUT PASS"],
["BUG1234 DEBUG TEST : passes/text.html = TIMEOUT PASS"],
"")
self.assert_lines_lint(
["SLOW DEFER BUG1234 : misc/passing.html = PASS"],
["SLOW DEFER BUG1234 : passes/text.html = PASS"],
"")
self.assert_lines_lint(
["WONTFIX SKIP : misc/passing.html = TIMEOUT"],
["WONTFIX SKIP : passes/text.html = TIMEOUT"],
"")
def test_valid_modifiers(self):
self.assert_lines_lint(
["INVALID-MODIFIER : misc/passing.html = PASS"],
["INVALID-MODIFIER : passes/text.html = PASS"],
"Invalid modifier for test: invalid-modifier "
"misc/passing.html [test/expectations] [5]")
"passes/text.html [test/expectations] [5]")
self.assert_lines_lint(
["SKIP : misc/passing.html = PASS"],
["SKIP : passes/text.html = PASS"],
"Test lacks BUG modifier. "
"misc/passing.html [test/expectations] [2]")
"passes/text.html [test/expectations] [2]")
self.assert_lines_lint(
["WONTFIX DEFER : misc/passing.html = PASS"],
["WONTFIX DEFER : passes/text.html = PASS"],
"Test cannot be both DEFER and WONTFIX. "
"misc/passing.html [test/expectations] [5]")
"passes/text.html [test/expectations] [5]")
def test_expectation_errors(self):
self.assert_lines_lint(
["missing expectations"],
"Missing expectations. ['missing expectations'] [test/expectations] [5]")
self.assert_lines_lint(
["SLOW : misc/passing.html = TIMEOUT"],
["SLOW : passes/text.html = TIMEOUT"],
"A test can not be both slow and timeout. "
"If it times out indefinitely, then it should be just timeout. "
"misc/passing.html [test/expectations] [5]")
"passes/text.html [test/expectations] [5]")
self.assert_lines_lint(
["does/not/exist.html = FAIL"],
"Path does not exist. does/not/exist.html [test/expectations] [2]")
def test_parse_expectations(self):
self.assert_lines_lint(
["misc/passing.html = PASS"],
["passes/text.html = PASS"],
"")
self.assert_lines_lint(
["misc/passing.html = UNSUPPORTED"],
["passes/text.html = UNSUPPORTED"],
"Unsupported expectation: unsupported "
"misc/passing.html [test/expectations] [5]")
"passes/text.html [test/expectations] [5]")
self.assert_lines_lint(
["misc/passing.html = PASS UNSUPPORTED"],
["passes/text.html = PASS UNSUPPORTED"],
"Unsupported expectation: unsupported "
"misc/passing.html [test/expectations] [5]")
"passes/text.html [test/expectations] [5]")
def test_already_seen_test(self):
self.assert_lines_lint(
["misc/passing.html = PASS",
"misc/passing.html = TIMEOUT"],
["passes/text.html = PASS",
"passes/text.html = TIMEOUT"],
"Duplicate expectation. %s [test/expectations] [5]" % self._test_file)
def test_tab(self):
self.assert_lines_lint(
["\tmisc/passing.html = PASS"],
["\tpasses/text.html = PASS"],
"Line contains tab character. [whitespace/tab] [5]")
if __name__ == '__main__':
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment