8000 Backend test python script improvement and docs by mhoff12358 · Pull Request #8816 · google/filament · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Backend test python script improvement and docs #8816

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 9, 2025
Merged

Backend test python script improvement and docs #8816

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 63 additions & 0 deletions filament/backend/test/README.md
8000
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# Backend Unit Tests

These are tests that ensure the Filament backend is working properly on various operating systems
and graphics backends.

The majority of these tests generate images that are then compared against a known golden image.

## Running with a specific graphics library

Run with `-a<backend>` to run with a specific backend, such as `-avulkan` for vulkan or `-ametal`
for metal.

## Image comparisons

The expected images are stored as PNG files in the `expected_images` subdirectory of the test source
code. When cmake is run it will copy this directory to the build output creating
`images/expected_images` inside the same directory as the unit test binary.

The unit tests will then write their resulting images into `images/actual_images` in order to make
the tests easier to debug.

### Python utility for updating golden images and comparing results

Inside the unit test source code directory there is a python script called
`move_actual_images_to_expected.py`.
It will display the actual and expected images side-by-side and copy the actual image into the
source tree's `expected_images` directory if the user approves the change.

#### Directories

The `-r` flag is required and should be the directory where the test binary is.

If not running the script from the directory it's in, `-s` should be the source code's
`expected_images` directory.

#### Configuring compare/move behavior

The `-c` flag has the tool display the actual and expected images side-by-side. By default it
does this by opening both with the OS's default behavior. `-p` can be used to specify a different
terminal program.

The `-m` flag has the tool move the actual image to overwrite the expected image in the source
tree. If the images are also being compared then the move will only happen if the user approves the
change.

After updating the expected images, cmake will need to be run again to copy them to the binary's
directory. Also, currently some platforms can't load images to compare and so have the hash
hardcoded into the test source code, so for now those need to be updated by running the test and
copying the value manually.

#### Picking which tests to compare

The `-x` flag causes the tool to check the `test_detail.xml` file and only process the images who
failed a comparison. `--gtest_output=xml` needs to be passed to the test binary to generate this
file.

The `-t` argument takes a list of images to compare, provided as the file name without the `.png`
suffix.

## Unsupported tests

If a test depends on a feature that is unsupported by the current environment, it will start with
the `SKIP_IF` macro. This will cause the test to not be run.
137 changes: 80 additions & 57 deletions filament/backend/test/move_actual_images_to_expected.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,26 +25,40 @@ def get_latest_failed_images(self) -> typing.List[str]:

return failed_images

def handle_failed_image(self, failed_image):
self.show_images(failed_image)
print(f'Update {failed_image}\'s expected image? y/n')
while True:
user_input = input()
if user_input == 'y':
self.move_actual_to_source([failed_image])
break
8000 elif user_input == 'n':
break

def handle_all_failed_images(self):
for failed_image in self.get_latest_failed_images():
self.handle_failed_image(failed_image)

def show_images(self, failed_image):
def process(self, test_prefixes: typing.List[str], compare: bool, move: bool,
compare_program: str = ''):
for test in test_prefixes:
self.handle_image(test, compare, move, compare_program)

def handle_image(self, failed_image: str, compare: bool, move: bool, compare_program: str):
if compare:
self.show_images(failed_image, compare_program)

# If the image can be moved, prompt the user after the comparison. But always wait for
# user input to not spam them with images.
if move:
print(f'Update {failed_image}\'s expected image? y/n')
while True:
user_input = input()
if user_input == 'y':
break
elif user_input == 'n':
move = False
break
else:
print(f'Move to next?')
input()

if move:
self.move_actual_to_source([failed_image])

def show_images(self, failed_image: str, command: str):
# TODO: Test more on non-mac systems
open_command: str
os_name = platform.system().lower()
if 'windows' in os_name:
if command:
open_command = command
elif 'windows' in os_name:
open_command = 'start'
elif 'osx' in os_name or 'darwin' in os_name:
open_command = 'open'
Expand All @@ -63,12 +77,6 @@ def move_actual_to_source(self, file_prefixes: typing.List[str]):
replacement=TestResults.EXPECTED_SUFFIX,
output_path=self.source_expected_directory, prefixes=file_prefixes)

def batch_move(self, prefixes: typing.Optional[typing.List[str]] = None):
replace_file_names(path=self.actual_directory, removed=TestResults.ACTUAL_SUFFIX,
replacement=TestResults.EXPECTED_SUFFIX,
output_path=self.source_expected_directory, prefixes=prefixes)


def match_suffix(file_name: str, suffix: str, accepted_prefixes: typing.List[str]) -> str:
"""
Check if the file name is one of the searched for ones with the given suffix and if so return
Expand Down Expand Up @@ -105,52 +113,67 @@ def replace_file_names(path: str, removed: str, replacement: str = '', output_pa
print(f'Failed to find {unfound_prefix}_actual.png')


if __name__ == "__main__":
def main():
parser = argparse.ArgumentParser(prog='Backend Test File Renamer',
description='Moves actual generated test images to the '
'expected images directory, to update the test '
'requirements. test_cases accepts multiple '
'arguments that should be the name of the '
'expected image file without the .png suffix. '
'Also --all can be passed to copy all images.\n'
'Remember to sync CMake after running this to '
'move the new expected images to the binary '
'directory.')
description=
'Compares and moves actual generated test images to the expected images directory to '
'update the test requirements. You need to use -r to specify the directory with the test '
'binary.\nYou likely want the flags -cmx to compare and prompt to move every failed image'
'\nRemember to sync CMake after running this to move the new expected images to the binary '
'directory.')
parser.add_argument('-r', '--results_path',
help='The path with the generated images directory, which should be where '
'the test binary was run.')
parser.add_argument('-s', '--source_expected_path', default="./expected_images",
help='The directory that updated expected images should be written to, '
'which should be the source directory copy.')
# The mutually exclusive options for how to process the actual images
parser.add_argument('-b', '--batch', action='extend', nargs='*',
help='If true copy all actual images to the source expected image '
'directory.')
parser.add_argument('-a', '--all', action='store_true',
help='If true, visually compare all generated images.')
parser.add_argument('-t', '--tests', action='store_true',
help='If true use a test_detail.xml file that exists in the results_path '
'directory to visually compare all images that failed a test.')
parser.add_argument('-c', '--compare', action='extend', nargs='*',
help='A list of image names to visually compare (without the .png suffix).')
parser.add_argument('-c', '--compare', action='store_true',
help='If true, actual and expected images will be displayed to the user '
 8000 9;for comparison.')
parser.add_argument('-p', '--compare_program', default=None,
help='The terminal program that should be used to open images when '
'comparing them')
parser.add_argument('-m', '--move_files', action='store_true',
help='If true, the actual images will be copied to overwrite the source '
'tree\'s expected image for that test. If the --compare flag is also '
'present the user will be prompted for each copy.')
# The mutually exclusive options for what images to compare.
parser.add_argument('-t', '--tests', action='extend', nargs='*',
help='The list of test images to compare, provided as the name of the '
'actual image file without the .png suffix.')
parser.add_argument('-x', '--xml', action='store_true',
help='If true use a test_detail.xml file generated by the test binary to '
'compare all images that failed a test. Remember to pass '
'`--gtest_output=xml` to the test binary to generate this file.')

args = parser.parse_args()
if not args.results_path:
raise AssertionError("No result path provided")
results_path = args.results_path

# Catch argument mistakes
if not args.move_files and not args.compare:
print("Neither instructed to move or compare images")
return
if args.compare_program and not args.compare:
print("Compare program provided but not comparing images")
return

results_path = args.results_path
results = TestResults(results_directory=results_path,
source_expected_directory=args.source_expected_path)

if args.all:
results.batch_move()
elif args.tests:
results.handle_all_failed_images()
elif args.compare:
for file_prefix in args.compare:
results.show_images(file_prefix)
else:
results.batch_move(args.batch)
print("--------------------------------------------------------")
print("REMEMBER TO RESYNC CMAKE AND UPDATE HASHES IN TEST FILES")
print("--------------------------------------------------------")
test_prefixes = args.tests
if args.xml:
test_prefixes = results.get_latest_failed_images()

results.process(test_prefixes, compare=args.compare, move=args.move_files,
compare_program=args.compare_program)

if args.move_files:
print("--------------------------------------------------------")
print("REMEMBER TO RESYNC CMAKE AND UPDATE HASHES IN TEST FILES")
print("--------------------------------------------------------")


if __name__ == "__main__":
main()
Loading
0