testing.rst 3.1KB

1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374
  1. Testing with ``pytest``
  2. #######################
  3. For ``dds``'s more rigorous test suite, we use the ``pytest`` testing framework.
  4. These tests are stored in the ``tests/`` directory and written in ``test_*.py``
  5. files.
  6. The test suite can be run separately without ``dds-ci`` by executing ``pytest``
  7. from within the :doc:`Poetry virtual environment <env>`::
  8. $ pytest tests/
  9. Note that individual tests can take between a few seconds and a few minutes to
  10. execute, so it may be useful to execute only a subset of the tests based on the
  11. functionality you want to test. Refer to
  12. `the pytest documentation <https://docs.pytest.org/en/latest/>` for more
  13. information about using and executing ``pytest``. If you are running the full
  14. test suite, you may also want to pass the ``-n`` argument with a number of
  15. parallel jobs to execute.
  16. .. highlight:: python
  17. Writing Tests
  18. *************
  19. If a particular aspect of ``dds`` can be tested in isolation and within a few
  20. dozen milliseconds, you should prefer to test it as a unit test in a
  21. ``*.test.cpp`` file. The ``pytest`` tests are intended to perform full
  22. end-to-end feature and error handling tests.
  23. Tests are grouped into individual Python files in the ``tests/`` directory. Any
  24. Python file containing tests must have a filename beginning with ``test_``.
  25. Individual test functions should begin with ``test_``. All test functions should
  26. be properly type-annotated and successfully check via ``mypy``.
  27. The ``dds`` test suite has access to a set of test fixtures that can be used
  28. throughout tests to perform complex setup and teardown for complete test-by-test
  29. isolation.
  30. Here is a simple test that simple executes ``dds`` with ``--help``::
  31. def test_get_help(dds: DDSWrapper) -> None:
  32. dds.run(['--help'])
  33. In this test function, :func:`the dds object is a test fixture
  34. <dds_ci.testing.fixtures.dds>` that wraps the ``dds`` executable under test.
  35. Testing Error Handling
  36. **********************
  37. It is important that ``dds`` handle errors correctly, of course, including user
  38. error. It is not simply enough to check that a certain operation fails: We must
  39. be sure that it fails *correctly*. To check that the correct code path is
  40. executed, ``dds`` can write a file containing a simple constant string
  41. designating the error handling path that was taken. The file will be written to
  42. the path indicated by the ``DDS_WRITE_ERROR_MARKER`` environment variable.
  43. For examples of these error strings, search for usage of ``write_error_marker``
  44. in the ``dds`` source code. These should only execute within error-handling
  45. contexts, should appear near the log messages that issue diagnostics, and should
  46. be specific to the error at hand.
  47. To write a test that checks for a given error-handling path, use the
  48. :func:`~dds_ci.testing.error.expect_error_marker` context manager function::
  49. def test_sdist_invalid_project(tmp_project: Project) -> None:
  50. # Trying to create a package archive from a project without a
  51. # package.json5 is invalid. Check that it creates the correct
  52. # error-message string
  53. with error.expect_error_marker('no-package-json5'):
  54. tmp_project.pkg_create()