Testing

Does the implementation conform to the specifications?

Automated testing is a valued tool in software development, however it's value doesn't diminish when maintaining systems, as regression testing ensures the functionality is unaffected by the changes.

When it comes to automated tests there are multiple schools of thought.

The Test Spectrum

• Unit Tests
• Integration Tests
• End-to-end Tests

Note: Avoid over-testing and writing bad tests

At the smallest most isolated test is the unit test. These should be fast to run and are focused on behaviour of a given unit.

At the other end of the spectrum is full end-to-end tests involving multiple systems and a more involved test environment.

Between these are the integration tests which test the combined functionally of multiple units.

Avoid over testing and redundant tests.

References:

https://martinfowler.com/articles/practical-test-pyramid.html

Code Coverage

Robustness

• Measurement of tests
• As with any metric, avoid overoptimizing it (Goodhart's law)

In conjunction with tests one might one to measure how well these test cover the codebase. Code coverage is the metric for how many lines/branches/expressions are hit by the tests.

This can be usual to find areas or branches that or untested.

However as with any metric avoid over-optimizing it, as trying to have perfect test coverage can hurt the development process and inevitably the software quality itself. Gating pull/merge-requests by code coverage can result in stagnation.

References:

• https://sohl-dickstein.github.io/2022/11/06/strong-Goodhart.html

Property Based Testing

Robustness

• QuickCheck --- Haskell 1999

Property based testing is way to the test based on properties of the input. As such the test framework can generate arbitrary inputs that satisfy the properties and how the test perform.

When a test is failed the given test instance is and can be handled as a specific edge case.

Fuzzing

Robustness

• Blackbox test for trying to crash the program

Fuzzing is akin to property based tests, but instead looks at the program as black box and tries to 'brute force' the program to fail. This is usual in testing parsing and validation of arbitrary user input.

Fuzzing usually runs more slowly.

Other Tools

• Valgrind
• GDB / LLDB