Remove assertions (and helper methods) that operate on (non-public) attributes

[sebastianbergmann]

The following methods should be removed:

• assertAttributeContains()
• assertAttributeNotContains()
• assertAttributeContainsOnly()
• assertAttributeNotContainsOnly()
• assertAttributeCount()
• assertAttributeNotCount()
• assertAttributeEquals()
• assertAttributeNotEquals()
• assertAttributeEmpty()
• assertAttributeNotEmpty()
• assertAttributeGreaterThan()
• assertAttributeGreaterThanOrEqual()
• assertAttributeLessThan()
• assertAttributeLessThanOrEqual()
• assertAttributeSame()
• assertAttributeNotSame()
• assertAttributeInstanceOf()
• assertAttributeNotInstanceOf()
• assertAttributeInternalType()
• assertAttributeNotInternalType()
• attribute()
• attributeEqualTo()
• readAttribute()
• getStaticAttribute()
• getObjectAttribute

[localheinz]

@sebastianbergmann

Interesting!

When you have the time, can you share what the plans are? Should these be replaced by using non-specialized (or domain-specific) assertions in userland code?

[sebastianbergmann]

These assertions make it too easy to test implementation details (private and protected attributes) that should not be tested directly.

[localheinz]

@sebastianbergmann

Ha, I missed the Attribute part. Let’s kill them with 🔥!

[gmponos]

Is there any related discussion that explain the thought behind this decision?

[sebastianbergmann]

A test should not depend on private implementation details. It was a bad idea to make a bad testing practice this convenient.

[gmponos]

These assertions make it too easy to test implementation details (private and protected attributes) that should not be tested directly.
A test should not depend on private implementation details. It was a bad idea to make a bad testing practice this convenient.

I know that this is true but on the other hand this is just a misuse of a very helpful feature let me give you one of my use cases:

namespace Whatever;

class ResetPasswordHandler
{
    /** @var ResetPasswordRepository */
    private $resetPasswordRepository;

    public function __construct(ResetPasswordRepository $resetPasswordRepository)
    {
        $this->resetPasswordRepository = $resetPasswordRepository;
    }

    public function handle(string $oneTimeToken, string $email)
    {
        /** @var ResetPassword $resetPassword */
        $resetPassword = $this->resetPasswordRepository->findOneBy(['email' => $email]);

        if (!$resetPassword->hasReachedAttemptLimit() && $resetPassword->getToken() !== $oneTimeToken) {
            $resetPassword->increaseAttempts();
            $this->resetPasswordRepository->save($resetPassword);
            return false;
        } else {
            return true;
        }
    }
}

/** My entity class */
class ResetPassword
{
    private CONST MAX_ATTEMPTS = 3;

    private $uuid;
    private $attempts;
    private $lastAttempt;
    private $email;
    private $token;

    public function __construct($uuid, $email, $token, $attempts, \DateTimeInterface $lastAttempt)
    {
        $this->uuid = $uuid;
        $this->attempts = $attempts;
        $this->lastAttempt = $lastAttempt;
        $this->email = $email;
        $this->token = $token;
    }

    public function hasReachedAttemptLimit()
    {
        return $this->attempts >= self::MAX_ATTEMPTS;
    }

    public function increaseAttempts()
    {
        $this->attempts++;
        $this->lastAttempt = new \DateTime();
    }

    /** @return string */
    public function getToken()
    {
        return $this->token;
    }
}

class ResetPasswordHandlerUnitTest extends TestCase
{
    private $repository;
    private $handler;

    public function setUp()
    {
        $this->repository = $this->getMockBuilder(ResetPasswordHandler::class)->disableOriginalConstructor()->getMock();
        $this->handler = new ResetPasswordHandler($this->repository);
    }

    public function testThatItIncreasedTheCount()
    {
        $entity = new ResetPassword('12234', 'test@test.com', 'whatevertoken', 0, new \DateTime());
        $this->repository->expects($this->once())->method('findOneBy')->willReturn($entity);
        $this->handler->handle('notmatch', 'test@test.com');
        $this->assertAttributeSame(1, 'attempts', $entity);
    }
}

If you remove these functions in order for me to test that the developer has not made anything stupid like commenting a line like this:

    public function increaseAttempts()
    {
	    // $this->attempts++;
        $this->lastAttempt = new \DateTime();
    }

I will need to create a getter function for attempts which honestly I hate doing. I mean you shouldn't add functions just in order to test your code. Another solution would be to extend this entity under a Test namespace and add the getter there.

So again if a developer wants to actually test the private details of an object by removing this we are forcing them follow even worse paths.

Note that I guess that most probably you will disagree with how the example is structured but similar scenarios happen regularly in everyday life and hence these are some pretty useful functions

[localheinz]

@gmponos

How about testing it like this instead?

final class ResetPasswordTest extends TestCase
{
    public function testHasReachedAttemptLimitReturnsFalseWhenAttemptLimitHasNotBeenReached(): void
    {
        $attempts = 0;

        $resetPassword = new ResetPassword(
            '12234',
            'test@test.com',
            'whatevertoken',
            $attempts,
            new \DateTimeImmutable()
        );

        $resetPassword->increaseAttempts();

        $this->assertFalse($resetPassword->hasReachedAttemptLimit());
    }

    public function testHasReachedAttemptLimitReturnsTrueWhenAttemptLimitHasBeenReached(): void
    {
        $attempts = ResetPassword::MAX_ATTEMPTS - 1;

        $resetPassword = new ResetPassword(
            '12234',
            'test@test.com',
            'whatevertoken',
            $attempts,
            new \DateTimeImmutable()
        );

        $resetPassword->increaseAttempts();

        $this->assertTrue($resetPassword->hasReachedAttemptLimit());
    }
}

There are always alternatives! 👍

[gmponos]

Yea.. ok.. in this case you got me.. but the example was just for the sake of the argument... For instance what if he comments the $this->lastAttempt = new \DateTime();...?

There will be cases that this will make testing much harder driving developers to harder and worse paths... :(

[epdenouden]

Hi @gmponos!

Step away from the PHP for a minute. What behaviour do you want to test? How about this:

1. do a first attempt to reset the password
2. assert that the maximum number of attempts has not been reached
3. do a second attempt to reset the password
4. assert that the maximum number of attempts has not been reached
5. do a third attempt to reset the password
6. assert that the maximum number of attempts has not been reached
7. do a fourth attempt to reset the password
8. assert that the maximum number of attempts has been reached

I know that this is true but on the other hand this is just a misuse of a very helpful feature let me give you one of my use cases:
[...]
I will need to create a getter function for attempts which honestly I hate doing. I mean you shouldn't add functions just in order to test your code.

The testing problem is quickly solved by looking at the difference between testing implementation versus testing behaviour of a system. When I started working on the test execution reordering and its cache I ran into this question. The TestSuiteSorter and ResultCacheExtension are designed to work without any user output and to have as little effect as possible.

My first tests relied heavily on --debug output and even reflection. This caused me a lot of grief later on, when refactoring the TestSuiteSorter forced me to rewrite a lot of tests. Apart from all the extra work the testing confidence went out the window, as I had a lot fewer stable tests during the refactoring! Not fun.

The cleanest solution turned out to be to only test the observable external behaviour:

• the TestSuiteSorter should give me: TestSuites that are sorted in a different order. I can observe this by looking at the order the tests are actually run. I can observe this with --debug (not great) or by watching a streaming log format like TeamCity (much better!)
• the ResultCacheExtension should give me: a result cache file that matches the results of the previous run