Skip to content

Pester, coverage in Appveyor - #2499

Merged
potatoqualitee merged 11 commits into
dataplat:developmentfrom
niphlod:appveyor_coverage
Oct 24, 2017
Merged

Pester, coverage in Appveyor#2499
potatoqualitee merged 11 commits into
dataplat:developmentfrom
niphlod:appveyor_coverage

Conversation

@niphlod

@niphlod niphlod commented Oct 23, 2017

Copy link
Copy Markdown
Contributor

Type of Change

  • Bug fix (non-breaking change, fixes #)
  • New feature (non-breaking change, adds functionality)
  • Breaking change (effects multiple commands or functionality)
  • Ran manual Pester test and has passed (`.\tests\manual.pester.ps1)
  • Adding code coverage to existing functionality
  • Pester test is included
  • Nunit test is included
  • Documentation
  • Build system

Purpose 1

Complementing #2411, this brings code coverage to codecov.io for ps1 files.
This will make codecov.io the entry point for tests creators to see what is currently covered (with all intricacies of PS coverage, see this) and what is not, line by line.

Coincidentally, it will also bring closer attention to what is tested and how to "the public".
On top of it, it'll - probably - foster:

  • "secure" adoption by users in their scripts if the coverage for the function is above average
  • "careful" adoption by users in their scripts if the coverage is below average
  • "screaming headless torsos" users asking for tests to be added for whatever function is not covered

Purpose 2

Bringing "autodetection" of the correct build in the appveyor build matrix without further fiddling with pester.groups.ps1.

For "unaware of the details" people, let's go back one step: running two concurrent instances on appveyor has proven to be resource-intensive and prone to random failures.
A build matrix has been prepared to let tests which DO NOT NEED both instances up and running to have more resources at their disposal.

Those builds are "scenarios":

scenario instances
2008R2 only 2008R2 active
2016 only 2016 active
2016_service only 2016 active, tied to some disrupting tests that restart the instance
default both instances active

Everyone should try to NOT end up in the default scenario, which should be needed only for Copy-* functions.

The autodetection logic follows these rules:

  • if the scenario in pester.groups.ps1 is present
    • if the scenario is not "autodetect" (e.g. 2016_service)
      • run tests tied to that scenario
    • if the scenario is "autodetect" (e.g. 2008R2)
      • run tests files which contain ONLY the autodetected string (ATM $script:instance1 on 2008R2, $script:instance2 on 2016)
  • if the scenario is not present
    • run any leftover (which, ATM, makes the default scenario run everything that is not specifically tied (2016_service), containing BOTH $script:instance1 and $script:instance2)

Additional care has been taken if for some reason in the future one wants to write a test scenario that needs to go separately on multiple builds (i.e. a test file on 2008R2 and a test file on 2016, not ending up in the default scenario): it's sufficient to create:

  • Get-DbaFoo.whatever.Tests.ps1 (containing only references to $script:instance1)
  • Get-DbaFoo.whatever2.Tests.ps1(containing only references to $script:instance2)

Approach

Coverage is brought you by inferencing dependencies automatically (no AST here yet, see Get-CoverageIndications within appveyor.pester.ps1), as sending every function to pester's CodeCoverage parameter takes up zillions of resources (both CPU and RAM).
The report to codecov.io is custom-made following codecov.io documentation, as the default jacoco one in latest pester releases does not bring line-by-line information (we should probably raise a PR to pester's repo, adding a new output format).

Autodetection is brought by scanning the test files for occurrences of $script:instance1 on 2008R2, $script:instance2 on 2016, logic is in appveyor.pester.ps1

Improvements, in later PRs

@wsmelton wsmelton left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually understood a portion of this ⛅️

@potatoqualitee
potatoqualitee merged commit c3a9f13 into dataplat:development Oct 24, 2017
@niphlod
niphlod deleted the appveyor_coverage branch November 27, 2017 21:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants