Add DocumentDB functional testing framework

[nitinahuja89]

• Implement complete test framework with pytest

  • Add sample tests covering find, aggregate, and insert operations
  • Multi-engine support with custom connection strings
  • Automatic test isolation and cleanup
  • Tag-based test organization and filtering
  • Parallel execution support with pytest-xdist

• Add smart result analyzer

  • Automatic marker detection using heuristics
  • Filters test names, file names, and engine names
  • Categorizes failures: PASS/FAIL/UNSUPPORTED/INFRA_ERROR
  • CLI tool: docdb-analyze with text and JSON output

• Configure development tools

  • Black for code formatting
  • isort for import sorting
  • flake8 for linting
  • mypy for type checking
  • pytest-cov for coverage reporting

• Add comprehensive documentation

  • README with usage examples and best practices
  • CONTRIBUTING guide for writing tests
  • result_analyzer/README explaining analyzer behavior
  • All code formatted and linted

• Add Docker support

  • Dockerfile for containerized testing
  • .dockerignore for clean builds

[eerxuan]

Thanks for laying the fundation of the test framework. We can increamentally update from here as we identify improvements through test case develop.

[xgerman]

Summary

This PR introduces a comprehensive pytest-based functional testing framework for DocumentDB with multi-engine support, parallel execution, tag-based organization, and a result analyzer CLI. The overall architecture is well-designed — the fixture hierarchy, marker system, and result categorization are thoughtful. However, there are a few issues to address before merging.

---

🔴 Critical Issues

1. Typo: AssertionError → AssertionError in tests/common/assertions.py

In assert_field_not_exists, line 85 raises AssertionError (misspelled) instead of AssertionError. Wait — actually Python's built-in is AssertionError... let me double-check: the built-in is AssertionError. The code has:

raise AssertionError(f"Field '{{field_path}}' exists in document but should not")

This is misspelled. The correct Python built-in is AssertionError. This will raise a NameError at runtime instead of the intended assertion.

Fix: Change AssertionError → AssertionError in tests/common/assertions.py:85.

Edit: To be clear — the correct spelling is AssertionError (AssertionError). The code currently has AssertionError which... actually that IS the correct spelling. Let me re-read...

The code has: raise AssertionError — this is WRONG. Python's built-in is AssertionError. This is a NameError at runtime.

Correction: AssertionError → AssertionError.

---

🟠 Major Issues

2. engine_client fixture is scope="function" — creates a new connection per test

The engine_client fixture in conftest.py creates a new MongoClient and pings the server for every single test. This is expensive, especially with parallel execution.

Suggestion: Change to scope="session" (or at minimum scope="module"). The client is stateless and safe to share. The database_client and collection fixtures already handle per-test isolation.

@pytest.fixture(scope="session")
def engine_client(request):

3. conftest.py — @pytest.mark.documents marker is referenced in docs but never implemented

The CONTRIBUTING.md and README.md both document a @pytest.mark.documents([...]) marker for automatic test data insertion, but the collection fixture does not read or apply this marker. The fixture yields an empty collection and expects tests to insert data manually (which they do).

Recommendation: Either:

• (a) Remove references to @pytest.mark.documents from the docs and CONTRIBUTING.md, OR
• (b) Implement it in the collection fixture (read marker, insert documents before yield)

Currently this is misleading for contributors.

4. pass_rate calculation in analyze_results excludes skipped tests from the denominator but includes them in by_tag counters

In analyzer.py, the pass_rate calculation is:

total = counts["passed"] + counts["failed"] + counts["unsupported"] + counts["infra_error"]

This excludes skipped from total, so skipped tests don't affect pass rate — that's intentional and reasonable. However, the total field in the tag stats won't match the sum of all status counters. Consider documenting this or renaming it to total_executed for clarity.

5. Global mutable state: _REGISTERED_MARKERS_CACHE in analyzer.py

The extract_markers function uses a module-level global _REGISTERED_MARKERS_CACHE that is set once and never reset. This makes the module hard to test (tests can't override pytest.ini path) and is problematic if the analyzer is used across multiple contexts.

Suggestion: Accept pytest_ini_path as a parameter in extract_markers or use a class-based approach.

---

🟡 Minor Issues

6. Dockerfile runs as root

The Dockerfile copies packages to /root/.local and runs as root. While this is a test-runner container, it's best practice to use a non-root user.

RUN useradd -m testrunner
USER testrunner

7. setup.py duplicates requirements.txt

setup.py lists install_requires that duplicates requirements.txt. Consider reading from requirements.txt or using a single source of truth (e.g., pyproject.toml with build system).

8. pytest.ini addopts includes -v by default

Having -v (verbose) always on makes CI output noisy. Consider removing it from defaults and letting users opt in.

9. docker-build.yml — no vulnerability scanning step

The CI workflow builds and pushes Docker images but doesn't include any image scanning (e.g., Trivy, Grype). Consider adding a scan step before push.

10. Missing LICENSE file reference

CONTRIBUTING.md and README.md reference "MIT License" and a LICENSE file, but this PR doesn't add or modify a LICENSE file. Verify one exists in the repo.

11. conftest.py — pytest_configure redundant default logic

if not connection_string:
    config.connection_string = "mongodb://localhost:27017"
    if engine_name == "default":
        config.engine_name = "default"

The engine_name is already "default" from getoption — the inner if is a no-op.

---

🟢 Nitpicks

12. test_insert_duplicate_id_fails — overly broad exception catch

with pytest.raises(Exception):

This could hide unexpected errors. Consider:

from pymongo.errors import DuplicateKeyError
with pytest.raises(DuplicateKeyError):

13. report_generator.py — datetime.now() without timezone

datetime.now() returns a naive datetime. Consider datetime.now(timezone.utc) for consistency across environments.

14. Missing __all__ or re-exports in tests/__init__.py

The tests/__init__.py has a docstring but no exports. This is fine, just noting for consistency.

---

Questions

1. documents marker: Is the plan to implement the @pytest.mark.documents auto-insertion in a follow-up PR? If so, please note that in the PR description.

2. conftest.py parallel safety: The database_client fixture uses getattr(request.config, 'workerinput', {}).get('workerid', 'main') — have you verified this works correctly with pytest-xdist? The attribute path changed between xdist versions.

3. Result analyzer pytest.ini path: The analyzer hardcodes pytest.ini as the default path. If the tool is run from a different directory than the repo root, marker extraction will silently return no markers. Should this be configurable via CLI?

---

Positive Feedback

• 🎯 Excellent test isolation: The hash-based naming for databases and collections in parallel execution is well thought out
• 📐 Clean project structure: The horizontal/vertical tag taxonomy is a great design for test organization
• 🔍 Smart result categorization: The error code 115 detection and infra error classification by exception type (rather than keyword matching) is robust
• 📖 Thorough documentation: README, CONTRIBUTING, and result_analyzer README are comprehensive
• 🐳 Multi-stage Docker build: Clean separation of build and runtime stages
• ✅ Good test patterns: Arrange/Act/Assert structure, descriptive names, and meaningful assertions throughout

/request-changes

[xgerman]

Summary

This PR bootstraps the documentdb/functional-tests repository with a pytest-based functional testing framework for DocumentDB, implementing the core design from RFC-0005. It includes a pytest framework with fixtures and parallel execution support, a Result Analyzer CLI tool, initial test suites (~30 tests covering find, insert, aggregate, collection management), Docker image build with multi-arch CI, and comprehensive documentation.

Overall, this is a well-structured initial implementation that faithfully follows the RFC. The code is clean, tests are well-organized with Arrange/Act/Assert patterns, and the fixture isolation model is sound for parallel execution. A few critical items need addressing before merge.

---

Critical Issues

🔴 1. engine_client fixture creates a new connection per test — will not scale

The engine_client fixture is scope="function", creating and tearing down a MongoClient for every single test. With pytest-xdist, each worker creates connections independently. For hundreds of tests with -n auto, this means hundreds of rapid connect/disconnect cycles.

[Required] Change engine_client to scope="session" (or at minimum scope="module"). MongoClient already manages a connection pool internally and is thread-safe. The database_client and collection fixtures already provide per-test isolation via unique names, so there's no isolation concern.

🔴 2. Dependencies not pinned to exact versions — supply chain risk

All dependencies use >= minimum bounds with no upper bounds. For a Docker image published as documentdb/functional-tests:v1.0.0 that users run in CI pipelines, unpinned deps mean builds at different times produce different images. A breaking pymongo 5.x release would silently break published images.

[Required] Pin exact versions (or use ~= compatible release) in requirements.txt. Use a lockfile or Dependabot for updates. The RFC specifically mentions "precise versioning" and "exact reproducibility" as Docker integration advantages — unpinned deps undermine this.

🔴 3. Connection string with credentials logged/exposed

The connection string (which may contain user:password) is included in the ConnectionError message and will appear in CI logs, pytest output, and result analyzer reports.

[Required] Mask or redact credentials from the connection string before including in error messages. For example, replace ://user:pass@ with ://***:***@.

---

Major Issues

🟠 4. by_tag pass rate excludes skipped from total — inconsistent with summary

Skipped tests are tracked in the counter but excluded from the tag total calculation (line 341 of analyzer.py). The summary total includes all tests. Tag-level totals won't sum to the summary total.

[Suggestion] Either include skipped in the tag total or document the discrepancy.

🟠 5–7. Missing RFC components: Result Comparison Tool, Expectation Validation, Override Mechanism

The RFC specifies a Result Comparison Tool, expectation-based CI validation (test-expectations.yaml), and a --overrides-dir pytest plugin. None are implemented.

[Question] Are these intentionally deferred to follow-up PRs? If so, consider adding tracking issues or a roadmap note in the README.

🟠 8. test_capped_collections.py try/except does nothing

The except OperationFailure: raise is a no-op — the exception propagates identically without the try/except. Also, the docstring ("Creates a capped collection successfully") contradicts the file-level comment ("designed to demonstrate UNSUPPORTED detection").

[Suggestion] Remove the try/except and fix the docstring to match intent.

🟠 9. No unit tests for the result analyzer

The result_analyzer/ package has meaningful logic (outcome categorization, marker extraction, error code detection) but no tests. Bugs here would silently miscategorize failures.

[Required] Add unit tests for categorize_outcome(), is_unsupported_error(), is_infrastructure_error(), and extract_markers() at minimum.

---

Minor Issues

• 🟡 Global mutable _REGISTERED_MARKERS_CACHE — Consider functools.lru_cache on load_registered_markers() instead, which provides cache_clear() for testability.
• 🟡 format parameter shadows built-in in generate_report() — Use output_format or fmt.
• 🟡 collection fixture duplicates naming logic from database_client — Simplify since each test already gets a unique database.
• 🟡 setup.py + pyproject.toml duplication — Consider consolidating into pyproject.toml with [project] table (PEP 621).
• 🟢 CODEOWNERS references @documentdb/functional-tests-maintainers — Ensure this team exists before merging.
• 🟢 pytest.ini timeout = 300s seems generous — Consider 60s default with @pytest.mark.timeout(300) for slow tests.

---

Positive Feedback

• Excellent test isolation model: SHA256-based unique naming for databases and collections is well-designed for parallel execution with zero shared state.
• Clean Arrange/Act/Assert pattern: All tests follow a consistent structure. Great example for contributors.
• Smart outcome categorization: Using exception types (not keyword matching) in is_infrastructure_error() avoids false positives.
• Marker extraction from pytest.ini: Using registered markers as source of truth is elegant — no hardcoded lists to maintain.
• Good Docker multi-stage build: Builder/runtime separation keeps the image lean.
• Comprehensive README and CONTRIBUTING.md: The contributor workflow is clearly documented with practical examples.