Python 3.12 · pandas · data quality · KPI reconciliation · reporting · Jupyter · pytest · Ruff · GitHub Actions
A compact, tested Data/BI workflow that turns an intentionally imperfect CSV into validated records, auditable rejections, reconciled KPIs, reporting summaries, charts and a clean verification notebook.
This repository is part of the DataTideHH portfolio for my IHK retraining in Data and Process Analysis. Its purpose is not to imitate an enterprise platform. It demonstrates a small workflow that can be inspected, executed, tested and explained end to end.
Portfolio status: Core scope complete. Future changes are limited mainly to dependency compatibility, defect correction and documentation clarity. See
docs/project-status.md.
| Capability | Evidence in this repository |
|---|---|
| Reproducible Python setup | Python 3.12 project metadata, explicit dependency groups and platform-specific setup documentation |
| Data-quality controls | Required columns, controlled type conversion, missing values, ranges, duplicate handling and row-level rejection reasons |
| Data lineage | Original CSV row retained as source_row in validated and rejected outputs |
| KPI logic | Module-level result counts, learner counts, average scores, pass/fail counts and pass rates |
| Reconciliation | Persisted KPI output is recalculated from cleaned records and rejected when values differ |
| Reporting | Deterministic summaries plus Matplotlib SVG charts |
| Notebook discipline | Clean notebooks without committed outputs, execution counts, local paths or IDE timestamps |
| Automated verification | pytest, Ruff, bytecode compilation and end-to-end execution on Ubuntu and Windows |
| Scope discipline | Synthetic data, explicit limitations and no production-scale claims |
data/raw/training_results.csv
│
▼
validation and controlled cleaning
│
├── cleaned_results.csv
├── rejected_results.csv
├── module_kpis.csv
└── quality_report.json
│
▼
KPI recalculation and reconciliation
│
├── rejection_reason_summary.csv
├── reporting_summary.json
├── average_score_by_module.svg
└── pass_rate_by_module.svg
│
▼
clean reporting notebook with repeated assertions
The reporting layer does not trust module_kpis.csv blindly. It derives the expected KPIs again from cleaned_results.csv and stops if the persisted and recalculated values differ.
A technical reviewer can verify the main workflow without reading every example module.
py -3.12 -m venv .venv
& ".\.venv\Scripts\Activate.ps1"
python -m pip install --upgrade pip
python -m pip install -r requirements-dev.txtpython3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements-dev.txtDetailed setup and troubleshooting are documented in docs/setup.md.
python -m data_quality `
--input "data/raw/training_results.csv" `
--output ".ci-output/data-quality"python -m data_quality \
--input data/raw/training_results.csv \
--output .ci-output/data-qualitypython -m reporting `
--input ".ci-output/data-quality" `
--output ".ci-output/reporting"python -m reporting \
--input .ci-output/data-quality \
--output .ci-output/reportingpython -m pytest
python -m ruff check main.py data_quality reporting examples tests
python -m ruff format --check main.py data_quality reporting examples testsThe committed synthetic fixture contains 15 source rows with deliberately introduced quality problems.
| Control | Verified value |
|---|---|
| Input rows | 15 |
| Accepted rows | 8 |
| Rejected rows | 7 |
| Exact duplicate rows removed | 1 |
| Modules | 4 |
| Overall average score | 70.00% |
| Overall pass rate | 62.50% |
| Distinct rejection reasons | 7 |
| KPI reconciliation | passed |
These numbers are automated control values for the fixture, not claims about real learners or business operations.
The package in data_quality/ implements:
- required-column validation
- missing-value detection
- strict ISO date parsing
- controlled numeric conversion
- positive
max_score - non-negative scores and pass thresholds
- scores not exceeding their maximum
- pass thresholds not exceeding their maximum
- exact duplicate removal
- rejection of conflicting records sharing one result ID
- identifier and whitespace normalisation
Invalid rows are not silently dropped. They remain visible in rejected_results.csv with explicit pipe-separated reason codes.
Accepted records include:
source_rowfor lineage to the raw CSV- normalised identifiers and module names
score_percentage- a non-null Boolean
passed
The complete contract is documented in docs/data-quality-workflow.md.
The package in reporting/ validates the generated files, recalculates KPIs, summarises rejection reasons and writes deterministic reporting outputs.
Fresh SVG files are generated by the workflow during every CI run. The committed reference charts show the expected result for the synthetic fixture.
Detailed reporting behaviour is documented in docs/reporting-notebook.md.
notebooks/reporting_verification.ipynb reads generated workflow outputs rather than embedding a second copy of the source data.
It displays and verifies:
- reconciled module KPIs
- rejection-reason counts
- average-score chart
- pass-rate chart
- expected reporting control totals
- a final verification marker
dataspell_test.ipynb remains a smaller interpreter and package-import check.
Both committed notebooks are kept free of outputs, execution counts, absolute local paths and IDE-specific timestamps. CI executes temporary copies only.
The workflow in .github/workflows/python-quality.yml runs with Python 3.12 on:
- Ubuntu 24.04
- Windows 2025
Each matrix job performs:
- dependency installation
- bytecode compilation
- Ruff lint and format checks
- the complete pytest suite
- the deterministic baseline entry point
- the full data-quality workflow
- source control-total verification
- reporting and KPI reconciliation
- reporting control-total and SVG verification
- the bounded optional ML example
- both notebook executions
- short-lived artifact upload
- a final quality gate
This is a quality-assurance workflow, not a deployment or release pipeline.
python-data-basics/
├── .github/workflows/python-quality.yml
├── data/raw/training_results.csv
├── data_quality/
├── reporting/
├── notebooks/reporting_verification.ipynb
├── docs/
│ ├── assets/
│ ├── data-quality-workflow.md
│ ├── project-status.md
│ ├── reporting-notebook.md
│ └── setup.md
├── examples/
├── tests/
├── dataspell_test.ipynb
├── main.py
├── pyproject.toml
├── requirements.txt
├── requirements-dev.txt
└── README.md
The repository retains a few bounded learning examples without presenting them as the main portfolio result:
examples/01_csv_pandas_basics.py— CSV and grouped pandas operationsexamples/02_json_basics.py— nested JSON normalisationexamples/03_api_request_basics.py— public API request without credentialsexamples/04_ollama_local_api_basics.py— optional localhost-only requestexamples/optional/logistic_regression_basics.py— scikit-learn API demonstration without a model-quality claim
main.py is a deterministic environment and pandas sanity check, separate from the substantive data-quality and reporting workflow.
Only synthetic learning data and public endpoints belong in this repository. The committed fixture contains no real learner, customer or company data.
Excluded content includes:
- virtual environments and caches
.envfiles and credentials- API keys and OAuth tokens
- personal or customer data
- IDE metadata
- executed notebook copies
- generated local workflow outputs
This repository does not claim:
- a production package, ETL platform or semantic model
- streaming or distributed processing
- production orchestration, observability or deployment
- regulatory validation
- an interactive dashboard or Power BI report
- a validated predictive model
- statistical inference from the synthetic fixture
The value is the complete, testable chain from imperfect input through validation, auditable rejection, KPI reconciliation and documented reporting.
sql-server-docker-basics— reproducible SQL Server analytics lab, relational integrity and star schemahamburg-district-data-basics— public Hamburg data and Power BI preparationopen-meteo-germany-weather-ranking— public API-to-CSV scoring workflowflask-country-data-api— validated ingestion, persistence and API delivery
Portfolio overview: datatidehh.github.io/DataTideHH · Website: datatidehh.de · LinkedIn: linkedin.com/in/datatidehh