Export metadata and feature data tables by epbrenner · Pull Request #34 · JRaviLab/amRdata

epbrenner · 2026-07-28T22:22:02Z

Testing:

For this PR, I have included bundled test data in /inst/extdata/Campy_testdata.zip. Extract this into a single directory of your choosing. This bundled data is not intended to the ultimate example data for final package release.

For data_curation.R:
Run
prepareGenomes("Staphylococcus argenteus")
then
exportTables("data/Staphylococcus_argenteus/Sar.duckdb")

For data_processing.R:
Unzip the Campy test data first, then run
exportProcessedData("inst/extdata/Campy_testdata/Cco_parquet.duckdb")

There are several thrilling parameters to fiddle around with.

What this does:

Adds the ability to export basic stats into human readable CSVs from data_curation.R workflow.
Adds the ability to export all the processed feature data into a variety of formats from data_processing.R workflow.

What's the point?

Makes amRdata more independently useful to people even if they don't run the full amR suite. For example, you could export feature matrices for custom modeling workflows outside of ours, or just get an idea of what distributions of different QC and AST labels are across different taxa.

In data_curation.R, adding ability to export genome stats and QC metadata into human readable CSVs. `load_tables` allows you to run "example <- exportTables("data/Staphylococcus_aureus/Sau.duckdb", load_tables=TRUE)" and import the tables into R so you can play around with them directly. By default, it just saves the tables as exported CSVs. In data_processing.R, added a similar exporter that allows you to save out all the feature data and other DuckDB-connected tables, with the ability to output sequences too, and append AST phenotype data to the feature tables. Tables can be saved out as csv (default), tsv, Parquet, and xlsx. Added `writexl` to Suggests in DESCRIPTION. Fixed some reversions where I broke things in the PR this branch is based on. Will add those same fixes to that PR momentarily to keep things aligned.

Ran data processing workflow for Campylobacter coli. This can be unzipped and used as the test data for exportProcessedData() rather than having to run it all beforehand. Data curation step still requires that prepareGenomes() be run for a test bug, and Staphylococcus argenteus is a good use case for this.

jananiravi

Few quick comments. Will test it out tomorrow.

Clarified a few names based on code review feedback.

jananiravi · 2026-07-29T20:16:49Z

Few quick comments. Will test it out tomorrow.

@epbrenner prepareGenomes + exportTables --> they run locally with Sar.

jananiravi · 2026-07-29T20:21:50Z

After running the campy --> metadata.csv

what's TET doing in resistant_classes? genome_drug.resistance_phenotype marks them as susceptible. @epbrenner @AbhirupaGhosh

AbhirupaGhosh · 2026-07-29T20:27:42Z

After running the campy --> metadata.csv
what's TET doing in `resistant_classes`? genome_drug.resistance_phenotype marks them as susceptible. @epbrenner @AbhirupaGhosh

resistant_classes is a column that adds all the drug classes the genome is resistant to. If there are more than 1, say TET and FLQ, it will show TET_FLQ and it shows susceptible to the genomes that are resistant to none.

jananiravi

is this going to be merged to qc_filtering and then qc_filtering --> main? that might be helpful to avoid duplicate reviews.

jananiravi · 2026-07-29T20:38:59Z

+      count_if_present("metadata"),
+      count_if_present("genome_data"),
+      count_if_present("amr_phenotype"),
+      count_if_present("metadata_qc"),
+      count_if_present("metadata_qc_rejections"),
+      count_if_present("filtered"),
+      count_if_present("files"),
+      count_if_present("bac_data")


how do these checks tie back to basic_tables? is the latter not exported? so should "files" be added to basic_tables?

Refactored to use a default_tables list rather than duplicating this way. There was logic for the checks, but upon careful review, the logic belonged in the garbage.

eboyer221

When I tested this, it looks like genome IDs such as 263.1, 263.10, and 263.100 are being written to the CSV as plain numbers rather than text, since they happen to look numeric. When I read that export back (with readr::read_csv(), pandas, or Excel's defaults) all three collapsed to the same value, since trailing zeros after a decimal don't count (263.1 and 263.10 parse as numerically identical). So the three distinct genomes merged under one ID, and it'll happen for any taxon with more than ~9 genomes. I also tried forcing quotes around every value (quote = "all") to see if that would prevent it, but the collision still happened on read. Quoting only protects against commas breaking the file format, it doesn't tell the reader "treat this as text."

This affects every genome_id-keyed table both export functions produce (genome_data, metadata, amr_phenotype_wide, gene_count, etc.).
Two options I see: (1) add a clear warning telling users to force col_types/dtype=str on ID columns when reading CSV/TSV back, or (2) recommend Parquet as the default/safe export format for programmatic reuse, since arrow preserves the real column type - I verified this did not occur with Parquet.

epbrenner

Oh I have to submit my own review to my own PR to actually post these comments, huh? Oops.

jananiravi · 2026-07-29T21:29:20Z

When I tested this, it looks like genome IDs such as 263.1, 263.10, and 263.100 are being written to the CSV as plain numbers rather than text, since they happen to look numeric. When I read that export back (with readr::read_csv(), pandas, or Excel's defaults) all three collapsed to the same value, since trailing zeros after a decimal don't count (263.1 and 263.10 parse as numerically identical). So the three distinct genomes merged under one ID, and it'll happen for any taxon with more than ~9 genomes. I also tried forcing quotes around every value (quote = "all") to see if that would prevent it, but the collision still happened on read. Quoting only protects against commas breaking the file format, it doesn't tell the reader "treat this as text."

This affects every genome_id-keyed table both export functions produce (genome_data, metadata, amr_phenotype_wide, gene_count, etc.).
Two options I see: (1) add a clear warning telling users to force col_types/dtype=str on ID columns when reading CSV/TSV back, or (2) recommend Parquet as the default/safe export format for programmatic reuse, since arrow preserves the real column type - I verified this did not occur with Parquet.

@epbrenner @AbhirupaGhosh didnt we have this fix in already?

Co-authored-by: Emily Boyer <130874527+eboyer221@users.noreply.github.com>

epbrenner · 2026-07-31T22:14:35Z

When I tested this, it looks like genome IDs such as 263.1, 263.10, and 263.100 are being written to the CSV as plain numbers rather than text, since they happen to look numeric. When I read that export back (with readr::read_csv(), pandas, or Excel's defaults) all three collapsed to the same value, since trailing zeros after a decimal don't count (263.1 and 263.10 parse as numerically identical). So the three distinct genomes merged under one ID, and it'll happen for any taxon with more than ~9 genomes. I also tried forcing quotes around every value (quote = "all") to see if that would prevent it, but the collision still happened on read. Quoting only protects against commas breaking the file format, it doesn't tell the reader "treat this as text."
This affects every genome_id-keyed table both export functions produce (genome_data, metadata, amr_phenotype_wide, gene_count, etc.).
Two options I see: (1) add a clear warning telling users to force col_types/dtype=str on ID columns when reading CSV/TSV back, or (2) recommend Parquet as the default/safe export format for programmatic reuse, since arrow preserves the real column type - I verified this did not occur with Parquet.

@epbrenner @AbhirupaGhosh didnt we have this fix in already?

Yeah, relatively easy solution, just requires special handling of the columns of interest. Patched in the latest commit.

with `devtools::document()`

jananiravi

minor edits, comments added. yet to run line-by-line; that's the reason I didn't approve yet, but looks good.

jananiravi · 2026-08-02T00:57:36Z

+        genome_id = as.character(`genome.genome_id`),
+        antibiotic = as.character(`genome_drug.antibiotic`),
+        phenotype = as.character(`genome_drug.resistant_phenotype`)


Need this special treatment for anything else? accessions that end in a.x as well? (all of them?)

jananiravi · 2026-08-02T01:01:03Z

+exportTables <- function(duckdb_path,
+                         output_dir = NULL,
+                         tables = NULL,
+                         skip_tables = c(NULL),


Suggested change

skip_tables = c(NULL),

skip_tables = NULL,

jananiravi · 2026-08-02T01:02:35Z

+    stop("No tables left to export after applying skip_tables.")
+  }
+
+  preserve_id_text <- function(df) {


I think the two preserve_id_text function definitions aren't identical. Can you check why there are two and if they match? One would be future-proof.

jananiravi · 2026-08-02T01:03:06Z

+  }
+
+  # Save trailing zeroes
+  preserve_id_text <- function(df) {


one more present!

jananiravi · 2026-08-02T01:05:33Z

    "-o", output_dir_cont,
    "--clean-mode", "strict",
    "--merge_paralogs",
+    "--refind-mode off",


this will eventually inherit the many off-related changes downstream, I suppose. 🤔

epbrenner added 2 commits July 28, 2026 13:22

epbrenner requested review from AbhirupaGhosh, amcim, eboyer221 and jananiravi July 28, 2026 22:22

epbrenner self-assigned this Jul 28, 2026

epbrenner added the enhancement New feature or request label Jul 28, 2026

jananiravi reviewed Jul 29, 2026

View reviewed changes

Updated variable/function names

5a944a9

Clarified a few names based on code review feedback.

jananiravi previously approved these changes Jul 29, 2026

View reviewed changes

Merge branch 'main' into export_tables

6ecae92

jananiravi dismissed their stale review via 6ecae92 July 29, 2026 20:17

jananiravi reviewed Jul 29, 2026

View reviewed changes

eboyer221 requested changes Jul 29, 2026

View reviewed changes

Comment thread R/data_curation.R Outdated

epbrenner commented Jul 29, 2026

View reviewed changes

Comment thread R/data_curation.R

Comment thread R/data_curation.R Outdated

Comment thread R/data_curation.R Outdated

Comment thread R/data_curation.R Outdated

Comment thread R/data_curation.R Outdated

Comment thread R/data_curation.R Outdated

Comment thread R/data_curation.R Outdated

epbrenner and others added 2 commits July 31, 2026 13:55

Apply suggestion from @eboyer221

c082308

Co-authored-by: Emily Boyer <130874527+eboyer221@users.noreply.github.com>

Code review fixes

66038e3

epbrenner requested a review from eboyer221 July 31, 2026 22:18

jananiravi added 2 commits August 1, 2026 18:02

Merge branch 'main' into export_tables

7a59169

regenerated namespace and Rd files

e4acc7b

with `devtools::document()`

jananiravi reviewed Aug 2, 2026

View reviewed changes

Uh oh!

Conversation

epbrenner commented Jul 28, 2026

Testing:

What this does:

What's the point?

Uh oh!

jananiravi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jananiravi commented Jul 29, 2026

Uh oh!

jananiravi commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AbhirupaGhosh commented Jul 29, 2026

Uh oh!

jananiravi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jananiravi Jul 29, 2026

Choose a reason for hiding this comment

Uh oh!

epbrenner Jul 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

eboyer221 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

epbrenner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jananiravi commented Jul 29, 2026

Uh oh!

epbrenner commented Jul 31, 2026

Uh oh!

jananiravi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jananiravi Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

jananiravi Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

jananiravi Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

jananiravi Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

jananiravi Aug 2, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

jananiravi commented Jul 29, 2026 •

edited

Loading

jananiravi left a comment •

edited

Loading