Read a job's data, keep its rows, and record what was done

The shared data step of every analysis template. It reads a registered dataset (or an hvtiRdatabuild analysis set), resolves the patient identifier, drops the medical record number columns, keeps the rows where selects and checks that rows are unique on key.

Usage

read_job_data(
  cfg,
  dataset = "study",
  analysis_set = NULL,
  where = NULL,
  id = "ccfid",
  key = NULL,
  join = NULL,
  join_vars = NULL,
  reduce = NULL,
  join_key = NULL,
  one_row_per_patient = FALSE,
  imputation = NULL
)

Arguments

cfg: Study configuration, from study_config.
dataset: Name of a dataset registered in _study.yml; "built" and "study" both name the study dataset, and the selection records "study".
analysis_set: Name of an analysis set written by hvtiRdatabuild::write_analysis_set(), or NULL to read dataset whole. Analysis sets derive from the study dataset only.
where: Rows to keep: NULL, one condition from quote(), or a list from rlang::exprs(), all of which must hold. Conditions follow dplyr::filter(): a row where a condition is NA is dropped. A value from outside the data, written .env$min_age or as a name that is not a column, is fixed into the condition when the data are read, so the recorded condition rebuilds the same rows wherever it runs. Only a plain name and .env$x are fixed in; x$y and x@y are kept as written, and a name for a function stays a name. Such a value must be a vector, NULL, a symbol or a call; a data frame, list, environment or S4 object stops, since it would be saved whole. A condition that mentions the id column or a column named MRN or eMRN (ignoring case), directly or as .data$x or .data[["x"]], stops before any row is filtered, because each condition is saved, values included, in the job's output. Exclude those patients in the dataset build, or with an hvtiRdatabuild analysis set, instead. A value from outside the data is fixed in before this check, so .data[[nm]] is judged by the column nm names. A column reached any other way, as by .data[[paste0(...)]] or a string lookup such as get(), stops too, since it cannot be known. A function bound to such a lookup under another name is read as that lookup. Values are checked as well as names, after every name check: the check covers literal values (including a vector fixed in from outside) of five or more characters that equal a value of the identifier, MRN or eMRN columns in the data, and folds only arithmetic, c(), paste/paste0 and numeric or character coercion of constants (such as 4730000000 + 1) to check their result. So a wrapper function or a copy of the identifier under another name is caught when it is compared with such a value. The check never calls any other function: a value produced by any other call, or computed from a data column, as in id2 / 2 != 2365000000.5, is not checked. A threshold that happens to equal a patient's identifier is refused too, and the message says so; numbers are compared as whole numbers where they are whole, and NA never matches. Thresholds shorter than five characters, such as 1, 18 or 2015, are not checked, so a study keyed on shorter identifiers relies on the name checks.
id: The patient identifier column. When it is the default "ccfid" and absent, MRN and then eMRN are used.
key: Columns that make a row unique. NULL (the default) uses the key registered for dataset, and id, one row per patient, when none is registered. Add a visit time or date for repeated measures. A key that differs from the registered one is noted in the record. Templates pass their KEY, a study choice the analyst reviews, so the default serves a direct call. With join, the cohort must be one row per patient. A key of cohort columns is checked on the cohort, and the result is keyed as reduce says; a key that names a joined column, such as a visit time only the joined dataset carries, is checked on the joined rows and keys the result.
join: Name of one registered ancillary dataset (such as echoes or labs) to join to the cohort on id, or NULL. The cohort, dataset or analysis_set, decides the patients: joined records of other patients are dropped and counted. A dataset registered with a kind other than "ancillary" is refused.
join_vars: Cohort columns each joined row carries, the id always among them; NULL carries all of them. A column both datasets have stops, so list only the cohort columns the job needs.
reduce: NULL keeps one row per joined record, keyed on the joined dataset's key. list(rule = "first", by = "echo_date") keeps one row per cohort patient, keyed on id: the record with the smallest by ("first"), the largest ("last"), or the one nearest a cohort column (rule = "nearest" with to = "dt_surg"). A patient with no record keeps the row with the joined columns missing. by may name several columns, such as c("echo_date", "echo_seq"): each later one breaks a tie on those before it, in the same direction, and a missing tie-break value loses the tie. Records with no value in the first by column are left out and counted, and a patient whose chosen record still ties with another stops the read, since picking one would be a hidden choice.
join_key: Columns that make the joined dataset's rows unique, overriding its registered key. A join needs one or the other.
one_row_per_patient: TRUE for a job that models one row per patient, such as a hazard, logistic or random forest fit. Such a job stops on a join without reduce, before any data are read, since the long form would count every joined record as a patient, and on rows kept that repeat a patient, as a dataset of repeated records read whole would, after the rows are selected. FALSE, the default, accepts the long form, as descriptive jobs and jobs that model repeated measures need.
imputation: NULL, the default, or the name of the column that numbers the imputations of stacked multiple imputations, one row per patient per imputation. With one_row_per_patient = TRUE the rows are then counted within each imputation, so a patient who appears once in every imputation passes, and one who appears twice in the same imputation still stops the read. The column must be in the data read, and in key for the rows to be unique. It is matched ignoring case, as key is. It must hold no missing value and must not be the patient identifier. The record says how many imputations the rows hold. It is not inferred from key, since a visit column in key makes a long dataset, not an imputation.

Value

A list:

data, the selected rows;
record, a data frame of step and value to print, whose "selection" attribute holds the settings used. Its "Source" row names the file read, the dated version for a dataset registered as one. When the dataset's source has been rebuilt since it was registered, the registered version is read and a final "Note" row says so, naming hvtiRutilities::update_manifest(), the call that registers the new file. A stale analysis set read in a draft gets a "Note" row the same way. With join, rows name the joined dataset and the rows read from it, count its records outside the cohort and the cohort patients with none, and name the reduction;
provenance, the read's provenance record;
provenance_join, the joined dataset's provenance record, naming the version read, or NULL without join;
attrition, an analysis set's per-rule attrition table, or NULL for a dataset read whole.

The "selection" attribute is a list that a job saves in its hand-off, so a downstream job can rebuild the same rows:

dataset, with "built" recorded as "study", and analysis_set, as given;
where, the exact text of each condition, values included, used to rebuild the rows; it stays inside the study and is never printed;
where_shown, the same conditions as a report may show them, with the values of any condition on a key column, or using .data or a string lookup, replaced;
id and key, the resolved column names, the key being the joined result's when there is a join;
imputation, the resolved name of the imputation column in the data's own spelling, present only when imputation was given, so a job can pass it on to a fit;
join, join_vars, reduce and join_key, the join as read, with join_vars, join_key and reduce's columns spelled as the data spell them and reduce's fields in name order, so the same join written two ways records the same; and cohort_key, the cohort's own key. All five are present only with a join;
rows and patients, the counts kept;
key_hash, a SHA-256 hash of the kept key values, with a join the joined dataset's key values too, so a downstream job can tell that it rebuilt the same patients, and with reduce the same chosen records, and not only the same counts.

Details

Direct identifiers are dropped unless one is the identifier, from a joined dataset too: columns named, ignoring case, ccfid, MRN, eMRN, a Social Security number (ssn, soc_sec, social_security, pt_ssn), and, held as text, a patient name (name, first_name, pt_name, surname) or contact details (address, street, phone, email). Any other column whose values are shaped like Social Security numbers is dropped too, and the record notes it by name; a column named as a date or time is not read for that shape. A KEY column is never dropped. The record lists every column dropped. An explicit id or key matches its column ignoring case, because hvtiRutilities::read_built() lowercases column names. Identifier, key and date values are not printed: the record holds counts, and a where condition that mentions a key column, uses .data or calls a string lookup such as get() is shown, in the record and in error messages, with its values replaced by <value>. So is a condition on id in a selection saved before such conditions were refused. Every setting is checked before the data are read, and join's registration too; only what needs the data, such as whether a column exists, is checked after. With join, where applies to the joined rows, so a condition may name a column from either dataset, though not a cohort column join_vars leaves out; the joined dataset's identifier values are refused in it as the cohort's are. With reduce, a condition that names a column of the joined dataset filters its records before one is chosen per patient, so rule = "last" with echo_type == "TTE" keeps each patient's last such echo, and a patient left with no record is counted as having none. The other conditions filter the reduced rows. The data table shows them in that order; the selection records them as written.

Examples

# \donttest{
root <- file.path(tempdir(), "job-data-example")
dir.create(root)
hvtiRutilities::study_setup(root, "Example", 1L, adopt = TRUE, cleanup = FALSE, renv = FALSE)
#> Study: /tmp/RtmpcLEZJZ/job-data-example
#> 
#> [x] _study.yml — study: Example
#> [ ] renv.lock — no renv.lock; run renv::init() in the study project
#> [ ] manifest.yaml — no manifest.yaml; register_data() creates it
#> [ ] dataset — no default dataset registered; run register_data()
#> [ ] provenance — no .qmd/.Rmd sources found; 0 sidecars
#> 
#> 0 .R  |  0 .qmd/.Rmd  |  0 .sas  |  0 provenance sidecars
d <- data.frame(ccfid = 1:4, age = c(15, 40, 55, 70))
utils::write.csv(d, file.path(hvtiRutilities::study_dir("datasets", root), "built.csv"),
                 row.names = FALSE)
hvtiRutilities::register_data(root, "built.csv")
#> Study: /tmp/RtmpcLEZJZ/job-data-example
#> 
#> [x] _study.yml — study: Example
#> [ ] renv.lock — no renv.lock; run renv::init() in the study project
#> [x] manifest.yaml — 1 dataset entry verified by checksum
#> [x] dataset — built.csv, registered as built_20261022.parquet
#> [ ] provenance — no .qmd/.Rmd sources found; 0 sidecars
#> 
#> 0 .R  |  0 .qmd/.Rmd  |  0 .sas  |  0 provenance sidecars
cfg <- hvtiRutilities::study_config(start = root)
job <- read_job_data(cfg, where = quote(age >= 18))
job$record
#>                  step                                    value
#> 1              Source dataset `study` (built_20261022.parquet)
#> 2           Rows read                                        4
#> 3                  ID                                  `ccfid`
#> 4 Identifiers dropped                                     none
#> 5         `age >= 18`                                removed 1
#> 6           Rows kept                     3 rows on 3 patients
unlink(root, recursive = TRUE)
# }