if (requireNamespace("hvtiRutilities", quietly = TRUE)) {
library("hvtiRutilities")
} else {
pkgload::load_all(export_all = FALSE, helpers = FALSE, quiet = TRUE)
}
#>
#> hvtiRutilities 1.5.9
#>
#> Type hvtiRutilities.news() to see new features, changes, and bug fixes.
#> The Problem: Datasets Drift
A dataset pulled from Epic today is not the same file you will get if you pull it again tomorrow — even when no one intentionally changed anything. Backfilled lab results, corrected ICD codes, and updated patient status all mean that re-running an extraction silently changes the underlying numbers.
The standard defence starts with three habits:
-
Name every dataset by its extract date, such as
cohort_20240115.csvorlabs_20240801.sas7bdat. Never overwrite an existing dated file; a new pull gets a new filename. - Record a SHA-256 checksum the moment a file lands on disk. A single changed byte produces a completely different hash.
- Verify the checksums before every analysis run so that any drift is caught immediately, before results are generated.
For an ordinary file manifest, two functions implement those habits:
| Function | Purpose |
|---|---|
update_manifest() |
Register a dataset (or update its entry) in manifest.yaml
|
verify_manifest() |
Check every registered dataset before an analysis runs |
A registered study keeps each version of its dataset as a dated file, so a later rebuild never replaces the one a paper used. We return to that workflow after the checksum example.
In a Study: Rebuild, Then Update
Most of the time you do not call any of the functions below directly. A study registers its dataset once, and from then on the cycle is two steps.
Rebuild
datasets/built.sas7bdatas often as you need. Jobs are not affected: they read the registered version, a dated parquet such asbuilt_20260915.parquet, andread_built()says that a newer file is waiting.-
When the rebuild is right, register it:
hvtiRutilities::update_manifest()Run it from anywhere in the study. It converts the new file to
built_20261007.parquet, keeps every earlier version, and prints what it did. Commitmanifest.yaml.
Every registered version stays on disk, so a manuscript revision can be reproduced from the version its paper used. The rest of this vignette explains the checksums underneath, and how to record a file outside a study.
Setup
For this vignette we work in a temporary directory that mimics a real project’s datasets/ folder.
datasets_dir <- file.path(tempdir(), "datasets")
dir.create(datasets_dir, showWarnings = FALSE)
manifest_path <- file.path(tempdir(), "manifest.yaml")Registering Datasets
CSV — automatic row count
Generate a synthetic cohort (using generate_survival_data()) and write it as a date-stamped CSV.
set.seed(42)
cohort <- generate_survival_data(n = 200, seed = 42)
cohort_file <- file.path(datasets_dir, "cohort_20240115.csv")
write.csv(cohort, cohort_file, row.names = FALSE)Register it in the manifest. Row count is detected automatically from the CSV header; only the SHA-256 is computed from the raw bytes.
update_manifest(
file = cohort_file,
manifest_path = manifest_path,
extract_date = "2024-01-15",
source = "Epic EMR, query v4.2, ICD mapping v3.2",
sort_key = "ccfid"
)Add a second file — a simulated labs extract pulled on the same date.
set.seed(7)
labs <- data.frame(
ccfid = cohort$ccfid,
hgb = round(rnorm(200, mean = 13.5, sd = 1.8), 1),
egfr = round(rnorm(200, mean = 72, sd = 18), 0),
bnp = round(exp(rnorm(200, mean = 4.8, sd = 0.9)), 0)
)
labs_file <- file.path(datasets_dir, "labs_20240115.csv")
write.csv(labs, labs_file, row.names = FALSE)
update_manifest(
file = labs_file,
manifest_path = manifest_path,
extract_date = "2024-01-15",
source = "Epic EMR, labs module — Hgb, eGFR, BNP",
sort_key = "ccfid"
)Inspecting the manifest
The manifest is a plain YAML file committed to version control alongside your analysis scripts (but not alongside the data files, which stay out of git).
cat(paste(readLines(manifest_path), collapse = "\n"))
#> datasets:
#> - file: cohort_20240115.csv
#> extract_date: '2024-01-15'
#> n_rows: 200
#> sha256: f3db9dd8a47765003a2509c54068f8736b0fd8c2f0b0425808422cc11f0bdfcd
#> role: source
#> source: Epic EMR, query v4.2, ICD mapping v3.2
#> sort_key: ccfid
#> - file: labs_20240115.csv
#> extract_date: '2024-01-15'
#> n_rows: 200
#> sha256: 035acf2e981692554ba8e2c07f8b61f9d6ceb834343612d50d5487aed062b380
#> role: source
#> source: Epic EMR, labs module — Hgb, eGFR, BNP
#> sort_key: ccfidEach entry records exactly what is needed to re-verify the file later: extract_date, n_rows, sha256, and any provenance notes you supply.
SAS datasets — automatic row count
Files exported directly from SAS (.sas7bdat) are handled the same way. update_manifest() calls haven::read_sas() internally to count rows; the checksum is always computed from the raw file bytes.
update_manifest(
file = here::here("datasets", "cohort_20240115.sas7bdat"),
manifest_path = here::here("manifest.yaml"),
extract_date = "2024-01-15",
source = "SAS CORR registry extract, query v3.1",
sort_key = "pat_id"
)Excel workbooks — automatic row count
Adjudication logs, crosswalk tables, and other curator-maintained datasets often arrive as .xlsx files. update_manifest() calls readxl::read_excel() to count rows.
update_manifest(
file = here::here("datasets", "adjudication_20240115.xlsx"),
manifest_path = here::here("manifest.yaml"),
extract_date = "2024-01-15",
source = "Clinical Events Committee adjudication log, v2"
)Other formats — supply n_rows explicitly
For RDS, Feather, Parquet, or any format not listed above, pass n_rows directly. The SHA-256 is still computed from the raw bytes.
update_manifest(
file = here::here("datasets", "cohort_20240115.rds"),
manifest_path = here::here("manifest.yaml"),
extract_date = "2024-01-15",
n_rows = nrow(readRDS(here::here("datasets", "cohort_20240115.rds"))),
source = "RDS cached from Epic pull"
)Verifying Before Every Analysis
Place verify_manifest() at the very top of every analysis script and Quarto document — before any data are loaded and before any results are produced.
verify_manifest(
manifest_path = manifest_path,
data_dir = datasets_dir
)The function prints one confirmation line per file and returns invisibly. If everything is clean, your script continues.
Typical script header
library(hvtiRutilities)
library(here)
# Abort immediately if any registered dataset has changed
verify_manifest(here("manifest.yaml"))
# Only reached if all checksums pass
cohort <- read.csv(here("datasets", "cohort_20240115.csv"))
labs <- read.csv(here("datasets", "labs_20240115.csv"))What Happens When Data Changes
Suppose the data warehouse silently updates cohort_20240115.csv overnight. verify_manifest() catches it the moment the next analysis run starts.
# Simulate a silent dataset change
cohort_modified <- cohort
cohort_modified$age[1] <- cohort_modified$age[1] + 1L
write.csv(cohort_modified, cohort_file, row.names = FALSE)
# verify_manifest stops immediately
verify_manifest(
manifest_path = manifest_path,
data_dir = datasets_dir
)
#> Error:
#> ! STOP: manifest verification failed for:
#> cohort_20240115.csv: SHA-256 mismatch
#> expected: f3db9dd8a47765003a2509c54068f8736b0fd8c2f0b0425808422cc11f0bdfcd
#> actual: 5557af305fedf53ed5ee74c6a0aace29e4974a549a2a44fc5ff8c173d40a4e1a
#> If this file was rebuilt on purpose, run hvtiRutilities::update_manifest() to register the new version.The error message names the affected file and shows both the expected and actual SHA-256 values, giving you an unambiguous audit trail.
Collecting all failures at once
During development it can be useful to see every problem before fixing any of them. Set stop_on_error = FALSE to collect warnings instead of stopping.
report <- verify_manifest(
manifest_path = manifest_path,
data_dir = datasets_dir,
stop_on_error = FALSE
)
#> Warning: STOP: manifest verification failed for:
#> cohort_20240115.csv: SHA-256 mismatch
#> expected: f3db9dd8a47765003a2509c54068f8736b0fd8c2f0b0425808422cc11f0bdfcd
#> actual: 5557af305fedf53ed5ee74c6a0aace29e4974a549a2a44fc5ff8c173d40a4e1a
#> If this file was rebuilt on purpose, run hvtiRutilities::update_manifest() to register the new version.
report[report$status == "FAIL", c("file", "message")]
#> file
#> 1 cohort_20240115.csv
#> message
#> 1 SHA-256 mismatch\n expected: f3db9dd8a47765003a2509c54068f8736b0fd8c2f0b0425808422cc11f0bdfcd\n actual: 5557af305fedf53ed5ee74c6a0aace29e4974a549a2a44fc5ff8c173d40a4e1a\n If this file was rebuilt on purpose, run hvtiRutilities::update_manifest() to register the new version.Correcting a file
A changed checksum tells you that the bytes moved. It does not tell you which bytes were correct, and an already overwritten file cannot be reconstructed from its checksum. Keep the mutable work under a working name while the statistical programmer builds and validates it, and register it as a dated file only when it is right. Registering is the boundary: a dated file is not rewritten afterwards.
The rest of this vignette still needs its original temporary file. This reset is safe only because the fixture is disposable; it is not a recovery method for a study whose registered file was overwritten.
write.csv(cohort, cohort_file, row.names = FALSE)Looking at a Rebuild Before Registering It
After a rebuild, jobs keep reading the registered version. Before you register the new file, review_built() shows how it differs from the version jobs read. It reads both files and writes nothing.
When the working file is unchanged since the last registration, it says so. When it differs, it prints the comparison of rows, columns, types and labels and points to update_manifest(), which registers the file as the next version.
hvtiRutilities::update_manifest()review_built() describes how the data changed. It does not certify that the new file is correct; that judgement is yours.
Running a Job on the Working File
Sometimes a job needs the rebuilt data before it has a dated version, for example a descriptive report you want to look at while the build is still being checked. read_built(working = TRUE) reads the working file, and so does setting HVTI_DATA=working for the render.
HVTI_DATA=working quarto render 10_descriptive/table1.qmdAny read that is not of a dated version is a working read, whether or not you asked for one. It is allowed in the interactive console and in jobs in the study’s datasets and descriptive folders. In distribution, analysis and graph jobs it stops with a message that says to register the dataset or unset HVTI_DATA. A working read signals hvtiRutilities_working_read, a message that says the results are a draft. Treat the report that comes out as a draft too.
A study still on a legacy registration, one with no dated version, has only its working file to read. Its analysis jobs stop until update_manifest() converts it to a dated version.
How a Version Was Produced
Each registered version records where its data came from, under a provenance: map in manifest.yaml keyed by the parquet name.
-
origin: bdmeans a build left a build record,<stem>.build.yml, beside the working file, and that record describes exactly the file registered.update_manifest()moves it next to the dated parquet and the manifest notes its name and checksum, with the job, view and as-of date it carries. -
origin: legacymeans there was no such record. Every version registered before this scheme reads as legacy.
verify_manifest() checks each build record against its recorded checksum. The record is a file beside the dataset on the share. It is not committed: only manifest.yaml goes into Git.
A New Pull for a Different Project
When the same registry is pulled again months later for a different analysis, it gets a new filename — never overwriting the original.
set.seed(99)
cohort_aug <- generate_survival_data(n = 315, seed = 99)
cohort_aug_file <- file.path(datasets_dir, "cohort_20240801.csv")
write.csv(cohort_aug, cohort_aug_file, row.names = FALSE)
update_manifest(
file = cohort_aug_file,
manifest_path = manifest_path,
extract_date = "2024-08-01",
source = "Epic EMR, query v4.2, ICD mapping v3.3 — LVAD sub-study cohort",
sort_key = "ccfid"
)The manifest now tracks all three files independently.
m <- yaml::read_yaml(manifest_path)
do.call(rbind, lapply(m$datasets, function(e) {
data.frame(file = e$file, extract_date = e$extract_date,
n_rows = e$n_rows, stringsAsFactors = FALSE)
}))
#> file extract_date n_rows
#> 1 cohort_20240115.csv 2024-01-15 200
#> 2 labs_20240115.csv 2024-01-15 200
#> 3 cohort_20240801.csv 2024-08-01 315Verification covers all of them in a single call.
verify_manifest(
manifest_path = manifest_path,
data_dir = datasets_dir
)What Changed Inside the File?
The manifest answers one question, and it answers it absolutely: did this file change? A single altered byte flips the checksum. But a failed verify_manifest() tells you nothing about what moved, and that is the question you actually have to answer before you decide whether the change is benign.
compare_datasets() answers it. Give it the two versions and it reports the four kinds of drift that matter:
cohort_v1 <- generate_survival_data(n = 200, seed = 42)
cohort_v2 <- generate_survival_data(n = 315, seed = 99)
# Changes a real re-pull might bring
cohort_v2$creatinine <- 1.1
cohort_v2$reop <- NULL
cohort_v2$dead <- as.integer(cohort_v2$dead) > 0
labelled::var_label(cohort_v2$gfr_bs) <- "eGFR at discharge (mL/min/1.73m2)"
compare_datasets(cohort_v1, cohort_v2)
#> Dataset Comparison
#> Rows: 200 -> 315
#> Columns added: creatinine
#> Columns dropped: reop
#> Type changes:
#> dead: integer -> logical
#> Label changes:
#> dead: 'Death indicator (1=dead, 0=censored)' -> 'dead'
#> gfr_bs: 'Baseline eGFR (mL/min/1.73m2)' -> 'eGFR at discharge (mL/min/1.73m2)'Read that output in order of how much trouble each line will cause you.
Added and dropped columns are the easy ones; a model referencing reop will fail loudly on the next run. The type change on dead is worse, because integer to logical is a change most code will accept silently. Notice it also cost the column its label, which is how a type change usually announces itself in the audit.
The label change on gfr_bs is the one to watch. Baseline eGFR and eGFR at discharge are different measurements under the same column name, and nothing but the label will tell you they were swapped. Checksums catch it as “changed”; only the label comparison tells you the variable now means something else.
The result is a list, so a script can act on it rather than print it:
drift <- compare_datasets(cohort_v1, cohort_v2)
drift$cols_dropped
#> [1] "reop"
drift$label_changes
#> variable old_label
#> 1 dead Death indicator (1=dead, 0=censored)
#> 2 gfr_bs Baseline eGFR (mL/min/1.73m2)
#> new_label
#> 1 dead
#> 2 eGFR at discharge (mL/min/1.73m2)Use the two tools together: verify_manifest() at the top of the script to stop the run, then compare_datasets() to work out why. See PROC CONTENTS and PROC MEANS in R for more on inspecting an extract once you know it moved.
Policy Recommendations
| Stage | Action |
|---|---|
| While rebuilding | Work on the working file; jobs keep reading the registered version |
| When the rebuild looks right | Run review_built(), then update_manifest(); commit manifest.yaml
|
| Top of every analysis script | Call verify_manifest() — before loading any data |
| At manuscript submission | Tag the git commit; freeze manifest.yaml; do not re-pull data |
| For an unregistered legacy file | Use update_manifest() to record its current bytes; a checksum cannot recover an overwritten version |
.gitignore |
Add datasets/ — commit the manifest, not the data |
Session Information
sessionInfo()
#> R version 4.6.1 (2026-06-24)
#> Platform: x86_64-pc-linux-gnu
#> Running under: Ubuntu 24.04.5 LTS
#>
#> Matrix products: default
#> BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
#> LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so; LAPACK version 3.12.0
#>
#> locale:
#> [1] LC_CTYPE=C.UTF-8 LC_NUMERIC=C LC_TIME=C.UTF-8
#> [4] LC_COLLATE=C.UTF-8 LC_MONETARY=C.UTF-8 LC_MESSAGES=C.UTF-8
#> [7] LC_PAPER=C.UTF-8 LC_NAME=C LC_ADDRESS=C
#> [10] LC_TELEPHONE=C LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
#>
#> time zone: UTC
#> tzcode source: system (glibc)
#>
#> attached base packages:
#> [1] stats graphics grDevices utils datasets methods base
#>
#> other attached packages:
#> [1] hvtiRutilities_1.5.9
#>
#> loaded via a namespace (and not attached):
#> [1] vctrs_0.7.3 cli_3.6.6 knitr_1.52 rlang_1.3.0
#> [5] xfun_0.62 otel_0.2.0 forcats_1.0.1 haven_2.5.5
#> [9] generics_0.1.4 jsonlite_2.0.0 glue_1.8.1 htmltools_0.5.9
#> [13] hms_1.1.4 rmarkdown_2.32 evaluate_1.0.5 tibble_3.3.1
#> [17] fastmap_1.2.0 yaml_2.3.12 lifecycle_1.0.5 compiler_4.6.1
#> [21] dplyr_1.2.1 pkgconfig_2.0.3 labelled_2.16.1 digest_0.6.39
#> [25] R6_2.6.1 tidyselect_1.2.1 pillar_1.11.1 magrittr_2.0.5
#> [29] tools_4.6.1