Skip to contents

Changelog

Source: NEWS.md

ducklake (development version)

ducklake 0.9.0

  • New view_table_changes() opens a table’s change feed in an interactive viewer, an htmlwidget. A sidebar lists the schema changes, the columns and cells that changed, the rows inserted, updated, and deleted, and the snapshots in range (each with its author and message), with a count on every entry. The main panel shows the selected part as a table. An update is one row, shown from its old or its new side, with the changed cells highlighted and both values on hover, and a cells view lists every change as snapshot, rowid, column, old, new. It takes the lazy feed from get_table_changes(), so dplyr filters run in DuckDB before anything is collected, or that feed collected into a data frame. The layout follows Hadley Wickham’s data-diff (https://github.com/hadley/data-diff), with thanks. htmlwidgets joins Suggests, with jsonlite for the viewer’s tests.

  • get_table_changes() defaults start and end to the table’s full history, and its result carries a ducklake_changes attribute (table, lake, range) that view_table_changes() reads. The attribute survives dplyr verbs on the lazy table and is dropped by collect().

  • New create_check() and run_checks() (experimental) keep data checks in the lake. A check is a view, in a schema of its own, that returns the rows breaking a rule, with the rule’s label as its comment, so the rules live in the catalog, are versioned with the data, and run from any client of the lake. create_check() takes the rule as it is said (cyl %in% c(4, 6, 8), dose != 0), keeps the rows where it is false, and writes the schema, the view, and the label as one snapshot; a rule that is easier to state as its failure is a pipeline into create_view(). run_checks() returns one row per check: its label and how many rows fail. Inside with_transaction() the counts include pending writes, so a load that fails a check can be rolled back before it commits, and on a snapshot-pinned attach the rules and the data are both read as of that snapshot. vignette("data-checks") walks through it and sets the approach next to affirm and data-dict. DuckLake enforces NOT NULL and nothing else, and its catalog takes no custom tags from SQL, which is why the rules are views.

  • set_table_comment() comments a view as well as a table. It used to send COMMENT ON TABLE for every name, which DuckLake refuses for a view, so get_table_comments() could read a view’s comment but nothing in the package could write one.

  • create_view() keeps a view’s comment when it replaces the view. DuckLake stores the replacement as a new catalog entry and keys the comment to the entry, so CREATE OR REPLACE VIEW by itself drops it. create_view() reads the comment first and sets it again in the same snapshot, as replace_table() does for a table’s comments. A check’s label is its view’s comment, so revising a rule keeps the label.

  • get_table_comments() reads comments through DuckDB’s catalog functions (duckdb_tables(), duckdb_views(), duckdb_columns()) instead of the DuckLake metadata tables. It used to return the lake’s latest committed comments whatever the session was looking at, which was wrong in two situations. On a lake attached with snapshot_version or snapshot_time it returned present-day comments, so collect() restored labels that were not in force at that snapshot. Inside an open transaction it missed the comments that transaction had set, so a replace_table() there put the older committed comments back over them, and create_table() from a query did not inherit them. All of these now follow what the session sees.

  • A time-travel read restores the variable labels in force at its snapshot. collect() on get_ducklake_table_version() or get_ducklake_table_asof() used to attach today’s labels to the historical rows, so a label reworded since showed its new text and a label added since appeared where none had been. The lazy table now records the version or timestamp it asked for, and collect() reads the column comments as of that snapshot from the catalog’s versioned rows, finding the table under the name and id it had then. A table written from a time-travel read still takes the present-day comments, as restore_table_version() does.

  • replace_table() inside a transaction no longer fails on a table that has table-scoped options. DuckLake cannot set options on a table created in the open transaction, so the rewrite is meant to go ahead and warn with the set_ducklake_option() calls to run after the commit, but the warning itself raised a cli pluralization error, which rolled the whole transaction back. Outside a transaction, replace_table() and restore_table_version() now carry target_file_size, parquet_row_group_size_bytes, and parquet_version over. DuckLake reports the sizes as a bare number of bytes and the version as V2, forms set_option() refuses, so the reapply failed after the data had already been replaced and the option stayed behind on the dropped table.

  • replace_table() respects partition and sort keys changed earlier in the same transaction. It reads the keys to carry over from DuckLake’s metadata tables, which hold committed rows only, and no other surface shows pending keys. A table given keys with set_table_partitioning() or set_table_sorting() and rewritten in one transaction therefore lost them, and keys removed with reset_table_partitioning() or reset_table_sorting() came back. Those four functions now note what they did while a transaction is open, tagged with its transaction id so that nothing outlives it, and the rewrite uses the note. Keys changed with a raw ALTER TABLE statement in the same transaction are still not seen.

  • restore_table_version() refuses to run inside an open transaction, and says why, before it touches anything. It commits for itself, so its own BEGIN failed there with “cannot start a transaction within a transaction”, under a hint about snapshots that did not apply, and the error handler’s rollback discarded the caller’s uncommitted work.

  • The lifecycle stage is now stable, with ducklake on CRAN since 0.6.0 (published 2026-09-09): the interface is settled, and any breaking change will come with a deprecation cycle.

ducklake 0.8.0

  • commit_transaction() and with_transaction() name the snapshot this connection committed, read from DuckLake’s last_committed_snapshot(), instead of the lake’s newest snapshot right after the commit. The two agree on a lake with one writer, but with several sessions writing at once the newest snapshot could be a neighbor’s, and a transaction that changed nothing could be reported as the commit of a snapshot that was not its own. The “no changes” confirmation still names the snapshot the lake stands at. An extension without the function falls back to the previous reading.

  • attach_ducklake() labels the creation snapshot of a lake it creates. DuckLake writes snapshot 0 itself when it makes a lake, with no author and no commit message, so every history began with an <NA> row. The new commit_message (default "Create lake") and author arguments are written onto snapshot 0 right after the lake is created, with the same metadata update set_snapshot_metadata() uses, because DuckLake does not take set_commit_message() for that snapshot. A lake that already exists is left alone, and so is a read-only, snapshot-pinned, or in-transaction attach; commit_message = NULL keeps the old behavior. set_snapshot_metadata() gains snapshot_id, so the creation snapshot of an existing lake, or any other snapshot, can be labeled after the fact; it still fills blanks only unless overwrite = TRUE, and its confirmation now names the snapshot.

  • New ducklake.author option: the author recorded on every snapshot the session commits without naming one, through commit_transaction(), with_transaction(), restore_table_version(), and the creation snapshot attach_ducklake() labels. An author argument still wins. set_snapshot_metadata() does not read it, and a write made outside a transaction records no author, as before.

  • The metadata readers honor the metadata_schema a lake was attached with. They looked in the catalog’s default schema whatever the attach said, so set_snapshot_metadata(), get_table_comments(), get_table_partitions(), get_table_sorting(), get_table_info(), and get_metadata_table() failed on such a lake; the schema is now kept with the lake’s registry entry and used by all of them.

  • plot_snapshots() classifies each snapshot by what it did to the table being drawn. A transaction that updated one table in place and rebuilt another used to color both as “created”, because the whole snapshot was classified by the highest-precedence change it carried; the single-table timeline and the swimlane now read only the change-map entries that name the table, so the updated table shows a data change and the rebuilt one a creation. Swimlane lanes are schema-qualified whenever the lake keeps tables outside main, so bronze.dm and silver.dm no longer share a lane; a lake with everything in main keeps bare names.

  • The Clinical Trial Data Lake article is rewritten, and it is now a pkgdown-only article (vignettes/articles/), so it can use packages a CRAN vignette cannot declare: dplyneage for column lineage, haven for the XPT transfer it loads, admiral and pharmaversesdtm for the derivations. The lake is laid out as bronze, silver, gold, and results schemas. SDTM arrives as an XPT transfer, so the silver blank-to-NA step has blanks to convert; pharmaversesdtm stores missing values as NA already, which made the old version of that step a no-op. ADSL and ADAE are derived the way the admiral 1.4 templates do it, with imputation flags, Y/N population flags, derive_var_trtemfl(), and occurrence flags from derive_var_extreme_flag() under restrict_derivation(). A treatment-emergent adverse event summary is stored in the lake with its lineage drawn, and a correction flows from SDTM through ADaM to that summary in one transaction. The ADPC dataset and the section that stored define.xml and JSON blobs in table cells are gone, and the wording on audit trails names the elements the FDA 2024 Q&A lists rather than claiming Part 11 compliance. The eight packages only that vignette used leave Suggests (admiral, pharmaversesdtm, DiagrammeR, jsonlite, lubridate, purrr, stringr, tidyr).

  • The ducklake DuckDB extension no longer needs an install step. The code never required one: attach_ducklake() has always installed it on first use, after a message naming the directory. The README and the articles now say so, and install_ducklake() is documented as the way to download the extension ahead of time (container images, CI, machines that are offline when the lake is attached). A first-use install now also clears the answer ducklake_extension_available() cached, so it reports TRUE afterwards. The hint that follows an install into a temporary directory did not show on macOS for a first-use install, because the directory does not exist until INSTALL creates it and the check only recognized existing paths; it shows now. No load-time check was added: the package neither downloads nor starts DuckDB when it is loaded.

  • The minimum duckdb version is now 1.5.5, the release that settled where the R package keeps downloaded extensions: ~/.duckdb when it exists (duckdb offers to create it the first time it connects in an interactive session), DUCKDB_R_HOME or the duckdb.home option when set, otherwise a per-session temporary directory (?duckdb::duckdb_storage). ducklake_extension_available() now opens its probe on the directory duckdb would resolve for a new connection, so it finds an existing ~/.duckdb without triggering that offer, and the hint printed after an install into a temporary directory describes these options.

  • with_transaction() and commit_transaction() confirm a commit with one line naming the snapshot it created, with the author and commit message when they were given: “Committed snapshot 3 (Data Engineer): Add the Motor Trend car data”. A transaction that changed nothing says so, and begin_transaction() is silent. The two lines they replace (“Transaction started.”, “Transaction committed.”) said nothing about the snapshot; the new one gives the version number that time travel uses.

  • The Getting Started article (vignette("ducklake")) is rewritten as one short session: install, attach a lake, add a table, read it back, change it, see its history, and detach, with the reasons to wrap changes in with_transaction() explained along the way. Its recipes move to two new articles. “Loading Data” (vignette("loading-data")) collects the ways data gets into a lake, from data frames and files to registering Parquet in place and migrating from DuckDB or Iceberg. “Views, Comments, and Labels” (vignette("views-comments-labels")) covers the query logic and documentation that live in the catalog, and its labels example uses labelled::set_variable_labels(), so labelled is now a suggested package.

ducklake 0.7.0

  • New article “Choosing a Deployment” (vignette("deployment")): what a lake looks like on disk, which catalog backend fits one person, a team on a shared drive, or many users on object storage, where data can go and the Windows limits, how reader and writer roles map onto catalog grants and storage permissions, the day-one settings (extension persistence, schemas, retention policy, commit messages), and upgrading.

  • New get_ducklake_info() describes an attached lake in one row: backend, catalog, data path, DuckLake format version, extension version, encryption, and current snapshot.

  • New set_column_not_null() sets or drops a NOT NULL constraint, the one constraint DuckLake supports.

  • attach_ducklake() creates a local lake_path (and the directory of a local catalog file) when it does not exist and creating the lake is allowed, instead of failing with DuckDB’s “Cannot open file” error.

  • ?set_ducklake_option lists every option DuckLake 1.0 persists, with its default and what it controls.

  • New cookbook recipes migrate an existing DuckDB database into a lake with COPY FROM DATABASE and exchange tables with an Iceberg catalog; the time-travel vignette documents the rowid and snapshot_id hidden columns.

  • backup_ducklake() copies the catalog with DuckDB’s COPY FROM DATABASE while the lake stays attached, a consistent snapshot taken inside one transaction, instead of shutting the connection down to release file locks and copying the file. Nothing is detached any more: other attached lakes, in-memory secrets, and a connection registered with set_ducklake_connection() (whose locks the old approach could not release, leaving a 0-byte catalog) are left as they are. A SQLite catalog is copied into a SQLite file. Restoring a backup is documented with create = FALSE, so a mistyped path is an error rather than a new lake.

  • New package option ducklake.verbose: set it to FALSE to silence the confirmations the package emits after each operation (“Transaction committed.”, “Added column …”). Warnings, errors, and notices about extension downloads stay on. The messages carry the condition class ducklake_message.

  • Schema-qualified table names work end to end. Every function that takes a table name accepts "schema.table": get_ducklake_table() hands dbplyr a proper table path for it (the duckdb driver’s tbl() turned a dotted name into raw SQL that rows_*() could not write to), and the metadata readers (get_table_comments(), get_table_partitions(), get_table_sorting(), get_table_info(), list_table_snapshots(), get_table_changes(), list_ducklake_files()) resolve the schema instead of matching the bare name. Functions with a schema_name argument take the schema from either place. The readers gain a schema_name column. New create_schema() and drop_schema() manage schemas, the natural home for medallion layers; the README example now keeps bronze, silver, and gold in schemas of their own.

  • attach_ducklake() gains create (FALSE opens an existing lake and errors on a wrong path or name instead of creating a new, empty lake) and metadata_schema (several lakes in one PostgreSQL database, each in its own schema).

  • New set_ducklake_retry() sets how DuckLake retries a transaction that races with another writer, and the transactions vignette explains which concurrent changes conflict and which are retried.

  • create_table() and replace_table() now run dplyr pipelines inside DuckDB. A lazy table on the package’s connection is written with CREATE TABLE ... AS; a replacement is materialized in DuckDB’s temporary storage first (the query may read the table it replaces) and the table is rebuilt from it. Rows no longer pass through R, so silver and gold layers derived from large bronze tables cost DuckDB memory, not R memory. Column comments follow the data: each output column keeps the comment of the same-named column in the tables the query reads, so variable labels survive a pipeline as they did through the old collect path. Data frames, and lazy tables on other connections, load as before.

  • replace_table() and restore_table_version() now carry a table’s metadata over to the rewritten table: the table comment, column comments (and so variable labels), partition keys, sort order, and table-scoped options. Both go through DROP + CREATE, which gives the table a new id, and DuckLake keeps all of these against the id, so they were silently lost before. Partition and sort keys are set before the rows are written, so the rewrite itself lands partitioned and sorted. Table-scoped options are re-set right after the rewrite commits (DuckLake cannot set options on a table created in the open transaction); inside a transaction you opened yourself they cannot be re-set, and a warning lists the calls to make after committing. New get_table_sorting() reads a table’s sort keys from the catalog, the counterpart of get_table_partitions().

  • The set_table_partitioning() recipe for re-partitioning existing data (set the keys, then rewrite with replace_table()) now works; before, the rewrite dropped the keys it was meant to apply.

  • checkpoint_ducklake() no longer claims to expire old snapshots and reclaim their files on its own. A checkpoint does so only when the lake carries a retention policy (expire_older_than and delete_older_than, set with set_ducklake_option()); without one it flushes and compacts but keeps every snapshot and every file. The storage and data-inlining vignettes and the cookbook now show the policy.

  • set_snapshot_metadata() fills in only empty fields by default and stops when a supplied field already has a value; pass overwrite = TRUE to replace one. It writes to the catalog outside DuckLake’s transaction model, and an overwrite leaves no trace of the previous value, so the audited path is metadata set at commit time with with_transaction() or commit_transaction(). The documentation now says so plainly.

  • The minimum duckdb version is now 1.5.2, the release that ships DuckLake 1.0. The extension built for DuckDB 1.5.1 writes the earlier 0.4 catalog format, so a lake created there needs a one-time migration after the upgrade: attach_ducklake() gains automatic_migration = TRUE for that (DuckLake’s AUTOMATIC_MIGRATION option).

  • Where the ducklake extension is installed depends on the duckdb R package: from 1.5.2 on it is a per-session temporary directory unless DUCKDB_R_HOME (or the duckdb.home option, or an existing ~/.duckdb) points somewhere durable, so “install once per machine” needs that setting. install_ducklake() and the automatic extension installs now report the directory they used and say when it is temporary; the README, ducklake_extension_available(), and create_storage_secret(persistent = TRUE) describe the setting.

  • The clinical trial vignette’s silver layer called admiral::convert_blanks_to_na() on a lazy lake table, which leaves the table untouched, so the cleaning it described never ran. The conversion now runs inside DuckDB through a small dbplyr helper. (The pharmaverse test data already stores missing values as NA, so the vignette’s output does not change; XPT exports do carry blanks.)

  • Vignettes follow the package’s own guidance on how to change a table: derived columns are declared with add_table_column() and filled with ducklake_exec(), corrections use rows_update() and rows_delete(), and replace_table() is kept for bulk rewrites.

  • get_ducklake_table_asof() and get_ducklake_table_version() return the same tbl_ducklake class as get_ducklake_table(), so collect() restores stored column labels on time-travel reads too.

  • list_table_snapshots(table_name) matches snapshots by parsing the change map rather than by a regular expression over its printed form, and resolves table ids within the table’s schema.

  • Functions that read a named lake’s metadata (get_metadata_table(), list_table_snapshots(), get_table_partitions(), get_table_comments(), plot_snapshots()) resolve the catalog backend from that lake rather than from the current database, so a PostgreSQL or MySQL lake that is attached but not current is qualified correctly.

  • ducklake_exec(.quiet = FALSE) emits its SQL trace as messages instead of printing it, and no longer prints the lazy table itself, which ran a preview query. show_ducklake_query() prints through cli.

  • The documentation notes that the aws and azure DuckDB extensions are not available on Windows for the duckdb R package, so create_storage_secret(provider = "credential_chain") and Azure secrets do not work there. The README now says Quack needs duckdb 1.5.3 or newer, not 1.5.4.

  • add_table_column() now works with logical, Date, and POSIXct defaults. DuckLake accepts only plain constants in a DEFAULT clause and rejected the typed literals the package rendered (TRUE, DATE '...', TIMESTAMP '...') as “non-literal”; such values are now passed as quoted strings, which DuckLake converts to the column type.

  • A failed schema evolution statement no longer leaves the shared connection in an aborted transaction. Some DuckLake DDL failures do that even in autocommit mode, so that every later statement failed with “Current transaction is aborted”; the wrappers now roll back before re-raising the error when they did not inherit a transaction.

  • The unreferenced and broken inst/examples/with_transaction_demo.R was removed; the with_transaction() examples cover it.

  • New meta_encryption_key argument in attach_ducklake() encrypts the DuckDB catalog database file itself with AES-256-GCM (DuckLake forwards META_-prefixed options to the metadata catalog). The key is set when the catalog is created and required on every later attach. The catalog is where encrypted = TRUE stores its Parquet keys, so encrypting it closes that loop; pass askpass::askpass() as the value to be prompted instead of writing the key in code (#46, suggested by @frankpopham).

  • The "duckdb" backend now honors catalog_connection_string as the path for its catalog file, so a single-writer lake can keep the catalog on local disk while lake_path points at object storage. The documentation already promised this argument worked for the duckdb backend; now it does (#44).

  • attach_ducklake() stops early when the duckdb backend would create its catalog file on object storage, which DuckDB cannot write and which previously surfaced as a confusing IO error. The message points to catalog_connection_string, to read_only = TRUE (attaching an existing remote catalog stays supported), and to the other backends. httpfs is now loaded up front whenever a remote path is involved instead of relying on DuckDB’s mid-statement autoload.

  • The create_storage_secret() example and the storage vignette attached an S3 lake with the default backend and no catalog path, which would put the catalog file itself on S3. Both now show the split layout, and backup_ducklake() finds a catalog that lives outside lake_path.

  • create_storage_secret(provider = "credential_chain") now loads DuckDB’s aws extension itself for "s3", "gcs", and "r2" secrets, installing it on first use. The provider lives in that extension, and DuckDB’s automatic mid-statement install of it could fail, leaving CREATE SECRET erroring with “Install it first” (#43).

ducklake 0.6.0

CRAN release: 2026-09-09

First CRAN release.

  • New ducklake_extension_available() reports whether the ducklake DuckDB extension is installed and loadable. It probes with automatic extension installation switched off, so it never downloads anything, and it is what the package’s own examples, tests, and vignettes gate on.

  • Every example that can run against a temporary lake now does, guarded by @examplesIf ducklake_extension_available(). Only the cases that need outside infrastructure stay unevaluated: Quack servers, PostgreSQL and MySQL catalogs, and remote URLs.

  • backup_ducklake() no longer needs the fs package. It was the one place in the package that called a suggested dependency unconditionally, so backups failed for anyone without fs installed.

  • The package now tells you before it installs a DuckDB extension for you. attach_ducklake() and the functions that load httpfs, quack, or a backend extension used to download into the extension cache in your home directory without a word.

  • Fixed the with_transaction() rollback example, which reused a table name that already existed and so failed before reaching the error it meant to demonstrate.

ducklake 0.5.0

  • New rows_upsert() completes the dplyr rows_* family: rows that match on the key columns are updated and the rest are inserted, as one atomic MERGE INTO statement (DuckLake tables have no primary keys, so the ON CONFLICT upsert other databases use does not apply). One call is one snapshot, with the updates and inserts recorded individually in the change feed.

  • New merge_into() exposes the full SQL MERGE surface for the cases rows_upsert() cannot express: conditional matched and not-matched clauses, deleting matched rows, and delete_missing = TRUE to drop target rows absent from the source (a staging-table sync). DuckLake currently allows one update/delete action per MERGE statement, so syncs that need both run as MERGE plus DELETE inside a single transaction and snapshot.

  • New table documentation family, built for labelled-data workflows: set_table_comment() and set_column_comments() store descriptions in the lake’s catalog (COMMENT ON), and get_table_comments() reads them back as a tidy data frame. create_table() gains a labels argument (default TRUE) that stores haven/labelled variable labels as column comments at load time, and collect() on a lake table reattaches stored comments as label attributes – so gtsummary, gt, and other label-aware tools work as if the data never left R, and every other client of the lake can read the same documentation.

  • New create_view() stores a dplyr pipeline as a SQL view in the lake: shared business logic that reads current data and that every client – R, Python, or plain SQL – sees identically. drop_view() removes one. SQL macros stay unwrapped on purpose (a macro body is raw SQL and unreachable from dplyr pipelines); the cookbook shows the DBI::dbExecute() escape hatch.

  • New list_ducklake_tables() answers “what is in this lake?” with a tidy frame of tables and views.

  • New schema evolution family: add_table_column() (with optional default, which DuckLake applies to existing rows too), drop_table_column(), rename_table_column(), set_column_type() (widening promotions only, with the add-copy-drop-rename recipe in the error when a change would narrow), and rename_ducklake_table(). All are metadata-only ALTER TABLE operations: no data files are rewritten, and earlier snapshots keep the earlier schema. Until now schema changes went through replace_table(), which collects the whole table into R; its documentation now points here, and it remains the tool for bulk data transformations. Derived columns combine the two styles: add_table_column() then a mutate() pipeline through ducklake_exec() fills the column with an in-database UPDATE.

  • New plotting functions, all requiring the suggested ggplot2 package and shown in action in a new “Visualizing Your Lake” vignette: plot_snapshots() draws a table’s snapshot history as a commit-log timeline (snapshots in order, with authors, commit messages, and inline markers for long idle gaps) or, without a table name, the whole lake as a swimlane with one row per table; plot_table_changes() draws the rows each snapshot inserted, updated, and deleted as diverging bars; and plot_table_files() draws each table’s Parquet file count and size on disk.

  • New get_table_info() returns per-table file statistics (data and delete file counts and sizes) from the DuckLake catalog, wrapping DuckLake’s ducklake_table_info() function.

  • New add_data_files() registers existing Parquet files with a table without copying or rewriting them – the migration path for data that is already in Parquet. A vector of files is registered atomically in one snapshot, and create = TRUE can bootstrap a new target table directly from the Parquet schema. list_ducklake_files() shows the files backing a table, optionally as of a past snapshot.

  • New sorted-table support: set_table_sorting() and reset_table_sorting() manage a table’s declared sort order, the complement to partitioning for pruning on high-cardinality columns.

  • New set_ducklake_option() and get_ducklake_options() expose DuckLake’s full option system (parquet_compression, target_file_size, sort_on_insert, require_commit_message, …) at lake, schema, or table scope. set_inlining_row_limit() now builds on the same internals.

  • New create_storage_secret() stores object-storage credentials (S3, GCS, R2, Azure) via DuckDB’s secrets manager, so a lake’s lake_path can live on cloud storage. backup_ducklake() now errors clearly for remote data paths instead of failing partway through.

  • attach_ducklake() gains snapshot_version and snapshot_time arguments to attach a lake pinned to a historical snapshot – a frozen, read-only view for reproducing past analyses.

  • replace_table() now runs its drop and create as one transaction, so a failed create no longer leaves the table dropped. When the caller has already opened a transaction, replace_table() defers to it as before.

  • set_snapshot_metadata() now validates ducklake_name like the rest of the package, and resolves the catalog backend from that name instead of the current database.

  • create_table() no longer leaves a temporary view registered on the shared connection when the CREATE statement fails.

  • replace_table(.quiet = FALSE) reports progress via messages (cli) rather than printing to the console, so it can be suppressed and captured like the rest of the package’s output.

  • The package now declares R (>= 4.1) explicitly (the tests and examples use the base pipe), and is prepared for CRAN: extension-dependent tests skip on CRAN, and vignettes evaluate only where the ducklake DuckDB extension can be loaded.

  • New targeted maintenance wrappers complement checkpoint_ducklake(): expire_snapshots() (with older_than, versions, and dry_run), merge_adjacent_files(), cleanup_old_files(), delete_orphaned_files(), and rewrite_data_files() (#16, suggested by @stefanlinner).

  • New partitioning support: set_table_partitioning() and reset_table_partitioning() manage a table’s partition keys (identity, year/month/day/hour, and bucket transforms), and get_table_partitions() lists the keys from the metadata catalog (#16, suggested by @stefanlinner).

  • New get_table_changes() exposes DuckLake’s data change feed: the exact inserts, deletes, and update pre/post images between two snapshots, as a lazy table that composes with dplyr verbs (#16, suggested by @stefanlinner).

  • Timestamps passed to the new functions as POSIXct are converted to UTC before interpolation, matching how DuckLake records snapshot times.

  • get_ducklake_table_asof() and restore_table_version() now also convert POSIXct timestamps to UTC. Previously they rendered local time, which DuckLake reads as UTC, silently shifting the queried instant by the UTC offset – a bare Sys.time() looked hours in the past (or future) unless the session’s timezone was UTC. Timestamps taken from list_table_snapshots()$snapshot_time are unaffected.

  • attach_ducklake() now collapses duplicate slashes in lake_path (remote URIs are untouched). DuckLake compares file paths as exact strings, so a doubled slash – which R’s tempdir() produces on macOS – made delete_orphaned_files() treat every live data file as orphaned.

  • The dplyr-to-DuckLake translation behind ducklake_exec() and show_ducklake_query() is now built from dbplyr’s structured query objects (dbplyr::sql_build()) instead of pattern-matching rendered SQL text. Classification no longer depends on what the SQL happens to look like, which fixes several latent bugs:

    • A filtered read from another table (get_ducklake_table("staging") |> filter(...) |> ducklake_exec("target")) was translated into a DELETE on the target table; it now appends the matching rows, as intended.
    • Filter values containing SQL keywords (e.g. filter(note != "WHERE is it")) were refused as “too complex”; they now translate fine.
    • A mutate() that adds a new column is refused upfront with a pointer to replace_table(), instead of failing with a database binder error.

  • INSERT translations now list columns explicitly, so appends from another table match columns by name rather than by position, and joined or unioned sources can be appended in one step.

  • Pipelines that compile to a subquery over the target table (grouped filters, filtering on a just-mutated column), and clauses with no in-place equivalent (arrange(), head(), distinct()), are detected structurally and refused with a clear message rather than mistranslated.

  • dbplyr (>= 2.5.0) is now required; both dbplyr 2.5.x and the select-list format introduced in dbplyr 2.6.0 are supported.

ducklake 0.4.0

This release focuses on production hardiness: self-contained connection management, working detach/restore, SQL identifier safety, Quack remote access, and a documentation overhaul.

Quack remote protocol support

Added support for Quack, DuckDB’s client-server protocol, which became a core extension in DuckDB 1.5.3 (#20, @JavOrraca). A DuckLake served by one DuckDB instance can now be queried and modified by other R sessions over the network. For concurrent access this is a lighter-weight option than a PostgreSQL or SQLite catalog, since the whole setup stays inside DuckDB and DuckLake.

  • attach_quack() connects to a remote Quack server and attaches it as a catalog in the current session.
  • detach_quack() disconnects from a remote Quack server.
  • install_quack() installs the Quack DuckDB extension.
  • quack_query() runs a one-off query against a remote Quack server and returns a data.frame.
  • quack_serve() serves the current session, including an attached DuckLake, to other clients over Quack.
  • quack_stop() stops a running Quack server.

Production hardening

New Features

  • attach_ducklake() gains an encrypted argument: pass encrypted = TRUE to have DuckLake encrypt the Parquet files it writes (#18). Note that the encryption keys are stored in the catalog database, so protect the catalog. The httpfs extension is loaded automatically for encrypted lakes, since on some platforms (notably Windows) DuckDB’s built-in crypto module is read-only.
  • restore_table_version() now works. It previously generated a RESTORE TABLE statement that does not exist in DuckLake and failed on every call. It now recreates the table from a time-travel read inside a transaction, recording the restore as a new snapshot so history is preserved. It also gains author and commit_message arguments so the restore snapshot carries full audit-trail metadata.
  • get_ducklake_backend() gains a ducklake_name argument and tracks each attached lake separately, so sessions with several lakes on different catalog backends resolve backend-specific behaviour correctly.

Bug Fixes

  • detach_ducklake() now actually detaches. Previously the DETACH ran while the lake was still the session’s current database, which DuckDB refuses, and the error was silently swallowed – the lake stayed attached. The session now switches back to the connection’s own catalog first. Relatedly, restoring a backup to a new location requires override_data_path = TRUE (as documented); the storage vignette example has been corrected.
  • Table names, lake names, and file paths are now quoted or validated before being interpolated into SQL (DBI::dbQuoteIdentifier() and friends), so names with spaces or quotes no longer produce malformed statements.
  • rows_insert(), rows_update(), and rows_delete() now also dispatch as S3 methods on tables returned by get_ducklake_table(). Previously, if dplyr was loaded after ducklake, dplyr’s generics masked ducklake’s wrappers and calls failed with conflict = "error" complaints; load order no longer matters.
  • rows_insert(), rows_update(), and rows_delete() now work inside with_transaction(), so several row operations can be grouped into a single snapshot with an author and commit message. Previously, passing a local data frame made dbplyr copy it to a temporary table inside its own transaction, which DuckDB rejects when one is already open. Local data frames are now sent as inline queries (dbplyr::copy_inline()), which is also faster for the small changesets these functions are designed for.
  • backup_ducklake() backs up every schema directory, not just main.
  • create_table() now converts factor columns to character (with a message) instead of failing with “unsupported type ENUM” – DuckLake does not support DuckDB’s ENUM type, which is what factors become.
  • The internal dplyr-to-SQL translation in ducklake_exec() no longer uses sink() (which could leak diverted output on error), and now refuses queries with subqueries or multiple WHERE clauses instead of generating incorrect SQL.
  • ducklake_exec() no longer executes its statement twice. The internal translation step also executed the SQL before ducklake_exec() ran it again, so every call created two snapshots and non-idempotent updates (e.g. v = v + 1) were applied twice.
  • show_ducklake_query() is now a true preview: it previously executed the translated statement against the lake while displaying it.
  • ducklake_exec() now translates any mutate() into an UPDATE, not just those that compile to CASE WHEN. Previously a simple transformation like mutate(v = round(v, 1)) fell through to an INSERT of the table’s own rows, silently duplicating the table. Plain self-reads with nothing to translate are now refused for the same reason, and UPDATE assignments containing commas inside function calls are parsed correctly.
  • list_table_snapshots(table_name) no longer misses snapshots created by rows_insert(), rows_update(), and rows_delete(). DuckLake records row-level changes against the table’s numeric id rather than its name; the filter now resolves and matches those ids, so the per-table audit trail is complete. Filtered listings also number their rows from 1 instead of leaking the row positions of the unfiltered result.

Connection management is now self-contained

ducklake now creates and manages its own DuckDB connection instead of reaching into duckplyr’s unexported internals. This removes the package’s last ::: calls and the duckplyr dependency entirely.

Breaking Changes

  • duckplyr is no longer a dependency. If you relied on ducklake sharing duckplyr’s default connection, register a connection explicitly with the new set_ducklake_connection().

New Features

  • set_ducklake_connection() (returning by popular demand, now safer): point ducklake at any DuckDB connection you manage — for example one shared with other DBI tools. Connections you supply are never closed by ducklake; only its own automatically created connection is shut down by detach_ducklake(shutdown = TRUE) and at session exit.

ducklake 0.3.0

DuckLake v1.0 Specification Alignment

This release aligns the package with the DuckLake v1.0 stable specification, which requires DuckDB v1.5.2+ (compatible with duckdb R package >= 1.5.1).

Breaking Changes

  • DuckDB version requirement bumped from 1.3.0 to 1.5.1 (duckdb R package) / 1.5.2 (DuckDB engine/CLI) to match DuckLake v1.0. install_ducklake() now enforces this at the engine level.

  • commit_transaction() and with_transaction() now use the official CALL ducklake.set_commit_message() API to set commit metadata within the transaction before COMMIT, consistent with the v1.0 specification. set_snapshot_metadata() retroactively updates the ducklake_snapshot_changes metadata table directly.

ducklake 0.2.0

Multi-Backend Catalog Support

DuckLake now supports PostgreSQL, SQLite, and MySQL as catalog backends in addition to DuckDB (#15, @stefanlinner). This aligns with the DuckLake 1.0 specification and enables concurrent multi-client access when using PostgreSQL or SQLite.

New Features

  • attach_ducklake() gains backend, catalog_connection_string, read_only, and override_data_path parameters for multi-backend support.
  • install_ducklake() gains a backend parameter to pre-install backend extensions (e.g., install_ducklake(backend = "postgres")).
  • New get_ducklake_backend() returns the active catalog backend type.
  • detach_ducklake() gains a shutdown parameter. By default it now performs a soft detach (SQL DETACH + USE memory;) instead of shutting down the connection, allowing backend switching within a session.
  • backup_ducklake() is now backend-aware: file-based backends (DuckDB, SQLite) get catalog + data copied; PostgreSQL/MySQL get data only with guidance to use pg_dump/mysqldump. Also fixes a pre-existing bug where catalog backups were silently 0 bytes due to DuckDB holding file locks during file.copy().

Breaking Changes

  • attach_ducklake() now requires lake_path (previously optional).
  • set_ducklake_connection() has been removed. The package now exclusively uses duckplyr’s singleton DuckDB connection.
  • detach_ducklake() no longer shuts down the DuckDB connection by default. Pass shutdown = TRUE for the previous behaviour.

Internal

ducklake 0.1.0

Initial release of ducklake, an R package for versioned data lake infrastructure built on DuckDB and DuckLake.

Features

Core Table Operations

Row-Level Operations

ACID Transactions

Time Travel

Metadata and Audit Trail

Connection Management

Query Execution

  • ducklake_exec() - Execute SQL with automatic assignment handling
  • show_ducklake_query() - Preview translated SQL queries
  • extract_assignments_from_sql() - Parse SQL table assignments

Backup and Maintenance

  • backup_ducklake() - Create incremental backups
  • Support for local and remote backup locations

Vignettes

  • Getting Started - Introduction to ducklake workflows
  • Clinical Trial Data Lake - Industry-specific use case
  • Modifying Tables - Comprehensive guide to row operations
  • Working with Transactions - ACID transaction patterns
  • Time Travel Queries - Historical data access
  • Storage and Backup Management - Data persistence strategies

Lifecycle

This package is currently in experimental status. The API may change as we gather feedback from early users, but core functionality is stable and ready for pilot projects.