Introduction
Most of the time a DuckLake lives in one place and one person works with it. But data analysis is usually a team activity: a few statisticians share the same study data, an analytics group works from one set of cleaned tables, several people review the same numbers before a deliverable goes out. The usual answer is to copy files around (a CSV here, an XPT there) and hope everyone is looking at the same version.
Quack offers a different answer. It is a protocol, a core DuckDB extension stable since DuckDB 2.0, that lets one DuckDB instance act as a server while other DuckDB instances connect to it as clients. A DuckLake held by the server can then be read and updated by other R sessions over the network, with the same snapshots and time travel you get locally. Several people can write at the same time without locking each other out.
This vignette shows the two halves of that picture: connecting to a lake someone else is serving, and serving a lake of your own. The audience here is the everyday R user, so the examples stay close to ordinary analysis work and do not assume any server administration experience.
The code below is shown with its expected output but is not run when the vignette is built, because Quack needs a live server.
Installing the extension
Quack is a core extension, so DuckDB loads it automatically the first time it is used. On a machine that has no internet access at query time you can install it ahead of time:
install_quack()
#> Installed quack extension.
#> Loaded quack extension.Connecting to a shared lake
This is the common case. A colleague, or a shared workstation, runs a Quack server that holds the team’s DuckLake. You connect to it from your own R session and pull the data you need.
quack_query() sends a single SQL query to the server and
returns the result as a data frame. The lake is named by its catalog on
the server, which is the name it was attached under (here,
trial):
adsl <- quack_query(
"quack:trial-server.example.org",
"SELECT * FROM trial.adsl",
token = "super_secret"
)
adsl
#> USUBJID SAFFL AGE
#> 1 01-001 Y 45
#> 2 01-002 N 52The address starts with quack:. A bare host name works
too and the scheme is added for you. The default port is 9494; include a
different one as quack:host:port. The token is
whatever password the server was started with.
Once a query comes back you have an ordinary data frame, so the rest of your analysis is the dplyr and base R you already use:
adsl |>
group_by(SAFFL) |>
summarise(n = n(), mean_age = mean(AGE))
#> # A tibble: 2 x 3
#> SAFFL n mean_age
#> <chr> <int> <dbl>
#> 1 N 1 52
#> 2 Y 1 45For large tables it is faster to let the server do the filtering and summarising and send back only what you need, rather than pulling every row across the network:
quack_query(
"quack:trial-server.example.org",
"SELECT USUBJID, AGE FROM trial.adsl WHERE SAFFL = 'Y' AND AGE >= 18",
token = "super_secret"
)
#> USUBJID AGE
#> 1 01-001 45Because Quack supports concurrent writers, you can also add data to the shared lake. The query runs on the server, so it refers to the lake by its catalog name and returns the number of rows affected:
quack_query(
"quack:trial-server.example.org",
"INSERT INTO trial.adsl VALUES ('01-099', 'Y', 70)",
token = "super_secret"
)
#> Count
#> 1 1Connecting to a plain database
If the server is sharing a regular DuckDB database rather than a
DuckLake, you can attach it as a catalog and work with it through the
familiar get_ducklake_table() and dplyr path.
attach_quack() gives the remote a local name; its tables
are then reached as name.table:
attach_quack("warehouse", "quack:data.example.org", token = "super_secret")
get_ducklake_table("warehouse.sales") |>
filter(region == "EMEA") |>
collect()
detach_quack("warehouse")A served DuckLake is a separate catalog on the server, not part of
its default database, so its tables are not exposed this way. Reach a
served DuckLake with quack_query() and the lake’s catalog
name, as in the previous section, or attach the lake itself through its
served catalog as the section after next shows.
DuckDB 2.0 also has a CONNECT statement that points a
whole session at a server. Leave it alone on the package’s connection:
on the duckdb R client it breaks every statement that follows, with no
way back short of a new connection (duckdb/duckdb#27269).
Serving a lake of your own
The other half of the picture is publishing a lake so others can connect to it. Say you have built a small medallion lake and a few colleagues want to read from it. You attach the lake as usual, then start a server.
attach_ducklake("trial", lake_path = "~/lakes/trial")
with_transaction(
create_table(mtcars, "vehicles"),
author = "You",
commit_message = "Seed the shared lake"
)
install_quack()
quack_serve(token = "super_secret")
#> Quack server listening on "quack:localhost".The server runs in the background of the DuckDB instance, so your R session stays free for other work. Anyone who can reach the address can now connect with the client code from the previous section, pointing at your machine and using the same token.
Choose the address to match who should reach the server.
"quack:localhost" (the default) stays on your own machine,
which is useful for trying things out. To accept connections from other
machines, listen on a host name or address they can route to.
When you are finished, stop the server:
quack_stop()
#> Quack server on "quack:localhost" stopped.A note on access
A token is the only thing standing between your data and anyone who
can reach the server. Pick a strong one. A server is never open: when
you start one without a token, DuckDB generates a random one and
quack_serve() returns it in auth_token, and
create_secret_if_not_exists = TRUE keeps a generated token
in a secret so later servers reuse it. On an untrusted network, serve
HTTPS with the certificate and key quack_generate_keys()
writes and hand clients the fingerprint it reports as
ssl_fingerprint, or put the server behind a TLS-terminating
reverse proxy. The DuckDB Quack
documentation covers these deployment options.
Serving the catalog instead of the tables
The server above serves a lake’s tables: clients query them, and only the server touches the files. DuckLake 1.1 adds a second arrangement, experimental for now, where a Quack server holds a lake’s catalog open and every client attaches the lake through it. The catalog is a plain DuckDB file, which on its own admits one process at a time; served, it takes any number of sessions, each of them a full client of the lake that reads and writes the Parquet files itself. That makes the arrangement a fit for a team on a shared drive that would rather not run PostgreSQL.
The server is a DuckDB process that opens the catalog file and serves it. Any DuckDB client will do; from R:
con <- DBI::dbConnect(duckdb::duckdb(), dbdir = "/shared/lakes/trial/catalog.duckdb")
DBI::dbExecute(con, "INSTALL ducklake; LOAD ducklake; INSTALL httpfs; LOAD httpfs; INSTALL quack; LOAD quack")
DBI::dbGetQuery(con, "CALL quack_serve('quack:catalog.example.org', token = 'super_secret', allow_other_hostname = true)")Each client attaches the lake with backend = "quack",
the server’s address as the catalog connection string, and the data path
as every client reaches it:
attach_ducklake(
"trial",
backend = "quack",
catalog_connection_string = "quack:catalog.example.org",
quack_token = "super_secret",
lake_path = "/shared/lakes/trial/data"
)
with_transaction(
create_table(mtcars, "vehicles"),
author = "You",
commit_message = "Seed the shared lake"
)
list_table_snapshots("vehicles")From here on nothing is different: snapshots, time travel, comments,
and the maintenance functions all go through the served catalog, and
commits run on the server. Back up the catalog by stopping the server
and copying its file; backup_ducklake() copies the data
files only.
Quack or a shared catalog?
ducklake already supports PostgreSQL and SQLite catalogs for
concurrent access (see ?attach_ducklake and the package
NEWS). Quack covers a similar need from two angles, and the right choice
depends on where your data lives and who can reach it.
| Shared catalog (PostgreSQL or SQLite) | Catalog served over Quack | Tables served over Quack | |
|---|---|---|---|
| What clients connect to | The catalog database and the Parquet storage directly | One DuckDB server for the catalog, the Parquet storage directly | One DuckDB server |
| Storage access each client needs | Read and write access to the same data path | Read and write access to the same data path | None; only the server touches the files |
| Extra service to run | A PostgreSQL server (SQLite needs none) | A DuckDB server process | A DuckDB server process |
| Good fit when | Everyone can mount the same shared storage | Everyone can mount the storage and nobody wants to run PostgreSQL | The data sits on one machine and you want to share access to it |
A common pattern is to use two of them: a PostgreSQL catalog so the lake itself supports many writers, and a Quack server in front so analysts who cannot mount the storage can still connect.
Caveats
Quack reached its stable release in DuckDB 2.0, the version ducklake requires, so the server and every client speak the same settled protocol. For very high write volumes a purpose built database is still the better tool, but for sharing a study lake across a small team Quack is a light way to get everyone onto the same versioned data.
Learn more
- Quack overview, the DuckDB documentation.
-
?quack_serve,?attach_quack, and?quack_queryfor the function reference. - The Storage and Backup Management vignette for how a DuckLake is laid out on disk.
