Skip to contents

Flush inlined data to Parquet files

Source: R/data_inlining.R
flush_inlined_data.Rd

Materialises data that has been stored inline in the catalog database into Parquet files on the data path. This includes both inlined inserts and inlined deletions.

Usage

flush_inlined_data(ducklake_name = NULL, table_name = NULL, schema_name = NULL)

Arguments

ducklake_name

Name of the attached DuckLake catalog. If NULL, the current database is used.

table_name

Optional table name, optionally qualified as "schema.table". When provided, only flushes inlined data for that table.

schema_name

Optional schema name. When provided, only flushes inlined data for tables in that schema.

Value

A data frame with columns schema_name, table_name, and rows_flushed. Tables with no inlined data are omitted.

Details

Flushing writes inlined rows to consolidated Parquet files and cleans up the inlined data tables. Time-travel information is preserved: flushed rows that had been deleted will produce a partial deletion file with snapshot metadata.

Tables with auto_compact set to FALSE are skipped when flushing an entire lake or schema. Use an explicit table_name to flush those tables.

If a table has a sort order defined, the flushed Parquet file will be sorted by those keys.

See also

set_inlining_row_limit(), checkpoint_ducklake()

Other data inlining: checkpoint_ducklake(), get_inlining_row_limit(), set_inlining_row_limit()

Other maintenance: backup_ducklake(), checkpoint_ducklake(), cleanup_old_files(), delete_orphaned_files(), expire_snapshots(), get_table_info(), list_ducklake_files(), merge_adjacent_files(), plot_table_files(), rewrite_data_files()

Examples

lake_dir <- tempfile("flush_lake_")
dir.create(lake_dir)
attach_ducklake("flush_lake", lake_path = lake_dir,
                data_inlining_row_limit = 10)
create_table(head(mtcars, 3), "cars")

# Flush everything
flush_inlined_data()
#> Flushed 3 rows from 1 table to Parquet.
#>   schema_name table_name rows_flushed
#> 1        main       cars            3

# Flush a specific table
flush_inlined_data(table_name = "cars")
#> No inlined data to flush.
#> [1] schema_name  table_name   rows_flushed
#> <0 rows> (or 0-length row.names)

detach_ducklake("flush_lake", shutdown = TRUE)
unlink(lake_dir, recursive = TRUE)