Skip to contents

Flush inlined data to Parquet files

Source: R/data_inlining.R
flush_inlined_data.Rd

Materialises data that has been stored inline in the catalog database into Parquet files on the data path. This includes both inlined inserts and inlined deletions.

Usage

flush_inlined_data(ducklake_name = NULL, table_name = NULL, schema_name = NULL)

Arguments

ducklake_name

Name of the attached DuckLake catalog. If NULL, the current database is used.

table_name

Optional table name. When provided, only flushes inlined data for that table.

schema_name

Optional schema name. When provided, only flushes inlined data for tables in that schema.

Value

A data frame with columns schema_name, table_name, and rows_flushed. Tables with no inlined data are omitted.

Details

Flushing writes inlined rows to consolidated Parquet files and cleans up the inlined data tables. Time-travel information is preserved: flushed rows that had been deleted will produce a partial deletion file with snapshot metadata.

Tables with auto_compact set to FALSE are skipped when flushing an entire lake or schema. Use an explicit table_name to flush those tables.

If a table has a sort order defined, the flushed Parquet file will be sorted by those keys.

See also

set_inlining_row_limit(), checkpoint_ducklake()

Other data inlining: checkpoint_ducklake(), get_inlining_row_limit(), set_inlining_row_limit()

Other maintenance: backup_ducklake(), checkpoint_ducklake(), cleanup_old_files(), delete_orphaned_files(), expire_snapshots(), merge_adjacent_files(), rewrite_data_files()

Examples

if (FALSE) { # \dontrun{
# Flush everything
flush_inlined_data()

# Flush a specific table
flush_inlined_data(table_name = "readings")

# Flush a specific schema
flush_inlined_data(schema_name = "staging")
} # }