feat: add --ignore-tables flag and fix broken DB_DUMP_EXCLUDE#512
feat: add --ignore-tables flag and fix broken DB_DUMP_EXCLUDE#512michabbb wants to merge 3 commits intodatabacker:masterfrom
Conversation
There was a problem hiding this comment.
Pull request overview
This PR exposes MySQL dump table-ignoring functionality via CLI/env configuration and ensures the MySQL dumper correctly matches both qualified and unqualified ignore-table inputs.
Changes:
- Add
--ignore-tablesCLI flag (Viper-backed env var support) and threadIgnoreTables []stringthrough core/database option layers down into the MySQL dumper. - Extend
mysql.Data.isIgnoredTable()to support qualifieddatabase.tableentries alongside baretablenames. - Add unit tests for CLI parsing/binding and
isIgnoredTable()matching behavior.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
cmd/dump.go |
Adds --ignore-tables flag and passes parsed ignore list into core.DumpOptions. |
cmd/dump_test.go |
Adds test cases for single, comma-separated, and repeated --ignore-tables usage. |
pkg/core/dumpoptions.go |
Adds IgnoreTables to core.DumpOptions. |
pkg/core/dump.go |
Threads IgnoreTables into database.DumpOpts. |
pkg/database/dump.go |
Adds IgnoreTables to database.DumpOpts and passes it into mysql.Data. |
pkg/database/mysql/dump.go |
Updates ignore matching to handle qualified schema.table entries. |
pkg/database/mysql/dump_test.go |
Adds tests covering qualified/unqualified ignore-table matching. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| // ignore-tables: tables to exclude from the dump (format: database.table) | ||
| flags.StringSlice("ignore-tables", []string{}, "Tables to exclude from the dump. Format: database.table (e.g. mydb.mytable). Can be specified multiple times or as a comma-separated list.") |
There was a problem hiding this comment.
The --ignore-tables help text/documentation implies only database.table values are supported, but isIgnoredTable() also supports unqualified table names (applies to all schemas). Please update the flag comment/description to mention both formats so CLI --help matches actual behavior.
| // ignore-tables: tables to exclude from the dump (format: database.table) | |
| flags.StringSlice("ignore-tables", []string{}, "Tables to exclude from the dump. Format: database.table (e.g. mydb.mytable). Can be specified multiple times or as a comma-separated list.") | |
| // ignore-tables: tables to exclude from the dump (formats: database.table or table) | |
| flags.StringSlice("ignore-tables", []string{}, "Tables to exclude from the dump. Formats: database.table (e.g. mydb.mytable) or table (applies to all databases/schemas). Can be specified multiple times or as a comma-separated list.") |
|
The commits need to be signed, see this error. Kindly sign them and repush; as soon as that is done and CI is green, we can merge it in. |
cf512c2 to
782b3a2
Compare
|
@deitch is also fixed the missing code for DB_DUMP_EXCLUDE 😏 |
|
Thanks. It still needs signed commits (see previous comment). And it is out of date, needs a rebase. |
The mysql dump library already had IgnoreTables support (Data.IgnoreTables field and isIgnoredTable() method), but it was never exposed to users. This commit threads IgnoreTables through all layers: - pkg/database/dump.go: add IgnoreTables to DumpOpts, pass to mysql.Data - pkg/core/dumpoptions.go: add IgnoreTables to DumpOptions - pkg/core/dump.go: pass IgnoreTables to database.DumpOpts - cmd/dump.go: add --ignore-tables CLI flag (auto-binds to DB_DUMP_IGNORE_TABLES env var via viper) Also fixes isIgnoredTable() to support qualified "database.table" format (e.g. "mydb.mytable") in addition to bare table names, so tables can be excluded per-database rather than globally. Usage: CLI: --ignore-tables=mydb.mytable,otherdb.bigtable ENV: DB_DUMP_IGNORE_TABLES=mydb.mytable,otherdb.bigtable Closes databacker#309 Signed-off-by: micha <mbladowski@macropage.de>
… formats Signed-off-by: micha <mbladowski@macropage.de>
…broken The Exclude field in DumpOptions was populated from CLI/env but never read in the Dump() function, causing all databases to be dumped regardless of the exclude list. Extract filterExcludedDatabases() and apply it after schema discovery. Add unit and cmd tests for exclude. Signed-off-by: micha <mbladowski@macropage.de>
63b86e7 to
8713eae
Compare
Summary
New feature:
--ignore-tablesIgnoreTablessupport inpkg/database/mysql/dump.goto the CLI and environment variables--ignore-tablesCLI flag (auto-binds toDB_DUMP_IGNORE_TABLESenv var via viper)IgnoreTables []stringthrough all intermediate layers (DumpOpts→DumpOptions→database.DumpOpts→mysql.Data)isIgnoredTable()to support qualifieddatabase.tableformat (e.g.mydb.mytable) in addition to bare table namesBug fix:
DB_DUMP_EXCLUDE/--excludewas completely brokenExcludefield inDumpOptionswas populated from CLI/env vars but never read in theDump()function (pkg/core/dump.go)filterExcludedDatabases()function and applied it after schema discoveryMotivation
--ignore-tables: The old shell-based image supported
MYSQLDUMP_OPTS="--ignore-table=db.table", but this was lost in the Go rewrite (v1.0+). The underlyingmysql.Data.IgnoreTablesfield andisIgnoredTable()method already existed but were never connected to any user-facing configuration. This was reported in #309.DB_DUMP_EXCLUDE: The exclude feature was documented and accepted via CLI/env vars, but the filtering logic was never implemented in
pkg/core/dump.go. Theopts.Excludefield was set but never read, causing disk-full incidents in production.Usage
Exclude databases
Ignore tables
Both qualified (
database.table) and unqualified (table) formats are supported:backuppc.hosts— ignoreshostsonly in thebackuppcdatabasehosts— ignoreshostsin all databasesTest plan
cmd/dump_test.gofor--exclude: single, comma-separated, multiple flagspkg/core/dump_test.goforfilterExcludedDatabases()cmd/dump_test.gofor--ignore-tablespkg/database/mysql/dump_test.goforisIgnoredTable()Closes #309