Skip to content

feat: add --ignore-tables flag and fix broken DB_DUMP_EXCLUDE#512

Open
michabbb wants to merge 3 commits intodatabacker:masterfrom
michabbb:feature/ignore-tables
Open

feat: add --ignore-tables flag and fix broken DB_DUMP_EXCLUDE#512
michabbb wants to merge 3 commits intodatabacker:masterfrom
michabbb:feature/ignore-tables

Conversation

@michabbb
Copy link
Copy Markdown

@michabbb michabbb commented Mar 26, 2026

Summary

New feature: --ignore-tables

  • Wires up the existing but unused IgnoreTables support in pkg/database/mysql/dump.go to the CLI and environment variables
  • Adds --ignore-tables CLI flag (auto-binds to DB_DUMP_IGNORE_TABLES env var via viper)
  • Threads IgnoreTables []string through all intermediate layers (DumpOptsDumpOptionsdatabase.DumpOptsmysql.Data)
  • Fixes isIgnoredTable() to support qualified database.table format (e.g. mydb.mytable) in addition to bare table names

Bug fix: DB_DUMP_EXCLUDE / --exclude was completely broken

  • The Exclude field in DumpOptions was populated from CLI/env vars but never read in the Dump() function (pkg/core/dump.go)
  • All databases were dumped regardless of the exclude list — the field was dead code
  • Extracted filterExcludedDatabases() function and applied it after schema discovery
  • Fixed documentation typo (space-separated → comma-separated format)

Motivation

--ignore-tables: The old shell-based image supported MYSQLDUMP_OPTS="--ignore-table=db.table", but this was lost in the Go rewrite (v1.0+). The underlying mysql.Data.IgnoreTables field and isIgnoredTable() method already existed but were never connected to any user-facing configuration. This was reported in #309.

DB_DUMP_EXCLUDE: The exclude feature was documented and accepted via CLI/env vars, but the filtering logic was never implemented in pkg/core/dump.go. The opts.Exclude field was set but never read, causing disk-full incidents in production.

Usage

Exclude databases

# CLI flag
dump --exclude=tempdb,archivedb

# Environment variable
DB_DUMP_EXCLUDE=tempdb,archivedb

Ignore tables

# CLI flag
dump --ignore-tables=mydb.mytable,otherdb.bigtable

# or multiple flags
dump --ignore-tables=mydb.mytable --ignore-tables=otherdb.bigtable

# Environment variable (e.g. docker-compose)
DB_DUMP_IGNORE_TABLES=mydb.mytable,otherdb.bigtable

Both qualified (database.table) and unqualified (table) formats are supported:

  • backuppc.hosts — ignores hosts only in the backuppc database
  • hosts — ignores hosts in all databases

Test plan

  • Added 3 test cases to cmd/dump_test.go for --exclude: single, comma-separated, multiple flags
  • Added 7 test cases in pkg/core/dump_test.go for filterExcludedDatabases()
  • Added 3 test cases to cmd/dump_test.go for --ignore-tables
  • Added 10 test cases in pkg/database/mysql/dump_test.go for isIgnoredTable()
  • All existing tests pass
  • Verified with real MySQL: DB_DUMP_EXCLUDE now correctly excludes databases from the dump

Closes #309

Copilot AI review requested due to automatic review settings March 26, 2026 14:33
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR exposes MySQL dump table-ignoring functionality via CLI/env configuration and ensures the MySQL dumper correctly matches both qualified and unqualified ignore-table inputs.

Changes:

  • Add --ignore-tables CLI flag (Viper-backed env var support) and thread IgnoreTables []string through core/database option layers down into the MySQL dumper.
  • Extend mysql.Data.isIgnoredTable() to support qualified database.table entries alongside bare table names.
  • Add unit tests for CLI parsing/binding and isIgnoredTable() matching behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
cmd/dump.go Adds --ignore-tables flag and passes parsed ignore list into core.DumpOptions.
cmd/dump_test.go Adds test cases for single, comma-separated, and repeated --ignore-tables usage.
pkg/core/dumpoptions.go Adds IgnoreTables to core.DumpOptions.
pkg/core/dump.go Threads IgnoreTables into database.DumpOpts.
pkg/database/dump.go Adds IgnoreTables to database.DumpOpts and passes it into mysql.Data.
pkg/database/mysql/dump.go Updates ignore matching to handle qualified schema.table entries.
pkg/database/mysql/dump_test.go Adds tests covering qualified/unqualified ignore-table matching.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cmd/dump.go Outdated
Comment on lines +375 to +376
// ignore-tables: tables to exclude from the dump (format: database.table)
flags.StringSlice("ignore-tables", []string{}, "Tables to exclude from the dump. Format: database.table (e.g. mydb.mytable). Can be specified multiple times or as a comma-separated list.")
Copy link

Copilot AI Mar 26, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The --ignore-tables help text/documentation implies only database.table values are supported, but isIgnoredTable() also supports unqualified table names (applies to all schemas). Please update the flag comment/description to mention both formats so CLI --help matches actual behavior.

Suggested change
// ignore-tables: tables to exclude from the dump (format: database.table)
flags.StringSlice("ignore-tables", []string{}, "Tables to exclude from the dump. Format: database.table (e.g. mydb.mytable). Can be specified multiple times or as a comma-separated list.")
// ignore-tables: tables to exclude from the dump (formats: database.table or table)
flags.StringSlice("ignore-tables", []string{}, "Tables to exclude from the dump. Formats: database.table (e.g. mydb.mytable) or table (applies to all databases/schemas). Can be specified multiple times or as a comma-separated list.")

Copilot uses AI. Check for mistakes.
deitch
deitch previously approved these changes Mar 27, 2026
Copy link
Copy Markdown
Collaborator

@deitch deitch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Solid PR, thank you.

@deitch
Copy link
Copy Markdown
Collaborator

deitch commented Mar 27, 2026

The commits need to be signed, see this error. Kindly sign them and repush; as soon as that is done and CI is green, we can merge it in.

@michabbb michabbb force-pushed the feature/ignore-tables branch from cf512c2 to 782b3a2 Compare March 27, 2026 11:26
@michabbb michabbb changed the title feat: add --ignore-tables flag to exclude tables from dump feat: add --ignore-tables flag and fix broken DB_DUMP_EXCLUDE Mar 27, 2026
@michabbb
Copy link
Copy Markdown
Author

@deitch is also fixed the missing code for DB_DUMP_EXCLUDE 😏

@deitch
Copy link
Copy Markdown
Collaborator

deitch commented Mar 27, 2026

Thanks. It still needs signed commits (see previous comment). And it is out of date, needs a rebase.

The mysql dump library already had IgnoreTables support (Data.IgnoreTables
field and isIgnoredTable() method), but it was never exposed to users.

This commit threads IgnoreTables through all layers:
- pkg/database/dump.go: add IgnoreTables to DumpOpts, pass to mysql.Data
- pkg/core/dumpoptions.go: add IgnoreTables to DumpOptions
- pkg/core/dump.go: pass IgnoreTables to database.DumpOpts
- cmd/dump.go: add --ignore-tables CLI flag (auto-binds to
  DB_DUMP_IGNORE_TABLES env var via viper)

Also fixes isIgnoredTable() to support qualified "database.table" format
(e.g. "mydb.mytable") in addition to bare table names, so tables can be
excluded per-database rather than globally.

Usage:
  CLI: --ignore-tables=mydb.mytable,otherdb.bigtable
  ENV: DB_DUMP_IGNORE_TABLES=mydb.mytable,otherdb.bigtable

Closes databacker#309

Signed-off-by: micha <mbladowski@macropage.de>
… formats

Signed-off-by: micha <mbladowski@macropage.de>
…broken

The Exclude field in DumpOptions was populated from CLI/env but never
read in the Dump() function, causing all databases to be dumped
regardless of the exclude list. Extract filterExcludedDatabases() and
apply it after schema discovery. Add unit and cmd tests for exclude.

Signed-off-by: micha <mbladowski@macropage.de>
@michabbb michabbb force-pushed the feature/ignore-tables branch from 63b86e7 to 8713eae Compare March 27, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

How to ignore tables? in 1.0.4rc?

3 participants