Conversation
whitehawk
force-pushed
the
ADBDEV-7984
branch
2 times, most recently
from
efefe82 to
d4a9c4d
Compare
List of changes: Implement redo module for orphaned files removal (#1414) 1. Implement redo module for pending deletes. This module is responsible for operations that are related to redo process: - Inserting XLOG_PENDING_DELETE xlog record in xlog when checkpointer requests it. - Parsing XLOG_SMGR_CREATE xlog to retrieve relfilenodes, that should be added to pending deletes hash table on redo. - Replaying XLOG_PENDING_DELETE by adding items to pending deletes redo hash table. - Removing nodes from pending deletes redo hash table for committed or aborted transactions. - Dropping orphaned files basing on redo hash table with pending deletes in the end of the recovery process. 2. Add unit test for the module. 3. Add GUC 'gp_track_pending_delete'. Ticket: ADBDEV-7304 Implement storage pending deletes module (#1460) The module contains functions to maintain doubly linked lists of (RelFileNodePendingDelete, transaction id) pairs for all backends in shared memory. The shared memory can be initialized using the PdlShmemSize and PdlShmemInit functions. Backend can add and remove pairs to its own list using PdlShmemAdd and PdlShmemRemove respectively. The backends lists can be got in the format suitable for XLOG_PENDING_DELETE using PdlXLogShmemDump. When backend stops, the module cleanups its pending deletes list. The size argument is removed from PdlXLogShmemDump, because this value can be calculated by caller using the function return value. Ticket: ADBDEV-7303 Support new XLOG_SMGR_CREATE_PDL WAL record (#1497) Problem description: XLOG_SMGR_CREATE WAL record doesn't contain information about relstorage for the created relation. Orphaned files removal feature requires knowing the relstorage info, otherwise it can't properly handle the removal of all orphaned files for AO tables. Fix: In order to store relation's relstorage in xlog record and keep backward compatibility with previous versions we introduce a new xlog record type XLOG_SMGR_CREATE_PDL. This new record type contains info about relstorage and is used instead of XLOG_SMGR_CREATE. Plus, this patch updates 'log_smgrcreate()' - now it creates XLOG_SMGR_CREATE_PDL record and flushes it right after the creation (otherwise, in case of a crash after file creation, the file may be orphaned). No special tests are presented in this patch, as the added functionality will be tested later together with other parts for the orphaned file removal feature. At this point, it is enough to pass the current standard test set. Integrate storage_pending_deletes module into the core code (#1520) Extend the PendingRelDelete structure. Now it stores the shmemPtr pointer to the corresponding shared pending deletes list node. The pointer is filled in RelationCreateStorage as a return value of PdlShmemAdd. The pointer is set as invalid in RelationDropStorage, because storage dropping can't lead to orphaned relfilenode. Replace pfree for the pendingDeletes list entry with a new PendingRelDeleteFree function which calls PdlShmemRemove before pfree when shmemPtr is valid. Add initialization of shared memory for the storage_pending_deletes module. Increase number of LWLocks by MaxBackends to add LWLocks for the module. Fix Assert in the dsm_create function, because the module is used in the stand-alone mode, for example, when initdb runs 'postgres --single'. No special tests are presented in this patch, as the added functionality will be tested later together with other parts for the orphaned file removal feature. At this point, it is enough to pass the current standard test set. Ticket: ADBDEV-7410 Support XLOG_PENDING_DELETE record in xlog_desc() (#1536) Tests are not added, because xlog_desc is used for debug purposes only. Ticket: ADBDEV-7409 The XLOG_SMGR_CREATE_PDL WAL record should be always linked to XID (#1547) When a table was created right after transaction beginning, then the XLOG_SMGR_CREATE_PDL record was added with InvalidTransactionId, because XLogInsert calls GetCurrentTransactionIdIfAny and the transaction id has not been got at this moment. Get the transaction id before the log_smgrcreate function is called. No special tests are presented in this patch, as the added functionality will be tested later together with other parts for the orphaned file removal feature. At this point, it is enough to pass the current standard test set. Ticket: ADBDEV-7458 Process XLOG_PENDING_DELETE record on Standby (#1569) Problem description: After following scenario: 1. primary segment started a transaction; 2. primary segment created a relation; 3. primary segment did a checkpoint; 4. mirror created a restartpoint by timeout; 5. both primary and mirror crashed and, then, recovered; primary has removed the orphaned file, but mirror didn't. Root cause: On step 5, mirror started replaying WAL from the restartpoint, so it didn't meet the XLOG_SMGR_CREATE_PDL record (which was at the moment of table creation, before the restartpoint). The information about the table's relfilenode is also stored in the XLOG_PENDING_DELETE WAL record. But the mirror skipped the processing of the XLOG_PENDING_DELETE record. Fix: Enable processing of the XLOG_PENDING_DELETE record by the mirror. This record is created on each checkpoint, so if the mirror starts from a restartpoint, it will be one of the first records to replay. Mirror will obtain the relfilenode information from this record and will remove the orphaned file. Enable orphaned files removal (#1478) This patch: - Adds clean up of orphaned files into `StartupXLOG()`. - Adds function RemovePendingDeletesForPreparedTransactions(), which removes prepared transactions xids from redo pending deletes before the orphaned files cleanup is performed. Orphaned files removal feature should consider prepared transactions xids and remove them from redo pending deletes, because otherwise some crash scenarios (for ex. 'crash_recovery_dtm' isolation2 test) would drop files, that shouldn't be dropped. No special tests are added. For now, it is enough to pass the standard test set. Plus, abi-check is fixed. Add tests for orphaned files (#1556) The tests check that orphaned files are not left for all access methods, before and after checkpoint, on segments and on coordinator. Ticket: ADBDEV-7305 Add test cases for orphaned files removal when mirror promotion happens (#1737) This patch introduces additional test cases for the orphaned files removal feature, when primary segment goes down completely without immediate crash recovery, and mirror promotion happens. Plus, this patch updates the ignore file for abi check, as the baseline has changed. Co-authored-by: Andrey Sokolov <a.sokolov@arenadata.io> Co-authored-by: Roman Eskin <r.eskin@arenadata.io> (cherry picked from commit 3ef385f) (cherry picked from commit 7804831) (cherry picked from commit ceea92e)
whitehawk
force-pushed
the
ADBDEV-7984
branch
from
d4a9c4d to
7ada356
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
ADBDEV-7984 - DRAFT for CI