diff options
Diffstat (limited to 'src/mongo/db/repl')
106 files changed, 725 insertions, 3030 deletions
diff --git a/src/mongo/db/repl/README.md b/src/mongo/db/repl/README.md index 08af0e4073a..c0526e9d966 100644 --- a/src/mongo/db/repl/README.md +++ b/src/mongo/db/repl/README.md @@ -361,8 +361,6 @@ If the server does not support `hello`, the `helloOk` flag is ignored. A new dri not see "helloOk: true" in the response and continue to send `isMaster` on this connection. Old drivers will not specify this flag at all, so the behavior remains the same. -Communication between nodes in the cluster is always done using `hello`, never with `isMaster`. - ## Communication Each node has a copy of the **`ReplicaSetConfig`** in the `ReplicationCoordinator` that lists all @@ -510,7 +508,7 @@ assigns itself a priority takeover timeout proportional to its rank. After that node will check if it's eligible to run for election and if so will begin an election. The timeout is simply: `(election timeout) * (priority rank + 1)`. -Heartbeat threads belong to the +Heartbeat threads belong to the [`ReplCoordThreadPool`](https://github.com/mongodb/mongo/blob/674d57fc70d80dedbfd634ce00ca4b967ea89646/src/mongo/db/mongod_main.cpp#L944) connection pool started by the [`ReplicationCoordinator`](https://github.com/mongodb/mongo/blob/674d57fc70d80dedbfd634ce00ca4b967ea89646/src/mongo/db/mongod_main.cpp#L986). @@ -1642,15 +1640,15 @@ The `initialSyncTransientErrorRetryPeriodSeconds` is also used to control retrie fetcher and all network operations in initial sync which take place after the data cloning has started. -As of v4.4, initial syncing a node with [two-phase index builds](https://github.com/mongodb/mongo/blob/0a7641e69031fcfdf25a1780a3b62bca5f59d68f/src/mongo/db/catalog/README.md#replica-set-index-builds) -will immediately build all ready indexes from the sync source and setup the index builder threads -for any unfinished index builds. -[See here](https://github.com/mongodb/mongo/blob/85d75907fd12c2360cf16b97f941386f343ca6fc/src/mongo/db/repl/collection_cloner.cpp#L247-L301). +As of v4.4, initial syncing a node with [two-phase index builds](https://github.com/mongodb/mongo/blob/0a7641e69031fcfdf25a1780a3b62bca5f59d68f/src/mongo/db/catalog/README.md#replica-set-index-builds) +will immediately build all ready indexes from the sync source and setup the index builder threads +for any unfinished index builds. +[See here](https://github.com/mongodb/mongo/blob/85d75907fd12c2360cf16b97f941386f343ca6fc/src/mongo/db/repl/collection_cloner.cpp#L247-L301). -This is necessary to avoid a scenario where the primary node cannot satisfy the index builds commit -quorum if it depends on the initial syncing nodes vote. Prior to this, initial syncing nodes would -start the index build when they came across the `commitIndexBuild` oplog entry, which is only -observable once the index builds commit quorum has been satisfied. +This is necessary to avoid a scenario where the primary node cannot satisfy the index builds commit +quorum if it depends on the initial syncing nodes vote. Prior to this, initial syncing nodes would +start the index build when they came across the `commitIndexBuild` oplog entry, which is only +observable once the index builds commit quorum has been satisfied. [See this test for an example](https://github.com/mongodb/mongo/blob/f495bdead326a06a76f8a980e44092deb096a21d/jstests/noPassthrough/commit_quorum_does_not_hang_with_initial_sync.js). ## Oplog application phase @@ -1899,7 +1897,7 @@ Once the files are moved, we switch storage one more time, back to the original now has the downloaded files. We use the `InitialSyncFileMover` to delete the move marker and the entire .initialsync directory; a restart of the server at or after this point will not involve any more FCBIS work. Then we release the global lock, and retrieve the last applied -OpTime and WallTime from the top of the oplog (the latest entry in the oplog) in +OpTime and WallTime from the top of the oplog (the latest entry in the oplog) in `_updateLastAppliedOptime`. The initial sync attempt is now considered successful, and we call `_finishCallback`. This acts @@ -2210,7 +2208,7 @@ value has yet to be loaded from disk, the FCV is set to `kUnsetDefaultLastLTSBeh indicates that the server will be using the last-LTS feature set as to ensure compatibility with other nodes in the replica set. -As part of initial sync, the in-memory FCV value is always initially set to be +As part of initial sync, the in-memory FCV value is always initially set to be `kUnsetDefaultLastLTSBehavior`. This is to ensure compatibility between the sync source and sync target. If the sync source is actually in a different feature compatibility version, we will find out when we clone the `admin.system.version` collection. @@ -2221,7 +2219,7 @@ if it has not yet received the `replSetInitiate` command. ## setFeatureCompatibilityVersion The FCV can be set using the `setFeatureCompatibilityVersion` admin command to one of the following: -* The version of the last-LTS (Long Term Support) +* The version of the last-LTS (Long Term Support) * Indicates to the server to use the feature set compatible with the last LTS release version. * The version of the last-continuous release * Indicates to the server to use the feature set compatible with the last continuous release @@ -2265,8 +2263,8 @@ kUpgradingFrom_5_0_To_5_1: } kDowngradingFrom_5_1_To_5_0: -{ - version: 5.0, +{ + version: 5.0, targetVersion: 5.0, previousVersion: 5.1 } @@ -2329,11 +2327,11 @@ transaction. For a prepared transaction, we have the following guarantee: `prepa **`currentCommittedSnapshot`**: An optime maintained in `ReplicationCoordinator` that is used to serve majority reads and is always guaranteed to be <= `lastCommittedOpTime`. When `eMRC=true`, this -is currently [set to the stable optime](https://github.com/mongodb/mongo/blob/00fbc981646d9e6ebc391f45a31f4070d4466753/src/mongo/db/repl/replication_coordinator_impl.cpp#L4945). +is currently [set to the stable optime](https://github.com/mongodb/mongo/blob/00fbc981646d9e6ebc391f45a31f4070d4466753/src/mongo/db/repl/replication_coordinator_impl.cpp#L4945). Since it is reset every time we recalculate the stable optime, it will also be up to date. -When `eMRC=false`, this [is set](https://github.com/mongodb/mongo/blob/00fbc981646d9e6ebc391f45a31f4070d4466753/src/mongo/db/repl/replication_coordinator_impl.cpp#L4952-L4961) -to the minimum of the stable optime and the `lastCommittedOpTime`, even though it is not used to +When `eMRC=false`, this [is set](https://github.com/mongodb/mongo/blob/00fbc981646d9e6ebc391f45a31f4070d4466753/src/mongo/db/repl/replication_coordinator_impl.cpp#L4952-L4961) +to the minimum of the stable optime and the `lastCommittedOpTime`, even though it is not used to serve majority reads in that case. **`initialDataTimestamp`**: A timestamp used to indicate the timestamp at which history “begins”. @@ -2388,7 +2386,7 @@ populated internally from the `currentCommittedSnapshot` timestamp inside `Repli **`stable_timestamp`**: The newest timestamp at which the storage engine is allowed to take a checkpoint, which can be thought of as a consistent snapshot of the data. Replication informs the storage engine of where it is safe to take its next checkpoint. This timestamp is guaranteed to be -majority committed so that RTT rollback can use it. In the case when +majority committed so that RTT rollback can use it. In the case when [`eMRC=false`](#enableMajorityReadConcern-flag), the stable timestamp may not be majority committed, which is why we must use the Rollback via Refetch rollback algorithm. diff --git a/src/mongo/db/repl/SConscript b/src/mongo/db/repl/SConscript index bf3ff27ed2e..10783f244cd 100644 --- a/src/mongo/db/repl/SConscript +++ b/src/mongo/db/repl/SConscript @@ -40,16 +40,6 @@ env.Library( ) env.Library( - target='oplog_constraint_violation_logger', - source=[ - 'oplog_constraint_violation_logger.cpp', - ], - LIBDEPS_PRIVATE=[ - '$BUILD_DIR/mongo/base', - ], -) - -env.Library( target='oplog', source=[ 'apply_ops.cpp', @@ -65,13 +55,11 @@ env.Library( '$BUILD_DIR/mongo/base', '$BUILD_DIR/mongo/db/catalog/catalog_helpers', '$BUILD_DIR/mongo/db/catalog/database_holder', - '$BUILD_DIR/mongo/db/catalog/health_log_interface', '$BUILD_DIR/mongo/db/catalog/import_collection_oplog_entry', '$BUILD_DIR/mongo/db/catalog/index_build_oplog_entry', '$BUILD_DIR/mongo/db/catalog/local_oplog_info', '$BUILD_DIR/mongo/db/catalog/multi_index_block', '$BUILD_DIR/mongo/db/commands/feature_compatibility_parsers', - '$BUILD_DIR/mongo/db/concurrency/exception_util', '$BUILD_DIR/mongo/db/db_raii', '$BUILD_DIR/mongo/db/dbdirectclient', '$BUILD_DIR/mongo/db/dbhelpers', @@ -88,7 +76,6 @@ env.Library( '$BUILD_DIR/mongo/rpc/command_status', 'dbcheck', 'image_collection_entry', - 'oplog_constraint_violation_logger', 'repl_coordinator_interface', 'repl_server_parameters', 'repl_settings', @@ -130,7 +117,7 @@ env.Library( '$BUILD_DIR/mongo/db/concurrency/lock_manager', '$BUILD_DIR/mongo/db/db_raii', '$BUILD_DIR/mongo/db/dbhelpers', - '$BUILD_DIR/mongo/db/index/index_access_method', + '$BUILD_DIR/mongo/db/index/index_descriptor', '$BUILD_DIR/mongo/db/query_exec', '$BUILD_DIR/mongo/util/fail_point', 'oplog', @@ -150,6 +137,7 @@ env.Library( LIBDEPS=[ '$BUILD_DIR/mongo/client/connection_pool', '$BUILD_DIR/mongo/client/fetcher', + '$BUILD_DIR/mongo/db/concurrency/write_conflict_exception', '$BUILD_DIR/mongo/db/index_builds_coordinator_interface', '$BUILD_DIR/mongo/db/service_context', '$BUILD_DIR/mongo/util/concurrency/thread_pool', @@ -167,7 +155,6 @@ env.Library( 'sync_source_resolver', ], LIBDEPS_PRIVATE=[ - '$BUILD_DIR/mongo/db/concurrency/exception_util', '$BUILD_DIR/mongo/db/op_observer', '$BUILD_DIR/mongo/executor/thread_pool_task_executor', 'repl_server_parameters', @@ -208,8 +195,7 @@ env.Library( ], LIBDEPS=[ '$BUILD_DIR/mongo/base', - '$BUILD_DIR/mongo/bson/bson_validate', - '$BUILD_DIR/mongo/db/catalog/health_log_interface', + '$BUILD_DIR/mongo/db/catalog/health_log', '$BUILD_DIR/mongo/db/db_raii', '$BUILD_DIR/mongo/idl/idl_parser', ], @@ -218,7 +204,6 @@ env.Library( '$BUILD_DIR/mongo/db/query_exec', '$BUILD_DIR/mongo/db/record_id_helpers', '$BUILD_DIR/mongo/util/md5', - 'repl_server_parameters', ], ) @@ -261,7 +246,6 @@ env.Library( '$BUILD_DIR/mongo/db/catalog/database_holder', '$BUILD_DIR/mongo/db/catalog/multi_index_block', '$BUILD_DIR/mongo/db/common', - '$BUILD_DIR/mongo/db/concurrency/exception_util', '$BUILD_DIR/mongo/db/dbhelpers', '$BUILD_DIR/mongo/db/index_builds_coordinator_interface', '$BUILD_DIR/mongo/db/multitenancy', @@ -603,7 +587,8 @@ env.Library( LIBDEPS=[ '$BUILD_DIR/mongo/db/auth/authorization_manager_global', '$BUILD_DIR/mongo/db/concurrency/lock_manager', - '$BUILD_DIR/mongo/db/query/query_stats/query_stats', + '$BUILD_DIR/mongo/db/concurrency/write_conflict_exception', + '$BUILD_DIR/mongo/db/curop', '$BUILD_DIR/mongo/db/query_exec', '$BUILD_DIR/mongo/db/stats/timer_stats', '$BUILD_DIR/mongo/db/storage/storage_options', @@ -620,8 +605,6 @@ env.Library( ], LIBDEPS_PRIVATE=[ '$BUILD_DIR/mongo/db/commands/mongod_fsync', - '$BUILD_DIR/mongo/db/concurrency/exception_util', - '$BUILD_DIR/mongo/db/curop_metrics', '$BUILD_DIR/mongo/db/storage/storage_control', 'repl_server_parameters', 'replication_auth', @@ -718,7 +701,7 @@ env.Library( '$BUILD_DIR/mongo/db/commands/mongod_fcv', '$BUILD_DIR/mongo/db/common', '$BUILD_DIR/mongo/db/concurrency/lock_manager', - '$BUILD_DIR/mongo/db/index/index_access_method', + '$BUILD_DIR/mongo/db/index/index_descriptor', '$BUILD_DIR/mongo/db/kill_sessions_local', '$BUILD_DIR/mongo/db/mongod_options', '$BUILD_DIR/mongo/db/prepare_conflict_tracker', @@ -748,7 +731,6 @@ env.Library( '$BUILD_DIR/mongo/db/index_builds_coordinator_interface', '$BUILD_DIR/mongo/db/storage/journal_flusher', '$BUILD_DIR/mongo/idl/server_parameter', - 'delayable_timeout_callback', 'repl_server_parameters', 'replica_set_aware_service', 'split_horizon', @@ -1067,7 +1049,6 @@ env.Library( '$BUILD_DIR/mongo/db/catalog/document_validation', '$BUILD_DIR/mongo/db/commands/list_collections_filter', '$BUILD_DIR/mongo/db/ops/write_ops_exec', - '$BUILD_DIR/mongo/idl/cluster_parameter_synchronization_helpers', '$BUILD_DIR/mongo/rpc/metadata', '$BUILD_DIR/mongo/util/progress_meter', 'oplog', @@ -1360,7 +1341,6 @@ env.Library( "tenant_migration_recipient_entry_helpers.cpp", ], LIBDEPS=[ - '$BUILD_DIR/mongo/db/keys_collection_util', '$BUILD_DIR/mongo/util/future_util', 'repl_coordinator_interface', 'repl_server_parameters', @@ -1369,7 +1349,6 @@ env.Library( LIBDEPS_PRIVATE=[ "$BUILD_DIR/mongo/base", "$BUILD_DIR/mongo/db/catalog_raii", - "$BUILD_DIR/mongo/db/concurrency/exception_util", "$BUILD_DIR/mongo/db/dbhelpers", "$BUILD_DIR/mongo/db/namespace_string", '$BUILD_DIR/mongo/db/service_context', @@ -1399,14 +1378,12 @@ env.Library( ], LIBDEPS_PRIVATE=[ '$BUILD_DIR/mongo/client/clientdriver_network', - '$BUILD_DIR/mongo/db/concurrency/exception_util', '$BUILD_DIR/mongo/db/multitenancy', '$BUILD_DIR/mongo/db/ops/write_ops_exec', '$BUILD_DIR/mongo/db/pipeline/process_interface/mongo_process_interface', '$BUILD_DIR/mongo/db/storage/wiredtiger/storage_wiredtiger_import', '$BUILD_DIR/mongo/db/transaction', '$BUILD_DIR/mongo/executor/scoped_task_executor', - '$BUILD_DIR/mongo/idl/cluster_parameter_synchronization_helpers', 'cloner_utils', 'oplog', 'oplog_application_interface', @@ -1452,7 +1429,6 @@ env.Library( ], LIBDEPS_PRIVATE=[ '$BUILD_DIR/mongo/db/catalog/local_oplog_info', - '$BUILD_DIR/mongo/db/concurrency/exception_util', ], ) @@ -1471,7 +1447,6 @@ env.Library( ], LIBDEPS_PRIVATE=[ "$BUILD_DIR/mongo/db/catalog/local_oplog_info", - "$BUILD_DIR/mongo/db/concurrency/exception_util", "$BUILD_DIR/mongo/db/index_builds_coordinator_interface", ], ) @@ -1489,11 +1464,12 @@ env.Library( '$BUILD_DIR/mongo/db/auth/auth', '$BUILD_DIR/mongo/db/cloner', '$BUILD_DIR/mongo/db/concurrency/lock_manager', + '$BUILD_DIR/mongo/db/curop', + '$BUILD_DIR/mongo/db/free_mon/free_mon_mongod', '$BUILD_DIR/mongo/db/kill_sessions_local', '$BUILD_DIR/mongo/db/logical_time', '$BUILD_DIR/mongo/db/not_primary_error_tracker', '$BUILD_DIR/mongo/db/op_observer', - '$BUILD_DIR/mongo/db/query/query_stats/query_stats', '$BUILD_DIR/mongo/db/query_exec', '$BUILD_DIR/mongo/db/service_context', '$BUILD_DIR/mongo/db/stats/counters', @@ -1576,8 +1552,6 @@ if wiredtiger: LIBDEPS=[ '$BUILD_DIR/mongo/db/catalog/database_holder', '$BUILD_DIR/mongo/db/catalog/document_validation', - '$BUILD_DIR/mongo/db/catalog/health_log', - '$BUILD_DIR/mongo/db/catalog/health_log_interface', '$BUILD_DIR/mongo/db/index_builds_coordinator_interface', '$BUILD_DIR/mongo/db/multitenancy', '$BUILD_DIR/mongo/db/service_context_d_test_fixture', @@ -1638,7 +1612,6 @@ if wiredtiger: 'abstract_async_component_test.cpp', 'apply_ops_test.cpp', 'check_quorum_for_config_change_test.cpp', - 'delayable_timeout_callback_test.cpp', 'drop_pending_collection_reaper_test.cpp', 'idempotency_document_structure_test.cpp', 'idempotency_update_sequence_test.cpp', @@ -1694,13 +1667,12 @@ if wiredtiger: '$BUILD_DIR/mongo/db/auth/authmocks', '$BUILD_DIR/mongo/db/auth/authorization_manager_global', '$BUILD_DIR/mongo/db/catalog/catalog_helpers', - '$BUILD_DIR/mongo/db/catalog/health_log', '$BUILD_DIR/mongo/db/catalog_raii', '$BUILD_DIR/mongo/db/commands/feature_compatibility_parsers', '$BUILD_DIR/mongo/db/commands/mongod_fcv', '$BUILD_DIR/mongo/db/commands/txn_cmd_request', '$BUILD_DIR/mongo/db/dbdirectclient', - '$BUILD_DIR/mongo/db/index/index_access_method', + '$BUILD_DIR/mongo/db/index/index_access_methods', '$BUILD_DIR/mongo/db/index_build_entry_helpers', '$BUILD_DIR/mongo/db/index_builds_coordinator_mongod', '$BUILD_DIR/mongo/db/logical_session_id_helpers', @@ -1729,7 +1701,6 @@ if wiredtiger: '$BUILD_DIR/mongo/util/concurrency/thread_pool', 'abstract_async_component', 'data_replicator_external_state_mock', - 'delayable_timeout_callback', 'drop_pending_collection_reaper', 'idempotency_test_fixture', 'idempotency_test_util', @@ -1984,7 +1955,7 @@ env.Library( ], LIBDEPS_PRIVATE=[ '$BUILD_DIR/mongo/base', - '$BUILD_DIR/mongo/db/concurrency/exception_util', + '$BUILD_DIR/mongo/db/concurrency/write_conflict_exception', 'abstract_async_component', 'cloner_utils', 'oplog', @@ -2006,14 +1977,3 @@ env.Library( 'oplog_entry', ], ) - -env.Library( - target='delayable_timeout_callback', - source=[ - 'delayable_timeout_callback.cpp', - ], - LIBDEPS=[ - '$BUILD_DIR/mongo/base', - '$BUILD_DIR/mongo/executor/task_executor_interface', - ], -) diff --git a/src/mongo/db/repl/all_database_cloner.cpp b/src/mongo/db/repl/all_database_cloner.cpp index 6f6d73cefec..b5856b7aef0 100644 --- a/src/mongo/db/repl/all_database_cloner.cpp +++ b/src/mongo/db/repl/all_database_cloner.cpp @@ -59,17 +59,16 @@ BaseCloner::ClonerStages AllDatabaseCloner::getStages() { } Status AllDatabaseCloner::ensurePrimaryOrSecondary( - const executor::RemoteCommandResponse& helloReply) { - if (!helloReply.isOK()) { - LOGV2(21054, "Cannot reconnect because 'hello' command failed"); - return helloReply.status; + const executor::RemoteCommandResponse& isMasterReply) { + if (!isMasterReply.isOK()) { + LOGV2(21054, "Cannot reconnect because isMaster command failed"); + return isMasterReply.status; } - if (helloReply.data["isWritablePrimary"].trueValue() || - helloReply.data["secondary"].trueValue()) + if (isMasterReply.data["ismaster"].trueValue() || isMasterReply.data["secondary"].trueValue()) return Status::OK(); // There is a window during startup where a node has an invalid configuration and will have - // an "hello" response the same as a removed node. So we must check to see if the node is + // an isMaster response the same as a removed node. So we must check to see if the node is // removed by checking local configuration. auto memberData = ReplicationCoordinator::get(getGlobalServiceContext())->getMemberData(); auto syncSourceIter = std::find_if( @@ -110,8 +109,8 @@ BaseCloner::AfterStageBehavior AllDatabaseCloner::connectStage() { // handle the reconnect itself. This is necessary to get correct backoff behavior. if (client->getServerHostAndPort() != getSource()) { client->setHandshakeValidationHook( - [this](const executor::RemoteCommandResponse& helloReply) { - return ensurePrimaryOrSecondary(helloReply); + [this](const executor::RemoteCommandResponse& isMasterReply) { + return ensurePrimaryOrSecondary(isMasterReply); }); uassertStatusOK(client->connect(getSource(), StringData(), boost::none)); } else { diff --git a/src/mongo/db/repl/all_database_cloner.h b/src/mongo/db/repl/all_database_cloner.h index e61c09030b7..d538af3869c 100644 --- a/src/mongo/db/repl/all_database_cloner.h +++ b/src/mongo/db/repl/all_database_cloner.h @@ -82,7 +82,7 @@ private: * Validation function to ensure we connect only to primary or secondary nodes. * * Because the cloner connection is separate from the usual inter-node connection pool and - * did not have the 'hangUpOnStepDown:false' flag set in the initial "hello" request, we + * did not have the 'hangUpOnStepDown:false' flag set in the initial isMaster request, we * will always disconnect if the sync source transitions to a state other than PRIMARY * or SECONDARY. It will not disconnect on a PRIMARY to SECONDARY or SECONDARY to PRIMARY * transition because we no longer do that (the flag name is anachronistic). After @@ -99,7 +99,7 @@ private: * would succeed and we would have an inconsistent node. If other data was added we would * invariant during oplog application with a NamespaceNotFound error. */ - Status ensurePrimaryOrSecondary(const executor::RemoteCommandResponse& helloReply); + Status ensurePrimaryOrSecondary(const executor::RemoteCommandResponse& isMasterReply); /** * Stage function that makes a connection to the sync source. diff --git a/src/mongo/db/repl/apply_ops.cpp b/src/mongo/db/repl/apply_ops.cpp index e9ae0d4858c..5f71c23b3ab 100644 --- a/src/mongo/db/repl/apply_ops.cpp +++ b/src/mongo/db/repl/apply_ops.cpp @@ -39,8 +39,8 @@ #include "mongo/db/catalog/database_holder.h" #include "mongo/db/catalog/document_validation.h" #include "mongo/db/client.h" -#include "mongo/db/concurrency/exception_util.h" #include "mongo/db/concurrency/lock_state.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" diff --git a/src/mongo/db/repl/apply_ops_test.cpp b/src/mongo/db/repl/apply_ops_test.cpp index 1c0e3a1bee0..baf5a770d7a 100644 --- a/src/mongo/db/repl/apply_ops_test.cpp +++ b/src/mongo/db/repl/apply_ops_test.cpp @@ -346,7 +346,6 @@ OplogEntry makeOplogEntry(OpTypeEnum opType, NamespaceString("a.a"), // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version oField, // o boost::none, // o2 diff --git a/src/mongo/db/repl/bgsync.cpp b/src/mongo/db/repl/bgsync.cpp index fbc450e59b8..8217e38090a 100644 --- a/src/mongo/db/repl/bgsync.cpp +++ b/src/mongo/db/repl/bgsync.cpp @@ -41,8 +41,8 @@ #include "mongo/client/connection_pool.h" #include "mongo/db/auth/authorization_session.h" #include "mongo/db/client.h" -#include "mongo/db/concurrency/exception_util.h" #include "mongo/db/concurrency/replication_state_transition_lock_guard.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/index_builds_coordinator.h" #include "mongo/db/repl/data_replicator_external_state_impl.h" @@ -822,15 +822,15 @@ void BackgroundSync::_runRollbackViaRecoverToCheckpoint( if (_state != ProducerState::Running) { return; } - _rollback = std::make_unique<RollbackImpl>( - localOplog, &remoteOplog, storageInterface, _replicationProcess, _replCoord); } + _rollback = std::make_unique<RollbackImpl>( + localOplog, &remoteOplog, storageInterface, _replicationProcess, _replCoord); + LOGV2(21104, "Scheduling rollback (sync source: {syncSource})", "Scheduling rollback", "syncSource"_attr = source); - auto status = _rollback->runRollback(opCtx); if (status.isOK()) { LOGV2(21105, "Rollback successful"); diff --git a/src/mongo/db/repl/collection_bulk_loader_impl.cpp b/src/mongo/db/repl/collection_bulk_loader_impl.cpp index eb39fdadd55..c0b3365621e 100644 --- a/src/mongo/db/repl/collection_bulk_loader_impl.cpp +++ b/src/mongo/db/repl/collection_bulk_loader_impl.cpp @@ -39,7 +39,7 @@ #include "mongo/db/catalog/index_catalog.h" #include "mongo/db/client.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" #include "mongo/db/jsobj.h" #include "mongo/db/operation_context.h" @@ -82,54 +82,49 @@ Status CollectionBulkLoaderImpl::init(const std::vector<BSONObj>& secondaryIndex // locks as yielding a MODE_X/MODE_S lock isn't allowed. _secondaryIndexesBlock->setIndexBuildMethod(IndexBuildMethod::kForeground); _idIndexBlock->setIndexBuildMethod(IndexBuildMethod::kForeground); - return writeConflictRetry(_opCtx.get(), - "CollectionBulkLoader::init", - _collection->getNss().ns(), - [&secondaryIndexSpecs, this] { - WriteUnitOfWork wuow(_opCtx.get()); - // All writes in CollectionBulkLoaderImpl should be - // unreplicated. The opCtx is accessed indirectly through - // _secondaryIndexesBlock. - UnreplicatedWritesBlock uwb(_opCtx.get()); - // This enforces the buildIndexes setting in the replica set - // configuration. - CollectionWriter collWriter(_opCtx.get(), *_collection); - auto indexCatalog = - collWriter.getWritableCollection()->getIndexCatalog(); - auto specs = indexCatalog->removeExistingIndexesNoChecks( - _opCtx.get(), collWriter.get(), secondaryIndexSpecs); - if (specs.size()) { - _secondaryIndexesBlock->ignoreUniqueConstraint(); - auto status = _secondaryIndexesBlock - ->init(_opCtx.get(), - collWriter, - specs, - MultiIndexBlock::kNoopOnInitFn, - /*forRecovery=*/false) - .getStatus(); - if (!status.isOK()) { - return status; - } - } else { - _secondaryIndexesBlock.reset(); - } - if (!_idIndexSpec.isEmpty()) { - auto status = _idIndexBlock - ->init(_opCtx.get(), - collWriter, - _idIndexSpec, - MultiIndexBlock::kNoopOnInitFn) - .getStatus(); - if (!status.isOK()) { - return status; - } - } else { - _idIndexBlock.reset(); - } - - wuow.commit(); - return Status::OK(); - }); + return writeConflictRetry( + _opCtx.get(), + "CollectionBulkLoader::init", + _collection->getNss().ns(), + [&secondaryIndexSpecs, this] { + WriteUnitOfWork wuow(_opCtx.get()); + // All writes in CollectionBulkLoaderImpl should be unreplicated. + // The opCtx is accessed indirectly through _secondaryIndexesBlock. + UnreplicatedWritesBlock uwb(_opCtx.get()); + // This enforces the buildIndexes setting in the replica set configuration. + CollectionWriter collWriter(_opCtx.get(), *_collection); + auto indexCatalog = collWriter.getWritableCollection()->getIndexCatalog(); + auto specs = indexCatalog->removeExistingIndexesNoChecks( + _opCtx.get(), collWriter.get(), secondaryIndexSpecs); + if (specs.size()) { + _secondaryIndexesBlock->ignoreUniqueConstraint(); + auto status = + _secondaryIndexesBlock + ->init(_opCtx.get(), collWriter, specs, MultiIndexBlock::kNoopOnInitFn) + .getStatus(); + if (!status.isOK()) { + return status; + } + } else { + _secondaryIndexesBlock.reset(); + } + if (!_idIndexSpec.isEmpty()) { + auto status = _idIndexBlock + ->init(_opCtx.get(), + collWriter, + _idIndexSpec, + MultiIndexBlock::kNoopOnInitFn) + .getStatus(); + if (!status.isOK()) { + return status; + } + } else { + _idIndexBlock.reset(); + } + + wuow.commit(); + return Status::OK(); + }); }); } @@ -276,52 +271,26 @@ Status CollectionBulkLoaderImpl::commit() { if (_idIndexBlock) { // Do not do inside a WriteUnitOfWork (required by dumpInsertsFromBulk). auto status = _idIndexBlock->dumpInsertsFromBulk( - _opCtx.get(), **_collection, [&](const RecordId& rid) { - writeConflictRetry( + _opCtx.get(), _collection->getCollection(), [&](const RecordId& rid) { + return writeConflictRetry( _opCtx.get(), "CollectionBulkLoaderImpl::commit", _nss.ns(), [this, &rid] { WriteUnitOfWork wunit(_opCtx.get()); - - auto doc = (*_collection)->docFor(_opCtx.get(), rid); - - // Delete the document before committing the index. If we were to delete - // the document after committing the index, it's possible that the we - // may unindex a record with the same key but a different RecordId. - (*_collection)->getRecordStore()->deleteRecord(_opCtx.get(), rid); - - auto indexIt = - (*_collection) - ->getIndexCatalog() - ->getIndexIterator(_opCtx.get(), - IndexCatalog::InclusionPolicy::kReady); - while (auto entry = indexIt->next()) { - if (entry->descriptor()->isIdIndex()) { - continue; - } - - SharedBufferFragmentBuilder pooledBuilder{ - KeyString::HeapBuilder::kHeapAllocatorDefaultBytes}; - - InsertDeleteOptions options; - options.dupsAllowed = !entry->descriptor()->unique(); - - entry->accessMethod()->remove( - _opCtx.get(), - pooledBuilder, - **_collection, - doc.value(), - rid, - false /* logIfError */, - options, - nullptr /* numDeleted */, - // Initial sync can build an index over a collection with - // duplicates, so we need to check the RecordId of the docuemnt - // we are unindexing. See SERVER-17487 for more details. - CheckRecordId::On); - } - + // If we were to delete the document after committing the index build, + // it's possible that the storage engine unindexes a different record + // with the same key, but different RecordId. By deleting the document + // before committing the index build, the index removal code uses + // 'dupsAllowed', which forces the storage engine to only unindex + // records that match the same key and RecordId. + (*_collection) + ->deleteDocument(_opCtx.get(), + kUninitializedStmtId, + rid, + nullptr /** OpDebug **/, + false /* fromMigrate */, + true /* noWarn */); wunit.commit(); + return Status::OK(); }); - return Status::OK(); }); if (!status.isOK()) { return status; diff --git a/src/mongo/db/repl/collection_cloner.cpp b/src/mongo/db/repl/collection_cloner.cpp index ed2bccf8bb9..0066240cc2f 100644 --- a/src/mongo/db/repl/collection_cloner.cpp +++ b/src/mongo/db/repl/collection_cloner.cpp @@ -29,8 +29,6 @@ #define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kReplicationInitialSync -#include "mongo/db/index/index_descriptor_fwd.h" -#include "mongo/db/service_context.h" #include "mongo/platform/basic.h" #include "mongo/base/string_data.h" @@ -208,19 +206,8 @@ BaseCloner::AfterStageBehavior CollectionCloner::listIndexesStage() { "source"_attr = getSource()); } - const auto storageEngine = getGlobalServiceContext()->getStorageEngine(); // Parse the index specs into their respective state, ready or unfinished. for (auto&& spec : indexSpecs) { - // Sanitize storage engine options to remove options which might not apply to this node. See - // SERVER-68122. - if (auto storageEngineElem = spec.getField(IndexDescriptor::kStorageEngineFieldName)) { - auto sanitizedStorageEngineOpts = - storageEngine->getSanitizedStorageOptionsForSecondaryReplication( - storageEngineElem.embeddedObject()); - spec = spec.addFields( - BSON(IndexDescriptor::kStorageEngineFieldName << sanitizedStorageEngineOpts)); - } - if (spec.hasField("clustered")) { invariant(_collectionOptions.clusteredIndex); invariant(spec.getBoolField("clustered") == true); diff --git a/src/mongo/db/repl/data_replicator_external_state.h b/src/mongo/db/repl/data_replicator_external_state.h index d5be160d5cf..87826b0f199 100644 --- a/src/mongo/db/repl/data_replicator_external_state.h +++ b/src/mongo/db/repl/data_replicator_external_state.h @@ -30,7 +30,6 @@ #pragma once #include "mongo/base/status_with.h" -#include "mongo/db/repl/last_vote.h" #include "mongo/db/repl/multiapplier.h" #include "mongo/db/repl/oplog_applier.h" #include "mongo/db/repl/oplog_buffer.h" @@ -145,12 +144,6 @@ public: virtual Status storeLocalConfigDocument(OperationContext* opCtx, const BSONObj& config) = 0; /** - * Returns the current stored replica set "last vote" if there is one, or an error why there - * isn't. - */ - virtual StatusWith<LastVote> loadLocalLastVoteDocument(OperationContext* opCtx) const = 0; - - /** * Returns the replication journal listener. */ virtual JournalListener* getReplicationJournalListener() = 0; diff --git a/src/mongo/db/repl/data_replicator_external_state_impl.cpp b/src/mongo/db/repl/data_replicator_external_state_impl.cpp index 9bd60084aed..00c924ff1ea 100644 --- a/src/mongo/db/repl/data_replicator_external_state_impl.cpp +++ b/src/mongo/db/repl/data_replicator_external_state_impl.cpp @@ -175,11 +175,6 @@ Status DataReplicatorExternalStateImpl::storeLocalConfigDocument(OperationContex opCtx, config, false /* write oplog entry */); } -StatusWith<LastVote> DataReplicatorExternalStateImpl::loadLocalLastVoteDocument( - OperationContext* opCtx) const { - return _replicationCoordinatorExternalState->loadLocalLastVoteDocument(opCtx); -} - JournalListener* DataReplicatorExternalStateImpl::getReplicationJournalListener() { return _replicationCoordinatorExternalState->getReplicationJournalListener(); } diff --git a/src/mongo/db/repl/data_replicator_external_state_impl.h b/src/mongo/db/repl/data_replicator_external_state_impl.h index 9cd2364927e..c408c484dc9 100644 --- a/src/mongo/db/repl/data_replicator_external_state_impl.h +++ b/src/mongo/db/repl/data_replicator_external_state_impl.h @@ -87,8 +87,6 @@ public: Status storeLocalConfigDocument(OperationContext* opCtx, const BSONObj& config) override; - StatusWith<LastVote> loadLocalLastVoteDocument(OperationContext* opCtx) const override; - JournalListener* getReplicationJournalListener() override; protected: diff --git a/src/mongo/db/repl/data_replicator_external_state_mock.cpp b/src/mongo/db/repl/data_replicator_external_state_mock.cpp index 617f4f24098..ddcfc701ca6 100644 --- a/src/mongo/db/repl/data_replicator_external_state_mock.cpp +++ b/src/mongo/db/repl/data_replicator_external_state_mock.cpp @@ -147,10 +147,5 @@ JournalListener* DataReplicatorExternalStateMock::getReplicationJournalListener( return nullptr; } -StatusWith<LastVote> DataReplicatorExternalStateMock::loadLocalLastVoteDocument( - OperationContext* opCtx) const { - return StatusWith<LastVote>(ErrorCodes::NoMatchingDocument, "mock"); -} - } // namespace repl } // namespace mongo diff --git a/src/mongo/db/repl/data_replicator_external_state_mock.h b/src/mongo/db/repl/data_replicator_external_state_mock.h index beb7ecdc28d..535ee513102 100644 --- a/src/mongo/db/repl/data_replicator_external_state_mock.h +++ b/src/mongo/db/repl/data_replicator_external_state_mock.h @@ -77,8 +77,6 @@ public: Status storeLocalConfigDocument(OperationContext* opCtx, const BSONObj& config) override; - StatusWith<LastVote> loadLocalLastVoteDocument(OperationContext* opCtx) const override; - JournalListener* getReplicationJournalListener() override; // Task executor. diff --git a/src/mongo/db/repl/database_cloner.cpp b/src/mongo/db/repl/database_cloner.cpp index 2c5087fabfd..adeeeb0afb6 100644 --- a/src/mongo/db/repl/database_cloner.cpp +++ b/src/mongo/db/repl/database_cloner.cpp @@ -29,7 +29,6 @@ #define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kReplicationInitialSync -#include "mongo/db/service_context.h" #include "mongo/platform/basic.h" #include "mongo/base/string_data.h" @@ -70,7 +69,6 @@ BaseCloner::AfterStageBehavior DatabaseCloner::listCollectionsStage() { BSONObj res; auto collectionInfos = getClient()->getCollectionInfos(_dbName, ListCollectionsFilter::makeTypeCollectionFilter()); - const auto storageEngine = getGlobalServiceContext()->getStorageEngine(); stdx::unordered_set<std::string> seen; for (auto&& info : collectionInfos) { @@ -106,13 +104,6 @@ BaseCloner::AfterStageBehavior DatabaseCloner::listCollectionsStage() { << "'" << result.getName() << "': " << info, isDuplicate); - // Sanitize storage engine options to remove options which might not apply to this node. See - // SERVER-68122. - auto sanitizedStorageOptions = - storageEngine->getSanitizedStorageOptionsForSecondaryReplication( - result.getOptions().storageEngine); - result.getOptions().storageEngine = sanitizedStorageOptions; - // While UUID is a member of CollectionOptions, listCollections does not return the // collectionUUID there as part of the options, but instead places it in the 'info' field. // We need to move it back to CollectionOptions to create the collection properly. diff --git a/src/mongo/db/repl/dbcheck.cpp b/src/mongo/db/repl/dbcheck.cpp index cc140a71817..6fb94689544 100644 --- a/src/mongo/db/repl/dbcheck.cpp +++ b/src/mongo/db/repl/dbcheck.cpp @@ -27,16 +27,15 @@ * it in the license file. */ -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kCommand - #include "mongo/platform/basic.h" #include "mongo/bson/simple_bsonelement_comparator.h" #include "mongo/db/catalog/collection_catalog.h" #include "mongo/db/catalog/database.h" #include "mongo/db/catalog/database_holder.h" -#include "mongo/db/catalog/health_log_interface.h" +#include "mongo/db/catalog/health_log.h" #include "mongo/db/catalog/index_catalog.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/operation_context.h" #include "mongo/db/query/internal_plans.h" @@ -45,13 +44,10 @@ #include "mongo/db/repl/dbcheck_gen.h" #include "mongo/db/repl/oplog.h" #include "mongo/db/repl/optime.h" -#include "mongo/db/repl/repl_server_parameters_gen.h" -#include "mongo/logv2/log.h" namespace mongo { MONGO_FAIL_POINT_DEFINE(SleepDbCheckInBatch); -MONGO_FAIL_POINT_DEFINE(hangAfterGeneratingHashForExtraIndexKeysCheck); namespace { @@ -386,7 +382,7 @@ Status dbCheckBatchOnSecondary(OperationContext* opCtx, "dbCheck failed", OplogEntriesEnum::Batch, BSON("success" << false << "info" << msg)); - HealthLogInterface::get(opCtx)->log(*logEntry); + HealthLog::get(opCtx).log(*logEntry); return Status::OK(); } @@ -412,21 +408,13 @@ Status dbCheckBatchOnSecondary(OperationContext* opCtx, (batchesProcessed % gDbCheckHealthLogEveryNBatches.load() == 0)) { // On debug builds, health-log every batch result; on release builds, health-log // every N batches. - HealthLogInterface::get(opCtx)->log(*logEntry); - } - - if (MONGO_unlikely(hangAfterGeneratingHashForExtraIndexKeysCheck.shouldFail())) { - LOGV2_DEBUG(3083200, - 3, - "Hanging due to hangAfterGeneratingHashForExtraIndexKeysCheck failpoint"); - // hangAfterGeneratingHashForExtraIndexKeysCheck.pauseWhileSet(opCtx); - opCtx->sleepFor(Milliseconds(1000)); + HealthLog::get(opCtx).log(*logEntry); } } catch (const DBException& exception) { // In case of an error, report it to the health log, auto logEntry = dbCheckErrorHealthLogEntry( entry.getNss(), msg, OplogEntriesEnum::Batch, exception.toStatus(), entry.toBSON()); - HealthLogInterface::get(opCtx)->log(*logEntry); + HealthLog::get(opCtx).log(*logEntry); return Status::OK(); } return Status::OK(); @@ -450,79 +438,10 @@ Status dbCheckOplogCommand(OperationContext* opCtx, auto type = OplogEntries_parse(IDLParserErrorContext("type"), cmd.getStringField("type")); IDLParserErrorContext ctx("o"); - auto skipDbCheck = mode != OplogApplication::Mode::kSecondary; - std::string oplogApplicationMode; - if (mode == OplogApplication::Mode::kInitialSync) { - oplogApplicationMode = "initial sync"; - } else if (mode == OplogApplication::Mode::kUnstableRecovering) { - oplogApplicationMode = "unstable recovering"; - } else if (mode == OplogApplication::Mode::kStableRecovering) { - oplogApplicationMode = "stable recovering"; - } else if (mode == OplogApplication::Mode::kApplyOpsCmd) { - oplogApplicationMode = "applyOps"; - } else { - oplogApplicationMode = "secondary"; - } switch (type) { case OplogEntriesEnum::Batch: { auto invocation = DbCheckOplogBatch::parse(ctx, cmd); - - /* - // TODO SERVER-78399: Clean up handling minKey/maxKey once feature flag is removed. - // If the dbcheck oplog entry doesn't contain batchStart, convert minKey to a BSONObj to - // be used as batchStart. - BSONObj batchStart, batchEnd, batchId; - if (!invocation.getBatchStart()) { - batchStart = BSON("_id" << invocation.getMinKey().elem()); - } else { - batchStart = invocation.getBatchStart().get(); - } - if (!invocation.getBatchEnd()) { - batchEnd = BSON("_id" << invocation.getMaxKey().elem()); - } else { - batchEnd = invocation.getBatchEnd().get(); - } - */ - - if (!skipDbCheck && !repl::skipApplyingDbCheckBatchOnSecondary.load()) { - return dbCheckBatchOnSecondary(opCtx, opTime, invocation); - } - - // TODO SERVER-89921: Uncomment once the relevant tickets are backported. - /* - if (invocation.getBatchId()) { - batchId = invocation.getBatchId().get().toBSON(); - } - - BSONObjBuilder data; - data.append("batchStart", batchStart); - data.append("batchEnd", batchEnd); - - if (!batchId.isEmpty()) { - data.append("batchId", batchId); - } - */ - - auto warningMsg = "cannot execute dbcheck due to ongoing " + oplogApplicationMode; - if (repl::skipApplyingDbCheckBatchOnSecondary.load()) { - warningMsg = - "skipping applying dbcheck batch because the " - "'skipApplyingDbCheckBatchOnSecondary' parameter is on"; - } - - LOGV2_DEBUG(8888500, 3, "skipping applying dbcheck batch", "reason"_attr = warningMsg); - // TODO SERVER-89921: Uncomment these logging attributes once the relevant tickets are - // backported. - //"batchStart"_attr = batchStart, - //"batchEnd"_attr = batchEnd, - //"batchId"_attr = batchId); - - auto healthLogEntry = mongo::dbCheckHealthLogEntry( - invocation.getNss(), SeverityEnum::Warning, warningMsg, type, boost::none /*data*/); - - HealthLogInterface::get(Client::getCurrent()->getServiceContext()) - ->log(*healthLogEntry); - return Status::OK(); + return dbCheckBatchOnSecondary(opCtx, opTime, invocation); } case OplogEntriesEnum::Collection: { // TODO SERVER-61963. @@ -532,15 +451,9 @@ Status dbCheckOplogCommand(OperationContext* opCtx, // fallthrough case OplogEntriesEnum::Stop: const auto healthLogEntry = mongo::dbCheckHealthLogEntry( - boost::none /*nss*/, - skipDbCheck ? SeverityEnum::Warning : SeverityEnum::Info, - skipDbCheck ? "cannot execute dbcheck due to ongoing " + oplogApplicationMode : "", - type, - boost::none /*data*/ + boost::none /*nss*/, SeverityEnum::Info, "", type, boost::none /*data*/ ); - // TODO: need to change this - HealthLogInterface::get(Client::getCurrent()->getServiceContext()) - ->log(*healthLogEntry); + HealthLog::get(Client::getCurrent()->getServiceContext()).log(*healthLogEntry); return Status::OK(); } diff --git a/src/mongo/db/repl/delayable_timeout_callback.cpp b/src/mongo/db/repl/delayable_timeout_callback.cpp deleted file mode 100644 index d84d7099779..00000000000 --- a/src/mongo/db/repl/delayable_timeout_callback.cpp +++ /dev/null @@ -1,192 +0,0 @@ -/** - * Copyright (C) 2022-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kReplication - -#include "mongo/db/repl/delayable_timeout_callback.h" -#include "mongo/logv2/log.h" - -namespace mongo { -namespace repl { - -DelayableTimeoutCallback::~DelayableTimeoutCallback() { - cancel(); -} - -void DelayableTimeoutCallback::cancel() { - stdx::lock_guard lk(_mutex); - _cancel(lk); -} - -void DelayableTimeoutCallback::_cancel(WithLock) { - if (_cbHandle) { - _executor->cancel(_cbHandle); - _cbHandle = executor::TaskExecutor::CallbackHandle(); - _nextCall = Date_t(); - } - invariant(_nextCall == Date_t()); -} - -Date_t DelayableTimeoutCallback::getNextCall() const { - stdx::lock_guard lk(_mutex); - return _nextCall; -} - -bool DelayableTimeoutCallback::isActive() const { - return getNextCall() != Date_t(); -} - -Status DelayableTimeoutCallback::scheduleAt(Date_t when) { - stdx::lock_guard lk(_mutex); - return _scheduleAt(lk, when); -} - -Status DelayableTimeoutCallback::_scheduleAt(WithLock lk, Date_t when) { - if (_cbHandle && when < _nextCall) { - LOGV2_DEBUG(6602300, - 3, - "Moving a delayable timeout call backwards, which is inefficient", - "timerName"_attr = _timerName, - "when"_attr = when, - "nextCall"_attr = _nextCall); - _cancel(lk); - } - return _delayUntil(lk, when); -} - -Status DelayableTimeoutCallback::delayUntil(Date_t when) { - stdx::lock_guard lk(_mutex); - return _delayUntil(lk, when); -} - -Status DelayableTimeoutCallback::_delayUntil(WithLock lk, Date_t when) { - if (!_cbHandle) { - // No timeout is active; just schedule it - return _reschedule(lk, when); - } - if (when == _nextCall) { - LOGV2_DEBUG(6602301, - 5, - "'Rescheduling' to same time", - "timerName"_attr = _timerName, - "when"_attr = when, - "nextCall"_attr = _nextCall); - } - _nextCall = when; - return Status::OK(); -} - -void DelayableTimeoutCallback::_handleTimeout(const executor::TaskExecutor::CallbackArgs& args) { - { - stdx::lock_guard lk(_mutex); - if (args.myHandle != _cbHandle) { - // This is normal when scheduleAt() or cancel() is used. - LOGV2_DEBUG(6602302, - 5, - "DelayableTimeoutCallback::_handleTimeout got a timeout after a new handle " - "was scheduled", - "timerName"_attr = _timerName); - return; - } - Date_t now = _executor->now(); - if (args.status == ErrorCodes::CallbackCanceled) { - // If args.status is CallbackCanceled yet the handles matched, that means the - // executor canceled the callback itself, probably as part of shutdown. We - // do not want to reschedule in this case, nor call the callback. - _cbHandle = executor::TaskExecutor::CallbackHandle(); - _nextCall = Date_t(); - return; - } else if (_nextCall > now) { - Status status = _reschedule(lk, _nextCall); - if (!status.isOK()) { - LOGV2_DEBUG(6602303, - 2, - "DelayableTimeoutCallback::_handleTimeout unable to schedule", - "timerName"_attr = _timerName, - "error"_attr = status); - fassert(6602305, status == ErrorCodes::ShutdownInProgress); - } - return; - } - _cbHandle = executor::TaskExecutor::CallbackHandle(); - _nextCall = Date_t(); - } - _callback(args); -} - -Status DelayableTimeoutCallback::_reschedule(WithLock, Date_t when) { - // We clear _cbHandle and _nextCall in advance so if scheduleWorkAt fails for any reason - // (including by exception), the invariant that _cbHandle and _nextCall are clear when no - // callback is scheduled is maintained. - _cbHandle = executor::TaskExecutor::CallbackHandle(); - _nextCall = Date_t(); - auto cbh = _executor->scheduleWorkAt( - when, [this](const executor::TaskExecutor::CallbackArgs& args) { _handleTimeout(args); }); - if (cbh == ErrorCodes::ShutdownInProgress) { - return cbh.getStatus(); - } - _nextCall = when; - _cbHandle = fassert(6602304, cbh); - return Status::OK(); -} - -void DelayableTimeoutCallbackWithJitter::_resetRandomization(WithLock) { - _lastRandomizationTime = Date_t(); - _currentJitter = Milliseconds(0); -} - -Status DelayableTimeoutCallbackWithJitter::scheduleAt(Date_t when) { - stdx::lock_guard lk(_mutex); - _resetRandomization(lk); - return _scheduleAt(lk, when); -} - -Status DelayableTimeoutCallbackWithJitter::delayUntil(Date_t when) { - stdx::lock_guard lk(_mutex); - _resetRandomization(lk); - return _delayUntil(lk, when); -} - -Status DelayableTimeoutCallbackWithJitter::delayUntilWithJitter(Date_t when, - Milliseconds jitterUpperBound) { - if (jitterUpperBound == Milliseconds::zero()) - return delayUntil(when); - stdx::lock_guard lk(_mutex); - Date_t now = _getExecutor()->now(); - Milliseconds elapsed = now - _lastRandomizationTime; - if (_lastRandomizationTime == Date_t() || elapsed < Milliseconds::zero() || - elapsed >= jitterUpperBound || jitterUpperBound < _currentJitter) { - _lastRandomizationTime = now; - _currentJitter = Milliseconds(_randomSource(durationCount<Milliseconds>(jitterUpperBound))); - } - return _delayUntil(lk, when + _currentJitter); -} - -} // namespace repl -} // namespace mongo diff --git a/src/mongo/db/repl/delayable_timeout_callback.h b/src/mongo/db/repl/delayable_timeout_callback.h deleted file mode 100644 index d1cb8e29f95..00000000000 --- a/src/mongo/db/repl/delayable_timeout_callback.h +++ /dev/null @@ -1,159 +0,0 @@ -/** - * Copyright (C) 2022-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ -#pragma once - -#include <string> - -#include "mongo/base/status.h" -#include "mongo/executor/task_executor.h" -#include "mongo/util/time_support.h" - -namespace mongo { -namespace repl { - -/** - * The DelayableTimeoutCallback is a utility class which allows a callback to be scheduled on an - * executor at a given time, and then that time pushed back (later) arbitrarily often without - * rescheduling the call on the executor. The callback is never called with CallbackCanceled or - * ShutdownInProgress. - * - * All methods are thread safe, though isActive() and getNextCall() may return stale information - * if external synchronization is not used. The callback is called without any locks held. - */ -class DelayableTimeoutCallback { -public: - /** - * Creates a DelayableTimeoutCallback with the given executor and callback function. The - * DelayableTimeoutCallback is inactive (the callback is not scheduled) when constructed. - */ - DelayableTimeoutCallback(executor::TaskExecutor* executor, - executor::TaskExecutor::CallbackFn callback, - std::string timerName = std::string()) - : _executor(executor), _callback(std::move(callback)), _timerName(std::move(timerName)){}; - - ~DelayableTimeoutCallback(); - - /** - * If the timeout is scheduled, cancel it. The callback function will not be called. - */ - void cancel(); - - /** - * Schedule the timeout to occur at "when", regardless of if or when it is already scheduled. - * If it is already scheduled to occur after "when", it is canceled and rescheduled - * - * Returns status of the attempt to schedule on the executor. - */ - Status scheduleAt(Date_t when); - - /** - * Schedule the timeout to occur at "when" if it is not scheduled or scheduled to occur before - * "when". If it is already scheduled to occur before "when", this call has no effect. - * - * Returns status of the attempt to schedule on the executor. - */ - Status delayUntil(Date_t when); - - /** - * Returns whether the callback is scheduled at all. - */ - bool isActive() const; - - /** - * Returns when the next call to the passed-in callback will be made, or Date_t() if inactive. - */ - Date_t getNextCall() const; - -protected: - void _cancel(WithLock); - Status _scheduleAt(WithLock, Date_t when); - Status _delayUntil(WithLock, Date_t when); - executor::TaskExecutor* _getExecutor() { - return _executor; - } - - mutable Mutex _mutex = MONGO_MAKE_LATCH("DelayableTimeoutCallback"); - -private: - void _handleTimeout(const executor::TaskExecutor::CallbackArgs& cbData); - Status _reschedule(WithLock, Date_t when); - - executor::TaskExecutor* _executor; - executor::TaskExecutor::CallbackHandle _cbHandle; - const executor::TaskExecutor::CallbackFn _callback; - Date_t _nextCall; - - // Timer name is used only for logging. - const std::string _timerName; -}; - -/** - * DelayableTimeoutCallbackWithJitter is a slight variation on DelayableTimeoutCallback - * which adds some additional random time to delays. Since the callback may be delayed at - * intervals much shorter than the random time, this would naively result in the timeout - * either being moved backwards often, or if we forbid moving it backwards, ending up quickly - * moving to the maximum jitter (which isn't very random). To avoid that, we only recompute - * the jitter every maximum jitter interval -- e.g. if the max jitter is 10 seconds and we - * add 3 seconds jitter at time T, we will add 3 seconds jitter to every subsequent call until - * time T + 10. - * - * The typical purpose of the jitter is to prevent two timers receiving delay calls at the same - * times from firing at the same time. - * - * Synchronization of the randomSource is up to the caller; it is provided externally to - * avoid having a separate random number generator per timer. The randomSource function will - * be called with the maximum jitter value passed to delayUntilWithJitter; it should return - * a value in the range [0, maxJitter) or [0, maxJitter] depending on what you want the - * actual jitter range to be. - */ -class DelayableTimeoutCallbackWithJitter : public DelayableTimeoutCallback { -public: - using RandomSource = std::function<int64_t(int64_t)>; - - DelayableTimeoutCallbackWithJitter(executor::TaskExecutor* executor, - executor::TaskExecutor::CallbackFn callback, - RandomSource randomSource, - std::string timerName = std::string()) - : DelayableTimeoutCallback(executor, std::move(callback), timerName), - _randomSource(std::move(randomSource)) {} - - Status scheduleAt(Date_t when); - Status delayUntil(Date_t when); - Status delayUntilWithJitter(Date_t when, Milliseconds maxJitter); - -private: - void _resetRandomization(WithLock); - - RandomSource _randomSource; - Date_t _lastRandomizationTime; - Milliseconds _currentJitter; -}; - -} // namespace repl -} // namespace mongo diff --git a/src/mongo/db/repl/delayable_timeout_callback_test.cpp b/src/mongo/db/repl/delayable_timeout_callback_test.cpp deleted file mode 100644 index b293d2793bb..00000000000 --- a/src/mongo/db/repl/delayable_timeout_callback_test.cpp +++ /dev/null @@ -1,360 +0,0 @@ -/** - * Copyright (C) 2022-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/db/repl/delayable_timeout_callback.h" -#include "mongo/executor/thread_pool_task_executor_test_fixture.h" - -namespace mongo { -namespace repl { -template <typename T> -class DelayableTimeoutCallbackBaseTest : public unittest::Test { -protected: - void setUp() override { - auto network = std::make_unique<executor::NetworkInterfaceMock>(); - _net = network.get(); - _executor = makeSharedThreadPoolTestExecutor(std::move(network)); - _executor->startup(); - createDelayableTimeoutCallback(); - } - - void tearDown() override { - _delayableTimeoutCallback = boost::none; - _executor->shutdown(); - _executor->join(); - _executor.reset(); - } - - void callback(const mongo::executor::TaskExecutor::CallbackArgs& cbData) { - callbackRan++; - if (!cbData.status.isOK()) { - callbackRanWithError++; - } - } - - void createDelayableTimeoutCallback() { - MONGO_UNREACHABLE; - } - - -protected: - boost::optional<T> _delayableTimeoutCallback; - executor::NetworkInterfaceMock* _net; - std::shared_ptr<executor::TaskExecutor> _executor; - int callbackRan = 0; - int callbackRanWithError = 0; -}; - -template <> -void DelayableTimeoutCallbackBaseTest<DelayableTimeoutCallback>::createDelayableTimeoutCallback() { - _delayableTimeoutCallback.emplace( - _executor.get(), [this](const mongo::executor::TaskExecutor::CallbackArgs& cbData) { - this->callback(cbData); - }); -} - -template <> -void DelayableTimeoutCallbackBaseTest< - DelayableTimeoutCallbackWithJitter>::createDelayableTimeoutCallback() { - _delayableTimeoutCallback.emplace( - _executor.get(), - [this](const mongo::executor::TaskExecutor::CallbackArgs& cbData) { - this->callback(cbData); - }, - [](int64_t limit) { - static int64_t notVeryRandom = 0; - notVeryRandom += 10; - return notVeryRandom % limit; - }); -} - -typedef DelayableTimeoutCallbackBaseTest<DelayableTimeoutCallback> DelayableTimeoutCallbackTest; -typedef DelayableTimeoutCallbackBaseTest<DelayableTimeoutCallbackWithJitter> - DelayableTimeoutCallbackWithJitterTest; - -TEST_F(DelayableTimeoutCallbackTest, ScheduleAtSchedulesFirstCallback) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() + Seconds(2))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, DelayUntilSchedulesFirstCallback) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->delayUntil(_net->now() + Seconds(2))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, ScheduleAtMovesCallbackLater) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() + Seconds(2))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() + Seconds(2))); - - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, DelayUntilMovesCallbackLater) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->delayUntil(_net->now() + Seconds(2))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - - ASSERT_OK(_delayableTimeoutCallback->delayUntil(_net->now() + Seconds(2))); - - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, ScheduleAtMovesCallbackEarlier) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() + Seconds(3))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() + Seconds(1))); - - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, DelayUntilDoesNotMoveCallbackEarlier) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->delayUntil(_net->now() + Seconds(3))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - - ASSERT_OK(_delayableTimeoutCallback->delayUntil(_net->now() + Seconds(1))); - - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, ScheduleAtInPastRunsImmediately) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - // Make sure there's a past to schedule in. - _net->runUntil(_net->now() + Days(1)); - - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() - Seconds(1))); - - // Needed to trigger anything scheduled. - _net->runReadyNetworkOperations(); - - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, DelayUntilInPastRunsImmediately) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - // Make sure there's a past to schedule in. - _net->runUntil(_net->now() + Days(1)); - - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - ASSERT_OK(_delayableTimeoutCallback->delayUntil(_net->now() - Seconds(1))); - - // Needed to trigger anything scheduled. - _net->runReadyNetworkOperations(); - - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, Cancellation) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() + Seconds(2))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - _delayableTimeoutCallback->cancel(); - ASSERT_EQ(0, callbackRan); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_EQ(0, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackTest, Shutdown) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->scheduleAt(_net->now() + Seconds(2))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(0, callbackRan); - _net->runUntil(_net->now() + Seconds(1)); - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - - _executor->shutdown(); - - // This makes sure the executor processes the shutdown. - _net->runReadyNetworkOperations(); - - ASSERT_EQ(0, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackWithJitterTest, DelayUntilWithJitter) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK(_delayableTimeoutCallback->delayUntilWithJitter(_net->now() + Seconds(10), - Milliseconds(100))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - // Our "random" generator is just 10,20,30,... - ASSERT_EQ(_net->now() + Milliseconds(10010), _delayableTimeoutCallback->getNextCall()); - ASSERT_EQ(0, callbackRan); - - // Setting it again in the same time shouldn't change jitter. - ASSERT_OK(_delayableTimeoutCallback->delayUntilWithJitter(_net->now() + Seconds(10), - Milliseconds(100))); - ASSERT_EQ(_net->now() + Milliseconds(10010), _delayableTimeoutCallback->getNextCall()); - ASSERT_EQ(0, callbackRan); - - // Move forward less than the max jitter shouldn't change jitter. - for (int i = 0; i < 3; i++) { - _net->runUntil(_net->now() + Milliseconds(25)); - ASSERT_OK(_delayableTimeoutCallback->delayUntilWithJitter(_net->now() + Seconds(10), - Milliseconds(100))); - ASSERT_EQ(_net->now() + Milliseconds(10010), _delayableTimeoutCallback->getNextCall()); - ASSERT_EQ(0, callbackRan); - } - - // Move forward to the max jitter should recalculate jitter. - _net->runUntil(_net->now() + Milliseconds(25)); - ASSERT_OK(_delayableTimeoutCallback->delayUntilWithJitter(_net->now() + Seconds(10), - Milliseconds(100))); - ASSERT_EQ(_net->now() + Milliseconds(10020), _delayableTimeoutCallback->getNextCall()); - ASSERT_EQ(0, callbackRan); - - // Setting max jitter to less than actual jitter should recalculate jitter. - ASSERT_OK(_delayableTimeoutCallback->delayUntilWithJitter(_net->now() + Seconds(10), - Milliseconds(19))); - // Jitter value will be 30 % 19 = 11 - ASSERT_EQ(_net->now() + Milliseconds(10011), _delayableTimeoutCallback->getNextCall()); - ASSERT_EQ(0, callbackRan); - - _net->runUntil(_net->now() + Milliseconds(10011)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -TEST_F(DelayableTimeoutCallbackWithJitterTest, DelayUntilWithZeroJitter) { - executor::NetworkInterfaceMock::InNetworkGuard guard(_net); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); - - ASSERT_OK( - _delayableTimeoutCallback->delayUntilWithJitter(_net->now() + Seconds(10), Seconds(0))); - - ASSERT_TRUE(_delayableTimeoutCallback->isActive()); - ASSERT_EQ(_net->now() + Milliseconds(10000), _delayableTimeoutCallback->getNextCall()); - ASSERT_EQ(0, callbackRan); - - _net->runUntil(_net->now() + Milliseconds(10000)); - ASSERT_EQ(1, callbackRan); - ASSERT_EQ(0, callbackRanWithError); - ASSERT_FALSE(_delayableTimeoutCallback->isActive()); -} - -} // namespace repl -} // namespace mongo diff --git a/src/mongo/db/repl/idempotency_test_fixture.cpp b/src/mongo/db/repl/idempotency_test_fixture.cpp index d9a30cee036..0043ff54e3e 100644 --- a/src/mongo/db/repl/idempotency_test_fixture.cpp +++ b/src/mongo/db/repl/idempotency_test_fixture.cpp @@ -43,6 +43,7 @@ #include "mongo/db/catalog/index_catalog.h" #include "mongo/db/client.h" #include "mongo/db/concurrency/d_concurrency.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" #include "mongo/db/db_raii.h" #include "mongo/db/index_builds_coordinator.h" @@ -404,10 +405,8 @@ CollectionState IdempotencyTest::validate(const NamespaceString& nss) { nss, CollectionValidation::ValidateMode::kForegroundFull, CollectionValidation::RepairMode::kNone, - /*additionalOptions=*/{}, &validateResults, - &bob, - /*logDiagnostics=*/false)); + &bob)); ASSERT_TRUE(validateResults.valid); } diff --git a/src/mongo/db/repl/initial_syncer.cpp b/src/mongo/db/repl/initial_syncer.cpp index d8a2f5104d7..02fb4af0be5 100644 --- a/src/mongo/db/repl/initial_syncer.cpp +++ b/src/mongo/db/repl/initial_syncer.cpp @@ -548,7 +548,6 @@ void InitialSyncer::_setUp_inlock(OperationContext* opCtx, std::uint32_t initial _stats.initialSyncStart = _exec->now(); _stats.maxFailedInitialSyncAttempts = initialSyncMaxAttempts; _stats.failedInitialSyncAttempts = 0; - _stats.exec = std::weak_ptr<executor::TaskExecutor>(_exec); _allowedOutageDuration = Seconds(initialSyncTransientErrorRetryPeriodSeconds.load()); } @@ -1394,63 +1393,53 @@ void InitialSyncer::_lastOplogEntryFetcherCallbackForStopTimestamp( std::shared_ptr<OnCompletionGuard> onCompletionGuard) { OpTimeAndWallTime resultOpTimeAndWallTime = {OpTime(), Date_t()}; { - { - stdx::lock_guard<Latch> lock(_mutex); - auto status = _checkForShutdownAndConvertStatus_inlock( - result.getStatus(), "error fetching last oplog entry for stop timestamp"); - if (_shouldRetryError(lock, status)) { - auto scheduleStatus = - (*_attemptExec) - ->scheduleWork( - [this, onCompletionGuard](executor::TaskExecutor::CallbackArgs args) { - // It is not valid to schedule the retry from within this callback, - // hence we schedule a lambda to schedule the retry. - stdx::lock_guard<Latch> lock(_mutex); - // Since the stopTimestamp is retrieved after we have done all the - // work of retrieving collection data, we handle retries within this - // class by retrying for - // 'initialSyncTransientErrorRetryPeriodSeconds' (default 24 hours). - // This is the same retry strategy used when retrieving collection - // data, and avoids retrieving all the data and then throwing it - // away due to a transient network outage. - auto status = _scheduleLastOplogEntryFetcher_inlock( - [=](const StatusWith<mongo::Fetcher::QueryResponse>& status, - mongo::Fetcher::NextAction*, - mongo::BSONObjBuilder*) { - _lastOplogEntryFetcherCallbackForStopTimestamp( - status, onCompletionGuard); - }, - kInitialSyncerHandlesRetries); - if (!status.isOK()) { - onCompletionGuard->setResultAndCancelRemainingWork_inlock( - lock, status); - } - }); - if (scheduleStatus.isOK()) - return; - // If scheduling failed, we're shutting down and cannot retry. - // So just continue with the original failed status. - } - if (!status.isOK()) { - onCompletionGuard->setResultAndCancelRemainingWork_inlock(lock, status); - return; - } - - auto&& optimeStatus = parseOpTimeAndWallTime(result); - if (!optimeStatus.isOK()) { - onCompletionGuard->setResultAndCancelRemainingWork_inlock(lock, - optimeStatus.getStatus()); + stdx::lock_guard<Latch> lock(_mutex); + auto status = _checkForShutdownAndConvertStatus_inlock( + result.getStatus(), "error fetching last oplog entry for stop timestamp"); + if (_shouldRetryError(lock, status)) { + auto scheduleStatus = + (*_attemptExec) + ->scheduleWork([this, + onCompletionGuard](executor::TaskExecutor::CallbackArgs args) { + // It is not valid to schedule the retry from within this callback, + // hence we schedule a lambda to schedule the retry. + stdx::lock_guard<Latch> lock(_mutex); + // Since the stopTimestamp is retrieved after we have done all the work of + // retrieving collection data, we handle retries within this class by + // retrying for 'initialSyncTransientErrorRetryPeriodSeconds' (default 24 + // hours). This is the same retry strategy used when retrieving collection + // data, and avoids retrieving all the data and then throwing it away due to + // a transient network outage. + auto status = _scheduleLastOplogEntryFetcher_inlock( + [=](const StatusWith<mongo::Fetcher::QueryResponse>& status, + mongo::Fetcher::NextAction*, + mongo::BSONObjBuilder*) { + _lastOplogEntryFetcherCallbackForStopTimestamp(status, + onCompletionGuard); + }, + kInitialSyncerHandlesRetries); + if (!status.isOK()) { + onCompletionGuard->setResultAndCancelRemainingWork_inlock(lock, status); + } + }); + if (scheduleStatus.isOK()) return; - } - resultOpTimeAndWallTime = optimeStatus.getValue(); + // If scheduling failed, we're shutting down and cannot retry. + // So just continue with the original failed status. + } + if (!status.isOK()) { + onCompletionGuard->setResultAndCancelRemainingWork_inlock(lock, status); + return; } - // Release the _mutex to write to disk. - auto opCtx = makeOpCtx(); - _replicationProcess->getConsistencyMarkers()->setMinValid( - opCtx.get(), resultOpTimeAndWallTime.opTime, true); + auto&& optimeStatus = parseOpTimeAndWallTime(result); + if (!optimeStatus.isOK()) { + onCompletionGuard->setResultAndCancelRemainingWork_inlock(lock, + optimeStatus.getStatus()); + return; + } + resultOpTimeAndWallTime = optimeStatus.getValue(); - stdx::lock_guard<Latch> lock(_mutex); _initialSyncState->stopTimestamp = resultOpTimeAndWallTime.opTime.getTimestamp(); // If the beginFetchingTimestamp is different from the stopTimestamp, it indicates that @@ -2197,10 +2186,9 @@ void InitialSyncer::Stats::append(BSONObjBuilder* builder) const { builder->appendNumber("maxFailedInitialSyncAttempts", static_cast<long long>(maxFailedInitialSyncAttempts)); - auto e = exec.lock(); if (initialSyncStart != Date_t()) { builder->appendDate("initialSyncStart", initialSyncStart); - auto elapsedDurationEnd = e ? e->now() : Date_t::now(); + auto elapsedDurationEnd = Date_t::now(); if (initialSyncEnd != Date_t()) { builder->appendDate("initialSyncEnd", initialSyncEnd); elapsedDurationEnd = initialSyncEnd; diff --git a/src/mongo/db/repl/initial_syncer.h b/src/mongo/db/repl/initial_syncer.h index 3a124b6b02d..177122e12d9 100644 --- a/src/mongo/db/repl/initial_syncer.h +++ b/src/mongo/db/repl/initial_syncer.h @@ -140,7 +140,6 @@ public: Date_t initialSyncStart; Date_t initialSyncEnd; std::vector<InitialSyncer::InitialSyncAttemptInfo> initialSyncAttemptInfos; - std::weak_ptr<executor::TaskExecutor> exec; std::string toString() const; BSONObj toBSON() const; diff --git a/src/mongo/db/repl/initial_syncer_test.cpp b/src/mongo/db/repl/initial_syncer_test.cpp index 0fc3afc5538..147d0571ceb 100644 --- a/src/mongo/db/repl/initial_syncer_test.cpp +++ b/src/mongo/db/repl/initial_syncer_test.cpp @@ -642,7 +642,6 @@ OplogEntry makeOplogEntry(int t, NamespaceString("a.a"), // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert version, // version oField, // o boost::none, // o2 @@ -4511,10 +4510,6 @@ TEST_F(InitialSyncerTest, TestRemainingInitialSyncEstimatedMillisMetric) { // Wait for the server to have reached the end of cloning collection 'a.a'. The size of this // collection is expected to equal 'dbSize'. hangDuringCloningFailPoint->waitForTimesEntered(timesEntered + 1); - { - executor::NetworkInterfaceMock::InNetworkGuard guard(net); - net->runUntil(Date_t::now() + Seconds(1)); - } auto progress = initialSyncer->getInitialSyncProgress(); LOGV2(5301701, "Progress in middle of cloning", "progress"_attr = progress); { @@ -4610,7 +4605,7 @@ TEST_F(InitialSyncerTest, GetInitialSyncProgressReturnsCorrectProgress) { ASSERT_FALSE(progress.hasField("InitialSyncEnd")); ASSERT_EQUALS(progress.getIntField("failedInitialSyncAttempts"), 0) << progress; ASSERT_EQUALS(progress.getIntField("maxFailedInitialSyncAttempts"), 2) << progress; - ASSERT_EQUALS(progress["totalInitialSyncElapsedMillis"].type(), NumberInt) << progress; + ASSERT_EQUALS(progress["totalInitialSyncElapsedMillis"].type(), NumberLong) << progress; ASSERT_EQUALS(progress.getIntField("approxTotalDataSize"), 0) << progress; ASSERT_EQUALS(progress.getIntField("approxTotalBytesCopied"), 0) << progress; ASSERT_EQUALS(progress["initialSyncStart"].type(), Date) << progress; @@ -4678,7 +4673,7 @@ TEST_F(InitialSyncerTest, GetInitialSyncProgressReturnsCorrectProgress) { ASSERT_FALSE(progress.hasField("InitialSyncEnd")); ASSERT_EQUALS(progress.getIntField("failedInitialSyncAttempts"), 1) << progress; ASSERT_EQUALS(progress.getIntField("maxFailedInitialSyncAttempts"), 2) << progress; - ASSERT_EQUALS(progress["totalInitialSyncElapsedMillis"].type(), NumberInt) << progress; + ASSERT_EQUALS(progress["totalInitialSyncElapsedMillis"].type(), NumberLong) << progress; ASSERT_EQUALS(progress.getIntField("approxTotalDataSize"), 0) << progress; ASSERT_EQUALS(progress.getIntField("approxTotalBytesCopied"), 0) << progress; ASSERT_EQUALS(progress["initialSyncStart"].type(), Date) << progress; @@ -4780,7 +4775,7 @@ TEST_F(InitialSyncerTest, GetInitialSyncProgressReturnsCorrectProgress) { ASSERT_EQUALS(progress.nFields(), 14) << progress; ASSERT_EQUALS(progress.getIntField("failedInitialSyncAttempts"), 1) << progress; ASSERT_EQUALS(progress.getIntField("maxFailedInitialSyncAttempts"), 2) << progress; - ASSERT_EQUALS(progress["totalInitialSyncElapsedMillis"].type(), NumberInt) << progress; + ASSERT_EQUALS(progress["totalInitialSyncElapsedMillis"].type(), NumberLong) << progress; ASSERT_EQUALS(progress.getIntField("approxTotalDataSize"), 10) << progress; ASSERT_EQUALS(progress.getIntField("approxTotalBytesCopied"), 10) << progress; ASSERT_EQUALS(progress["initialSyncOplogStart"].timestamp(), Timestamp(1, 1)) << progress; diff --git a/src/mongo/db/repl/isself.cpp b/src/mongo/db/repl/isself.cpp index 1e0ec7dfab7..243a1c065f0 100644 --- a/src/mongo/db/repl/isself.cpp +++ b/src/mongo/db/repl/isself.cpp @@ -163,17 +163,10 @@ std::vector<std::string> getAddrsForHost(const std::string& iporhost, } // namespace -bool isSelf(const HostAndPort& hostAndPort, ServiceContext* const ctx, Milliseconds timeout) { - if (isSelfFastPath(hostAndPort)) { - return true; - } - return isSelfSlowPath(hostAndPort, ctx, timeout); -} - -bool isSelfFastPath(const HostAndPort& hostAndPort) { +bool isSelf(const HostAndPort& hostAndPort, ServiceContext* const ctx) { if (MONGO_unlikely(failIsSelfCheck.shouldFail())) { LOGV2(356490, - "failIsSelfCheck failpoint activated, returning false from isSelfFastPath", + "failIsSelfCheck failpoint activated, returning false from isSelf", "hostAndPort"_attr = hostAndPort); return false; } @@ -229,30 +222,18 @@ bool isSelfFastPath(const HostAndPort& hostAndPort) { } } } - return false; -} -bool isSelfSlowPath(const HostAndPort& hostAndPort, - ServiceContext* const ctx, - Milliseconds timeout) { ctx->waitForStartupComplete(); - if (MONGO_unlikely(failIsSelfCheck.shouldFail())) { - LOGV2(6605000, - "failIsSelfCheck failpoint activated, returning false from isSelfSlowPath", - "hostAndPort"_attr = hostAndPort); - return false; - } try { DBClientConnection conn; - double timeoutSeconds = static_cast<double>(durationCount<Milliseconds>(timeout)) / 1000.0; - conn.setSoTimeout(timeoutSeconds); - - // We need to avoid the "hello" call triggered by a normal connect, which would cause a - // deadlock. 'isSelf' is called by the Replication Coordinator when validating a replica set - // configuration document, but the "hello" command requires a lock on the replication - // coordinator to execute. As such we call we call 'connectSocketOnly', which does not call - // "hello". + conn.setSoTimeout(30); // 30 second timeout + + // We need to avoid the isMaster call triggered by a normal connect, which would + // cause a deadlock. 'isSelf' is called by the Replication Coordinator when validating + // a replica set configuration document, but the 'isMaster' command requires a lock on the + // replication coordinator to execute. As such we call we call 'connectSocketOnly', which + // does not call 'isMaster'. auto connectSocketResult = conn.connectSocketOnly(hostAndPort, boost::none); if (!connectSocketResult.isOK()) { LOGV2(4834700, diff --git a/src/mongo/db/repl/isself.h b/src/mongo/db/repl/isself.h index fb61c8a6cb3..84264978720 100644 --- a/src/mongo/db/repl/isself.h +++ b/src/mongo/db/repl/isself.h @@ -33,7 +33,6 @@ #include <vector> #include "mongo/bson/oid.h" -#include "mongo/util/duration.h" namespace mongo { struct HostAndPort; @@ -50,21 +49,7 @@ extern OID instanceId; /** * Returns true if "hostAndPort" identifies this instance. */ -bool isSelf(const HostAndPort& hostAndPort, - ServiceContext* ctx, - Milliseconds timeout = Seconds(30)); - -/** - * Returns true if "hostAndPort" identifies this instance by checking our bound IP addresses, - * without going out to the network and running the _isSelf command on the node. - */ -bool isSelfFastPath(const HostAndPort& hostAndPort); - -/** - * Returns true if "hostAndPort" identifies this instance by running the _isSelf command on the - * node. - */ -bool isSelfSlowPath(const HostAndPort& hostAndPort, ServiceContext* ctx, Milliseconds timeout); +bool isSelf(const HostAndPort& hostAndPort, ServiceContext* ctx); /** * Returns all the IP addresses bound to the network interfaces of this machine. diff --git a/src/mongo/db/repl/isself_test.cpp b/src/mongo/db/repl/isself_test.cpp index 91ad0ad3803..511ef122ecf 100644 --- a/src/mongo/db/repl/isself_test.cpp +++ b/src/mongo/db/repl/isself_test.cpp @@ -55,7 +55,6 @@ TEST_F(ServiceContextTest, DetectsSameHostIPv4) { // Fastpath should agree with the result of getBoundAddrs // since it uses it... for (std::vector<string>::const_iterator it = addrs.begin(); it != addrs.end(); ++it) { - ASSERT(isSelfFastPath(HostAndPort(*it, serverGlobalParams.port))); ASSERT(isSelf(HostAndPort(*it, serverGlobalParams.port), getGlobalServiceContext())); } #else @@ -73,7 +72,6 @@ TEST_F(ServiceContextTest, DetectsSameHostIPv6) { // Fastpath should agree with the result of getBoundAddrs // since it uses it... for (std::vector<string>::const_iterator it = addrs.begin(); it != addrs.end(); ++it) { - ASSERT(isSelfFastPath(HostAndPort(*it, serverGlobalParams.port))); ASSERT(isSelf(HostAndPort(*it, serverGlobalParams.port), getGlobalServiceContext())); } #else diff --git a/src/mongo/db/repl/member_config.h b/src/mongo/db/repl/member_config.h index cc668fef79c..e68c24b6f2d 100644 --- a/src/mongo/db/repl/member_config.h +++ b/src/mongo/db/repl/member_config.h @@ -115,7 +115,7 @@ public: } /** - * Gets the horizon name for which the parameters (captured during the first `hello`) + * Gets the horizon name for which the parameters (captured during the first `isMaster`) * correspond. */ StringData determineHorizon(const SplitHorizon::Parameters& params) const { @@ -194,7 +194,7 @@ public: } /** - * Returns true if this member is hidden (not reported by "hello", not electable). + * Returns true if this member is hidden (not reported by isMaster, not electable). */ bool isHidden() const { return getHidden(); diff --git a/src/mongo/db/repl/multiapplier_test.cpp b/src/mongo/db/repl/multiapplier_test.cpp index e2424ca1e36..e49eff8992d 100644 --- a/src/mongo/db/repl/multiapplier_test.cpp +++ b/src/mongo/db/repl/multiapplier_test.cpp @@ -73,7 +73,6 @@ OplogEntry makeOplogEntry(int ts) { NamespaceString("a.a"), // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version BSONObj(), // o boost::none, // o2 diff --git a/src/mongo/db/repl/noop_writer.cpp b/src/mongo/db/repl/noop_writer.cpp index ad3fdd197ed..b3adb9da2cf 100644 --- a/src/mongo/db/repl/noop_writer.cpp +++ b/src/mongo/db/repl/noop_writer.cpp @@ -35,7 +35,7 @@ #include "mongo/db/commands.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" #include "mongo/db/namespace_string.h" #include "mongo/db/op_observer.h" diff --git a/src/mongo/db/repl/oplog.cpp b/src/mongo/db/repl/oplog.cpp index 9b750af05ed..453c17c6688 100644 --- a/src/mongo/db/repl/oplog.cpp +++ b/src/mongo/db/repl/oplog.cpp @@ -45,7 +45,6 @@ #include "mongo/db/auth/action_type.h" #include "mongo/db/auth/authorization_manager.h" #include "mongo/db/auth/privilege.h" -#include "mongo/db/catalog/backwards_compatible_collection_options_util.h" #include "mongo/db/catalog/capped_utils.h" #include "mongo/db/catalog/coll_mod.h" #include "mongo/db/catalog/collection.h" @@ -56,7 +55,6 @@ #include "mongo/db/catalog/drop_collection.h" #include "mongo/db/catalog/drop_database.h" #include "mongo/db/catalog/drop_indexes.h" -#include "mongo/db/catalog/health_log_interface.h" #include "mongo/db/catalog/import_collection_oplog_entry_gen.h" #include "mongo/db/catalog/local_oplog_info.h" #include "mongo/db/catalog/multi_index_block.h" @@ -64,9 +62,8 @@ #include "mongo/db/client.h" #include "mongo/db/coll_mod_gen.h" #include "mongo/db/commands.h" -#include "mongo/db/commands/create_gen.h" #include "mongo/db/commands/feature_compatibility_version_parser.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" #include "mongo/db/dbhelpers.h" @@ -120,7 +117,6 @@ using std::string; using std::stringstream; using std::unique_ptr; using std::vector; -using namespace std::string_literals; using IndexVersion = IndexDescriptor::IndexVersion; @@ -293,6 +289,23 @@ void writeToImageCollection(OperationContext* opCtx, DisableDocumentValidation documentValidationDisabler( opCtx, DocumentValidationSettings::kDisableInternalValidation); + BSONObj existingImageEntryBson; + Helpers::findOne(opCtx, + autoColl.getCollection(), + BSON("_id" << imageEntry.get_id().toBSON() << "ts" << imageEntry.getTs()), + existingImageEntryBson); + if (!existingImageEntryBson.isEmpty()) { + auto existingImageEntry = repl::ImageEntry::parse( + IDLParserErrorContext("writeToImageCollection"), existingImageEntryBson); + uassert( + 6652600, + str::stream() + << "Found an existing findAndModify image entry with unexpected content. Found: " + << existingImageEntry.toBSON() << ". Expected: " << imageEntry.toBSON(), + existingImageEntry.toBSON().woCompare(imageEntry.toBSON()) == 0); + return; + } + UpdateRequest request; request.setNamespaceString(NamespaceString::kConfigImagesNamespace); request.setQuery( @@ -793,21 +806,6 @@ NamespaceString extractNsFromUUIDorNs(OperationContext* opCtx, return ui ? extractNsFromUUID(opCtx, ui.get()) : extractNs(ns.db(), cmd); } -BSONObj getObjWithSanitizedStorageEngineOptions(OperationContext* opCtx, const BSONObj& cmd) { - static_assert( - CreateCommand::kStorageEngineFieldName == IndexDescriptor::kStorageEngineFieldName, - "Expected storage engine options field to be the same for collections and indexes."); - - if (auto storageEngineElem = cmd[IndexDescriptor::kStorageEngineFieldName]) { - auto storageEngine = opCtx->getServiceContext()->getStorageEngine(); - auto engineObj = storageEngineElem.embeddedObject(); - auto sanitizedObj = - storageEngine->getSanitizedStorageOptionsForSecondaryReplication(engineObj); - return cmd.addFields(BSON(IndexDescriptor::kStorageEngineFieldName << sanitizedObj)); - } - return cmd; -} - using OpApplyFn = std::function<Status( OperationContext* opCtx, const OplogEntry& entry, OplogApplication::Mode mode)>; @@ -832,9 +830,7 @@ const StringMap<ApplyOpMetadata> kOpsMap = { {"create", {[](OperationContext* opCtx, const OplogEntry& entry, OplogApplication::Mode mode) -> Status { const auto& ui = entry.getUuid(); - // Sanitize storage engine options to remove options which might not apply to this node. - // See SERVER-68122. - const auto cmd = getObjWithSanitizedStorageEngineOptions(opCtx, entry.getObject()); + const auto& cmd = entry.getObject(); const NamespaceString nss(extractNs(entry.getNss().db(), cmd)); // Mode SECONDARY steady state replication should not allow create collection to rename an @@ -844,23 +840,6 @@ const StringMap<ApplyOpMetadata> kOpsMap = { // complete. const bool allowRenameOutOfTheWay = (mode != repl::OplogApplication::Mode::kSecondary); - // Check whether there is an open but empty database where the name conflicts with the new - // collection's database name. It is possible for a secondary's in-memory database state - // to diverge from the primary's, if the primary rolls back the dropDatabase oplog entry - // after closing its own in-memory database state. In this case, the primary may accept - // creating a new database with a conflicting name to what the secondary still has open. - // It is okay to simply close the empty database on the secondary in this case. - auto duplicates = DatabaseHolder::get(opCtx)->getNamesWithConflictingCasing( - TenantDatabaseName(boost::none, nss.db())); - if (duplicates.size() == 1) { - auto dupDatabaseIt = duplicates.begin(); - if (CollectionCatalog::get(opCtx) - ->getAllCollectionUUIDsFromDb(*dupDatabaseIt) - .size() == 0) { - fassert(7727801, dropDatabaseForApplyOps(opCtx, dupDatabaseIt->dbName()).isOK()); - } - } - Lock::DBLock dbLock(opCtx, nss.db(), MODE_IX); if (auto idIndexElem = cmd["idIndex"]) { // Remove "idIndex" field from command. @@ -895,10 +874,7 @@ const StringMap<ApplyOpMetadata> kOpsMap = { {ErrorCodes::NamespaceExists}}}, {"createIndexes", {[](OperationContext* opCtx, const OplogEntry& entry, OplogApplication::Mode mode) -> Status { - // Sanitize storage engine options to remove options which might not apply to this node. - // See SERVER-68122. - const auto cmd = getObjWithSanitizedStorageEngineOptions(opCtx, entry.getObject()); - + const auto& cmd = entry.getObject(); if (OplogApplication::Mode::kApplyOpsCmd == mode) { return {ErrorCodes::CommandNotSupported, "The createIndexes operation is not supported in applyOps mode"}; @@ -933,12 +909,6 @@ const StringMap<ApplyOpMetadata> kOpsMap = { "Error parsing 'startIndexBuild' oplog entry"); } - // Sanitize storage engine options to remove options which might not apply to this node. - // See SERVER-68122. - for (auto& spec : swOplogEntry.getValue().indexSpecs) { - spec = getObjWithSanitizedStorageEngineOptions(opCtx, spec); - } - IndexBuildsCoordinator::ApplicationMode applicationMode = IndexBuildsCoordinator::ApplicationMode::kNormal; if (mode == OplogApplication::Mode::kInitialSync) { @@ -989,8 +959,7 @@ const StringMap<ApplyOpMetadata> kOpsMap = { {ErrorCodes::NamespaceNotFound}}}, {"collMod", {[](OperationContext* opCtx, const OplogEntry& entry, OplogApplication::Mode mode) -> Status { - const auto cmd = - backwards_compatible_collection_options::parseCollModCmdFromOplogEntry(entry); + const auto& cmd = entry.getObject(); auto opMsg = OpMsgRequest::fromDBAndBody(entry.getNss().db(), cmd); auto collModCmd = CollMod::parse(IDLParserErrorContext("collModOplogEntry"), opMsg); const auto nssOrUUID([&collModCmd, &entry, mode]() -> NamespaceStringOrUUID { @@ -1141,8 +1110,6 @@ void writeChangeStreamPreImage(OperationContext* opCtx, constexpr StringData OplogApplication::kInitialSyncOplogApplicationMode; constexpr StringData OplogApplication::kRecoveringOplogApplicationMode; -constexpr StringData OplogApplication::kStableRecoveringOplogApplicationMode; -constexpr StringData OplogApplication::kUnstableRecoveringOplogApplicationMode; constexpr StringData OplogApplication::kSecondaryOplogApplicationMode; constexpr StringData OplogApplication::kApplyOpsCmdOplogApplicationMode; @@ -1150,10 +1117,8 @@ StringData OplogApplication::modeToString(OplogApplication::Mode mode) { switch (mode) { case OplogApplication::Mode::kInitialSync: return OplogApplication::kInitialSyncOplogApplicationMode; - case OplogApplication::Mode::kUnstableRecovering: - return OplogApplication::kUnstableRecoveringOplogApplicationMode; - case OplogApplication::Mode::kStableRecovering: - return OplogApplication::kStableRecoveringOplogApplicationMode; + case OplogApplication::Mode::kRecovering: + return OplogApplication::kRecoveringOplogApplicationMode; case OplogApplication::Mode::kSecondary: return OplogApplication::kSecondaryOplogApplicationMode; case OplogApplication::Mode::kApplyOpsCmd: @@ -1166,9 +1131,7 @@ StatusWith<OplogApplication::Mode> OplogApplication::parseMode(const std::string if (mode == OplogApplication::kInitialSyncOplogApplicationMode) { return OplogApplication::Mode::kInitialSync; } else if (mode == OplogApplication::kRecoveringOplogApplicationMode) { - // This only being used in applyOps command which is controlled by the client, so it should - // be unstable. - return OplogApplication::Mode::kUnstableRecovering; + return OplogApplication::Mode::kRecovering; } else if (mode == OplogApplication::kSecondaryOplogApplicationMode) { return OplogApplication::Mode::kSecondary; } else if (mode == OplogApplication::kApplyOpsCmdOplogApplicationMode) { @@ -1180,76 +1143,6 @@ StatusWith<OplogApplication::Mode> OplogApplication::parseMode(const std::string MONGO_UNREACHABLE; } -void OplogApplication::checkOnOplogFailureForRecovery(OperationContext* opCtx, - const mongo::NamespaceString& nss, - const mongo::BSONObj& oplogEntry, - const std::string& errorMsg) { - const bool isReplicaSet = - repl::ReplicationCoordinator::get(opCtx->getServiceContext())->getReplicationMode() == - repl::ReplicationCoordinator::modeReplSet; - // Relax the constraints of oplog application if the node is not a replica set member or the - // node is in the middle of a backup and restore process. - if (!isReplicaSet || storageGlobalParams.restore) { - return; - } - - // During the recovery process, certain configuration collections such as - // 'config.image_collections' are handled differently, which may result in encountering oplog - // application failures in common scenarios, and therefore assert statements are not used. - if (nss.isConfigDB()) { - LOGV2_DEBUG( - 5415002, - 1, - "Error applying operation while recovering from stable checkpoint. This is related to " - "one of the configuration collections so this error might be benign.", - "oplogEntry"_attr = oplogEntry, - "error"_attr = errorMsg); - } else if (getTestCommandsEnabled()) { - // Only fassert in test environment. - LOGV2_FATAL(5415000, - "Error applying operation while recovering from stable " - "checkpoint. This can lead to data corruption.", - "oplogEntry"_attr = oplogEntry, - "error"_attr = errorMsg); - } else { - LOGV2_WARNING(5415001, - "Error applying operation while recovering from stable " - "checkpoint. This can lead to data corruption.", - "oplogEntry"_attr = oplogEntry, - "error"_attr = errorMsg); - } -} - -// Logger for oplog constraint violations. -OplogConstraintViolationLogger* oplogConstraintViolationLogger; - -MONGO_INITIALIZER(CreateOplogConstraintViolationLogger)(InitializerContext* context) { - oplogConstraintViolationLogger = new OplogConstraintViolationLogger(); -} - -void logOplogConstraintViolation(OperationContext* opCtx, - const NamespaceString& nss, - OplogConstraintViolationEnum type, - const std::string& operation, - const BSONObj& opObj, - boost::optional<Status> status) { - // Log the violation. - oplogConstraintViolationLogger->logViolationIfReady(type, opObj, status); - - // Write a new entry to the health log. - HealthLogEntry entry; - entry.setNss(nss); - entry.setTimestamp(Date_t::now()); - // Oplog constraint violations should always be marked as warning. - entry.setSeverity(SeverityEnum::Warning); - entry.setScope(ScopeEnum::Document); - entry.setMsg(toString(type)); - entry.setOperation(operation); - entry.setData(opObj); - - HealthLogInterface::get(opCtx->getServiceContext())->log(entry); -} - // @return failure status if an update should have happened and the document DNE. // See replset initial sync code. Status applyOperation_inlock(OperationContext* opCtx, @@ -1286,22 +1179,11 @@ Status applyOperation_inlock(OperationContext* opCtx, return Status::OK(); } - const bool inStableRecovery = mode == OplogApplication::Mode::kStableRecovering; NamespaceString requestNss; CollectionPtr collection = nullptr; if (auto uuid = op.getUuid()) { auto catalog = CollectionCatalog::get(opCtx); collection = catalog->lookupCollectionByUUID(opCtx, uuid.get()); - if (!collection && inStableRecovery) { - repl::OplogApplication::checkOnOplogFailureForRecovery( - opCtx, - op.getNss(), - redact(op.toBSONForLogging()), - str::stream() - << "(NamespaceNotFound): Failed to apply operation due to missing collection (" - << uuid.value() << ")"); - } - uassert(ErrorCodes::NamespaceNotFound, str::stream() << "Failed to apply operation due to missing collection (" << uuid.get() << "): " << redact(opOrGroupedInserts.toBSON()), @@ -1369,7 +1251,7 @@ Status applyOperation_inlock(OperationContext* opCtx, case ReplicationCoordinator::modeNone: { // Only assign timestamps on standalones during replication recovery when // started with the 'recoverFromOplogAsStandalone' flag. - return OplogApplication::inRecovering(mode); + return mode == OplogApplication::Mode::kRecovering; } } } @@ -1388,7 +1270,8 @@ Status applyOperation_inlock(OperationContext* opCtx, // correct pre-image for them. return collection && collection->isChangeStreamPreAndPostImagesEnabled() && isDataConsistent && - (OplogApplication::inRecovering(mode) || mode == OplogApplication::Mode::kSecondary) && + (mode == OplogApplication::Mode::kRecovering || + mode == OplogApplication::Mode::kSecondary) && !op.getFromMigrate().get_value_or(false) && !requestNss.isTemporaryReshardingCollection(); }; @@ -1528,23 +1411,10 @@ Status applyOperation_inlock(OperationContext* opCtx, return status; } if (mode == OplogApplication::Mode::kSecondary) { - const auto& opObj = redact(op.toBSONForLogging()); - opCounters->gotInsertOnExistingDoc(); - logOplogConstraintViolation( - opCtx, - op.getNss(), - OplogConstraintViolationEnum::kInsertOnExistingDoc, - "insert", - opObj, - boost::none /* status */); - if (oplogApplicationEnforcesSteadyStateConstraints) { return status; } - } else if (inStableRecovery) { - repl::OplogApplication::checkOnOplogFailureForRecovery( - opCtx, op.getNss(), redact(op.toBSONForLogging()), redact(status)); } // Continue to the next block to retry the operation as an upsert. needToDoUpsert = true; @@ -1619,13 +1489,13 @@ Status applyOperation_inlock(OperationContext* opCtx, request.setNamespaceString(requestNss); request.setQuery(updateCriteria); // If we are in steady state and the update is on a timeseries bucket collection, we can - // enable some optimizations in diff application. In some cases, like during tenant - // migration or $_internalApplyOplogUpdate update, we can for some reason generate - // entries for timeseries bucket collections which still rely on the idempotency - // guarantee, which then means we shouldn't apply these optimizations. + // enable some optimizations in diff application. In some cases, during tenant + // migration, we can for some reason generate entries for timeseries bucket collections + // which still rely on the idempotency guarantee, which then means we shouldn't apply + // these optimizations. write_ops::UpdateModification::DiffOptions options; if (mode == OplogApplication::Mode::kSecondary && collection->getTimeseriesOptions() && - !op.getCheckExistenceForDiffInsert() && !op.getFromTenantMigration()) { + !op.getFromTenantMigration()) { options.mustCheckExistenceForInsertOperations = false; } auto updateMod = write_ops::UpdateModification::parseFromOplogEntry(o, options); @@ -1762,15 +1632,10 @@ Status applyOperation_inlock(OperationContext* opCtx, !ur.upsertedId.isEmpty() && !(collection && collection->isCapped())) { // This indicates we upconverted an update to an upsert, and it did indeed // upsert. In steady state mode this is unexpected. - const auto& opObj = redact(op.toBSONForLogging()); - + LOGV2_WARNING(2170001, + "update needed to be converted to upsert", + "op"_attr = redact(op.toBSONForLogging())); opCounters->gotUpdateOnMissingDoc(); - logOplogConstraintViolation(opCtx, - op.getNss(), - OplogConstraintViolationEnum::kUpdateOnMissingDoc, - "update", - opObj, - boost::none /* status */); // We shouldn't be doing upserts in secondary mode when enforcing steady state // constraints. @@ -1807,10 +1672,6 @@ Status applyOperation_inlock(OperationContext* opCtx, }); if (!status.isOK()) { - if (inStableRecovery) { - repl::OplogApplication::checkOnOplogFailureForRecovery( - opCtx, op.getNss(), redact(op.toBSONForLogging()), redact(status)); - } return status; } @@ -1892,22 +1753,16 @@ Status applyOperation_inlock(OperationContext* opCtx, writeChangeStreamPreImage(opCtx, collection, op, *(result.requestedPreImage)); } - if (result.nDeleted == 0 && inStableRecovery) { - repl::OplogApplication::checkOnOplogFailureForRecovery( - opCtx, - op.getNss(), - redact(op.toBSONForLogging()), - !collection ? str::stream() - << "(NamespaceNotFound): Failed to apply operation due " - "to missing collection (" - << requestNss << ")" - : "Applied a delete which did not delete anything."s); - } // It is legal for a delete operation on the pre-images collection to delete zero // documents - pre-image collections are not guaranteed to contain the same set of // documents at all times. if (result.nDeleted == 0 && mode == OplogApplication::Mode::kSecondary && !requestNss.isChangeStreamPreImagesCollection()) { + LOGV2_WARNING(2170002, + "Applied a delete which did not delete anything in steady state " + "replication", + "op"_attr = redact(op.toBSONForLogging())); + // In FCV 4.4, each node is responsible for deleting the excess documents in // capped collections. This implies that capped deletes may not be synchronized // between nodes at times. When upgraded to FCV 5.0, the primary will generate @@ -1920,25 +1775,11 @@ Status applyOperation_inlock(OperationContext* opCtx, // capped collections when oplog application is enforcing steady state // constraints. bool isCapped = false; - const auto& opObj = redact(op.toBSONForLogging()); if (collection) { isCapped = collection->isCapped(); opCounters->gotDeleteWasEmpty(); - logOplogConstraintViolation(opCtx, - op.getNss(), - OplogConstraintViolationEnum::kDeleteWasEmpty, - "delete", - opObj, - boost::none /* status */); } else { opCounters->gotDeleteFromMissingNamespace(); - logOplogConstraintViolation( - opCtx, - op.getNss(), - OplogConstraintViolationEnum::kDeleteOnMissingNs, - "delete", - opObj, - boost::none /* status */); } if (!isCapped) { @@ -2057,7 +1898,7 @@ Status applyCommand_inlock(OperationContext* opCtx, case ReplicationCoordinator::modeNone: { // Only assign timestamps on standalones during replication recovery when // started with 'recoverFromOplogAsStandalone'. - return OplogApplication::inRecovering(mode); + return mode == OplogApplication::Mode::kRecovering; } } MONGO_UNREACHABLE; @@ -2128,17 +1969,12 @@ Status applyCommand_inlock(OperationContext* opCtx, "aborting index build and retrying", logAttrs(ns)); } else { - opCtx->recoveryUnit()->abandonSnapshot(); - auto lockState = opCtx->lockState(); Locker::LockSnapshot lockSnapshot; - bool canSaveState = lockState->canSaveLockState(); - if (canSaveState) { - lockState->saveLockStateAndUnlock(&lockSnapshot); - } + auto locksReleased = lockState->saveLockStateAndUnlock(&lockSnapshot); ScopeGuard guard{[&] { - if (canSaveState) { + if (locksReleased) { invariant(!lockState->isLocked()); lockState->restoreLockState(lockSnapshot); } @@ -2154,6 +1990,7 @@ Status applyCommand_inlock(OperationContext* opCtx, IndexBuildsCoordinator::get(opCtx)->awaitNoIndexBuildInProgressForCollection( opCtx, swUUID.get()); + opCtx->recoveryUnit()->abandonSnapshot(); opCtx->checkForInterrupt(); LOGV2_DEBUG( @@ -2191,15 +2028,12 @@ Status applyCommand_inlock(OperationContext* opCtx, if (mode == OplogApplication::Mode::kSecondary && status.code() != ErrorCodes::IndexNotFound) { - const auto& opObj = redact(entry.toBSONForLogging()); + LOGV2_WARNING(2170000, + "Acceptable error during oplog application", + "db"_attr = nss.db(), + "error"_attr = status, + "oplogEntry"_attr = redact(entry.toBSONForLogging())); opCounters->gotAcceptableErrorInCommand(); - logOplogConstraintViolation( - opCtx, - entry.getNss(), - OplogConstraintViolationEnum::kAcceptableErrorInCommand, - "command", - opObj, - status); } else { LOGV2_DEBUG(51776, 1, diff --git a/src/mongo/db/repl/oplog.h b/src/mongo/db/repl/oplog.h index 3df367954b0..c3c9c2de13a 100644 --- a/src/mongo/db/repl/oplog.h +++ b/src/mongo/db/repl/oplog.h @@ -38,7 +38,6 @@ #include "mongo/bson/timestamp.h" #include "mongo/db/catalog/collection_options.h" #include "mongo/db/logical_session_id.h" -#include "mongo/db/repl/oplog_constraint_violation_logger.h" #include "mongo/db/repl/oplog_entry.h" #include "mongo/db/repl/oplog_entry_or_grouped_inserts.h" #include "mongo/db/repl/optime.h" @@ -172,10 +171,7 @@ using IncrementOpsAppliedStatsFn = std::function<void()>; class OplogApplication { public: static constexpr StringData kInitialSyncOplogApplicationMode = "InitialSync"_sd; - // This only being used in 'applyOps' command when sent by client. static constexpr StringData kRecoveringOplogApplicationMode = "Recovering"_sd; - static constexpr StringData kStableRecoveringOplogApplicationMode = "StableRecovering"_sd; - static constexpr StringData kUnstableRecoveringOplogApplicationMode = "UnstableRecovering"_sd; static constexpr StringData kSecondaryOplogApplicationMode = "Secondary"_sd; static constexpr StringData kApplyOpsCmdOplogApplicationMode = "ApplyOps"_sd; @@ -184,12 +180,9 @@ public: kInitialSync, // Used when we are applying oplog operations to recover the database state following an - // clean/unclean shutdown, or when we are recovering from the oplog after we rollback to a + // unclean shutdown, or when we are recovering from the oplog after we rollback to a // checkpoint. - // If recovering from a unstable stable checkpoint. - kUnstableRecovering, - // If recovering from a stable checkpoint.~ - kStableRecovering, + kRecovering, // Used when a secondary node is applying oplog operations from the primary during steady // state replication. @@ -200,20 +193,9 @@ public: kApplyOpsCmd }; - static bool inRecovering(Mode mode) { - return mode == Mode::kUnstableRecovering || mode == Mode::kStableRecovering; - } - static StringData modeToString(Mode mode); static StatusWith<Mode> parseMode(const std::string& mode); - - // Server will crash on oplog application failure during recovery from stable checkpoint in the - // test environment. - static void checkOnOplogFailureForRecovery(OperationContext* opCtx, - const mongo::NamespaceString& nss, - const mongo::BSONObj& oplogEntry, - const std::string& errorMsg); }; inline std::ostream& operator<<(std::ostream& s, OplogApplication::Mode mode) { @@ -221,16 +203,6 @@ inline std::ostream& operator<<(std::ostream& s, OplogApplication::Mode mode) { } /** - * Logs an oplog constraint violation and writes an entry into the health log. - */ -void logOplogConstraintViolation(OperationContext* opCtx, - const NamespaceString& nss, - OplogConstraintViolationEnum type, - const std::string& operation, - const BSONObj& opObj, - boost::optional<Status> status); - -/** * Used for applying from an oplog entry or grouped inserts. * @param opOrGroupedInserts a single oplog entry or grouped inserts to be applied. * @param alwaysUpsert convert some updates to upserts for idempotency reasons diff --git a/src/mongo/db/repl/oplog_applier.h b/src/mongo/db/repl/oplog_applier.h index 252070077a2..b582b282efd 100644 --- a/src/mongo/db/repl/oplog_applier.h +++ b/src/mongo/db/repl/oplog_applier.h @@ -67,10 +67,10 @@ public: Options() = delete; explicit Options(OplogApplication::Mode inputMode) : mode(inputMode), - allowNamespaceNotFoundErrorsOnCrudOps(inputMode == - OplogApplication::Mode::kInitialSync || - OplogApplication::inRecovering(inputMode)), - skipWritesToOplog(OplogApplication::inRecovering(inputMode)) {} + allowNamespaceNotFoundErrorsOnCrudOps( + inputMode == OplogApplication::Mode::kInitialSync || + inputMode == OplogApplication::Mode::kRecovering), + skipWritesToOplog(inputMode == OplogApplication::Mode::kRecovering) {} // Used to determine which operations should be applied. Only initial sync will set this to // be something other than the null optime. diff --git a/src/mongo/db/repl/oplog_applier_impl.cpp b/src/mongo/db/repl/oplog_applier_impl.cpp index 3ce566c713b..c5ca80bb8d6 100644 --- a/src/mongo/db/repl/oplog_applier_impl.cpp +++ b/src/mongo/db/repl/oplog_applier_impl.cpp @@ -38,6 +38,7 @@ #include "mongo/db/catalog/document_validation.h" #include "mongo/db/catalog_raii.h" #include "mongo/db/client.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/logical_session_id.h" #include "mongo/db/repl/apply_ops.h" @@ -103,11 +104,6 @@ Status finishAndLogApply(OperationContext* opCtx, attrs.add("duration", Milliseconds(opDuration)); - // Obtain storage specific statistics and log them if they exist. - CurOp::get(opCtx)->debug().storageStats = - opCtx->recoveryUnit()->computeOperationStatisticsSinceLastCall(); - CurOp::get(opCtx)->debug().reportStorageStats(&attrs); - LOGV2(51801, "Applied op", attrs); } } @@ -151,9 +147,6 @@ protected: // We have to use setMyLastAppliedOpTimeAndWallTimeForward since this thread races with // ReplicationExternalStateImpl::onTransitionToPrimary. _replCoord->setMyLastAppliedOpTimeAndWallTimeForward(newOpTimeAndWallTime); - // We know we're at a no-holes point and we've already advanced visibility; we need - // to notify waiters since we changed the lastAppliedSnapshot. - signalOplogWaiters(); } void _recordDurable(const OpTimeAndWallTime& newOpTimeAndWallTime) { diff --git a/src/mongo/db/repl/oplog_applier_impl_test.cpp b/src/mongo/db/repl/oplog_applier_impl_test.cpp index 26e142f6372..f9c41a19877 100644 --- a/src/mongo/db/repl/oplog_applier_impl_test.cpp +++ b/src/mongo/db/repl/oplog_applier_impl_test.cpp @@ -44,6 +44,7 @@ #include "mongo/db/client.h" #include "mongo/db/commands/feature_compatibility_version_parser.h" #include "mongo/db/concurrency/d_concurrency.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" @@ -67,6 +68,7 @@ #include "mongo/db/session_txn_record_gen.h" #include "mongo/db/stats/counters.h" #include "mongo/db/transaction_participant_gen.h" +#include "mongo/idl/server_parameter_test_util.h" #include "mongo/platform/mutex.h" #include "mongo/unittest/death_test.h" #include "mongo/unittest/unittest.h" @@ -128,7 +130,7 @@ TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, NamespaceString otherNss("test.othername"); auto op = makeOplogEntry(OpTypeEnum::kDelete, otherNss, {}); int prevDeleteFromMissing = replOpCounters.getDeleteFromMissingNamespace()->load(); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, otherNss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, false); auto postDeleteFromMissing = replOpCounters.getDeleteFromMissingNamespace()->load(); ASSERT_EQ(1, postDeleteFromMissing - prevDeleteFromMissing); @@ -166,7 +168,7 @@ TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, NamespaceString otherNss(nss.getSisterNS("othername")); auto op = makeOplogEntry(OpTypeEnum::kDelete, otherNss, kUuid); int prevDeleteFromMissing = replOpCounters.getDeleteFromMissingNamespace()->load(); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, false); auto postDeleteFromMissing = replOpCounters.getDeleteFromMissingNamespace()->load(); ASSERT_EQ(1, postDeleteFromMissing - prevDeleteFromMissing); @@ -210,7 +212,7 @@ TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, // implicitly create the collection. auto op = makeOplogEntry(OpTypeEnum::kDelete, nss, {}); int prevDeleteFromMissing = replOpCounters.getDeleteFromMissingNamespace()->load(); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, false); ASSERT_FALSE(collectionExists(_opCtx.get(), nss)); auto postDeleteFromMissing = replOpCounters.getDeleteFromMissingNamespace()->load(); ASSERT_EQ(1, postDeleteFromMissing - prevDeleteFromMissing); @@ -238,7 +240,7 @@ TEST_F(OplogApplierImplTest, applyOplogEntryOrGroupedInsertsInsertDocumentCollec const NamespaceString nss("test.t"); repl::createCollection(_opCtx.get(), nss, {}); auto op = makeOplogEntry(OpTypeEnum::kInsert, nss, {}); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, true); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, true); } TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, @@ -247,7 +249,7 @@ TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, repl::createCollection(_opCtx.get(), nss, {}); auto op = makeOplogEntry(OpTypeEnum::kDelete, nss, {}); int prevDeleteWasEmpty = replOpCounters.getDeleteWasEmpty()->load(); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, false); auto postDeleteWasEmpty = replOpCounters.getDeleteWasEmpty()->load(); ASSERT_EQ(1, postDeleteWasEmpty - prevDeleteWasEmpty); @@ -273,7 +275,7 @@ TEST_F(OplogApplierImplTest, applyOplogEntryOrGroupedInsertsDeleteDocumentCollec createCollection(_opCtx.get(), nss, createRecordPreImageCollectionOptions()); ASSERT_OK(getStorageInterface()->insertDocument(_opCtx.get(), nss, {BSON("_id" << 0)}, 0)); auto op = makeOplogEntry(OpTypeEnum::kDelete, nss, {}); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, true); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, true); } TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, @@ -283,7 +285,7 @@ TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, ASSERT_OK(getStorageInterface()->insertDocument(_opCtx.get(), nss, {BSON("_id" << 0)}, 0)); auto op = makeOplogEntry(OpTypeEnum::kInsert, nss, uuid); int prevInsertOnExistingDoc = replOpCounters.getInsertOnExistingDoc()->load(); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, false); auto postInsertOnExistingDoc = replOpCounters.getInsertOnExistingDoc()->load(); ASSERT_EQ(1, postInsertOnExistingDoc - prevInsertOnExistingDoc); @@ -300,7 +302,7 @@ TEST_F(OplogApplierImplTestEnableSteadyStateConstraints, auto uuid = createCollectionWithUuid(_opCtx.get(), nss); ASSERT_OK(getStorageInterface()->insertDocument(_opCtx.get(), nss, {BSON("_id" << 0)}, 0)); auto op = makeOplogEntry(OpTypeEnum::kInsert, nss, uuid); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::DuplicateKey, op, nss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::DuplicateKey, op, false); } TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, @@ -310,7 +312,7 @@ TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, auto op = makeOplogEntry( repl::OpTypeEnum::kUpdate, nss, uuid, BSON("$set" << BSON("a" << 1)), BSON("_id" << 0)); int prevUpdateOnMissingDoc = replOpCounters.getUpdateOnMissingDoc()->load(); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, true); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, true); auto postUpdateOnMissingDoc = replOpCounters.getUpdateOnMissingDoc()->load(); ASSERT_EQ(1, postUpdateOnMissingDoc - prevUpdateOnMissingDoc); @@ -327,8 +329,7 @@ TEST_F(OplogApplierImplTestEnableSteadyStateConstraints, auto uuid = createCollectionWithUuid(_opCtx.get(), nss); auto op = makeOplogEntry( repl::OpTypeEnum::kUpdate, nss, uuid, BSON("$set" << BSON("a" << 1)), BSON("_id" << 0)); - _testApplyOplogEntryOrGroupedInsertsCrudOperation( - ErrorCodes::UpdateOperationFailed, op, nss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::UpdateOperationFailed, op, false); } TEST_F(OplogApplierImplTest, applyOplogEntryOrGroupedInsertsInsertDocumentCollectionLockedByUUID) { @@ -337,7 +338,7 @@ TEST_F(OplogApplierImplTest, applyOplogEntryOrGroupedInsertsInsertDocumentCollec // Test that the collection to lock is determined by the UUID and not the 'ns' field. NamespaceString otherNss(nss.getSisterNS("othername")); auto op = makeOplogEntry(OpTypeEnum::kInsert, otherNss, uuid); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, true); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, true); } TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, @@ -351,7 +352,7 @@ TEST_F(OplogApplierImplTestDisableSteadyStateConstraints, NamespaceString otherNss(nss.getSisterNS("othername")); auto op = makeOplogEntry(OpTypeEnum::kDelete, otherNss, options.uuid); int prevDeleteWasEmpty = replOpCounters.getDeleteWasEmpty()->load(); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, false); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, false); auto postDeleteWasEmpty = replOpCounters.getDeleteWasEmpty()->load(); ASSERT_EQ(1, postDeleteWasEmpty - prevDeleteWasEmpty); @@ -388,7 +389,7 @@ TEST_F(OplogApplierImplTest, applyOplogEntryOrGroupedInsertsDeleteDocumentCollec // Test that the collection to lock is determined by the UUID and not the 'ns' field. NamespaceString otherNss(nss.getSisterNS("othername")); auto op = makeOplogEntry(OpTypeEnum::kDelete, otherNss, options.uuid); - _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, nss, true); + _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::OK, op, true); } TEST_F(OplogApplierImplTest, applyOplogEntryToRecordChangeStreamPreImages) { @@ -421,14 +422,11 @@ TEST_F(OplogApplierImplTest, applyOplogEntryToRecordChangeStreamPreImages) { } }; generateTestCasesForOperations(OplogApplication::Mode::kSecondary, {}, true); - generateTestCasesForOperations(OplogApplication::Mode::kUnstableRecovering, {}, true); - generateTestCasesForOperations(OplogApplication::Mode::kStableRecovering, {}, true); + generateTestCasesForOperations(OplogApplication::Mode::kRecovering, {}, true); generateTestCasesForOperations(OplogApplication::Mode::kInitialSync, {}, false); const auto kFromMigrate{true}; generateTestCasesForOperations(OplogApplication::Mode::kSecondary, kFromMigrate, false); - generateTestCasesForOperations( - OplogApplication::Mode::kUnstableRecovering, kFromMigrate, false); - generateTestCasesForOperations(OplogApplication::Mode::kStableRecovering, kFromMigrate, false); + generateTestCasesForOperations(OplogApplication::Mode::kRecovering, kFromMigrate, false); generateTestCasesForOperations(OplogApplication::Mode::kInitialSync, kFromMigrate, false); int docId{0}; @@ -862,7 +860,7 @@ TEST_F(MultiOplogEntryOplogApplierImplTest, MultiApplyUnpreparedTransactionAllAt ReplicationCoordinator::get(_opCtx.get()), getConsistencyMarkers(), getStorageInterface(), - repl::OplogApplier::Options(repl::OplogApplication::Mode::kStableRecovering), + repl::OplogApplier::Options(repl::OplogApplication::Mode::kRecovering), _writerPool.get()); // Apply both inserts and the commit in a single batch. We expect no oplog entries to @@ -1344,7 +1342,7 @@ TEST_F(MultiOplogEntryPreparedTransactionTest, MultiApplyPreparedTransactionReco ReplicationCoordinator::get(_opCtx.get()), getConsistencyMarkers(), getStorageInterface(), - repl::OplogApplier::Options(repl::OplogApplication::Mode::kStableRecovering), + repl::OplogApplier::Options(repl::OplogApplication::Mode::kRecovering), _writerPool.get()); // Apply a batch with the insert operations. This should have no effect, because this is @@ -1578,7 +1576,7 @@ TEST_F(MultiOplogEntryPreparedTransactionTest, ReplicationCoordinator::get(_opCtx.get()), getConsistencyMarkers(), getStorageInterface(), - repl::OplogApplier::Options(repl::OplogApplication::Mode::kStableRecovering), + repl::OplogApplier::Options(repl::OplogApplication::Mode::kRecovering), _writerPool.get()); const auto expectedStartOpTime = _singlePrepareApplyOp->getOpTime(); @@ -2687,7 +2685,6 @@ public: ns, // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert 0, // version object, // o object2, // o2 @@ -2721,7 +2718,6 @@ public: ns, // namespace boost::none, // uuid true, // fromMigrate - boost::none, // checkExistenceForDiffInsert 0, // version object, // o object2, // o2 diff --git a/src/mongo/db/repl/oplog_applier_impl_test_fixture.cpp b/src/mongo/db/repl/oplog_applier_impl_test_fixture.cpp index 4afe05e7bc5..949a69c2a29 100644 --- a/src/mongo/db/repl/oplog_applier_impl_test_fixture.cpp +++ b/src/mongo/db/repl/oplog_applier_impl_test_fixture.cpp @@ -32,9 +32,7 @@ #include "mongo/db/repl/oplog_applier_impl_test_fixture.h" #include "mongo/db/catalog/document_validation.h" -#include "mongo/db/catalog/health_log.h" -#include "mongo/db/catalog/health_log_interface.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" @@ -184,22 +182,14 @@ void OplogApplierImplTest::setUp() { // This is necessary to generate ghost timestamps for index builds that are not 0, since 0 is an // invalid timestamp. VectorClockMutable::get(_opCtx.get())->tickClusterTimeTo(LogicalTime(Timestamp(1, 0))); - - HealthLogInterface::set(serviceContext, std::make_unique<HealthLog>()); - HealthLogInterface::get(serviceContext)->startup(); } void OplogApplierImplTest::tearDown() { - HealthLogInterface::get(serviceContext)->shutdown(); _opCtx.reset(); _consistencyMarkers = {}; DropPendingCollectionReaper::set(serviceContext, {}); StorageInterface::set(serviceContext, {}); ServiceContextMongoDTest::tearDown(); - - for (auto serverParamController : _serverParamControllers) { - serverParamController.reset(); - } } ReplicationConsistencyMarkers* OplogApplierImplTest::getConsistencyMarkers() const { @@ -223,10 +213,7 @@ Status OplogApplierImplTest::_applyOplogEntryOrGroupedInsertsWrapper( } void OplogApplierImplTest::_testApplyOplogEntryOrGroupedInsertsCrudOperation( - ErrorCodes::Error expectedError, - const OplogEntry& op, - const NamespaceString& targetNss, - bool expectedApplyOpCalled) { + ErrorCodes::Error expectedError, const OplogEntry& op, bool expectedApplyOpCalled) { bool applyOpCalled = false; auto checkOpCtx = [](OperationContext* opCtx) { @@ -241,14 +228,9 @@ void OplogApplierImplTest::_testApplyOplogEntryOrGroupedInsertsCrudOperation( _opObserver->onInsertsFn = [&](OperationContext* opCtx, const NamespaceString& nss, const std::vector<BSONObj>& docs) { - // Other threads may be calling into the opObserver. Only assert if we are writing to - // the target ns, otherwise skip these asserts. - if (targetNss != nss) { - return Status::OK(); - } - applyOpCalled = true; checkOpCtx(opCtx); + ASSERT_EQUALS(NamespaceString("test.t"), nss); ASSERT_EQUALS(1U, docs.size()); // For upserts we don't know the intended value of the document. if (op.getOpType() == repl::OpTypeEnum::kInsert) { @@ -262,28 +244,18 @@ void OplogApplierImplTest::_testApplyOplogEntryOrGroupedInsertsCrudOperation( const boost::optional<UUID>& uuid, StmtId stmtId, const OplogDeleteEntryArgs& args) { - // Other threads may be calling into the opObserver. Only assert if we are writing to - // the target ns, otherwise skip these asserts. - if (targetNss != nss) { - return Status::OK(); - } - applyOpCalled = true; checkOpCtx(opCtx); + ASSERT_EQUALS(NamespaceString("test.t"), nss); ASSERT(args.deletedDoc); ASSERT_BSONOBJ_EQ(op.getObject(), *(args.deletedDoc)); return Status::OK(); }; _opObserver->onUpdateFn = [&](OperationContext* opCtx, const OplogUpdateEntryArgs& args) { - // Other threads may be calling into the opObserver. Only assert if we are writing to - // the target ns, otherwise skip these asserts. - if (targetNss != args.nss) { - return Status::OK(); - } - applyOpCalled = true; checkOpCtx(opCtx); + ASSERT_EQUALS(NamespaceString("test.t"), args.nss); return Status::OK(); }; @@ -441,7 +413,6 @@ OplogEntry makeOplogEntry(OpTypeEnum opType, nss, // namespace uuid, // uuid fromMigrate, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version o, // o o2, // o2 diff --git a/src/mongo/db/repl/oplog_applier_impl_test_fixture.h b/src/mongo/db/repl/oplog_applier_impl_test_fixture.h index 28709a29b2c..e1b188232ae 100644 --- a/src/mongo/db/repl/oplog_applier_impl_test_fixture.h +++ b/src/mongo/db/repl/oplog_applier_impl_test_fixture.h @@ -38,7 +38,6 @@ #include "mongo/db/repl/replication_consistency_markers.h" #include "mongo/db/service_context_d_test_fixture.h" #include "mongo/db/session_txn_record_gen.h" -#include "mongo/idl/server_parameter_test_util.h" namespace mongo { @@ -203,7 +202,6 @@ protected: void _testApplyOplogEntryOrGroupedInsertsCrudOperation(ErrorCodes::Error expectedError, const OplogEntry& op, - const NamespaceString& targetNss, bool expectedApplyOpCalled); Status _applyOplogEntryOrGroupedInsertsWrapper(OperationContext* opCtx, @@ -215,11 +213,6 @@ protected: ServiceContext* serviceContext; OplogApplierImplOpObserver* _opObserver = nullptr; - template <typename T> - inline void setServerParameter(const std::string& name, T value) { - _serverParamControllers.push_back(ServerParameterControllerForTest(name, value)); - } - OpTime nextOpTime() { static long long lastSecond = 1; return OpTime(Timestamp(Seconds(lastSecond++), 0), 1LL); @@ -239,8 +232,6 @@ protected: Status runOpsInitialSync(std::vector<OplogEntry> ops); UUID kUuid{UUID::gen()}; - - std::vector<ServerParameterControllerForTest> _serverParamControllers; }; // Utility class to allow easily scanning a collection. Scans in forward order, returns diff --git a/src/mongo/db/repl/oplog_applier_utils.cpp b/src/mongo/db/repl/oplog_applier_utils.cpp index 5b1061e3998..65f8298af89 100644 --- a/src/mongo/db/repl/oplog_applier_utils.cpp +++ b/src/mongo/db/repl/oplog_applier_utils.cpp @@ -34,9 +34,8 @@ #include "mongo/db/catalog/collection.h" #include "mongo/db/catalog/collection_catalog.h" #include "mongo/db/catalog/document_validation.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" -#include "mongo/db/curop_metrics.h" #include "mongo/db/db_raii.h" #include "mongo/db/repl/oplog_applier_utils.h" #include "mongo/db/repl/repl_server_parameters_gen.h" @@ -200,11 +199,9 @@ Status OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon( OpCounters* opCounters) { invariant(DocumentValidationSettings::get(opCtx).isSchemaValidationDisabled()); + auto op = entryOrGroupedInserts.getOp(); // Count each log op application as a separate operation, for reporting purposes CurOp individualOp(opCtx); - ON_BLOCK_EXIT([opCtx]() { recordCurOpMetricsOplogApplication(opCtx); }); - - auto op = entryOrGroupedInserts.getOp(); const NamespaceString nss(op.getNss()); auto opType = op.getOpType(); if (opType == OpTypeEnum::kNoop) { @@ -256,15 +253,7 @@ Status OplogApplierUtils::applyOplogEntryOrGroupedInsertsCommon( !oplogApplicationEnforcesSteadyStateConstraints && oplogApplicationMode == OplogApplication::Mode::kSecondary) { if (opCounters) { - const auto& opObj = redact(op.toBSONForLogging()); opCounters->gotDeleteFromMissingNamespace(); - logOplogConstraintViolation( - opCtx, - op.getNss(), - OplogConstraintViolationEnum::kDeleteOnMissingNs, - "delete", - opObj, - boost::none /* status */); } return Status::OK(); } @@ -315,7 +304,6 @@ Status OplogApplierUtils::applyOplogBatchCommon( InsertGroup insertGroup( ops, opCtx, oplogApplicationMode, isDataConsistent, applyOplogEntryOrGroupedInserts); - const bool inStableRecovery = oplogApplicationMode == OplogApplication::Mode::kStableRecovering; for (auto it = ops->cbegin(); it != ops->cend(); ++it) { const OplogEntry& entry = **it; @@ -335,18 +323,9 @@ Status OplogApplierUtils::applyOplogBatchCommon( if (!status.isOK()) { // Tried to apply an update operation but the document is missing, there must be // a delete operation for the document later in the oplog. - // Server will crash on oplog application failure during recovery from stable - // checkpoint in the test environment. if (status == ErrorCodes::UpdateOperationFailed && (oplogApplicationMode == OplogApplication::Mode::kInitialSync || - OplogApplication::inRecovering(oplogApplicationMode))) { - if (inStableRecovery) { - repl::OplogApplication::checkOnOplogFailureForRecovery( - opCtx, - entry.getNss(), - redact(entry.toBSONForLogging()), - redact(status)); - } + oplogApplicationMode == OplogApplication::Mode::kRecovering)) { continue; } @@ -360,14 +339,8 @@ Status OplogApplierUtils::applyOplogBatchCommon( } catch (const DBException& e) { // SERVER-24927 If we have a NamespaceNotFound exception, then this document will be // dropped before initial sync or recovery ends anyways and we should ignore it. - // Server will crash on oplog application failure during recovery from stable checkpoint - // in the test environment. if (e.code() == ErrorCodes::NamespaceNotFound && entry.isCrudOpType() && allowNamespaceNotFoundErrorsOnCrudOps) { - if (inStableRecovery) { - repl::OplogApplication::checkOnOplogFailureForRecovery( - opCtx, entry.getNss(), redact(entry.toBSONForLogging()), redact(e)); - } continue; } diff --git a/src/mongo/db/repl/oplog_batcher_test_fixture.cpp b/src/mongo/db/repl/oplog_batcher_test_fixture.cpp index cb21043383f..c05ce1c4de0 100644 --- a/src/mongo/db/repl/oplog_batcher_test_fixture.cpp +++ b/src/mongo/db/repl/oplog_batcher_test_fixture.cpp @@ -185,7 +185,6 @@ OplogEntry makeInsertOplogEntry(int t, const NamespaceString& nss, boost::option nss, // namespace uuid, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version oField, // o boost::none, // o2 @@ -219,7 +218,6 @@ OplogEntry makeUpdateOplogEntry(int t, nss, // namespace uuid, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version oField, // o boost::none, // o2 @@ -244,7 +242,6 @@ OplogEntry makeNoopOplogEntry(int t, const StringData& msg) { NamespaceString(""), // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version oField, // o boost::none, // o2 @@ -281,7 +278,6 @@ OplogEntry makeApplyOpsOplogEntry(int t, bool prepare, const std::vector<OplogEn nss, // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version oField.obj(), // o boost::none, // o2 @@ -318,7 +314,6 @@ OplogEntry makeCommitTransactionOplogEntry(int t, StringData dbName, bool prepar nss, // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version oField, // o boost::none, // o2 @@ -382,7 +377,6 @@ OplogEntry makeLargeTransactionOplogEntries(int t, nss, // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version oField, // o boost::none, // o2 diff --git a/src/mongo/db/repl/oplog_buffer_collection_test.cpp b/src/mongo/db/repl/oplog_buffer_collection_test.cpp index a320cc56acc..f93278f45e3 100644 --- a/src/mongo/db/repl/oplog_buffer_collection_test.cpp +++ b/src/mongo/db/repl/oplog_buffer_collection_test.cpp @@ -33,6 +33,7 @@ #include "mongo/db/catalog/database.h" #include "mongo/db/client.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/json.h" diff --git a/src/mongo/db/repl/oplog_constraint_violation_logger.cpp b/src/mongo/db/repl/oplog_constraint_violation_logger.cpp deleted file mode 100644 index f887393dbdf..00000000000 --- a/src/mongo/db/repl/oplog_constraint_violation_logger.cpp +++ /dev/null @@ -1,90 +0,0 @@ -/** - * Copyright (C) 2023-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kReplication - -#include "mongo/db/repl/oplog_constraint_violation_logger.h" -#include "mongo/logv2/log.h" - -namespace mongo { -namespace repl { - -// Default interval set to 10 minutes. -const Seconds OplogConstraintViolationLogger::kPeriodicLogTimeout(60 * 10); - -StringData toString(OplogConstraintViolationEnum type) { - switch (type) { - case OplogConstraintViolationEnum::kInsertOnExistingDoc: - return kInsertOnExistingDocMsg; - case OplogConstraintViolationEnum::kUpdateOnMissingDoc: - return kUpdateOnMissingDocMsg; - case OplogConstraintViolationEnum::kDeleteWasEmpty: - return kDeleteWasEmptyMsg; - case OplogConstraintViolationEnum::kDeleteOnMissingNs: - return kDeleteOnMissingNs; - case OplogConstraintViolationEnum::kAcceptableErrorInCommand: - return kAcceptableErrorInCommand; - default: - return ""; - } -} - -void OplogConstraintViolationLogger::logViolationIfReady(OplogConstraintViolationEnum type, - const BSONObj& obj, - boost::optional<Status> status) { - const auto index = static_cast<int>(type); - - stdx::lock_guard lk(_mutex); - const auto lastLog = _lastLogTimes[index]; - const auto now = Date_t::now(); - - if (now < lastLog + OplogConstraintViolationLogger::kPeriodicLogTimeout) { - // We have logged this violation already within the last 10 minutes. - return; - } - - if (!status) { - LOGV2_WARNING(7149000, - "Potential replication constraint violation during steady state replication", - "msg"_attr = toString(type), - "obj"_attr = obj); - } else { - LOGV2_WARNING(7149001, - "Potential replication constraint violation during steady state replication", - "msg"_attr = toString(type), - "obj"_attr = obj, - "status"_attr = *status); - } - - // Update the last log time to now. - _lastLogTimes[index] = now; -} - -} // namespace repl -} // namespace mongo diff --git a/src/mongo/db/repl/oplog_constraint_violation_logger.h b/src/mongo/db/repl/oplog_constraint_violation_logger.h deleted file mode 100644 index 321544e28d9..00000000000 --- a/src/mongo/db/repl/oplog_constraint_violation_logger.h +++ /dev/null @@ -1,85 +0,0 @@ -/** - * Copyright (C) 2023-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include <boost/optional.hpp> - -#include "mongo/base/status.h" -#include "mongo/bson/bsonobj.h" -#include "mongo/platform/mutex.h" -#include "mongo/util/time_support.h" - -namespace mongo { -namespace repl { - -static constexpr StringData kInsertOnExistingDocMsg = "attempted to insert on existing doc"_sd; -static constexpr StringData kUpdateOnMissingDocMsg = - "ran update as upsert and failed to match any documents"_sd; -static constexpr StringData kDeleteWasEmptyMsg = "applied a delete that did not delete anything"_sd; -static constexpr StringData kDeleteOnMissingNs = "applied a delete on missing namespace"_sd; -static constexpr StringData kAcceptableErrorInCommand = - "received an acceptable error during oplog application"_sd; - -enum class OplogConstraintViolationEnum { - kInsertOnExistingDoc = 0, - kUpdateOnMissingDoc, - kDeleteWasEmpty, - kDeleteOnMissingNs, - kAcceptableErrorInCommand, - NUM_VIOLATION_TYPES, -}; - -// Returns a string describing the constraint violation of the given type. -StringData toString(OplogConstraintViolationEnum type); - -/** - * Logs oplog constraint violation occurrences. - * - * To avoid flooding the logs if continuous oplog constraint violations occur, we will only log - * once every 10 minutes per each oplog constraint violation type. - */ -class OplogConstraintViolationLogger { -public: - // Minimum period of time before logging another warning log message, set to 10min. - static const Seconds kPeriodicLogTimeout; - - void logViolationIfReady(OplogConstraintViolationEnum type, - const BSONObj& obj, - boost::optional<Status> status); - -private: - mutable Mutex _mutex = MONGO_MAKE_LATCH("OplogConstraintViolationLogger::mutex"); - - std::vector<Date_t> _lastLogTimes = std::vector<Date_t>( - static_cast<int>(OplogConstraintViolationEnum::NUM_VIOLATION_TYPES)); // (M) -}; - -} // namespace repl -} // namespace mongo diff --git a/src/mongo/db/repl/oplog_entry.cpp b/src/mongo/db/repl/oplog_entry.cpp index 7f7363be06e..f55e7352b79 100644 --- a/src/mongo/db/repl/oplog_entry.cpp +++ b/src/mongo/db/repl/oplog_entry.cpp @@ -53,7 +53,6 @@ BSONObj makeOplogEntryDoc(OpTime opTime, const NamespaceString& nss, const boost::optional<UUID>& uuid, const boost::optional<bool>& fromMigrate, - const boost::optional<bool>& checkExistenceForDiffInsert, int64_t version, const BSONObj& oField, const boost::optional<BSONObj>& o2Field, @@ -90,10 +89,6 @@ BSONObj makeOplogEntryDoc(OpTime opTime, if (fromMigrate) { builder.append(OplogEntryBase::kFromMigrateFieldName, fromMigrate.get()); } - if (checkExistenceForDiffInsert) { - builder.append(OplogEntryBase::kCheckExistenceForDiffInsertFieldName, - checkExistenceForDiffInsert.value()); - } builder.append(OplogEntryBase::kObjectFieldName, oField); if (o2Field) { builder.append(OplogEntryBase::kObject2FieldName, o2Field.get()); @@ -328,7 +323,6 @@ DurableOplogEntry::DurableOplogEntry(OpTime opTime, const NamespaceString& nss, const boost::optional<UUID>& uuid, const boost::optional<bool>& fromMigrate, - const boost::optional<bool>& checkExistenceForDiffInsert, int version, const BSONObj& oField, const boost::optional<BSONObj>& o2Field, @@ -349,7 +343,6 @@ DurableOplogEntry::DurableOplogEntry(OpTime opTime, nss, uuid, fromMigrate, - checkExistenceForDiffInsert, version, oField, o2Field, @@ -646,10 +639,6 @@ const boost::optional<bool> OplogEntry::getFromMigrate() const& { return _entry.getFromMigrate(); } -bool OplogEntry::getCheckExistenceForDiffInsert() const& { - return _entry.getCheckExistenceForDiffInsert().get_value_or(false); -} - const boost::optional<mongo::UUID>& OplogEntry::getFromTenantMigration() const& { return _entry.getFromTenantMigration(); } diff --git a/src/mongo/db/repl/oplog_entry.h b/src/mongo/db/repl/oplog_entry.h index 7552faa3f9e..40092e47f4e 100644 --- a/src/mongo/db/repl/oplog_entry.h +++ b/src/mongo/db/repl/oplog_entry.h @@ -87,13 +87,11 @@ public: o.parseProtected(ctxt, bsonObject); return o; } - - const BSONObj& getPostImageDocumentKey() const { - return _postImageDocumentKey; + const BSONObj& getPreImageDocumentKey() const { + return _preImageDocumentKey; } - - void setPostImageDocumentKey(BSONObj value) { - _postImageDocumentKey = std::move(value); + void setPreImageDocumentKey(BSONObj value) { + _preImageDocumentKey = std::move(value); } const BSONObj& getPreImage() const { @@ -217,8 +215,7 @@ public: } private: - // Stores the post image _id + shard key values. - BSONObj _postImageDocumentKey; + BSONObj _preImageDocumentKey; // Used for storing the pre-image and post-image for the operation in-memory regardless of where // the images should be persisted. @@ -371,10 +368,6 @@ public: getDurableReplOperation().setFromMigrate(value); } - void setCheckExistenceForDiffInsert() & { - getDurableReplOperation().setCheckExistenceForDiffInsert(true); - } - /** * Same as setFromMigrate but only set when it is true. */ @@ -392,7 +385,6 @@ class DurableOplogEntry : private MutableOplogEntry { public: // Make field names accessible. using MutableOplogEntry::k_idFieldName; - using MutableOplogEntry::kCheckExistenceForDiffInsertFieldName; using MutableOplogEntry::kDestinedRecipientFieldName; using MutableOplogEntry::kDurableReplOperationFieldName; using MutableOplogEntry::kFromMigrateFieldName; @@ -420,7 +412,6 @@ public: // Make serialize() and getters accessible. using MutableOplogEntry::get_id; - using MutableOplogEntry::getCheckExistenceForDiffInsert; using MutableOplogEntry::getDestinedRecipient; using MutableOplogEntry::getDurableReplOperation; using MutableOplogEntry::getFromMigrate; @@ -487,7 +478,6 @@ public: const NamespaceString& nss, const boost::optional<UUID>& uuid, const boost::optional<bool>& fromMigrate, - const boost::optional<bool>& checkExistenceForDiffInsert, int version, const BSONObj& oField, const boost::optional<BSONObj>& o2Field, @@ -646,8 +636,6 @@ public: static constexpr auto kDurableReplOperationFieldName = DurableOplogEntry::kDurableReplOperationFieldName; static constexpr auto kFromMigrateFieldName = DurableOplogEntry::kFromMigrateFieldName; - static constexpr auto kCheckExistenceForDiffInsertFieldName = - DurableOplogEntry::kCheckExistenceForDiffInsertFieldName; static constexpr auto kFromTenantMigrationFieldName = DurableOplogEntry::kFromTenantMigrationFieldName; static constexpr auto kHashFieldName = DurableOplogEntry::kHashFieldName; @@ -720,7 +708,6 @@ public: const boost::optional<std::int64_t> getHash() const&; std::int64_t getVersion() const; const boost::optional<bool> getFromMigrate() const&; - bool getCheckExistenceForDiffInsert() const&; const boost::optional<mongo::UUID>& getFromTenantMigration() const&; const boost::optional<mongo::repl::OpTime>& getPrevWriteOpTimeInTransaction() const&; const boost::optional<mongo::repl::OpTime>& getPostImageOpTime() const&; diff --git a/src/mongo/db/repl/oplog_entry.idl b/src/mongo/db/repl/oplog_entry.idl index b193903bbc9..7c1ba09f320 100644 --- a/src/mongo/db/repl/oplog_entry.idl +++ b/src/mongo/db/repl/oplog_entry.idl @@ -131,11 +131,6 @@ structs: type: bool optional: true description: "An operation caused by a chunk migration" - checkExistenceForDiffInsert: - type: bool - optional: true - description: "Marks that checks on field existence are needed for insert operations - in the diff update." OplogEntryBase: description: A document in which the server stores an oplog entry. diff --git a/src/mongo/db/repl/oplog_entry_test_helpers.cpp b/src/mongo/db/repl/oplog_entry_test_helpers.cpp index 1592e10da3c..870287dd8f6 100644 --- a/src/mongo/db/repl/oplog_entry_test_helpers.cpp +++ b/src/mongo/db/repl/oplog_entry_test_helpers.cpp @@ -67,7 +67,6 @@ repl::OplogEntry makeOplogEntry(repl::OpTime opTime, nss, // namespace uuid, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert repl::OplogEntry::kOplogVersion, // version object, // o object2, // o2 diff --git a/src/mongo/db/repl/oplog_fetcher.cpp b/src/mongo/db/repl/oplog_fetcher.cpp index 4747e392542..816b5f19fa6 100644 --- a/src/mongo/db/repl/oplog_fetcher.cpp +++ b/src/mongo/db/repl/oplog_fetcher.cpp @@ -719,21 +719,8 @@ StatusWith<OplogFetcher::Documents> OplogFetcher::_getNextBatch() { auto lastCommittedWithCurrentTerm = _dataReplicatorExternalState->getCurrentTermAndLastCommittedOpTime(); if (lastCommittedWithCurrentTerm.value != OpTime::kUninitializedTerm) { - if (!_cursor->isExhaust() && lastCommittedWithCurrentTerm.opTime.isNull()) { - // For non-exhaust cursors, only set the lastKnownCommittedOpTime when it is not - // a null opTime. This is to avoid sending null opTime again and again and - // triggering oplog empty batches every single time in case we can't advance our - // commit point (e.g. during initial sync). - _cursor->setCurrentTermAndLastCommittedOpTime( - lastCommittedWithCurrentTerm.value, boost::none); - } else { - // For exhaust cursors, it is safe to set a null lastKnownCommittedOpTime in the - // initial getMore because the sync source will update the exhaust cursor's - // lastKnownCommittedOpTime to the commit point sent in the last response after - // each oplog batch. - _cursor->setCurrentTermAndLastCommittedOpTime( - lastCommittedWithCurrentTerm.value, lastCommittedWithCurrentTerm.opTime); - } + _cursor->setCurrentTermAndLastCommittedOpTime(lastCommittedWithCurrentTerm.value, + lastCommittedWithCurrentTerm.opTime); } _cursor->more(); } diff --git a/src/mongo/db/repl/oplog_fetcher_test.cpp b/src/mongo/db/repl/oplog_fetcher_test.cpp index f05229254d2..96003452793 100644 --- a/src/mongo/db/repl/oplog_fetcher_test.cpp +++ b/src/mongo/db/repl/oplog_fetcher_test.cpp @@ -91,7 +91,6 @@ BSONObj makeNoopOplogEntry(OpTime opTime) { NamespaceString("test.t"), // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert repl::OplogEntry::kOplogVersion, // version BSONObj(), // o boost::none, // o2 @@ -203,21 +202,16 @@ void validateGetMoreCommand(Message m, ASSERT_EQ(cursorId, msg.body.getIntField("getMore")); ASSERT_EQUALS(timeout, msg.body.getIntField("maxTimeMS")); - // In unittests, lastCommittedWithCurrentTerm.value should always be a valid term. + // In unittests, lastCommittedWithCurrentTerm should always be default to valid and non-null. // The case when currentTerm is kUninitializedTerm is tested separately in // GetMoreQueryDoesNotContainTermIfGetCurrentTermAndLastCommittedOpTimeReturnsUninitializedTerm. invariant(lastCommittedWithCurrentTerm.value != OpTime::kUninitializedTerm); + invariant(!lastCommittedWithCurrentTerm.opTime.isNull()); ASSERT_EQUALS(lastCommittedWithCurrentTerm.value, msg.body["term"].numberLong()); - if (!exhaustSupported && lastCommittedWithCurrentTerm.opTime.isNull()) { - // Test that we don't attach the lastKnownCommittedOpTime field for non-exhaust cursors when - // the lastCommittedOpTime is null. - ASSERT_FALSE(msg.body.hasField("lastKnownCommittedOpTime")); - } else { - ASSERT_EQUALS(lastCommittedWithCurrentTerm.opTime.getTimestamp(), - msg.body["lastKnownCommittedOpTime"]["ts"].timestamp()); - ASSERT_EQUALS(lastCommittedWithCurrentTerm.opTime.getTerm(), - msg.body["lastKnownCommittedOpTime"]["t"].numberLong()); - } + ASSERT_EQUALS(lastCommittedWithCurrentTerm.opTime.getTimestamp(), + msg.body["lastKnownCommittedOpTime"]["ts"].timestamp()); + ASSERT_EQUALS(lastCommittedWithCurrentTerm.opTime.getTerm(), + msg.body["lastKnownCommittedOpTime"]["t"].numberLong()); if (exhaustSupported) { ASSERT_TRUE(OpMsg::isFlagSet(m, OpMsg::kExhaustSupported)); @@ -1586,14 +1580,6 @@ TEST_F(OplogFetcherTest, OplogFetcherWorksWithoutExhaust) { // Update lastFetched before it is updated by getting the next batch. lastFetched = oplogFetcher->getLastOpTimeFetched_forTest(); - // Set a null lastCommittedOpTime to test that non-exhaust cursors don't attach a null - // lastKnownCommittedOpTime. This must be done before we issue the response to the find request - // so that the first getMore request (made immediately after processSingleRequestResponse) can - // pick this up. - dataReplicatorExternalState->lastCommittedOpTime = OpTime(); - auto firstGetMoreTermAndLastCommittedOpTime = - dataReplicatorExternalState->getCurrentTermAndLastCommittedOpTime(); - // Creating the cursor will succeed. After this, the cursor will be blocked on call() for the // getMore command. auto m = processSingleRequestResponse(oplogFetcher->getDBClientConnection_forTest(), @@ -1613,15 +1599,6 @@ TEST_F(OplogFetcherTest, OplogFetcherWorksWithoutExhaust) { auto fourthEntry = makeNoopOplogEntry({{Seconds(458), 0}, lastFetched.getTerm()}); auto secondBatch = {thirdEntry, fourthEntry}; - - // Reset the lastCommittedOpTime to non-null. This must be done before we issue the response to - // the first getMore request so that the second getMore request (made immediately after - // processSingleRequestResponse) can pick this up. - dataReplicatorExternalState->lastCommittedOpTime = {{9999, 0}, - dataReplicatorExternalState->currentTerm}; - auto secondGetMoreTermAndLastCommittedOpTime = - dataReplicatorExternalState->getCurrentTermAndLastCommittedOpTime(); - // moreToCome would be set to false if oplogFetcherUsesExhaust was set to false. After this, // the cursor will be blocked on call() for the next getMore command. m = processSingleRequestResponse( @@ -1632,7 +1609,7 @@ TEST_F(OplogFetcherTest, OplogFetcherWorksWithoutExhaust) { validateGetMoreCommand(m, cursorId, durationCount<Milliseconds>(oplogFetcher->getAwaitDataTimeout_forTest()), - firstGetMoreTermAndLastCommittedOpTime, + dataReplicatorExternalState->getCurrentTermAndLastCommittedOpTime(), false /* exhaustSupported */); // Update lastFetched since it should have been updated after getting the last batch. @@ -1652,7 +1629,7 @@ TEST_F(OplogFetcherTest, OplogFetcherWorksWithoutExhaust) { validateGetMoreCommand(m, cursorId, durationCount<Milliseconds>(oplogFetcher->getAwaitDataTimeout_forTest()), - secondGetMoreTermAndLastCommittedOpTime, + dataReplicatorExternalState->getCurrentTermAndLastCommittedOpTime(), false /* exhaustSupported */); // Update lastFetched since it should have been updated after getting the last batch. diff --git a/src/mongo/db/repl/primary_only_service.cpp b/src/mongo/db/repl/primary_only_service.cpp index 24e3b534aef..6a093ac6490 100644 --- a/src/mongo/db/repl/primary_only_service.cpp +++ b/src/mongo/db/repl/primary_only_service.cpp @@ -38,6 +38,7 @@ #include "mongo/db/auth/authorization_session.h" #include "mongo/db/catalog_raii.h" #include "mongo/db/client.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/dbdirectclient.h" #include "mongo/db/ops/write_ops.h" #include "mongo/db/repl/repl_client_info.h" @@ -536,8 +537,8 @@ PrimaryOnlyService::getOrCreateInstance(OperationContext* opCtx, BSONObj initial return {newInstance, true}; } -std::pair<boost::optional<std::shared_ptr<PrimaryOnlyService::Instance>>, bool> -PrimaryOnlyService::lookupInstance(OperationContext* opCtx, const InstanceID& id) { +boost::optional<std::shared_ptr<PrimaryOnlyService::Instance>> PrimaryOnlyService::lookupInstance( + OperationContext* opCtx, const InstanceID& id) { // If this operation is holding any database locks, then it must have opted into getting // interrupted at stepdown to prevent deadlocks. invariant(!opCtx->lockState()->isLocked() || opCtx->shouldAlwaysInterruptAtStepDownOrUp() || @@ -547,7 +548,7 @@ PrimaryOnlyService::lookupInstance(OperationContext* opCtx, const InstanceID& id _waitForStateNotRebuilding(opCtx, lk); if (_state == State::kShutdown || _state == State::kPaused) { - return {boost::none, true}; + return boost::none; } if (_state == State::kRebuildFailed) { uassertStatusOK(_rebuildStatus); @@ -557,10 +558,10 @@ PrimaryOnlyService::lookupInstance(OperationContext* opCtx, const InstanceID& id auto it = _activeInstances.find(id); if (it == _activeInstances.end()) { - return {boost::none, false}; + return boost::none; } - return {it->second.getInstance(), false}; + return it->second.getInstance(); } std::vector<std::shared_ptr<PrimaryOnlyService::Instance>> PrimaryOnlyService::getAllInstances( diff --git a/src/mongo/db/repl/primary_only_service.h b/src/mongo/db/repl/primary_only_service.h index 5e30444af12..3ac82044484 100644 --- a/src/mongo/db/repl/primary_only_service.h +++ b/src/mongo/db/repl/primary_only_service.h @@ -156,17 +156,15 @@ public: * Same functionality as PrimaryOnlyService::lookupInstance, but returns a pointer of * the proper derived class for the Instance. */ - static std::pair<boost::optional<std::shared_ptr<InstanceType>>, bool> lookup( - OperationContext* opCtx, PrimaryOnlyService* service, const InstanceID& id) { - auto [instance, isPausedOrShutdown] = service->lookupInstance(opCtx, id); + static boost::optional<std::shared_ptr<InstanceType>> lookup(OperationContext* opCtx, + PrimaryOnlyService* service, + const InstanceID& id) { + auto instance = service->lookupInstance(opCtx, id); if (!instance) { - return {boost::none, isPausedOrShutdown}; + return boost::none; } - // If there is an active instance, the service must be running. - invariant(!isPausedOrShutdown); - - return {checked_pointer_cast<InstanceType>(instance.get()), isPausedOrShutdown}; + return checked_pointer_cast<InstanceType>(instance.get()); } /** @@ -308,14 +306,12 @@ protected: virtual std::shared_ptr<Instance> constructInstance(BSONObj initialState) = 0; /** - * Given an InstanceId returns the corresponding running Instance object (or boost::none if - * there is none), as well as a boolean flag indicating whether the service is paused (i.e. - * stepped down) or shutdown, in which case all the instances have been released so we will - * always return boost::none. If the service state is kRebuilding, we will first wait - * (interruptibly on the opCtx) for the rebuild to complete. + * Given an InstanceId returns the corresponding running Instance object, or boost::none if + * there is none. If the service is in State::kRebuilding, will wait (interruptibly on the + * opCtx) for the rebuild to complete. */ - std::pair<boost::optional<std::shared_ptr<Instance>>, bool> lookupInstance( - OperationContext* opCtx, const InstanceID& id); + boost::optional<std::shared_ptr<Instance>> lookupInstance(OperationContext* opCtx, + const InstanceID& id); /** * Extracts an InstanceID from the _id field of the given 'initialState' object. If an Instance diff --git a/src/mongo/db/repl/primary_only_service_op_observer.h b/src/mongo/db/repl/primary_only_service_op_observer.h index 76c225a483c..5f552149f98 100644 --- a/src/mongo/db/repl/primary_only_service_op_observer.h +++ b/src/mongo/db/repl/primary_only_service_op_observer.h @@ -208,10 +208,6 @@ public: size_t numberOfPrePostImagesToWrite, Date_t wallClockTime) final {} - void onTransactionPrepareNonPrimary(OperationContext* opCtx, - const std::vector<repl::OplogEntry>& statements, - const repl::OpTime& prepareOpTime) final {} - void onTransactionAbort(OperationContext* opCtx, boost::optional<OplogSlot> abortOplogEntryOpTime) final {} diff --git a/src/mongo/db/repl/primary_only_service_test.cpp b/src/mongo/db/repl/primary_only_service_test.cpp index ae8d85390ee..90e420bdafb 100644 --- a/src/mongo/db/repl/primary_only_service_test.cpp +++ b/src/mongo/db/repl/primary_only_service_test.cpp @@ -520,21 +520,16 @@ TEST_F(PrimaryOnlyServiceTest, LookupInstance) { ASSERT(instance.get()); ASSERT_EQ(0, instance->getID()); - auto [instance2, isPausedOrShutdown2] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_TRUE(instance2); - ASSERT_FALSE(isPausedOrShutdown2); + auto instance2 = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)).get(); - ASSERT_EQ(instance.get(), instance2.value().get()); + ASSERT_EQ(instance.get(), instance2.get()); TestServiceHangDuringInitialization.setMode(FailPoint::off); instance->getCompletionFuture().get(); // Shouldn't be able to look up instance after it has completed running. - auto [instance3, isPausedOrShutdown3] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_FALSE(instance3); - ASSERT_FALSE(isPausedOrShutdown3); + auto instance3 = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); + ASSERT_FALSE(instance3.is_initialized()); } TEST_F(PrimaryOnlyServiceTest, LookupInstanceInterruptible) { @@ -569,11 +564,9 @@ TEST_F(PrimaryOnlyServiceTest, LookupInstanceHoldingISLock) { opCtx->setAlwaysInterruptAtStepDownOrUp(); ASSERT_FALSE(opCtx->lockState()->wasGlobalLockTakenInModeConflictingWithWrites()); - auto [instance2, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_TRUE(instance2); - ASSERT_FALSE(isPausedOrShutdown); - ASSERT_EQ(instance.get(), instance2.value().get()); + auto instance2 = + TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)).get(); + ASSERT_EQ(instance.get(), instance2.get()); } TestServiceHangDuringInitialization.setMode(FailPoint::off); @@ -593,11 +586,9 @@ TEST_F(PrimaryOnlyServiceTest, LookupInstanceHoldingIXLock) { { Lock::GlobalLock lk(opCtx.get(), MODE_IX); ASSERT_FALSE(opCtx->shouldAlwaysInterruptAtStepDownOrUp()); - auto [instance2, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_TRUE(instance2); - ASSERT_FALSE(isPausedOrShutdown); - ASSERT_EQ(instance.get(), instance2.value().get()); + auto instance2 = + TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)).get(); + ASSERT_EQ(instance.get(), instance2.get()); } TestServiceHangDuringInitialization.setMode(FailPoint::off); @@ -640,11 +631,9 @@ TEST_F(PrimaryOnlyServiceTest, LookupInstanceAfterStepDownReturnsNone) { stepDown(); - auto [instance2, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); + auto instance2 = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_FALSE(instance2); - ASSERT_TRUE(isPausedOrShutdown); + ASSERT_EQ(instance2, boost::none); TestServiceHangDuringInitialization.setMode(FailPoint::off); ASSERT_EQ(ErrorCodes::Interrupted, instance->getCompletionFuture().getNoThrow()); @@ -665,11 +654,9 @@ TEST_F(PrimaryOnlyServiceTest, LookupInstanceAfterShutDownReturnsNone) { shutdown(); - auto [instance2, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); + auto instance2 = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_FALSE(instance2); - ASSERT_TRUE(isPausedOrShutdown); + ASSERT_EQ(instance2, boost::none); ASSERT_EQ(ErrorCodes::Interrupted, instance->getCompletionFuture().getNoThrow()); } @@ -851,10 +838,8 @@ TEST_F(PrimaryOnlyServiceTest, StepDownBeforePersisted) { auto opCtx = makeOperationContext(); // Since the Instance never wrote its state document, it shouldn't be recreated on stepUp. - auto [recreatedInstance, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_FALSE(recreatedInstance); - ASSERT_FALSE(isPausedOrShutdown); + auto recreatedInstance = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); + ASSERT(!recreatedInstance.is_initialized()); } TEST_F(PrimaryOnlyServiceTest, RecreateInstanceOnStepUp) { @@ -880,13 +865,11 @@ TEST_F(PrimaryOnlyServiceTest, RecreateInstanceOnStepUp) { { auto opCtx = makeOperationContext(); - auto [recreatedInstance, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_TRUE(recreatedInstance); - ASSERT_FALSE(isPausedOrShutdown); - ASSERT_EQ(TestService::State::kOne, (*recreatedInstance)->getInitialState()); + auto recreatedInstance = + TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)).get(); + ASSERT_EQ(TestService::State::kOne, recreatedInstance->getInitialState()); TestServiceHangDuringStateTwo.waitForTimesEntered(++stateTwoFPTimesEntered); - ASSERT_EQ(TestService::State::kTwo, (*recreatedInstance)->getState()); + ASSERT_EQ(TestService::State::kTwo, recreatedInstance->getState()); } stepDown(); @@ -900,19 +883,17 @@ TEST_F(PrimaryOnlyServiceTest, RecreateInstanceOnStepUp) { { auto opCtx = makeOperationContext(); - auto [recreatedInstance, isPausedOrShutdown1] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_TRUE(recreatedInstance); - ASSERT_FALSE(isPausedOrShutdown1); - ASSERT_EQ(TestService::State::kTwo, (*recreatedInstance)->getInitialState()); + auto recreatedInstance = + TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)).get(); + ASSERT_EQ(TestService::State::kTwo, recreatedInstance->getInitialState()); TestServiceHangDuringStateOne.setMode(FailPoint::off); - (*recreatedInstance)->getCompletionFuture().get(); - ASSERT_EQ(TestService::State::kDone, (*recreatedInstance)->getState()); + recreatedInstance->getCompletionFuture().get(); + ASSERT_EQ(TestService::State::kDone, recreatedInstance->getState()); + - auto [nonExistentInstance, isPausedOrShutdown2] = + auto nonExistentInstance = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_FALSE(nonExistentInstance); - ASSERT_FALSE(isPausedOrShutdown2); + ASSERT(!nonExistentInstance.is_initialized()); } stepDown(); @@ -922,10 +903,9 @@ TEST_F(PrimaryOnlyServiceTest, RecreateInstanceOnStepUp) { auto opCtx = makeOperationContext(); // No Instance should be recreated since the previous run completed successfully and deleted // its state document. - auto [nonExistentInstance, isPausedOrShutdown] = + auto nonExistentInstance = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_FALSE(nonExistentInstance); - ASSERT_FALSE(isPausedOrShutdown); + ASSERT(!nonExistentInstance.is_initialized()); } } @@ -981,16 +961,13 @@ TEST_F(PrimaryOnlyServiceTest, StepDownBeforeRebuildingInstances) { TestServiceHangDuringStateOne.waitForTimesEntered(++stateOneFPTimesEntered); auto opCtx = makeOperationContext(); - auto [instance, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_TRUE(instance); - ASSERT_FALSE(isPausedOrShutdown); - ASSERT_EQ(TestService::State::kOne, (*instance)->getInitialState()); - ASSERT_EQ(TestService::State::kOne, (*instance)->getState()); + auto instance = TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)).get(); + ASSERT_EQ(TestService::State::kOne, instance->getInitialState()); + ASSERT_EQ(TestService::State::kOne, instance->getState()); TestServiceHangDuringStateOne.setMode(FailPoint::off); - (*instance)->getCompletionFuture().get(); + instance->getCompletionFuture().get(); } TEST_F(PrimaryOnlyServiceTest, RecreateInstancesFails) { @@ -1034,10 +1011,8 @@ TEST_F(PrimaryOnlyServiceTest, RecreateInstancesFails) { // After stepping down we are in a consistent state again, but cannot create or lookup // instances because we are not primary. auto opCtx = makeOperationContext(); - auto [instance, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_FALSE(instance); - ASSERT_TRUE(isPausedOrShutdown); + ASSERT_FALSE(TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)) + .is_initialized()); ASSERT_THROWS_CODE(TestService::Instance::getOrCreate( opCtx.get(), _service, BSON("_id" << 0 << "state" << 0)), DBException, @@ -1054,15 +1029,13 @@ TEST_F(PrimaryOnlyServiceTest, RecreateInstancesFails) { { // Instance should be recreated successfully. auto opCtx = makeOperationContext(); - auto [instance, isPausedOrShutdown] = - TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)); - ASSERT_TRUE(instance); - ASSERT_FALSE(isPausedOrShutdown); - ASSERT_EQ(TestService::State::kOne, (*instance)->getInitialState()); - ASSERT_EQ(TestService::State::kOne, (*instance)->getState()); + auto instance = + TestService::Instance::lookup(opCtx.get(), _service, BSON("_id" << 0)).get(); + ASSERT_EQ(TestService::State::kOne, instance->getInitialState()); + ASSERT_EQ(TestService::State::kOne, instance->getState()); TestServiceHangDuringStateOne.setMode(FailPoint::off); - (*instance)->getCompletionFuture().get(); - ASSERT_EQ(TestService::State::kDone, (*instance)->getState()); + instance->getCompletionFuture().get(); + ASSERT_EQ(TestService::State::kDone, instance->getState()); } } diff --git a/src/mongo/db/repl/repl_server_parameters.idl b/src/mongo/db/repl/repl_server_parameters.idl index e178b4f4154..227685978ee 100644 --- a/src/mongo/db/repl/repl_server_parameters.idl +++ b/src/mongo/db/repl/repl_server_parameters.idl @@ -654,26 +654,6 @@ server_parameters: default: expr: 15 * 60 * 1000 - unsupportedSyncSource: - description: >- - **Not a supported feature**. Specifies the host/port for a node to use as a sync source. - It is a fatal error to specify a node that is not a part of the replica set config or to - specify the node itself. - set_at: startup - cpp_vartype: std::string - cpp_varname: unsupportedSyncSource - default: "" - validator: { callback: 'validateHostAndPort' } - - skipApplyingDbCheckBatchOnSecondary: - description: >- - Parameter for whether dbcheck batches should be applied on secondaries. - set_at: [ startup, runtime ] - cpp_vartype: AtomicWord<bool> - cpp_varname: skipApplyingDbCheckBatchOnSecondary - default: false - - feature_flags: featureFlagRetryableFindAndModify: description: >- diff --git a/src/mongo/db/repl/repl_set_commands.cpp b/src/mongo/db/repl/repl_set_commands.cpp index a7e27182584..2724219e764 100644 --- a/src/mongo/db/repl/repl_set_commands.cpp +++ b/src/mongo/db/repl/repl_set_commands.cpp @@ -47,6 +47,7 @@ #include "mongo/db/auth/authorization_session.h" #include "mongo/db/commands.h" #include "mongo/db/commands/server_status_metric.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/op_observer.h" #include "mongo/db/repl/drop_pending_collection_reaper.h" @@ -153,17 +154,14 @@ public: result.append("lastStableRecoveryTimestamp", ts.get()); } } else { - LOGV2_WARNING( - 6100700, - "Failed to get last stable recovery timestamp due to lock acquire timeout. " - "Note this is expected if shutdown is in progress."); + LOGV2_WARNING(6100700, + "Failed to get last stable recovery timestamp due to {error}", + "error"_attr = "lock acquire timeout"_sd); } } catch (const ExceptionForCat<ErrorCategory::Interruption>& ex) { - LOGV2_WARNING( - 6100701, - "Failed to get last stable recovery timestamp due to cancellation error. Note " - "this is expected if shutdown is in progress.", - "error"_attr = redact(ex)); + LOGV2_WARNING(6100701, + "Failed to get last stable recovery timestamp due to {error}", + "error"_attr = redact(ex)); } return true; } else if (cmdObj.hasElement("restartHeartbeats")) { diff --git a/src/mongo/db/repl/repl_set_config_checks.cpp b/src/mongo/db/repl/repl_set_config_checks.cpp index cb769f382f8..3be0e88e5d8 100644 --- a/src/mongo/db/repl/repl_set_config_checks.cpp +++ b/src/mongo/db/repl/repl_set_config_checks.cpp @@ -321,21 +321,11 @@ StatusWith<int> findSelfInConfig(ReplicationCoordinatorExternalState* externalSt for (ReplSetConfig::MemberIterator iter = newConfig.membersBegin(); iter != newConfig.membersEnd(); ++iter) { - if (externalState->isSelfFastPath(iter->getHostAndPort())) { + if (externalState->isSelf(iter->getHostAndPort(), ctx)) { meConfigs.push_back(iter); } } if (meConfigs.empty()) { - // No self-hosts were found using the fastpath; check with the slow path. - for (ReplSetConfig::MemberIterator iter = newConfig.membersBegin(); - iter != newConfig.membersEnd(); - ++iter) { - if (externalState->isSelfSlowPath(iter->getHostAndPort(), ctx, Seconds(30))) { - meConfigs.push_back(iter); - } - } - } - if (meConfigs.empty()) { return StatusWith<int>(ErrorCodes::NodeNotFound, str::stream() << "No host described in new configuration with " << newConfig.getConfigVersionAndTerm().toString() diff --git a/src/mongo/db/repl/repl_set_config_checks_test.cpp b/src/mongo/db/repl/repl_set_config_checks_test.cpp index d491bdd9131..21cb645cf10 100644 --- a/src/mongo/db/repl/repl_set_config_checks_test.cpp +++ b/src/mongo/db/repl/repl_set_config_checks_test.cpp @@ -1288,60 +1288,6 @@ TEST_F(ServiceContextTest, FindSelfInConfig) { .getStatus()); } -TEST_F(ServiceContextTest, FindSelfInConfigFastAndSlow) { - ReplSetConfig newConfig; - newConfig = ReplSetConfig::parse(BSON("_id" - << "rs0" - << "version" << 2 << "protocolVersion" << 1 << "members" - << BSON_ARRAY(BSON("_id" << 1 << "host" - << "h1") - << BSON("_id" << 2 << "host" - << "h2") - << BSON("_id" << 3 << "host" - << "h3")))); - - { - // Present once only on the slow path, but fast enough to be found - ReplicationCoordinatorExternalStateMock presentOnceExternalState; - presentOnceExternalState.addSelfSlow(HostAndPort("h2"), Seconds(29)); - ASSERT_EQUALS(1, - unittest::assertGet(findSelfInConfig( - &presentOnceExternalState, newConfig, getServiceContext()))); - } - - { - // Present twice only on the slow path, but fast enough to be found both times. - ReplicationCoordinatorExternalStateMock presentTwiceExternalState; - presentTwiceExternalState.addSelfSlow(HostAndPort("h2"), Seconds(29)); - presentTwiceExternalState.addSelfSlow(HostAndPort("h3"), Seconds(29)); - ASSERT_EQUALS(ErrorCodes::InvalidReplicaSetConfig, - findSelfInConfig(&presentTwiceExternalState, newConfig, getServiceContext()) - .getStatus()); - } - - { - // Present once on the fast path, once on the slow path. This is expected to erroneously - // succeed, because we should not check the slow path if we got a unique result on the fast - // path. - ReplicationCoordinatorExternalStateMock presentFastAndSlowExternalState; - presentFastAndSlowExternalState.addSelf(HostAndPort("h2")); - presentFastAndSlowExternalState.addSelfSlow(HostAndPort("h3"), Seconds(29)); - ASSERT_EQUALS(1, - unittest::assertGet(findSelfInConfig( - &presentFastAndSlowExternalState, newConfig, getServiceContext()))); - } - - { - // Present only on the slow path, with a long timeout. This will fail. - ReplicationCoordinatorExternalStateMock presentLongTimeoutExternalState; - presentLongTimeoutExternalState.addSelfSlow(HostAndPort("h2"), Seconds(31)); - ASSERT_EQUALS( - ErrorCodes::NodeNotFound, - findSelfInConfig(&presentLongTimeoutExternalState, newConfig, getServiceContext()) - .getStatus()); - } -} - } // namespace } // namespace repl } // namespace mongo diff --git a/src/mongo/db/repl/replica_set_aware_service.h b/src/mongo/db/repl/replica_set_aware_service.h index 255beda9d0f..3f099610a5c 100644 --- a/src/mongo/db/repl/replica_set_aware_service.h +++ b/src/mongo/db/repl/replica_set_aware_service.h @@ -132,8 +132,6 @@ public: /** * Called as part of ReplicationCoordinator shutdown. - * Note that it is possible that we are still a writable primary after onShutdown() has been - * called (see SERVER-81115). */ virtual void onShutdown() = 0; diff --git a/src/mongo/db/repl/replication_consistency_markers_impl.cpp b/src/mongo/db/repl/replication_consistency_markers_impl.cpp index 8e7994669de..dde621e310d 100644 --- a/src/mongo/db/repl/replication_consistency_markers_impl.cpp +++ b/src/mongo/db/repl/replication_consistency_markers_impl.cpp @@ -37,6 +37,7 @@ #include "mongo/db/catalog_raii.h" #include "mongo/db/concurrency/d_concurrency.h" #include "mongo/db/concurrency/lock_state.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/repl/optime.h" #include "mongo/db/repl/replication_coordinator.h" #include "mongo/db/repl/storage_interface.h" diff --git a/src/mongo/db/repl/replication_consistency_markers_impl_test.cpp b/src/mongo/db/repl/replication_consistency_markers_impl_test.cpp index 27339c1e53c..3d58ef476e4 100644 --- a/src/mongo/db/repl/replication_consistency_markers_impl_test.cpp +++ b/src/mongo/db/repl/replication_consistency_markers_impl_test.cpp @@ -35,7 +35,7 @@ #include "mongo/db/client.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/namespace_string.h" diff --git a/src/mongo/db/repl/replication_coordinator.h b/src/mongo/db/repl/replication_coordinator.h index b909074b36a..01966a7722a 100644 --- a/src/mongo/db/repl/replication_coordinator.h +++ b/src/mongo/db/repl/replication_coordinator.h @@ -150,10 +150,8 @@ public: * Does whatever cleanup is required to stop replication, including instructing the other * components of the replication system to shut down and stop any threads they are using, * blocking until all replication-related shutdown tasks are complete. - * The parameter `shutdownTimeElapsedBuilder` is for adding time elapsed of tasks done - * in this function into one single builder that records the time elapsed during shutdown. */ - virtual void shutdown(OperationContext* opCtx, BSONObjBuilder* shutdownTimeElapsedBuilder) = 0; + virtual void shutdown(OperationContext* opCtx) = 0; /** * Returns a reference to the parsed command line arguments that are related to replication. @@ -382,11 +380,6 @@ public: * necessarily commit in sequential order. It is also used when we finish oplog batch * application on secondaries, to avoid any potential race conditions around setting the * applied optime from more than one thread. - * - * Since the last applied op time and wall time might not be visible (i.e. there may be - * "oplog holes" from oplog entries with earlier timestamps which commit after this one) - * this method does not notify oplog waiters. Callers which know the new lastApplied is at - * a no-holes point should call signalOplogWaiters after calling this method. */ virtual void setMyLastAppliedOpTimeAndWallTimeForward( const OpTimeAndWallTime& opTimeAndWallTime) = 0; diff --git a/src/mongo/db/repl/replication_coordinator_external_state.h b/src/mongo/db/repl/replication_coordinator_external_state.h index eec7564971c..9bf96582627 100644 --- a/src/mongo/db/repl/replication_coordinator_external_state.h +++ b/src/mongo/db/repl/replication_coordinator_external_state.h @@ -146,20 +146,6 @@ public: virtual bool isSelf(const HostAndPort& host, ServiceContext* service) = 0; /** - * Returns true if "host" is one of the network identities of this node, without actually - * going out to the network and checking. - */ - virtual bool isSelfFastPath(const HostAndPort& host) = 0; - - /** - * Returns true if "host" is one of the network identities of this node, without - * checking the fast path first. - */ - virtual bool isSelfSlowPath(const HostAndPort& host, - ServiceContext* service, - Milliseconds timeout) = 0; - - /** * Gets the replica set config document from local storage, or returns an error. */ virtual StatusWith<BSONObj> loadLocalConfigDocument(OperationContext* opCtx) = 0; diff --git a/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp b/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp index a8f42d8f1dd..22cff587223 100644 --- a/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp +++ b/src/mongo/db/repl/replication_coordinator_external_state_impl.cpp @@ -51,10 +51,11 @@ #include "mongo/db/commands/rwc_defaults_commands_gen.h" #include "mongo/db/commands/server_status_metric.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" #include "mongo/db/dbhelpers.h" +#include "mongo/db/free_mon/free_mon_mongod.h" #include "mongo/db/index_builds_coordinator.h" #include "mongo/db/jsobj.h" #include "mongo/db/kill_sessions_local.h" @@ -536,6 +537,8 @@ OpTime ReplicationCoordinatorExternalStateImpl::onTransitionToPrimary(OperationC IndexBuildsCoordinator::get(opCtx)->onStepUp(opCtx); + notifyFreeMonitoringOnTransitionToPrimary(); + // It is only necessary to check the system indexes on the first transition to primary. // On subsequent transitions to primary the indexes will have already been created. static std::once_flag verifySystemIndexesOnce; @@ -698,16 +701,7 @@ Status ReplicationCoordinatorExternalStateImpl::storeLocalLastVoteDocument( // don't want to have this process interrupted due to us stepping down, since we // want to be able to cast our vote for a new primary right away. Both the write's lock // acquisition and the "waitUntilDurable" lock acquisition must be uninterruptible. - // - // It is not safe to take an uninterruptible lock during STARTUP2, so we only take this lock - // if we are primary or secondary. We do not have the RSTL but that is OK because we never - // move in to STARTUP2 from PRIMARY or SECONDARY, so the consequence of a stale state is - // only that we don't take an uninterruptible lock when we should. - auto* replCoord = ReplicationCoordinator::get(opCtx); - - boost::optional<UninterruptibleLockGuard> noInterrupt; - if (replCoord->isInPrimaryOrSecondaryState_UNSAFE()) - noInterrupt.emplace(opCtx->lockState()); + UninterruptibleLockGuard noInterrupt(opCtx->lockState()); Status status = writeConflictRetry( opCtx, @@ -802,16 +796,6 @@ bool ReplicationCoordinatorExternalStateImpl::isSelf(const HostAndPort& host, Se return repl::isSelf(host, ctx); } -bool ReplicationCoordinatorExternalStateImpl::isSelfFastPath(const HostAndPort& host) { - return repl::isSelfFastPath(host); -} - -bool ReplicationCoordinatorExternalStateImpl::isSelfSlowPath(const HostAndPort& host, - ServiceContext* ctx, - Milliseconds timeout) { - return repl::isSelfSlowPath(host, ctx, timeout); -} - HostAndPort ReplicationCoordinatorExternalStateImpl::getClientHostAndPort( const OperationContext* opCtx) { return HostAndPort(opCtx->getClient()->clientAddress(true)); @@ -910,9 +894,9 @@ void ReplicationCoordinatorExternalStateImpl::_shardingOnTransitionToPrimaryHook if (status.isOK()) { // Load the clusterId into memory. Use local readConcern, since we can't use - // majority/snapshot readConcern in drain mode because the global lock prevents - // replication. This is safe, since if the clusterId write is rolled back, any writes - // that depend on it will also be rolled back. + // majority readConcern in drain mode because the global lock prevents replication. + // This is safe, since if the clusterId write is rolled back, any writes that depend + // on it will also be rolled back. // // Since we *just* wrote the cluster ID to the config.version document (via the call // to ShardingCatalogManager::initializeConfigDatabaseIfNeeded above), this read can @@ -935,43 +919,30 @@ void ReplicationCoordinatorExternalStateImpl::_shardingOnTransitionToPrimaryHook PeriodicShardedIndexConsistencyChecker::get(_service).onStepUp(_service); TransactionCoordinatorService::get(_service)->onStepUp(opCtx); - } else if (serverGlobalParams.clusterRole == ClusterRole::ShardServer) { - if (ShardingState::get(opCtx)->enabled()) { - Status status = ShardingStateRecovery::recover(opCtx); - VectorClockMutable::get(opCtx)->recoverDirect(opCtx); - - // If the node is shutting down or it lost quorum just as it was becoming primary, don't - // run the sharding onStepUp machinery. The onStepDown counterpart to these methods is - // already idempotent, so the machinery will remain in the stepped down state. - if (ErrorCodes::isShutdownError(status.code()) || - ErrorCodes::isNotPrimaryError(status.code())) { - return; - } - fassert(40107, status); - - const auto configsvrConnStr = - Grid::get(opCtx)->shardRegistry()->getConfigShard()->getConnString(); - ShardingInitializationMongoD::get(opCtx)->updateShardIdentityConfigString( - opCtx, configsvrConnStr); - - CatalogCacheLoader::get(_service).onStepUp(); - ChunkSplitter::get(_service).onStepUp(); - PeriodicBalancerConfigRefresher::get(_service).onStepUp(_service); - TransactionCoordinatorService::get(_service)->onStepUp(opCtx); - - // Note, these must be done after the configOpTime is recovered via - // ShardingStateRecovery::recover above, because they may trigger filtering metadata - // refreshes which should use the recovered configOpTime. - migrationutil::resubmitRangeDeletionsOnStepUp(_service); - migrationutil::resumeMigrationCoordinationsOnStepUp(opCtx); - migrationutil::resumeMigrationRecipientsOnStepUp(opCtx); - - const bool scheduleAsyncRefresh = true; - resharding::clearFilteringMetadata(opCtx, scheduleAsyncRefresh); + } else if (ShardingState::get(opCtx)->enabled()) { + Status status = ShardingStateRecovery::recover(opCtx); + VectorClockMutable::get(opCtx)->recoverDirect(opCtx); + + // If the node is shutting down or it lost quorum just as it was becoming primary, don't + // run the sharding onStepUp machinery. The onStepDown counterpart to these methods is + // already idempotent, so the machinery will remain in the stepped down state. + if (ErrorCodes::isShutdownError(status.code()) || + ErrorCodes::isNotPrimaryError(status.code())) { + return; } - // The code above will only be executed after a stepdown happens, however the code below - // needs to be executed also on startup, and the enabled check might fail in shards during - // startup. Create uuid index on config.rangeDeletions if needed + fassert(40107, status); + + const auto configsvrConnStr = + Grid::get(opCtx)->shardRegistry()->getConfigShard()->getConnString(); + ShardingInitializationMongoD::get(opCtx)->updateShardIdentityConfigString(opCtx, + configsvrConnStr); + + CatalogCacheLoader::get(_service).onStepUp(); + ChunkSplitter::get(_service).onStepUp(); + PeriodicBalancerConfigRefresher::get(_service).onStepUp(_service); + TransactionCoordinatorService::get(_service)->onStepUp(opCtx); + + // Create uuid index on config.rangeDeletions if needed auto minKeyFieldName = RangeDeletionTask::kRangeFieldName + "." + ChunkRange::kMinKey; auto maxKeyFieldName = RangeDeletionTask::kRangeFieldName + "." + ChunkRange::kMaxKey; Status indexStatus = createIndexOnConfigCollection( @@ -994,6 +965,16 @@ void ReplicationCoordinatorExternalStateImpl::_shardingOnTransitionToPrimaryHook indexStatus.withContext("Failed to create index on config.rangeDeletions on " "shard's first transition to primary")); } + + // Note, these must be done after the configOpTime is recovered via + // ShardingStateRecovery::recover above, because they may trigger filtering metadata + // refreshes which should use the recovered configOpTime. + migrationutil::resubmitRangeDeletionsOnStepUp(_service); + migrationutil::resumeMigrationCoordinationsOnStepUp(opCtx); + migrationutil::resumeMigrationRecipientsOnStepUp(opCtx); + + const bool scheduleAsyncRefresh = true; + resharding::clearFilteringMetadata(opCtx, scheduleAsyncRefresh); } else { // unsharded if (auto validator = LogicalTimeValidator::get(_service)) { validator->enableKeyGenerator(opCtx, true); diff --git a/src/mongo/db/repl/replication_coordinator_external_state_impl.h b/src/mongo/db/repl/replication_coordinator_external_state_impl.h index 20eb8c61984..9a1e448f636 100644 --- a/src/mongo/db/repl/replication_coordinator_external_state_impl.h +++ b/src/mongo/db/repl/replication_coordinator_external_state_impl.h @@ -81,10 +81,6 @@ public: OpTime onTransitionToPrimary(OperationContext* opCtx) override; virtual void forwardSecondaryProgress(); virtual bool isSelf(const HostAndPort& host, ServiceContext* service); - bool isSelfFastPath(const HostAndPort& host) final; - bool isSelfSlowPath(const HostAndPort& host, - ServiceContext* service, - Milliseconds timeout) final; Status createLocalLastVoteCollection(OperationContext* opCtx) final; virtual StatusWith<BSONObj> loadLocalConfigDocument(OperationContext* opCtx); virtual Status storeLocalConfigDocument(OperationContext* opCtx, diff --git a/src/mongo/db/repl/replication_coordinator_external_state_mock.cpp b/src/mongo/db/repl/replication_coordinator_external_state_mock.cpp index 5f449c1ec51..86aaa76c00f 100644 --- a/src/mongo/db/repl/replication_coordinator_external_state_mock.cpp +++ b/src/mongo/db/repl/replication_coordinator_external_state_mock.cpp @@ -96,24 +96,9 @@ void ReplicationCoordinatorExternalStateMock::forwardSecondaryProgress() {} bool ReplicationCoordinatorExternalStateMock::isSelf(const HostAndPort& host, ServiceContext* const service) { - return sequenceContains(_selfHosts, host) || _selfHostsSlow.find(host) != _selfHostsSlow.end(); -} - -bool ReplicationCoordinatorExternalStateMock::isSelfFastPath(const HostAndPort& host) { return sequenceContains(_selfHosts, host); } -bool ReplicationCoordinatorExternalStateMock::isSelfSlowPath(const HostAndPort& host, - ServiceContext* const service, - Milliseconds timeout) { - if (sequenceContains(_selfHosts, host)) - return true; - auto iter = _selfHostsSlow.find(host); - if (iter == _selfHostsSlow.end()) - return false; - return iter->second <= timeout; -} - void ReplicationCoordinatorExternalStateMock::addSelf(const HostAndPort& host) { _selfHosts.push_back(host); } @@ -122,11 +107,6 @@ void ReplicationCoordinatorExternalStateMock::clearSelfHosts() { _selfHosts.clear(); } -void ReplicationCoordinatorExternalStateMock::addSelfSlow(const HostAndPort& host, - Milliseconds timeout) { - _selfHostsSlow.emplace(host, timeout); -} - HostAndPort ReplicationCoordinatorExternalStateMock::getClientHostAndPort( const OperationContext* opCtx) { return _clientHostAndPort; diff --git a/src/mongo/db/repl/replication_coordinator_external_state_mock.h b/src/mongo/db/repl/replication_coordinator_external_state_mock.h index fd8327f4fee..ecd0f072fed 100644 --- a/src/mongo/db/repl/replication_coordinator_external_state_mock.h +++ b/src/mongo/db/repl/replication_coordinator_external_state_mock.h @@ -73,10 +73,6 @@ public: OpTime onTransitionToPrimary(OperationContext* opCtx) override; virtual void forwardSecondaryProgress(); virtual bool isSelf(const HostAndPort& host, ServiceContext* service); - bool isSelfFastPath(const HostAndPort& host) final; - bool isSelfSlowPath(const HostAndPort& host, - ServiceContext* service, - Milliseconds timeout) final; virtual HostAndPort getClientHostAndPort(const OperationContext* opCtx); virtual StatusWith<BSONObj> loadLocalConfigDocument(OperationContext* opCtx); virtual Status storeLocalConfigDocument(OperationContext* opCtx, @@ -109,20 +105,13 @@ public: /** * Adds "host" to the list of hosts that this mock will match when responding to "isSelf" - * messages, including "isSelfFastPath" and "isSelfSlowPath". + * messages. */ void addSelf(const HostAndPort& host); /** - * Adds "host" to the list of hosts that this mock will match when responding to - * "isSelfSlowPath" messages with a timeout less than or equal to that given, - * but not "isSelfFastPath" messages. - */ - void addSelfSlow(const HostAndPort& host, Milliseconds timeout); - - /** * Remove all hosts from the list of hosts that this mock will match when responding to "isSelf" - * messages. Clears both regular and slow hosts. + * messages. */ void clearSelfHosts(); @@ -219,7 +208,6 @@ private: StatusWith<OpTime> _lastOpTime; StatusWith<Date_t> _lastWallTime; std::vector<HostAndPort> _selfHosts; - stdx::unordered_map<HostAndPort, Milliseconds> _selfHostsSlow; bool _canAcquireGlobalSharedLock; Status _storeLocalConfigDocumentStatus; Status _storeLocalLastVoteDocumentStatus; diff --git a/src/mongo/db/repl/replication_coordinator_impl.cpp b/src/mongo/db/repl/replication_coordinator_impl.cpp index 5f433b99548..5112a683f48 100644 --- a/src/mongo/db/repl/replication_coordinator_impl.cpp +++ b/src/mongo/db/repl/replication_coordinator_impl.cpp @@ -60,7 +60,6 @@ #include "mongo/db/curop.h" #include "mongo/db/curop_failpoint_helpers.h" #include "mongo/db/dbdirectclient.h" -#include "mongo/db/exec/scoped_timer.h" #include "mongo/db/index/index_descriptor.h" #include "mongo/db/index_builds_coordinator.h" #include "mongo/db/kill_sessions_local.h" @@ -111,6 +110,7 @@ #include "mongo/util/assert_util.h" #include "mongo/util/fail_point.h" #include "mongo/util/scopeguard.h" +#include "mongo/util/stacktrace.h" #include "mongo/util/testing_proctor.h" #include "mongo/util/time_support.h" #include "mongo/util/timer.h" @@ -183,13 +183,6 @@ ServerStatusMetricField<Counter64> displayNumAutoReconfigs( "repl.reconfig.numAutoReconfigsForRemovalOfNewlyAddedFields", &numAutoReconfigsForRemovalOfNewlyAddedFields); -Atomic64Metric replicationWaiterListMetric; -ServerStatusMetricField<Atomic64Metric> displayReplicationWaiterListMetric( - "repl.waiters.replication", &replicationWaiterListMetric); -Atomic64Metric opTimeWaiterListMetric; -ServerStatusMetricField<Atomic64Metric> displayOpTimeWaiterListMetric("repl.waiters.opTime", - &opTimeWaiterListMetric); - using namespace fmt::literals; using CallbackArgs = executor::TaskExecutor::CallbackArgs; @@ -227,24 +220,15 @@ constexpr StringData kQuiesceModeShutdownMessage = } // namespace -ReplicationCoordinatorImpl::WaiterList::WaiterList(Atomic64Metric& waiterCountMetric) - : _waiterCountMetric(waiterCountMetric) {} - -void ReplicationCoordinatorImpl::WaiterList::_updateMetric_inlock() { - _waiterCountMetric.set(_waiters.size()); -} - void ReplicationCoordinatorImpl::WaiterList::add_inlock(const OpTime& opTime, SharedWaiterHandle waiter) { _waiters.emplace(opTime, std::move(waiter)); - _updateMetric_inlock(); } SharedSemiFuture<void> ReplicationCoordinatorImpl::WaiterList::add_inlock( const OpTime& opTime, boost::optional<WriteConcernOptions> wc) { auto pf = makePromiseFuture<void>(); _waiters.emplace(opTime, std::make_shared<Waiter>(std::move(pf.promise), std::move(wc))); - _updateMetric_inlock(); return std::move(pf.future); } @@ -252,7 +236,6 @@ bool ReplicationCoordinatorImpl::WaiterList::remove_inlock(SharedWaiterHandle wa for (auto iter = _waiters.begin(); iter != _waiters.end(); iter++) { if (iter->second == waiter) { _waiters.erase(iter); - _updateMetric_inlock(); return true; } } @@ -276,7 +259,6 @@ void ReplicationCoordinatorImpl::WaiterList::setValueIf_inlock(Func&& func, it = _waiters.erase(it); } } - _updateMetric_inlock(); } void ReplicationCoordinatorImpl::WaiterList::setValueAll_inlock() { @@ -284,7 +266,6 @@ void ReplicationCoordinatorImpl::WaiterList::setValueAll_inlock() { waiter->promise.emplaceValue(); } _waiters.clear(); - _updateMetric_inlock(); } void ReplicationCoordinatorImpl::WaiterList::setErrorAll_inlock(Status status) { @@ -293,7 +274,6 @@ void ReplicationCoordinatorImpl::WaiterList::setErrorAll_inlock(Status status) { waiter->promise.setError(status); } _waiters.clear(); - _updateMetric_inlock(); } namespace { @@ -312,7 +292,6 @@ InitialSyncerInterface::Options createInitialSyncerOptions( externalState](const OpTimeAndWallTime& opTimeAndWallTime) { // Note that setting the last applied opTime forward also advances the global timestamp. replCoord->setMyLastAppliedOpTimeAndWallTimeForward(opTimeAndWallTime); - signalOplogWaiters(); // The oplog application phase of initial sync starts timestamping writes, causing // WiredTiger to pin this data in memory. Advancing the oldest timestamp in step with the // last applied optime here will permit WiredTiger to evict this data as it sees fit. @@ -349,8 +328,6 @@ ReplicationCoordinatorImpl::ReplicationCoordinatorImpl( _topCoord(std::move(topCoord)), _replExecutor(std::move(executor)), _externalState(std::move(externalState)), - _replicationWaiterList(replicationWaiterListMetric), - _opTimeWaiterList(opTimeWaiterListMetric), _inShutdown(false), _memberState(MemberState::RS_STARTUP), _rsConfigState(kConfigPreStart), @@ -359,16 +336,6 @@ ReplicationCoordinatorImpl::ReplicationCoordinatorImpl( _readWriteAbility(std::make_unique<ReadWriteAbility>(!settings.usingReplSets())), _replicationProcess(replicationProcess), _storage(storage), - _handleLivenessTimeoutCallback(_replExecutor.get(), - [this](const executor::TaskExecutor::CallbackArgs& args) { - _handleLivenessTimeout(args); - }), - _handleElectionTimeoutCallback( - _replExecutor.get(), - [this](const executor::TaskExecutor::CallbackArgs&) { - _startElectSelfIfEligibleV1(StartElectionReasonEnum::kElectionTimeout); - }, - [this](int64_t limit) { return _nextRandomInt64_inlock(limit); }), _random(prngSeed) { _termShadow.store(OpTime::kUninitializedTerm); @@ -415,7 +382,10 @@ ReplSetConfig ReplicationCoordinatorImpl::getReplicaSetConfig_forTest() { Date_t ReplicationCoordinatorImpl::getElectionTimeout_forTest() const { stdx::lock_guard<Latch> lk(_mutex); - return _handleElectionTimeoutCallback.getNextCall(); + if (!_handleElectionTimeoutCbh.isValid()) { + return Date_t(); + } + return _handleElectionTimeoutWhen; } Milliseconds ReplicationCoordinatorImpl::getRandomizedElectionOffset_forTest() { @@ -444,10 +414,6 @@ executor::TaskExecutor::CallbackHandle ReplicationCoordinatorImpl::getCatchupTak return _catchupTakeoverCbh; } -int64_t ReplicationCoordinatorImpl::getLastHorizonChange_forTest() const { - return _lastHorizonTopologyChange; -} - OpTime ReplicationCoordinatorImpl::getCurrentCommittedSnapshotOpTime() const { stdx::lock_guard<Latch> lk(_mutex); return _getCurrentCommittedSnapshotOpTime_inlock(); @@ -556,8 +522,8 @@ bool ReplicationCoordinatorImpl::_startLoadLocalConfig( LOGV2(4280504, "Cleaning up any partially applied oplog batches & reading last op from oplog"); // Read the last op from the oplog after cleaning up any partially applied batches. - auto stableTimestamp = - _replicationProcess->getReplicationRecovery()->recoverFromOplog(opCtx, boost::none); + const auto stableTimestamp = boost::none; + _replicationProcess->getReplicationRecovery()->recoverFromOplog(opCtx, stableTimestamp); LOGV2(4280505, "Creating any necessary TenantMigrationAccessBlockers for unfinished migrations"); @@ -569,9 +535,7 @@ bool ReplicationCoordinatorImpl::_startLoadLocalConfig( tenant_migration_access_blocker::recoverTenantMigrationAccessBlockers(opCtx); LOGV2(4280506, "Reconstructing prepared transactions"); - reconstructPreparedTransactions(opCtx, - stableTimestamp ? OplogApplication::Mode::kStableRecovering - : OplogApplication::Mode::kUnstableRecovering); + reconstructPreparedTransactions(opCtx, OplogApplication::Mode::kRecovering); const auto lastOpTimeAndWallTimeResult = _externalState->loadLastOpTimeAndWallTime(opCtx); @@ -864,7 +828,6 @@ void ReplicationCoordinatorImpl::_initialSyncerCompletionFunction( const auto lastApplied = opTimeStatus.getValue(); _setMyLastAppliedOpTimeAndWallTime(lock, lastApplied, false); - signalOplogWaiters(); _topCoord->resetMaintenanceCount(); } @@ -926,11 +889,6 @@ void ReplicationCoordinatorImpl::_startDataReplication(OperationContext* opCtx) void ReplicationCoordinatorImpl::startup(OperationContext* opCtx, StorageEngine::LastShutdownState lastShutdownState) { - // Initialize the cached pointer to the oplog collection. We want to do this even as standalone - // so accesses to the cached pointer in replica set nodes started as standalone still work - // (mainly AutoGetOplog). In case the oplog doesn't exist, it is just initialized to null. - acquireOplogCollectionForLogging(opCtx); - if (!isReplEnabled()) { if (ReplSettings::shouldRecoverFromOplogAsStandalone()) { uassert(ErrorCodes::InvalidOptions, @@ -1048,10 +1006,6 @@ bool ReplicationCoordinatorImpl::enterQuiesceModeIfSecondary(Milliseconds quiesc return false; } - // Cancel any ongoing election so that the node cannot become primary once in quiesce mode, - // and do not wait for cancellation to complete. - _cancelElectionIfNeeded(lk); - _inQuiesceMode = true; _quiesceDeadline = _replExecutor->now() + quiesceTime; @@ -1066,8 +1020,7 @@ bool ReplicationCoordinatorImpl::inQuiesceMode() const { return _inQuiesceMode; } -void ReplicationCoordinatorImpl::shutdown(OperationContext* opCtx, - BSONObjBuilder* shutdownTimeElapsedBuilder) { +void ReplicationCoordinatorImpl::shutdown(OperationContext* opCtx) { // Shutdown must: // * prevent new threads from blocking in awaitReplication // * wake up all existing threads blocking in awaitReplication @@ -1077,14 +1030,8 @@ void ReplicationCoordinatorImpl::shutdown(OperationContext* opCtx, return; } - { - auto scopedTimer = - createTimeElapsedBuilderScopedTimer(opCtx->getServiceContext()->getFastClockSource(), - "Shut down the replica set aware services", - shutdownTimeElapsedBuilder); - LOGV2(5074000, "Shutting down the replica set aware services."); - ReplicaSetAwareServiceRegistry::get(_service).onShutdown(); - } + LOGV2(5074000, "Shutting down the replica set aware services."); + ReplicaSetAwareServiceRegistry::get(_service).onShutdown(); LOGV2(21328, "Shutting down replication subsystems"); @@ -1102,19 +1049,11 @@ void ReplicationCoordinatorImpl::shutdown(OperationContext* opCtx, } if (_rsConfigState == kConfigStartingUp) { // Wait until we are finished starting up, so that we can cleanly shut everything down. - auto scopedTimer = createTimeElapsedBuilderScopedTimer( - opCtx->getServiceContext()->getFastClockSource(), - "Wait for startup to complete before shutting down", - shutdownTimeElapsedBuilder); lk.unlock(); _waitForStartUpComplete(); lk.lock(); fassert(18823, _rsConfigState != kConfigStartingUp); } - auto scopedTimer = - createTimeElapsedBuilderScopedTimer(opCtx->getServiceContext()->getFastClockSource(), - "Shut down replication", - shutdownTimeElapsedBuilder); _replicationWaiterList.setErrorAll_inlock( {ErrorCodes::ShutdownInProgress, "Replication is being shut down"}); _opTimeWaiterList.setErrorAll_inlock( @@ -1126,10 +1065,6 @@ void ReplicationCoordinatorImpl::shutdown(OperationContext* opCtx, // joining the replication executor is blocking so it must be run outside of the mutex if (initialSyncerCopy) { - auto scopedTimer = - createTimeElapsedBuilderScopedTimer(opCtx->getServiceContext()->getFastClockSource(), - "Shut down initial syncer", - shutdownTimeElapsedBuilder); LOGV2_DEBUG( 21329, 1, "ReplicationCoordinatorImpl::shutdown calling InitialSyncer::shutdown"); const auto status = initialSyncerCopy->shutdown(); @@ -1142,28 +1077,9 @@ void ReplicationCoordinatorImpl::shutdown(OperationContext* opCtx, initialSyncerCopy->join(); initialSyncerCopy.reset(); } - - { - auto scopedTimer = - createTimeElapsedBuilderScopedTimer(opCtx->getServiceContext()->getFastClockSource(), - "Shut down external state", - shutdownTimeElapsedBuilder); - _externalState->shutdown(opCtx); - } - { - auto scopedTimer = - createTimeElapsedBuilderScopedTimer(opCtx->getServiceContext()->getFastClockSource(), - "Shut down replication executor", - shutdownTimeElapsedBuilder); - _replExecutor->shutdown(); - } - { - auto scopedTimer = - createTimeElapsedBuilderScopedTimer(opCtx->getServiceContext()->getFastClockSource(), - "Join replication executor", - shutdownTimeElapsedBuilder); - _replExecutor->join(); - } + _externalState->shutdown(opCtx); + _replExecutor->shutdown(); + _replExecutor->join(); } const ReplSettings& ReplicationCoordinatorImpl::getSettings() const { @@ -1483,7 +1399,6 @@ void ReplicationCoordinatorImpl::setMyLastAppliedOpTimeAndWallTime( stdx::unique_lock<Latch> lock(_mutex); // The optime passed to this function is required to represent a consistent database state. _setMyLastAppliedOpTimeAndWallTime(lock, opTimeAndWallTime, false); - signalOplogWaiters(); _reportUpstream_inlock(std::move(lock)); } @@ -1557,6 +1472,9 @@ void ReplicationCoordinatorImpl::_setMyLastAppliedOpTimeAndWallTime( }, opTime); + // Notify the oplog waiters after updating the local snapshot. + signalOplogWaiters(); + if (opTime.isNull()) { return; } @@ -1940,7 +1858,7 @@ Status ReplicationCoordinatorImpl::_setLastOptime(WithLock lk, _wakeReadyWaiters(lk, std::max(args.appliedOpTime, args.durableOpTime)); } - _rescheduleLivenessUpdate_inlock(args.memberId); + _cancelAndRescheduleLivenessUpdate_inlock(args.memberId); return Status::OK(); } @@ -2391,10 +2309,6 @@ ReplicationCoordinatorImpl::_getHelloResponseFuture( prevCounter <= topologyVersionCounter); if (prevCounter < topologyVersionCounter) { - uassert(ErrorCodes::SplitHorizonChange, - "Stale horizon detected, we have since received a reconfig that changed the " - "horizon mappings.", - prevCounter >= _lastHorizonTopologyChange); // The received hello command contains a stale topology version so we respond // immediately with a more current topology version. return SharedSemiFuture<SharedHelloResponse>( @@ -2655,28 +2569,33 @@ ReplicationCoordinatorImpl::AutoGetRstlForStepUpStepDown::AutoGetRstlForStepUpSt deadline = start + Seconds(rstlTimeout); // cap deadline } - _rstlLock.emplace(_opCtx, MODE_X, ReplicationStateTransitionLockGuard::EnqueueOnly()); - - ON_BLOCK_EXIT([&] { _stopAndWaitForKillOpThread(); }); - _startKillOpThread(); - - // Wait for RSTL to be acquired. - _rstlLock->waitForLockUntil(deadline, [opCtx, rstlTimeout, start] { - if (rstlTimeout <= 0 || Date_t::now() - start < Seconds{rstlTimeout}) { - return; + try { + // Enqueues RSTL in X mode. + _rstlLock.emplace(_opCtx, MODE_X, ReplicationStateTransitionLockGuard::EnqueueOnly()); + + ON_BLOCK_EXIT([&] { _stopAndWaitForKillOpThread(); }); + _startKillOpThread(); + + // Wait for RSTL to be acquired. + _rstlLock->waitForLockUntil(deadline); + + } catch (const ExceptionFor<ErrorCodes::LockTimeout>&) { + if (rstlTimeout > 0 && Date_t::now() - start >= Seconds(rstlTimeout)) { + // Dump all locks to identify which thread(s) are holding RSTL. + getGlobalLockManager()->dump(); + + auto lockerInfo = + opCtx->lockState()->getLockerInfo(CurOp::get(opCtx)->getLockStatsBase()); + BSONObjBuilder lockRep; + lockerInfo->stats.report(&lockRep); + LOGV2_FATAL(5675600, + "Time out exceeded waiting for RSTL, stepUp/stepDown is not possible " + "thus calling abort() to allow cluster to progress.", + "lockRep"_attr = lockRep.obj()); } - - // Dump all locks to identify which thread(s) are holding RSTL. - getGlobalLockManager()->dump(); - - auto lockerInfo = opCtx->lockState()->getLockerInfo(CurOp::get(opCtx)->getLockStatsBase()); - BSONObjBuilder lockRep; - lockerInfo->stats.report(&lockRep); - LOGV2_FATAL(5675600, - "Time out exceeded waiting for RSTL, stepUp/stepDown is not possible thus " - "calling abort() to allow cluster to progress", - "lockRep"_attr = lockRep.obj()); - }); + // Rethrow to keep processing as before at a higher layer. + throw; + } }; void ReplicationCoordinatorImpl::AutoGetRstlForStepUpStepDown::_startKillOpThread() { @@ -3291,15 +3210,13 @@ Status ReplicationCoordinatorImpl::processReplSetGetStatus( lastStableRecoveryTimestamp = _storage->getLastStableRecoveryTimestamp(_service); } else { LOGV2_WARNING(6100702, - "Failed to get last stable recovery timestamp due to lock acquire " - "timeout. Note this is expected if shutdown is in progress."); + "Failed to get last stable recovery timestamp due to {error}", + "error"_attr = "lock acquire timeout"_sd); } } catch (const ExceptionForCat<ErrorCategory::Interruption>& ex) { - LOGV2_WARNING( - 6100703, - "Failed to get last stable recovery timestamp due to cancellation error. Note this is " - "expected if shutdown is in progress.", - "error"_attr = redact(ex)); + LOGV2_WARNING(6100703, + "Failed to get last stable recovery timestamp due to {error}", + "error"_attr = redact(ex)); } stdx::lock_guard<Latch> lk(_mutex); @@ -3862,7 +3779,6 @@ Status ReplicationCoordinatorImpl::_doReplSetReconfig(OperationContext* opCtx, const auto rwcDefaults = ReadWriteConcernDefaults::get(opCtx->getServiceContext()).getDefault(opCtx); const auto wcDefault = rwcDefaults.getDefaultWriteConcern(); - // Default WC can be 'boost::none' if the implicit default is used and set to 'w:1'. if (wcDefault) { auto validateWCStatus = newConfig.validateWriteConcern(wcDefault.get()); if (!validateWCStatus.isOK()) { @@ -4462,8 +4378,6 @@ void ReplicationCoordinatorImpl::_errorOnPromisesIfHorizonChanged(WithLock lk, promise->setError({ErrorCodes::SplitHorizonChange, "Received a reconfig that changed the horizon mappings."}); } - _topCoord->incrementTopologyVersion(); - _lastHorizonTopologyChange = _topCoord->getTopologyVersion().getCounter(); _sniToValidConfigPromiseMap.clear(); HelloMetrics::get(opCtx)->resetNumAwaitingTopologyChanges(); } @@ -4478,10 +4392,6 @@ void ReplicationCoordinatorImpl::_errorOnPromisesIfHorizonChanged(WithLock lk, promise->setError({ErrorCodes::SplitHorizonChange, "Received a reconfig that changed the horizon mappings."}); } - // Increment topology version to mark a horizon change, since a reconfig doesn't - // increment the topology version until the end. - _topCoord->incrementTopologyVersion(); - _lastHorizonTopologyChange = _topCoord->getTopologyVersion().getCounter(); _createHorizonTopologyChangePromiseMapping(lk); HelloMetrics::get(opCtx)->resetNumAwaitingTopologyChanges(); } @@ -5737,8 +5647,6 @@ Status ReplicationCoordinatorImpl::processReplSetRequestVotes( LastVote lastVote{args.getTerm(), args.getCandidateIndex()}; Status status = _externalState->storeLocalLastVoteDocument(opCtx, lastVote); if (!status.isOK()) { - // Note the topology coordinator has already advanced its last vote at this point, - // so this node will not be able to vote in this election; this is a "spoiled" vote. LOGV2_ERROR(21428, "replSetRequestVotes failed to store LastVote document", "error"_attr = status); diff --git a/src/mongo/db/repl/replication_coordinator_impl.h b/src/mongo/db/repl/replication_coordinator_impl.h index 6bf3a850ce5..fafe33221b9 100644 --- a/src/mongo/db/repl/replication_coordinator_impl.h +++ b/src/mongo/db/repl/replication_coordinator_impl.h @@ -38,7 +38,6 @@ #include "mongo/bson/timestamp.h" #include "mongo/db/concurrency/d_concurrency.h" #include "mongo/db/concurrency/replication_state_transition_lock_guard.h" -#include "mongo/db/repl/delayable_timeout_callback.h" #include "mongo/db/repl/initial_syncer.h" #include "mongo/db/repl/initial_syncer_interface.h" #include "mongo/db/repl/member_state.h" @@ -113,8 +112,7 @@ public: virtual bool inQuiesceMode() const override; - virtual void shutdown(OperationContext* opCtx, - BSONObjBuilder* shutdownTimeElapsedBuilder) override; + virtual void shutdown(OperationContext* opCtx) override; virtual const ReplSettings& getSettings() const override; @@ -468,11 +466,6 @@ public: executor::TaskExecutor::CallbackHandle getCatchupTakeoverCbh_forTest() const; /** - * Returns the cached horizon topology version from most recent SplitHorizonChange. - */ - int64_t getLastHorizonChange_forTest() const; - - /** * Simple wrappers around _setLastOptime to make it easier to test. */ Status setLastAppliedOptime_forTest(long long cfgVer, @@ -769,9 +762,6 @@ private: class WaiterList { public: - WaiterList() = delete; - WaiterList(Atomic64Metric& waiterCountMetric); - // Adds waiter into the list. void add_inlock(const OpTime& opTime, SharedWaiterHandle waiter); // Adds a waiter into the list and returns the future of the waiter's promise. @@ -789,13 +779,8 @@ private: void setErrorAll_inlock(Status status); private: - void _updateMetric_inlock(); - // Waiters sorted by OpTime. std::multimap<OpTime, SharedWaiterHandle> _waiters; - // We keep a separate count outside _waiters.size() in order to avoid having to - // take a lock to read the metric. - Atomic64Metric& _waiterCountMetric; }; enum class HeartbeatState { kScheduled = 0, kSent = 1 }; @@ -1161,13 +1146,6 @@ private: */ Milliseconds _getRandomizedElectionOffset_inlock(); - /* - * Return the upper bound of the offset amount returned by _getRandomizedElectionOffset - * This is actually off by one, that is, the election offset is in the half-open range - * [0, electionOffsetUpperBound) - */ - long long _getElectionOffsetUpperBound_inlock(); - /** * Starts a heartbeat for each member in the current config. Called while holding _mutex. */ @@ -1508,10 +1486,8 @@ private: /** * Bottom half of _scheduleNextLivenessUpdate. * Must be called with _mutex held. - * If reschedule is true, will recompute the liveness update even if a timeout is - * already pending. */ - void _scheduleNextLivenessUpdate_inlock(bool reschedule); + void _scheduleNextLivenessUpdate_inlock(); /** * Callback which marks downed nodes as down, triggers a stepdown if a majority of nodes are no @@ -1520,11 +1496,11 @@ private: void _handleLivenessTimeout(const executor::TaskExecutor::CallbackArgs& cbData); /** - * If "updatedMemberId" is the current _earliestMemberId, calls _scheduleNextLivenessUpdate to - * schedule a new one. + * If "updatedMemberId" is the current _earliestMemberId, cancels the current + * _handleLivenessTimeout callback and calls _scheduleNextLivenessUpdate to schedule a new one. * Returns immediately otherwise. */ - void _rescheduleLivenessUpdate_inlock(int updatedMemberId); + void _cancelAndRescheduleLivenessUpdate_inlock(int updatedMemberId); /** * Cancels all outstanding _priorityTakeover callbacks. @@ -1776,10 +1752,15 @@ private: stdx::condition_variable _currentCommittedSnapshotCond; // (M) // Callback Handle used to cancel a scheduled LivenessTimeout callback. - DelayableTimeoutCallback _handleLivenessTimeoutCallback; // (S) + executor::TaskExecutor::CallbackHandle _handleLivenessTimeoutCbh; // (M) + + // Callback Handle used to cancel a scheduled ElectionTimeout callback. + executor::TaskExecutor::CallbackHandle _handleElectionTimeoutCbh; // (M) - // Used to manage scheduling and canceling election timeouts. - DelayableTimeoutCallbackWithJitter _handleElectionTimeoutCallback; // (M) + // Election timeout callback will not run before this time. + // If this date is Date_t(), the callback is either unscheduled or canceled. + // Used for testing only. + Date_t _handleElectionTimeoutWhen; // (M) // Callback Handle used to cancel a scheduled PriorityTakeover callback. executor::TaskExecutor::CallbackHandle _priorityTakeoverCbh; // (M) @@ -1849,9 +1830,6 @@ private: // The cached value of the 'counter' field in the server's TopologyVersion. AtomicWord<int64_t> _cachedTopologyVersionCounter; // (S) - // The cached value of the topology from the most recent SplitHorizonChange. - int64_t _lastHorizonTopologyChange{-1}; // (M) - // This should be set during sharding initialization. boost::optional<bool> _wasCWWCSetOnConfigServerOnStartup; diff --git a/src/mongo/db/repl/replication_coordinator_impl_elect_v1.cpp b/src/mongo/db/repl/replication_coordinator_impl_elect_v1.cpp index e87cf7f7837..0c05b9f4a0c 100644 --- a/src/mongo/db/repl/replication_coordinator_impl_elect_v1.cpp +++ b/src/mongo/db/repl/replication_coordinator_impl_elect_v1.cpp @@ -46,8 +46,6 @@ namespace mongo { namespace repl { MONGO_FAIL_POINT_DEFINE(hangInWritingLastVoteForDryRun); -MONGO_FAIL_POINT_DEFINE(electionHangsBeforeUpdateMemberState); -MONGO_FAIL_POINT_DEFINE(hangBeforeOnVoteRequestCompleteCallback); class ReplicationCoordinatorImpl::ElectionState::LoseElectionGuardV1 { LoseElectionGuardV1(const LoseElectionGuardV1&) = delete; @@ -136,12 +134,7 @@ ReplicationCoordinatorImpl::ElectionState::getElectionDryRunFinishedEvent(WithLo void ReplicationCoordinatorImpl::ElectionState::cancel(WithLock) { _isCanceled = true; - // This check is necessary because _voteRequester is only initialized in _startVoteRequester. - // Since we don't hold mutex during the entire election process, it is possible to get here - // before _startVoteRequester is ever called. - if (_voteRequester) { - _voteRequester->cancel(); - } + _voteRequester->cancel(); } void ReplicationCoordinatorImpl::ElectionState::start(WithLock lk, StartElectionReasonEnum reason) { @@ -397,16 +390,13 @@ void ReplicationCoordinatorImpl::ElectionState::_requestVotesForRealElection( _replExecutor ->onEvent(nextPhaseEvh.getValue(), [=](const executor::TaskExecutor::CallbackArgs&) { - if (MONGO_unlikely(hangBeforeOnVoteRequestCompleteCallback.shouldFail())) { - LOGV2(7277400, - "Hang due to hangBeforeOnVoteRequestCompleteCallback failpoint"); - hangBeforeOnVoteRequestCompleteCallback.pauseWhileSet(); - } _onVoteRequestComplete(newTerm, reason); }) .status_with_transitional_ignore(); } +MONGO_FAIL_POINT_DEFINE(electionHangsBeforeUpdateMemberState); + void ReplicationCoordinatorImpl::ElectionState::_onVoteRequestComplete( long long newTerm, StartElectionReasonEnum reason) { stdx::lock_guard<Latch> lk(_repl->_mutex); diff --git a/src/mongo/db/repl/replication_coordinator_impl_elect_v1_test.cpp b/src/mongo/db/repl/replication_coordinator_impl_elect_v1_test.cpp index d71d606c656..84b45c40fa5 100644 --- a/src/mongo/db/repl/replication_coordinator_impl_elect_v1_test.cpp +++ b/src/mongo/db/repl/replication_coordinator_impl_elect_v1_test.cpp @@ -227,8 +227,7 @@ TEST_F(ReplCoordTest, ElectionSucceedsWhenNodeIsTheOnlyElectableNode) { const auto opCtxPtr = makeOperationContext(); auto& opCtx = *opCtxPtr; - // Since we're still in drain mode, expect that we report isWritablePrimary:false, - // issecondary:true. + // Since we're still in drain mode, expect that we report ismaster: false, issecondary:true. auto helloResponse = getReplCoord()->awaitHelloResponse(opCtxPtr.get(), {}, boost::none, boost::none); ASSERT_FALSE(helloResponse->isWritablePrimary()) << helloResponse->toBSON().toString(); @@ -285,8 +284,7 @@ TEST_F(ReplCoordTest, ElectionSucceedsWhenNodeIsTheOnlyNode) { const auto opCtxPtr = makeOperationContext(); auto& opCtx = *opCtxPtr; - // Since we're still in drain mode, expect that we report isWritablePrimary:false, - // issecondary:true. + // Since we're still in drain mode, expect that we report ismaster: false, issecondary:true. auto helloResponse = getReplCoord()->awaitHelloResponse(opCtxPtr.get(), {}, boost::none, boost::none); ASSERT_FALSE(helloResponse->isWritablePrimary()) << helloResponse->toBSON().toString(); diff --git a/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp b/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp index ce5c77a1e7d..060f4d6fc9b 100644 --- a/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp +++ b/src/mongo/db/repl/replication_coordinator_impl_heartbeat.cpp @@ -77,19 +77,16 @@ MONGO_FAIL_POINT_DEFINE(blockHeartbeatStepdown); MONGO_FAIL_POINT_DEFINE(blockHeartbeatReconfigFinish); MONGO_FAIL_POINT_DEFINE(hangAfterTrackingNewHandleInHandleHeartbeatResponseForTest); MONGO_FAIL_POINT_DEFINE(waitForPostActionCompleteInHbReconfig); -MONGO_FAIL_POINT_DEFINE(pauseInHandleHeartbeatResponse); } // namespace using executor::RemoteCommandRequest; -long long ReplicationCoordinatorImpl::_getElectionOffsetUpperBound_inlock() { +Milliseconds ReplicationCoordinatorImpl::_getRandomizedElectionOffset_inlock() { long long electionTimeout = durationCount<Milliseconds>(_rsConfig.getElectionTimeoutPeriod()); - return electionTimeout * _externalState->getElectionTimeoutOffsetLimitFraction(); -} + long long randomOffsetUpperBound = + electionTimeout * _externalState->getElectionTimeoutOffsetLimitFraction(); -Milliseconds ReplicationCoordinatorImpl::_getRandomizedElectionOffset_inlock() { - long long randomOffsetUpperBound = _getElectionOffsetUpperBound_inlock(); // Avoid divide by zero error in random number generator. if (randomOffsetUpperBound == 0) { return Milliseconds(0); @@ -185,12 +182,6 @@ void ReplicationCoordinatorImpl::handleHeartbeatResponse_forTest(BSONObj respons void ReplicationCoordinatorImpl::_handleHeartbeatResponse( const executor::TaskExecutor::RemoteCommandCallbackArgs& cbData, const std::string& setName) { - pauseInHandleHeartbeatResponse.executeIf( - [](const BSONObj& data) { pauseInHandleHeartbeatResponse.pauseWhileSet(); }, - [&cbData](const BSONObj& data) -> bool { - StringData dtarget = data["target"].valueStringDataSafe(); - return dtarget == cbData.request.target.toString(); - }); stdx::unique_lock<Latch> lk(_mutex); // remove handle from queued heartbeats @@ -201,15 +192,7 @@ void ReplicationCoordinatorImpl::_handleHeartbeatResponse( Status responseStatus = cbData.response.status; const HostAndPort& target = cbData.request.target; - // It is possible that the callback was canceled after handleHeartbeatResponse was called but - // before it got the lock above. - // - // In this case, the responseStatus will be OK and we can process the heartbeat. However, if - // we do so, cancelling heartbeats no longer establishes a barrier after which all heartbeats - // processed are "new" (sent subsequent to the cancel), which is something we care about for - // catchup takeover. So if we detect this situation (by checking if the handle was canceled) - // we will NOT process the 'stale' heartbeat. - if (responseStatus == ErrorCodes::CallbackCanceled || cbData.myHandle.isCanceled()) { + if (responseStatus == ErrorCodes::CallbackCanceled) { LOGV2_FOR_HEARTBEATS(4615619, 2, "Received response to heartbeat (requestId: {requestId}) from " @@ -1061,7 +1044,9 @@ void ReplicationCoordinatorImpl::_cancelHeartbeats_inlock() { // Heartbeat callbacks will remove themselves from _heartbeatHandles when they execute with // CallbackCanceled status, so it's better to leave the handles in the list, for now. - _handleLivenessTimeoutCallback.cancel(); + if (_handleLivenessTimeoutCbh.isValid()) { + _replExecutor->cancel(_handleLivenessTimeoutCbh); + } } void ReplicationCoordinatorImpl::restartScheduledHeartbeats_forTest() { @@ -1110,12 +1095,16 @@ void ReplicationCoordinatorImpl::_startHeartbeats_inlock() { _topCoord->restartHeartbeat(now, target); } - _scheduleNextLivenessUpdate_inlock(/* reschedule = */ false); + _scheduleNextLivenessUpdate_inlock(); } void ReplicationCoordinatorImpl::_handleLivenessTimeout( const executor::TaskExecutor::CallbackArgs& cbData) { stdx::unique_lock<Latch> lk(_mutex); + // Only reset the callback handle if it matches, otherwise more will be coming through + if (cbData.myHandle == _handleLivenessTimeoutCbh) { + _handleLivenessTimeoutCbh = CallbackHandle(); + } if (!cbData.status.isOK()) { return; } @@ -1127,10 +1116,10 @@ void ReplicationCoordinatorImpl::_handleLivenessTimeout( lk = _handleHeartbeatResponseAction_inlock( action, StatusWith(ReplSetHeartbeatResponse()), std::move(lk)); - _scheduleNextLivenessUpdate_inlock(/* reschedule = */ false); + _scheduleNextLivenessUpdate_inlock(); } -void ReplicationCoordinatorImpl::_scheduleNextLivenessUpdate_inlock(bool reschedule) { +void ReplicationCoordinatorImpl::_scheduleNextLivenessUpdate_inlock() { // Scan liveness table for earliest date; schedule a run at (that date plus election // timeout). Date_t earliestDate; @@ -1143,7 +1132,7 @@ void ReplicationCoordinatorImpl::_scheduleNextLivenessUpdate_inlock(bool resched return; } - if (!reschedule && _handleLivenessTimeoutCallback.isActive()) { + if (_handleLivenessTimeoutCbh.isValid() && !_handleLivenessTimeoutCbh.isCanceled()) { // don't bother to schedule; one is already scheduled and pending. return; } @@ -1156,21 +1145,31 @@ void ReplicationCoordinatorImpl::_scheduleNextLivenessUpdate_inlock(bool resched "nextTimeout"_attr = nextTimeout); // It is possible we will schedule the next timeout in the past. - // DelayableTimeoutCallback schedules its work immediately if it's given a time <= now(). + // ThreadPoolTaskExecutor::_scheduleWorkAt() schedules its work immediately if it's given a + // time <= now(). // If we missed the timeout, it means that on our last check the earliest live member was // just barely fresh and it has become stale since then. We must schedule another liveness // check to continue conducting liveness checks and be able to step down from primary if we // lose contact with a majority of nodes. - // We ignore shutdown errors; any other error triggers an fassert. - _handleLivenessTimeoutCallback.delayUntil(nextTimeout).ignore(); + auto cbh = + _scheduleWorkAt(nextTimeout, [=](const executor::TaskExecutor::CallbackArgs& cbData) { + _handleLivenessTimeout(cbData); + }); + if (!cbh) { + return; + } + _handleLivenessTimeoutCbh = cbh; _earliestMemberId = earliestMemberId.getData(); } -void ReplicationCoordinatorImpl::_rescheduleLivenessUpdate_inlock(int updatedMemberId) { +void ReplicationCoordinatorImpl::_cancelAndRescheduleLivenessUpdate_inlock(int updatedMemberId) { if ((_earliestMemberId != -1) && (_earliestMemberId != updatedMemberId)) { return; } - _scheduleNextLivenessUpdate_inlock(/* reschedule = */ true); + if (_handleLivenessTimeoutCbh.isValid()) { + _replExecutor->cancel(_handleLivenessTimeoutCbh); + } + _scheduleNextLivenessUpdate_inlock(); } void ReplicationCoordinatorImpl::_cancelPriorityTakeover_inlock() { @@ -1204,8 +1203,7 @@ void ReplicationCoordinatorImpl::_cancelAndRescheduleElectionTimeout_inlock() { // the logs. int cancelAndRescheduleLogLevel = 5; static auto logThrottleTime = _replExecutor->now(); - auto oldWhen = _handleElectionTimeoutCallback.getNextCall(); - const bool wasActive = oldWhen != Date_t(); + const bool wasActive = _handleElectionTimeoutCbh.isValid(); auto now = _replExecutor->now(); const bool doNotReschedule = _inShutdown || !_memberState.secondary() || _selfIndex < 0 || !_rsConfig.getMemberAt(_selfIndex).isElectable(); @@ -1214,40 +1212,48 @@ void ReplicationCoordinatorImpl::_cancelAndRescheduleElectionTimeout_inlock() { cancelAndRescheduleLogLevel = 4; logThrottleTime = now; } - if (wasActive && doNotReschedule) { + if (wasActive) { LOGV2_FOR_ELECTION(4615649, cancelAndRescheduleLogLevel, "Canceling election timeout callback at {when}", "Canceling election timeout callback", - "when"_attr = oldWhen); - _handleElectionTimeoutCallback.cancel(); + "when"_attr = _handleElectionTimeoutWhen); + _replExecutor->cancel(_handleElectionTimeoutCbh); + _handleElectionTimeoutCbh = CallbackHandle(); + _handleElectionTimeoutWhen = Date_t(); } if (doNotReschedule) return; - Milliseconds upperBound = Milliseconds(_getElectionOffsetUpperBound_inlock()); - auto requestedWhen = now + _rsConfig.getElectionTimeoutPeriod(); - invariant(requestedWhen > now); - Status delayStatus = - _handleElectionTimeoutCallback.delayUntilWithJitter(requestedWhen, upperBound); - Date_t when = _handleElectionTimeoutCallback.getNextCall(); + Milliseconds randomOffset = _getRandomizedElectionOffset_inlock(); + auto when = now + _rsConfig.getElectionTimeoutPeriod() + randomOffset; + invariant(when > now); if (wasActive) { // The log level here is 4 once per second, otherwise 5. LOGV2_FOR_ELECTION(4615650, cancelAndRescheduleLogLevel, - "Rescheduled election timeout callback", - "when"_attr = when, - "requestedWhen"_attr = requestedWhen, - "error"_attr = delayStatus); + "Rescheduling election timeout callback at {when}", + "Rescheduling election timeout callback", + "when"_attr = when); } else { LOGV2_FOR_ELECTION(4615651, 4, - "Scheduled election timeout callback", - "when"_attr = when, - "requestedWhen"_attr = requestedWhen, - "error"_attr = delayStatus); + "Scheduling election timeout callback at {when}", + "Scheduling election timeout callback", + "when"_attr = when); } + _handleElectionTimeoutWhen = when; + _handleElectionTimeoutCbh = + _scheduleWorkAt(when, [=](const mongo::executor::TaskExecutor::CallbackArgs& cbData) { + stdx::lock_guard<Latch> lk(_mutex); + if (_handleElectionTimeoutCbh == cbData.myHandle) { + // This lets _cancelAndRescheduleElectionTimeout_inlock know the callback + // has happened. + _handleElectionTimeoutCbh = CallbackHandle(); + } + _startElectSelfIfEligibleV1(lk, StartElectionReasonEnum::kElectionTimeout); + }); } void ReplicationCoordinatorImpl::_startElectSelfIfEligibleV1(StartElectionReasonEnum reason) { @@ -1271,7 +1277,7 @@ void ReplicationCoordinatorImpl::_startElectSelfIfEligibleV1(WithLock lk, _cancelCatchupTakeover_inlock(); _cancelPriorityTakeover_inlock(); _cancelAndRescheduleElectionTimeout_inlock(); - if (_inShutdown || _inQuiesceMode) { + if (_inShutdown) { LOGV2_FOR_ELECTION(4615654, 0, "Not starting an election, since we are shutting down"); return; } diff --git a/src/mongo/db/repl/replication_coordinator_impl_test.cpp b/src/mongo/db/repl/replication_coordinator_impl_test.cpp index f0ece048ca3..2a106fca9c0 100644 --- a/src/mongo/db/repl/replication_coordinator_impl_test.cpp +++ b/src/mongo/db/repl/replication_coordinator_impl_test.cpp @@ -85,9 +85,6 @@ namespace mongo { namespace repl { -extern Atomic64Metric replicationWaiterListMetric; -extern Atomic64Metric opTimeWaiterListMetric; - namespace { using executor::NetworkInterfaceMock; @@ -139,7 +136,7 @@ std::shared_ptr<const repl::HelloResponse> awaitHelloWithNewOpCtx( return replCoord->awaitHelloResponse(newOpCtx.get(), horizonParams, topologyVersion, deadline); } -TEST_F(ReplCoordTest, IsWritablePrimaryFalseDuringStepdown) { +TEST_F(ReplCoordTest, IsMasterIsFalseDuringStepdown) { BSONObj configObj = BSON("_id" << "mySet" << "version" << 1 << "members" @@ -164,13 +161,13 @@ TEST_F(ReplCoordTest, IsWritablePrimaryFalseDuringStepdown) { replCoord->updateTerm_forTest(replCoord->getTerm() + 1, &updateTermResult); ASSERT(TopologyCoordinator::UpdateTermResult::kTriggerStepDown == updateTermResult); - // Test that "isWritablePrimary" is immediately false, although "secondary" is not yet true. + // Test that "ismaster" is immediately false, although "secondary" is not yet true. auto opCtx = makeOperationContext(); const auto response = getReplCoord()->awaitHelloResponse(opCtx.get(), {}, boost::none, boost::none); ASSERT_TRUE(response->isConfigSet()); - BSONObj responseObj = response->toBSON(false /*useLegacyResponseFields*/); - ASSERT_FALSE(responseObj["isWritablePrimary"].Bool()); + BSONObj responseObj = response->toBSON(); + ASSERT_FALSE(responseObj["ismaster"].Bool()); ASSERT_FALSE(responseObj["secondary"].Bool()); ASSERT_FALSE(responseObj.hasField("isreplicaset")); @@ -3520,10 +3517,10 @@ TEST_F(ReplCoordTest, AwaitHelloResponseReturnsOnStepDown) { responseAfterDisablingWrites->getTopologyVersion(); ASSERT_EQUALS(topologyVersionAfterDisablingWrites->getCounter(), expectedCounter); ASSERT_EQUALS(topologyVersionAfterDisablingWrites->getProcessId(), expectedProcessId); - // We expect the server to increment the TopologyVersion and respond to waiting hellos once - // we disable writes on the node that is stepping down from primary. At this time, the - // 'isWritablePrimary' response field will be false but the node will have yet to transition - // to secondary. + // We expect the server to increment the TopologyVersion and respond to waiting hellos + // once we disable writes on the node that is stepping down from primary. At this time, + // the 'ismaster' response field will be false but the node will have yet to transition to + // secondary. ASSERT_FALSE(responseAfterDisablingWrites->isWritablePrimary()); ASSERT_FALSE(responseAfterDisablingWrites->isSecondary()); ASSERT_EQUALS(responseAfterDisablingWrites->getPrimary().host(), "node1"); @@ -4077,131 +4074,6 @@ TEST_F(ReplCoordTest, AwaitHelloResponseReturnsErrorOnHorizonChange) { getHelloThread.join(); } -TEST_F(ReplCoordTest, ServerUassertAfterStaleHorizonTopology) { - init(); - assertStartSuccess(BSON("_id" - << "mySet" - << "version" << 2 << "members" - << BSON_ARRAY(BSON("host" - << "node1:12345" - << "_id" << 0) - << BSON("host" - << "node2:12345" - << "_id" << 1))), - HostAndPort("node1", 12345)); - - // Become primary. - ASSERT_OK(getReplCoord()->setFollowerMode(MemberState::RS_SECONDARY)); - replCoordSetMyLastAppliedAndDurableOpTime(OpTimeWithTermOne(100, 1), Date_t() + Seconds(100)); - simulateSuccessfulV1Election(); - ASSERT(getReplCoord()->getMemberState().primary()); - - auto maxAwaitTime = Milliseconds(5000); - auto deadline = getNet()->now() + maxAwaitTime; - auto opCtx = makeOperationContext(); - - auto topologyVersionBeforeReconfig = getTopoCoord().getTopologyVersion(); - // awaitHelloResponse blocks and waits on a future when the request TopologyVersion equals - // the current TopologyVersion of the server. - stdx::thread getHelloThread([&] { - ASSERT_THROWS_CODE( - awaitHelloWithNewOpCtx(getReplCoord(), topologyVersionBeforeReconfig, {}, deadline), - AssertionException, - ErrorCodes::SplitHorizonChange); - }); - - auto lastHorizonBeforeReconfig = getReplCoord()->getLastHorizonChange_forTest(); - ASSERT_EQUALS(lastHorizonBeforeReconfig, -1); - - BSONObjBuilder garbage; - ReplSetReconfigArgs args; - // Use force to bypass the oplog commitment check, which we're not worried about testing here. - args.force = true; - // Do a reconfig that changes the SplitHorizon and also adds a third node. This should respond - // to all waiting hello requests with an error. - args.newConfigObj = BSON("_id" - << "mySet" - << "version" << 3 << "protocolVersion" << 1 << "members" - << BSON_ARRAY(BSON("_id" << 0 << "host" - << "node1:12345" - << "priority" << 3 << "horizons" - << BSON("testhorizon" - << "test.monkey.example.com:24")) - << BSON("_id" << 1 << "host" - << "node2:12345" - << "horizons" - << BSON("testhorizon" - << "test.giraffe.example.com:25")) - << BSON("_id" - << 2 << "host" - << "node3:12345" - << "horizons" - << BSON("testhorizon" - << "test.elephant.example.com:26")))); - stdx::thread reconfigThread([&] { - Status status(ErrorCodes::InternalError, "Not Set"); - status = getReplCoord()->processReplSetReconfig(opCtx.get(), args, &garbage); - ASSERT_OK(status); - }); - replyToReceivedHeartbeatV1(); - reconfigThread.join(); - getHelloThread.join(); - - // After reconfig, the last horizon change topology counter should come out greater than the - // input topology counter , but less than the output topology counter. - ASSERT_GREATER_THAN(getReplCoord()->getLastHorizonChange_forTest(), - topologyVersionBeforeReconfig.getCounter()); - ASSERT_GREATER_THAN(getTopoCoord().getTopologyVersion().getCounter(), - getReplCoord()->getLastHorizonChange_forTest()); - ASSERT_GREATER_THAN(getReplCoord()->getLastHorizonChange_forTest(), lastHorizonBeforeReconfig); - - // Send hello with a TopologyVersion older than the TopologyVersion of the last horizon change. - auto requestTopologyVersion = - TopologyVersion(getTopoCoord().getTopologyVersion().getProcessId(), - getReplCoord()->getLastHorizonChange_forTest() - 1); - - // AwaitHelloResponse should throw uassert with SplitHorizonChange if topology version - // corresponds to a stale horizon. - ASSERT_THROWS_CODE(awaitHelloWithNewOpCtx(getReplCoord(), requestTopologyVersion, {}, deadline), - DBException, - ErrorCodes::SplitHorizonChange); - - // Send hello with a TopologyVersion version equal to the TopologyVersion of the last horizon - // change. - auto expectedTopologyVersion = getTopoCoord().getTopologyVersion(); - requestTopologyVersion = TopologyVersion(expectedTopologyVersion.getProcessId(), - getReplCoord()->getLastHorizonChange_forTest()); - - ASSERT_GREATER_THAN(expectedTopologyVersion.getCounter(), requestTopologyVersion.getCounter()); - // AwaitHelloResponse should return with a helloResponse that matches expectedTopologyVersion. - // Since expectedTopologyVersion > requestTopologyVersion, the call is non-blocking and will - // return immediately. - auto response = awaitHelloWithNewOpCtx(getReplCoord(), requestTopologyVersion, {}, deadline); - auto responseTopologyVersion = response->getTopologyVersion(); - ASSERT_EQUALS(responseTopologyVersion->getCounter(), expectedTopologyVersion.getCounter()); - ASSERT_EQUALS(responseTopologyVersion->getProcessId(), expectedTopologyVersion.getProcessId()); - - // Setup instance where lastHorizonChange topology counter < request topology counter < server - // topology counter. For server topology counter to be greater than both request's and horizon, - // we must increment server topology. - getTopoCoord().incrementTopologyVersion(); - expectedTopologyVersion = getTopoCoord().getTopologyVersion(); - // Send hello with a TopologyVersion version greater than the TopologyVersion of the last - // horizon change. - requestTopologyVersion = TopologyVersion(getTopoCoord().getTopologyVersion().getProcessId(), - getReplCoord()->getLastHorizonChange_forTest() + 1); - - ASSERT_GREATER_THAN(expectedTopologyVersion.getCounter(), requestTopologyVersion.getCounter()); - ASSERT_GREATER_THAN(requestTopologyVersion.getCounter(), - getReplCoord()->getLastHorizonChange_forTest()); - - // AwaitHelloResponse should return with a helloResponse that matches expectedTopologyVersion. - response = awaitHelloWithNewOpCtx(getReplCoord(), requestTopologyVersion, {}, deadline); - responseTopologyVersion = response->getTopologyVersion(); - ASSERT_EQUALS(responseTopologyVersion->getCounter(), expectedTopologyVersion.getCounter()); - ASSERT_EQUALS(responseTopologyVersion->getProcessId(), expectedTopologyVersion.getProcessId()); -} - TEST_F(ReplCoordTest, NonAwaitableHelloReturnsNoConfigsOnNodeWithUninitializedConfig) { start(); auto opCtx = makeOperationContext(); @@ -4847,9 +4719,6 @@ TEST_F(ReplCoordTest, AwaitHelloRespondsCorrectlyWhenNodeRemovedAndReadded) { }); waitForHelloFailPoint->waitForTimesEntered(timesEnteredFailPoint + 2); - auto lastHorizonBeforeReconfig = getReplCoord()->getLastHorizonChange_forTest(); - ASSERT_EQUALS(lastHorizonBeforeReconfig, -1); - const auto newHorizonNodeOne = "newhorizon.com:100"; const auto newHorizonNodeTwo = "newhorizon.com:200"; @@ -4874,22 +4743,10 @@ TEST_F(ReplCoordTest, AwaitHelloRespondsCorrectlyWhenNodeRemovedAndReadded) { }); replyToReceivedHeartbeatV1(); reconfigThread.join(); - ASSERT_OK( getReplCoord()->waitForMemberState(opCtx.get(), MemberState::RS_SECONDARY, Seconds(1))); getHelloThread.join(); - ASSERT_GREATER_THAN(getReplCoord()->getLastHorizonChange_forTest(), lastHorizonBeforeReconfig); - // Send hello with a TopologyVersion older than the TopologyVersion of the last horizon change. - auto requestTopologyVersion = - TopologyVersion(getTopoCoord().getTopologyVersion().getProcessId(), - getReplCoord()->getLastHorizonChange_forTest() - 1); - // AwaitHelloResponse should throw uassert with SplitHorizonChange if topology version - // corresponds to a stale horizon. - ASSERT_THROWS_CODE(awaitHelloWithNewOpCtx(getReplCoord(), requestTopologyVersion, {}, deadline), - DBException, - ErrorCodes::SplitHorizonChange); - stdx::thread getHelloThreadNewHorizon([&] { const auto expectedTopologyVersion = getTopoCoord().getTopologyVersion(); // Sending a hello on the rejoined node should return the appropriate horizon view. @@ -5179,8 +5036,8 @@ TEST_F(ReplCoordTest, HelloResponseMentionsLackOfReplicaSetConfig) { const auto response = getReplCoord()->awaitHelloResponse(opCtx.get(), {}, boost::none, boost::none); ASSERT_FALSE(response->isConfigSet()); - BSONObj responseObj = response->toBSON(false /*useLegacyResponseFields*/); - ASSERT_FALSE(responseObj["isWritablePrimary"].Bool()); + BSONObj responseObj = response->toBSON(); + ASSERT_FALSE(responseObj["ismaster"].Bool()); ASSERT_FALSE(responseObj["secondary"].Bool()); ASSERT_TRUE(responseObj["isreplicaset"].Bool()); ASSERT_EQUALS("Does not have a valid replica set config", responseObj["info"].String()); @@ -5320,7 +5177,7 @@ TEST_F(ReplCoordTest, LogAMessageWhenShutDownBeforeReplicationStartUpFinished) { startCapturingLogMessages(); { auto opCtx = makeOperationContext(); - getReplCoord()->shutdown(opCtx.get(), nullptr /* shutdownTimeElapsedBuilder */); + getReplCoord()->shutdown(opCtx.get()); } stopCapturingLogMessages(); ASSERT_EQUALS(1, @@ -5901,140 +5758,6 @@ TEST_F(ReplCoordTest, awaiter.reset(); } - -// We need to wait for replication to start waiting before the waiter metric increases. We -// return the metric value from the function for the convenience of the assert macro below, -// which allows us to get a nice assert message without repeating the value. -template <typename T, typename U> -U expectMetricIncreaseTo(T& metric, U value) { - // If this doesn't go in 10 seconds, something's seriously wrong; even if just a slow machine, - // the test will likely fail anyway. - constexpr auto timeout = Seconds(10); - const auto deadline = Date_t::now() + timeout; - U lastValue = metric.get(); - U curValue = metric.get(); - while (curValue < value && Date_t::now() <= deadline) { - sleepFor(Milliseconds{10}); - curValue = metric.get(); - lastValue = curValue; - } - return curValue; -} - -#define ASSERT_METRIC_INCREASE_TO(metric, value) \ - ASSERT_EQ(expectMetricIncreaseTo(metric, value), value) - -TEST_F(ReplCoordTest, ReplicationWaiterMetrics) { - assertStartSuccess(BSON("_id" - << "mySet" - << "version" << 2 << "members" - << BSON_ARRAY(BSON("host" - << "node1:12345" - << "_id" << 0) - << BSON("host" - << "node2:12345" - << "_id" << 1) - << BSON("host" - << "node3:12345" - << "_id" << 2))), - HostAndPort("node1", 12345)); - ASSERT_OK(getReplCoord()->setFollowerMode(MemberState::RS_SECONDARY)); - replCoordSetMyLastAppliedOpTime(OpTimeWithTermOne(100, 1), Date_t() + Seconds(100)); - replCoordSetMyLastDurableOpTime(OpTimeWithTermOne(100, 1), Date_t() + Seconds(100)); - simulateSuccessfulV1Election(); - - ReplicationAwaiter awaiter1(getReplCoord(), getServiceContext()); - ReplicationAwaiter awaiter2(getReplCoord(), getServiceContext()); - - OpTimeWithTermOne time1(100, 1); - OpTimeWithTermOne time2(100, 2); - OpTimeWithTermOne time3(100, 3); - - WriteConcernOptions writeConcern; - writeConcern.wTimeout = WriteConcernOptions::kNoTimeout; - writeConcern.w = 2; - - WriteConcernOptions writeConcernLocal; - writeConcernLocal.wTimeout = WriteConcernOptions::kNoTimeout; - writeConcernLocal.w = 1; - writeConcernLocal.syncMode = WriteConcernOptions::SyncMode::UNSET; - - // 2 waiters waiting for 2 nodes to reach time1. - awaiter1.setOpTime(time1); - awaiter1.setWriteConcern(writeConcern); - awaiter1.start(); - ASSERT_METRIC_INCREASE_TO(replicationWaiterListMetric, 1); - ASSERT_EQ(opTimeWaiterListMetric.get(), 0); - - awaiter2.setOpTime(time1); - awaiter2.setWriteConcern(writeConcern); - awaiter2.start(); - ASSERT_METRIC_INCREASE_TO(replicationWaiterListMetric, 2); - ASSERT_EQ(opTimeWaiterListMetric.get(), 0); - - replCoordSetMyLastAppliedOpTime(time1, Date_t() + Seconds(100)); - replCoordSetMyLastDurableOpTime(time1, Date_t() + Seconds(100)); - ASSERT_OK(getReplCoord()->setLastAppliedOptime_forTest(2, 1, time1)); - ReplicationCoordinator::StatusAndDuration statusAndDur = awaiter1.getResult(); - ASSERT_OK(statusAndDur.status); - statusAndDur = awaiter2.getResult(); - ASSERT_OK(statusAndDur.status); - ASSERT_EQ(replicationWaiterListMetric.get(), 0); - ASSERT_EQ(opTimeWaiterListMetric.get(), 0); - awaiter1.reset(); - awaiter2.reset(); - - // 2 nodes waiting for time2, but only locally. - awaiter1.setOpTime(time2); - awaiter1.setWriteConcern(writeConcernLocal); - awaiter1.start(); - ASSERT_METRIC_INCREASE_TO(opTimeWaiterListMetric, 1); - ASSERT_EQ(replicationWaiterListMetric.get(), 0); - - awaiter2.setOpTime(time2); - awaiter2.setWriteConcern(writeConcernLocal); - awaiter2.start(); - ASSERT_METRIC_INCREASE_TO(opTimeWaiterListMetric, 2); - ASSERT_EQ(replicationWaiterListMetric.get(), 0); - - replCoordSetMyLastAppliedOpTime(time2, Date_t() + Seconds(100)); - replCoordSetMyLastDurableOpTime(time2, Date_t() + Seconds(100)); - ASSERT_OK(getReplCoord()->setLastAppliedOptime_forTest(2, 1, time2)); - statusAndDur = awaiter1.getResult(); - ASSERT_OK(statusAndDur.status); - statusAndDur = awaiter2.getResult(); - ASSERT_OK(statusAndDur.status); - awaiter1.reset(); - awaiter2.reset(); - ASSERT_EQ(replicationWaiterListMetric.get(), 0); - ASSERT_EQ(opTimeWaiterListMetric.get(), 0); - - // 2 nodes waiting for time3, one local one not, but we're going to step down first. - awaiter1.setOpTime(time3); - awaiter1.setWriteConcern(writeConcernLocal); - awaiter1.start(); - ASSERT_METRIC_INCREASE_TO(opTimeWaiterListMetric, 1); - ASSERT_EQ(replicationWaiterListMetric.get(), 0); - - awaiter2.setOpTime(time3); - awaiter2.setWriteConcern(writeConcern); - awaiter2.start(); - ASSERT_METRIC_INCREASE_TO(replicationWaiterListMetric, 1); - ASSERT_EQ(opTimeWaiterListMetric.get(), 1); - - const auto opCtx = makeOperationContext(); - getReplCoord()->stepDown(opCtx.get(), true, Milliseconds(0), Milliseconds(1000)); - - statusAndDur = awaiter1.getResult(); - ASSERT_EQUALS(ErrorCodes::PrimarySteppedDown, statusAndDur.status); - statusAndDur = awaiter2.getResult(); - ASSERT_EQUALS(ErrorCodes::PrimarySteppedDown, statusAndDur.status); - awaiter1.reset(); - awaiter2.reset(); - ASSERT_EQ(replicationWaiterListMetric.get(), 0); - ASSERT_EQ(opTimeWaiterListMetric.get(), 0); -} - TEST_F(ReplCoordTest, NodeReturnsFromMajorityWriteConcernOnlyOnceTheWriteAppearsInACommittedSnapShot) { // Test that we can satisfy majority write concern can only be @@ -7201,14 +6924,15 @@ TEST_F(ReplCoordTest, CancelAndRescheduleElectionTimeoutLogging) { // Setting mode to secondary should schedule the election timeout. ReplicationCoordinatorImpl* replCoord = getReplCoord(); ASSERT_OK(replCoord->setFollowerMode(MemberState::RS_SECONDARY)); - ASSERT_EQ(1, countTextFormatLogLinesContaining("Scheduled election timeout callback")); - ASSERT_EQ(0, countTextFormatLogLinesContaining("Rescheduled election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Scheduling election timeout callback")); + ASSERT_EQ(0, countTextFormatLogLinesContaining("Rescheduling election timeout callback")); ASSERT_EQ(0, countTextFormatLogLinesContaining("Canceling election timeout callback")); // Scheduling again should produce the "rescheduled", not the "scheduled", message . replCoord->cancelAndRescheduleElectionTimeout(); - ASSERT_EQ(1, countTextFormatLogLinesContaining("Scheduled election timeout callback")); - ASSERT_EQ(1, countTextFormatLogLinesContaining("Rescheduled election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Scheduling election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Rescheduling election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Canceling election timeout callback")); auto net = getNet(); net->enterNetwork(); @@ -7236,8 +6960,9 @@ TEST_F(ReplCoordTest, CancelAndRescheduleElectionTimeoutLogging) { net->exitNetwork(); // The election should have scheduled (not rescheduled) another timeout. - ASSERT_EQ(2, countTextFormatLogLinesContaining("Scheduled election timeout callback")); - ASSERT_EQ(1, countTextFormatLogLinesContaining("Rescheduled election timeout callback")); + ASSERT_EQ(2, countTextFormatLogLinesContaining("Scheduling election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Rescheduling election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Canceling election timeout callback")); auto replElectionReducedSeverityGuard = unittest::MinimumLoggedSeverityGuard{ logv2::LogComponent::kReplicationElection, logv2::LogSeverity::Debug(4)}; @@ -7248,8 +6973,9 @@ TEST_F(ReplCoordTest, CancelAndRescheduleElectionTimeoutLogging) { replCoord->cancelAndRescheduleElectionTimeout(); // We should not see this reschedule because it should be at log level 5. - ASSERT_EQ(2, countTextFormatLogLinesContaining("Scheduled election timeout callback")); - ASSERT_EQ(1, countTextFormatLogLinesContaining("Rescheduled election timeout callback")); + ASSERT_EQ(2, countTextFormatLogLinesContaining("Scheduling election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Rescheduling election timeout callback")); + ASSERT_EQ(1, countTextFormatLogLinesContaining("Canceling election timeout callback")); net->enterNetwork(); until = electionTimeoutWhen + Milliseconds(1001); @@ -7260,8 +6986,9 @@ TEST_F(ReplCoordTest, CancelAndRescheduleElectionTimeoutLogging) { stopCapturingLogMessages(); // We should see this reschedule at level 4 because it has been over 1 sec since we logged // at level 4. - ASSERT_EQ(2, countTextFormatLogLinesContaining("Scheduled election timeout callback")); - ASSERT_EQ(2, countTextFormatLogLinesContaining("Rescheduled election timeout callback")); + ASSERT_EQ(2, countTextFormatLogLinesContaining("Scheduling election timeout callback")); + ASSERT_EQ(2, countTextFormatLogLinesContaining("Rescheduling election timeout callback")); + ASSERT_EQ(2, countTextFormatLogLinesContaining("Canceling election timeout callback")); } TEST_F(ReplCoordTest, ZeroCommittedSnapshotAfterClearingCommittedSnapshot) { diff --git a/src/mongo/db/repl/replication_coordinator_mock.cpp b/src/mongo/db/repl/replication_coordinator_mock.cpp index 30e902806fc..22118c874aa 100644 --- a/src/mongo/db/repl/replication_coordinator_mock.cpp +++ b/src/mongo/db/repl/replication_coordinator_mock.cpp @@ -94,8 +94,7 @@ bool ReplicationCoordinatorMock::inQuiesceMode() const { return false; } -void ReplicationCoordinatorMock::shutdown(OperationContext*, - BSONObjBuilder* shutdownTimeElapsedBuilder) { +void ReplicationCoordinatorMock::shutdown(OperationContext*) { // TODO } @@ -241,8 +240,6 @@ void ReplicationCoordinatorMock::_setMyLastAppliedOpTimeAndWallTime( _myLastAppliedOpTime = opTimeAndWallTime.opTime; _myLastAppliedWallTime = opTimeAndWallTime.wallTime; - setCurrentCommittedSnapshotOpTime(opTimeAndWallTime.opTime); - if (auto storageEngine = _service->getStorageEngine()) { if (auto snapshotManager = storageEngine->getSnapshotManager()) { snapshotManager->setCommittedSnapshot(opTimeAndWallTime.opTime.getTimestamp()); diff --git a/src/mongo/db/repl/replication_coordinator_mock.h b/src/mongo/db/repl/replication_coordinator_mock.h index e3c0da86ee9..8e48a9bd8d7 100644 --- a/src/mongo/db/repl/replication_coordinator_mock.h +++ b/src/mongo/db/repl/replication_coordinator_mock.h @@ -75,7 +75,7 @@ public: virtual bool inQuiesceMode() const; - virtual void shutdown(OperationContext* opCtx, BSONObjBuilder* shutdownTimeElapsedBuilder); + virtual void shutdown(OperationContext* opCtx); virtual void appendDiagnosticBSON(BSONObjBuilder* bob) override {} diff --git a/src/mongo/db/repl/replication_coordinator_noop.cpp b/src/mongo/db/repl/replication_coordinator_noop.cpp index a44d4d50ab6..66f46c4fd92 100644 --- a/src/mongo/db/repl/replication_coordinator_noop.cpp +++ b/src/mongo/db/repl/replication_coordinator_noop.cpp @@ -50,8 +50,7 @@ bool ReplicationCoordinatorNoOp::inQuiesceMode() const { MONGO_UNREACHABLE; } -void ReplicationCoordinatorNoOp::shutdown(OperationContext* opCtx, - BSONObjBuilder* shutdownTimeElapsedBuilder) {} +void ReplicationCoordinatorNoOp::shutdown(OperationContext* opCtx) {} ReplicationCoordinator::Mode ReplicationCoordinatorNoOp::getReplicationMode() const { return modeReplSet; diff --git a/src/mongo/db/repl/replication_coordinator_noop.h b/src/mongo/db/repl/replication_coordinator_noop.h index 190898748fa..baf4b586d2b 100644 --- a/src/mongo/db/repl/replication_coordinator_noop.h +++ b/src/mongo/db/repl/replication_coordinator_noop.h @@ -55,7 +55,7 @@ public: bool inQuiesceMode() const final; - void shutdown(OperationContext* opCtx, BSONObjBuilder* shutdownTimeElapsedBuilder) final; + void shutdown(OperationContext* opCtx) final; ServiceContext* getServiceContext() final { return _service; diff --git a/src/mongo/db/repl/replication_coordinator_test_fixture.cpp b/src/mongo/db/repl/replication_coordinator_test_fixture.cpp index ac83a0163dc..d1bcdcc92ab 100644 --- a/src/mongo/db/repl/replication_coordinator_test_fixture.cpp +++ b/src/mongo/db/repl/replication_coordinator_test_fixture.cpp @@ -454,7 +454,7 @@ void ReplCoordTest::runSingleNodeElection(OperationContext* opCtx) { void ReplCoordTest::shutdown(OperationContext* opCtx) { invariant(_callShutdown); _net->exitNetwork(); - _repl->shutdown(opCtx, nullptr /* shutdownTimeElapsedBuilder */); + _repl->shutdown(opCtx); _callShutdown = false; } diff --git a/src/mongo/db/repl/replication_info.cpp b/src/mongo/db/repl/replication_info.cpp index 39179d71193..2172307d791 100644 --- a/src/mongo/db/repl/replication_info.cpp +++ b/src/mongo/db/repl/replication_info.cpp @@ -248,11 +248,11 @@ public: BSONObjBuilder result; result.append("latestOptime", replCoord->getMyLastAppliedOpTime().getTimestamp()); - auto earliestOplogTimestampFetch = [&]() -> Timestamp { + auto earliestOplogTimestampFetch = [&]() -> StatusWith<Timestamp> { auto oplog = CollectionCatalog::get(opCtx)->lookupCollectionByNamespaceForRead( opCtx, NamespaceString::kRsOplogNamespace); if (!oplog) { - return Timestamp(); + return StatusWith<Timestamp>(ErrorCodes::NamespaceNotFound, "oplog doesn't exist"); } // Try to get the lock. If it's already locked, immediately return null timestamp. @@ -282,13 +282,13 @@ public: return o["ts"].timestamp(); } } - if (!swEarliestOplogTimestamp.isOK()) { - return Timestamp(); - } - return swEarliestOplogTimestamp.getValue(); + + return swEarliestOplogTimestamp; }(); - result.append("earliestOptime", earliestOplogTimestampFetch); + uassert( + 17347, "Problem reading earliest entry from oplog", earliestOplogTimestampFetch.isOK()); + result.append("earliestOptime", earliestOplogTimestampFetch.getValue()); return result.obj(); } @@ -367,14 +367,6 @@ public: sessionTagsToSet |= transport::Session::kKeepOpen; } - // Negotiate compressors before logging metadata so we can include the result in the log - // line. - auto result = replyBuilder->getBodyBuilder(); - if (opCtx->getClient()->session()) { - MessageCompressorManager::forSession(opCtx->getClient()->session()) - .serverNegotiate(cmd.getCompression(), &result); - } - auto client = opCtx->getClient(); if (ClientMetadata::tryFinalize(client)) { audit::logClientMetadata(client); @@ -447,6 +439,8 @@ public: !clientTopologyVersion && !maxAwaitTimeMS); } + auto result = replyBuilder->getBodyBuilder(); + // Try to parse the optional 'helloOk' field. This should be provided on the initial // handshake for an incoming connection if the client supports the hello command. Clients // that specify 'helloOk' do not rely on "not master" error message parsing, which means @@ -506,6 +500,11 @@ public: param->append(opCtx, result, kAutomationServiceDescriptorFieldName); } + if (opCtx->getClient()->session()) { + MessageCompressorManager::forSession(opCtx->getClient()->session()) + .serverNegotiate(cmd.getCompression(), &result); + } + if (opCtx->isExhaust()) { LOGV2_DEBUG(23905, 3, "Using exhaust for isMaster or hello protocol"); diff --git a/src/mongo/db/repl/replication_recovery.cpp b/src/mongo/db/repl/replication_recovery.cpp index f8f983e936f..80388fd2f3e 100644 --- a/src/mongo/db/repl/replication_recovery.cpp +++ b/src/mongo/db/repl/replication_recovery.cpp @@ -332,7 +332,9 @@ void ReplicationRecoveryImpl::recoverFromOplogAsStandalone(OperationContext* opC // We support only recovery from stable checkpoints during initial sync. invariant(!_duringInitialSync || recoveryTS); - boost::optional<Timestamp> stableTimestamp = boost::none; + // Initialize the cached pointer to the oplog collection. + acquireOplogCollectionForLogging(opCtx); + if (recoveryTS || startupRecoveryForRestore) { if (startupRecoveryForRestore && !recoveryTS) { LOGV2_WARNING(5576601, @@ -343,7 +345,8 @@ void ReplicationRecoveryImpl::recoverFromOplogAsStandalone(OperationContext* opC // We pass in "none" for the stable timestamp so that recoverFromOplog asks storage // for the recoveryTimestamp just like on replica set recovery. - stableTimestamp = recoverFromOplog(opCtx, boost::none); + const auto stableTimestamp = boost::none; + recoverFromOplog(opCtx, stableTimestamp); } else { if (gTakeUnstableCheckpointOnShutdown) { // Ensure 'recoverFromOplogAsStandalone' with 'takeUnstableCheckpointOnShutdown' @@ -363,10 +366,7 @@ void ReplicationRecoveryImpl::recoverFromOplogAsStandalone(OperationContext* opC if (!_duringInitialSync) { // Initial sync will reconstruct prepared transactions when it is completely done. - reconstructPreparedTransactions(opCtx, - stableTimestamp - ? OplogApplication::Mode::kStableRecovering - : OplogApplication::Mode::kUnstableRecovering); + reconstructPreparedTransactions(opCtx, OplogApplication::Mode::kRecovering); // Two-phase index builds are built in the background, which may still be in-progress after // recovering from the oplog. To prevent crashing the server, skip enabling read-only mode. @@ -392,6 +392,9 @@ void ReplicationRecoveryImpl::recoverFromOplogUpTo(OperationContext* opCtx, Time "Cannot use 'recoverToOplogTimestamp' without a stable checkpoint"); } + // Initialize the cached pointer to the oplog collection. + acquireOplogCollectionForLogging(opCtx); + // This may take an IS lock on the oplog collection. _truncateOplogIfNeededAndThenClearOplogTruncateAfterPoint(opCtx, &recoveryTS); @@ -435,14 +438,14 @@ void ReplicationRecoveryImpl::recoverFromOplogUpTo(OperationContext* opCtx, Time invariant(appliedUpTo <= endPoint); } - reconstructPreparedTransactions(opCtx, OplogApplication::Mode::kStableRecovering); + reconstructPreparedTransactions(opCtx, OplogApplication::Mode::kRecovering); } -boost::optional<Timestamp> ReplicationRecoveryImpl::recoverFromOplog( - OperationContext* opCtx, boost::optional<Timestamp> stableTimestamp) try { +void ReplicationRecoveryImpl::recoverFromOplog(OperationContext* opCtx, + boost::optional<Timestamp> stableTimestamp) try { if (_consistencyMarkers->getInitialSyncFlag(opCtx)) { LOGV2(21542, "No recovery needed. Initial sync flag set"); - return stableTimestamp; // Initial Sync will take over so no cleanup is needed. + return; // Initial Sync will take over so no cleanup is needed. } const auto serviceCtx = getGlobalServiceContext(); @@ -484,7 +487,7 @@ boost::optional<Timestamp> ReplicationRecoveryImpl::recoverFromOplog( // Oplog is empty. There are no oplog entries to apply, so we exit recovery and go into // initial sync. LOGV2(21543, "No oplog entries to apply for recovery. Oplog is empty"); - return stableTimestamp; + return; } fassert(40290, topOfOplogSW); const auto topOfOplog = topOfOplogSW.getValue(); @@ -498,7 +501,6 @@ boost::optional<Timestamp> ReplicationRecoveryImpl::recoverFromOplog( _recoverFromUnstableCheckpoint( opCtx, _consistencyMarkers->getAppliedThrough(opCtx), topOfOplog); } - return stableTimestamp; } catch (...) { LOGV2_FATAL_CONTINUE(21570, "Caught exception during replication recovery: {error}", @@ -711,10 +713,6 @@ Timestamp ReplicationRecoveryImpl::_applyOplogOperations(OperationContext* opCtx RecoveryOplogApplierStats stats; - auto oplogApplicationMode = (recoveryMode == RecoveryMode::kStartupFromStableTimestamp || - recoveryMode == RecoveryMode::kRollbackFromStableTimestamp) - ? OplogApplication::Mode::kStableRecovering - : OplogApplication::Mode::kUnstableRecovering; auto writerPool = makeReplWriterPool(); auto* replCoord = ReplicationCoordinator::get(opCtx); OplogApplierImpl oplogApplier(nullptr, @@ -723,7 +721,7 @@ Timestamp ReplicationRecoveryImpl::_applyOplogOperations(OperationContext* opCtx replCoord, _consistencyMarkers, _storageInterface, - OplogApplier::Options(oplogApplicationMode), + OplogApplier::Options(OplogApplication::Mode::kRecovering), writerPool.get()); OplogApplier::BatchLimits batchLimits; diff --git a/src/mongo/db/repl/replication_recovery.h b/src/mongo/db/repl/replication_recovery.h index 6eb7782de00..c6c4a000918 100644 --- a/src/mongo/db/repl/replication_recovery.h +++ b/src/mongo/db/repl/replication_recovery.h @@ -53,12 +53,9 @@ public: /** * Recovers the data on disk from the oplog. If the provided stable timestamp is not "none", * this function assumes the data reflects that timestamp. - * Returns the provided stable timestamp. If the provided stable timestamp is "none", this - * function might try to ask storage for the last stable timestamp if it exists before doing - * recovery which will be returned after performing successful recovery. */ - virtual boost::optional<Timestamp> recoverFromOplog( - OperationContext* opCtx, boost::optional<Timestamp> stableTimestamp) = 0; + virtual void recoverFromOplog(OperationContext* opCtx, + boost::optional<Timestamp> stableTimestamp) = 0; /** * Recovers the data on disk from the oplog and puts the node in readOnly mode. If @@ -82,8 +79,8 @@ public: ReplicationRecoveryImpl(StorageInterface* storageInterface, ReplicationConsistencyMarkers* consistencyMarkers); - boost::optional<Timestamp> recoverFromOplog( - OperationContext* opCtx, boost::optional<Timestamp> stableTimestamp) override; + void recoverFromOplog(OperationContext* opCtx, + boost::optional<Timestamp> stableTimestamp) override; void recoverFromOplogAsStandalone(OperationContext* opCtx, bool duringInitialSync = false) override; diff --git a/src/mongo/db/repl/replication_recovery_mock.h b/src/mongo/db/repl/replication_recovery_mock.h index a34ce54500a..97072a4fdd9 100644 --- a/src/mongo/db/repl/replication_recovery_mock.h +++ b/src/mongo/db/repl/replication_recovery_mock.h @@ -42,10 +42,8 @@ class ReplicationRecoveryMock : public ReplicationRecovery { public: ReplicationRecoveryMock() = default; - boost::optional<Timestamp> recoverFromOplog( - OperationContext* opCtx, boost::optional<Timestamp> stableTimestamp) override { - return stableTimestamp; - } + void recoverFromOplog(OperationContext* opCtx, + boost::optional<Timestamp> stableTimestamp) override {} void recoverFromOplogAsStandalone(OperationContext* opCtx, bool duringInitialSync = false) override {} diff --git a/src/mongo/db/repl/replication_recovery_test.cpp b/src/mongo/db/repl/replication_recovery_test.cpp index 3e23497ec65..00bd8b1c35b 100644 --- a/src/mongo/db/repl/replication_recovery_test.cpp +++ b/src/mongo/db/repl/replication_recovery_test.cpp @@ -261,7 +261,6 @@ repl::OplogEntry _makeOplogEntry(repl::OpTime opTime, testNs, // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert repl::OplogEntry::kOplogVersion, // version object, // o object2, // o2 @@ -703,7 +702,7 @@ TEST_F(ReplicationRecoveryTest, testRecoveryToStableAppliesDocumentsWithNoAppliedThrough(false); } -TEST_F(ReplicationRecoveryTest, UnstableRecoveryIgnoresDroppedCollections) { +TEST_F(ReplicationRecoveryTest, RecoveryIgnoresDroppedCollections) { ReplicationRecoveryImpl recovery(getStorageInterface(), getConsistencyMarkers()); auto opCtx = getOperationContext(); @@ -715,7 +714,7 @@ TEST_F(ReplicationRecoveryTest, UnstableRecoveryIgnoresDroppedCollections) { ASSERT_FALSE(autoColl.getCollection()); } - // Not setting a stable timestamp in order to perform unstable recovery, + getStorageInterfaceRecovery()->setRecoveryTimestamp(Timestamp(2, 2)); recovery.recoverFromOplog(opCtx, boost::none); _assertDocsInOplog(opCtx, {1, 2, 3, 4, 5}); @@ -726,24 +725,6 @@ TEST_F(ReplicationRecoveryTest, UnstableRecoveryIgnoresDroppedCollections) { ASSERT_EQ(getConsistencyMarkers()->getOplogTruncateAfterPoint(opCtx), Timestamp()); } -DEATH_TEST_REGEX_F(ReplicationRecoveryTest, - StableRecoveryCrashOnDroppedCollectionsInTests, - "Fatal assertion.*5415000") { - ReplicationRecoveryImpl recovery(getStorageInterface(), getConsistencyMarkers()); - auto opCtx = getOperationContext(); - - _setUpOplog(opCtx, getStorageInterface(), {1, 2, 3, 4, 5}); - - ASSERT_OK(getStorageInterface()->dropCollection(opCtx, testNs)); - { - AutoGetCollectionForReadCommand autoColl(opCtx, testNs); - ASSERT_FALSE(autoColl.getCollection()); - } - - getStorageInterfaceRecovery()->setRecoveryTimestamp(Timestamp(2, 2)); - recovery.recoverFromOplog(opCtx, boost::none); -} - TEST_F(ReplicationRecoveryTest, RecoveryAppliesDocumentsWhenAppliedThroughIsBehindAfterTruncation) { ReplicationRecoveryImpl recovery(getStorageInterface(), getConsistencyMarkers()); auto opCtx = getOperationContext(); @@ -1231,7 +1212,7 @@ TEST_F(ReplicationRecoveryTest, ASSERT_EQ(getConsistencyMarkers()->getOplogTruncateAfterPoint(opCtx), Timestamp()); } -TEST_F(ReplicationRecoveryTest, RecoverFromOplogUpTo) { +TEST_F(ReplicationRecoveryTest, RecoverFromOplogUpToBeforeEndOfOplog) { ReplicationRecoveryImpl recovery(getStorageInterface(), getConsistencyMarkers()); auto opCtx = getOperationContext(); @@ -1241,16 +1222,8 @@ TEST_F(ReplicationRecoveryTest, RecoverFromOplogUpTo) { // Recovers operations with timestamps: 3, 4, 5. recovery.recoverFromOplogUpTo(opCtx, Timestamp(5, 5)); _assertDocsInTestCollection(opCtx, {3, 4, 5}); -} - -TEST_F(ReplicationRecoveryTest, RecoverFromOplogUpToBeforeEndOfOplog) { - ReplicationRecoveryImpl recovery(getStorageInterface(), getConsistencyMarkers()); - auto opCtx = getOperationContext(); - _setUpOplog(opCtx, getStorageInterface(), {2, 3, 4, 5, 6, 7, 8, 9, 10}); - getStorageInterfaceRecovery()->setRecoveryTimestamp(Timestamp(2, 2)); - - // Recovers operations with timestamps: 3, 4, 5, 6, 7, 8, 9. + // Recovers operations with timestamps: 6, 7, 8, 9. recovery.recoverFromOplogUpTo(opCtx, Timestamp(9, 9)); _assertDocsInTestCollection(opCtx, {3, 4, 5, 6, 7, 8, 9}); } @@ -1316,6 +1289,8 @@ TEST_F(ReplicationRecoveryTest, RecoverFromOplogUpToDoesNotExceedEndPoint) { _setUpOplog(opCtx, getStorageInterface(), {2, 5, 10}); getStorageInterfaceRecovery()->setRecoveryTimestamp(Timestamp(2, 2)); + recovery.recoverFromOplogUpTo(opCtx, Timestamp(9, 9)); + recovery.recoverFromOplogUpTo(opCtx, Timestamp(15, 15)); } diff --git a/src/mongo/db/repl/rollback_impl.cpp b/src/mongo/db/repl/rollback_impl.cpp index 5ea811dff36..1cec7be6b30 100644 --- a/src/mongo/db/repl/rollback_impl.cpp +++ b/src/mongo/db/repl/rollback_impl.cpp @@ -42,8 +42,8 @@ #include "mongo/db/catalog/import_collection_oplog_entry_gen.h" #include "mongo/db/commands.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" #include "mongo/db/concurrency/replication_state_transition_lock_guard.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" #include "mongo/db/dbhelpers.h" @@ -655,7 +655,7 @@ void RollbackImpl::_runPhaseFromAbortToReconstructPreparedTxns( // transactions were aborted (i.e. the in-memory counts were rolled-back) before computing // collection counts, reconstruct the prepared transactions now, adding on any additional counts // to the now corrected record store. - reconstructPreparedTransactions(opCtx, OplogApplication::Mode::kStableRecovering); + reconstructPreparedTransactions(opCtx, OplogApplication::Mode::kRecovering); } void RollbackImpl::_correctRecordStoreCounts(OperationContext* opCtx) { diff --git a/src/mongo/db/repl/rollback_impl_test.cpp b/src/mongo/db/repl/rollback_impl_test.cpp index 336dfb07db4..af9e2d85dad 100644 --- a/src/mongo/db/repl/rollback_impl_test.cpp +++ b/src/mongo/db/repl/rollback_impl_test.cpp @@ -1491,14 +1491,13 @@ RollbackImplTest::_setUpUnpreparedTransactionForCountTest(UUID collId) { insertOp2Obj = insertOp2Obj.removeField("wall"); auto partialApplyOpsObj = BSON("applyOps" << BSON_ARRAY(insertOp2Obj) << "partialTxn" << true); - DurableOplogEntry partialApplyOpsOplogEntry(partialApplyOpsOpTime, // opTime - 1LL, // hash - OpTypeEnum::kCommand, // opType - boost::none, // tenant id - adminCmdNss, // nss - boost::none, // uuid - boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert + DurableOplogEntry partialApplyOpsOplogEntry(partialApplyOpsOpTime, // opTime + 1LL, // hash + OpTypeEnum::kCommand, // opType + boost::none, // tenant id + adminCmdNss, // nss + boost::none, // uuid + boost::none, // fromMigrate OplogEntry::kOplogVersion, // version partialApplyOpsObj, // oField boost::none, // o2Field @@ -1535,7 +1534,6 @@ RollbackImplTest::_setUpUnpreparedTransactionForCountTest(UUID collId) { adminCmdNss, // nss boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version commitApplyOpsObj, // oField boost::none, // o2Field diff --git a/src/mongo/db/repl/rs_rollback.cpp b/src/mongo/db/repl/rs_rollback.cpp index d1d460dcd83..35951278ee1 100644 --- a/src/mongo/db/repl/rs_rollback.cpp +++ b/src/mongo/db/repl/rs_rollback.cpp @@ -49,8 +49,8 @@ #include "mongo/db/client.h" #include "mongo/db/commands.h" #include "mongo/db/commands/txn_cmds_gen.h" -#include "mongo/db/concurrency/exception_util.h" #include "mongo/db/concurrency/replication_state_transition_lock_guard.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/exec/working_set_common.h" @@ -860,10 +860,9 @@ void dropIndex(OperationContext* opCtx, const string& indexName, NamespaceString& nss) { IndexCatalog* indexCatalog = collection->getIndexCatalog(); - auto indexDescriptor = indexCatalog->findIndexByName( - opCtx, - indexName, - IndexCatalog::InclusionPolicy::kReady | IndexCatalog::InclusionPolicy::kUnfinished); + bool includeUnfinishedIndexes = true; + auto indexDescriptor = + indexCatalog->findIndexByName(opCtx, indexName, includeUnfinishedIndexes); if (!indexDescriptor) { LOGV2_WARNING(21725, "Rollback failed to drop index {indexName} in {namespace}: index not found.", @@ -999,7 +998,7 @@ void rollbackDropIndexes(OperationContext* opCtx, "uuid"_attr = uuid, "indexName"_attr = indexName); - createIndexForApplyOps(opCtx, indexSpec, *nss, OplogApplication::Mode::kStableRecovering); + createIndexForApplyOps(opCtx, indexSpec, *nss, OplogApplication::Mode::kRecovering); LOGV2_DEBUG(21676, 1, diff --git a/src/mongo/db/repl/session_update_tracker.cpp b/src/mongo/db/repl/session_update_tracker.cpp index a5939923519..87c12cf5463 100644 --- a/src/mongo/db/repl/session_update_tracker.cpp +++ b/src/mongo/db/repl/session_update_tracker.cpp @@ -62,7 +62,6 @@ OplogEntry createOplogEntryForTransactionTableUpdate(repl::OpTime opTime, NamespaceString::kSessionTransactionsTableNamespace, boost::none, // uuid false, // fromMigrate - boost::none, // checkExistenceForDiffInsert repl::OplogEntry::kOplogVersion, updateBSON, o2Field, @@ -213,7 +212,7 @@ boost::optional<std::vector<OplogEntry>> SessionUpdateTracker::_updateSessionInf return {}; } - if (!entry.getObject2() || entry.getObject2()->isEmpty()) { + if (!entry.getObject2()) { return {}; } } diff --git a/src/mongo/db/repl/storage_interface_impl.cpp b/src/mongo/db/repl/storage_interface_impl.cpp index 0c91ba3b592..4aedc7c2473 100644 --- a/src/mongo/db/repl/storage_interface_impl.cpp +++ b/src/mongo/db/repl/storage_interface_impl.cpp @@ -53,8 +53,8 @@ #include "mongo/db/catalog/index_catalog.h" #include "mongo/db/client.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" #include "mongo/db/concurrency/lock_state.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/curop.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbhelpers.h" @@ -638,9 +638,7 @@ Status StorageInterfaceImpl::setIndexIsMultikey(OperationContext* opCtx, } auto idx = collection->getIndexCatalog()->findIndexByName( - opCtx, - indexName, - IndexCatalog::InclusionPolicy::kReady | IndexCatalog::InclusionPolicy::kUnfinished); + opCtx, indexName, true /* includeUnfinishedIndexes */); if (!idx) { return Status(ErrorCodes::IndexNotFound, str::stream() @@ -788,8 +786,9 @@ StatusWith<std::vector<BSONObj>> _findOrDeleteDocuments( // Use index scan. auto indexCatalog = collection->getIndexCatalog(); invariant(indexCatalog); - const IndexDescriptor* indexDescriptor = indexCatalog->findIndexByName( - opCtx, *indexName, IndexCatalog::InclusionPolicy::kReady); + bool includeUnfinishedIndexes = false; + const IndexDescriptor* indexDescriptor = + indexCatalog->findIndexByName(opCtx, *indexName, includeUnfinishedIndexes); if (!indexDescriptor) { return Result(ErrorCodes::IndexNotFound, str::stream() << "Index not found, ns:" << nsOrUUID.toString() diff --git a/src/mongo/db/repl/storage_interface_impl_test.cpp b/src/mongo/db/repl/storage_interface_impl_test.cpp index 3c942ed7361..de039c3705c 100644 --- a/src/mongo/db/repl/storage_interface_impl_test.cpp +++ b/src/mongo/db/repl/storage_interface_impl_test.cpp @@ -42,7 +42,7 @@ #include "mongo/db/catalog/validate_results.h" #include "mongo/db/client.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/index/index_access_method.h" #include "mongo/db/index/index_descriptor.h" diff --git a/src/mongo/db/repl/storage_timestamp_test.cpp b/src/mongo/db/repl/storage_timestamp_test.cpp index 6431be37aed..7919675a946 100644 --- a/src/mongo/db/repl/storage_timestamp_test.cpp +++ b/src/mongo/db/repl/storage_timestamp_test.cpp @@ -46,7 +46,7 @@ #include "mongo/db/catalog/index_catalog.h" #include "mongo/db/catalog/multi_index_block.h" #include "mongo/db/client.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" #include "mongo/db/dbhelpers.h" @@ -90,6 +90,7 @@ #include "mongo/db/transaction_participant_gen.h" #include "mongo/db/vector_clock_mutable.h" #include "mongo/dbtests/dbtests.h" +#include "mongo/idl/server_parameter_test_util.h" #include "mongo/logv2/log.h" #include "mongo/rpc/get_status_from_command_result.h" #include "mongo/stdx/future.h" @@ -1473,7 +1474,7 @@ TEST_F(StorageTimestampTest, SecondarySetWildcardIndexMultikeyOnInsert) { _coordinatorMock, _consistencyMarkers, storageInterface, - repl::OplogApplier::Options(repl::OplogApplication::Mode::kStableRecovering), + repl::OplogApplier::Options(repl::OplogApplication::Mode::kRecovering), writerPool.get()); uassertStatusOK(oplogApplier.applyOplogBatch(_opCtx, ops)); @@ -1571,7 +1572,7 @@ TEST_F(StorageTimestampTest, SecondarySetWildcardIndexMultikeyOnUpdate) { _coordinatorMock, _consistencyMarkers, storageInterface, - repl::OplogApplier::Options(repl::OplogApplication::Mode::kStableRecovering), + repl::OplogApplier::Options(repl::OplogApplication::Mode::kRecovering), writerPool.get()); uassertStatusOK(oplogApplier.applyOplogBatch(_opCtx, ops)); @@ -2703,10 +2704,8 @@ TEST_F(StorageTimestampTest, IndexBuildsResolveErrorsDuringStateChangeToPrimary) } auto indexCatalog = collection->getIndexCatalog(); - buildingIndex = indexCatalog->getEntry(indexCatalog->findIndexByName( - _opCtx, - "a_1_b_1", - IndexCatalog::InclusionPolicy::kReady | IndexCatalog::InclusionPolicy::kUnfinished)); + buildingIndex = indexCatalog->getEntry( + indexCatalog->findIndexByName(_opCtx, "a_1_b_1", /* includeUnfinished */ true)); ASSERT(buildingIndex); ASSERT_OK(indexer.insertAllDocumentsInCollection(_opCtx, collection.get())); diff --git a/src/mongo/db/repl/sync_source_resolver.cpp b/src/mongo/db/repl/sync_source_resolver.cpp index b5a41a6537b..69437154c6f 100644 --- a/src/mongo/db/repl/sync_source_resolver.cpp +++ b/src/mongo/db/repl/sync_source_resolver.cpp @@ -172,8 +172,7 @@ std::unique_ptr<Fetcher> SyncSourceResolver::_makeFirstOplogEntryFetcher( << "projection" << BSON(OplogEntryBase::kTimestampFieldName << 1 << OplogEntryBase::kTermFieldName << 1) - << ReadConcernArgs::kReadConcernFieldName << ReadConcernArgs::kLocal << "term" - << -1), + << ReadConcernArgs::kReadConcernFieldName << ReadConcernArgs::kLocal), [=](const StatusWith<Fetcher::QueryResponse>& response, Fetcher::NextAction*, BSONObjBuilder*) { diff --git a/src/mongo/db/repl/sync_source_resolver_test.cpp b/src/mongo/db/repl/sync_source_resolver_test.cpp index 460e8c033d4..65eddde1cf4 100644 --- a/src/mongo/db/repl/sync_source_resolver_test.cpp +++ b/src/mongo/db/repl/sync_source_resolver_test.cpp @@ -318,7 +318,6 @@ BSONObj _makeOplogEntry(Timestamp ts, long long term) { NamespaceString("a.a"), // namespace boost::none, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert repl::OplogEntry::kOplogVersion, // version BSONObj(), // o boost::none, // o2 diff --git a/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp b/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp index 954e88d044a..e1790384202 100644 --- a/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp +++ b/src/mongo/db/repl/tenant_migration_access_blocker_util.cpp @@ -35,7 +35,7 @@ #include "mongo/db/repl/tenant_migration_access_blocker_util.h" #include "mongo/db/catalog_raii.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/op_observer.h" #include "mongo/db/persistent_task_store.h" #include "mongo/db/repl/tenant_migration_access_blocker_registry.h" diff --git a/src/mongo/db/repl/tenant_migration_donor_op_observer.h b/src/mongo/db/repl/tenant_migration_donor_op_observer.h index 403d2b6b486..43992f5e040 100644 --- a/src/mongo/db/repl/tenant_migration_donor_op_observer.h +++ b/src/mongo/db/repl/tenant_migration_donor_op_observer.h @@ -188,10 +188,6 @@ public: Timestamp commitTimestamp, const std::vector<repl::ReplOperation>& statements) noexcept final {} - void onTransactionPrepareNonPrimary(OperationContext* opCtx, - const std::vector<repl::OplogEntry>& statements, - const repl::OpTime& prepareOpTime) final {} - std::unique_ptr<ApplyOpsOplogSlotAndOperationAssignment> preTransactionPrepare( OperationContext* opCtx, const std::vector<OplogSlot>& reservedSlots, diff --git a/src/mongo/db/repl/tenant_migration_donor_service.cpp b/src/mongo/db/repl/tenant_migration_donor_service.cpp index 5fa0c560284..e7912da9577 100644 --- a/src/mongo/db/repl/tenant_migration_donor_service.cpp +++ b/src/mongo/db/repl/tenant_migration_donor_service.cpp @@ -36,11 +36,10 @@ #include "mongo/config.h" #include "mongo/db/commands/tenant_migration_donor_cmds_gen.h" #include "mongo/db/commands/tenant_migration_recipient_cmds_gen.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/index_builds_coordinator.h" -#include "mongo/db/keys_collection_util.h" #include "mongo/db/persistent_task_store.h" #include "mongo/db/query/find_command_gen.h" #include "mongo/db/repl/repl_server_parameters_gen.h" @@ -68,7 +67,6 @@ MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeLeavingAbortingIndexBuildsStat MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeLeavingBlockingState); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeLeavingDataSyncState); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeFetchingKeys); -MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationDonorBeforeStoringExternalClusterTimeKeyDocs); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationDonorBeforeWaitingForKeysToReplicate); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationDonorBeforeMarkingStateGarbageCollectable); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationDonorAfterMarkingStateGarbageCollectable); @@ -79,7 +77,6 @@ MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeInsertingDonorStateDoc); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeCreatingStateDocumentTTLIndex); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeCreatingExternalKeysTTLIndex); MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeLeavingCommittedState); -MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationAfterUpdatingToCommittedState); const std::string kTTLIndexName = "TenantMigrationDonorTTLIndex"; const std::string kExternalKeysTTLIndexName = "ExternalKeysTTLIndex"; @@ -620,10 +617,6 @@ ExecutorFuture<repl::OpTime> TenantMigrationDonorService::Instance::_updateState wuow.commit(); - if (nextState == TenantMigrationDonorStateEnum::kCommitted) { - pauseTenantMigrationAfterUpdatingToCommittedState.pauseWhileSet(); - } - updateOpTime = oplogSlot; }); @@ -939,13 +932,13 @@ SemiFuture<void> TenantMigrationDonorService::Instance::run( return _waitForRecipientToBecomeConsistentAndEnterBlockingState( executor, recipientTargeterRS, abortToken); }) - .then([this, self = shared_from_this(), executor, recipientTargeterRS, abortToken, token] { + .then([this, self = shared_from_this(), executor, recipientTargeterRS, abortToken] { LOGV2(6104905, "Waiting for recipient to reach the block timestamp.", "migrationId"_attr = _migrationUuid, "tenantId"_attr = _tenantId); return _waitForRecipientToReachBlockTimestampAndEnterCommittedState( - executor, recipientTargeterRS, abortToken, token); + executor, recipientTargeterRS, abortToken); }) // Note from here on the migration cannot be aborted, so only the token from the primary // only service should be used. @@ -1071,34 +1064,32 @@ TenantMigrationDonorService::Instance::_fetchAndStoreRecipientClusterTimeKeyDocs std::make_shared<std::vector<ExternalKeysCollectionDocument>>(); auto fetchStatus = std::make_shared<boost::optional<Status>>(); - auto fetcherCallback = [this, - self = shared_from_this(), - fetchStatus, - keyDocs]( - const Fetcher::QueryResponseStatus& dataStatus, - Fetcher::NextAction* nextAction, - BSONObjBuilder* getMoreBob) { - // Throw out any accumulated results on error - if (!dataStatus.isOK()) { - *fetchStatus = dataStatus.getStatus(); - keyDocs->clear(); - return; - } + auto fetcherCallback = + [this, self = shared_from_this(), fetchStatus, keyDocs]( + const Fetcher::QueryResponseStatus& dataStatus, + Fetcher::NextAction* nextAction, + BSONObjBuilder* getMoreBob) { + // Throw out any accumulated results on error + if (!dataStatus.isOK()) { + *fetchStatus = dataStatus.getStatus(); + keyDocs->clear(); + return; + } - const auto& data = dataStatus.getValue(); - for (const BSONObj& doc : data.documents) { - keyDocs->push_back( - keys_collection_util::makeExternalClusterTimeKeyDoc( - doc.getOwned(), _migrationUuid, boost::none /* expireAt */)); - } - *fetchStatus = Status::OK(); + const auto& data = dataStatus.getValue(); + for (const BSONObj& doc : data.documents) { + keyDocs->push_back( + tenant_migration_util::makeExternalClusterTimeKeyDoc( + _migrationUuid, doc.getOwned())); + } + *fetchStatus = Status::OK(); - if (!getMoreBob) { - return; - } - getMoreBob->append("getMore", data.cursorId); - getMoreBob->append("collection", data.nss.coll()); - }; + if (!getMoreBob) { + return; + } + getMoreBob->append("getMore", data.cursorId); + getMoreBob->append("collection", data.nss.coll()); + }; auto fetcher = std::make_shared<Fetcher>( _recipientCmdExecutor.get(), @@ -1153,11 +1144,8 @@ TenantMigrationDonorService::Instance::_fetchAndStoreRecipientClusterTimeKeyDocs .then([this, self = shared_from_this(), executor, token](auto keyDocs) { checkForTokenInterrupt(token); - auto opCtx = cc().makeOperationContext(); - pauseTenantMigrationDonorBeforeStoringExternalClusterTimeKeyDocs - .pauseWhileSet(opCtx.get()); - return keys_collection_util::storeExternalClusterTimeKeyDocs( - opCtx.get(), std::move(keyDocs)); + return tenant_migration_util::storeExternalClusterTimeKeyDocs( + std::move(keyDocs)); }) .then([this, self = shared_from_this(), token](repl::OpTime lastKeyOpTime) { pauseTenantMigrationDonorBeforeWaitingForKeysToReplicate.pauseWhileSet(); @@ -1231,7 +1219,6 @@ ExecutorFuture<void> TenantMigrationDonorService::Instance::_waitForRecipientToReachBlockTimestampAndEnterCommittedState( const std::shared_ptr<executor::ScopedTaskExecutor>& executor, std::shared_ptr<RemoteCommandTargeter> recipientTargeterRS, - const CancellationToken& abortToken, const CancellationToken& token) { { stdx::lock_guard<Latch> lg(_mutex); @@ -1241,6 +1228,7 @@ TenantMigrationDonorService::Instance::_waitForRecipientToReachBlockTimestampAnd invariant(_stateDoc.getBlockTimestamp()); } + // Source to cancel the timeout if the operation completed in time. CancellationSource cancelTimeoutSource; CancellationSource recipientSyncDataSource(token); @@ -1296,21 +1284,18 @@ TenantMigrationDonorService::Instance::_waitForRecipientToReachBlockTimestampAnd uasserted(ErrorCodes::InternalError, "simulate a tenant migration error"); } }) - .then([this, self = shared_from_this(), executor, abortToken, token] { - // Last chance to abort - checkForTokenInterrupt(abortToken); - + .then([this, self = shared_from_this(), executor, token] { // Enter "commit" state. LOGV2(6104908, "Entering 'committed' state.", "migrationId"_attr = _migrationUuid, "tenantId"_attr = _tenantId); - // Ignore the abort token once we've entered the committed state return _updateStateDoc(executor, TenantMigrationDonorStateEnum::kCommitted, token) .then([this, self = shared_from_this(), executor, token](repl::OpTime opTime) { return _waitForMajorityWriteConcern(executor, std::move(opTime), token) .then([this, self = shared_from_this()] { pauseTenantMigrationBeforeLeavingCommittedState.pauseWhileSet(); + stdx::lock_guard<Latch> lg(_mutex); // If interrupt is called at some point during execution, it is // possible that interrupt() will fulfill the promise before we diff --git a/src/mongo/db/repl/tenant_migration_donor_service.h b/src/mongo/db/repl/tenant_migration_donor_service.h index 11fe2d6ffab..87931e03343 100644 --- a/src/mongo/db/repl/tenant_migration_donor_service.h +++ b/src/mongo/db/repl/tenant_migration_donor_service.h @@ -188,7 +188,6 @@ public: ExecutorFuture<void> _waitForRecipientToReachBlockTimestampAndEnterCommittedState( const std::shared_ptr<executor::ScopedTaskExecutor>& executor, std::shared_ptr<RemoteCommandTargeter> recipientTargeterRS, - const CancellationToken& abortToken, const CancellationToken& token); ExecutorFuture<void> _handleErrorOrEnterAbortedState( diff --git a/src/mongo/db/repl/tenant_migration_recipient_entry_helpers.cpp b/src/mongo/db/repl/tenant_migration_recipient_entry_helpers.cpp index bc798479097..ce994c1581e 100644 --- a/src/mongo/db/repl/tenant_migration_recipient_entry_helpers.cpp +++ b/src/mongo/db/repl/tenant_migration_recipient_entry_helpers.cpp @@ -33,7 +33,7 @@ #include "mongo/db/catalog/database.h" #include "mongo/db/catalog_raii.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/namespace_string.h" diff --git a/src/mongo/db/repl/tenant_migration_recipient_op_observer.h b/src/mongo/db/repl/tenant_migration_recipient_op_observer.h index 7f5181a6d4a..dd42ff6581f 100644 --- a/src/mongo/db/repl/tenant_migration_recipient_op_observer.h +++ b/src/mongo/db/repl/tenant_migration_recipient_op_observer.h @@ -207,10 +207,6 @@ public: size_t numberOfPrePostImagesToWrite, Date_t wallClockTime) final {} - void onTransactionPrepareNonPrimary(OperationContext* opCtx, - const std::vector<repl::OplogEntry>& statements, - const repl::OpTime& prepareOpTime) final {} - void onTransactionAbort(OperationContext* opCtx, boost::optional<OplogSlot> abortOplogEntryOpTime) final {} diff --git a/src/mongo/db/repl/tenant_migration_recipient_service.cpp b/src/mongo/db/repl/tenant_migration_recipient_service.cpp index 2fcfd6cbffa..3f32411c555 100644 --- a/src/mongo/db/repl/tenant_migration_recipient_service.cpp +++ b/src/mongo/db/repl/tenant_migration_recipient_service.cpp @@ -43,10 +43,9 @@ #include "mongo/db/client.h" #include "mongo/db/commands/tenant_migration_donor_cmds_gen.h" #include "mongo/db/commands/test_commands_enabled.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" -#include "mongo/db/keys_collection_util.h" #include "mongo/db/namespace_string.h" #include "mongo/db/op_observer.h" #include "mongo/db/ops/write_ops_exec.h" @@ -257,10 +256,6 @@ public: MONGO_UNREACHABLE; } - StatusWith<LastVote> loadLocalLastVoteDocument(OperationContext* opCtx) const final { - MONGO_UNREACHABLE; - } - JournalListener* getReplicationJournalListener() final { MONGO_UNREACHABLE; } @@ -2286,12 +2281,11 @@ void TenantMigrationRecipientService::Instance::_fetchAndStoreDonorClusterTimeKe auto cursor = _client->find(std::move(findRequest), _readPreference); while (cursor->more()) { const auto doc = cursor->nextSafe().getOwned(); - keyDocs.push_back(keys_collection_util::makeExternalClusterTimeKeyDoc( - doc, _migrationUuid, boost::none /* expireAt */)); + keyDocs.push_back( + tenant_migration_util::makeExternalClusterTimeKeyDoc(_migrationUuid, doc)); } - auto opCtx = cc().makeOperationContext(); - keys_collection_util::storeExternalClusterTimeKeyDocs(opCtx.get(), std::move(keyDocs)); + tenant_migration_util::storeExternalClusterTimeKeyDocs(std::move(keyDocs)); } void TenantMigrationRecipientService::Instance::_compareRecipientAndDonorFCV() const { diff --git a/src/mongo/db/repl/tenant_migration_recipient_service_test.cpp b/src/mongo/db/repl/tenant_migration_recipient_service_test.cpp index 6d3dbaee1c2..589b3e63cd0 100644 --- a/src/mongo/db/repl/tenant_migration_recipient_service_test.cpp +++ b/src/mongo/db/repl/tenant_migration_recipient_service_test.cpp @@ -94,7 +94,6 @@ OplogEntry makeOplogEntry(OpTime opTime, nss, // namespace uuid, // uuid boost::none, // fromMigrate - boost::none, // checkExistenceForDiffInsert OplogEntry::kOplogVersion, // version o, // o o2, // o2 diff --git a/src/mongo/db/repl/tenant_migration_shard_merge_util.cpp b/src/mongo/db/repl/tenant_migration_shard_merge_util.cpp index 2ef76d6839f..05779a48618 100644 --- a/src/mongo/db/repl/tenant_migration_shard_merge_util.cpp +++ b/src/mongo/db/repl/tenant_migration_shard_merge_util.cpp @@ -41,7 +41,7 @@ #include "mongo/db/catalog/create_collection.h" #include "mongo/db/catalog/uncommitted_catalog_updates.h" #include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/cursor_server_params_gen.h" #include "mongo/db/db_raii.h" #include "mongo/db/multitenancy.h" diff --git a/src/mongo/db/repl/tenant_migration_util.cpp b/src/mongo/db/repl/tenant_migration_util.cpp index 48e21715a4c..4386cdf1cce 100644 --- a/src/mongo/db/repl/tenant_migration_util.cpp +++ b/src/mongo/db/repl/tenant_migration_util.cpp @@ -32,7 +32,7 @@ #include "mongo/bson/json.h" #include "mongo/bson/mutable/algorithm.h" #include "mongo/bson/mutable/document.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbdirectclient.h" #include "mongo/db/dbhelpers.h" @@ -62,11 +62,50 @@ const std::set<std::string> kSensitiveFieldNames{"donorCertificateForRecipient", "recipientCertificateForDonor"}; MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeMarkingExternalKeysGarbageCollectable); +MONGO_FAIL_POINT_DEFINE(pauseTenantMigrationBeforeStoringExternalClusterTimeKeyDocs); } // namespace const Backoff kExponentialBackoff(Seconds(1), Milliseconds::max()); +ExternalKeysCollectionDocument makeExternalClusterTimeKeyDoc(UUID migrationId, BSONObj keyDoc) { + auto originalKeyDoc = KeysCollectionDocument::parse(IDLParserErrorContext("keyDoc"), keyDoc); + + ExternalKeysCollectionDocument externalKeyDoc( + OID::gen(), originalKeyDoc.getKeyId(), migrationId); + externalKeyDoc.setKeysCollectionDocumentBase(originalKeyDoc.getKeysCollectionDocumentBase()); + + return externalKeyDoc; +} + +repl::OpTime storeExternalClusterTimeKeyDocs(std::vector<ExternalKeysCollectionDocument> keyDocs) { + auto opCtxHolder = cc().makeOperationContext(); + auto opCtx = opCtxHolder.get(); + auto nss = NamespaceString::kExternalKeysCollectionNamespace; + + pauseTenantMigrationBeforeStoringExternalClusterTimeKeyDocs.pauseWhileSet(opCtx); + + for (auto& keyDoc : keyDocs) { + AutoGetCollection collection(opCtx, nss, MODE_IX); + + writeConflictRetry(opCtx, "CloneExternalKeyDocs", nss.ns(), [&] { + // Note that each external key's _id is generated by the migration, so this upsert can + // only insert. + const auto filter = + BSON(ExternalKeysCollectionDocument::kIdFieldName << keyDoc.getId()); + const auto updateMod = keyDoc.toBSON(); + + Helpers::upsert(opCtx, + nss.ns(), + filter, + updateMod, + /*fromMigrate=*/false); + }); + } + + return repl::ReplClientInfo::forClient(opCtx->getClient()).getLastOp(); +} + void createOplogViewForTenantMigrations(OperationContext* opCtx, Database* db) { writeConflictRetry( opCtx, "createDonorOplogView", NamespaceString::kTenantMigrationOplogView.ns(), [&] { diff --git a/src/mongo/db/repl/tenant_migration_util.h b/src/mongo/db/repl/tenant_migration_util.h index 40421caa5df..44e2ac67bbf 100644 --- a/src/mongo/db/repl/tenant_migration_util.h +++ b/src/mongo/db/repl/tenant_migration_util.h @@ -202,6 +202,19 @@ inline void protocolStorageOptionsCompatibilityCheck(OperationContext* opCtx, !opCtx->getServiceContext()->getStorageEngine()->isUsingDirectoryForIndexes()); } +/* + * Creates an ExternalKeysCollectionDocument representing an config.external_validation_keys + * document from the given the admin.system.keys document BSONObj. + */ +ExternalKeysCollectionDocument makeExternalClusterTimeKeyDoc(UUID migrationId, BSONObj keyDoc); + +/* + * For each given ExternalKeysCollectionDocument, inserts it if there is not an existing document in + * config.external_validation_keys for it with the same keyId and replicaSetName. Otherwise, + * updates the ttlExpiresAt of the existing document if it is less than the new ttlExpiresAt. + */ +repl::OpTime storeExternalClusterTimeKeyDocs(std::vector<ExternalKeysCollectionDocument> keyDocs); + /** * Sets the "ttlExpiresAt" field for the external keys so they can be garbage collected by the ttl * monitor. diff --git a/src/mongo/db/repl/tenant_oplog_applier.cpp b/src/mongo/db/repl/tenant_oplog_applier.cpp index 731f83d6929..38eb4edfd71 100644 --- a/src/mongo/db/repl/tenant_oplog_applier.cpp +++ b/src/mongo/db/repl/tenant_oplog_applier.cpp @@ -38,7 +38,7 @@ #include "mongo/db/auth/authorization_session.h" #include "mongo/db/catalog/document_validation.h" #include "mongo/db/catalog_raii.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/db_raii.h" #include "mongo/db/dbhelpers.h" #include "mongo/db/op_observer.h" diff --git a/src/mongo/db/repl/topology_coordinator.cpp b/src/mongo/db/repl/topology_coordinator.cpp index 630fefa541c..47a043e093e 100644 --- a/src/mongo/db/repl/topology_coordinator.cpp +++ b/src/mongo/db/repl/topology_coordinator.cpp @@ -267,23 +267,8 @@ void TopologyCoordinator::_setSyncSource(HostAndPort newSyncSource, HostAndPort TopologyCoordinator::chooseNewSyncSource(Date_t now, const OpTime& lastOpTimeFetched, ReadPreference readPreference) { - // If we are not a member of the current replica set configuration, no sync source is valid. - if (_selfIndex == -1) { - LOGV2_DEBUG( - 21778, 1, "Cannot sync from any members because we are not in the replica set config"); - return HostAndPort(); - } - - // Check to see if we should choose a sync source because 'unsupportedSyncSource' was - // set. - auto maybeSyncSource = _chooseSyncSourceUnsupportedSyncSourceParameter(now); - if (maybeSyncSource) { - _setSyncSource(*maybeSyncSource, now, false /* fromReplSetSyncFrom */); - return _syncSource; - } - // Check to see if we should choose a sync source because the 'replSetSyncFrom' command was set. - maybeSyncSource = _chooseSyncSourceReplSetSyncFrom(now); + auto maybeSyncSource = _chooseSyncSourceReplSetSyncFrom(now); if (maybeSyncSource) { // If we have a forced sync source via 'replSetSyncFrom', set the _replSetSyncFromSet flag // to true. @@ -541,7 +526,9 @@ bool TopologyCoordinator::_isEligibleSyncSource(int candidateIndex, } boost::optional<HostAndPort> TopologyCoordinator::_chooseSyncSourceReplSetSyncFrom(Date_t now) { - invariant(_selfIndex != -1, "Unexpectedly not in the replica set config"); + if (_selfIndex == -1) { + return boost::none; + } if (_forceSyncSourceIndex == -1) { return boost::none; @@ -557,43 +544,13 @@ boost::optional<HostAndPort> TopologyCoordinator::_chooseSyncSourceReplSetSyncFr return syncSource; } -boost::optional<HostAndPort> TopologyCoordinator::_chooseSyncSourceUnsupportedSyncSourceParameter( - Date_t now) { - invariant(_selfIndex != -1, "Unexpectedly not in the replica set config"); - - auto syncSourceStr = repl::unsupportedSyncSource; - if (syncSourceStr.empty()) { - return boost::none; - } - auto syncSource = HostAndPort(syncSourceStr); - const int syncSourceIndex = _rsConfig.findMemberIndexByHostAndPort(syncSource); - if (syncSourceIndex < 0) { - LOGV2_FATAL( - 7785600, - "Selecting node specified in 'unsupportedSyncSource' parameter failed due to host " - "and port not in replica set config.", - "unsupportedSyncSource"_attr = syncSourceStr); - } - - if (_selfIndex == syncSourceIndex) { - LOGV2_FATAL( - 7785601, - "Node specified in 'unsupportedSyncSource' parameter is self: cannot select self as " - "a sync source", - "unsupportedSyncSource"_attr = syncSourceStr); - } - - LOGV2(7785602, - "Choosing sync source candidate specified by 'unsupportedSyncSource' parameter", - "syncSource"_attr = syncSourceStr, - "syncsourceobj"_attr = syncSource); - std::string msg(str::stream() << "syncing from: " << syncSourceStr << " by request"); - setMyHeartbeatMessage(now, msg); - return syncSource; -} - boost::optional<HostAndPort> TopologyCoordinator::_chooseSyncSourceInitialChecks(Date_t now) { - invariant(_selfIndex != -1, "Unexpectedly not in the replica set config"); + // If we are not a member of the current replica set configuration, no sync source is valid. + if (_selfIndex == -1) { + LOGV2_DEBUG( + 21778, 1, "Cannot sync from any members because we are not in the replica set config"); + return HostAndPort(); + } if (auto sfp = forceSyncSourceCandidate.scoped(); MONGO_unlikely(sfp.isActive())) { const auto& data = sfp.getData(); @@ -748,13 +705,6 @@ void TopologyCoordinator::prepareSyncFromResponse(const HostAndPort& target, return; } - if (!repl::unsupportedSyncSource.empty()) { - *result = - Status(ErrorCodes::IllegalOperation, - "replSetSyncFrom may not be used when 'unsupportedSyncSource' parameter is set"); - return; - } - ReplSetConfig::MemberIterator targetConfig = _rsConfig.membersEnd(); int targetIndex = 0; for (ReplSetConfig::MemberIterator it = _rsConfig.membersBegin(); it != _rsConfig.membersEnd(); @@ -3161,13 +3111,6 @@ TopologyCoordinator::_shouldChangeSyncSourceInitialChecks(const HostAndPort& cur return {ChangeSyncSourceDecision::kNo, -1}; } - if (!repl::unsupportedSyncSource.empty()) { - LOGV2(7785604, - "Not choosing new sync source because sync source is forced via " - "'unsupportedSyncSource' parameter"); - return {ChangeSyncSourceDecision::kNo, -1}; - } - // If the user requested a sync source change, return kYes. if (_forceSyncSourceIndex != -1) { LOGV2(21829, @@ -3361,11 +3304,10 @@ bool TopologyCoordinator::shouldChangeSyncSourceDueToPingTime(const HostAndPort& // If we find an eligible sync source that is significantly closer than our current sync source, // return true. - // Do not re-evaluate our sync source if it was set via the replSetSyncFrom command, the - // forceSyncSourceCandidate failpoint, or the 'unsupportedSyncSource' server parameter. + // Do not re-evaluate our sync source if it was set via the replSetSyncFrom command or the + // forceSyncSourceCandidate failpoint. auto sfp = forceSyncSourceCandidate.scoped(); - if (_replSetSyncFromSet || MONGO_unlikely(sfp.isActive()) || - !repl::unsupportedSyncSource.empty()) { + if (_replSetSyncFromSet || MONGO_unlikely(sfp.isActive())) { return false; } @@ -3543,7 +3485,7 @@ void TopologyCoordinator::processReplSetRequestVotes(const ReplSetRequestVotesAr if (!args.isADryRun()) { _lastVote.setTerm(args.getTerm()); _lastVote.setCandidateIndex(args.getCandidateIndex()); - LOGV2_DEBUG(5972100, 1, "Voting yes in election"); + LOGV2_DEBUG(5972100, 0, "Voting yes in election"); } response->setVoteGranted(true); } diff --git a/src/mongo/db/repl/topology_coordinator.h b/src/mongo/db/repl/topology_coordinator.h index a9b8cd8ab16..3d3084ddf76 100644 --- a/src/mongo/db/repl/topology_coordinator.h +++ b/src/mongo/db/repl/topology_coordinator.h @@ -906,9 +906,6 @@ private: // Returns a HostAndPort if one is forced via the 'replSetSyncFrom' command. boost::optional<HostAndPort> _chooseSyncSourceReplSetSyncFrom(Date_t now); - // Returns a HostAndPort if one is forced via the 'unsupportedSyncSource' startup parameter. - boost::optional<HostAndPort> _chooseSyncSourceUnsupportedSyncSourceParameter(Date_t now); - // Does preliminary checks involved in choosing sync source // * Do we have a valid configuration? // * Is the 'forceSyncSourceCandidate' failpoint enabled? diff --git a/src/mongo/db/repl/topology_coordinator_v1_test.cpp b/src/mongo/db/repl/topology_coordinator_v1_test.cpp index e29bf1edc1e..3898035b08c 100644 --- a/src/mongo/db/repl/topology_coordinator_v1_test.cpp +++ b/src/mongo/db/repl/topology_coordinator_v1_test.cpp @@ -4888,107 +4888,6 @@ TEST_F(ReevalSyncSourceTest, NoChangeWhenSyncSourceForcedByFailPoint) { ReadPreference::Nearest)); } -// Test that we will select the node specified by the 'unsupportedSyncSource' parameter as a sync -// source even if it is farther away. -TEST_F(ReevalSyncSourceTest, ChooseSyncSourceForcedByStartupParameterEvenIfFarther) { - RAIIServerParameterControllerForTest syncSourceParamGuard{"unsupportedSyncSource", - "host2:27017"}; - - // Make the desired host much farther away. - getTopoCoord().setPing_forTest(HostAndPort("host2"), pingTime); - getTopoCoord().setPing_forTest(HostAndPort("host3"), significantlyCloserPingTime); - - // Select a sync source. - auto syncSource = - getTopoCoord().chooseNewSyncSource(now()++, OpTime(), ReadPreference::Nearest); - ASSERT_EQ(syncSource, HostAndPort("host2:27017")); -} - -// Test that we will not change from the node specified by the 'unsupportedSyncSource' parameter -// due to ping time. -TEST_F(ReevalSyncSourceTest, NoChangeDueToPingTimeWhenSyncSourceForcedByStartupParameter) { - RAIIServerParameterControllerForTest syncSourceParamGuard{"unsupportedSyncSource", - "host2:27017"}; - // Select a sync source. - auto syncSource = - getTopoCoord().chooseNewSyncSource(now()++, OpTime(), ReadPreference::Nearest); - ASSERT_EQ(syncSource, HostAndPort("host2:27017")); - - // Set up so that without forcing the sync source, the node otherwise would have changed sync - // sources. - getTopoCoord().setPing_forTest(HostAndPort("host2"), pingTime); - getTopoCoord().setPing_forTest(HostAndPort("host3"), significantlyCloserPingTime); - - ASSERT_FALSE(getTopoCoord().shouldChangeSyncSourceDueToPingTime(HostAndPort("host2"), - MemberState::RS_SECONDARY, - lastOpTimeFetched, - now(), - ReadPreference::Nearest)); -} - -// Test that we will not change sync sources due to the replSetSyncFrom command being run when -// the 'unsupportedSyncSource' startup parameter is set - that is, the parameter should always -// take priority. -TEST_F(ReevalSyncSourceTest, NoChangeDueToReplSetSyncFromWhenSyncSourceForcedByStartupParameter) { - RAIIServerParameterControllerForTest syncSourceParamGuard{"unsupportedSyncSource", - "host2:27017"}; - // Select a sync source. - auto syncSource = - getTopoCoord().chooseNewSyncSource(now()++, OpTime(), ReadPreference::Nearest); - ASSERT_EQ(syncSource, HostAndPort("host2:27017")); - - // Simulate calling replSetSyncFrom and selecting host3. - BSONObjBuilder response; - auto result = Status::OK(); - getTopoCoord().prepareSyncFromResponse(HostAndPort("host3:27017"), &response, &result); - - // Assert the command failed. - ASSERT_EQ(result.code(), ErrorCodes::IllegalOperation); - - // Reselect a sync source, and confirm it hasn't changed. - syncSource = getTopoCoord().chooseNewSyncSource(now()++, OpTime(), ReadPreference::Nearest); - ASSERT_EQ(syncSource, HostAndPort("host2:27017")); -} - -// Test that if a node is REMOVED but has 'unsupportedSyncSource' specified, we select no sync -// source. -TEST_F(ReevalSyncSourceTest, RemovedNodeSpecifiesSyncSourceStartupParameter) { - RAIIServerParameterControllerForTest syncSourceParamGuard{"unsupportedSyncSource", - "host2:27017"}; - // Remove ourselves from the config. - updateConfig(BSON("_id" - << "rs0" - << "version" << 2 << "members" - << BSON_ARRAY(BSON("_id" << 1 << "host" - << "host2:27017") - << BSON("_id" << 2 << "host" - << "host3:27017"))), - -1); - // Confirm we were actually removed. - ASSERT_EQUALS(MemberState::RS_REMOVED, getTopoCoord().getMemberState().s); - // Confirm we select no sync source. - auto syncSource = - getTopoCoord().chooseNewSyncSource(now()++, OpTime(), ReadPreference::Nearest); - ASSERT_EQUALS(syncSource, HostAndPort()); -} - -// Test that we crash if the 'unsupportedSyncSource' parameter specifies a node that is not in the -// replica set config. -DEATH_TEST_F(ReevalSyncSourceTest, CrashOnSyncSourceParameterNotInReplSet, "7785600") { - RAIIServerParameterControllerForTest syncSourceParamGuard{"unsupportedSyncSource", - "host4:27017"}; - auto syncSource = - getTopoCoord().chooseNewSyncSource(now()++, OpTime(), ReadPreference::Nearest); -} - -// Test that we crash if the 'unsupportedSyncSource' parameter specifies ourself as a node. -DEATH_TEST_F(ReevalSyncSourceTest, CrashOnSyncSourceParameterIsSelf, "7785601") { - RAIIServerParameterControllerForTest syncSourceParamGuard{"unsupportedSyncSource", - "host1:27017"}; - auto syncSource = - getTopoCoord().chooseNewSyncSource(now()++, OpTime(), ReadPreference::Nearest); -} - class HeartbeatResponseReconfigTestV1 : public TopoCoordTest { public: virtual void setUp() { diff --git a/src/mongo/db/repl/topology_version_observer_test.cpp b/src/mongo/db/repl/topology_version_observer_test.cpp index d7ee56b4778..6d7d54d57b8 100644 --- a/src/mongo/db/repl/topology_version_observer_test.cpp +++ b/src/mongo/db/repl/topology_version_observer_test.cpp @@ -42,9 +42,7 @@ #include "mongo/db/repl/replication_coordinator_impl.h" #include "mongo/db/repl/replication_coordinator_test_fixture.h" #include "mongo/db/repl/topology_version_observer.h" -#include "mongo/logv2/log.h" #include "mongo/unittest/barrier.h" -#include "mongo/unittest/log_test.h" #include "mongo/unittest/unittest.h" #include "mongo/util/assert_util.h" #include "mongo/util/clock_source.h" @@ -120,9 +118,6 @@ protected: const Milliseconds sleepTime = Milliseconds(100); std::unique_ptr<TopologyVersionObserver> observer; - - unittest::MinimumLoggedSeverityGuard severityGuard{logv2::LogComponent::kDefault, - logv2::LogSeverity::Debug(4)}; }; @@ -145,15 +140,11 @@ TEST_F(TopologyVersionObserverTest, UpdateCache) { auto electionTimeoutWhen = getReplCoord()->getElectionTimeout_forTest(); simulateSuccessfulV1ElectionWithoutExitingDrainMode(electionTimeoutWhen, opCtx.get()); - auto sleepCounter = 0; // Wait for the observer to update its cache while (observer->getCached()->getTopologyVersion()->getCounter() == cachedResponse->getTopologyVersion()->getCounter()) { sleepFor(sleepTime); - // Make sure the test doesn't wait here for longer than 15 seconds. - ASSERT_LTE(sleepCounter++, 150); } - LOGV2(9326401, "Observer topology incremented after successful election"); auto newResponse = observer->getCached(); ASSERT(newResponse && newResponse->getTopologyVersion()); diff --git a/src/mongo/db/repl/transaction_oplog_application.cpp b/src/mongo/db/repl/transaction_oplog_application.cpp index 20d8459a18b..af8fa188671 100644 --- a/src/mongo/db/repl/transaction_oplog_application.cpp +++ b/src/mongo/db/repl/transaction_oplog_application.cpp @@ -33,13 +33,11 @@ #include "mongo/db/repl/transaction_oplog_application.h" -#include "mongo/db/catalog/document_validation.h" #include "mongo/db/catalog_raii.h" #include "mongo/db/commands/txn_cmds_gen.h" -#include "mongo/db/concurrency/exception_util.h" +#include "mongo/db/concurrency/write_conflict_exception.h" #include "mongo/db/dbdirectclient.h" #include "mongo/db/index_builds_coordinator.h" -#include "mongo/db/op_observer.h" #include "mongo/db/repl/apply_ops.h" #include "mongo/db/repl/storage_interface_impl.h" #include "mongo/db/repl/timestamp_block.h" @@ -90,26 +88,9 @@ Status _applyOperationsForTransaction(OperationContext* opCtx, } } catch (const DBException& ex) { // Ignore NamespaceNotFound errors if we are in initial sync or recovering mode. - // During recovery we reconsutuct prepared transactions at the end after applying all - // the oplogs, so 'NamespaceNotFound' error shouldn't be hit whether it is a stable or - // unstable recovery. However we have some scenarios when this error should be skipped: - // 1- This code path can be called while applying commit oplog during unstable recovery - // when 'startupRecoveryForRestore' is set. - // 2- During selective backup: - // - During restore when 'recoverFromOplogAsStandalone' is set which is usually be - // done in a stable recovery mode. - // - After the restore finished as the standalone node started with the flag - // 'takeUnstableCheckpointOnShutdown' so after restarting the node as a replica - // set member it will go through unstable recovery. const bool ignoreException = ex.code() == ErrorCodes::NamespaceNotFound && (oplogApplicationMode == repl::OplogApplication::Mode::kInitialSync || - repl::OplogApplication::inRecovering(oplogApplicationMode)); - - if (ex.code() == ErrorCodes::NamespaceNotFound && - oplogApplicationMode == repl::OplogApplication::Mode::kStableRecovering) { - repl::OplogApplication::checkOnOplogFailureForRecovery( - opCtx, op.getNss(), redact(op.toBSONForLogging()), redact(ex)); - } + oplogApplicationMode == repl::OplogApplication::Mode::kRecovering); if (!ignoreException) { LOGV2_DEBUG( @@ -149,7 +130,7 @@ Status _applyTransactionFromOplogChain(OperationContext* opCtx, repl::OplogApplication::Mode mode, Timestamp commitTimestamp, Timestamp durableTimestamp) { - invariant(repl::OplogApplication::inRecovering(mode)); + invariant(mode == repl::OplogApplication::Mode::kRecovering); auto ops = readTransactionOperationsFromOplogChain(opCtx, entry, {}); @@ -208,8 +189,7 @@ Status applyCommitTransaction(OperationContext* opCtx, invariant(commitCommand.getCommitTimestamp()); switch (mode) { - case repl::OplogApplication::Mode::kUnstableRecovering: - case repl::OplogApplication::Mode::kStableRecovering: { + case repl::OplogApplication::Mode::kRecovering: { return _applyTransactionFromOplogChain(opCtx, entry, mode, @@ -256,8 +236,7 @@ Status applyAbortTransaction(OperationContext* opCtx, const OplogEntry& entry, repl::OplogApplication::Mode mode) { switch (mode) { - case repl::OplogApplication::Mode::kUnstableRecovering: - case repl::OplogApplication::Mode::kStableRecovering: { + case repl::OplogApplication::Mode::kRecovering: { // We don't put transactions into the prepare state until the end of recovery, // so there is no transaction to abort. return Status::OK(); @@ -409,7 +388,7 @@ Status _applyPrepareTransaction(OperationContext* opCtx, // The prepare time of the transaction is set explicitly below. auto ops = readTransactionOperationsFromOplogChain(opCtx, entry, {}); - if (repl::OplogApplication::inRecovering(mode) || + if (mode == repl::OplogApplication::Mode::kRecovering || mode == repl::OplogApplication::Mode::kInitialSync) { // We might replay a prepared transaction behind oldest timestamp. Note that since this is // scoped to the storage transaction, and readTransactionOperationsFromOplogChain implicitly @@ -486,7 +465,7 @@ Status _applyPrepareTransaction(OperationContext* opCtx, // Set this in case the application of any ops need to use the prepare timestamp of this // transaction. It should be cleared automatically when the transaction finishes. - if (repl::OplogApplication::inRecovering(mode) || + if (mode == repl::OplogApplication::Mode::kRecovering || mode == repl::OplogApplication::Mode::kInitialSync) { txnParticipant.setPrepareOpTimeForRecovery(opCtx, entry.getOpTime()); } @@ -519,11 +498,6 @@ Status _applyPrepareTransaction(OperationContext* opCtx, } txnParticipant.prepareTransaction(opCtx, entry.getOpTime()); - - auto opObserver = opCtx->getServiceContext()->getOpObserver(); - invariant(opObserver); - opObserver->onTransactionPrepareNonPrimary(opCtx, ops, entry.getOpTime()); - // Prepare transaction success. abortOnError.dismiss(); @@ -539,8 +513,6 @@ void _reconstructPreparedTransaction(OperationContext* opCtx, const OplogEntry& prepareEntry, repl::OplogApplication::Mode mode) { repl::UnreplicatedWritesBlock uwb(opCtx); - // The transaction may have been prepared originally with document validation bypassed. - DisableDocumentValidation validationDisabler(opCtx); // Snapshot transaction can never conflict with the PBWM lock. opCtx->lockState()->setShouldConflictWithSecondaryBatchApplication(false); @@ -571,8 +543,7 @@ Status applyPrepareTransaction(OperationContext* opCtx, const OplogEntry& entry, repl::OplogApplication::Mode mode) { switch (mode) { - case repl::OplogApplication::Mode::kUnstableRecovering: - case repl::OplogApplication::Mode::kStableRecovering: { + case repl::OplogApplication::Mode::kRecovering: { if (!serverGlobalParams.enableMajorityReadConcern) { LOGV2_ERROR( 21850, @@ -642,10 +613,7 @@ void reconstructPreparedTransactions(OperationContext* opCtx, repl::OplogApplica AlternativeClientRegion acr(newClient); const auto newOpCtx = cc().makeOperationContext(); - // Ignore interruptions while reconstructing prepared transactions, so that we do not - // fassert and crash due to interruptions inside this call. - newOpCtx->runWithoutInterruptionExceptAtGlobalShutdown( - [&] { _reconstructPreparedTransaction(newOpCtx.get(), prepareOplogEntry, mode); }); + _reconstructPreparedTransaction(newOpCtx.get(), prepareOplogEntry, mode); } } } |