diff options
| author | Lucas de Castro Borges <lucas@gnuabordo.com.br> | 2025-02-14 14:26:38 -0300 |
|---|---|---|
| committer | Lucas de Castro Borges <lucas@gnuabordo.com.br> | 2025-02-14 14:26:38 -0300 |
| commit | 294bc6ecabf14c09c9bc8644704921dcf97cb44e (patch) | |
| tree | 279b1e0bab53901a1647ac63c1c724f0f789a663 /src/mongo/db/free_mon | |
| parent | 70be7c27a251621187a1de533462ae2bb1e3bd39 (diff) | |
| parent | 1e917fd798aa25b7066d4b414b51184f13d5a092 (diff) | |
Update upstream source from tag 'upstream/6.0.10'debian/6.0.10-1
Update to upstream version '6.0.10'
with Debian dir 2d176fa254eee97b139f712fec5709641335a8c3
Diffstat (limited to 'src/mongo/db/free_mon')
30 files changed, 0 insertions, 7184 deletions
diff --git a/src/mongo/db/free_mon/README.md b/src/mongo/db/free_mon/README.md deleted file mode 100644 index 18f101a58d1..00000000000 --- a/src/mongo/db/free_mon/README.md +++ /dev/null @@ -1,130 +0,0 @@ -# Free Monitoring - -## Table of Contents - -- [Free Monitoring](#free-monitoring) - - [Table of Contents](#table-of-contents) - - [High Level Overview](#high-level-overview) - -## High Level Overview - -Free Monitoring is a way for MongoDB Community users to enable Cloud Monitoring on their database. -To use Free Monitoring, a customer must first register by issuing the command -`db.enableFreeMonitoring()`. - -The entire Free Monitoring subsystem is controlled by an object of type -[`FreeMonController`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.h#L53). -The `FreeMonController` lives as a decoration on the `ServiceContext`. The `FreeMonController` has a -two collections of collectors - the -[`_registrationCollectors`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.h#L197) -which collect data at registration time, and the -[`_metricCollectors`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.h#L200) -which collect data periodically. It also owns a -[`FreeMonProcessor`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.h#L304) -which under the hood contains a multi-producer priority queue, and a -[`FreeMonNetworkInterface`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_network.h#L40) -which is a way for the subsystem to send and recieve packets to the cloud endpoint. - -When the server first starts, if Free Monitoring is enabled (using a command line parameter -`enableFreeMonitoring`), the FreeMonController is initialized on server startup through the mongod -main function which calls -[`startFreeMonitoring`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_mongod.cpp#L310). -This function creates the -[`FreeMonNetworkInterface`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_mongod.cpp#L322), -initializes the controller, determines the Registration type, and calls -[`start`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_mongod.cpp#L346-L348) -on the controller. The -[`start`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.h#L59-L65) -function initializes the processor, creates a thread for it to run on, and performs registration if -the user has performed registration. - -If the user has not performed registration, the metrics collector begins collecting data. It stores -this data in a MetricsBuffer, capable of holding up to 10 data points. When the user performs -registration, a call to -[`registerServerStartup`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.cpp#L79-L84) -is made, placing a -[`FreeMonMessageWithPayload`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_message.h#L255) -object in the processor's queue. A `FreeMonMessageWithPayload` is an expanded subclass of -[`FreeMonMessage`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_message.h#L146), -which represents a message sent to the -[`FreeMonProcessor`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.h#L304) -actor to process and make a decision. For more reading on the actor model that the Free Monitoring -system is based on, see [here](https://en.wikipedia.org/wiki/Actor_model). A `FreeMonMessage` has -two significant properties, a type and a deadline. The type determines how the queue responds when -processing the message. The code for how the queue processes messages can be found -[here](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L156-L269). -The deadline determines the priority of the message in the queue. The deadline represents both the -waiting period the queue must take to process a message and a priority the queue uses to determine -the order in which messages are processed. For example, a message with a deadline of now can be -processed before something with a deadline of an hour from now. - -In the call to -[`registerServerStartup`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.cpp#L79-L84), -a message of type `RegisterServer` is sent to the queue. Included with the message is a payload of -the registration type that the server should perform. The queue processes this message and calls -[`doServerRegister`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L324) -with the message object. The function processes the `RegistrationType` and if it is -`RegisterOnStart`, sends a `RegisterCommand` message to the queue. If the `RegistrationType` is -something else, we try to determine whether we are a primary or secondary in a replica set and send -a message depending on the state of free monitoring in the set. The full logic with comments is -[here](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L330-L367), -but know that it may create a `RegisterCommand` message to the queue in certain cases. The function -[`doServerRegister`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L328) -also creates a message of type `MetricsCollect` in the queue. - -When the queue processes the -[`RegisterCommand`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L173-L175) -message, it calls -[`doCommandRegister`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L406) -which uses the -[`FreeMonNetworkInterface`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_mongod.cpp#L322) -to send a message over the wire to the cloud endpoint. It also writes the registration state -(`FreeMonRegistrationStatus::kPending`) and registration information out to disk, calling -[`writeState`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L299), -which invokes functions on the -[`FreeMonStorage`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_storage.h#L43) -class. The job of the `FreeMonStorage` class is to provide an interface for different functions to -interact with the storage subsystem. Whenever a change is made to the registration state - -registration completes or registration is cancelled because of an endpoint error - the processor -writes this information out to disk. - -The queue also processes the first `MetricsCollect` command around this time. The queue reads the -Metrics collect method and calls -[`doMetricsCollect`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L714-L726) -which fires the `_metrics` collectors to collect and stores the data in a -[`MetricsBuffer`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.h#L185-L225). -The buffer can hold 10 data points at a time, so if the data has not been synced to the cloud -endpoint by the time the 11th data point is collected, then the buffer will remove the last item -from the queue. The function -[`doMetricsCollect`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L724-L725) -creates another message of type `MetricsCollect` with a deadline of the specified -`_metricsGatherInterval` for collection. - -The way for a queue to trigger sending new metrics to the server is by sending a message of type -`MetricsSend`. This occurs on a few occasions - when registration information has been successfully -[sent to the cloud -endpoint](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L666), -when metrics information has [successfully been sent to the cloud -endpoint](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L865-L866), -and when metrics information has [failed to send to the -endpoint](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.cpp#L886-L888). -In the first case, a message is created with a deadline of now. In the second case the message is -sent with a deadline that is tracked by the -[`MetricsRetryCounter`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_processor.h#L153-L183) -object. The retry object is used to track any failures the processor encountered when sending -metrics; if enough failures have occured in a row, then the processor stops sending the metrics. In -the third case, the `MetricsRetryCounter` object is incremented to indicate failure. If it has not -exceeded the retry limit, then the message is again sent with a deadline tracked by the retry -object. - -The last notable part of `FreeMonitoring` is the -[`FreeMonOpObserver`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_op_observer.h#L40). -This `OpObserver` watches the namespace where the free monitoring registration information is stored -(the `admin` database and the `system.version` collection) and sends a message to the processor if -there is a change to the document for the registration information. For example, if someone updates -the document, the OpObserver calls -[`notifyOnUpsert`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.h#L133-L138) -in the controller, the controller queues a -[`NotifyOnUpsert`](https://github.com/mongodb/mongo/blob/r4.4.0/src/mongo/db/free_mon/free_mon_controller.cpp#L111) -command in the processor. When the processor reads that message, it reads the updated registration -state from disk and updates the free monitoring subsystem based on the new information. diff --git a/src/mongo/db/free_mon/SConscript b/src/mongo/db/free_mon/SConscript deleted file mode 100644 index c1fe6d83627..00000000000 --- a/src/mongo/db/free_mon/SConscript +++ /dev/null @@ -1,88 +0,0 @@ -# -*- mode: python -*- -Import("env") -Import("free_monitoring") - -env = env.Clone() - -fmEnv = env.Clone() -fmEnv.InjectThirdParty(libraries=['snappy']) - -fmEnv.Library( - target='free_mon', - source=[ - 'free_mon_processor.cpp', - 'free_mon_queue.cpp', - 'free_mon_op_observer.cpp', - 'free_mon_storage.cpp', - 'free_mon_controller.cpp', - 'free_mon_protocol.idl', - 'free_mon_commands.idl', - 'free_mon_storage.idl', - ], - LIBDEPS=[ - '$BUILD_DIR/mongo/base', - '$BUILD_DIR/mongo/db/concurrency/lock_manager', - '$BUILD_DIR/mongo/db/dbhelpers', - '$BUILD_DIR/mongo/db/ftdc/ftdc', - '$BUILD_DIR/mongo/idl/idl_parser', - '$BUILD_DIR/third_party/shim_snappy', - ], -) - -if free_monitoring == "on": - fmEnv.Library( - target='free_mon_mongod', - source=[ - 'free_mon_commands.cpp', - 'free_mon_mongod.cpp', - 'free_mon_mongod.idl', - 'free_mon_options.cpp', - 'free_mon_options.idl', - 'free_mon_status.cpp', - ], - LIBDEPS=[ - '$BUILD_DIR/mongo/db/commands/server_status', - '$BUILD_DIR/mongo/db/ftdc/ftdc_server', - '$BUILD_DIR/mongo/util/options_parser/options_parser', - 'free_mon', - ], - LIBDEPS_PRIVATE=[ - '$BUILD_DIR/mongo/client/clientdriver_network', - '$BUILD_DIR/mongo/util/concurrency/thread_pool', - '$BUILD_DIR/mongo/util/net/http_client', - ], - ) -else: - fmEnv.Library( - target='free_mon_mongod', - source=[ - 'free_mon_commands_stub.cpp', - 'free_mon_stub.cpp', - ], - LIBDEPS_PRIVATE=[ - '$BUILD_DIR/mongo/base', - '$BUILD_DIR/mongo/db/auth/auth', - '$BUILD_DIR/mongo/db/auth/authprivilege', - '$BUILD_DIR/mongo/db/commands', - 'free_mon', - ], - ) - - -fmEnv.CppUnitTest( - target='db_free_mon_test', - source=[ - 'free_mon_controller_test.cpp', - 'free_mon_queue_test.cpp', - 'free_mon_storage_test.cpp', - ], - LIBDEPS=[ - '$BUILD_DIR/mongo/db/auth/authmocks', - '$BUILD_DIR/mongo/db/repl/replmocks', - '$BUILD_DIR/mongo/db/repl/storage_interface_impl', - '$BUILD_DIR/mongo/db/service_context_d_test_fixture', - '$BUILD_DIR/mongo/executor/thread_pool_task_executor_test_fixture', - '$BUILD_DIR/mongo/util/clock_source_mock', - 'free_mon', - ], -) diff --git a/src/mongo/db/free_mon/free_mon_commands.cpp b/src/mongo/db/free_mon/free_mon_commands.cpp deleted file mode 100644 index 78cc3f3a420..00000000000 --- a/src/mongo/db/free_mon/free_mon_commands.cpp +++ /dev/null @@ -1,167 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/platform/basic.h" - -#include "mongo/db/auth/authorization_session.h" -#include "mongo/db/commands.h" -#include "mongo/db/free_mon/free_mon_commands_gen.h" -#include "mongo/db/free_mon/free_mon_controller.h" -#include "mongo/db/free_mon/free_mon_options.h" -#include "mongo/db/free_mon/free_mon_storage.h" - -namespace mongo { - -namespace { - -const auto kRegisterSyncTimeout = Milliseconds{5000}; - -/** - * Indicates the current status of Free Monitoring. - */ -class GetFreeMonitoringStatusCommand : public BasicCommand { -public: - GetFreeMonitoringStatusCommand() : BasicCommand("getFreeMonitoringStatus") {} - - bool adminOnly() const override { - return true; - } - - AllowedOnSecondary secondaryAllowed(ServiceContext*) const final { - return AllowedOnSecondary::kAlways; - } - - bool supportsWriteConcern(const BSONObj& cmd) const final { - return false; - } - - std::string help() const final { - return "Indicates free monitoring status"; - } - - Status checkAuthForCommand(Client* client, - const std::string& dbname, - const BSONObj& cmdObj) const final { - if (!AuthorizationSession::get(client)->isAuthorizedForActionsOnResource( - ResourcePattern::forClusterResource(), ActionType::checkFreeMonitoringStatus)) { - return Status(ErrorCodes::Unauthorized, "Unauthorized"); - } - return Status::OK(); - } - - bool run(OperationContext* opCtx, - const std::string& dbname, - const BSONObj& cmdObj, - BSONObjBuilder& result) final { - // Command has no members, invoke the parser to confirm that. - IDLParserErrorContext ctx("getFreeMonitoringStatus"); - GetFreeMonitoringStatus::parse(ctx, cmdObj); - - if (globalFreeMonParams.freeMonitoringState == EnableCloudStateEnum::kOff) { - result.append("state", "disabled"); - return true; - } - - auto* controller = FreeMonController::get(opCtx->getServiceContext()); - if (!controller) { - result.append("state", "disabled"); - } else { - controller->getStatus(opCtx, &result); - } - return true; - } -} getFreeMonitoringStatusCommand; - -/** - * Enables or disables Free Monitoring service. - */ -class SetFreeMonitoringCommand : public BasicCommand { -public: - SetFreeMonitoringCommand() : BasicCommand("setFreeMonitoring") {} - - bool adminOnly() const override { - return true; - } - - AllowedOnSecondary secondaryAllowed(ServiceContext*) const final { - return AllowedOnSecondary::kNever; - } - - bool supportsWriteConcern(const BSONObj& cmd) const final { - return false; - } - - std::string help() const final { - return "enable or disable Free Monitoring"; - } - - Status checkAuthForCommand(Client* client, - const std::string& dbname, - const BSONObj& cmdObj) const final { - if (!AuthorizationSession::get(client)->isAuthorizedForActionsOnResource( - ResourcePattern::forClusterResource(), ActionType::setFreeMonitoring)) { - return Status(ErrorCodes::Unauthorized, "Unauthorized"); - } - return Status::OK(); - } - - bool run(OperationContext* opCtx, - const std::string& dbname, - const BSONObj& cmdObj, - BSONObjBuilder& result) final { - IDLParserErrorContext ctx("setFreeMonitoring"); - auto cmd = SetFreeMonitoring::parse(ctx, cmdObj); - - auto* controller = FreeMonController::get(opCtx->getServiceContext()); - if (!controller) { - // Pending operation. - uasserted(50840, - "Free Monitoring has been disabled via the command-line and/or config file"); - } - - boost::optional<Status> optStatus = boost::none; - if (cmd.getAction() == SetFreeMonActionEnum::enable) { - optStatus = controller->registerServerCommand(kRegisterSyncTimeout); - } else { - optStatus = controller->unregisterServerCommand(kRegisterSyncTimeout); - } - - if (optStatus) { - // Completed within timeout. - uassertStatusOK(*optStatus); - } else { - // Pending operation. - } - return true; - } - -} setFreeMonitoringCmd; - -} // namespace -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_commands.idl b/src/mongo/db/free_mon/free_mon_commands.idl deleted file mode 100644 index 2fddd12a2c3..00000000000 --- a/src/mongo/db/free_mon/free_mon_commands.idl +++ /dev/null @@ -1,61 +0,0 @@ -# Copyright (C) 2018-present MongoDB, Inc. -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the Server Side Public License, version 1, -# as published by MongoDB, Inc. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# Server Side Public License for more details. -# -# You should have received a copy of the Server Side Public License -# along with this program. If not, see -# <http://www.mongodb.com/licensing/server-side-public-license>. -# -# As a special exception, the copyright holders give permission to link the -# code of portions of this program with the OpenSSL library under certain -# conditions as described in each individual source file and distribute -# linked combinations including the program with the OpenSSL library. You -# must comply with the Server Side Public License in all respects for -# all of the code used other than as permitted herein. If you modify file(s) -# with this exception, you may extend this exception to your version of the -# file(s), but you are not obligated to do so. If you do not wish to do so, -# delete this exception statement from your version. If you delete this -# exception statement from all source files in the program, then also delete -# it in the license file. -# -global: - cpp_namespace: "mongo" - - -imports: - - "mongo/idl/basic_types.idl" - - -enums: - SetFreeMonAction: - description: "Action types" - type: string - values: - enable: "enable" - disable: "disable" - - -commands: - setFreeMonitoring: - description: "setFreeMonitoring Command" - command_name: setFreeMonitoring - namespace: ignored - api_version: "" - fields: - action: - description: "Action to take" - type: SetFreeMonAction - - getFreeMonitoringStatus: - description: "getFreeMonitoringStatus Command" - command_name: getFreeMonitoringStatus - namespace: ignored - api_version: "" - diff --git a/src/mongo/db/free_mon/free_mon_commands_stub.cpp b/src/mongo/db/free_mon/free_mon_commands_stub.cpp deleted file mode 100644 index 4b8494e5eea..00000000000 --- a/src/mongo/db/free_mon/free_mon_commands_stub.cpp +++ /dev/null @@ -1,92 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/platform/basic.h" - -#include "mongo/db/auth/authorization_session.h" -#include "mongo/db/commands.h" -#include "mongo/db/free_mon/free_mon_commands_gen.h" - -namespace mongo { - -namespace { - -/** - * Indicates the current status of Free Monitoring. - * - * Note that this file is only built when --enable-free-mon=off - * and it will return a static result of {status:"disabled"}. - */ -class GetFreeMonitoringStatusCommandStub : public BasicCommand { -public: - GetFreeMonitoringStatusCommandStub() : BasicCommand("getFreeMonitoringStatus") {} - - bool adminOnly() const override { - return true; - } - - AllowedOnSecondary secondaryAllowed(ServiceContext*) const final { - return AllowedOnSecondary::kAlways; - } - - bool supportsWriteConcern(const BSONObj& cmd) const final { - return false; - } - - std::string help() const final { - return "Indicates free monitoring status"; - } - - Status checkAuthForCommand(Client* client, - const std::string& dbname, - const BSONObj& cmdObj) const final { - if (!AuthorizationSession::get(client)->isAuthorizedForActionsOnResource( - ResourcePattern::forClusterResource(), ActionType::checkFreeMonitoringStatus)) { - return Status(ErrorCodes::Unauthorized, "Unauthorized"); - } - return Status::OK(); - } - - bool run(OperationContext* opCtx, - const std::string& dbname, - const BSONObj& cmdObj, - BSONObjBuilder& result) final { - // Command has no members, invoke the parser to confirm that. - IDLParserErrorContext ctx("getFreeMonitoringStatus"); - GetFreeMonitoringStatus::parse(ctx, cmdObj); - - result.append("state", "disabled"); - result.append("message", - "Free Monitoring support is not available in this build of MongoDB"); - return true; - } -} getFreeMonitoringStatusCommand; - -} // namespace -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_controller.cpp b/src/mongo/db/free_mon/free_mon_controller.cpp deleted file mode 100644 index 8c3565a8a11..00000000000 --- a/src/mongo/db/free_mon/free_mon_controller.cpp +++ /dev/null @@ -1,249 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kControl - -#include "mongo/platform/basic.h" - -#include "mongo/db/free_mon/free_mon_controller.h" - -#include "mongo/logv2/log.h" - -namespace mongo { - -namespace { - -const auto getFreeMonController = - ServiceContext::declareDecoration<synchronized_value<std::unique_ptr<FreeMonController>>>(); - -} // namespace - -FreeMonController* FreeMonController::get(ServiceContext* serviceContext) { - return getFreeMonController(serviceContext)->get(); -} - -void FreeMonController::init(ServiceContext* serviceContext, - std::unique_ptr<FreeMonController> controller) { - auto fmcContainer = getFreeMonController(serviceContext).synchronize(); - // Since FreeMonController::get() provides raw pointers, the FreeMonController can only be - // set once without producing memory leaks. - invariant(!fmcContainer->get()); - fmcContainer = std::move(controller); -} - - -FreeMonNetworkInterface::~FreeMonNetworkInterface() = default; - -void FreeMonController::addRegistrationCollector( - std::unique_ptr<FreeMonCollectorInterface> collector) { - { - stdx::lock_guard<Latch> lock(_mutex); - invariant(_state == State::kNotStarted); - - _registrationCollectors.add(std::move(collector)); - } -} - -void FreeMonController::addMetricsCollector(std::unique_ptr<FreeMonCollectorInterface> collector) { - { - stdx::lock_guard<Latch> lock(_mutex); - invariant(_state == State::kNotStarted); - - _metricCollectors.add(std::move(collector)); - } -} - -void FreeMonController::registerServerStartup(RegistrationType registrationType, - std::vector<std::string>& tags) { - _enqueue(FreeMonMessageWithPayload<FreeMonMessageType::RegisterServer>::createNow( - FreeMonMessageWithPayload<FreeMonMessageType::RegisterServer>::payload_type( - registrationType, tags))); -} - -boost::optional<Status> FreeMonController::registerServerCommand(Milliseconds timeout) { - auto msg = FreeMonRegisterCommandMessage::createNow({std::vector<std::string>(), boost::none}); - _enqueue(msg); - - if (timeout > Milliseconds::min()) { - return msg->wait_for(timeout); - } - - return Status::OK(); -} - -boost::optional<Status> FreeMonController::unregisterServerCommand(Milliseconds timeout) { - auto msg = - FreeMonWaitableMessageWithPayload<FreeMonMessageType::UnregisterCommand>::createNow(true); - _enqueue(msg); - - if (timeout > Milliseconds::min()) { - return msg->wait_for(timeout); - } - - return Status::OK(); -} - -void FreeMonController::notifyOnUpsert(const BSONObj& doc) { - invariant(doc.isOwned()); - _enqueue(FreeMonMessageWithPayload<FreeMonMessageType::NotifyOnUpsert>::createNow(doc)); -} - - -void FreeMonController::notifyOnDelete() { - _enqueue(FreeMonMessage::createNow(FreeMonMessageType::NotifyOnDelete)); -} - - -void FreeMonController::notifyOnTransitionToPrimary() { - _enqueue(FreeMonMessage::createNow(FreeMonMessageType::OnTransitionToPrimary)); -} - -void FreeMonController::notifyOnRollback() { - _enqueue(FreeMonMessage::createNow(FreeMonMessageType::NotifyOnRollback)); -} - -void FreeMonController::_enqueue(std::shared_ptr<FreeMonMessage> msg) { - { - stdx::lock_guard<Latch> lock(_mutex); - invariant(_state == State::kStarted); - } - - _processor->enqueue(std::move(msg)); -} - -void FreeMonController::start(RegistrationType registrationType, - std::vector<std::string>& tags, - Seconds gatherMetricsInterval) { - { - stdx::lock_guard<Latch> lock(_mutex); - - invariant(_state == State::kNotStarted); - } - - // Start the agent - _processor = std::make_shared<FreeMonProcessor>(_registrationCollectors, - _metricCollectors, - _network.get(), - _useCrankForTest, - gatherMetricsInterval); - - _thread = stdx::thread([this] { _processor->run(); }); - - { - stdx::lock_guard<Latch> lock(_mutex); - - invariant(_state == State::kNotStarted); - _state = State::kStarted; - } - - if (registrationType != RegistrationType::DoNotRegister) { - registerServerStartup(registrationType, tags); - } -} - -void FreeMonController::stop() { - // Stop the agent - LOGV2(20609, "Shutting down free monitoring"); - - { - stdx::lock_guard<Latch> lock(_mutex); - - bool started = (_state == State::kStarted); - - invariant(_state == State::kNotStarted || _state == State::kStarted); - - if (!started) { - _state = State::kDone; - return; - } - - _state = State::kStopRequested; - - // Tell the processor to stop - _processor->stop(); - } - - _thread.join(); - - { - stdx::lock_guard<Latch> lock(_mutex); - - _state = State::kDone; - } -} - -void FreeMonController::turnCrankForTest(size_t countMessagesToIgnore) { - { - stdx::lock_guard<Latch> lock(_mutex); - invariant(_state == State::kStarted); - } - - LOGV2(20610, "Turning Crank", "count"_attr = countMessagesToIgnore); - - _processor->turnCrankForTest(countMessagesToIgnore); -} - -void FreeMonController::deprioritizeFirstMessageForTest(FreeMonMessageType type) { - { - stdx::lock_guard<Latch> lock(_mutex); - invariant(_state == State::kStarted); - } - - LOGV2(5167901, "Deprioritize message", "type"_attr = static_cast<int>(type)); - - _processor->deprioritizeFirstMessageForTest(type); -} - -void FreeMonController::getStatus(OperationContext* opCtx, BSONObjBuilder* status) { - { - stdx::lock_guard<Latch> lock(_mutex); - - if (_state != State::kStarted) { - status->append("state", "disabled"); - return; - } - } - - _processor->getStatus(opCtx, status, FreeMonProcessor::FreeMonGetStatusEnum::kCommandStatus); -} - -void FreeMonController::getServerStatus(OperationContext* opCtx, BSONObjBuilder* status) { - { - stdx::lock_guard<Latch> lock(_mutex); - - if (_state != State::kStarted) { - status->append("state", "disabled"); - return; - } - } - - _processor->getStatus(opCtx, status, FreeMonProcessor::FreeMonGetStatusEnum::kServerStatus); -} - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_controller.h b/src/mongo/db/free_mon/free_mon_controller.h deleted file mode 100644 index f734231eeaa..00000000000 --- a/src/mongo/db/free_mon/free_mon_controller.h +++ /dev/null @@ -1,220 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include <memory> -#include <mutex> -#include <string> -#include <utility> -#include <vector> - -#include "mongo/base/status.h" -#include "mongo/db/client.h" -#include "mongo/db/free_mon/free_mon_message.h" -#include "mongo/db/free_mon/free_mon_network.h" -#include "mongo/db/free_mon/free_mon_processor.h" -#include "mongo/db/service_context.h" -#include "mongo/stdx/thread.h" -#include "mongo/util/duration.h" - -namespace mongo { - -/** - * Manages and control Free Monitoring. This is the entry point for non-free monitoring components - * into free-monitoring. - */ -class FreeMonController { -public: - explicit FreeMonController(std::unique_ptr<FreeMonNetworkInterface> network, - bool useCrankForTest = false) - : _network(std::move(network)), _useCrankForTest(useCrankForTest) {} - - /** - * Initializes free monitoring. - * Start free monitoring thread in the background. - */ - void start(RegistrationType registrationType, - std::vector<std::string>& tags, - Seconds gatherMetricsInterval); - - /** - * Stops free monitoring thread. - */ - void stop(); - - /** - * Turn the crank of the message queue by ignoring deadlines for N messages. - */ - void turnCrankForTest(size_t countMessagesToIgnore); - - /** - * Deproritize the first message to force interleavings of messages. - */ - void deprioritizeFirstMessageForTest(FreeMonMessageType type); - - /** - * Add a metric collector to collect on registration - */ - void addRegistrationCollector(std::unique_ptr<FreeMonCollectorInterface> collector); - - /** - * Add a metric collector to collect periodically - */ - void addMetricsCollector(std::unique_ptr<FreeMonCollectorInterface> collector); - - /** - * Get the FreeMonController from ServiceContext. - */ - static FreeMonController* get(ServiceContext* serviceContext); - - /** - * Initialize the FreeMonController decoration in the ServiceContext. - */ - static void init(ServiceContext* serviceContext, std::unique_ptr<FreeMonController> controller); - - /** - * Start registration of mongod with remote service. - * - * Only sends one remote registration at a time. - * Returns after timeout if registrations is not complete. Registration continues though. - */ - void registerServerStartup(RegistrationType registrationType, std::vector<std::string>& tags); - - /** - * Start registration of mongod with remote service. - * - * Only sends one remote registration at a time. - * Returns after timeout if registrations is not complete. Registration continues though. - * Update is synchronous with 10sec timeout - * kicks off register, and once register is done kicks off metrics upload - */ - boost::optional<Status> registerServerCommand(Milliseconds timeout); - - /** - * Stop registration of mongod with remote service. - * - * As with registerServerCommand() above, but undoes registration. - * On complettion of this command, no further metrics will be transmitted. - */ - boost::optional<Status> unregisterServerCommand(Milliseconds timeout); - - /** - * Populates an info blob for use by {getFreeMonitoringStatus: 1} - */ - void getStatus(OperationContext* opCtx, BSONObjBuilder* status); - - /** - * Populates an info blob for use by {serverStatus: 1} - */ - void getServerStatus(OperationContext* opCtx, BSONObjBuilder* status); - - /** - * Notify on upsert. - * - * Updates and inserts are treated as the same. - */ - void notifyOnUpsert(const BSONObj& doc); - - /** - * Notify on document delete or drop collection. - */ - void notifyOnDelete(); - - /** - * Notify that we local instance has become a primary. - */ - void notifyOnTransitionToPrimary(); - - /** - * Notify that storage has rolled back - */ - void notifyOnRollback(); - -private: - void _enqueue(std::shared_ptr<FreeMonMessage> msg); - -private: - /** - * Private enum to track state. - * - * +-----------------------------------------------------------+ - * | v - * +-------------+ +----------+ +----------------+ +-------+ - * | kNotStarted | --> | kStarted | --> | kStopRequested | --> | kDone | - * +-------------+ +----------+ +----------------+ +-------+ - */ - enum class State { - /** - * Initial state. Either start() or stop() can be called next. - */ - kNotStarted, - - /** - * start() has been called. stop() should be called next. - */ - kStarted, - - /** - * stop() has been called, and the background thread is in progress of shutting down - */ - kStopRequested, - - /** - * Controller has been stopped. - */ - kDone, - }; - - // Controller state - State _state{State::kNotStarted}; - - // Mutext to protect internal state - Mutex _mutex = MONGO_MAKE_LATCH("FreeMonController::_mutex"); - - // Set of registration collectors - FreeMonCollectorCollection _registrationCollectors; - - // Set of metric collectors - FreeMonCollectorCollection _metricCollectors; - - // Network interface - std::unique_ptr<FreeMonNetworkInterface> _network; - - // Background thead for agent - stdx::thread _thread; - - // Crank for test - bool _useCrankForTest; - - // Background agent - std::shared_ptr<FreeMonProcessor> _processor; -}; - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_controller_test.cpp b/src/mongo/db/free_mon/free_mon_controller_test.cpp deleted file mode 100644 index e074e73c7f0..00000000000 --- a/src/mongo/db/free_mon/free_mon_controller_test.cpp +++ /dev/null @@ -1,1675 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kControl - -#include "mongo/platform/basic.h" - -#include <boost/filesystem.hpp> -#include <future> -#include <iostream> -#include <memory> -#include <snappy.h> - -#include "mongo/db/free_mon/free_mon_controller.h" -#include "mongo/db/free_mon/free_mon_storage.h" - -#include "mongo/base/data_type_validated.h" -#include "mongo/bson/bson_validate.h" -#include "mongo/bson/bsonmisc.h" -#include "mongo/bson/bsonobjbuilder.h" -#include "mongo/db/client.h" -#include "mongo/db/free_mon/free_mon_op_observer.h" -#include "mongo/db/ftdc/collector.h" -#include "mongo/db/ftdc/config.h" -#include "mongo/db/ftdc/constants.h" -#include "mongo/db/ftdc/controller.h" -#include "mongo/db/ftdc/ftdc_test.h" -#include "mongo/db/jsobj.h" -#include "mongo/db/op_observer_noop.h" -#include "mongo/db/op_observer_registry.h" -#include "mongo/db/repl/replication_coordinator_mock.h" -#include "mongo/db/repl/storage_interface.h" -#include "mongo/db/repl/storage_interface_impl.h" -#include "mongo/db/service_context.h" -#include "mongo/db/service_context_d_test_fixture.h" -#include "mongo/executor/network_interface_mock.h" -#include "mongo/executor/thread_pool_task_executor_test_fixture.h" -#include "mongo/logv2/log.h" -#include "mongo/rpc/object_check.h" -#include "mongo/unittest/barrier.h" -#include "mongo/unittest/temp_dir.h" -#include "mongo/unittest/unittest.h" -#include "mongo/util/clock_source.h" -#include "mongo/util/hex.h" - - -namespace mongo { -namespace { - -auto makeRandom() { - auto seed = SecureRandom().nextInt64(); - LOGV2(24189, "PseudoRandom()", "seed"_attr = seed); - return PseudoRandom(seed); -} - -class FreeMonMetricsCollectorMock : public FreeMonCollectorInterface { -public: - ~FreeMonMetricsCollectorMock() { - // ASSERT_TRUE(_state == State::kStarted); - } - - void collect(OperationContext* opCtx, BSONObjBuilder& builder) final { - _state = State::kStarted; - - builder.append("mock", "some data"); - - { - stdx::lock_guard<Latch> lck(_mutex); - - ++_counter; - - if (_counter == _wait) { - _condvar.notify_all(); - } - } - } - - std::string name() const final { - return "mock"; - } - - void setSignalOnCount(int c) { - _wait = c; - } - - std::uint32_t count() { - stdx::lock_guard<Latch> lck(_mutex); - return _counter; - } - - void wait() { - stdx::unique_lock<Latch> lck(_mutex); - while (_counter < _wait) { - _condvar.wait(lck); - } - } - -private: - /** - * Private enum to ensure caller uses class correctly. - */ - enum class State { - kNotStarted, - kStarted, - }; - - // state - State _state{State::kNotStarted}; - - std::uint32_t _counter{0}; - - Mutex _mutex = MONGO_MAKE_LATCH("FreeMonMetricsCollectorMock::_mutex"); - stdx::condition_variable _condvar; - std::uint32_t _wait{0}; -}; - -BSONArray decompressMetrics(ConstDataRange cdr) { - std::string outBuffer; - snappy::Uncompress(cdr.data(), cdr.length(), &outBuffer); - - ConstDataRange raw(outBuffer.data(), outBuffer.data() + outBuffer.size()); - auto swObj = raw.readNoThrow<Validated<BSONObj>>(); - ASSERT_OK(swObj.getStatus()); - - return BSONArray(swObj.getValue().val["data"].Obj().getOwned()); -} - -/** - * Countdown latch that propagates a message. - */ -template <typename T> -class CountdownLatchResult { -public: - CountdownLatchResult(uint32_t count) : _count(count) {} - - /** - * Set the count of events to wait for. - */ - void reset(uint32_t count) { - stdx::lock_guard<Latch> lock(_mutex); - ASSERT_EQ(_count, 0UL); - ASSERT_GT(count, 0UL); - - _count = count; - _payload = T(); - } - - /** - * Set the payload and signal waiter. - */ - void set(T payload) { - stdx::lock_guard<Latch> lock(_mutex); - - if (_count > 0) { - --_count; - if (_count == 0) { - _payload = std::move(payload); - _condvar.notify_one(); - } - } - } - - /** - * Waits for duration until N events have occured. - * - * Returns boost::none on timeout. - */ - boost::optional<T> wait_for(Milliseconds duration) { - stdx::unique_lock<Latch> lock(_mutex); - - if (!_condvar.wait_for( - lock, duration.toSystemDuration(), [this]() { return _count == 0; })) { - return {}; - } - - return _payload; - } - -private: - // Condition variable to signal consumer - stdx::condition_variable _condvar; - - // Lock for condition variable and to protect state - Mutex _mutex = MONGO_MAKE_LATCH("CountdownLatchResult::_mutex"); - - // Count to wait fore - uint32_t _count; - - // Provided payload - T _payload; -}; - -class FreeMonNetworkInterfaceMock final : public FreeMonNetworkInterface { -public: - struct Options { - // If sync = true, then execute the callback immediately and the subsequent future chain - // This allows us to ensure the follow up functions to a network request are executed - // before anything else is processed by FreeMonProcessor - bool doSync{false}; - - // Faults to inject for registration - bool failRegisterHttp{false}; - bool invalidRegister{false}; - bool haltRegister{false}; - - // Faults to inject for metrics - bool haltMetrics{false}; - bool fail2MetricsUploads{false}; - bool permanentlyDeleteAfter3{false}; - - bool resendRegistrationAfter3{false}; - }; - - explicit FreeMonNetworkInterfaceMock(executor::ThreadPoolTaskExecutor* threadPool, - Options options) - : _threadPool(threadPool), _options(options), _countdownMetrics(0) {} - - Future<FreeMonRegistrationResponse> sendRegistrationAsync( - const FreeMonRegistrationRequest& req) final { - LOGV2(20611, "Sending Registration ..."); - - _registers.addAndFetch(1); - - auto pf = makePromiseFuture<FreeMonRegistrationResponse>(); - if (_options.doSync) { - pf.promise.setFrom(doRegister(req)); - } else { - auto swSchedule = _threadPool->scheduleWork( - [sharedPromise = std::move(pf.promise), req, this]( - const executor::TaskExecutor::CallbackArgs& cbArgs) mutable { - sharedPromise.setWith([&] { return doRegister(req); }); - }); - - ASSERT_OK(swSchedule.getStatus()); - } - - return std::move(pf.future); - } - - StatusWith<FreeMonRegistrationResponse> doRegister(const FreeMonRegistrationRequest& req) { - - if (_options.failRegisterHttp) { - return Status(ErrorCodes::FreeMonHttpTemporaryFailure, "Mock failure"); - } - - auto resp = FreeMonRegistrationResponse(); - resp.setVersion(1); - - if (_options.invalidRegister) { - resp.setVersion(42); - } - - resp.setId("regId123"); - - if (_options.haltRegister) { - resp.setHaltMetricsUploading(true); - } - - resp.setReportingInterval(1); - - return resp; - } - - - Future<FreeMonMetricsResponse> sendMetricsAsync(const FreeMonMetricsRequest& req) final { - LOGV2(20612, "Sending Metrics ..."); - - _metrics.addAndFetch(1); - - auto pf = makePromiseFuture<FreeMonMetricsResponse>(); - if (_options.doSync) { - pf.promise.setFrom(doMetrics(req)); - } else { - auto swSchedule = _threadPool->scheduleWork( - [sharedPromise = std::move(pf.promise), req, this]( - const executor::TaskExecutor::CallbackArgs& cbArgs) mutable { - sharedPromise.setWith([&] { return doMetrics(req); }); - }); - - ASSERT_OK(swSchedule.getStatus()); - } - - return std::move(pf.future); - } - - StatusWith<FreeMonMetricsResponse> doMetrics(const FreeMonMetricsRequest& req) { - auto cdr = req.getMetrics(); - - { - stdx::lock_guard<Latch> lock(_metricsLock); - auto metrics = decompressMetrics(cdr); - _lastMetrics = metrics; - _countdownMetrics.set(metrics); - } - - if (_options.fail2MetricsUploads && _metrics.loadRelaxed() < 3) { - return Status(ErrorCodes::FreeMonHttpTemporaryFailure, "Mock failure"); - } - - auto resp = FreeMonMetricsResponse(); - resp.setVersion(1); - resp.setReportingInterval(1); - - resp.setId("metricsId456"_sd); - - if (_options.haltMetrics) { - resp.setHaltMetricsUploading(true); - } - - if (_options.permanentlyDeleteAfter3 && _metrics.loadRelaxed() > 3) { - resp.setPermanentlyDelete(true); - } - - if (_options.resendRegistrationAfter3 && _metrics.loadRelaxed() == 3) { - resp.setResendRegistration(true); - } - - return resp; - } - - int32_t getRegistersCalls() const { - return _registers.load(); - } - - int32_t getMetricsCalls() const { - return _metrics.load(); - } - - boost::optional<BSONArray> waitMetricsCalls(uint32_t count, Milliseconds wait) { - _countdownMetrics.reset(count); - return _countdownMetrics.wait_for(wait); - } - - BSONArray getLastMetrics() { - stdx::lock_guard<Latch> lock(_metricsLock); - return _lastMetrics; - } - - -private: - AtomicWord<int> _registers; - AtomicWord<int> _metrics; - - executor::ThreadPoolTaskExecutor* _threadPool; - - Mutex _metricsLock = MONGO_MAKE_LATCH("FreeMonNetworkInterfaceMock::_metricsLock"); - BSONArray _lastMetrics; - - Options _options; - - CountdownLatchResult<BSONArray> _countdownMetrics; -}; - -class FreeMonControllerTest : public ServiceContextMongoDTest { - -protected: - void setUp() override; - void tearDown() override; - -protected: - /** - * Looks up the current ReplicationCoordinator. - * The result is cast to a ReplicationCoordinatorMock to provide access to test features. - */ - repl::ReplicationCoordinatorMock* _getReplCoord() const; - - ServiceContext::UniqueOperationContext _opCtx; - - executor::NetworkInterfaceMock* _mockNetwork{nullptr}; - - std::unique_ptr<executor::ThreadPoolTaskExecutor> _mockThreadPool; -}; - -void FreeMonControllerTest::setUp() { - ServiceContextMongoDTest::setUp(); - auto service = getServiceContext(); - - repl::ReplicationCoordinator::set(service, - std::make_unique<repl::ReplicationCoordinatorMock>(service)); - - // Set up a NetworkInterfaceMock. Note, unlike NetworkInterfaceASIO, which has its own pool of - // threads, tasks in the NetworkInterfaceMock must be carried out synchronously by the (single) - // thread the unit test is running on. - auto netForFixedTaskExecutor = std::make_unique<executor::NetworkInterfaceMock>(); - _mockNetwork = netForFixedTaskExecutor.get(); - - // Set up a ThreadPoolTaskExecutor. Note, for local tasks this TaskExecutor uses a - // ThreadPoolMock, and for remote tasks it uses the NetworkInterfaceMock created above. However, - // note that the ThreadPoolMock uses the NetworkInterfaceMock's threads to run tasks, which is - // again just the (single) thread the unit test is running on. Therefore, all tasks, local and - // remote, must be carried out synchronously by the test thread. - _mockThreadPool = makeThreadPoolTestExecutor(std::move(netForFixedTaskExecutor)); - - _mockThreadPool->startup(); - - _opCtx = cc().makeOperationContext(); - - //_storage = std::make_unique<repl::StorageInterfaceImpl>(); - repl::StorageInterface::set(service, std::make_unique<repl::StorageInterfaceImpl>()); - - // Transition to PRIMARY so that the server can accept writes. - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_PRIMARY)); - - repl::createOplog(_opCtx.get()); - - // Create collection with one document. - CollectionOptions collectionOptions; - collectionOptions.uuid = UUID::gen(); - - auto statusCC = repl::StorageInterface::get(service)->createCollection( - _opCtx.get(), NamespaceString("admin", "system.version"), collectionOptions); - ASSERT_OK(statusCC); -} - -void FreeMonControllerTest::tearDown() { - _opCtx = {}; - ServiceContextMongoDTest::tearDown(); -} - -repl::ReplicationCoordinatorMock* FreeMonControllerTest::_getReplCoord() const { - auto replCoord = repl::ReplicationCoordinator::get(_opCtx.get()); - ASSERT(replCoord) << "No ReplicationCoordinator installed"; - auto replCoordMock = dynamic_cast<repl::ReplicationCoordinatorMock*>(replCoord); - ASSERT(replCoordMock) << "Unexpected type for installed ReplicationCoordinator"; - return replCoordMock; -} - -#define ASSERT_RANGE(target, lower, upper) \ - { \ - auto __x = counter.getNextDuration(); \ - ASSERT_GTE(__x, target + lower); \ - ASSERT_LTE(__x, target + upper); \ - } - - -// Positive: Ensure deadlines sort properly -TEST(FreeMonRetryTest, TestRegistration) { - auto random = makeRandom(); - RegistrationRetryCounter counter(random); - counter.reset(); - - ASSERT_EQ(counter.getNextDuration(), Seconds(1)); - ASSERT_EQ(counter.getNextDuration(), Seconds(1)); - - for (int j = 0; j < 3; j++) { - // Fail requests - for (int i = 1; i <= 10; ++i) { - ASSERT_TRUE(counter.incrementError()); - - int64_t base = pow(2, i); - ASSERT_RANGE(Seconds(base), Seconds(2), Seconds(10)); - } - - ASSERT_TRUE(counter.incrementError()); - ASSERT_RANGE(Seconds(1024), Seconds(60), Seconds(120)); - ASSERT_TRUE(counter.incrementError()); - ASSERT_RANGE(Seconds(1024), Seconds(60), Seconds(120)); - - counter.reset(); - } - - // Validate max timeout - - auto characterizeJitter = [](Seconds jitter1, Seconds jitter2) { - static constexpr size_t kStage1Retries = 10; - static constexpr auto kTMax = Days{2}; - auto t = Seconds(0); - auto base = Seconds(1); - size_t i = 0; - for (; t < kTMax; ++i) { - if (i < kStage1Retries) { - base *= 2; - t += base + jitter1; - } else { - t += base + jitter2; - } - } - return i; - }; - // If jitter is small as possible, we'd expect trueMax increments before false. - const auto trueMax = characterizeJitter(Seconds{2}, Seconds{60}); - // If jitter is large as possible, we'd expect trueMin increments before false. - const auto trueMin = characterizeJitter(Seconds{9}, Seconds{119}); - - // LOGV2(20613, "trueMin:{trueMin}", "trueMin"_attr = trueMin); - // LOGV2(20614, "trueMax:{trueMax}", "trueMax"_attr = trueMax); - - for (int j = 0; j < 30; j++) { - // std::cout << "j: " << j << "\n"; - // Fail requests - size_t trueCount = 0; - while (counter.incrementError()) { - ++trueCount; - } - ASSERT_GTE(trueCount, trueMin); - ASSERT_LTE(trueCount, trueMax); - counter.reset(); - } -} - -// Positive: Ensure deadlines sort properly -TEST(FreeMonRetryTest, TestMetrics) { - auto random = makeRandom(); - MetricsRetryCounter counter(random); - counter.reset(); - - ASSERT_EQ(counter.getNextDuration(), Seconds(1)); - ASSERT_EQ(counter.getNextDuration(), Seconds(1)); - - int32_t minTime = 1; - for (int j = 0; j < 3; j++) { - // Fail requests - for (int i = 0; i <= 6; ++i) { - ASSERT_TRUE(counter.incrementError()); - - int64_t base = pow(2, i); - ASSERT_RANGE(Seconds(base), Seconds(minTime / 2), Seconds(minTime)); - } - - ASSERT_TRUE(counter.incrementError()); - ASSERT_RANGE(Seconds(64), Seconds(minTime / 2), Seconds(minTime)); - ASSERT_TRUE(counter.incrementError()); - ASSERT_RANGE(Seconds(64), Seconds(minTime / 2), Seconds(minTime)); - - counter.reset(); - } - - // Validate max timeout - static size_t expectation = [] { - // There's technically a jitter in the MetricsRetryCounter but its default - // magnitude rounds to 0, so we make an exact expectation. - size_t iters = 0; - static constexpr auto kDurationMax = Days{7}; - auto t = Seconds{0}; - auto base = Seconds{1}; - for (; t < kDurationMax; ++iters) { - if (iters < 6) - base *= 2; - t += base; - } - return iters; - }(); - - for (int j = 0; j < 30; j++) { - // Fail requests - int iters = 0; - while (counter.incrementError()) { - ++iters; - } - ASSERT_EQ(iters, expectation); - counter.reset(); - } -} - -// Positive: Ensure the response is validated correctly -TEST(FreeMonProcessorTest, TestRegistrationResponseValidation) { - ASSERT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // max reporting interval - ASSERT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 30 * 60 * 60 * 24LL)))); - - // Positive: version 2 - ASSERT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 2LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Positive: empty registration id string - ASSERT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" - << "" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: bad protocol version - ASSERT_NOT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 42LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: halt uploading - ASSERT_NOT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << true << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: large registartation id - ASSERT_NOT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" << std::string(5000, 'a') - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: large URL - ASSERT_NOT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" << std::string(5000, 'b') << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: large message - ASSERT_NOT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" << std::string(5000, 'c') << "reportingInterval" << 1LL)))); - - // Negative: too small a reporting interval - ASSERT_NOT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 0LL)))); - - // Negative: too large a reporting interval - ASSERT_NOT_OK(FreeMonProcessor::validateRegistrationResponse(FreeMonRegistrationResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << (60LL * 60 * 24 * 30 + 1LL))))); -} - - -// Positive: Ensure the response is validated correctly -TEST(FreeMonProcessorTest, TestMetricsResponseValidation) { - ASSERT_OK(FreeMonProcessor::validateMetricsResponse( - FreeMonMetricsResponse::parse(IDLParserErrorContext("foo"), - - BSON("version" << 1LL << "haltMetricsUploading" << false - << "permanentlyDelete" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Positive: Support version 2 - ASSERT_OK(FreeMonProcessor::validateMetricsResponse( - FreeMonMetricsResponse::parse(IDLParserErrorContext("foo"), - - BSON("version" << 2LL << "haltMetricsUploading" << false - << "permanentlyDelete" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Positive: Add resendRegistration - ASSERT_OK(FreeMonProcessor::validateMetricsResponse(FreeMonMetricsResponse::parse( - IDLParserErrorContext("foo"), - - BSON("version" << 2LL << "haltMetricsUploading" << false << "permanentlyDelete" << false - << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL << "resendRegistration" << true)))); - - - // Positive: max reporting interval - ASSERT_OK(FreeMonProcessor::validateMetricsResponse(FreeMonMetricsResponse::parse( - IDLParserErrorContext("foo"), - - BSON("version" << 1LL << "haltMetricsUploading" << false << "permanentlyDelete" << false - << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 60 * 60 * 24 * 30LL)))); - - // Negative: bad protocol version - ASSERT_NOT_OK(FreeMonProcessor::validateMetricsResponse( - FreeMonMetricsResponse::parse(IDLParserErrorContext("foo"), - BSON("version" << 42LL << "haltMetricsUploading" << false - << "permanentlyDelete" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: halt uploading - ASSERT_NOT_OK(FreeMonProcessor::validateMetricsResponse( - FreeMonMetricsResponse::parse(IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << true - << "permanentlyDelete" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: large registartation id - ASSERT_NOT_OK(FreeMonProcessor::validateMetricsResponse(FreeMonMetricsResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "permanentlyDelete" << false - << "id" << std::string(5000, 'a') << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: large URL - ASSERT_NOT_OK(FreeMonProcessor::validateMetricsResponse(FreeMonMetricsResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false - - << "permanentlyDelete" << false << "id" - << "mock123" - << "informationalURL" << std::string(5000, 'b') << "message" - << "msg456" - << "reportingInterval" << 1LL)))); - - // Negative: large message - ASSERT_NOT_OK(FreeMonProcessor::validateMetricsResponse(FreeMonMetricsResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "permanentlyDelete" << false - << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" << std::string(5000, 'c') << "reportingInterval" << 1LL)))); - - // Negative: too small a reporting interval - ASSERT_NOT_OK(FreeMonProcessor::validateMetricsResponse( - FreeMonMetricsResponse::parse(IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false - << "permanentlyDelete" << false << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << 0LL)))); - - // Negative: too large a reporting interval - ASSERT_NOT_OK(FreeMonProcessor::validateMetricsResponse(FreeMonMetricsResponse::parse( - IDLParserErrorContext("foo"), - BSON("version" << 1LL << "haltMetricsUploading" << false << "permanentlyDelete" << false - << "id" - << "mock123" - << "informationalURL" - << "http://www.example.com/123" - << "message" - << "msg456" - << "reportingInterval" << (60LL * 60 * 24 * 30 + 1LL))))); -} - -/** - * Fluent class that encapsulates how many turns of a crank is needed to do a particular operation. - * - * All commands take 1 turn except registerCommand and metricsSend since these have a HTTP send an - * HTTP receive. - */ -class Turner { -public: - Turner() = default; - - Turner& registerServer() { - return inc(1, 1); - } - - Turner& registerCommand(size_t count = 1) { - return inc(2, count); - } - - Turner& unRegisterCommand() { - return inc(1, 1); - } - - Turner& collect(size_t count = 1) { - return inc(1, count); - } - - Turner& metricsSend(size_t count = 1) { - return inc(2, count); - } - - Turner& onTransitionToPrimary() { - return inc(1, 1); - } - - Turner& notifyUpsert() { - return inc(1, 1); - } - - Turner& notifyDelete() { - return inc(1, 1); - } - - Turner& notifyOnRollback() { - return inc(1, 1); - } - - operator size_t() { - return _count; - } - -private: - Turner& inc(size_t perOperatioCost, size_t numberOfOperations) { - _count += (perOperatioCost * numberOfOperations); - return *this; - } - -private: - size_t _count; -}; - -/** - * Utility class to manage controller setup and lifecycle for testing. - */ -struct ControllerHolder { - ControllerHolder(executor::ThreadPoolTaskExecutor* pool, - FreeMonNetworkInterfaceMock::Options opts, - bool useCrankForTest = true) { - auto registerCollectorUnique = std::make_unique<FreeMonMetricsCollectorMock>(); - auto metricsCollectorUnique = std::make_unique<FreeMonMetricsCollectorMock>(); - - // If we want to manually turn the crank the queue, we must process the messages - // synchronously - if (useCrankForTest) { - opts.doSync = true; - } - - ASSERT_EQ(opts.doSync, useCrankForTest); - - auto networkUnique = - std::unique_ptr<FreeMonNetworkInterface>(new FreeMonNetworkInterfaceMock(pool, opts)); - network = static_cast<FreeMonNetworkInterfaceMock*>(networkUnique.get()); - controller = std::make_unique<FreeMonController>(std::move(networkUnique), useCrankForTest); - - registerCollector = registerCollectorUnique.get(); - metricsCollector = metricsCollectorUnique.get(); - - controller->addRegistrationCollector(std::move(registerCollectorUnique)); - controller->addMetricsCollector(std::move(metricsCollectorUnique)); - } - - ~ControllerHolder() { - controller->stop(); - } - - void start(RegistrationType registrationType) { - std::vector<std::string> tags; - controller->start(registrationType, tags, Seconds(1)); - } - - - FreeMonController* operator->() { - return controller.get(); - } - - FreeMonMetricsCollectorMock* registerCollector; - FreeMonMetricsCollectorMock* metricsCollector; - FreeMonNetworkInterfaceMock* network; - - std::unique_ptr<FreeMonController> controller; -}; - -// Positive: Test Register works -TEST_F(FreeMonControllerTest, TestRegister) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - controller.start(RegistrationType::DoNotRegister); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerCommand()); - - ASSERT_TRUE(!FreeMonStorage::read(_opCtx.get()).get().getRegistrationId().empty()); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 0UL); -} - -// Negatve: Test Register times out if network stack drops messages -TEST_F(FreeMonControllerTest, TestRegisterTimeout) { - - FreeMonNetworkInterfaceMock::Options opts; - opts.failRegisterHttp = true; - - ControllerHolder controller(_mockThreadPool.get(), opts); - - controller.start(RegistrationType::DoNotRegister); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - controller->turnCrankForTest(Turner().registerCommand(2)); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::pending); - ASSERT_GTE(controller.network->getRegistersCalls(), 2); - ASSERT_GTE(controller.registerCollector->count(), 2UL); -} - -// Negatve: Test Register fails if the registration is wrong -TEST_F(FreeMonControllerTest, TestRegisterFail) { - - FreeMonNetworkInterfaceMock::Options opts; - opts.invalidRegister = true; - ControllerHolder controller(_mockThreadPool.get(), opts); - - controller.start(RegistrationType::DoNotRegister); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - controller->turnCrankForTest(Turner().registerCommand(1)); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::disabled); - ASSERT_EQ(controller.network->getRegistersCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); -} - -// Positive: Ensure registration halts -TEST_F(FreeMonControllerTest, TestRegisterHalts) { - - FreeMonNetworkInterfaceMock::Options opts; - opts.haltRegister = true; - ControllerHolder controller(_mockThreadPool.get(), opts); - - controller.start(RegistrationType::DoNotRegister); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - controller->turnCrankForTest(Turner().registerCommand()); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::disabled); - ASSERT_EQ(controller.network->getRegistersCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); -} - -// Positive: Test Metrics works on server register -TEST_F(FreeMonControllerTest, TestMetrics) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - controller.start(RegistrationType::RegisterOnStart); - - controller->turnCrankForTest( - Turner().registerServer().registerCommand().collect(2).metricsSend()); - - ASSERT_TRUE(!FreeMonStorage::read(_opCtx.get()).get().getRegistrationId().empty()); - - ASSERT_GTE(controller.network->getRegistersCalls(), 1); - ASSERT_GTE(controller.network->getMetricsCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 1UL); -} - - -// Positive: Test Metrics is collected but no registration happens on empty storage -TEST_F(FreeMonControllerTest, TestMetricsWithEmptyStorage) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - controller->turnCrankForTest(Turner().registerServer().collect(4)); - - ASSERT_GTE(controller.network->getRegistersCalls(), 0); - ASSERT_GTE(controller.network->getMetricsCalls(), 0); - - ASSERT_EQ(controller.registerCollector->count(), 0UL); - ASSERT_GTE(controller.metricsCollector->count(), 4UL); -} - -FreeMonStorageState initStorage(StorageStateEnum e) { - FreeMonStorageState storage; - storage.setVersion(1UL); - - storage.setRegistrationId("Foo"); - storage.setState(e); - storage.setInformationalURL("http://www.example.com"); - storage.setMessage("Hello World"); - storage.setUserReminder(""); - return storage; -} - -// Positive: Test Metrics is collected and implicit registration happens when storage is initialized -TEST_F(FreeMonControllerTest, TestMetricsWithEnabledStorage) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - controller->turnCrankForTest( - Turner().registerServer().registerCommand().collect(2).metricsSend()); - - ASSERT_TRUE(!FreeMonStorage::read(_opCtx.get()).get().getRegistrationId().empty()); - - ASSERT_GTE(controller.network->getRegistersCalls(), 1); - ASSERT_GTE(controller.network->getMetricsCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 1UL); -} - -// Positive: Test Metrics is collected but no registration happens on disabled storage -TEST_F(FreeMonControllerTest, TestMetricsWithDisabledStorage) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::disabled)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - controller->turnCrankForTest(Turner().registerServer().collect(4)); - - ASSERT_GTE(controller.network->getRegistersCalls(), 0); - ASSERT_GTE(controller.network->getMetricsCalls(), 0); - - ASSERT_EQ(controller.registerCollector->count(), 0UL); - ASSERT_GTE(controller.metricsCollector->count(), 4UL); -} - - -// Positive: Test Metrics is collected but no registration happens on disabled storage until user -// registers -TEST_F(FreeMonControllerTest, TestMetricsWithDisabledStorageThenRegister) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::disabled)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - controller->turnCrankForTest(Turner().registerServer().metricsSend().collect(4)); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerCommand().metricsSend().collect(2).metricsSend()); - - ASSERT_GTE(controller.network->getRegistersCalls(), 1); - ASSERT_GTE(controller.network->getMetricsCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 4UL + 2UL); -} - -// Positive: Test Metrics is collected but no registration happens, then register, then Unregister, -// and finally register again -TEST_F(FreeMonControllerTest, TestMetricsWithDisabledStorageThenRegisterAndReregister) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::disabled)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - controller->turnCrankForTest(Turner().registerServer().metricsSend().collect(4)); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerCommand().collect(2).metricsSend()); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get())->getState() == StorageStateEnum::enabled); - - optionalStatus = controller->unregisterServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().unRegisterCommand().collect(3)); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get())->getState() == StorageStateEnum::disabled); - - optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerCommand().metricsSend().collect(2).metricsSend()); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get())->getState() == StorageStateEnum::enabled); - - ASSERT_GTE(controller.network->getRegistersCalls(), 2); - ASSERT_GTE(controller.network->getMetricsCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 2UL); - ASSERT_GTE(controller.metricsCollector->count(), 4UL + 3UL + 2UL); -} - -// Positive: Test DeRegister cancels a register that is in the middle of retrying -TEST_F(FreeMonControllerTest, TestMetricsUnregisterCancelsRegister) { - FreeMonNetworkInterfaceMock::Options opts; - opts.failRegisterHttp = true; - ControllerHolder controller(_mockThreadPool.get(), opts); - - controller.start(RegistrationType::DoNotRegister); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - controller->turnCrankForTest(Turner().registerCommand(2)); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::pending); - - ASSERT_GTE(controller.network->getRegistersCalls(), 2); - ASSERT_GTE(controller.registerCollector->count(), 2UL); - - optionalStatus = controller->unregisterServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().unRegisterCommand()); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::disabled); - - ASSERT_GTE(controller.network->getRegistersCalls(), 2); - ASSERT_GTE(controller.registerCollector->count(), 2UL); -} - -// Positive: Test Metrics halts -TEST_F(FreeMonControllerTest, TestMetricsHalt) { - FreeMonNetworkInterfaceMock::Options opts; - opts.haltMetrics = true; - ControllerHolder controller(_mockThreadPool.get(), opts); - - controller.start(RegistrationType::RegisterOnStart); - - controller->turnCrankForTest( - Turner().registerServer().registerCommand().metricsSend().collect(4).metricsSend()); - - ASSERT_TRUE(!FreeMonStorage::read(_opCtx.get()).get().getRegistrationId().empty()); - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::disabled); - - ASSERT_GTE(controller.network->getRegistersCalls(), 1); - ASSERT_GTE(controller.network->getMetricsCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 4UL); -} - - -// Positive: Test Metrics permanently deletes if requested -TEST_F(FreeMonControllerTest, TestMetricsPermanentlyDelete) { - FreeMonNetworkInterfaceMock::Options opts; - opts.permanentlyDeleteAfter3 = true; - ControllerHolder controller(_mockThreadPool.get(), opts); - - controller.start(RegistrationType::RegisterOnStart); - - controller->turnCrankForTest( - Turner().registerServer().registerCommand().collect(5).metricsSend(4)); - - ASSERT_FALSE(FreeMonStorage::read(_opCtx.get()).is_initialized()); - - ASSERT_GTE(controller.network->getRegistersCalls(), 1); - ASSERT_GTE(controller.network->getMetricsCalls(), 3); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 3UL); -} - -// Positive: ensure registration id rotates -TEST_F(FreeMonControllerTest, TestRegistrationIdRotatesAfterRegistration) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - controller->turnCrankForTest(Turner().registerServer().registerCommand().collect(2)); - - // Ensure registration rotated the id - ASSERT_EQ(FreeMonStorage::read(_opCtx.get())->getRegistrationId(), "regId123"); - - controller->turnCrankForTest(Turner().metricsSend().collect()); - - // Ensure metrics rotated the id - ASSERT_EQ(FreeMonStorage::read(_opCtx.get())->getRegistrationId(), "metricsId456"); - - ASSERT_GTE(controller.network->getRegistersCalls(), 1); - ASSERT_GTE(controller.network->getMetricsCalls(), 1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 1UL); -} - -// Positive: ensure pre-registration metrics batching occurs -// Positive: ensure we only get two metrics each time -TEST_F(FreeMonControllerTest, TestPreRegistrationMetricBatching) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().collect(4)); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerCommand().metricsSend()); - - // Ensure we sent all the metrics batched before registration - ASSERT_EQ(controller.network->getLastMetrics().nFields(), 4); - - controller->turnCrankForTest(Turner().metricsSend().collect(1)); - - // Ensure we only send 2 metrics in the normal happy case - ASSERT_EQ(controller.network->getLastMetrics().nFields(), 2); -} - -// Positive: resend registration in metrics response -TEST_F(FreeMonControllerTest, TestResendRegistration) { - FreeMonNetworkInterfaceMock::Options opts; - opts.resendRegistrationAfter3 = true; - - ControllerHolder controller(_mockThreadPool.get(), opts); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerServer().registerCommand().collect(2)); - - ASSERT_TRUE(!FreeMonStorage::read(_opCtx.get()).get().getRegistrationId().empty()); - - controller->turnCrankForTest( - Turner().metricsSend(3).collect(3).registerCommand().metricsSend(1)); - - ASSERT_EQ(controller.registerCollector->count(), 2UL); - ASSERT_GTE(controller.metricsCollector->count(), 4UL); -} - -#if 0 -// Negative: Test metrics buffers on failure, and retries and ensure 2 metrics occurs after a blip -// of an error -// Note: this test operates in real-time because it needs to test multiple retries matched with -// metrics collection. -TEST_F(FreeMonControllerTest, TestMetricBatchingOnErrorRealtime) { - FreeMonNetworkInterfaceMock::Options opts; - opts.fail2MetricsUploads = true; - ControllerHolder controller(_mockThreadPool.get(), opts, false); - - controller.start(RegistrationType::RegisterOnStart); - - // Ensure the second upload sends 1 samples - ASSERT_TRUE(controller.network->waitMetricsCalls(2, Seconds(5)).is_initialized()); - ASSERT_EQ(controller.network->getLastMetrics().nFields(), 2); - - // Ensure the third upload sends 3 samples because first failed - ASSERT_TRUE(controller.network->waitMetricsCalls(1, Seconds(5)).is_initialized()); - ASSERT_EQ(controller.network->getLastMetrics().nFields(), 4); - - // Ensure the fourth upload sends 2 samples - ASSERT_TRUE(controller.network->waitMetricsCalls(1, Seconds(5)).is_initialized()); - ASSERT_EQ(controller.network->getLastMetrics().nFields(), 2); -} -#endif - -class FreeMonControllerRSTest : public FreeMonControllerTest { -private: - void setUp() final; - void tearDown() final; -}; - -void FreeMonControllerRSTest::setUp() { - FreeMonControllerTest::setUp(); - auto service = getServiceContext(); - - // Set up an OpObserver to exercise repl integration - auto opObserver = std::make_unique<FreeMonOpObserver>(); - auto opObserverRegistry = dynamic_cast<OpObserverRegistry*>(service->getOpObserver()); - opObserverRegistry->addObserver(std::move(opObserver)); -} - -void FreeMonControllerRSTest::tearDown() { - FreeMonControllerTest::tearDown(); -} - -// Positive: Transition to primary -TEST_F(FreeMonControllerRSTest, TransitionToPrimary) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - // Now become a secondary, then primary, and see what happens when we become primary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_PRIMARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().collect(2)); - - controller->notifyOnTransitionToPrimary(); - - controller->turnCrankForTest(Turner().onTransitionToPrimary().registerCommand()); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).is_initialized()); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 2UL); -} - -// Positive: Test metrics works on secondary -TEST_F(FreeMonControllerRSTest, StartupOnSecondary) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - - // Now become a secondary, then primary, and see what happens when we become primary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().registerCommand().collect()); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).is_initialized()); - - // Validate the new registration id was not written - ASSERT_EQ(FreeMonStorage::read(_opCtx.get())->getRegistrationId(), "Foo"); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 1UL); -} - -// Positive: Test registration occurs on replicated insert from primary -TEST_F(FreeMonControllerRSTest, SecondaryStartOnInsert) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().collect(2)); - - controller->notifyOnUpsert(initStorage(StorageStateEnum::enabled).toBSON()); - - controller->turnCrankForTest(Turner().notifyUpsert().registerCommand().collect()); - - ASSERT_FALSE(FreeMonStorage::read(_opCtx.get()).is_initialized()); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 2UL); -} - -// Positive: Test registration occurs on replicated update from primary -TEST_F(FreeMonControllerRSTest, SecondaryStartOnUpdate) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::pending)); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().collect(2)); - - controller->notifyOnUpsert(initStorage(StorageStateEnum::enabled).toBSON()); - - controller->turnCrankForTest(Turner().notifyUpsert().registerCommand().collect()); - - // Since there is no local write, it remains pending - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::pending); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 2UL); -} - -// Positive: Test Metrics works on secondary after opObserver de-register -TEST_F(FreeMonControllerRSTest, SecondaryStopOnDeRegister) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().registerCommand().collect(1)); - - ASSERT_EQ(controller.metricsCollector->count(), 1UL); - - controller->notifyOnUpsert(initStorage(StorageStateEnum::disabled).toBSON()); - - controller->turnCrankForTest(Turner().notifyUpsert().collect().metricsSend()); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).is_initialized()); - - // Since there is no local write, it remains enabled - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::enabled); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_EQ(controller.metricsCollector->count(), 2UL); -} - -// Negative: Tricky: Primary becomes secondary during registration -TEST_F(FreeMonControllerRSTest, StepdownDuringRegistration) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerServer() + 1); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::pending); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - // Finish registration - controller->turnCrankForTest(1); - controller->turnCrankForTest(Turner().metricsSend().collect(2)); - - // Registration cannot write back to the local store so remain in pending - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::pending); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_EQ(controller.metricsCollector->count(), 2UL); -} - -// Negative: Tricky: Primary becomes secondary during metrics send -TEST_F(FreeMonControllerRSTest, StepdownDuringMetricsSend) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - auto optionalStatus = controller->registerServerCommand(Milliseconds::min()); - ASSERT(optionalStatus); - ASSERT_OK(*optionalStatus); - - controller->turnCrankForTest(Turner().registerServer().registerCommand().collect()); - - // Finish registration - controller->turnCrankForTest(Turner().collect(1) + 1); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - // Finish send - controller->turnCrankForTest(1); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_EQ(controller.metricsCollector->count(), 2UL); -} - -// Positive: Test Metrics works on secondary after opObserver delete of document -TEST_F(FreeMonControllerRSTest, SecondaryStopOnDocumentDrop) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().registerCommand().collect(1)); - - ASSERT_EQ(controller.metricsCollector->count(), 1UL); - - controller->notifyOnDelete(); - - // There is a race condition where sometimes metrics send sneaks in - controller->turnCrankForTest(Turner().notifyDelete().collect(3)); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).is_initialized()); - - // Since there is no local write, it remains enabled - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::enabled); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_GTE(controller.metricsCollector->count(), 2UL); -} - - -// Positive: Test Metrics works on secondary after opObserver delete of document between metrics -// send and metrics async complete -TEST_F(FreeMonControllerRSTest, SecondaryStopOnDocumentDropDuringCollect) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().registerCommand().collect(1)); - - ASSERT_EQ(controller.metricsCollector->count(), 1UL); - - // Crank the metrics send but not the complete - controller->turnCrankForTest(Turner().collect(1)); - - controller->notifyOnDelete(); - - // Move the notify delete above the async metrics complete - controller->deprioritizeFirstMessageForTest(FreeMonMessageType::AsyncMetricsComplete); - - // There is a race condition where sometimes metrics send sneaks in - // Crank the notifyDelete and the async metrics complete. - controller->turnCrankForTest(Turner().notifyDelete().collect(1)); - - controller->turnCrankForTest(Turner().metricsSend().collect(2)); - - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).is_initialized()); - - // Since there is no local write, it remains enabled - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::enabled); - - BSONObjBuilder builder; - controller->getServerStatus(_opCtx.get(), &builder); - auto obj = builder.obj(); - ASSERT_BSONOBJ_EQ(BSON("state" - << "undecided"), - obj); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_EQ(controller.metricsCollector->count(), 5UL); -} - - -// Negative: Test nice shutdown on bad update -TEST_F(FreeMonControllerRSTest, SecondaryStartOnBadUpdate) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest( - Turner().registerServer().registerCommand().metricsSend().collect(2)); - - controller->notifyOnUpsert(BSON("version" << 2LL)); - - controller->turnCrankForTest(Turner().notifyUpsert()); - - // Since there is no local write, it remains enabled - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::enabled); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_EQ(controller.metricsCollector->count(), 2UL); -} - -// Positive: On rollback, start registration if needed -TEST_F(FreeMonControllerRSTest, SecondaryRollbackStopMetrics) { - ControllerHolder controller(_mockThreadPool.get(), FreeMonNetworkInterfaceMock::Options()); - - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::disabled)); - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller.start(RegistrationType::RegisterAfterOnTransitionToPrimary); - - controller->turnCrankForTest(Turner().registerServer().collect(2)); - - ASSERT_EQ(controller.metricsCollector->count(), 2UL); - - // Simulate a rollback by writing out of band - // Cheat a little by flipping to primary to allow the write to succeed - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_PRIMARY)); - FreeMonStorage::replace(_opCtx.get(), initStorage(StorageStateEnum::enabled)); - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - controller->notifyOnRollback(); - - controller->turnCrankForTest( - Turner().notifyOnRollback().registerCommand().metricsSend().collect(2).metricsSend()); - - // Since there is no local write, it remains enabled - ASSERT_TRUE(FreeMonStorage::read(_opCtx.get()).get().getState() == StorageStateEnum::enabled); - - ASSERT_EQ(controller.registerCollector->count(), 1UL); - ASSERT_EQ(controller.metricsCollector->count(), 4UL); -} - -// TODO: tricky - OnUpser - disable - OnDelete - make sure registration halts -// TODO: tricky - OnDelete - make sure registration halts - -// TODO: Integration: Tricky - secondary as marked via command line - enableCloudFreeMOnitorig = -// false but a primary replicates a change to enable it - -// TODO: test SSL??? - - -// TODO: Positive: ensure optional fields are rotated - -} // namespace -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_message.h b/src/mongo/db/free_mon/free_mon_message.h deleted file mode 100644 index 2f8bab042c6..00000000000 --- a/src/mongo/db/free_mon/free_mon_message.h +++ /dev/null @@ -1,413 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include <condition_variable> -#include <vector> - -#include "mongo/db/free_mon/free_mon_protocol_gen.h" -#include "mongo/platform/mutex.h" -#include "mongo/stdx/condition_variable.h" -#include "mongo/util/duration.h" -#include "mongo/util/time_support.h" - -namespace mongo { - -/** - * Message types for free monitoring. - * - * Some are generated internally by FreeMonProcessor to handle async HTTP requests. - */ -enum class FreeMonMessageType { - /** - * Register server from command-line/config. - */ - RegisterServer, - - /** - * Register server from server command. - */ - RegisterCommand, - - /** - * Internal: Generated when an async registration HTTP request completes succesfully. - */ - AsyncRegisterComplete, - - /** - * Internal: Generated when an async registration HTTP request completes with an error. - */ - AsyncRegisterFail, - - /** - * Unregister server from server command. - */ - UnregisterCommand, - - /** - * Internal: Collect metrics and buffer them in-memory - */ - MetricsCollect, - - /** - * Internal: Send metrics to the cloud endpoint by beginning an async HTTP request. - */ - MetricsSend, - - /** - * Internal: Generated when an async metrics HTTP request completes succesfully. - */ - AsyncMetricsComplete, - - /** - * Internal: Generated when an async metrics HTTP request completes with an error. - */ - AsyncMetricsFail, - - /** - * Notify that the node has been made a primary replica. - */ - OnTransitionToPrimary, - - /** - * Notify that storage has received an insert or update. - */ - NotifyOnUpsert, - - /** - * Notify that storage has received a delete or drop collection. - */ - NotifyOnDelete, - - /** - * Notify that storage has been rolled back. - */ - NotifyOnRollback, -}; - -/** - * Supported types of registration that occur on server startup. - */ -enum class RegistrationType { - /** - * Do not register on start because it was not configured via commandline/config file. - */ - DoNotRegister, - - /** - * Register immediately on start since we are a standalone. - */ - RegisterOnStart, - - /** - * Register after transition to becoming primary because we are in a replica set, - * and Free Monitoring has been explicitly enabled. - */ - RegisterAfterOnTransitionToPrimary, - - /** - * As above, but only if we have been runtime enabled. - */ - RegisterAfterOnTransitionToPrimaryIfEnabled, -}; - -/** - * Message class that encapsulate a message to the FreeMonMessageProcessor - * - * Has a type and a deadline for when to start processing the message. - */ -class FreeMonMessage { -public: - virtual ~FreeMonMessage(); - - /** - * Create a message that should processed immediately. - */ - static std::shared_ptr<FreeMonMessage> createNow(FreeMonMessageType type) { - return std::make_shared<FreeMonMessage>(type, Date_t()); - } - - /** - * Create a message that should processed after the specified deadline. - */ - static std::shared_ptr<FreeMonMessage> createWithDeadline(FreeMonMessageType type, - Date_t deadline) { - return std::make_shared<FreeMonMessage>(type, deadline); - } - - FreeMonMessage(const FreeMonMessage&) = delete; - FreeMonMessage(FreeMonMessage&&) = default; - - /** - * Get the type of message. - */ - FreeMonMessageType getType() const { - return _type; - } - - /** - * Get the deadline for the message. - */ - Date_t getDeadline() const { - return _deadline; - } - - /** - * Get the unique message id for FIFO ordering messages with the same deadline. - */ - uint64_t getId() const { - return _id; - } - - /** - * Set the unique message id. - */ - void setId(uint64_t id) { - _id = id; - } - -public: - FreeMonMessage(FreeMonMessageType type, Date_t deadline) : _type(type), _deadline(deadline) {} - -private: - // Type of message - FreeMonMessageType _type; - - // Deadline for when to process message - Date_t _deadline; - - // Process-wide unique message id to ensure messages with the same deadlines are processed in - // FIFO order. - uint64_t _id{0}; -}; - - -/** - * Most messages have a simple payload, and this template ensures we create type-safe messages for - * each message type without copy-pasting repeatedly. - */ -template <FreeMonMessageType typeT> -struct FreeMonPayloadForMessage { - using payload_type = void; -}; - -template <> -struct FreeMonPayloadForMessage<FreeMonMessageType::AsyncRegisterComplete> { - using payload_type = FreeMonRegistrationResponse; -}; - -template <> -struct FreeMonPayloadForMessage<FreeMonMessageType::RegisterServer> { - using payload_type = std::pair<RegistrationType, std::vector<std::string>>; -}; - -template <> -struct FreeMonPayloadForMessage<FreeMonMessageType::AsyncRegisterFail> { - using payload_type = Status; -}; - -template <> -struct FreeMonPayloadForMessage<FreeMonMessageType::AsyncMetricsComplete> { - using payload_type = FreeMonMetricsResponse; -}; - -template <> -struct FreeMonPayloadForMessage<FreeMonMessageType::AsyncMetricsFail> { - using payload_type = Status; -}; - -template <> -struct FreeMonPayloadForMessage<FreeMonMessageType::NotifyOnUpsert> { - using payload_type = BSONObj; -}; - -/** - * Message with a generic payload based on the type of message. - */ -template <FreeMonMessageType typeT> -class FreeMonMessageWithPayload : public FreeMonMessage { -public: - using payload_type = typename FreeMonPayloadForMessage<typeT>::payload_type; - - /** - * Create a message that should processed immediately. - */ - static std::shared_ptr<FreeMonMessageWithPayload> createNow(payload_type t) { - return std::make_shared<FreeMonMessageWithPayload>(t, Date_t{}); - } - - /** - * Get message payload. - */ - const payload_type& getPayload() const { - return _t; - } - -public: - FreeMonMessageWithPayload(payload_type t, Date_t deadline) - : FreeMonMessage(typeT, deadline), _t(std::move(t)) {} - -private: - // Message payload - payload_type _t; -}; - -/** - * Single-shot class that encapsulates a Status and allows a caller to wait for a time. - * - * Basically, a single producer, single consumer queue with one event. - */ -class WaitableResult { -public: - WaitableResult() : _status(Status::OK()) {} - - /** - * Set Status and signal waiter. - */ - void set(Status status) { - stdx::lock_guard<Latch> lock(_mutex); - - invariant(!_set); - if (!_set) { - _set = true; - _status = std::move(status); - _condvar.notify_one(); - } - } - - /** - * Waits for duration until status has been set. - * - * Returns boost::none on timeout. - */ - boost::optional<Status> wait_for(Milliseconds duration) { - stdx::unique_lock<Latch> lock(_mutex); - - if (!_condvar.wait_for(lock, duration.toSystemDuration(), [this]() { return _set; })) { - return {}; - } - - return _status; - } - -private: - // Condition variable to signal consumer - stdx::condition_variable _condvar; - - // Lock for condition variable and to protect state - Mutex _mutex = MONGO_MAKE_LATCH("WaitableResult::_mutex"); - - // Indicates whether _status has been set - bool _set{false}; - - // Provided status - Status _status; -}; - -/** - * For the messages that the caller needs to wait on, this provides a mechanism to wait on messages - * to be processed. - */ -template <FreeMonMessageType typeT> -struct FreeMonWaitablePayloadForMessage { - using payload_type = void; -}; - -template <> -struct FreeMonWaitablePayloadForMessage<FreeMonMessageType::RegisterCommand> { - using payload_type = std::pair<std::vector<std::string>, boost::optional<std::string>>; -}; - -template <> -struct FreeMonWaitablePayloadForMessage<FreeMonMessageType::UnregisterCommand> { - // The parameter is unused but most not be void. - using payload_type = bool; -}; - -/** - * Message with a generic payload based on the type of message. - */ -template <FreeMonMessageType typeT> -class FreeMonWaitableMessageWithPayload : public FreeMonMessage { -public: - using payload_type = typename FreeMonWaitablePayloadForMessage<typeT>::payload_type; - - /** - * Create a message that should processed immediately. - */ - static std::shared_ptr<FreeMonWaitableMessageWithPayload> createNow(payload_type t) { - return std::make_shared<FreeMonWaitableMessageWithPayload>(t, Date_t()); - } - - /** - * Create a message that should processed immediately. - */ - static std::shared_ptr<FreeMonWaitableMessageWithPayload> createWithDeadline(payload_type t, - Date_t deadline) { - return std::make_shared<FreeMonWaitableMessageWithPayload>(t, deadline); - } - /** - * Get message payload. - */ - const payload_type& getPayload() const { - return _t; - } - - /** - * Set Status and signal waiter. - */ - void setStatus(Status status) { - _waitable.set(std::move(status)); - } - - /** - * Waits for duration until status has been set. - * - * Returns boost::none on timeout. - */ - boost::optional<Status> wait_for(Milliseconds duration) { - return _waitable.wait_for(duration); - } - -public: - FreeMonWaitableMessageWithPayload(payload_type t, Date_t deadline) - : FreeMonMessage(typeT, deadline), _t(std::move(t)) {} - -private: - // Message payload - payload_type _t; - - // WaitaleResult to notify caller - WaitableResult _waitable{}; -}; - -using FreeMonRegisterCommandMessage = - FreeMonWaitableMessageWithPayload<FreeMonMessageType::RegisterCommand>; -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_mongod.cpp b/src/mongo/db/free_mon/free_mon_mongod.cpp deleted file mode 100644 index eda9f8487ee..00000000000 --- a/src/mongo/db/free_mon/free_mon_mongod.cpp +++ /dev/null @@ -1,373 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kControl - -#include "mongo/platform/basic.h" - -#include "mongo/db/free_mon/free_mon_mongod.h" - -#include <mutex> -#include <snappy.h> -#include <string> - -#include "mongo/base/data_type_validated.h" -#include "mongo/base/error_codes.h" -#include "mongo/base/status.h" -#include "mongo/bson/bsonelement.h" -#include "mongo/bson/bsonmisc.h" -#include "mongo/bson/bsonobj.h" -#include "mongo/bson/bsonobjbuilder.h" -#include "mongo/bson/bsontypes.h" -#include "mongo/db/db_raii.h" -#include "mongo/db/free_mon/free_mon_controller.h" -#include "mongo/db/free_mon/free_mon_message.h" -#include "mongo/db/free_mon/free_mon_mongod_gen.h" -#include "mongo/db/free_mon/free_mon_network.h" -#include "mongo/db/free_mon/free_mon_op_observer.h" -#include "mongo/db/free_mon/free_mon_options.h" -#include "mongo/db/free_mon/free_mon_protocol_gen.h" -#include "mongo/db/free_mon/free_mon_storage.h" -#include "mongo/db/ftdc/ftdc_server.h" -#include "mongo/db/operation_context.h" -#include "mongo/db/repl/replication_coordinator.h" -#include "mongo/db/service_context.h" -#include "mongo/executor/network_interface_factory.h" -#include "mongo/executor/thread_pool_task_executor.h" -#include "mongo/rpc/object_check.h" -#include "mongo/util/assert_util.h" -#include "mongo/util/concurrency/thread_pool.h" -#include "mongo/util/future.h" -#include "mongo/util/net/http_client.h" -#include "mongo/util/testing_proctor.h" - -namespace mongo { - -namespace { - -constexpr Seconds kDefaultMetricsGatherInterval(60); - -auto makeTaskExecutor(ServiceContext* /*serviceContext*/) { - ThreadPool::Options tpOptions; - tpOptions.poolName = "FreeMonHTTP"; - tpOptions.maxThreads = 2; - tpOptions.onCreateThread = [](const std::string& threadName) { - Client::initThread(threadName.c_str()); - }; - return std::make_unique<executor::ThreadPoolTaskExecutor>( - std::make_unique<ThreadPool>(tpOptions), executor::makeNetworkInterface("FreeMonNet")); -} - -class FreeMonNetworkHttp final : public FreeMonNetworkInterface { -public: - explicit FreeMonNetworkHttp(ServiceContext* serviceContext) { - _executor = makeTaskExecutor(serviceContext); - _executor->startup(); - _client = HttpClient::create(); - _client->allowInsecureHTTP(TestingProctor::instance().isEnabled()); - _client->setHeaders({"Content-Type: application/octet-stream", - "Accept: application/octet-stream", - "Expect:"}); - } - - Future<FreeMonRegistrationResponse> sendRegistrationAsync( - const FreeMonRegistrationRequest& req) override { - BSONObj reqObj = req.toBSON(); - auto data = std::make_shared<std::vector<std::uint8_t>>( - reqObj.objdata(), reqObj.objdata() + reqObj.objsize()); - - return post("/register", data).then([](DataBuilder&& blob) { - if (!blob.size()) { - uasserted(ErrorCodes::FreeMonHttpTemporaryFailure, "Empty response received"); - } - - auto blobSize = blob.size(); - auto blobData = blob.release(); - ConstDataRange cdr(blobData.get(), blobSize); - BSONObj respObj = cdr.read<Validated<BSONObj>>(); - - auto resp = - FreeMonRegistrationResponse::parse(IDLParserErrorContext("response"), respObj); - - return resp; - }); - } - - Future<FreeMonMetricsResponse> sendMetricsAsync(const FreeMonMetricsRequest& req) override { - BSONObj reqObj = req.toBSON(); - auto data = std::make_shared<std::vector<std::uint8_t>>( - reqObj.objdata(), reqObj.objdata() + reqObj.objsize()); - - return post("/metrics", data).then([](DataBuilder&& blob) { - if (!blob.size()) { - uasserted(ErrorCodes::FreeMonHttpTemporaryFailure, "Empty response received"); - } - - auto blobSize = blob.size(); - auto blobData = blob.release(); - ConstDataRange cdr(blobData.get(), blobSize); - - BSONObj respObj = cdr.read<Validated<BSONObj>>(); - - auto resp = FreeMonMetricsResponse::parse(IDLParserErrorContext("response"), respObj); - - return resp; - }); - } - -private: - Future<DataBuilder> post(StringData path, - std::shared_ptr<std::vector<std::uint8_t>> data) const { - auto pf = makePromiseFuture<DataBuilder>(); - std::string url(FreeMonEndpointURL + path.toString()); - - auto status = _executor->scheduleWork( - [promise = std::move(pf.promise), url = std::move(url), data = std::move(data), this]( - const executor::TaskExecutor::CallbackArgs& cbArgs) mutable { - ConstDataRange cdr(data->data(), data->size()); - try { - auto result = this->_client->post(url, cdr); - promise.emplaceValue(std::move(result)); - } catch (...) { - promise.setError(exceptionToStatus()); - } - }); - - uassertStatusOK(status); - return std::move(pf.future); - } - -private: - std::unique_ptr<HttpClient> _client; - std::unique_ptr<executor::ThreadPoolTaskExecutor> _executor; -}; - -/** - * Collect the mms-automation state document from local.clustermanager during registration. - */ -class FreeMonLocalClusterManagerCollector : public FreeMonCollectorInterface { -public: - std::string name() const final { - return "clustermanager"; - } - - void collect(OperationContext* opCtx, BSONObjBuilder& builder) { - auto optionalObj = FreeMonStorage::readClusterManagerState(opCtx); - if (optionalObj.is_initialized()) { - builder.appendElements(optionalObj.get()); - } - } -}; - -/** - * Get the "storageEngine" section of "serverStatus" during registration. - */ -class FreeMonLocalStorageEngineStatusCollector : public FTDCSimpleInternalCommandCollector { -public: - FreeMonLocalStorageEngineStatusCollector() - : FTDCSimpleInternalCommandCollector( - "serverStatus", - "serverStatus", - "", - // Try to filter server status to make it cheaper to collect. Harmless if we gather - // extra - BSON("serverStatus" << 1 << "storageEngine" << true << "extra_info" << false - << "opLatencies" << false << "opcountersRepl" << false - << "opcounters" << false << "transactions" << false - << "connections" << false << "network" << false << "tcMalloc" - << false << "network" << false << "wiredTiger" << false - << "sharding" << false << "metrics" << false)) {} - - std::string name() const final { - return "storageEngine"; - } - - void collect(OperationContext* opCtx, BSONObjBuilder& builder) { - BSONObjBuilder localBuilder; - - FTDCSimpleInternalCommandCollector::collect(opCtx, localBuilder); - - BSONObj obj = localBuilder.obj(); - - builder.appendElements(obj["storageEngine"].Obj()); - } -}; - -/** - * Collect the UUIDs associated with the named collections (if available). - */ -class FreeMonNamespaceUUIDCollector : public FreeMonCollectorInterface { -public: - FreeMonNamespaceUUIDCollector(std::set<NamespaceString> namespaces) - : _namespaces(std::move(namespaces)) {} - - std::string name() const final { - return "uuid"; - } - - void collect(OperationContext* opCtx, BSONObjBuilder& builder) { - auto catalog = CollectionCatalog::get(opCtx); - for (auto& nss : _namespaces) { - auto optUUID = catalog->lookupUUIDByNSS(opCtx, nss); - if (optUUID) { - builder << nss.toString() << optUUID.get(); - } - } - } - -private: - std::set<NamespaceString> _namespaces; -}; - -} // namespace - -Status onValidateFreeMonEndpointURL(StringData str) { - // Check for http, not https here because testEnabled may not be set yet - if (!str.startsWith("http"_sd) != 0) { - return Status(ErrorCodes::BadValue, - "cloudFreeMonitoringEndpointURL only supports http:// URLs"); - } - - return Status::OK(); -} - -void registerCollectors(FreeMonController* controller) { - // These are collected only at registration - // - // CmdBuildInfo - controller->addRegistrationCollector(std::make_unique<FTDCSimpleInternalCommandCollector>( - "buildInfo", "buildInfo", "", BSON("buildInfo" << 1))); - - // HostInfoCmd - controller->addRegistrationCollector(std::make_unique<FTDCSimpleInternalCommandCollector>( - "hostInfo", "hostInfo", "", BSON("hostInfo" << 1))); - - // Add storageEngine section from serverStatus - controller->addRegistrationCollector( - std::make_unique<FreeMonLocalStorageEngineStatusCollector>()); - - // Gather one document from local.clustermanager - controller->addRegistrationCollector(std::make_unique<FreeMonLocalClusterManagerCollector>()); - - // These are periodically for metrics upload - // - controller->addMetricsCollector(std::make_unique<FTDCSimpleInternalCommandCollector>( - "getDiagnosticData", "diagnosticData", "", BSON("getDiagnosticData" << 1))); - - // These are collected at registration and as metrics periodically - // - if (repl::ReplicationCoordinator::get(getGlobalServiceContext())->getReplicationMode() != - repl::ReplicationCoordinator::modeNone) { - // CmdReplSetGetConfig - controller->addRegistrationCollector(std::make_unique<FTDCSimpleInternalCommandCollector>( - "replSetGetConfig", "replSetGetConfig", "", BSON("replSetGetConfig" << 1))); - - controller->addMetricsCollector(std::make_unique<FTDCSimpleInternalCommandCollector>( - "replSetGetConfig", "replSetGetConfig", "", BSON("replSetGetConfig" << 1))); - - // Collect UUID for certain collections. - std::set<NamespaceString> namespaces({NamespaceString("local.oplog.rs")}); - controller->addRegistrationCollector( - std::make_unique<FreeMonNamespaceUUIDCollector>(namespaces)); - controller->addMetricsCollector( - std::make_unique<FreeMonNamespaceUUIDCollector>(namespaces)); - } - - controller->addRegistrationCollector(std::make_unique<FTDCSimpleInternalCommandCollector>( - "isMaster", "isMaster", "", BSON("isMaster" << 1))); - - controller->addMetricsCollector(std::make_unique<FTDCSimpleInternalCommandCollector>( - "isMaster", "isMaster", "", BSON("isMaster" << 1))); -} - -void startFreeMonitoring(ServiceContext* serviceContext) { - if (globalFreeMonParams.freeMonitoringState == EnableCloudStateEnum::kOff) { - return; - } - - if (!TestingProctor::instance().isEnabled()) { - uassert(50774, - "ExportedFreeMonEndpointURL only supports https:// URLs", - FreeMonEndpointURL.compare(0, 5, "https") == 0); - } - - auto network = std::unique_ptr<FreeMonNetworkInterface>(new FreeMonNetworkHttp(serviceContext)); - - auto controller = std::make_unique<FreeMonController>(std::move(network)); - - auto controllerPtr = controller.get(); - - registerCollectors(controller.get()); - - // Install the new controller - FreeMonController::init(getGlobalServiceContext(), std::move(controller)); - - RegistrationType registrationType = RegistrationType::DoNotRegister; - if (globalFreeMonParams.freeMonitoringState == EnableCloudStateEnum::kOn) { - // If replication is enabled, we may need to register on becoming primary - if (repl::ReplicationCoordinator::get(getGlobalServiceContext())->getReplicationMode() != - repl::ReplicationCoordinator::modeNone) { - registrationType = RegistrationType::RegisterAfterOnTransitionToPrimary; - } else { - registrationType = RegistrationType::RegisterOnStart; - } - } else if (globalFreeMonParams.freeMonitoringState == EnableCloudStateEnum::kRuntime) { - registrationType = RegistrationType::RegisterAfterOnTransitionToPrimaryIfEnabled; - } - - controllerPtr->start(registrationType, - globalFreeMonParams.freeMonitoringTags, - Seconds(kDefaultMetricsGatherInterval)); -} - -void stopFreeMonitoring() { - if (globalFreeMonParams.freeMonitoringState == EnableCloudStateEnum::kOff) { - return; - } - - auto controller = FreeMonController::get(getGlobalServiceContext()); - - if (controller != nullptr) { - controller->stop(); - } -} - -void notifyFreeMonitoringOnTransitionToPrimary() { - auto controller = FreeMonController::get(getGlobalServiceContext()); - - if (controller != nullptr) { - controller->notifyOnTransitionToPrimary(); - } -} - -void setupFreeMonitoringOpObserver(OpObserverRegistry* registry) { - registry->addObserver(std::make_unique<FreeMonOpObserver>()); -} - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_mongod.h b/src/mongo/db/free_mon/free_mon_mongod.h deleted file mode 100644 index 41676ebf3e1..00000000000 --- a/src/mongo/db/free_mon/free_mon_mongod.h +++ /dev/null @@ -1,65 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include <string> - -#include "mongo/db/op_observer_registry.h" -#include "mongo/db/service_context.h" - -namespace mongo { - -/** - * Start Free Monitoring - * Starts 1 thread. - */ -void startFreeMonitoring(ServiceContext* serviceContext); - -/** - * Stop Free Monitoring - */ -void stopFreeMonitoring(); - -/** - * Notify free monitoring about a replica set member becoming primary - */ -void notifyFreeMonitoringOnTransitionToPrimary(); - -/** - * Setup Free Monitoring OpObserver. - * - * Called before free monitoring is started. - */ -void setupFreeMonitoringOpObserver(OpObserverRegistry* registry); - -Status onValidateFreeMonEndpointURL(StringData str); - - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_mongod.idl b/src/mongo/db/free_mon/free_mon_mongod.idl deleted file mode 100644 index b3b34dafb29..00000000000 --- a/src/mongo/db/free_mon/free_mon_mongod.idl +++ /dev/null @@ -1,47 +0,0 @@ -# Copyright (C) 2018-present MongoDB, Inc. -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the Server Side Public License, version 1, -# as published by MongoDB, Inc. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# Server Side Public License for more details. -# -# You should have received a copy of the Server Side Public License -# along with this program. If not, see -# <http://www.mongodb.com/licensing/server-side-public-license>. -# -# As a special exception, the copyright holders give permission to link the -# code of portions of this program with the OpenSSL library under certain -# conditions as described in each individual source file and distribute -# linked combinations including the program with the OpenSSL library. You -# must comply with the Server Side Public License in all respects for -# all of the code used other than as permitted herein. If you modify file(s) -# with this exception, you may extend this exception to your version of the -# file(s), but you are not obligated to do so. If you do not wish to do so, -# delete this exception statement from your version. If you delete this -# exception statement from all source files in the program, then also delete -# it in the license file. -# - -global: - cpp_namespace: "mongo" - cpp_includes: - - "mongo/db/free_mon/free_mon_mongod.h" - -imports: - - "mongo/idl/basic_types.idl" - -server_parameters: - - cloudFreeMonitoringEndpointURL: - description: "Suppress logging of warnings when non-SSL connections are accepted in preferSSL mode" - set_at: startup - default: "https://cloud.mongodb.com/freemonitoring/mongo" - cpp_vartype: std::string - cpp_varname: FreeMonEndpointURL - validator: - callback: "onValidateFreeMonEndpointURL" - diff --git a/src/mongo/db/free_mon/free_mon_network.h b/src/mongo/db/free_mon/free_mon_network.h deleted file mode 100644 index 2bf1ff66b91..00000000000 --- a/src/mongo/db/free_mon/free_mon_network.h +++ /dev/null @@ -1,59 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include "mongo/db/free_mon/free_mon_protocol_gen.h" -#include "mongo/util/future.h" - -namespace mongo { - -/** - * Makes HTTPS calls to cloud endpoint. - */ -class FreeMonNetworkInterface { -public: - virtual ~FreeMonNetworkInterface(); - - /** - * POSTs FreeMonRegistrationRequest to endpoint. - * - * Returns a FreeMonRegistrationResponse or throws an error on non-HTTP 200. - */ - virtual Future<FreeMonRegistrationResponse> sendRegistrationAsync( - const FreeMonRegistrationRequest& req) = 0; - - /** - * POSTs FreeMonMetricsRequest to endpoint. - * - * Returns a FreeMonMetricsResponse or throws an error on non-HTTP 200. - */ - virtual Future<FreeMonMetricsResponse> sendMetricsAsync(const FreeMonMetricsRequest& req) = 0; -}; -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_op_observer.cpp b/src/mongo/db/free_mon/free_mon_op_observer.cpp deleted file mode 100644 index 1a0785c1df8..00000000000 --- a/src/mongo/db/free_mon/free_mon_op_observer.cpp +++ /dev/null @@ -1,170 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/platform/basic.h" - -#include "mongo/db/free_mon/free_mon_op_observer.h" - -#include "mongo/db/free_mon/free_mon_controller.h" -#include "mongo/db/free_mon/free_mon_storage.h" -#include "mongo/db/operation_context.h" - -namespace mongo { -namespace { - -bool isStandaloneOrPrimary(OperationContext* opCtx) { - auto replCoord = repl::ReplicationCoordinator::get(opCtx); - const bool isReplSet = - replCoord->getReplicationMode() == repl::ReplicationCoordinator::modeReplSet; - return !isReplSet || - (repl::ReplicationCoordinator::get(opCtx)->getMemberState() == - repl::MemberState::RS_PRIMARY); -} - -const auto getFreeMonDeleteState = OperationContext::declareDecoration<bool>(); - -} // namespace - -FreeMonOpObserver::FreeMonOpObserver() = default; - -FreeMonOpObserver::~FreeMonOpObserver() = default; - -repl::OpTime FreeMonOpObserver::onDropCollection(OperationContext* opCtx, - const NamespaceString& collectionName, - const UUID& uuid, - std::uint64_t numRecords, - const CollectionDropType dropType) { - if (collectionName == NamespaceString::kServerConfigurationNamespace) { - auto controller = FreeMonController::get(opCtx->getServiceContext()); - - if (controller != nullptr) { - controller->notifyOnDelete(); - } - } - - return {}; -} - -void FreeMonOpObserver::onInserts(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - std::vector<InsertStatement>::const_iterator begin, - std::vector<InsertStatement>::const_iterator end, - bool fromMigrate) { - if (nss != NamespaceString::kServerConfigurationNamespace) { - return; - } - - if (isStandaloneOrPrimary(opCtx)) { - return; - } - - for (auto it = begin; it != end; ++it) { - const auto& insertedDoc = it->doc; - - if (auto idElem = insertedDoc["_id"]) { - if (idElem.str() == FreeMonStorage::kFreeMonDocIdKey) { - auto controller = FreeMonController::get(opCtx->getServiceContext()); - - if (controller != nullptr) { - controller->notifyOnUpsert(insertedDoc.getOwned()); - } - } - } - } -} - -void FreeMonOpObserver::onUpdate(OperationContext* opCtx, const OplogUpdateEntryArgs& args) { - if (args.nss != NamespaceString::kServerConfigurationNamespace) { - return; - } - - if (isStandaloneOrPrimary(opCtx)) { - return; - } - - if (args.updateArgs->updatedDoc["_id"].str() == FreeMonStorage::kFreeMonDocIdKey) { - auto controller = FreeMonController::get(opCtx->getServiceContext()); - - if (controller != nullptr) { - controller->notifyOnUpsert(args.updateArgs->updatedDoc.getOwned()); - } - } -} - -void FreeMonOpObserver::aboutToDelete(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - const BSONObj& doc) { - - bool isFreeMonDoc = (nss == NamespaceString::kServerConfigurationNamespace) && - (doc["_id"].str() == FreeMonStorage::kFreeMonDocIdKey); - - // Set a flag that indicates whether the document to be delete is the free monitoring state - // document - getFreeMonDeleteState(opCtx) = isFreeMonDoc; -} - -void FreeMonOpObserver::onDelete(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - StmtId stmtId, - const OplogDeleteEntryArgs& args) { - if (nss != NamespaceString::kServerConfigurationNamespace) { - return; - } - - if (isStandaloneOrPrimary(opCtx)) { - return; - } - - if (getFreeMonDeleteState(opCtx) == true) { - auto controller = FreeMonController::get(opCtx->getServiceContext()); - - if (controller != nullptr) { - controller->notifyOnDelete(); - } - } -} - -void FreeMonOpObserver::_onReplicationRollback(OperationContext* opCtx, - const RollbackObserverInfo& rbInfo) { - // Invalidate any in-memory auth data if necessary. - const auto& rollbackNamespaces = rbInfo.rollbackNamespaces; - if (rollbackNamespaces.count(NamespaceString::kServerConfigurationNamespace) == 1) { - auto controller = FreeMonController::get(opCtx->getServiceContext()); - - if (controller != nullptr) { - controller->notifyOnRollback(); - } - } -} - - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_op_observer.h b/src/mongo/db/free_mon/free_mon_op_observer.h deleted file mode 100644 index 498d728d187..00000000000 --- a/src/mongo/db/free_mon/free_mon_op_observer.h +++ /dev/null @@ -1,221 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include "mongo/db/op_observer.h" - -namespace mongo { - -/** - * OpObserver for Free Monitoring. Observes all secondary replication traffic and filters down to - * relevant entries for free monitoring. - */ -class FreeMonOpObserver final : public OpObserver { - FreeMonOpObserver(const FreeMonOpObserver&) = delete; - FreeMonOpObserver& operator=(const FreeMonOpObserver&) = delete; - -public: - FreeMonOpObserver(); - ~FreeMonOpObserver(); - - void onCreateIndex(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - BSONObj indexDoc, - bool fromMigrate) final {} - - void onStartIndexBuild(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& collUUID, - const UUID& indexBuildUUID, - const std::vector<BSONObj>& indexes, - bool fromMigrate) final {} - - void onStartIndexBuildSinglePhase(OperationContext* opCtx, const NamespaceString& nss) final {} - - void onAbortIndexBuildSinglePhase(OperationContext* opCtx, const NamespaceString& nss) final {} - - void onCommitIndexBuild(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& collUUID, - const UUID& indexBuildUUID, - const std::vector<BSONObj>& indexes, - bool fromMigrate) final {} - - void onAbortIndexBuild(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& collUUID, - const UUID& indexBuildUUID, - const std::vector<BSONObj>& indexes, - const Status& cause, - bool fromMigrate) final {} - - void onInserts(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - std::vector<InsertStatement>::const_iterator begin, - std::vector<InsertStatement>::const_iterator end, - bool fromMigrate) final; - - void onUpdate(OperationContext* opCtx, const OplogUpdateEntryArgs& args) final; - - void aboutToDelete(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - const BSONObj& doc) final; - - void onDelete(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - StmtId stmtId, - const OplogDeleteEntryArgs& args) final; - - void onInternalOpMessage(OperationContext* opCtx, - const NamespaceString& nss, - const boost::optional<UUID>& uuid, - const BSONObj& msgObj, - const boost::optional<BSONObj> o2MsgObj, - const boost::optional<repl::OpTime> preImageOpTime, - const boost::optional<repl::OpTime> postImageOpTime, - const boost::optional<repl::OpTime> prevWriteOpTimeInTransaction, - const boost::optional<OplogSlot> slot) final{}; - - void onCreateCollection(OperationContext* opCtx, - const CollectionPtr& coll, - const NamespaceString& collectionName, - const CollectionOptions& options, - const BSONObj& idIndex, - const OplogSlot& createOpTime, - bool fromMigrate) final {} - - void onCollMod(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - const BSONObj& collModCmd, - const CollectionOptions& oldCollOptions, - boost::optional<IndexCollModInfo> indexInfo) final {} - - void onDropDatabase(OperationContext* opCtx, const std::string& dbName) final {} - - using OpObserver::onDropCollection; - repl::OpTime onDropCollection(OperationContext* opCtx, - const NamespaceString& collectionName, - const UUID& uuid, - std::uint64_t numRecords, - CollectionDropType dropType) final; - - void onDropIndex(OperationContext* opCtx, - const NamespaceString& nss, - const UUID& uuid, - const std::string& indexName, - const BSONObj& indexInfo) final {} - - using OpObserver::onRenameCollection; - void onRenameCollection(OperationContext* opCtx, - const NamespaceString& fromCollection, - const NamespaceString& toCollection, - const UUID& uuid, - const boost::optional<UUID>& dropTargetUUID, - std::uint64_t numRecords, - bool stayTemp) final {} - - void onImportCollection(OperationContext* opCtx, - const UUID& importUUID, - const NamespaceString& nss, - long long numRecords, - long long dataSize, - const BSONObj& catalogEntry, - const BSONObj& storageMetadata, - bool isDryRun) final {} - - using OpObserver::preRenameCollection; - repl::OpTime preRenameCollection(OperationContext* opCtx, - const NamespaceString& fromCollection, - const NamespaceString& toCollection, - const UUID& uuid, - const boost::optional<UUID>& dropTargetUUID, - std::uint64_t numRecords, - bool stayTemp) final { - return repl::OpTime(); - } - void postRenameCollection(OperationContext* opCtx, - const NamespaceString& fromCollection, - const NamespaceString& toCollection, - const UUID& uuid, - const boost::optional<UUID>& dropTargetUUID, - bool stayTemp) final {} - void onApplyOps(OperationContext* opCtx, - const std::string& dbName, - const BSONObj& applyOpCmd) final {} - - void onEmptyCapped(OperationContext* opCtx, - const NamespaceString& collectionName, - const UUID& uuid) final {} - - void onUnpreparedTransactionCommit(OperationContext* opCtx, - std::vector<repl::ReplOperation>* statements, - size_t numberOfPrePostImagesToWrite) final {} - - void onPreparedTransactionCommit( - OperationContext* opCtx, - OplogSlot commitOplogEntryOpTime, - Timestamp commitTimestamp, - const std::vector<repl::ReplOperation>& statements) noexcept final {} - - std::unique_ptr<ApplyOpsOplogSlotAndOperationAssignment> preTransactionPrepare( - OperationContext* opCtx, - const std::vector<OplogSlot>& reservedSlots, - size_t numberOfPrePostImagesToWrite, - Date_t wallClockTime, - std::vector<repl::ReplOperation>* statements) final { - return nullptr; - } - - void onTransactionPrepare( - OperationContext* opCtx, - const std::vector<OplogSlot>& reservedSlots, - std::vector<repl::ReplOperation>* statements, - const ApplyOpsOplogSlotAndOperationAssignment* applyOpsOperationAssignment, - size_t numberOfPrePostImagesToWrite, - Date_t wallClockTime) final {} - - void onTransactionAbort(OperationContext* opCtx, - boost::optional<OplogSlot> abortOplogEntryOpTime) final {} - - void onBatchedWriteCommit(OperationContext* opCtx) final {} - - void onMajorityCommitPointUpdate(ServiceContext* service, - const repl::OpTime& newCommitPoint) final {} - -private: - void _onReplicationRollback(OperationContext* opCtx, const RollbackObserverInfo& rbInfo); -}; - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_options.cpp b/src/mongo/db/free_mon/free_mon_options.cpp deleted file mode 100644 index df1255e1200..00000000000 --- a/src/mongo/db/free_mon/free_mon_options.cpp +++ /dev/null @@ -1,99 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kFTDC - - -#include "mongo/platform/basic.h" - -#include "mongo/db/free_mon/free_mon_options.h" - -#include "mongo/base/error_codes.h" -#include "mongo/base/status.h" -#include "mongo/base/status_with.h" -#include "mongo/base/string_data.h" -#include "mongo/util/options_parser/startup_option_init.h" -#include "mongo/util/options_parser/startup_options.h" - -namespace mongo { - -FreeMonParams globalFreeMonParams; - -namespace optionenvironment { -class OptionSection; -class Environment; -} // namespace optionenvironment - -namespace moe = mongo::optionenvironment; - -namespace { - -constexpr StringData kEnableCloudState_on = "on"_sd; -constexpr StringData kEnableCloudState_off = "off"_sd; -constexpr StringData kEnableCloudState_runtime = "runtime"_sd; - -StatusWith<EnableCloudStateEnum> EnableCloudState_parse(StringData value) { - if (value == kEnableCloudState_on) { - return EnableCloudStateEnum::kOn; - } - if (value == kEnableCloudState_off) { - return EnableCloudStateEnum::kOff; - } - if (value == kEnableCloudState_runtime) { - return EnableCloudStateEnum::kRuntime; - } - - return Status(ErrorCodes::InvalidOptions, "Unrecognized state"); -} - -Status storeFreeMonitoringOptions(const moe::Environment& params) { - - if (params.count("cloud.monitoring.free.state")) { - auto swState = - EnableCloudState_parse(params["cloud.monitoring.free.state"].as<std::string>()); - if (!swState.isOK()) { - return swState.getStatus(); - } - globalFreeMonParams.freeMonitoringState = swState.getValue(); - } - - if (params.count("cloud.monitoring.free.tags")) { - globalFreeMonParams.freeMonitoringTags = - params["cloud.monitoring.free.tags"].as<std::vector<std::string>>(); - } - - return Status::OK(); -} - -MONGO_STARTUP_OPTIONS_STORE(FreeMonitoringOptions)(InitializerContext*) { - uassertStatusOK(storeFreeMonitoringOptions(moe::startupOptionsParsed)); -} - -} // namespace -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_options.h b/src/mongo/db/free_mon/free_mon_options.h deleted file mode 100644 index 19f707e8b65..00000000000 --- a/src/mongo/db/free_mon/free_mon_options.h +++ /dev/null @@ -1,56 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include <string> -#include <vector> - -namespace mongo { - -/** - * Free Moniting Command line choices - */ -enum class EnableCloudStateEnum : std::int32_t { - kOn, - kOff, - kRuntime, -}; - -/** - * Free Monitoring configuration options - */ -struct FreeMonParams { - std::vector<std::string> freeMonitoringTags; - EnableCloudStateEnum freeMonitoringState = EnableCloudStateEnum::kRuntime; -}; - -extern FreeMonParams globalFreeMonParams; - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_options.idl b/src/mongo/db/free_mon/free_mon_options.idl deleted file mode 100644 index ba7f472b467..00000000000 --- a/src/mongo/db/free_mon/free_mon_options.idl +++ /dev/null @@ -1,51 +0,0 @@ -# Copyright (C) 2018-present MongoDB, Inc. -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the Server Side Public License, version 1, -# as published by MongoDB, Inc. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# Server Side Public License for more details. -# -# You should have received a copy of the Server Side Public License -# along with this program. If not, see -# <http://www.mongodb.com/licensing/server-side-public-license>. -# -# As a special exception, the copyright holders give permission to link the -# code of portions of this program with the OpenSSL library under certain -# conditions as described in each individual source file and distribute -# linked combinations including the program with the OpenSSL library. You -# must comply with the Server Side Public License in all respects for -# all of the code used other than as permitted herein. If you modify file(s) -# with this exception, you may extend this exception to your version of the -# file(s), but you are not obligated to do so. If you do not wish to do so, -# delete this exception statement from your version. If you delete this -# exception statement from all source files in the program, then also delete -# it in the license file. -# - -global: - cpp_namespace: "mongo" - configs: - section: "Free Monitoring Options" - source: [ yaml, cli] - -imports: - - "mongo/idl/basic_types.idl" - -configs: - # Command Line: --enableFreeMonitoring=<on|runtime|off> - # YAML Name: cloud.monitoring.free=<on|runtime|off> - cloud.monitoring.free.state: - description: "Enable Cloud Free Monitoring (on|runtime|off)" - short_name: enableFreeMonitoring - arg_vartype: String - - # Command Line: --enableFreeMonitoringTag=array<string> - # YAML Name: cloud.monitoring.free.tag=array<string> - cloud.monitoring.free.tags: - description: "Cloud Free Monitoring Tags" - short_name: freeMonitoringTag - arg_vartype: StringVector diff --git a/src/mongo/db/free_mon/free_mon_processor.cpp b/src/mongo/db/free_mon/free_mon_processor.cpp deleted file mode 100644 index c91d62204b8..00000000000 --- a/src/mongo/db/free_mon/free_mon_processor.cpp +++ /dev/null @@ -1,1027 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#define MONGO_LOGV2_DEFAULT_COMPONENT ::mongo::logv2::LogComponent::kControl - -#include "mongo/platform/basic.h" - -#include "mongo/db/free_mon/free_mon_processor.h" - -#include <functional> -#include <numeric> -#include <snappy.h> -#include <tuple> -#include <utility> - -#include "mongo/base/data_range.h" -#include "mongo/base/status.h" -#include "mongo/base/string_data.h" -#include "mongo/bson/bsonobj.h" -#include "mongo/db/free_mon/free_mon_storage.h" -#include "mongo/db/service_context.h" -#include "mongo/idl/idl_parser.h" -#include "mongo/logv2/log.h" -#include "mongo/util/assert_util.h" - -namespace mongo { - -namespace { - -constexpr auto kMinProtocolVersion = 1; -constexpr auto kMaxProtocolVersion = 2; -constexpr auto kStorageVersion = 1; - -constexpr auto kRegistrationIdMaxLength = 4096; -constexpr auto kInformationalURLMaxLength = 4096; -constexpr auto kInformationalMessageMaxLength = 4096; -constexpr auto kUserReminderMaxLength = 4096; - -constexpr auto kReportingIntervalSecondsMin = 1; -constexpr auto kReportingIntervalSecondsMax = 30 * 60 * 60 * 24; - -constexpr auto kMetricsRequestArrayElement = "data"_sd; - -int64_t randomJitter(PseudoRandom& random, int64_t min, int64_t max) { - dassert(max > min); - return (std::abs(random.nextInt64()) % (max - min)) + min; -} - -} // namespace - -void RegistrationRetryCounter::reset() { - _current = _min; - _base = _min; - _retryCount = 0; - _total = Hours(0); -} - -bool RegistrationRetryCounter::incrementError() { - if (_retryCount < kStage1RetryCountMax) { - _base = 2 * _base; - _current = _base + Seconds(randomJitter(_random, kStage1JitterMin, kStage1JitterMax)); - ++_retryCount; - } else { - _current = _base + Seconds(randomJitter(_random, kStage2JitterMin, kStage2JitterMax)); - } - - _total += _current; - - if (_total > kStage2DurationMax) { - return false; - } - - return true; -} - -void MetricsRetryCounter::reset() { - _current = _min; - _base = _min; - _retryCount = 0; - _total = Hours(0); -} - -bool MetricsRetryCounter::incrementError() { - _base = static_cast<int>(pow(2, std::min(6, static_cast<int>(_retryCount)))) * _min; - _current = _base + Seconds(randomJitter(_random, _min.count() / 2, _min.count())); - ++_retryCount; - - _total += _current; - - if (_total > kDurationMax) { - return false; - } - - return true; -} - -FreeMonProcessor::FreeMonProcessor(FreeMonCollectorCollection& registration, - FreeMonCollectorCollection& metrics, - FreeMonNetworkInterface* network, - bool useCrankForTest, - Seconds metricsGatherInterval) - : _registration(registration), - _metrics(metrics), - _network(network), - _random(Date_t::now().asInt64()), - _registrationRetry(RegistrationRetryCounter(_random)), - _metricsRetry(MetricsRetryCounter(_random)), - _metricsGatherInterval(metricsGatherInterval), - _queue(useCrankForTest) { - _registrationRetry->reset(); - _metricsRetry->reset(); -} - -void FreeMonProcessor::enqueue(std::shared_ptr<FreeMonMessage> msg) { - _queue.enqueue(std::move(msg)); -} - -void FreeMonProcessor::stop() { - _queue.stop(); -} - -void FreeMonProcessor::turnCrankForTest(size_t countMessagesToIgnore) { - _countdown.reset(countMessagesToIgnore); - - _queue.turnCrankForTest(countMessagesToIgnore); - - _countdown.wait(); -} - -void FreeMonProcessor::deprioritizeFirstMessageForTest(FreeMonMessageType type) { - _queue.deprioritizeFirstMessageForTest(type); -} - -void FreeMonProcessor::run() { - try { - - Client::initThread("FreeMonProcessor"); - Client* client = &cc(); - - while (true) { - auto item = _queue.dequeue(client->getServiceContext()->getPreciseClockSource()); - if (!item.is_initialized()) { - // Shutdown was triggered - return; - } - - auto msg = item.get(); - - // Do work here - switch (msg->getType()) { - case FreeMonMessageType::RegisterCommand: { - doCommandRegister(client, msg); - break; - } - case FreeMonMessageType::RegisterServer: { - doServerRegister( - client, - checked_cast< - FreeMonMessageWithPayload<FreeMonMessageType::RegisterServer>*>( - msg.get())); - break; - } - case FreeMonMessageType::UnregisterCommand: { - doCommandUnregister(client, - checked_cast<FreeMonWaitableMessageWithPayload< - FreeMonMessageType::UnregisterCommand>*>(msg.get())); - break; - } - case FreeMonMessageType::AsyncRegisterComplete: { - doAsyncRegisterComplete( - client, - checked_cast< - FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterComplete>*>( - msg.get())); - break; - } - case FreeMonMessageType::AsyncRegisterFail: { - doAsyncRegisterFail( - client, - checked_cast< - FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterFail>*>( - msg.get())); - break; - } - case FreeMonMessageType::MetricsCollect: { - doMetricsCollect(client); - break; - } - case FreeMonMessageType::MetricsSend: { - doMetricsSend(client); - break; - } - case FreeMonMessageType::AsyncMetricsComplete: { - doAsyncMetricsComplete( - client, - checked_cast< - FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsComplete>*>( - msg.get())); - break; - } - case FreeMonMessageType::AsyncMetricsFail: { - doAsyncMetricsFail( - client, - checked_cast< - FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsFail>*>( - msg.get())); - break; - } - case FreeMonMessageType::OnTransitionToPrimary: { - doOnTransitionToPrimary(client); - break; - } - case FreeMonMessageType::NotifyOnUpsert: { - doNotifyOnUpsert( - client, - checked_cast< - FreeMonMessageWithPayload<FreeMonMessageType::NotifyOnUpsert>*>( - msg.get())); - break; - } - case FreeMonMessageType::NotifyOnDelete: { - doNotifyOnDelete(client); - break; - } - case FreeMonMessageType::NotifyOnRollback: { - doNotifyOnRollback(client); - break; - } - default: - MONGO_UNREACHABLE; - } - - // Record that we have finished processing the message for testing purposes. - _countdown.countDown(); - } - } catch (...) { - // Stop the queue - _queue.stop(); - - LOGV2_WARNING(20619, - "Uncaught exception in '{error}' in free monitoring subsystem. " - "Shutting down the free monitoring subsystem.", - "Uncaught exception in free monitoring subsystem. " - "Shutting down the free monitoring subsystem.", - "error"_attr = exceptionToStatus()); - } -} - -void FreeMonProcessor::readState(OperationContext* opCtx, bool updateInMemory) { - auto state = FreeMonStorage::read(opCtx); - - _lastReadState = state; - - if (state.is_initialized()) { - invariant(state.get().getVersion() == kStorageVersion); - - if (updateInMemory) { - _state = state.get(); - } - } else if (!state.is_initialized()) { - // Default the state - auto state = _state.synchronize(); - state->setVersion(kStorageVersion); - state->setState(StorageStateEnum::disabled); - state->setRegistrationId(""); - state->setInformationalURL(""); - state->setMessage(""); - state->setUserReminder(""); - } -} - -void FreeMonProcessor::readState(Client* client, bool updateInMemory) { - auto opCtx = client->makeOperationContext(); - readState(opCtx.get(), updateInMemory); -} - -void FreeMonProcessor::writeState(Client* client) { - - // Do a compare and swap - // Verify the document is the same as the one on disk, if it is the same, then do the update - // If the local document is different, then oh-well we do nothing, and wait until the next round - - // Has our in-memory state changed, if so consider writing - if (_lastReadState != _state.get()) { - - // The read and write are bound the same operation context - { - auto optCtx = client->makeOperationContext(); - - auto state = FreeMonStorage::read(optCtx.get()); - - // If our in-memory copy matches the last read, then write it to disk - if (state == _lastReadState) { - FreeMonStorage::replace(optCtx.get(), _state.get()); - - _lastReadState = boost::make_optional(_state.get()); - } - } - } -} - -void FreeMonProcessor::doServerRegister( - Client* client, const FreeMonMessageWithPayload<FreeMonMessageType::RegisterServer>* msg) { - - // Enqueue the first metrics gather first so we have something to send on intial registration - enqueue(FreeMonMessage::createNow(FreeMonMessageType::MetricsCollect)); - - // If we are asked to register now, then kick off a registration request - const auto regType = msg->getPayload().first; - if (regType == RegistrationType::RegisterOnStart) { - enqueue(FreeMonRegisterCommandMessage::createNow({msg->getPayload().second, boost::none})); - } else { - invariant((regType == RegistrationType::RegisterAfterOnTransitionToPrimary) || - (regType == RegistrationType::RegisterAfterOnTransitionToPrimaryIfEnabled)); - // Check if we need to wait to become primary: - // If the 'admin.system.version' has content, do not wait and just re-register - // If the collection is empty, wait until we become primary - // If we become secondary, OpObserver hooks will tell us our registration id - - auto optCtx = client->makeOperationContext(); - - // Check if there is an existing document - auto state = FreeMonStorage::read(optCtx.get()); - - // If there is no document, we may be: - // 1. in a replica set and may need to register after becoming primary since we cannot - // record the registration id until after becoming primary - // 2. a standalone which has never been registered - // - if (!state.is_initialized()) { - _registerOnTransitionToPrimary = regType; - } else { - // We are standalone or secondary, if we have a registration id, then send a - // registration notification, else wait for the user to register us. - if (state.get().getState() == StorageStateEnum::enabled) { - enqueue(FreeMonRegisterCommandMessage::createNow( - {msg->getPayload().second, boost::none})); - } - } - - // Ensure we read the state once. - // This is important on a disabled secondary so that the in-memory state knows we are - // disabled. - readState(optCtx.get()); - } -} - -namespace { -template <typename T> -std::unique_ptr<Future<void>> doAsyncCallback(FreeMonProcessor* proc, - Future<T> future, - std::function<void(const T&)> onSuccess, - std::function<void(Status)> onErrorFunc) { - - // Grab a weak_ptr to be sure that FreeMonProcessor is alive during the callback - std::weak_ptr<FreeMonProcessor> wpProc(proc->shared_from_this()); - - auto spError = std::make_shared<bool>(false); - - return std::make_unique<Future<void>>(std::move(future) - .onError([=](Status s) { - *(spError.get()) = true; - if (auto spProc = wpProc.lock()) { - onErrorFunc(s); - } - - return T(); - }) - .then([=](const auto& resp) { - // If we hit an error, then do not call onSuccess - if (*(spError.get()) == true) { - return; - } - - // Use a shared pointer here because the callback - // could return after we disappear - if (auto spProc = wpProc.lock()) { - onSuccess(resp); - } - })); -} -} // namespace - -void FreeMonProcessor::doCommandRegister(Client* client, - std::shared_ptr<FreeMonMessage> sharedMsg) { - auto msg = checked_cast<FreeMonRegisterCommandMessage*>(sharedMsg.get()); - - if (_futureRegistrationResponse) { - msg->setStatus(Status(ErrorCodes::FreeMonHttpInFlight, - "Free Monitoring Registration request in-flight already")); - return; - } - - _pendingRegisters.push_back(sharedMsg); - - readState(client); - - FreeMonRegistrationRequest req; - - if (msg->getPayload().second) { - req.setId(StringData(msg->getPayload().second.get())); - } else { - auto regid = _state->getRegistrationId(); - if (!regid.empty()) { - req.setId(regid); - } - } - - req.setVersion(kMaxProtocolVersion); - - req.setLocalTime(client->getServiceContext()->getPreciseClockSource()->now()); - - if (!msg->getPayload().first.empty()) { - // Cache the tags for subsequent retries - _tags = msg->getPayload().first; - } - - if (!_tags.empty()) { - req.setTags(transformVector(msg->getPayload().first)); - } - - // Collect the data - auto collect = _registration.collect(client); - - req.setPayload(std::get<0>(collect)); - - // Record that the registration is pending - _state->setState(StorageStateEnum::pending); - _registrationStatus = FreeMonRegistrationStatus::kPending; - - writeState(client); - - // Send the async request - _futureRegistrationResponse = doAsyncCallback<FreeMonRegistrationResponse>( - this, - _network->sendRegistrationAsync(req), - [this](const auto& resp) { - this->enqueue( - FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterComplete>::createNow( - resp)); - }, - [this](Status s) { - this->enqueue( - FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterFail>::createNow(s)); - }); -} - -Status FreeMonProcessor::validateRegistrationResponse(const FreeMonRegistrationResponse& resp) { - // Any validation failure stops registration from proceeding to upload - if (!(resp.getVersion() >= kMinProtocolVersion && resp.getVersion() <= kMaxProtocolVersion)) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() - << "Unexpected registration response protocol version, expected (" - << kMinProtocolVersion << ", " << kMaxProtocolVersion << "), received '" - << resp.getVersion() << "'"); - } - - if (resp.getId().size() >= kRegistrationIdMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Id is '" << resp.getId().size() - << "' bytes in length, maximum allowed length is '" - << kRegistrationIdMaxLength << "'"); - } - - if (resp.getInformationalURL().size() >= kInformationalURLMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "InformationURL is '" << resp.getInformationalURL().size() - << "' bytes in length, maximum allowed length is '" - << kInformationalURLMaxLength << "'"); - } - - if (resp.getMessage().size() >= kInformationalMessageMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Message is '" << resp.getMessage().size() - << "' bytes in length, maximum allowed length is '" - << kInformationalMessageMaxLength << "'"); - } - - if (resp.getUserReminder().is_initialized() && - resp.getUserReminder().get().size() >= kUserReminderMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "UserReminder is '" << resp.getUserReminder().get().size() - << "' bytes in length, maximum allowed length is '" - << kUserReminderMaxLength << "'"); - } - - if (resp.getReportingInterval() < kReportingIntervalSecondsMin || - resp.getReportingInterval() > kReportingIntervalSecondsMax) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Reporting Interval '" << resp.getReportingInterval() - << "' must be in the range [" << kReportingIntervalSecondsMin - << "," << kReportingIntervalSecondsMax << "]"); - } - - // Did cloud ask us to stop uploading? - if (resp.getHaltMetricsUploading()) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Halting metrics upload due to response"); - } - - return Status::OK(); -} - - -void FreeMonProcessor::notifyPendingRegisters(const Status s) { - for (auto&& pendingRegister : _pendingRegisters) { - (checked_cast<FreeMonRegisterCommandMessage*>(pendingRegister.get()))->setStatus(s); - } - _pendingRegisters.clear(); -} - - -Status FreeMonProcessor::validateMetricsResponse(const FreeMonMetricsResponse& resp) { - // Any validation failure stops registration from proceeding to upload - if (!(resp.getVersion() >= kMinProtocolVersion && resp.getVersion() <= kMaxProtocolVersion)) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Unexpected metrics response protocol version, expected (" - << kMinProtocolVersion << ", " << kMaxProtocolVersion - << "), received '" << resp.getVersion() << "'"); - } - - if (resp.getId().is_initialized() && resp.getId().get().size() >= kRegistrationIdMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Id is '" << resp.getId().get().size() - << "' bytes in length, maximum allowed length is '" - << kRegistrationIdMaxLength << "'"); - } - - if (resp.getInformationalURL().is_initialized() && - resp.getInformationalURL().get().size() >= kInformationalURLMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() - << "InformationURL is '" << resp.getInformationalURL().get().size() - << "' bytes in length, maximum allowed length is '" - << kInformationalURLMaxLength << "'"); - } - - if (resp.getMessage().is_initialized() && - resp.getMessage().get().size() >= kInformationalMessageMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Message is '" << resp.getMessage().get().size() - << "' bytes in length, maximum allowed length is '" - << kInformationalMessageMaxLength << "'"); - } - - if (resp.getUserReminder().is_initialized() && - resp.getUserReminder().get().size() >= kUserReminderMaxLength) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "UserReminder is '" << resp.getUserReminder().get().size() - << "' bytes in length, maximum allowed length is '" - << kUserReminderMaxLength << "'"); - } - - if (resp.getReportingInterval() < kReportingIntervalSecondsMin || - resp.getReportingInterval() > kReportingIntervalSecondsMax) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Reporting Interval '" << resp.getReportingInterval() - << "' must be in the range [" << kReportingIntervalSecondsMin - << "," << kReportingIntervalSecondsMax << "]"); - } - - // Did cloud ask us to stop uploading? - if (resp.getHaltMetricsUploading()) { - return Status(ErrorCodes::FreeMonHttpPermanentFailure, - str::stream() << "Halting metrics upload due to response"); - } - - return Status::OK(); -} - - -void FreeMonProcessor::doAsyncRegisterComplete( - Client* client, - const FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterComplete>* msg) { - - // Our request is no longer in-progress so delete it - _futureRegistrationResponse.reset(); - - if (_registrationStatus != FreeMonRegistrationStatus::kPending) { - notifyPendingRegisters(Status(ErrorCodes::BadValue, "Registration was canceled")); - - return; - } - - auto& resp = msg->getPayload(); - - Status s = validateRegistrationResponse(resp); - if (!s.isOK()) { - LOGV2_WARNING(20620, - "Free Monitoring registration halted due to {error}", - "Free Monitoring registration halted due to error", - "error"_attr = s); - - // Disable on any error - _state->setState(StorageStateEnum::disabled); - _registrationStatus = FreeMonRegistrationStatus::kDisabled; - - // Persist state - writeState(client); - - notifyPendingRegisters(s); - - // If validation fails, we do not retry - return; - } - - // Update in-memory state - _registrationRetry->setMin(Seconds(resp.getReportingInterval())); - _metricsGatherInterval = Seconds(resp.getReportingInterval()); - - { - auto state = _state.synchronize(); - state->setRegistrationId(resp.getId()); - - if (resp.getUserReminder().is_initialized()) { - state->setUserReminder(resp.getUserReminder().get()); - } else { - state->setUserReminder(""); - } - - state->setMessage(resp.getMessage()); - state->setInformationalURL(resp.getInformationalURL()); - - state->setState(StorageStateEnum::enabled); - } - - _registrationStatus = FreeMonRegistrationStatus::kEnabled; - - // Persist state - writeState(client); - - // Reset retry counter - _registrationRetry->reset(); - - // Notify waiters - notifyPendingRegisters(Status::OK()); - - LOGV2(20615, - "Free Monitoring is Enabled. Frequency: {interval} seconds", - "Free Monitoring is Enabled", - "interval"_attr = resp.getReportingInterval()); - - // Enqueue next metrics upload immediately to deliver a good experience - enqueue(FreeMonMessage::createNow(FreeMonMessageType::MetricsSend)); -} - -void FreeMonProcessor::doAsyncRegisterFail( - Client* client, const FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterFail>* msg) { - - // Our request is no longer in-progress so delete it - _futureRegistrationResponse.reset(); - - if (_registrationStatus != FreeMonRegistrationStatus::kPending) { - notifyPendingRegisters(Status(ErrorCodes::BadValue, "Registration was canceled")); - - return; - } - - if (!_registrationRetry->incrementError()) { - // We have exceeded our retry - LOGV2_WARNING(20621, "Free Monitoring is abandoning registration after excess retries"); - return; - } - - LOGV2_DEBUG(20616, - 1, - "Free Monitoring Registration Failed with status '{error}', retrying in {interval}", - "Free Monitoring Registration Failed, will retry after interval", - "error"_attr = msg->getPayload(), - "interval"_attr = _registrationRetry->getNextDuration()); - - // Enqueue a register retry - enqueue(FreeMonRegisterCommandMessage::createWithDeadline( - {_tags, boost::none}, _registrationRetry->getNextDeadline(client))); -} - -void FreeMonProcessor::doCommandUnregister( - Client* client, FreeMonWaitableMessageWithPayload<FreeMonMessageType::UnregisterCommand>* msg) { - // Treat this request as idempotent - readState(client); - - _state->setState(StorageStateEnum::disabled); - _registrationStatus = FreeMonRegistrationStatus::kDisabled; - - writeState(client); - - LOGV2(20617, "Free Monitoring is Disabled"); - - msg->setStatus(Status::OK()); -} - -void FreeMonProcessor::doMetricsCollect(Client* client) { - // Collect the time at the beginning so the time to collect does not affect the schedule - Date_t now = client->getServiceContext()->getPreciseClockSource()->now(); - - // Collect the data - auto collect = _metrics.collect(client); - - _metricsBuffer.push(std::get<0>(collect)); - - // Enqueue the next metrics collect based on when we started processing the last collection. - enqueue(FreeMonMessage::createWithDeadline(FreeMonMessageType::MetricsCollect, - now + _metricsGatherInterval)); -} - -std::string compressMetrics(MetricsBuffer& buffer) { - BSONObjBuilder builder; - - { - BSONArrayBuilder arrayBuilder(builder.subarrayStart(kMetricsRequestArrayElement)); - - for (const auto& obj : buffer) { - arrayBuilder.append(obj); - } - } - - BSONObj obj = builder.done(); - - std::string outBuffer; - snappy::Compress(obj.objdata(), obj.objsize(), &outBuffer); - - return outBuffer; -} - -void FreeMonProcessor::doMetricsSend(Client* client) { - // We want to read state from disk in case we asked to stop but otherwise - // use the in-memory state. It is important not to treat disk state as authoritative - // on secondaries. - readState(client, false); - - // Only continue metrics send if the local disk state (in-case user deleted local document) - // and in-memory status both say to continue. - if (_registrationStatus != FreeMonRegistrationStatus::kEnabled || - _state->getState() != StorageStateEnum::enabled) { - // If we are recently disabled, then stop sending metrics - return; - } - - // Build outbound request - FreeMonMetricsRequest req; - - req.setVersion(kMaxProtocolVersion); - req.setLocalTime(client->getServiceContext()->getPreciseClockSource()->now()); - req.setEncoding(MetricsEncodingEnum::snappy); - - req.setId(_state->getRegistrationId()); - - // Get the buffered metrics - auto metrics = compressMetrics(_metricsBuffer); - req.setMetrics(ConstDataRange(metrics.data(), metrics.size())); - - _lastMetricsSend = Date_t::now(); - - // Send the async request - doAsyncCallback<FreeMonMetricsResponse>( - this, - _network->sendMetricsAsync(req), - [this](const auto& resp) { - this->enqueue( - FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsComplete>::createNow( - resp)); - }, - [this](Status s) { - this->enqueue( - FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsFail>::createNow(s)); - }); -} - -void FreeMonProcessor::doAsyncMetricsComplete( - Client* client, - const FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsComplete>* msg) { - - // If we have disabled the store between the metrics send message and the metrcs complete - // message then it means that we need to stop processing metrics on this instance. We ignore the - // message entirely including an errors as the disabling of the store takes priority. - if (_lastReadState == boost::none) { - return; - } - - auto& resp = msg->getPayload(); - - Status s = validateMetricsResponse(resp); - if (!s.isOK()) { - LOGV2_WARNING(20622, - "Free Monitoring metrics uploading halted due to {error}", - "Free Monitoring metrics uploading halted due to error", - "error"_attr = s); - - // Disable free monitoring on validation errors - _state->setState(StorageStateEnum::disabled); - _registrationStatus = FreeMonRegistrationStatus::kDisabled; - - writeState(client); - - // If validation fails, we do not retry - return; - } - - // If cloud said delete, not just halt, so erase state - if (resp.getPermanentlyDelete() == true) { - auto opCtxUnique = client->makeOperationContext(); - FreeMonStorage::deleteState(opCtxUnique.get()); - - _state->setState(StorageStateEnum::pending); - _registrationStatus = FreeMonRegistrationStatus::kDisabled; - - // Clear out the in-memory state - _lastReadState = boost::none; - - return; - } - - // Update in-memory state of buffered metrics - // TODO: do we reset only the metrics we send or all pending on success? - - _metricsBuffer.reset(); - - { - auto state = _state.synchronize(); - - if (resp.getId().is_initialized()) { - state->setRegistrationId(resp.getId().get()); - } - - if (resp.getUserReminder().is_initialized()) { - state->setUserReminder(resp.getUserReminder().get()); - } - - if (resp.getInformationalURL().is_initialized()) { - state->setInformationalURL(resp.getInformationalURL().get()); - } - - if (resp.getMessage().is_initialized()) { - state->setMessage(resp.getMessage().get()); - } - } - - // Persist state - writeState(client); - - // Reset retry counter - _metricsGatherInterval = Seconds(resp.getReportingInterval()); - _metricsRetry->setMin(Seconds(resp.getReportingInterval())); - _metricsRetry->reset(); - - if (resp.getResendRegistration().is_initialized() && resp.getResendRegistration()) { - enqueue(FreeMonRegisterCommandMessage::createNow({_tags, boost::none})); - } else { - // Enqueue next metrics upload - enqueue(FreeMonMessage::createWithDeadline(FreeMonMessageType::MetricsSend, - _metricsRetry->getNextDeadline(client))); - } -} - -void FreeMonProcessor::doAsyncMetricsFail( - Client* client, const FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsFail>* msg) { - - if (!_metricsRetry->incrementError()) { - // We have exceeded our retry - LOGV2_WARNING(20623, "Free Monitoring is abandoning metrics upload after excess retries"); - return; - } - - LOGV2_DEBUG(20618, - 1, - "Free Monitoring Metrics upload failed with status {error}, retrying in {interval}", - "Free Monitoring Metrics upload failed, will retry after interval", - "error"_attr = msg->getPayload(), - "interval"_attr = _metricsRetry->getNextDuration()); - - // Enqueue next metrics upload - enqueue(FreeMonMessage::createWithDeadline(FreeMonMessageType::MetricsSend, - _metricsRetry->getNextDeadline(client))); -} - -void FreeMonProcessor::getStatus(OperationContext* opCtx, - BSONObjBuilder* status, - FreeMonGetStatusEnum mode) { - if (!_lastReadState.get()) { - // _state gets initialized by readState() regardless, - // use _lastReadState to differential "undecided" from default. - status->append("state", "undecided"); - return; - } - - if (mode == FreeMonGetStatusEnum::kServerStatus) { - status->append("state", StorageState_serializer(_state->getState())); - status->append("retryIntervalSecs", - durationCount<Seconds>(_metricsRetry->getNextDuration())); - auto lastMetricsSend = _lastMetricsSend.get(); - if (lastMetricsSend) { - status->append("lastRunTime", lastMetricsSend->toString()); - } - status->append("registerErrors", static_cast<long long>(_registrationRetry->getCount())); - status->append("metricsErrors", static_cast<long long>(_metricsRetry->getCount())); - } else { - auto state = _state.synchronize(); - status->append("state", StorageState_serializer(state->getState())); - status->append("message", state->getMessage()); - status->append("url", state->getInformationalURL()); - status->append("userReminder", state->getUserReminder()); - } -} - -void FreeMonProcessor::doOnTransitionToPrimary(Client* client) { - if (_registerOnTransitionToPrimary == RegistrationType::RegisterAfterOnTransitionToPrimary) { - enqueue( - FreeMonRegisterCommandMessage::createNow({std::vector<std::string>(), boost::none})); - - } else if (_registerOnTransitionToPrimary == - RegistrationType::RegisterAfterOnTransitionToPrimaryIfEnabled) { - readState(client); - if (_state->getState() == StorageStateEnum::enabled) { - enqueue(FreeMonRegisterCommandMessage::createNow( - {std::vector<std::string>(), boost::none})); - } - } - - // On transition to primary once - _registerOnTransitionToPrimary = RegistrationType::DoNotRegister; -} - -void FreeMonProcessor::processInMemoryStateChange(const FreeMonStorageState& originalState, - const FreeMonStorageState& newState) { - // Are we transition from disabled -> enabled? - if (originalState.getState() != newState.getState()) { - if (originalState.getState() != StorageStateEnum::enabled && - newState.getState() == StorageStateEnum::enabled) { - - // Secondary needs to start registration - enqueue(FreeMonRegisterCommandMessage::createNow( - {std::vector<std::string>(), newState.getRegistrationId().toString()})); - } - } -} - -void FreeMonProcessor::doNotifyOnUpsert( - Client* client, const FreeMonMessageWithPayload<FreeMonMessageType::NotifyOnUpsert>* msg) { - try { - const BSONObj& doc = msg->getPayload(); - auto newState = FreeMonStorageState::parse(IDLParserErrorContext("free_mon_storage"), doc); - - // Likely, the update changed something - if (newState != _state) { - uassert(50839, - str::stream() << "Unexpected free monitoring storage version " - << newState.getVersion(), - newState.getVersion() == kStorageVersion); - - processInMemoryStateChange(_state.get(), newState); - - // Note: enabled -> disabled is handled implicitly by register and send metrics checks - // after _state is updated below - - // Copy the fields - _state = newState; - } - - } catch (...) { - - // Stop the queue - _queue.stop(); - - LOGV2_WARNING(20624, - "Uncaught exception in '{exception}' in free monitoring op observer. " - "Shutting down the free monitoring subsystem.", - "exception"_attr = exceptionToStatus()); - } -} - -void FreeMonProcessor::doNotifyOnDelete(Client* client) { - // The config document was either deleted or the entire collection was dropped, we treat them - // the same and stop free monitoring. We continue collecting though. - - // So we mark the internal state as disabled which stop registration and metrics send - _state->setState(StorageStateEnum::pending); - _registrationStatus = FreeMonRegistrationStatus::kDisabled; - - // Clear out the in-memory state - _lastReadState = boost::none; -} - -void FreeMonProcessor::doNotifyOnRollback(Client* client) { - // We have rolled back, the state on disk reflects our new reality - // We should re-read the disk state and proceed. - - // copy the in-memory state - auto originalState = _state.get(); - - // Re-read state from disk - readState(client); - - auto newState = _state.get(); - - if (newState != originalState) { - processInMemoryStateChange(originalState, newState); - } -} - - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_processor.h b/src/mongo/db/free_mon/free_mon_processor.h deleted file mode 100644 index ec7c63fef45..00000000000 --- a/src/mongo/db/free_mon/free_mon_processor.h +++ /dev/null @@ -1,528 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ -#pragma once - -#include <boost/optional.hpp> -#include <cstdint> -#include <deque> -#include <memory> -#include <ratio> -#include <string> -#include <vector> - -#include "mongo/db/client.h" -#include "mongo/db/free_mon/free_mon_message.h" -#include "mongo/db/free_mon/free_mon_network.h" -#include "mongo/db/free_mon/free_mon_processor.h" -#include "mongo/db/free_mon/free_mon_protocol_gen.h" -#include "mongo/db/free_mon/free_mon_queue.h" -#include "mongo/db/free_mon/free_mon_storage_gen.h" -#include "mongo/db/ftdc/collector.h" -#include "mongo/db/service_context.h" -#include "mongo/util/clock_source.h" -#include "mongo/util/duration.h" -#include "mongo/util/future.h" -#include "mongo/util/synchronized_value.h" -#include "mongo/util/time_support.h" - -namespace mongo { -using FreeMonCollectorInterface = FTDCCollectorInterface; -using FreeMonCollectorCollection = FTDCCollectorCollection; - - -/** - * Reponsible for tracking when to send the next retry after errors are encountered. - */ -class RetryCounter { - const int64_t kMax = 60 * 60 * 24; - -public: - RetryCounter() : _min(1), _max(kMax) {} - virtual ~RetryCounter() = default; - - /** - * Set Minimum rety interval - */ - void setMin(Seconds s) { - _min = s; - reset(); - } - - /** - * Reset the retry interval, typically occurs after a succesfull message is sent. - */ - virtual void reset() = 0; - - /** - * Increment the error count and compute the next interval. - */ - virtual bool incrementError() = 0; - - /** - * Get the next retry duration. - */ - Seconds getNextDuration() const { - dassert(_current != Seconds(0)); - return _current; - } - - /** - * Get the next retry deadline - */ - Date_t getNextDeadline(Client* client) const { - return client->getServiceContext()->getPreciseClockSource()->now() + _current; - } - -protected: - // Current retry interval - Seconds _current; - - // Minimum retry interval - Seconds _min; - - // Maximum retry interval - Seconds _max; -}; - -/** - * Manage retries for registrations - */ -class RegistrationRetryCounter : public RetryCounter { -public: - explicit RegistrationRetryCounter(PseudoRandom& random) : _random(random) {} - - void reset() final; - - bool incrementError() final; - - size_t getCount() const { - return _retryCount; - } - -private: - // Random number generator for jitter - PseudoRandom& _random; - - // Retry count for stage 1 retry - size_t _retryCount{0}; - - // Total Seconds we have retried for - Seconds _total; - - // Last retry interval without jitter - Seconds _base; - - // Max Retry count - const size_t kStage1RetryCountMax{10}; - - const size_t kStage1JitterMin{2}; - const size_t kStage1JitterMax{10}; - - const Hours kStage2DurationMax{48}; - - const size_t kStage2JitterMin{60}; - const size_t kStage2JitterMax{120}; -}; - -/** - * Manage retries for metrics - */ -class MetricsRetryCounter : public RetryCounter { -public: - explicit MetricsRetryCounter(PseudoRandom& random) : _random(random) {} - - void reset() final; - - bool incrementError() final; - - size_t getCount() const { - return _retryCount; - } - -private: - // Random number generator for jitter - PseudoRandom& _random; - - // Retry count for stage 1 retry - size_t _retryCount{0}; - - // Total Seconds we have retried for - Seconds _total; - - // Last retry interval without jitter - Seconds _base; - - // Max Duration - const Hours kDurationMax{7 * 24}; -}; - -/** - * Simple bounded buffer of metrics to upload. - */ -class MetricsBuffer { -public: - using container_type = std::deque<BSONObj>; - - /** - * Add a metric to the buffer. Oldest metric will be discarded if buffer is at capacity. - */ - void push(BSONObj obj) { - if (_queue.size() == kMaxElements) { - _queue.pop_front(); - } - - _queue.push_back(obj); - } - - /** - * Flush the buffer down to kMinElements entries. The last entries are held for cloud. - */ - void reset() { - while (_queue.size() > kMinElements) { - _queue.pop_front(); - } - } - - container_type::iterator begin() { - return _queue.begin(); - } - container_type::iterator end() { - return _queue.end(); - } - -private: - // Bounded queue of metrics - container_type _queue; - - const size_t kMinElements = 1; - const size_t kMaxElements = 10; -}; - -/** - * Countdown latch for test support in FreeMonProcessor so that a crank can be turned manually. - */ -class FreeMonCountdownLatch { -public: - explicit FreeMonCountdownLatch() : _count(0) {} - - /** - * Reset countdown latch wait for N events. - */ - void reset(uint32_t count) { - stdx::lock_guard<Latch> lock(_mutex); - dassert(_count == 0); - dassert(count > 0); - _count = count; - } - - /** - * Count down an event. - */ - void countDown() { - stdx::lock_guard<Latch> lock(_mutex); - - if (_count > 0) { - --_count; - if (_count == 0) { - _condvar.notify_one(); - } - } - } - - /** - * Wait until the N events specified in reset have occured. - */ - void wait() { - stdx::unique_lock<Latch> lock(_mutex); - _condvar.wait(lock, [&] { return _count == 0; }); - } - -private: - // mutex to break count and cond var - Mutex _mutex = MONGO_MAKE_LATCH("FreeMonCountdownLatch::_mutex"); - - // cond var to signal and wait on - stdx::condition_variable _condvar; - - // count of events to wait for - size_t _count; -}; - -/** - * In-memory registration status - * - * Ensures primaries and secondaries register separately - */ -enum class FreeMonRegistrationStatus { - /** - * Free monitoring is not enabled - default state. - */ - kDisabled, - - /** - * Registration in progress. - */ - kPending, - - /** - * Free Monitoring is enabled. - */ - kEnabled, -}; - -/** - * Process in an Agent in a Agent/Message Passing model. - * - * Messages are given to it by enqueue, and the Processor processes messages with run(). - */ -class FreeMonProcessor : public std::enable_shared_from_this<FreeMonProcessor> { -public: - FreeMonProcessor(FreeMonCollectorCollection& registration, - FreeMonCollectorCollection& metrics, - FreeMonNetworkInterface* network, - bool useCrankForTest, - Seconds metricsGatherInterval); - - /** - * Enqueue a message to process - */ - void enqueue(std::shared_ptr<FreeMonMessage> msg); - - /** - * Stop processing messages. - */ - void stop(); - - /** - * Turn the crank of the message queue by ignoring deadlines for N messages. - */ - void turnCrankForTest(size_t countMessagesToIgnore); - - /** - * Deproritize the first message to force interleavings of messages. - */ - void deprioritizeFirstMessageForTest(FreeMonMessageType type); - - /** - * Processes messages forever - */ - void run(); - - /** - * Validate the registration response. Public for unit testing. - */ - static Status validateRegistrationResponse(const FreeMonRegistrationResponse& resp); - - /** - * Validate the metrics response. Public for unit testing. - */ - static Status validateMetricsResponse(const FreeMonMetricsResponse& resp); - -private: - /** - * Read the state from the database. - * - * Checks if the storage document has been delete locally or does not exist. If it is missing, - * generates a default disable state. - * - * If updateInMemory is true, update the state in memory with the state from disk. If false, do - * not update the state in memory from disk but instead treat the state in memory as - * authoritative. The is important for secondaries which may be in a different state for - * regsistration then there primary. - */ - void readState(OperationContext* opCtx, bool updateInMemory = true); - - /** - * Create a short-lived opCtx and read the state from the database. - */ - void readState(Client* client, bool updateInMemory = true); - - /** - * Write the state to disk if there are any changes. - */ - void writeState(Client* client); - - /** - * Process a registration from a command. - */ - void doCommandRegister(Client* client, std::shared_ptr<FreeMonMessage> sharedMsg); - - /** - * Process a registration from configuration. - */ - void doServerRegister(Client* client, - const FreeMonMessageWithPayload<FreeMonMessageType::RegisterServer>* msg); - - /** - * Process unregistration from a command. - */ - void doCommandUnregister( - Client* client, - FreeMonWaitableMessageWithPayload<FreeMonMessageType::UnregisterCommand>* msg); - - /** - * Process a successful HTTP request. - */ - void doAsyncRegisterComplete( - Client* client, - const FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterComplete>* msg); - - /** - * Process an unsuccessful HTTP request. - */ - void doAsyncRegisterFail( - Client* client, - const FreeMonMessageWithPayload<FreeMonMessageType::AsyncRegisterFail>* msg); - - /** - * Notify any command registers that are waiting. - */ - void notifyPendingRegisters(Status s); - - /** - * Upload collected metrics. - */ - void doMetricsCollect(Client* client); - - /** - * Upload gathered metrics. - */ - void doMetricsSend(Client* client); - - /** - * Process a successful HTTP request. - */ - void doAsyncMetricsComplete( - Client* client, - const FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsComplete>* msg); - - /** - * Process an unsuccessful HTTP request. - */ - void doAsyncMetricsFail( - Client* client, const FreeMonMessageWithPayload<FreeMonMessageType::AsyncMetricsFail>* msg); - - /** - * Process a change to become a replica set primary - */ - void doOnTransitionToPrimary(Client* client); - - /** - * Process a notification that storage has received insert or update. - */ - void doNotifyOnUpsert(Client* client, - const FreeMonMessageWithPayload<FreeMonMessageType::NotifyOnUpsert>* msg); - - /** - * Process a notification that storage has received delete or drop collection. - */ - void doNotifyOnDelete(Client* client); - - - /** - * Process a notification that storage has rolled back. - */ - void doNotifyOnRollback(Client* client); - - /** - * Process a in-memory state transition of state. - */ - void processInMemoryStateChange(const FreeMonStorageState& originalState, - const FreeMonStorageState& newState); - -protected: - friend class FreeMonController; - - enum FreeMonGetStatusEnum { - kServerStatus, - kCommandStatus, - }; - - /** - * Populate results for getFreeMonitoringStatus or serverStatus commands. - */ - void getStatus(OperationContext* opCtx, BSONObjBuilder* status, FreeMonGetStatusEnum mode); - -private: - // Collection of collectors to send on registration - FreeMonCollectorCollection& _registration; - - // Collection of collectors to send on each metrics call - FreeMonCollectorCollection& _metrics; - - // HTTP Network interface - FreeMonNetworkInterface* _network; - - // Random number generator for retries - PseudoRandom _random; - - // Registration Retry logic - synchronized_value<RegistrationRetryCounter> _registrationRetry; - - // Metrics Retry logic - synchronized_value<MetricsRetryCounter> _metricsRetry; - - // Interval for gathering metrics - Seconds _metricsGatherInterval; - - // Buffer of metrics to upload - MetricsBuffer _metricsBuffer; - - // When did we last send a metrics batch? - synchronized_value<boost::optional<Date_t>> _lastMetricsSend; - - // List of tags from server configuration registration - std::vector<std::string> _tags; - - // In-flight registration response - std::unique_ptr<Future<void>> _futureRegistrationResponse; - - // List of command registers waiting to be told about registration - std::vector<std::shared_ptr<FreeMonMessage>> _pendingRegisters; - - // Last read storage state - synchronized_value<boost::optional<FreeMonStorageState>> _lastReadState; - - // When we change to primary, do we register? - RegistrationType _registerOnTransitionToPrimary{RegistrationType::DoNotRegister}; - - // Pending update to disk - synchronized_value<FreeMonStorageState> _state; - - // In-memory registration status - FreeMonRegistrationStatus _registrationStatus{FreeMonRegistrationStatus::kDisabled}; - - // Countdown launch to support manual cranking - FreeMonCountdownLatch _countdown; - - // Message queue - FreeMonMessageQueue _queue; -}; - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_protocol.idl b/src/mongo/db/free_mon/free_mon_protocol.idl deleted file mode 100644 index 8e2e7899ed7..00000000000 --- a/src/mongo/db/free_mon/free_mon_protocol.idl +++ /dev/null @@ -1,149 +0,0 @@ -# Copyright (C) 2018-present MongoDB, Inc. -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the Server Side Public License, version 1, -# as published by MongoDB, Inc. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# Server Side Public License for more details. -# -# You should have received a copy of the Server Side Public License -# along with this program. If not, see -# <http://www.mongodb.com/licensing/server-side-public-license>. -# -# As a special exception, the copyright holders give permission to link the -# code of portions of this program with the OpenSSL library under certain -# conditions as described in each individual source file and distribute -# linked combinations including the program with the OpenSSL library. You -# must comply with the Server Side Public License in all respects for -# all of the code used other than as permitted herein. If you modify file(s) -# with this exception, you may extend this exception to your version of the -# file(s), but you are not obligated to do so. If you do not wish to do so, -# delete this exception statement from your version. If you delete this -# exception statement from all source files in the program, then also delete -# it in the license file. -# -global: - cpp_namespace: "mongo" - -imports: - - "mongo/idl/basic_types.idl" - - -enums: - MetricsEncoding: - description: "Metrics Encoding Methods" - type: string - values: - snappy: "snappy" - - -structs: - FreeMonRegistrationRequest: - description: "Registration Request to Cloud Server" - fields: - version: - description: "Protocol version, initial version is 1" - type: long - payload: - description: "Payload of registration information" - type: object - id: - description: "Existing Registration Id" - type: string - optional: true - localTime: - description: "Local time at registration send" - type: date - tags: - description: "Tags" - type: array<string> - optional: true - - FreeMonRegistrationResponse: - description: "Registration Response from Cloud Server" - fields: - version: - description: "Protocol version, initial version is 1" - type: long - haltMetricsUploading: - description: "True indicates it should not proceed to metrics uploading" - type: bool - id: - description: "Existing Registration Id" - type: string - informationalURL: - description: "Informational HTTP web page for metrics" - type: string - message: - description: "Informational message for shell to display to user" - type: string - reportingInterval: - description: "Metrics Reporting interval in seconds" - type: long - userReminder: - description: "Informational message to display to user to remind them about the service" - type: string - optional: true - - - FreeMonMetricsRequest: - description: "Metrics Request to Cloud Server" - fields: - version: - description: "Protocol version, initial version is 1" - type: long - id: - description: "Registration Id" - type: string - localTime: - description: "Local time at metrics send" - type: date - encoding: - description: "Compression Encoding" - type: MetricsEncoding - metrics: - description: "Metrics Blob" - type: bindata_generic - - # History - # ------- - # Version 2 - added resendRegistration bool - # - FreeMonMetricsResponse: - description: "Metrics Response from Cloud Server" - fields: - version: - description: "Protocol version, initial version is 1" - type: long - haltMetricsUploading: - description: "True indicates it should not proceed to metrics uploading" - type: bool - permanentlyDelete: - description: "True indicates it permanently delete the local state" - type: bool - reportingInterval: - description: "Metrics Reporting interval in seconds" - type: long - id: - description: "Existing Registration Id" - type: string - optional: true - message: - description: "Informational message for shell to display to user" - type: string - optional: true - informationalURL: - description: "Informational HTTP web page for metrics" - type: string - optional: true - userReminder: - description: "Message to display to user to remind them about service" - type: string - optional: true - resendRegistration: - description: "If true, resend registration to server" - type: bool - optional: true diff --git a/src/mongo/db/free_mon/free_mon_queue.cpp b/src/mongo/db/free_mon/free_mon_queue.cpp deleted file mode 100644 index 2e3582f663e..00000000000 --- a/src/mongo/db/free_mon/free_mon_queue.cpp +++ /dev/null @@ -1,234 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/platform/basic.h" - -#include "mongo/db/free_mon/free_mon_queue.h" - -#include <chrono> - -#include "mongo/util/concurrency/idle_thread_block.h" -#include "mongo/util/duration.h" - -namespace mongo { - -std::shared_ptr<FreeMonMessage> FreeMonPriorityQueue::top() const { - return _vector.front(); -} - -void FreeMonPriorityQueue::pop() { - std::pop_heap(_vector.begin(), _vector.end(), _comp); - _vector.pop_back(); -} - -void FreeMonPriorityQueue::push(std::shared_ptr<FreeMonMessage> item) { - _vector.push_back(item); - std::push_heap(_vector.begin(), _vector.end(), _comp); -} - -void FreeMonPriorityQueue::eraseByType(FreeMonMessageType type) { - - while (true) { - auto it = std::find_if(_vector.begin(), _vector.end(), [type](const auto& item) { - return item->getType() == type; - }); - - if (it == _vector.end()) { - break; - } - - _vector.erase(it); - } - - std::make_heap(_vector.begin(), _vector.end(), _comp); -} - - -FreeMonMessage::~FreeMonMessage() {} - -void FreeMonMessageQueue::enqueue(std::shared_ptr<FreeMonMessage> msg) { - { - stdx::lock_guard<Latch> lock(_mutex); - - // If we were stopped, drop messages - if (_stop) { - return; - } - - ++_counter; - msg->setId(_counter); - - if (msg->getType() == FreeMonMessageType::MetricsSend) { - _queue.eraseByType(FreeMonMessageType::MetricsSend); - } - - _queue.push(msg); - - // Signal the dequeue - _condvar.notify_one(); - } -} - -void FreeMonMessageQueue::deprioritizeFirstMessageForTest(FreeMonMessageType type) { - { - stdx::lock_guard<Latch> lock(_mutex); - - auto item = _queue.top(); - uassert(5167902, "Wrong message type", item->getType() == type); - - _queue.pop(); - - ++_counter; - item->setId(_counter); - _queue.push(item); - } -} - -boost::optional<std::shared_ptr<FreeMonMessage>> FreeMonMessageQueue::dequeue( - ClockSource* clockSource) { - { - stdx::unique_lock<Latch> lock(_mutex); - if (_stop) { - return {}; - } - - while (true) { - Date_t deadlineCV = Date_t::max(); - if (_useCrank) { - if (!_queue.empty() && _countMessagesIgnored < _countMessagesToIgnore) { - // For testing purposes, ignore the deadline - deadlineCV = Date_t(); - } else { - deadlineCV = clockSource->now() + Hours(1); - } - } else { - if (!_queue.empty()) { - deadlineCV = _queue.top()->getDeadline(); - } else { - deadlineCV = clockSource->now() + Hours(24); - } - } - - MONGO_IDLE_THREAD_BLOCK; - - _condvar.wait_until(lock, deadlineCV.toSystemTimePoint(), [this, clockSource]() { - if (_stop) { - return true; - } - - if (this->_queue.empty()) { - return false; - } - - // Always wake in test mode - if (_useCrank) { - if (_countMessagesIgnored < _countMessagesToIgnore) { - return true; - } else { - dassert(_countMessagesIgnored == _countMessagesToIgnore); - return false; - } - } - - auto deadlineMessage = this->_queue.top()->getDeadline(); - if (deadlineMessage <= Date_t()) { - return true; - } - - auto now = clockSource->now(); - - bool check = deadlineMessage < now; - return check; - }); - - if (_stop) { - return {}; - } - - // We were woken-up by a message being enqueue, go back to sleep and wait until crank is - // installed and turned. - if (_useCrank) { - if (_countMessagesIgnored == _countMessagesToIgnore) { - continue; - } - - dassert(_countMessagesIgnored <= _countMessagesToIgnore); - } - - // If the queue is not empty, return the message - // otherwise we need to go back to sleep in the hope we get a message. - if (!_queue.empty()) { - break; - } else if (_useCrank) { - dassert(0, "Was asked to wait for more messages then available"); - } - } - - _countMessagesIgnored++; - if (_useCrank && _countMessagesIgnored == _countMessagesToIgnore && _waitable) { - _waitable->set(Status::OK()); - } - - auto item = _queue.top(); - _queue.pop(); - return item; - } -} - -void FreeMonMessageQueue::stop() { - { - stdx::lock_guard<Latch> lock(_mutex); - - // We can be stopped twice in some situations: - // 1. Stop on unexpected error - // 2. Stop on clean shutdown - if (_stop == false) { - _stop = true; - _condvar.notify_one(); - } - } -} - -void FreeMonMessageQueue::turnCrankForTest(size_t countMessagesToIgnore) { - invariant(_useCrank); - - { - stdx::lock_guard<Latch> lock(_mutex); - - _waitable = std::make_unique<WaitableResult>(); - - _countMessagesIgnored = 0; - _countMessagesToIgnore = countMessagesToIgnore; - - _condvar.notify_one(); - } - - //_waitable->wait_for(Seconds(10)); -} -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_queue.h b/src/mongo/db/free_mon/free_mon_queue.h deleted file mode 100644 index de401e68841..00000000000 --- a/src/mongo/db/free_mon/free_mon_queue.h +++ /dev/null @@ -1,167 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include <boost/optional.hpp> -#include <memory> -#include <mutex> -#include <queue> -#include <vector> - -#include "mongo/db/free_mon/free_mon_message.h" -#include "mongo/util/clock_source.h" -#include "mongo/util/time_support.h" - -namespace mongo { - -/** - * Comparator for FreeMonMessage that will sort smallest deadlines at the beginning of a priority - * queue. The std::priority_queue is a max-heap. - */ -struct FreeMonMessageGreater { - bool operator()(const std::shared_ptr<FreeMonMessage>& left, - const std::shared_ptr<FreeMonMessage>& right) const { - if (left->getDeadline() > right->getDeadline()) { - return true; - } - - if (left->getDeadline() == right->getDeadline()) { - return left->getId() > right->getId(); - } - - return false; - } -}; - -/** - * Priority Queue with ability to remove items by filter. - */ -class FreeMonPriorityQueue { -public: - bool empty() const { - return _vector.empty(); - } - - /** - * Return the message at the top of the priority queue. - */ - std::shared_ptr<FreeMonMessage> top() const; - - /** - * Pop the message at the top of the priority queue. - */ - void pop(); - - /** - * Push a message into the priority queue. - */ - void push(std::shared_ptr<FreeMonMessage> item); - - /** - * Erase messages of a given type from the queue. - */ - void eraseByType(FreeMonMessageType type); - -private: - // Using shared_ptr because std::pop_heap does not support move-only types - std::vector<std::shared_ptr<FreeMonMessage>> _vector; - FreeMonMessageGreater _comp; -}; - -/** - * A multi-producer, single-consumer queue with deadlines. - * - * The smallest deadline sorts first. Messages with deadlines can be use as a timer mechanism. - */ -class FreeMonMessageQueue { -public: - FreeMonMessageQueue(bool useCrankForTest = false) : _useCrank(useCrankForTest) {} - - /** - * Enqueue a message and wake consumer if needed. - * - * Messages are dropped if the queue has been stopped. - */ - void enqueue(std::shared_ptr<FreeMonMessage> msg); - - /** - * Deque a message from the queue. - * - * Waits for a message to arrive. Returns boost::none if the queue has been stopped. - */ - boost::optional<std::shared_ptr<FreeMonMessage>> dequeue(ClockSource* clockSource); - - /** - * Stop the queue. - */ - void stop(); - - /** - * Turn the crank of the message queue by ignoring deadlines for N messages. - */ - void turnCrankForTest(size_t countMessagesToIgnore); - - /** - * Deproritize the first message to force interleavings of messages. - */ - void deprioritizeFirstMessageForTest(FreeMonMessageType type); - -private: - // Condition variable to signal consumer - stdx::condition_variable _condvar; - - // Lock for condition variable and to protect state - Mutex _mutex = MONGO_MAKE_LATCH("FreeMonMessageQueue::_mutex"); - - // Indicates whether queue has been stopped. - bool _stop{false}; - - // Priority queue of messages with shortest deadline first - FreeMonPriorityQueue _queue; - - // Use manual crank to process messages in-order instead of based on deadlines. - bool _useCrank{false}; - - // Stamp each message with a unique counter. This ensures that if two messages are queued with - // the same deadline, FIFO is achieved. - uint64_t _counter{0}; - - // Number of messages to ignore - size_t _countMessagesToIgnore{0}; - - // Number of messages that have been ignored - size_t _countMessagesIgnored{0}; - - // Waitable result for testing - std::unique_ptr<WaitableResult> _waitable; -}; - - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_queue_test.cpp b/src/mongo/db/free_mon/free_mon_queue_test.cpp deleted file mode 100644 index ad6104c5126..00000000000 --- a/src/mongo/db/free_mon/free_mon_queue_test.cpp +++ /dev/null @@ -1,169 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/platform/basic.h" - -#include "mongo/db/free_mon/free_mon_message.h" -#include "mongo/db/free_mon/free_mon_queue.h" -#include "mongo/db/service_context.h" -#include "mongo/db/service_context_d_test_fixture.h" -#include "mongo/executor/network_interface_mock.h" -#include "mongo/executor/thread_pool_task_executor.h" -#include "mongo/executor/thread_pool_task_executor_test_fixture.h" -#include "mongo/unittest/barrier.h" -#include "mongo/unittest/unittest.h" -#include "mongo/util/time_support.h" - -namespace mongo { -namespace { - -class FreeMonQueueTest : public ServiceContextMongoDTest { -private: - void setUp() final; - void tearDown() final; - -protected: - ServiceContext::UniqueOperationContext _opCtx; - - executor::NetworkInterfaceMock* _mockNetwork{nullptr}; - - std::unique_ptr<executor::ThreadPoolTaskExecutor> _mockThreadPool; -}; - -void FreeMonQueueTest::setUp() { - ServiceContextMongoDTest::setUp(); - - // Set up a NetworkInterfaceMock. Note, unlike NetworkInterfaceASIO, which has its own pool of - // threads, tasks in the NetworkInterfaceMock must be carried out synchronously by the (single) - // thread the unit test is running on. - auto netForFixedTaskExecutor = std::make_unique<executor::NetworkInterfaceMock>(); - _mockNetwork = netForFixedTaskExecutor.get(); - - // Set up a ThreadPoolTaskExecutor. Note, for local tasks this TaskExecutor uses a - // ThreadPoolMock, and for remote tasks it uses the NetworkInterfaceMock created above. However, - // note that the ThreadPoolMock uses the NetworkInterfaceMock's threads to run tasks, which is - // again just the (single) thread the unit test is running on. Therefore, all tasks, local and - // remote, must be carried out synchronously by the test thread. - _mockThreadPool = makeThreadPoolTestExecutor(std::move(netForFixedTaskExecutor)); - - _mockThreadPool->startup(); - - _opCtx = cc().makeOperationContext(); -} - -void FreeMonQueueTest::tearDown() { - _opCtx = {}; - - ServiceContextMongoDTest::tearDown(); -} - -// Postive: Can we enqueue and dequeue one item -TEST_F(FreeMonQueueTest, TestBasic) { - FreeMonMessageQueue queue; - - queue.enqueue(FreeMonMessage::createNow(FreeMonMessageType::RegisterServer)); - - auto item = queue.dequeue(_opCtx.get()->getServiceContext()->getPreciseClockSource()); - - ASSERT(item.get()->getType() == FreeMonMessageType::RegisterServer); -} - -Date_t fromNow(int millis) { - return getGlobalServiceContext()->getPreciseClockSource()->now() + Milliseconds(millis); -} - -// Positive: Ensure deadlines sort properly -TEST_F(FreeMonQueueTest, TestDeadlinePriority) { - FreeMonMessageQueue queue; - - queue.enqueue( - FreeMonMessage::createWithDeadline(FreeMonMessageType::RegisterServer, fromNow(5000))); - queue.enqueue( - FreeMonMessage::createWithDeadline(FreeMonMessageType::RegisterCommand, fromNow(50))); - - auto item = queue.dequeue(_opCtx.get()->getServiceContext()->getPreciseClockSource()).get(); - ASSERT(item->getType() == FreeMonMessageType::RegisterCommand); - - item = queue.dequeue(_opCtx.get()->getServiceContext()->getPreciseClockSource()).get(); - ASSERT(item->getType() == FreeMonMessageType::RegisterServer); -} - -// Positive: Ensure deadlines sort properly when they have the same deadlines -TEST_F(FreeMonQueueTest, TestFIFO) { - FreeMonMessageQueue queue; - - queue.enqueue(FreeMonMessage::createWithDeadline(FreeMonMessageType::RegisterServer, Date_t())); - queue.enqueue( - FreeMonMessage::createWithDeadline(FreeMonMessageType::AsyncRegisterComplete, Date_t())); - queue.enqueue( - FreeMonMessage::createWithDeadline(FreeMonMessageType::RegisterCommand, Date_t())); - - auto item = queue.dequeue(_opCtx.get()->getServiceContext()->getPreciseClockSource()).get(); - ASSERT(item->getType() == FreeMonMessageType::RegisterServer); - - item = queue.dequeue(_opCtx.get()->getServiceContext()->getPreciseClockSource()).get(); - ASSERT(item->getType() == FreeMonMessageType::AsyncRegisterComplete); - - item = queue.dequeue(_opCtx.get()->getServiceContext()->getPreciseClockSource()).get(); - ASSERT(item->getType() == FreeMonMessageType::RegisterCommand); -} - - -// Positive: Test Queue Stop -TEST_F(FreeMonQueueTest, TestQueueStop) { - FreeMonMessageQueue queue; - - queue.enqueue( - FreeMonMessage::createWithDeadline(FreeMonMessageType::RegisterServer, fromNow(50000))); - - unittest::Barrier barrier(2); - - auto swSchedule = - _mockThreadPool->scheduleWork([&](const executor::TaskExecutor::CallbackArgs& cbArgs) { - barrier.countDownAndWait(); - - // Try to dequeue from a stopped task queue - auto item = queue.dequeue(_opCtx.get()->getServiceContext()->getPreciseClockSource()); - ASSERT_FALSE(item.is_initialized()); - }); - - ASSERT_OK(swSchedule.getStatus()); - - // Stop the queue - queue.stop(); - - // Let our worker thread proceed - barrier.countDownAndWait(); - - _mockThreadPool->shutdown(); - _mockThreadPool->join(); -} - -} // namespace -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_status.cpp b/src/mongo/db/free_mon/free_mon_status.cpp deleted file mode 100644 index dc201956a8d..00000000000 --- a/src/mongo/db/free_mon/free_mon_status.cpp +++ /dev/null @@ -1,71 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/platform/basic.h" - -#include "mongo/db/commands/server_status.h" -#include "mongo/db/free_mon/free_mon_controller.h" -#include "mongo/db/free_mon/free_mon_options.h" - -namespace mongo { -namespace { - -class FreeMonServerStatus : public ServerStatusSection { -public: - FreeMonServerStatus() : ServerStatusSection("freeMonitoring") {} - - bool includeByDefault() const final { - return true; - } - - void addRequiredPrivileges(std::vector<Privilege>* out) final { - out->push_back(Privilege(ResourcePattern::forClusterResource(), - ActionType::checkFreeMonitoringStatus)); - } - - BSONObj generateSection(OperationContext* opCtx, const BSONElement& configElement) const final { - if (globalFreeMonParams.freeMonitoringState == EnableCloudStateEnum::kOff) { - return BSON("state" - << "disabled"); - } - - auto* controller = FreeMonController::get(opCtx->getServiceContext()); - if (!controller) { - return BSON("state" - << "disabled"); - } - - BSONObjBuilder builder; - controller->getServerStatus(opCtx, &builder); - return builder.obj(); - } -} freeMonServerStatus; - -} // namespace -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_storage.cpp b/src/mongo/db/free_mon/free_mon_storage.cpp deleted file mode 100644 index 89be39295e1..00000000000 --- a/src/mongo/db/free_mon/free_mon_storage.cpp +++ /dev/null @@ -1,142 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/db/free_mon/free_mon_storage.h" - -#include "mongo/base/status.h" -#include "mongo/base/string_data.h" -#include "mongo/bson/bsonelement.h" -#include "mongo/bson/bsonmisc.h" -#include "mongo/db/db_raii.h" -#include "mongo/db/namespace_string.h" -#include "mongo/db/operation_context.h" -#include "mongo/db/repl/replication_coordinator.h" -#include "mongo/db/repl/storage_interface.h" -#include "mongo/util/assert_util.h" - -namespace mongo { - -namespace { - -// mms-automation stores its document in local.clustermanager -static const NamespaceString localClusterManagerNss("local.clustermanager"); - -} // namespace - -constexpr StringData FreeMonStorage::kFreeMonDocIdKey; - -boost::optional<FreeMonStorageState> FreeMonStorage::read(OperationContext* opCtx) { - BSONObj deleteKey = BSON("_id" << kFreeMonDocIdKey); - BSONElement elementKey = deleteKey.firstElement(); - - auto storageInterface = repl::StorageInterface::get(opCtx); - - // Ensure we read without a timestamp. - invariant(RecoveryUnit::ReadSource::kNoTimestamp == - opCtx->recoveryUnit()->getTimestampReadSource()); - - AutoGetCollectionForRead autoRead(opCtx, NamespaceString::kServerConfigurationNamespace); - - auto swObj = storageInterface->findById( - opCtx, NamespaceString::kServerConfigurationNamespace, elementKey); - if (!swObj.isOK()) { - if (swObj.getStatus() == ErrorCodes::NoSuchKey || - swObj.getStatus() == ErrorCodes::NamespaceNotFound) { - return {}; - } - - uassertStatusOK(swObj.getStatus()); - } - - return FreeMonStorageState::parse(IDLParserErrorContext("FreeMonStorage"), swObj.getValue()); -} - -void FreeMonStorage::replace(OperationContext* opCtx, const FreeMonStorageState& doc) { - BSONObj deleteKey = BSON("_id" << kFreeMonDocIdKey); - BSONElement elementKey = deleteKey.firstElement(); - - BSONObj obj = doc.toBSON(); - - auto storageInterface = repl::StorageInterface::get(opCtx); - AutoGetCollection autoWrite(opCtx, NamespaceString::kServerConfigurationNamespace, MODE_IX); - - if (repl::ReplicationCoordinator::get(opCtx)->canAcceptWritesFor( - opCtx, NamespaceString::kServerConfigurationNamespace)) { - auto swObj = storageInterface->upsertById( - opCtx, NamespaceString::kServerConfigurationNamespace, elementKey, obj); - if (!swObj.isOK()) { - uassertStatusOK(swObj); - } - } -} - -void FreeMonStorage::deleteState(OperationContext* opCtx) { - BSONObj deleteKey = BSON("_id" << kFreeMonDocIdKey); - BSONElement elementKey = deleteKey.firstElement(); - - auto storageInterface = repl::StorageInterface::get(opCtx); - AutoGetCollection autoWrite(opCtx, NamespaceString::kServerConfigurationNamespace, MODE_IX); - - if (repl::ReplicationCoordinator::get(opCtx)->canAcceptWritesFor( - opCtx, NamespaceString::kServerConfigurationNamespace)) { - - auto swObj = storageInterface->deleteById( - opCtx, NamespaceString::kServerConfigurationNamespace, elementKey); - if (!swObj.isOK()) { - // Ignore errors about no document - if (swObj.getStatus() == ErrorCodes::NoSuchKey) { - return; - } - - uassertStatusOK(swObj); - } - } -} - -boost::optional<BSONObj> FreeMonStorage::readClusterManagerState(OperationContext* opCtx) { - auto storageInterface = repl::StorageInterface::get(opCtx); - - AutoGetCollectionForRead autoRead(opCtx, NamespaceString::kServerConfigurationNamespace); - - auto swObj = storageInterface->findSingleton(opCtx, localClusterManagerNss); - if (!swObj.isOK()) { - // Ignore errors about not-finding documents or having too many documents - if (swObj.getStatus() == ErrorCodes::NamespaceNotFound || - swObj.getStatus() == ErrorCodes::CollectionIsEmpty || - swObj.getStatus() == ErrorCodes::TooManyMatchingDocuments) { - return {}; - } - - uassertStatusOK(swObj.getStatus()); - } - - return swObj.getValue(); -} - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_storage.h b/src/mongo/db/free_mon/free_mon_storage.h deleted file mode 100644 index 8dffcc44068..00000000000 --- a/src/mongo/db/free_mon/free_mon_storage.h +++ /dev/null @@ -1,73 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#pragma once - -#include <boost/optional.hpp> - -#include "mongo/bson/bsonobj.h" -#include "mongo/db/free_mon/free_mon_storage_gen.h" -#include "mongo/db/operation_context.h" - -namespace mongo { - -/** - * Storage tier for Free Monitoring. Provides access to storage engine. - */ -class FreeMonStorage { -public: - /** - * The _id value in admin.system.version. - */ - static constexpr auto kFreeMonDocIdKey = "free_monitoring"_sd; - - /** - * Reads document from disk if it exists. - */ - static boost::optional<FreeMonStorageState> read(OperationContext* opCtx); - - /** - * Replaces document on disk with contents of document. Creates document if it does not exist. - */ - static void replace(OperationContext* opCtx, const FreeMonStorageState& doc); - - /** - * Deletes document on disk if it exists. - */ - static void deleteState(OperationContext* opCtx); - - /** - * Reads the singelton document from local.clustermanager. - * - * Returns nothing if there are more then one document or it does not exist. - */ - static boost::optional<BSONObj> readClusterManagerState(OperationContext* opCtx); -}; - -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_storage.idl b/src/mongo/db/free_mon/free_mon_storage.idl deleted file mode 100644 index 5c82bbeee2c..00000000000 --- a/src/mongo/db/free_mon/free_mon_storage.idl +++ /dev/null @@ -1,72 +0,0 @@ -# Copyright (C) 2018-present MongoDB, Inc. -# -# This program is free software: you can redistribute it and/or modify -# it under the terms of the Server Side Public License, version 1, -# as published by MongoDB, Inc. -# -# This program is distributed in the hope that it will be useful, -# but WITHOUT ANY WARRANTY; without even the implied warranty of -# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -# Server Side Public License for more details. -# -# You should have received a copy of the Server Side Public License -# along with this program. If not, see -# <http://www.mongodb.com/licensing/server-side-public-license>. -# -# As a special exception, the copyright holders give permission to link the -# code of portions of this program with the OpenSSL library under certain -# conditions as described in each individual source file and distribute -# linked combinations including the program with the OpenSSL library. You -# must comply with the Server Side Public License in all respects for -# all of the code used other than as permitted herein. If you modify file(s) -# with this exception, you may extend this exception to your version of the -# file(s), but you are not obligated to do so. If you do not wish to do so, -# delete this exception statement from your version. If you delete this -# exception statement from all source files in the program, then also delete -# it in the license file. -# -global: - cpp_namespace: "mongo" - -imports: - - "mongo/idl/basic_types.idl" - -enums: - StorageState: - description: "Action types" - type: string - values: - disabled: disabled - enabled: enabled - pending: pending - -structs: - FreeMonStorageState: - description: "Persisted document in admin.system.version" - strict: false - generate_comparison_operators: true - fields: - _id: - description: "Key of the Free Monitoring singleton document" - type: "string" - default: '"free_monitoring"' - version: - description: "Storage version, initial version is 1" - type: long - state: - description: "Indicates whether it is disabled or enabled" - type: StorageState - default: disabled - registrationId: - description: "Registration Id" - type: string - informationalURL: - description: "Informational HTTP web page for metrics" - type: string - message: - description: "Informational message for shell to display to user" - type: string - userReminder: - description: "Message to display to user to remind them about service" - type: string - diff --git a/src/mongo/db/free_mon/free_mon_storage_test.cpp b/src/mongo/db/free_mon/free_mon_storage_test.cpp deleted file mode 100644 index 06906389cdd..00000000000 --- a/src/mongo/db/free_mon/free_mon_storage_test.cpp +++ /dev/null @@ -1,272 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/platform/basic.h" - -#include "mongo/base/string_data.h" -#include "mongo/bson/bsonelement.h" -#include "mongo/bson/bsonmisc.h" -#include "mongo/db/catalog/collection_options.h" -#include "mongo/db/concurrency/d_concurrency.h" -#include "mongo/db/concurrency/lock_manager_defs.h" -#include "mongo/db/free_mon/free_mon_storage.h" -#include "mongo/db/namespace_string.h" -#include "mongo/db/repl/replication_coordinator_mock.h" -#include "mongo/db/repl/storage_interface.h" -#include "mongo/db/repl/storage_interface_impl.h" -#include "mongo/db/service_context.h" -#include "mongo/db/service_context_d_test_fixture.h" -#include "mongo/executor/network_interface_mock.h" -#include "mongo/executor/thread_pool_task_executor.h" -#include "mongo/executor/thread_pool_task_executor_test_fixture.h" -#include "mongo/unittest/unittest.h" -#include "mongo/util/uuid.h" - -namespace mongo { -namespace { - -class FreeMonStorageTest : public ServiceContextMongoDTest { -private: - void setUp() final; - void tearDown() final; - -protected: - /** - * Looks up the current ReplicationCoordinator. - * The result is cast to a ReplicationCoordinatorMock to provide access to test features. - */ - repl::ReplicationCoordinatorMock* _getReplCoord() const; - - ServiceContext::UniqueOperationContext _opCtx; - - executor::NetworkInterfaceMock* _mockNetwork{nullptr}; - - std::unique_ptr<executor::ThreadPoolTaskExecutor> _mockThreadPool; - - repl::StorageInterface* _storage{nullptr}; -}; - -void FreeMonStorageTest::setUp() { - ServiceContextMongoDTest::setUp(); - auto service = getServiceContext(); - - repl::ReplicationCoordinator::set(service, - std::make_unique<repl::ReplicationCoordinatorMock>(service)); - - _opCtx = cc().makeOperationContext(); - - repl::StorageInterface::set(service, std::make_unique<repl::StorageInterfaceImpl>()); - _storage = repl::StorageInterface::get(service); - - // Transition to PRIMARY so that the server can accept writes. - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_PRIMARY)); - - repl::createOplog(_opCtx.get()); -} - -void FreeMonStorageTest::tearDown() { - _opCtx = {}; - ServiceContextMongoDTest::tearDown(); -} - -repl::ReplicationCoordinatorMock* FreeMonStorageTest::_getReplCoord() const { - auto replCoord = repl::ReplicationCoordinator::get(_opCtx.get()); - ASSERT(replCoord) << "No ReplicationCoordinator installed"; - auto replCoordMock = dynamic_cast<repl::ReplicationCoordinatorMock*>(replCoord); - ASSERT(replCoordMock) << "Unexpected type for installed ReplicationCoordinator"; - return replCoordMock; -} - -// Positive: Test Storage works -TEST_F(FreeMonStorageTest, TestStorage) { - - // Validate no collection works - { - auto emptyDoc = FreeMonStorage::read(_opCtx.get()); - ASSERT_FALSE(emptyDoc.is_initialized()); - } - - // Create collection with one document. - CollectionOptions collectionOptions; - collectionOptions.uuid = UUID::gen(); - auto statusCC = _storage->createCollection( - _opCtx.get(), NamespaceString("admin", "system.version"), collectionOptions); - ASSERT_OK(statusCC); - - - FreeMonStorageState initialState = - FreeMonStorageState::parse(IDLParserErrorContext("foo"), - BSON("version" << 1LL << "state" - << "enabled" - << "registrationId" - << "1234" - << "informationalURL" - << "http://example.com" - << "message" - << "hello" - << "userReminder" - << "")); - - { - auto emptyDoc = FreeMonStorage::read(_opCtx.get()); - ASSERT_FALSE(emptyDoc.is_initialized()); - } - - FreeMonStorage::replace(_opCtx.get(), initialState); - - { - auto persistedDoc = FreeMonStorage::read(_opCtx.get()); - - ASSERT_TRUE(persistedDoc.is_initialized()); - - ASSERT_TRUE(persistedDoc == initialState); - } - - FreeMonStorage::deleteState(_opCtx.get()); - - { - auto emptyDoc = FreeMonStorage::read(_opCtx.get()); - ASSERT_FALSE(emptyDoc.is_initialized()); - } - - // Verfiy delete of nothing succeeds - FreeMonStorage::deleteState(_opCtx.get()); -} - - -// Positive: Test Storage works on a secondary -TEST_F(FreeMonStorageTest, TestSecondary) { - - // Create collection with one document. - CollectionOptions collectionOptions; - collectionOptions.uuid = UUID::gen(); - auto statusCC = _storage->createCollection( - _opCtx.get(), NamespaceString("admin", "system.version"), collectionOptions); - ASSERT_OK(statusCC); - - - FreeMonStorageState initialState = - FreeMonStorageState::parse(IDLParserErrorContext("foo"), - BSON("version" << 1LL << "state" - << "enabled" - << "registrationId" - << "1234" - << "informationalURL" - << "http://example.com" - << "message" - << "hello" - << "userReminder" - << "")); - - FreeMonStorage::replace(_opCtx.get(), initialState); - - { - auto persistedDoc = FreeMonStorage::read(_opCtx.get()); - - ASSERT_TRUE(persistedDoc.is_initialized()); - - ASSERT_TRUE(persistedDoc == initialState); - } - - // Now become a secondary - ASSERT_OK(_getReplCoord()->setFollowerMode(repl::MemberState::RS_SECONDARY)); - - FreeMonStorageState updatedState = - FreeMonStorageState::parse(IDLParserErrorContext("foo"), - BSON("version" << 2LL << "state" - << "enabled" - << "registrationId" - << "1234" - << "informationalURL" - << "http://example.com" - << "message" - << "hello" - << "userReminder" - << "")); - - - { - auto persistedDoc = FreeMonStorage::read(_opCtx.get()); - - ASSERT_TRUE(persistedDoc.is_initialized()); - - ASSERT_TRUE(persistedDoc == initialState); - } - - FreeMonStorage::deleteState(_opCtx.get()); - - { - auto persistedDoc = FreeMonStorage::read(_opCtx.get()); - ASSERT_TRUE(persistedDoc.is_initialized()); - } - - // Verfiy delete of nothing succeeds - FreeMonStorage::deleteState(_opCtx.get()); -} - -void insertDoc(OperationContext* optCtx, const NamespaceString nss, StringData id) { - auto storageInterface = repl::StorageInterface::get(optCtx); - - Lock::DBLock dblk(optCtx, nss.db(), MODE_IX); - Lock::CollectionLock lk(optCtx, nss, MODE_IX); - - BSONObj fakeDoc = BSON("_id" << id); - BSONElement elementKey = fakeDoc.firstElement(); - - ASSERT_OK(storageInterface->upsertById(optCtx, nss, elementKey, fakeDoc)); -} - -// Positive: Test local.clustermanager -TEST_F(FreeMonStorageTest, TestClusterManagerStorage) { - const NamespaceString localClusterManagerNss("local.clustermanager"); - - // Verify read of non-existent collection works - ASSERT_FALSE(FreeMonStorage::readClusterManagerState(_opCtx.get()).is_initialized()); - - CollectionOptions collectionOptions; - collectionOptions.uuid = UUID::gen(); - auto statusCC = - _storage->createCollection(_opCtx.get(), localClusterManagerNss, collectionOptions); - ASSERT_OK(statusCC); - - // Verify read of empty collection works - ASSERT_FALSE(FreeMonStorage::readClusterManagerState(_opCtx.get()).is_initialized()); - - insertDoc(_opCtx.get(), localClusterManagerNss, "foo1"); - - // Verify read of singleton collection works - ASSERT_TRUE(FreeMonStorage::readClusterManagerState(_opCtx.get()).is_initialized()); - - insertDoc(_opCtx.get(), localClusterManagerNss, "bar1"); - - // Verify read of two doc collection fails - ASSERT_FALSE(FreeMonStorage::readClusterManagerState(_opCtx.get()).is_initialized()); -} -} // namespace -} // namespace mongo diff --git a/src/mongo/db/free_mon/free_mon_stub.cpp b/src/mongo/db/free_mon/free_mon_stub.cpp deleted file mode 100644 index af50d7ce7df..00000000000 --- a/src/mongo/db/free_mon/free_mon_stub.cpp +++ /dev/null @@ -1,44 +0,0 @@ -/** - * Copyright (C) 2018-present MongoDB, Inc. - * - * This program is free software: you can redistribute it and/or modify - * it under the terms of the Server Side Public License, version 1, - * as published by MongoDB, Inc. - * - * This program is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the - * Server Side Public License for more details. - * - * You should have received a copy of the Server Side Public License - * along with this program. If not, see - * <http://www.mongodb.com/licensing/server-side-public-license>. - * - * As a special exception, the copyright holders give permission to link the - * code of portions of this program with the OpenSSL library under certain - * conditions as described in each individual source file and distribute - * linked combinations including the program with the OpenSSL library. You - * must comply with the Server Side Public License in all respects for - * all of the code used other than as permitted herein. If you modify file(s) - * with this exception, you may extend this exception to your version of the - * file(s), but you are not obligated to do so. If you do not wish to do so, - * delete this exception statement from your version. If you delete this - * exception statement from all source files in the program, then also delete - * it in the license file. - */ - -#include "mongo/db/free_mon/free_mon_mongod.h" - -#include "mongo/db/service_context.h" - -namespace mongo { - -void startFreeMonitoring(ServiceContext* serviceContext) {} - -void stopFreeMonitoring() {} - -void notifyFreeMonitoringOnTransitionToPrimary(){}; - -void setupFreeMonitoringOpObserver(OpObserverRegistry* registry) {} - -} // namespace mongo |
