Percona ClusterSync for MongoDB

Percona ClusterSync for MongoDB is a tool for replicating data from a source MongoDB cluster to a target MongoDB cluster. It supports cloning data, replicating changes, and managing collections and indexes.

Features

• Clone: Instantly transfer existing data from a source MongoDB to a target MongoDB.
• Real-Time Replication: Tail the oplog to keep your target cluster up to date.
• Namespace Filtering: Specify which databases and collections to include or exclude.
• Automatic Index Management: Ensure necessary indexes are created on the target.
• HTTP API: Start, finalize, pause, resume, and check replication status via REST endpoints.

Setup

Prerequisites

• Go 1.24 or later
• MongoDB 6.0 or later
• Python 3.13 or later (for testing)
• Poetry (for managing Python dependencies)

Installation

1. Clone the repository:

   git clone https://github.com/percona/percona-clustersync-mongodb.git
   cd percona-clustersync-mongodb

2. Build the project using the Makefile:

   make build

   Alternatively, you can install PCSM from the cloned repo using go install:

   go install .

   This will install pcsm into your GOBIN directory. If GOBIN is included in your PATH, you can run Percona ClusterSync for MongoDB by typing pcsm in your terminal.

3. Run the server:

   bin/pcsm --source <source-mongodb-uri> --target <target-mongodb-uri>

   Alternatively, you can use environment variables:

   export PCSM_SOURCE_URI=<source-mongodb-uri>
   export PCSM_TARGET_URI=<target-mongodb-uri>
   bin/pcsm

Usage

Starting the Replication

To start the replication process, you can either use the command-line interface or send a POST request to the /start endpoint with the desired options:

Using Command-Line Interface

bin/pcsm start

Using HTTP API

curl -X POST http://localhost:2242/start -d '{
    "includeNamespaces": ["db1.collection1", "db2.collection2"],
    "excludeNamespaces": ["db3.collection3", "db4.*"]
}'

Finalizing the Replication

To finalize the replication process, you can either use the command-line interface or send a POST request to the /finalize endpoint:

Using Command-Line Interface

bin/pcsm finalize

Using HTTP API

curl -X POST http://localhost:2242/finalize

Pausing the Replication

To pause the replication process, you can either use the command-line interface or send a POST request to the /pause endpoint:

Using Command-Line Interface

bin/pcsm pause

Using HTTP API

curl -X POST http://localhost:2242/pause

Resuming the Replication

To resume the replication process, you can either use the command-line interface or send a POST request to the /resume endpoint:

Using Command-Line Interface

bin/pcsm resume

Using HTTP API

curl -X POST http://localhost:2242/resume

Checking the Status

To check the current status of the replication process, you can either use the command-line interface or send a GET request to the /status endpoint:

Using Command-Line Interface

bin/pcsm status

Using HTTP API

curl http://localhost:2242/status

PCSM Options

When starting the PCSM server, you can use the following options:

• --port: The port on which the server will listen (default: 2242)
• --source: The MongoDB connection string for the source cluster
• --target: The MongoDB connection string for the target cluster
• --log-level: The log level (default: "info")
• --log-json: Output log in JSON format with disabled color
• --no-color: Disable log ASCI color

Example:

bin/pcsm \
    --source <source-mongodb-uri> \
    --target <target-mongodb-uri> \
    --port 2242 \
    --log-level debug \
    --log-json

Environment Variables

• PCSM_SOURCE_URI: MongoDB connection string for the source cluster.
• PCSM_TARGET_URI: MongoDB connection string for the target cluster.
• PCSM_MONGODB_CLI_OPERATION_TIMEOUT: Timeout for MongoDB client operations; accepts Go durations like 30s, 2m, 1h (default: 5m).

Log JSON Fields

When using the --log-json option, the logs will be output in JSON format with the following fields:

• time: Unix time when the log entry was created.
• level: Log level (e.g., "debug", "info", "warn", "error").
• message: Log message, if any.
• error: Error message, if any.
• s: Scope of the log entry.
• ns: Namespace (database.collection format).
• elapsed_secs: The duration in seconds for the specific operation to complete.

Example:

{ "level": "info",
  "s": "clone",
  "ns": "db_1.coll_1",
  "elapsed_secs": 0,
  "time": "2025-02-23 11:26:03.758",
  "message": "Cloned db_1.coll_1" }

{ "level": "info",
  "s": "pcsm",
  "elapsed_secs": 0,
  "time": "2025-02-23 11:26:03.857",
  "message": "Change replication stopped at 1740335163.1740335163 source cluster time" }

HTTP API

POST /start

Starts the replication process.

Request Body

• includeNamespaces (optional): List of namespaces to include in the replication.
• excludeNamespaces (optional): List of namespaces to exclude from the replication.

Example:

{
    "includeNamespaces": ["dbName.*", "anotherDB.collName1", "anotherDB.collName2"],
    "excludeNamespaces": ["dbName.collName"]
}

Response

• ok: Boolean indicating if the operation was successful.
• error (optional): Error message if the operation failed.

Example:

{ "ok": true }

POST /finalize

Finalizes the replication process.

Response

• ok: Boolean indicating if the operation was successful.
• error (optional): Error message if the operation failed.

Example:

{ "ok": true }

POST /pause

Pauses the replication process.

Response

• ok: Boolean indicating if the operation was successful.
• error (optional): Error message if the operation failed.

Example:

{ "ok": true }

POST /resume

Resumes the replication process.

Request Body

• fromFailure (optional): Allows PCSM to resume from failed state

Example:

{
    "fromFailure": true
}

Response

• ok: Boolean indicating if the operation was successful.
• error (optional): Error message if the operation failed.

Example:

{ "ok": true }

GET /status

The /status endpoint provides the current state of the PCSM replication process, including its progress, lag, and event processing details.

Response

• ok: indicates if the operation was successful.

• state: the current state of the replication.

• info: provides additional information about the current state.

• error (optional): the error message if the operation failed.

• lagTime: the current lag time in logical seconds between source and target clusters.

• eventsProcessed: the number of events processed.

• lastReplicatedOpTime: the last replicated operation time.

• initialSync.completed: indicates if the initial sync is completed.

• initialSync.lagTime: the lag time in logical seconds until the initial sync completed.

• initialSync.cloneCompleted: indicates if the cloning process is completed.

• initialSync.estimatedCloneSize: the estimated total size of the clone.

• initialSync.clonedSize: the size of the data that has been cloned.

Example:

{
    "ok": true,
    "state": "running",
    "info": "Initial Sync",

    "lagTime": 22,
    "eventsProcessed": 5000,
    "lastReplicatedOpTime": "1740335200.5",

    "initialSync": {
        "completed": false,
        "lagTime": 5,

        "cloneCompleted": false,
        "estimatedCloneSize": 5000000000,
        "clonedSize": 2500000000
    }
}

Testing

Prerequisites

• Install Poetry:

  curl -sSL https://install.python-poetry.org | python3 -

• Install the required Python packages:

  poetry install

Build for Testing

To build the project for testing, use the following command:

make test-build

Running Tests

To run the tests, use the following command:

poetry run pytest \
    --source-uri <source-mongodb-uri> \
    --target-uri <target-mongodb-uri> \
    --pcsm_url http://localhost:2242 \
    --pcsm-bin bin/pcsm_test

Alternatively, you can use environment variables:

export TEST_SOURCE_URI=<source-mongodb-uri>
export TEST_TARGET_URI=<target-mongodb-uri>
export TEST_PCSM_URL=http://localhost:2242
export TEST_PCSM_BIN=bin/pcsm_test
poetry run pytest

The --pcsm-bin flag or TEST_PCSM_BIN environment variable specifies the path to the PCSM binary. This allows the test suite to manage the PCSM process, ensuring it starts and stops as needed during the tests. If neither the flag nor the environment variable is provided, you must run PCSM externally before running the tests.

Contributing

Contributions are welcome. Please open a JIRA issue describing the proposed change, then submit a pull request on GitHub.

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.