Skip to content

clarify & correct Replicator target schema requirements #20859

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

clarify & correct Replicator target schema requirements #20859

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/current/_includes/molt/molt-setup.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,12 @@ When you run `molt fetch`, you can configure the following options for data load
- [Data load mode](#data-load-mode): Choose between `IMPORT INTO` and `COPY FROM`.
- [Fetch metrics](#fetch-metrics): Configure metrics collection during initial data load.

<div class="filters filters-big clearfix">
<button class="filter-button" data-scope="postgres">PostgreSQL</button>
<button class="filter-button" data-scope="mysql">MySQL</button>
<button class="filter-button" data-scope="oracle">Oracle</button>
</div>

### Connection strings

{% include molt/molt-connection-strings.md %}
Expand Down
24 changes: 12 additions & 12 deletions src/current/_includes/molt/replicator-flags-usage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ The following [MOLT Replicator]({% link molt/molt-replicator.md %}) flags are **
| Flag | Description |
|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--slotName` | **Required.** PostgreSQL replication slot name. Must match the slot name specified with `--pglogical-replication-slot-name` in the [MOLT Fetch command](#start-fetch). |
| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. |
| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. Schema name must be fully qualified in the format `database.schema`. |
| `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. |
| `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. |
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
Expand All @@ -14,7 +14,7 @@ The following [MOLT Replicator]({% link molt/molt-replicator.md %}) flags are **
<section class="filter-content" markdown="1" data-scope="mysql">
| Flag | Description |
|-------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. |
| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. Schema name must be fully qualified in the format `database.schema`. |
| `--defaultGTIDSet` | **Required.** Default GTID set for changefeed. |
| `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. |
| `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. |
Expand All @@ -26,16 +26,16 @@ You can find the starting GTID in the `cdc_cursor` field of the `fetch complete`
</section>

<section class="filter-content" markdown="1" data-scope="oracle">
| Flag | Description |
|-------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|
| `--sourceSchema` | **Required.** Source schema name on Oracle where tables will be replicated from. |
| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. |
| `--scn` | **Required.** Snapshot System Change Number (SCN) for the initial changefeed starting point. |
| `--backfillFromSCN` | **Required.** SCN of the earliest active transaction at the time of the snapshot. Ensures no transactions are skipped. |
| `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. |
| `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. |
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
| `--userscript` | Path to a userscript that enables table filtering from Oracle sources. Refer to [Table filter userscript](#table-filter-userscript). |
| Flag | Description |
|-------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------|
| `--sourceSchema` | **Required.** Source schema name on Oracle where tables will be replicated from. |
| `--targetSchema` | **Required.** Target schema name on CockroachDB where tables will be replicated. Schema name must be fully qualified in the format `database.schema`. |
| `--scn` | **Required.** Snapshot System Change Number (SCN) for the initial changefeed starting point. |
| `--backfillFromSCN` | **Required.** SCN of the earliest active transaction at the time of the snapshot. Ensures no transactions are skipped. |
| `--stagingSchema` | **Required.** Staging schema name for replication metadata and checkpoints. |
| `--stagingCreateSchema` | **Required.** Automatically create the staging schema if it does not exist. Include this flag when starting replication for the first time. |
| `--metricsAddr` | Enable Prometheus metrics at a specified `{host}:{port}`. Metrics are served at `http://{host}:{port}/_/varz`. |
| `--userscript` | Path to a userscript that enables table filtering from Oracle sources. Refer to [Table filter userscript](#table-filter-userscript). |

You can find the SCN values in the message `replication-only mode should include the following replicator flags` after the [initial data load](#start-fetch) completes.
</section>
Expand Down
2 changes: 1 addition & 1 deletion src/current/_includes/molt/replicator-flags.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@
| `--targetJitterTime` | `DURATION` | The time over which to jitter database pool disconnections.<br><br>**Default:** `15s` |
| `--targetMaxLifetime` | `DURATION` | The maximum lifetime of a database connection.<br><br>**Default:** `5m0s` |
| `--targetMaxPoolSize` | `INT` | The maximum number of target database connections.<br><br>**Default:** `128` |
| `--targetSchema` | `STRING` | The SQL database schema in the target cluster to update. |
| `--targetSchema` | `STRING` | The SQL database schema in the target cluster to update. CockroachDB schema names must be fully qualified in the format `database.schema`. |
Copy link

@Jeremyyang920 Jeremyyang920 Oct 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking at this now, i don't think this should be part of Global Flags since its really only for X2C

| `--targetStatementCacheSize` | `INT` | The maximum number of prepared statements to retain.<br><br>**Default:** `128` |
| `--taskGracePeriod` | `DURATION` | How long to allow for task cleanup when recovering from errors.<br><br>**Default:** `1m0s` |
| `--timestampLimit` | `INT` | The maximum number of source timestamps to coalesce into a target transaction.<br><br>**Default:** `1000` |
Expand Down
26 changes: 22 additions & 4 deletions src/current/molt/migrate-failback.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,12 @@ When you run `replicator`, you can configure the following options for replicati
</section>
- [Replicator metrics](#replicator-metrics): Monitor failback replication performance.

<div class="filters filters-big clearfix">
<button class="filter-button" data-scope="postgres">PostgreSQL</button>
<button class="filter-button" data-scope="mysql">MySQL</button>
<button class="filter-button" data-scope="oracle">Oracle</button>
</div>

### Connection strings

For failback, MOLT Replicator uses `--targetConn` to specify the destination database where you want to replicate CockroachDB changes, and `--stagingConn` for the CockroachDB staging database.
Expand Down Expand Up @@ -225,11 +231,19 @@ Create a CockroachDB changefeed to send changes to MOLT Replicator.

1. Create the CockroachDB changefeed pointing to the MOLT Replicator webhook endpoint. Use `cursor` to specify the logical timestamp from the preceding step.

{{site.data.alerts.callout_info}}
Ensure that only **one** changefeed points to MOLT Replicator at a time to avoid mixing streams of incoming data.
{{site.data.alerts.end}}
<section class="filter-content" markdown="1" data-scope="postgres">
The target schema is specified in the webhook URL path in the fully-qualified format `/database/schema`. For example, `/migration_schema/public` routes changes to the `public` schema in the `migration_schema` database.
</section>

{{site.data.alerts.callout_success}}
<section class="filter-content" markdown="1" data-scope="mysql">
The target schema is specified in the webhook URL path. For example, `/migration_schema` routes changes to the `migration_schema` schema.
</section>

<section class="filter-content" markdown="1" data-scope="oracle">
The target schema is specified in the webhook URL path. For example, `/MIGRATION_SCHEMA` routes changes to the `MIGRATION_SCHEMA` schema.
</section>

{{site.data.alerts.callout_info}}
For details on the webhook sink URI, refer to [Webhook sink]({% link {{ site.current_cloud_version }}/changefeed-sinks.md %}#webhook-sink).
{{site.data.alerts.end}}

Expand Down Expand Up @@ -268,6 +282,10 @@ Create a CockroachDB changefeed to send changes to MOLT Replicator.
1101234051444375553
~~~

{{site.data.alerts.callout_success}}
Ensure that only **one** changefeed points to MOLT Replicator at a time to avoid mixing streams of incoming data.
{{site.data.alerts.end}}

1. Monitor the changefeed status, specifying the job ID:

~~~ sql
Expand Down
6 changes: 6 additions & 0 deletions src/current/molt/migrate-load-replicate.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,12 @@ When you run `replicator`, you can configure the following options for replicati
</section>
- [Replicator metrics](#replicator-metrics): Monitor replication progress and performance.

<div class="filters filters-big clearfix">
<button class="filter-button" data-scope="postgres">PostgreSQL</button>
<button class="filter-button" data-scope="mysql">MySQL</button>
<button class="filter-button" data-scope="oracle">Oracle</button>
</div>

### Replication connection strings

MOLT Replicator uses `--sourceConn` and `--targetConn` to specify the source and target database connections.
Expand Down
4 changes: 3 additions & 1 deletion src/current/molt/migrate-resume-replication.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,10 +70,12 @@ replicator oraclelogminer \
--sourceConn $SOURCE \
--sourcePDBConn $SOURCE_PDB \
--sourceSchema migration_schema \
--targetSchema defaultdb.public \
--targetConn $TARGET \
--stagingSchema _replicator \
--metricsAddr :30005 \
--userscript table_filter.ts
--userscript table_filter.ts \
-v
~~~

{{site.data.alerts.callout_info}}
Expand Down
22 changes: 18 additions & 4 deletions src/current/molt/molt-replicator.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -269,8 +269,8 @@ These server certificates must correspond to the client certificates specified i

Encode client certificates for changefeed webhook URLs:

- **Webhook URLs**: Use both URL encoding and base64 encoding: `base64 -i ./client.crt | jq -R -r '@uri'`
- **Non-webhook contexts**: Use base64 encoding only: `base64 -w 0 ca.cert`
- Webhook URLs: Use both URL encoding and base64 encoding: `base64 -i ./client.crt | jq -R -r '@uri'`
- Non-webhook contexts: Use base64 encoding only: `base64 -w 0 ca.cert`

#### JWT authentication

Expand Down Expand Up @@ -430,6 +430,13 @@ Specify the source Oracle schema to replicate from:
~~~
</section>

Specify the target schema on CockroachDB in fully-qualified `database.schema` format:

{% include_cached copy-clipboard.html %}
~~~
--targetSchema defaultdb.public
~~~

To replicate from the correct position, specify the appropriate checkpoint value.

<section class="filter-content" markdown="1" data-scope="postgres">
Expand Down Expand Up @@ -476,6 +483,7 @@ At minimum, the `replicator` command should include the following flags:
replicator pglogical \
--sourceConn $SOURCE \
--targetConn $TARGET \
--targetSchema defaultdb.public \
--slotName molt_slot \
--stagingSchema _replicator \
--stagingCreateSchema
Expand All @@ -490,6 +498,7 @@ For detailed steps, refer to [Load and replicate]({% link molt/migrate-load-repl
replicator mylogical \
--sourceConn $SOURCE \
--targetConn $TARGET \
--targetSchema defaultdb.public \
--defaultGTIDSet '4c658ae6-e8ad-11ef-8449-0242ac140006:1-29' \
--stagingSchema _replicator \
--stagingCreateSchema
Expand All @@ -504,8 +513,9 @@ For detailed steps, refer to [Load and replicate]({% link molt/migrate-load-repl
replicator oraclelogminer \
--sourceConn $SOURCE \
--sourcePDBConn $SOURCE_PDB \
--sourceSchema migration_schema \
--targetConn $TARGET \
--sourceSchema migration_schema \
--targetSchema defaultdb.public \
--scn 26685786 \
--backfillFromSCN 26685444 \
--stagingSchema _replicator \
Expand Down Expand Up @@ -626,7 +636,11 @@ replicator start \
--tlsPrivateKey ./certs/server.key
~~~

For detailed steps, refer to [Migration failback]({% link molt/migrate-failback.md %}).
After starting `replicator`, create a CockroachDB changefeed to send changes to MOLT Replicator. For detailed steps, refer to [Migration failback]({% link molt/migrate-failback.md %}).

{{site.data.alerts.callout_info}}
When [creating the CockroachDB changefeed]({% link molt/migrate-failback.md %}#create-the-cockroachdb-changefeed) , you specify the target schema in the webhook URL path. For PostgreSQL targets, use the fully-qualified format `/database/schema` (for example, `/migration_schema/public`). For MySQL and Oracle targets, use just the schema name (for example, `/migration_schema`).
{{site.data.alerts.end}}

## Monitoring

Expand Down
Loading