MOLT Replicator Metrics

• Staging: Buffers mutations for ordered processing and crash recovery.
• Core sequencer: Processes staged mutations, maintains ordering guarantees, and coordinates transaction application.
• Target apply: Applies mutations to the target database.

​

Set up metrics

replicator start \
--targetConn $TARGET \
--stagingConn $STAGING \
--metricsAddr :30005
...

​

Metrics endpoints

curl http://localhost:30005/_/healthz

OK

​

Visualize metrics

​

Overall replication metrics

​

High-level performance metrics

• core_source_lag_seconds
  • Description: Age of the most recently received checkpoint. This represents the time from source commit to COMMIT event processing.
  • Interpretation: If consistently increasing, Replicator is falling behind in reading source changes, and cannot keep pace with database changes.
• target_apply_mutation_age_seconds
  • Description: End-to-end replication lag per mutation from source commit to target apply. Measures the difference between current wall time and the mutation’s .
  • Interpretation: Higher values mean that older mutations are being applied, and indicate end-to-end pipeline delays. Compare across tables to find bottlenecks.
• target_apply_queue_utilization_percent
  • Description: Percentage of target apply queue capacity utilization.
  • Interpretation: Values above 90 percent indicate severe backpressure throughout the pipeline, and potential data processing delays. Increase or investigate target database performance.

​

Replication lag

• source_commit_to_apply_lag_seconds
  • Description: Time delta between writing a mutation to the source and writing it to the target.
  • Interpretation: This may indicate the duration of a due to drainage. Low values (seconds or hundreds of milliseconds) would allow for minimal downtime on cutover.
• target_apply_transaction_lag_seconds
  • Description: Age of the transaction applied to the target table, measuring time from source commit to target apply.
  • Interpretation: Consistently high values indicate bottlenecks in the pipeline. Compare with core_source_lag_seconds to determine if the delay is in source read or target apply.

​

Replication pipeline metrics

​

Source read

• pglogical_dial_success_total
  • Description: Number of times Replicator successfully started logical replication (executed START_REPLICATION command).
  • Interpretation: Multiple successes may indicate reconnects. Monitor for connection stability.
• pglogical_dial_failure_total
  • Description: Number of times Replicator failed to start logical replication (failure to execute START_REPLICATION command).
  • Interpretation: Nonzero values indicate connection issues. Check network connectivity and source database health.
• mutation_total
  • Description: Total number of mutations processed, labeled by source and mutation type (insert/update/delete).
  • Interpretation: Use to monitor replication throughput and identify traffic patterns.
• pglogical_filtered_deletions_total
  • Description: Number of DELETE mutations filtered out before buffering when using the flag.
  • Interpretation: Use to verify that deletions are being filtered as expected for tables specified with --ignoreDeletionsForTable.

​

Core sequencer

• core_sweep_duration_seconds
  • Description: Duration of each schema sweep operation, which looks for and applies staged mutations.
  • Interpretation: Long durations indicate that large backlogs, slow staging reads, or slow target writes are affecting throughput.
• core_sweep_mutations_applied_total
  • Description: Total count of mutations read from staging and successfully applied to the target database during a sweep.
  • Interpretation: Use to monitor processing throughput. A flat line indicates no mutations are being applied.
• core_sweep_success_timestamp_seconds
  • Description: Wall time (Unix timestamp) at which a sweep attempt last succeeded.
  • Interpretation: If this value stops updating and becomes stale, it indicates that the sweep has stopped.
• core_parallelism_utilization_percent
  • Description: Percentage of the configured parallelism that is actively being used for concurrent transaction processing.
  • Interpretation: High utilization indicates bottlenecks in mutation processing.

​

Target apply

• target_apply_queue_size
  • Description: Number of transactions waiting in the target apply queue.
  • Interpretation: High values indicate target apply cannot keep up with incoming transactions.
• apply_duration_seconds
  • Description: Amount of time taken to successfully apply mutations to a table.
  • Interpretation: High values indicate target database performance issues or contention.
• apply_upserts_total
  • Description: Number of rows upserted to the target.
  • Interpretation: Use to monitor write throughput. Should grow steadily during active replication.
• apply_deletes_total
  • Description: Number of rows deleted from the target.
  • Interpretation: Use to monitor delete throughput. Compare with delete operations on the source database.
• apply_errors_total
  • Description: Number of times an error was encountered while applying mutations.
  • Interpretation: Growing error count indicates target database issues or constraint violations.
• apply_conflicts_total
  • Description: Number of rows that experienced a compare-and-set (CAS) conflict.
  • Interpretation: High counts indicate concurrent modifications or stale data conflicts. May require conflict resolution tuning.
• apply_resolves_total
  • Description: Number of rows that experienced a compare-and-set (CAS) conflict and were successfully resolved.
  • Interpretation: Compare with apply_conflicts_total to verify conflict resolution is working. Should be close to or equal to conflicts.

​

Userscript metrics

• script_invocations_total (counter)
  • Description: Number of times handler functions (such as onRowUpsert, onRowDelete, and onWrite) are invoked.
  • Interpretation: Use to confirm that userscripts are actively being called, and detect misconfigurations where scripts filter out all data or never run.

• script_rows_filtered_total (counter)
  • Description: Number of rows filtered out by the (for example, handlers that returned null or produced no output).
  • Interpretation: Use to identify scripts that unintentionally drop incoming data, and confirm that logic for filtering out data rows is working as intended.

• script_rows_processed_total (counter)
  • Description: Number of rows successfully processed and passed through the .
  • Interpretation: Use to measure how many rows are being transformed or routed successfully. Compare with script_rows_filtered_total to understand filtering ratios and validate script logic.

• script_exec_time_seconds (histogram)
  • Description: Measures the execution time of each function call.
  • Interpretation: Use to detect slow or inefficient userscripts that could introduce replication lag, and identify performance bottlenecks caused by complex transformations or external lookups.

• script_entry_wait_seconds (histogram)
  • Description: Measures the latency between a row entering the Replicator queue and the start of its execution inside the JavaScript runtime.
  • Interpretation: Use to detect whether userscripts are queuing up before execution (higher values indicate longer wait times), and monitor how busy the userscript runtime pool is under load.

• script_errors_total (counter)
  • Description: Number of errors that occurred during execution (for example, JavaScript exceptions or runtime errors).
  • Interpretation: Use to surface failing scripts or invalid assumptions about incoming data, and monitor script stability over time and catch regressions early.

​

Metrics snapshots

​

Enable metrics snapshotting

​

Step 1. Run Replicator with the snapshot flags

replicator pglogical \
--targetConn postgres://postgres:postgres@localhost:5432/molt?sslmode=disable \
--stagingConn postgres://root@localhost:26257/_replicator?sslmode=disable \
--slotName molt_slot \
--bindAddr 0.0.0.0:30004 \
--stagingSchema _replicator \
--stagingCreateSchema \
--disableAuthentication \
--tlsSelfSigned \
--stageMode crdb \
--bestEffortWindow 1s \
--flushSize 1000 \
--metricsAddr :30005 \
--metricsSnapshotPeriod 15s \
--metricsSnapshotCompression gzip \
--metricsSnapshotRetentionTime 168h \
-v