Configure resources for migration

This page describes the preparation part of the migration process where you configure resources for the migration.

At this stage, you do the following:

  1. Install command-line tools required for running commands at later stages of the migration process.
  2. Configure Firestore with MongoDB compatibility database.
  3. (Optional) Adjust the operation log (oplog) settings on your MongoDB compatible source.
  4. Create a Cloud Storage bucket for intermediate data storage.

Configure IAM permissions

Your account requires appropriate IAM roles in all services used in the migration process:

Install command-line tools

The migration procedure uses the gcloud CLI to configure and actuate the migration steps. Some of the steps in this procedure also require the use of the git and maven command-line tools.

If the following command-line tools aren't installed on your computer:

  1. Install the gcloud CLI by following instructions on the Install the gcloud CLI page.
  2. Install maven.
  3. Install git. Make sure that you have an account that allows command-line access to GitHub.

Source-specific configuration

MongoDB on Compute Engine

A MongoDB database that runs on Compute Engine VMs in a self-managed cluster isn't normally exposed to the public internet. The migration procedure will use Private Service Connect in Datastream to connect the Datastream pipeline to your source database.

Do the following:

  1. Configure Datastream Private Connectivity by following instructions outlined in Create a private connectivity configuration.

  2. Note the Configuration ID parameter of the created configuration. You will use it in later stages to set up required environment variables.

Azure Cosmos DB

Make sure that Azure CLI is installed on your computer.

Configure a destination Firestore with MongoDB compatibility database

  1. Make sure that your project has a Firestore with MongoDB compatibility database where you will migrate data from your source Mongo database. For more information about creating a database, see Create and manage databases.

  2. For the purpose of this migration, we recommend to create a username and password for the Firestore with MongoDB compatibility database to use with the SCRAM-SHA-256 authentication protocol. This username can be safely deleted after the migration completes, or you may choose to keep using these credentials to connect your Mongo clients to your new Firestore with MongoDB compatibility database.

You will use the name of this Firestore with MongoDB compatibility database and the user credentials in later steps.

Adjust the oplog window size of the source database

We recommend to adjust the oplog window of your source Mongo database to 3 days worth of total write traffic to this database. If the rate of write traffic exceeds the rate at which Datastream can consume changes from your database, this adjustment will prevent data loss.

The value might need to be adjusted further, depending on the pattern of the traffic and the volume of peak traffic. For example, if a week's worth of traffic is written in a short period of time, then Datastream might not capture the changes from the oplog fast enough before changes fall out of the oplog window. In this case you might need to resize the oplog window to 7 days worth of total write traffic.

Create a Cloud Storage bucket

Create a new Cloud Storage bucket in the following way:

  1. Choose a Cloud Storage region where the migration pipeline will run. We recommend to use the region where your destination Firestore with MongoDB compatibility database is located.

  2. Choose the name for this bucket. Example: mongo-migration-bucket. You will use this name in later steps.

  3. Create a new Cloud Storage bucket with the chosen name and in the chosen region by following instructions provided on the Create a bucket page in the Cloud Storage documentation.

What's next

Proceed to Configure environment variables.