How we migrated 360 Million records (1500+ tables) from DigitalOcean to AWS Aurora

Problem Statement

We are migrating a bulk messaging platform’s PostgreSQL database from DigitalOcean to AWS. The platform manages millions of records related to inbound and outbound messages. The migration must be efficient, cost-effective, and resilient. It should also ensure minimal downtime and the ability to resume from any failure point.

Key Challenges:

• The database contains over 1500+ tables with some tables exceeding 130 million records where we require a structured and resilient migration strategy.

• The entire application stack is moving to AWS to enhance security, scalability, and performance.

• The process must maintain data integrity and cost efficiency.

Solution Approach

Option 1: Manual Migration (Initial Evaluation)

At first, we considered a manual migration approach, which involved:

• Dumping the PostgreSQL database using pg_dump.

• Transferring the dump file to AWS.

• Restoring it on Aurora using pg_restore.

Limitations:

• Time-consuming for large datasets.

• High downtime due to the dump-and-restore process.

• No real-time replication, leading to data inconsistencies during migration.

Option 2: AWS DMS – The Optimal Choice

After evaluating the manual approach, we explored AWS Database Migration Service (DMS) which had the following benefits:

Schema Conversion & Transformation: Automatically migrated table structures.
Performance Efficiency: Faster and more scalable for large databases.

There are different DB migrations out of which we explored two migration types:

Big Bang Migration - Big Bang Migration is a strategy where all data are moved from the source system to the target system in a single effort. This approach is typically executed during off-peak hours to minimize disruption.

Zero Downtime Migration - Zero Downtime Migration is a strategy that allows for data migration without any interruption in service. This approach ensures that users can continue accessing the old system while the new system is being prepared.

Since we will migrate the database during off-peak hours, we decided to go with option 2 - AWS DMS - Big Bang Migration.

Architecture:

What are the Key Benefits we observed when using AWS DMS?

• Minimal Downtime – Supports continuous data replication with minimal service interruption.

• Automated Schema Conversion – Simplifies schema and data type mapping across database engines.

• Flexible Migration Options – Allows full load, ongoing replication, or change data capture (CDC) migration types.

• Scalability – Easily handles large-scale migrations with optimized resource allocation.

• Security – Provides built-in encryption, IAM-based authentication, and network isolation.

• Cost-Effective – The pay-as-you-go pricing model ensures efficient resource usage.

• Performance Optimization – Continuous monitoring, logging, and tuning options for database efficiency

Prerequisites of DMS migration:

1. An AWS account with IAM permissions for DMS.

2. An AWS RDS PostgreSQL instance set up as the target.

3. Networking and security groups configured to allow traffic between DigitalOcean and AWS.

What are the steps we followed to migrate using AWS DMS?

Step 1 - Created a DMS Replication Instance based on the workload (e.g., dms.t3.medium).

Step 2: Configured Source and Target Endpoints

Step 3: Created a Migration Task

Step 4: Started Migration and monitored the migration by checking AWS CloudWatch logs for any issues

Step 5: We completed the migration for 1538 tables holding nearly 360 million data successfully and all details related to the tables such as Schema name, Table name, Load state, Elapsed load time, Total rows migrated etc are found in Table statistics which we can use for DB data analysis.

Step 6: Once the migration is completed successfully, we validated data integrity by running the below query on both postgres and AWS RDS instances

SELECT schemaname, relname AS tablename, 
       (xpath('/row/count/text()', 
              query_to_xml(format('SELECT COUNT(*) AS count FROM %I.%I', schemaname, relname), false, true, '')) 
        )[1]::text::bigint AS exact_row_count
FROM pg_catalog.pg_stat_user_tables
ORDER BY schemaname, relname;

Time and Cost Estimation of the Migration:

• Migration Duration: Initial full load took 4.5 hours. Our Migration needed only one time full load migration.

• DMS Instance Used: dms.r5.large (with 2 vCPUs, 16GB RAM)

• Estimated Cost:

  The total migration cost was around $38.

  

This demonstrates how AWS DMS can migrate a large PostgreSQL database efficiently and cost-effectively

Conclusion

Migrating PostgreSQL from DigitalOcean to AWS RDS using AWS DMS is a structured process that ensures minimal downtime. With proper planning and monitoring, we achieved a smooth transition and leveraged AWS’s scalability and security.