Migrating a Live S3 Workload to GCP Without Waiting for Writes to Stop

Migrating a Live S3 Workload to GCP Without Waiting for Writes to Stop

AWS → GCP Storage Migration — Part 3

Using Amazon S3 events, SQS, and Google Storage Transfer Service to synchronize new objects while preparing for a controlled cutover.

Summary

The first two parts of this series dealt with migrations where the source could effectively be treated as a known dataset.

In Part 1, I migrated objects directly from Amazon S3 to Google Cloud Storage using an agentless Storage Transfer Service job.

In Part 2, I moved the same dataset from a POSIX filesystem using a Storage Transfer Agent.

Both worked.

But both avoided one of the more uncomfortable questions in a real migration:

What happens if the source application keeps writing data while you're migrating it?

Stopping an application for the entire duration of a large migration is rarely attractive. At sufficient scale, it may not even be practical.

So for the third PoC, I changed the problem.

Amazon S3 remained the source, but instead of treating the bucket as static, I configured S3 object-creation events to flow through Amazon SQS. Google Storage Transfer Service consumed that event stream and transferred newly created objects into Google Cloud Storage automatically.

The architecture became:

Application
     ↓
Amazon S3
     ↓ ObjectCreated
S3 Event Notification
     ↓
Amazon SQS
     ↓
Storage Transfer Service
     ↓
Google Cloud Storage

This changed the migration from a one-time copy into an incremental synchronization mechanism.

And that led to the most important lesson from the entire series:

At some point, migration stops being a data-copy problem and becomes a cutover problem.

The Problem With a Moving Source

Imagine an application storing user-generated objects in S3.

At 10:00 AM, the bucket contains:

1,000,000 objects

You begin migrating them.

At 10:30 AM, users are still using the application.

Now the source contains:

1,002,000 objects

At 11:00 AM:

1,005,000 objects

Your migration is progressing.

But the thing you're migrating is also changing.

This creates a moving target.

A batch transfer can copy the dataset that existed when the migration started, but that doesn't automatically solve the objects created afterward.

You could run another batch.

Then another.

Then another.

But eventually you need to answer:

When do the source and destination become close enough that I can actually switch the application?

That's a different problem from simply copying files.

Why Not Just Stop the Application?

The easiest migration architecture would be:

Stop application writes
        ↓
Copy everything
        ↓
Validate everything
        ↓
Switch application
        ↓
Resume writes

Technically, that's wonderfully simple.

Operationally, it might mean hours or days of downtime.

For a small dataset, perhaps that's acceptable.

For a large production workload, probably not.

Suppose the source contains tens or hundreds of terabytes.

Or petabytes.

Even with substantial transfer throughput, the bulk migration could take long enough that freezing the application for the entire transfer window becomes unrealistic.

So instead of trying to eliminate source changes, I wanted to capture them.

From Batch Transfer to Incremental Synchronization

The architecture I tested used events.

Whenever a new object was created in S3, S3 generated an event notification.

That notification was delivered to Amazon SQS.

Storage Transfer Service listened to the SQS event stream.

When an event appeared, the corresponding object could be transferred into GCS.

The flow became:

S3 PutObject
     ↓
ObjectCreated Event
     ↓
S3 Event Notification
     ↓
Amazon SQS
     ↓
Storage Transfer Service
     ↓
Google Cloud Storage

The important difference from Part 1 was that I didn't need to manually start another transfer every time the source changed.

The migration job remained active.

Waiting.

Why SQS Sits in the Middle

It might seem tempting to connect S3 directly to the destination migration process.

But the queue plays an important role.

S3 produces events.

Storage Transfer Service consumes them.

SQS separates those two systems.

Conceptually:

Producer
   │
   │ events
   ▼
 Queue
   │
   │ consumption
   ▼
Consumer

This means the producer doesn't need the consumer to process the event at exactly the moment it occurs.

The queue holds the event until it can be consumed.

For a migration workflow, that's useful because object creation and object transfer don't need to happen synchronously.

The application can continue writing to S3.

Migration happens alongside it.

Creating the Event Queue

For the PoC, I created a Standard Amazon SQS queue.

The configuration was intentionally simple:

Queue type:
Standard

Encryption:
SSE-SQS

Visibility timeout:
30 seconds

Message retention:
4 days

At this stage, the queue existed.

But S3 couldn't yet send anything to it.

That required a queue policy.

Restricting Who Could Publish Events

I didn't want the queue to accept arbitrary messages from unrelated S3 buckets.

The policy therefore restricted SQS:SendMessage to the intended S3 source.

Conceptually:

Specific S3 Bucket
        │
        │ SQS:SendMessage
        ▼
Migration Event Queue

The important conditions tied message publishing to:

• the expected AWS account,
• and the expected source bucket.

That meant the queue wasn't simply exposed as a generic event destination.

It existed specifically for the migration pipeline.

Connecting S3 to SQS

With the queue policy in place, I created an S3 event notification.

The event type was:

s3:ObjectCreated:*

The destination was the migration SQS queue.

So now:

Upload object to S3
        ↓
S3 generates ObjectCreated event
        ↓
SQS receives event

But before involving Google Cloud, I wanted to prove that this half of the architecture actually worked.

This is something I've become increasingly fond of when debugging integrations:

validate one boundary at a time.

Test the Event Pipeline Before Adding Another Cloud

Instead of configuring everything at once and hoping it worked, I tested:

S3
 ↓
SQS

independently.

I uploaded a small metadata object into the S3 bucket.

Then I manually polled the queue.

A message appeared containing information similar to:

{
  "Records": [
    {
      "eventSource": "aws:s3",
      "awsRegion": "ap-south-1",
      "eventName": "ObjectCreated:Put",
      "s3": {
        "bucket": {
          "name": "<SOURCE_BUCKET>"
        },
        "object": {
          "key": "metadata.txt",
          "size": 52
        }
      }
    }
  ]
}

That proved:

S3
 ↓
Event Notification
 ↓
SQS

PASS

Only after that did I introduce Storage Transfer Service.

This sounds like a small testing decision.

It saved a lot of ambiguity.

If the complete pipeline failed later, I already knew S3 → SQS was working.

The failure would have to be somewhere downstream.

Extending the Migration Identity

In Part 1, the AWS migration role required only:

s3:ListBucket
s3:GetObject

That was enough because Storage Transfer Service only needed to discover and read objects.

The event-driven architecture introduced another resource:

SQS.

So the role also needed:

sqs:ReceiveMessage
sqs:DeleteMessage
sqs:ChangeMessageVisibility

The permissions now represented two responsibilities.

Read the source

s3:ListBucket
s3:GetObject

Consume migration events

sqs:ReceiveMessage
sqs:DeleteMessage
sqs:ChangeMessageVisibility

Importantly, the trust relationship from Part 1 didn't need to change.

Google Storage Transfer Service was still assuming the same AWS IAM role through web identity federation.

The role simply gained the additional permissions required by the new architecture.

The Security Model Was Still Federated

I kept the same cross-cloud authentication model from Part 1.

There were still no permanent AWS access keys handed to Google.

The flow remained:

Google Storage Transfer identity
              ↓
AWS STS AssumeRoleWithWebIdentity
              ↓
Temporary AWS credentials
              ↓
S3 + SQS permissions

I like this because adding event-driven behavior didn't require weakening the identity architecture.

The role gained exactly the additional capabilities the new workflow required.

Nothing more.

Creating the Event-Driven Transfer

With S3 events flowing into SQS, I created another Storage Transfer Service job.

Source:

Amazon S3

Destination:

Google Cloud Storage

Authentication:

AWS IAM role
via identity federation

But instead of selecting a one-time batch schedule, I selected:

Event-driven

The SQS queue ARN became the event stream for the transfer.

After creation, the job didn't behave like the batch transfer from Part 1.

It didn't simply run and finish.

Its state remained:

Listening for events

That's exactly what I wanted.

The First Object

I started with the small metadata object.

The object existed in S3.

Its event appeared in SQS.

Storage Transfer Service detected the event.

The object appeared in GCS.

I didn't manually start a transfer.

I didn't click "Run."

The event triggered the migration path.

So the complete pipeline was now:

S3
 ↓
SQS
 ↓
Storage Transfer Service
 ↓
GCS

PASS

Now it was time to test something larger.

Uploading the First 400 MiB Object

I uploaded:

dataset-01.bin

to S3.

And then did nothing in Google Cloud.

That was the point of the experiment.

The event flow happened automatically:

dataset-01.bin
       ↓
S3 PutObject
       ↓
ObjectCreated
       ↓
SQS
       ↓
Storage Transfer Service
       ↓
GCS

The file eventually appeared in the destination.

Then I repeated the process with:

dataset-02.bin
dataset-03.bin

Again, no manual transfer initiation was required.

Eventually the destination contained:

dataset-01.bin
dataset-02.bin
dataset-03.bin
metadata.txt

Approximately:

1.26 GB
4 objects
0 transfer errors

And the transfer job remained active.

Listening for events

It hadn't "finished."

That was the expected behavior.

The job was waiting for the next source change.

This Isn't Replication

There's an important distinction worth making.

I wouldn't describe this architecture as a general-purpose replacement for synchronous cross-cloud replication.

It's an event-driven transfer mechanism.

There is time between:

Object created in S3

and:

Object available in GCS

That delay matters.

Applications need to be designed around the fact that the destination may temporarily lag behind the source.

For migration, that's often acceptable.

The objective isn't necessarily to make both systems instantaneously identical.

The objective is to continuously reduce the delta between them until cutover becomes manageable.

The Delta Is the Real Problem

This is where the migration strategy clicked for me.

Suppose you have:

100 TB

in S3.

You start the initial migration.

While that transfer is running, applications create another:

500 GB

of data.

Without incremental synchronization, your problem after the bulk transfer is:

500 GB delta

And the source is still changing.

With incremental synchronization running alongside the migration, those changes are continuously transferred.

By the time the bulk migration completes, the remaining delta can be much smaller.

That's important because the remaining delta determines how long the final cutover window needs to be.

Bulk Transfer + Incremental Sync

This leads to a much more realistic production pattern.

Existing S3 Dataset
        │
        │ Bulk Transfer
        ▼
       GCS

Meanwhile...

New S3 Objects
        │
        │ Events
        ▼
       SQS
        │
        ▼
Storage Transfer Service
        │
        ▼
       GCS

The bulk migration handles history.

The event stream handles change.

Together:

Historical Data
      +
New Changes
      ↓
Destination approaches source state

Now we're no longer trying to freeze the source for the entire migration.

We only need to freeze it briefly at the end.

The Cutover Window

Eventually, the application needs to stop writing to the old destination and begin writing to the new one.

This is where the migration becomes operational rather than purely technical.

The production pattern I arrived at was:

Initial Bulk Migration
        ↓
Incremental Synchronization
        ↓
Destination Validation
        ↓
Short Write Freeze
        ↓
Final Delta
        ↓
Final Integrity Validation
        ↓
Switch Application to GCS
        ↓
Monitor
        ↓
Rollback Window
        ↓
Decommission Source

The important word is:

short.

Instead of freezing writes while 100 TB moves, you freeze writes only long enough to close the final delta and prove the destination is ready.

That can dramatically change the downtime characteristics of a migration.

Why the Write Freeze Still Exists

It's tempting to think continuous synchronization removes downtime completely.

I wouldn't make that assumption.

At some point, you need a clean boundary.

Imagine:

T0: Application still writes to S3
T1: Final migration starts
T2: Application writes another object
T3: Validation begins
T4: Application switches to GCS

Which system owns the object written at T2?

Did its event reach SQS?

Was it transferred before validation?

Did the application switch before it arrived?

These are exactly the kinds of race conditions that make cutovers uncomfortable.

A short write freeze simplifies the final state.

Stop new writes
      ↓
Drain remaining events
      ↓
Transfer final delta
      ↓
Validate
      ↓
Switch

Sometimes a few minutes of controlled downtime is much safer than attempting a theoretically zero-downtime cutover with poorly understood race conditions.

Zero downtime isn't automatically better engineering.

Predictable downtime can be the safer trade-off.

Validation Didn't Change

Even though the migration architecture became more sophisticated, my validation principle stayed exactly the same.

I downloaded the four destination objects into a separate validation directory.

Then generated:

shasum -a 256 \
  dataset-01.bin \
  dataset-02.bin \
  dataset-03.bin \
  metadata.txt > continuous-destination-sha256.txt

And compared it with the original source manifest:

diff ~/source-sha256.txt continuous-destination-sha256.txt
echo $?

Result:

0

Again:

SHA-256 PASS

All four destination files matched the original source byte-for-byte.

Three migration architectures.

Same independent validation principle.

The Event Stream Needs Its Own Validation

Part 3 also taught me something that didn't exist in the earlier methods.

The migration now has two things to validate:

Data Plane
+
Event Plane

The data plane asks:

Did the object arrive correctly?

The event plane asks:

Did the change get detected at all?

Those are different failure modes.

An object can be perfectly transferable while its event is never generated.

An event can reach SQS while the migration identity cannot read the object.

Storage Transfer Service can receive an event while destination permissions prevent the write.

The complete pipeline contains several boundaries:

Application
    ↓
S3
    ↓
S3 Notification
    ↓
SQS
    ↓
STS Event Consumer
    ↓
S3 Read
    ↓
GCS Write

Testing S3 → SQS independently before introducing GCP was therefore more important than it initially seemed.

Event-Driven Systems Change How You Debug

Batch systems have a relatively simple question:

Did the job succeed?

Event-driven systems introduce more interesting questions.

Was the event generated?

Did it reach the queue?

Was it consumed?

Was it acknowledged?

Could the consumer read the corresponding object?

Did the destination write succeed?

Was the message retried?

Could duplicate events occur?

The architecture becomes more resilient in some ways.

It also becomes less visually obvious.

That's the trade.

Decoupling systems gives them independence.

It also means you need observability across the boundaries you've introduced.

Standard SQS Means You Design for Reality

The PoC used a Standard SQS queue.

With event-driven systems, I don't want to design downstream behavior around an assumption that every event will always appear exactly once in a perfectly ordered sequence.

Migration logic should tolerate the realities of asynchronous processing.

That means thinking about:

• duplicate events,
• retries,
• delayed processing,
• temporary failures,
• and idempotent behavior.

Even if the managed transfer service handles much of that machinery for me, I still need to understand the delivery model of the systems I'm connecting.

Managed doesn't mean consequence-free.

The Job Being "In Progress" Was Success

This is another subtle difference from Part 1.

Part 1 ended with:

SUCCESS

Part 3 remained:

In progress
Listening for events

Normally, an unfinished job sounds undesirable.

Here, it was exactly the desired state.

The transfer wasn't supposed to finish after one object.

It was supposed to remain available for future changes.

That's a useful reminder that operational states only make sense relative to the workload.

SUCCESS isn't always the desired state.

Sometimes:

Healthy + Waiting

is.

Don't Leave the Migration Running Forever

An event-driven migration pipeline is temporary infrastructure.

Once the application has successfully cut over to GCS, the old synchronization path should eventually disappear.

But not immediately.

I would preserve it through an agreed rollback window.

The lifecycle becomes:

Cutover
   ↓
Monitor destination
   ↓
Rollback window
   ↓
Migration accepted
   ↓
Disable event-driven transfer
   ↓
Remove S3 notification
   ↓
Delete migration SQS queue
   ↓
Remove temporary SQS permissions
   ↓
Eventually decommission source

The order matters.

Destroying the source-side migration infrastructure immediately after cutover removes options exactly when you may still need them.

Temporary Permissions Should Actually Be Temporary

Part 3 required adding SQS permissions to the migration role.

After the event-driven migration is no longer required, those permissions should be removed.

The role can return to the simpler Part 1 state:

s3:ListBucket
s3:GetObject

and lose:

sqs:ReceiveMessage
sqs:DeleteMessage
sqs:ChangeMessageVisibility

This is an easy detail to forget.

Migration infrastructure tends to be built under time pressure.

Then the migration succeeds.

Everyone moves on.

Six months later, temporary queues, IAM permissions, service accounts, and event notifications are still sitting around.

Cleanup is part of migration engineering.

Not an optional postscript.

What I Would Add for Production

The PoC demonstrated the architecture.

Production would require considerably more thought.

I'd want explicit monitoring for:

Event backlog

How many messages are waiting in SQS?

Event age

How old is the oldest unprocessed migration event?

Transfer errors

Which objects failed and why?

Source/destination delta

How far behind is GCS?

Object counts

Do source and destination inventories reconcile?

Bytes

Does total destination size match expectations?

Integrity

Can we validate checksums at scale?

Application behavior

Does the application actually work correctly against GCS?

The last one is easy to underestimate.

A perfect byte-for-byte migration can still produce a failed application migration.

Data validation and application validation are not interchangeable.

What About Deletes and Updates?

My PoC focused on object creation events.

That's important because "continuous synchronization" can sound broader than what was actually demonstrated.

The implemented event was:

s3:ObjectCreated:*

So the experiment proved:

Newly created S3 objects can trigger automatic transfer into GCS.

A production workload may require additional questions.

What happens when an object is deleted?

What happens when an existing object is overwritten?

What happens with versioned objects?

Should deletes propagate?

Should the destination preserve old versions?

Those behaviors need explicit design.

Never let the word "sync" hide the actual semantics you're implementing.

Three Patterns, Three Different Source Conditions

At the end of this series, the migration approaches now make much more sense to me when classified by the condition of the source.

Pattern 1 — Native and relatively static

Amazon S3
     ↓
Storage Transfer Service
     ↓
GCS

Use the managed, agentless path.

Pattern 2 — Private or filesystem-based

POSIX / NFS / Private Storage
            ↓
      Transfer Agent
            ↓
        Agent Pool
            ↓
 Storage Transfer Service
            ↓
           GCS

Put agents close to the data.

Pattern 3 — Active and continuously changing

Bulk S3 Transfer
        +
S3 Object Events
        ↓
       SQS
        ↓
Storage Transfer Service
        ↓
       GCS

Move the history while continuously reducing the change delta.

The technology isn't the starting point.

The source condition is.

The Migration Lifecycle Became Clearer

After implementing all three methods, I stopped thinking about migration as:

SOURCE
  ↓
COPY
  ↓
DESTINATION

A more realistic enterprise migration looks like:

DISCOVER
   ↓
ASSESS
   ↓
DESIGN
   ↓
PILOT
   ↓
BULK TRANSFER
   ↓
INCREMENTAL SYNC
   ↓
VALIDATE
   ↓
WRITE FREEZE
   ↓
FINAL DELTA
   ↓
FINAL VALIDATION
   ↓
CUTOVER
   ↓
MONITOR
   ↓
ROLLBACK WINDOW
   ↓
DECOMMISSION

The actual transfer occupies surprisingly little of that lifecycle.

Most of the difficult work happens around it.

The Principle That Survived All Three Methods

Part 1 used managed S3 transfer.

Part 2 used an agent.

Part 3 used events.

The architectures changed.

My definition of success didn't.

Transfer Job SUCCESS
        ≠
Migration Success

For me:

Migration Success
        =
Completeness
        +
Integrity
        +
Correctness
        +
Application Validation

Every implementation ended with independent SHA-256 validation.

All three returned:

diff exit code = 0

SHA-256 PASS

The migration tool told me the transfer worked.

The checksum told me the data agreed.

Those are different pieces of evidence.

I want both.

What This Series Changed for Me

When I started this PoC, I thought I was learning how to move storage from AWS to GCP.

Technically, I was.

But the more useful lessons had very little to do with copying files.

I learned that the migration architecture should follow the source architecture.

If the source is natively supported, keep the path managed and small.

If the source is private, put an agent close to the data.

If the source is active, don't pretend it will stop changing while you copy it.

I learned that identity is part of migration architecture.

That temporary credentials are better than casually distributing long-lived keys.

That read-only source permissions are often enough.

That event pipelines should be validated one boundary at a time.

That PoC throughput numbers need context.

That cleanup deserves to be part of the design.

And above all, I learned that moving data and migrating a system are not the same thing.

Final Thoughts

The first object I copied from S3 to GCS was easy.

The interesting part began when I asked:

What happens to the next object?

And the one after that?

And the object created while validation is running?

And the final object written before the application switches clouds?

Those questions changed the problem.

A storage migration isn't complete because every historical byte reached the destination.

It's complete when the destination can safely become the new source of truth.

That requires more than transfer.

It requires synchronization.

Validation.

A cutover boundary.

Monitoring.

Rollback planning.

And eventually, the confidence to turn the old path off.

That's where this series ends.

Not with:

COPY COMPLETE

but with something much more meaningful:

SOURCE
   ↓
MIGRATE
   ↓
VALIDATE
   ↓
CUTOVER
   ↓
DESTINATION BECOMES SOURCE OF TRUTH

And that, more than any green SUCCESS status, is what I would call a successful migration.