Migrating Amazon S3 to Google Cloud Storage Without Transfer Agents
Building a secure cross-cloud migration with Google Storage Transfer Service, identity federation, least-privilege access and independent data validation.
Migrating Amazon S3 to Google Cloud Storage Without Transfer Agents
AWS → GCP Storage Migration — Part 1
Building a secure cross-cloud migration with Google Storage Transfer Service, identity federation, least-privilege access, and independent data validation.
Summary
Moving objects from Amazon S3 to Google Cloud Storage sounds like one of those tasks that should be trivial.
There is a source bucket.
There is a destination bucket.
Copy the objects.
Wait for the green checkmark.
Done.
Once I started implementing the migration myself, I realised the interesting part wasn't copying the data at all.
The interesting questions were around it.
How should Google Cloud authenticate to AWS?
Do I really need a VM or transfer agent sitting between the two clouds?
How little access can I give the migration service?
How do I know the objects arriving in GCS are actually identical to what left S3?
And perhaps most importantly:
Does a transfer job reporting SUCCESS actually mean the migration succeeded?
This is Part 1 of my AWS → GCP storage migration series. I'll start with the simplest case: a dataset already stored in Amazon S3 and a Google Cloud Storage bucket waiting on the other side.
No transfer VM.
No manually managed transfer agent.
No permanent AWS access keys shared with Google.
Just S3, Storage Transfer Service, identity federation, GCS, and an independent way to prove that the data survived the journey.
The Problem I Wanted to Test
The scenario was straightforward:
Data currently exists in Amazon S3 and needs to be migrated into Google Cloud Storage.
Before thinking about tools, I wanted the test to answer a few practical questions.
Could the migration happen directly between the two cloud storage services?
Could it be done without maintaining transfer infrastructure?
Could cross-cloud authentication avoid permanent AWS credentials?
Could the source permissions remain read-only?
And could I independently verify the result instead of trusting the transfer service's status page?
That last question became more important than I expected.
Building a Dataset I Could Verify
I didn't want to test the migration with a couple of tiny text files.
That would prove the configuration worked, but it wouldn't make transfer progress or throughput particularly meaningful.
So I created three binary files of roughly 400 MiB each, along with a small metadata file.
mkdir -p ~/niveus-migration-demo
cd ~/niveus-migration-demo
mkfile 400m dataset-01.bin
mkfile 400m dataset-02.bin
mkfile 400m dataset-03.bin
echo "AWS S3 to Google Cloud Storage migration PoC" > metadata.txt
The resulting dataset was approximately:
dataset-01.bin ~400 MiB
dataset-02.bin ~400 MiB
dataset-03.bin ~400 MiB
metadata.txt small metadata object
Total:
~1.2 GiB
~1.26 GB
4 objects
I also wrote a small identifier into each binary file so they weren't simply three indistinguishable empty test files.
The exact dataset size wasn't important.
What mattered was having something large enough to observe the migration while still being small enough to repeatedly transfer and validate during a PoC.
I Created the Validation Before the Migration
This was one of the decisions I'm happiest with.
Before uploading anything to AWS, I generated SHA-256 hashes for every file.
shasum -a 256 \
dataset-01.bin \
dataset-02.bin \
dataset-03.bin \
metadata.txt > source-sha256.txt
That produced a manifest containing:
<SHA256> dataset-01.bin
<SHA256> dataset-02.bin
<SHA256> dataset-03.bin
<SHA256> metadata.txt
Then I moved that manifest outside the migration directory.
Why?
Because I didn't want the validation file itself to become part of the migration dataset.
The checksum file represented my independent source of truth.
That distinction matters.
If the migration platform tells me:
4 objects transferred successfully.
I've learned that the platform believes it transferred four objects.
I still haven't independently proven that the bytes at the destination are identical to the bytes at the source.
The SHA-256 manifest would let me do that later.
The Architecture
For this first method, the source was already Amazon S3.
Google Storage Transfer Service supports S3 directly, which meant there was no reason to introduce an intermediate transfer server.
The architecture ended up being pleasantly small:
AWS
Amazon S3
│
│ read
▼
AWS IAM Role
▲
│
Web Identity Federation
│
▼
Google Storage Transfer Service
│
│ write
▼
Google Cloud Storage
There is no VM in the middle.
No transfer agent to patch.
No temporary disk to size.
No custom copy application.
No server whose only purpose is to move bytes between two managed storage services.
For this particular source type, adding any of those would have created infrastructure without solving an additional problem.
That became the first lesson from the PoC:
If the migration service already understands the source natively, don't invent a transfer layer just because you can.
Preparing the Source
I created an Amazon S3 bucket in ap-south-1 (Mumbai).
For the PoC, the bucket was configured with:
Bucket type: General purpose
Public access: Blocked
ACLs: Disabled
Versioning: Disabled
Encryption: SSE-S3
Object Lock: Disabled
The four migration objects were uploaded:
dataset-01.bin
dataset-02.bin
dataset-03.bin
metadata.txt
The checksum manifest was deliberately kept outside the bucket.
At this point, AWS contained the source dataset.
Nothing had moved yet.
Preparing the Destination
On Google Cloud, I created a regional Cloud Storage bucket in asia-south1, also Mumbai.
The destination used:
Location: asia-south1
Storage class: Standard
Public access prevention: Enabled
Uniform bucket access: Enabled
Encryption: Google-managed
I intentionally kept the source and destination geographically close for the PoC.
The destination bucket started empty.
So the migration state was simple:
AWS S3
4 objects
~1.26 GB
↓
Google Cloud Storage
0 objects
Now came the more interesting part.
Authentication.
How Does Google Cloud Read a Private S3 Bucket?
The easiest approach would have been to create an AWS access key and secret access key and provide them to the migration service.
It would also have been the approach I liked least.
Long-lived credentials introduce another secret that needs to be:
- stored,
- protected,
- rotated,
- audited,
- and eventually revoked.
For a cross-cloud migration, I wanted something cleaner.
The solution was Web Identity Federation.
Instead of handing Google a permanent AWS credential, AWS trusts the identity used by Google Storage Transfer Service.
Conceptually:
Google Storage Transfer identity
│
▼
AWS STS AssumeRoleWithWebIdentity
│
▼
Temporary AWS credentials
│
▼
Read permitted S3 objects
Google proves its identity.
AWS evaluates the role's trust policy.
AWS STS issues temporary credentials.
Storage Transfer Service uses those credentials to access the source.
There is no permanent AWS access key sitting inside the migration configuration.
That's a much healthier security model.
Establishing the Trust Relationship
Storage Transfer Service exposes an identity that can be used as the trusted subject on the AWS side.
I retrieved the Storage Transfer Service account information from Google Cloud and used its subject ID when creating the AWS IAM role trust relationship.
Conceptually, the trust policy looked like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "accounts.google.com"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"accounts.google.com:sub": "<STORAGE_TRANSFER_SUBJECT_ID>"
}
}
}
]
}
This is an important distinction.
The role isn't saying:
"Any Google identity can access my bucket."
It trusts the specific subject expected from the Storage Transfer Service integration.
Once the trust relationship is established, Google can assume the role through AWS STS and receive temporary credentials.
Least Privilege Was Surprisingly Small
The next question was:
What does a migration service actually need to do to the source?
For a copy-only migration, not much.
It needs to know which objects exist.
And it needs to read them.
That's it.
So the source role received:
s3:ListBucket
s3:GetObject
It did not receive permission to:
Upload objects ✗
Modify objects ✗
Delete objects ✗
Administer S3 ✗
The simplified policy looked like:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::<SOURCE_BUCKET>"
},
{
"Effect": "Allow",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::<SOURCE_BUCKET>/*"
}
]
}
I like this architecture because the permissions describe the intent of the migration almost perfectly:
Discover the source objects and read them. Nothing else.
Even if the migration identity were misused, it wouldn't have permission to delete or modify the source dataset.
Creating the Transfer
With the identity relationship established, the actual Storage Transfer Service configuration became relatively boring.
And that's a good thing.
The source was:
Amazon S3
The destination was:
Google Cloud Storage
Authentication used:
AWS IAM role for identity federation
The job was configured as a one-time batch transfer.
For the PoC, I also selected conservative object behavior:
Overwrite: If different
Delete source objects: Never
Delete destination: Never
I enabled logging for successful and failed copies and started the transfer.
At this point, there wasn't much left to do.
Storage Transfer Service handled the data movement.
The Result
The transfer completed successfully.
The observed PoC result was:
Status: SUCCESS
Objects: 4 / 4
Transferred: 1.26 GB / 1.26 GB
Errors: 0
Skipped: 0
Duration: 44 seconds
Average throughput: 28.21 MB/s
That was encouraging.
But I want to be careful with these numbers.
This was a small PoC using a particular source region, destination region, dataset, network path, and point in time.
28.21 MB/s is an observation from this experiment. It is not a Storage Transfer Service performance guarantee.
A production migration could behave very differently depending on:
- object count,
- object size distribution,
- regions,
- source throttling,
- network conditions,
- service quotas,
- and dataset scale.
So I wasn't particularly interested in declaring one service "fast" based on a 1.26 GB experiment.
I was interested in something else.
Did the data arrive intact?
SUCCESS Wasn't Enough
The Google Cloud console said:
SUCCESS
All four objects were visible in the destination.
The transferred byte count matched the source.
Everything looked correct.
It would have been very easy to stop there.
Instead, I downloaded the destination objects into a completely separate validation directory.
mkdir -p ~/migration-validation
cd ~/migration-validation
Then I generated another SHA-256 manifest:
shasum -a 256 \
dataset-01.bin \
dataset-02.bin \
dataset-03.bin \
metadata.txt > destination-sha256.txt
Now I had two independently generated manifests.
One from before the migration:
source-sha256.txt
and one from the destination:
destination-sha256.txt
I compared them:
diff ~/source-sha256.txt destination-sha256.txt
echo $?
The result was:
0
No differences.
Source SHA-256
=
Destination SHA-256
PASS
The destination files were byte-for-byte identical to the original dataset.
Only then was I comfortable calling the migration successful.
Transfer Success and Migration Success Are Different Things
This became the most important lesson from the entire exercise.
A transfer platform can tell you:
Job completed
4 objects copied
1.26 GB transferred
0 errors
Those are useful operational signals.
But a migration has a broader definition of success.
For me, it looks more like:
Migration Success
=
Completeness
+
Integrity
+
Correctness
+
Application Validation
In this PoC, the checksum comparison independently proved integrity for the transferred files.
In a real migration, I would go further.
Object counts.
Total bytes.
Metadata.
Permissions where relevant.
Application behavior.
Business-level validation.
A green transfer job is evidence.
It isn't the entire proof.
What I'd Change for a Production Migration
This PoC intentionally kept the environment small.
A production migration would require considerably more preparation.
Before moving data, I'd want to understand:
- Total dataset size
- Number of objects
- Average and maximum object sizes
- Object change rate
- Available migration window
- Source request limits
- Network and regional considerations
- Encryption requirements
- Metadata requirements
- Compliance constraints
- Rollback expectations
I'd also want explicit answers to questions such as:
Is the source application still writing to S3?
Are deletes expected to propagate?
Do we need to preserve object metadata?
How long can the application tolerate a write freeze?
What proves that the application works against GCS after migration?
Those questions start pushing us beyond a simple batch copy.
And that's exactly where the later parts of this series go.
When I'd Use This Pattern
After implementing it, the decision rule for this first architecture became fairly simple.
If the source data is already in Amazon S3 and the goal is a managed migration into Google Cloud Storage, Storage Transfer Service is a natural starting point.
Especially when I want:
- No intermediate transfer VM
- No self-managed transfer agent
- Temporary federated AWS credentials
- Read-only source access
- Managed transfer orchestration
The architecture remains small:
S3
↓
Storage Transfer Service
↓
GCS
And small architectures are usually easier to reason about.
But this pattern depends on something important:
The source is already a storage system that Storage Transfer Service understands natively.
What happens when it isn't?
What I Learned
The biggest surprise from this implementation was how little infrastructure the migration actually required.
The difficult part wasn't moving bytes.
It was designing the trust boundary correctly and deciding what evidence I needed before calling the migration successful.
A few things I'll carry into future migrations:
Don't add infrastructure without a reason.
If the migration platform natively supports the source, an extra transfer server may simply become another component to operate.
Prefer temporary identity over permanent credentials.
Cross-cloud federation made the security model much cleaner than distributing AWS access keys.
Migration identities should be boring.
List.
Read.
Nothing else.
Measure performance, but don't turn a PoC into a benchmark.
A throughput number from one small experiment tells you what happened during that experiment. Nothing more.
And most importantly:
Never let the migration tool be the only system telling you that the migration worked.
Verify independently.
Next: What If the Data Isn't in S3?
This first migration was relatively clean because both ends were cloud-native object storage.
Real enterprise migrations aren't always that convenient.
Data may live on:
- NFS
- NAS
- POSIX filesystems
- Private storage appliances
- On-premises servers
- S3-compatible platforms
In those environments, Google Cloud cannot simply reach into the source through the S3 API.
Something needs to run close to the data.
That's where Storage Transfer Agents enter the architecture.
In Part 2, I'll migrate the same dataset from a POSIX filesystem into Google Cloud Storage using a Storage Transfer Agent running through Podman.
That experiment was considerably less smooth.
Authentication failed.
Container registry access needed fixing.
The source filesystem had to be mounted correctly.
And my ARM64 Mac ended up running an AMD64 transfer-agent image.
Which made it much more interesting.
Part 2: Migrating On-Prem and POSIX Storage to GCP with Storage Transfer Agents →