Migrating POSIX and Private Storage to GCP with Storage Transfer Agents
AWS → GCP Storage Migration — Part 2 What changes when your source data doesn't live behind a cloud-native object storage API.
Migrating POSIX and Private Storage to GCP with Storage Transfer Agents
AWS → GCP Storage Migration — Part 2
What changes when your source data doesn't live behind a cloud-native object storage API.
Summary
In Part 1 of this series, migrating Amazon S3 to Google Cloud Storage was surprisingly straightforward.
Google Storage Transfer Service already understood S3. There was no reason to deploy an intermediate VM or run a transfer agent. Once authentication and permissions were configured correctly, Google could retrieve the objects directly.
But enterprise data doesn't always live in S3.
It might be sitting on:
- an NFS share,
- a NAS appliance,
- a POSIX filesystem,
- private infrastructure,
- or storage that Google Cloud cannot directly access.
That changes the architecture.
For the second migration in my PoC, I used the same ~1.26 GB synthetic dataset from Part 1, but this time treated my local filesystem as the private source.
The migration path became:
POSIX Filesystem
↓
Podman
↓
Storage Transfer Agent
↓
Agent Pool
↓
Google Storage Transfer Service
↓
Google Cloud Storage
This implementation was considerably less smooth than the agentless S3 migration.
And that was probably the most useful part of it.
I ran into authentication problems, container registry failures, filesystem mount issues, and an architecture mismatch between my ARM64 Mac and the AMD64 Storage Transfer Agent image.
Eventually, all four files reached Google Cloud Storage successfully.
And once again, I refused to trust the green SUCCESS status until the SHA-256 hashes matched.
This is what changed when I introduced a transfer agent into the migration path.
Why Part 2 Exists
Part 1 solved a very specific problem:
Amazon S3
↓
Storage Transfer Service
↓
Google Cloud Storage
S3 exposes an API that Storage Transfer Service understands.
But imagine an enterprise migration where the source looks like this:
/data/archive
/data/applications
/data/backups
/data/shared
Those directories might live on a server inside a private network.
Google Cloud cannot simply call an S3 API and retrieve them.
Something needs access to that filesystem.
And that something needs to participate in the migration.
That's the role of the Storage Transfer Agent.
The Architecture Changed
The architecture for the second PoC looked like this:
Local / Private Environment
POSIX Filesystem
│
│ read-only mount
▼
Podman Linux VM
│
▼
Storage Transfer Agent
│
▼
Agent Pool
│
▼
Storage Transfer Service
│
▼
Google Cloud Storage
This is fundamentally different from Part 1.
The managed Storage Transfer Service still orchestrates the migration, but it can no longer directly reach the source.
The agent becomes the bridge between the private filesystem and Google's managed transfer infrastructure.
That immediately introduces something Part 1 didn't have:
infrastructure on the source side.
And infrastructure always brings assumptions with it.
Reusing the Same Dataset
I deliberately reused the same dataset from Part 1.
The source directory contained:
dataset-01.bin ~400 MiB
dataset-02.bin ~400 MiB
dataset-03.bin ~400 MiB
metadata.txt
Total:
~1.2 GiB
~1.26 GB
4 files
Using the same dataset mattered because I wanted each migration method to operate against an identical validation baseline.
The original SHA-256 manifest had already been generated before any migration:
source-sha256.txt
That file remained outside the migration directory.
So the source presented to the transfer agent contained only:
niveus-migration-demo/
├── dataset-01.bin
├── dataset-02.bin
├── dataset-03.bin
└── metadata.txt
The checksum manifest wasn't allowed to travel with the dataset it was supposed to validate.
Why Podman?
My development machine was macOS.
The Storage Transfer Agent needed a Linux environment.
I was already using Podman, so instead of provisioning a separate Linux VM purely for the PoC, I ran the agent through Podman's Linux virtual machine.
My environment looked roughly like:
Host:
macOS
Container runtime:
Podman
Podman VM:
AppleHV
CPU:
5
Memory:
8 GiB
Container architecture:
Linux ARM64
That last detail would become important later.
At this point, though, the first thing I wanted to prove was much simpler:
Can a Linux container actually see my source data?
Verify the Filesystem Before Blaming the Migration Service
Before installing the transfer agent, I tested the source mount with a minimal Alpine container.
Conceptually:
podman run --rm \
-v /path/to/niveus-migration-demo:/mnt/niveus-migration-demo:ro \
alpine \
ls -lh /mnt/niveus-migration-demo
Inside the container I expected to see:
dataset-01.bin
dataset-02.bin
dataset-03.bin
metadata.txt
And I did.
That small test eliminated several possible problems before the Storage Transfer Agent entered the picture.
It proved:
Host Filesystem
↓
Podman VM
↓
Container Mount
↓
Source Files
PASS
I also mounted the directory as:
:ro
Read-only.
The transfer agent's job was to migrate the source.
There was no reason for it to modify the source.
The filesystem permissions reflected that.
Creating the Agent Pool
Storage Transfer Agents don't operate independently.
They belong to an agent pool.
I created a dedicated pool for the PoC and configured it without an artificial bandwidth limit.
For this test:
Agent pool
│
└── 1 transfer agent
For a production migration, the architecture could contain multiple agents:
Agent Pool
┌──────────┼──────────┐
│ │ │
Agent 1 Agent 2 Agent 3
│ │ │
└──────────┼──────────┘
│
▼
Storage Transfer Service
Multiple agents can become relevant when throughput, scale, or redundancy requirements justify them.
But for a 1.26 GB PoC, one agent was enough.
Then Authentication Failed
This was the first real problem.
When I attempted to configure the environment for the transfer agent, the required Application Default Credentials weren't available.
The error pointed toward:
~/.config/gcloud/application_default_credentials.json
The credentials file didn't exist.
This is an easy detail to miss because being logged into the gcloud CLI and having Application Default Credentials (ADC) available are related concepts, but they aren't the same thing.
I had authenticated with Google Cloud.
The application still didn't have the credentials it expected.
The fix was:
gcloud auth application-default login
Then I validated ADC without printing the actual access token:
gcloud auth application-default print-access-token >/dev/null \
&& echo "ADC OK"
The important lesson wasn't the command.
It was understanding the authentication boundary.
When a containerized application says:
"I cannot find credentials"
the fact that your terminal can successfully run gcloud commands doesn't automatically prove that the application has access to the credentials it needs.
Then Podman Couldn't Pull the Agent
With ADC sorted, I encountered the next issue.
Podman attempted to retrieve the Storage Transfer Agent image and returned authentication errors similar to:
unable to retrieve auth token
unauthorized
authentication failed
Again, the migration hadn't even started.
The container runtime simply couldn't authenticate to the registry hosting the agent image.
I authenticated Podman using a Google access token:
gcloud auth print-access-token | \
podman login \
-u oauth2accesstoken \
--password-stdin \
gcr.io
After that:
Login Succeeded!
Now the agent image could be pulled.
This is exactly the sort of detail architecture diagrams hide.
The diagram says:
Podman
↓
Storage Transfer Agent
Reality says:
Which registry?
Which credentials?
Where are those credentials stored?
Can the container runtime actually use them?
The arrow is usually the easy part.
Everything required to make the arrow real is where the engineering happens.
Getting Credentials Into the Agent Environment
The agent itself also needed access to Google Cloud credentials.
For the PoC, I created a dedicated Google Cloud CLI container and authenticated Application Default Credentials inside it.
Conceptually:
podman run -ti --replace \
--name gcloud-config \
gcr.io/google.com/cloudsdktool/google-cloud-cli:stable \
gcloud auth application-default login
That container stored the ADC configuration.
The Storage Transfer Agent could then inherit the credential volume using:
--volumes-from gcloud-config
So the runtime arrangement became:
gcloud-config container
│
│ credentials
▼
Storage Transfer Agent
│
▼
Google Cloud
This was appropriate for the PoC.
For production, credential management deserves a much more deliberate security design.
Starting the Transfer Agent
With authentication working and the source filesystem mounted, I started the agent.
The important parts of the command were conceptually:
podman run \
-d \
--rm \
--name storage-transfer-agent \
--volumes-from gcloud-config \
-v /path/to/source:/mnt/niveus-migration-demo:ro \
gcr.io/cloud-ingest/tsop-agent:latest \
--project-id=<PROJECT_ID> \
--hostname=<AGENT_HOSTNAME> \
--agent-pool=<AGENT_POOL>
There are three relationships worth noticing.
The agent needs:
- Google Cloud credentials.
- Access to the source filesystem.
- Membership in the correct agent pool.
Once the container started, I checked:
podman ps
and then:
podman logs --tail 50 storage-transfer-agent
The logs confirmed the configured project, hostname, and agent pool.
More importantly, Google Cloud now showed:
1 agent connected
The bridge between my local filesystem and Storage Transfer Service was alive.
Then My Mac Reminded Me It Was ARM64
This was the most interesting infrastructure detail in the experiment.
My Mac was ARM64.
The Storage Transfer Agent image I was running was AMD64.
Podman warned me:
image platform (linux/amd64)
does not match expected platform (linux/arm64)
I also encountered:
rseq syscall failed with errno 38
The agent still started.
The migration still worked.
But this changed how I interpreted the performance results.
My architecture was effectively:
ARM64 Mac
↓
Podman Linux VM
↓
AMD64 container under emulation
↓
Storage Transfer Agent
That's perfectly acceptable for demonstrating functionality.
It is not an environment from which I would publish a serious performance benchmark.
This distinction matters.
A PoC can prove:
The architecture works.
It does not necessarily prove:
This is how fast the architecture will run in production.
Those are different claims.
Creating the POSIX Transfer Job
Once the agent was connected, I created a Storage Transfer job using:
Source:
POSIX filesystem
Destination:
Google Cloud Storage
The selected agent pool was the one containing my running transfer agent.
The source path was:
/mnt/niveus-migration-demo
And this is important:
That was the path inside the transfer-agent container.
Not the macOS path.
From the host's perspective, the source might be:
/Users/.../niveus-migration-demo
From the agent's perspective, it was:
/mnt/niveus-migration-demo
Containerized systems don't care what the path looks like on your laptop.
They care about the filesystem namespace visible inside the container.
That distinction is obvious once you understand containers.
It's also one of the easiest ways to configure a filesystem migration incorrectly.
Metadata Was Deliberately Kept Simple
POSIX files contain metadata beyond their contents.
Ownership.
Group IDs.
Permission modes.
Symlinks.
For this PoC, I wasn't trying to validate a complete enterprise filesystem-preservation strategy.
So I explicitly skipped:
GID
UID
Mode
Symlink handling
The goal was narrower:
Can the agent move the test files from the POSIX source into GCS and preserve their contents exactly?
A production filesystem migration would require a much more careful discussion about metadata semantics and how POSIX properties should map to the destination environment.
Again, the PoC was intentionally proving one thing at a time.
The Transfer Ran
Once the job started, the agent read the source filesystem and transferred the files through Storage Transfer Service into the destination GCS bucket.
The observed result was:
Status: SUCCESS
Files: 4 / 4
Transferred: 1.26 GB / 1.26 GB
Errors: 0
Skipped: 0
Duration: 2 minutes 14 seconds
Average throughput: 9.33 MB/s
At first glance, it's tempting to compare that with Part 1.
Part 1:
28.21 MB/s
Part 2:
9.33 MB/s
And conclude:
"Agent-based migration is slower."
I don't think the experiment supports that conclusion.
The environments weren't comparable.
Part 2 included:
- a macOS host,
- a Podman Linux VM,
- filesystem mounting,
- an ARM64 host,
- and an AMD64 agent running through emulation.
The number tells me what happened in this PoC.
It does not tell me the inherent performance difference between the two migration architectures.
That would require controlled benchmarking under comparable environments.
It's an important distinction because benchmark-looking numbers have a habit of escaping their original context.
The Green Checkmark Returned
Once again, Storage Transfer Service showed:
SUCCESS
And once again, I wasn't finished.
The destination contained:
dataset-01.bin
dataset-02.bin
dataset-03.bin
metadata.txt
Four files.
Approximately 1.26 GB.
Zero transfer errors.
All good signals.
But Part 1 had already established the rule for this series:
The system performing the migration should not be the only system verifying the migration.
So I created another independent validation directory.
SHA-256, Again
I downloaded the destination files and generated another manifest:
shasum -a 256 \
dataset-01.bin \
dataset-02.bin \
dataset-03.bin \
metadata.txt > agent-destination-sha256.txt
Then compared it with the original source manifest:
diff ~/source-sha256.txt agent-destination-sha256.txt
echo $?
Result:
0
Which meant:
Original Source
│
│ SHA-256
▼
Source Manifest
=
GCS Destination
│
│ SHA-256
▼
Destination Manifest
All four files matched byte-for-byte.
SHA-256 PASS
Only then did I consider Method 2 validated.
What Actually Changed From Part 1?
At first, Part 1 and Part 2 seem to solve the same problem.
Move data into GCS.
But architecturally they're very different.
Part 1
S3
↓
Storage Transfer Service
↓
GCS
Google understands the source natively.
Part 2
Private Filesystem
↓
Transfer Agent
↓
Agent Pool
↓
Storage Transfer Service
↓
GCS
Google cannot directly access the source.
So an agent must operate close to the data.
That additional component introduces:
- runtime requirements,
- authentication,
- filesystem access,
- connectivity,
- deployment,
- monitoring,
- and capacity planning.
The agent solves an access problem.
But like every architectural component, it comes with operational cost.
When I'd Use the Agent Pattern
This architecture becomes useful when the source lives somewhere Storage Transfer Service cannot access directly through a native cloud API.
Examples include:
POSIX filesystems
NFS
NAS
Private storage environments
S3-compatible storage platforms
The general pattern becomes:
Data
↓
Agent close to data
↓
Managed transfer orchestration
↓
Google Cloud Storage
The important phrase is:
close to the data.
For large enterprise migrations, moving the agent close to the source avoids introducing unnecessary intermediate storage layers and allows the migration architecture to work with private environments.
What I'd Do Differently in Production
My laptop was useful for proving the architecture.
I wouldn't use it as the production architecture.
For a real migration, I'd want to think carefully about:
Agent placement
Agents should run on infrastructure with reliable access to the source data and sufficient connectivity to Google Cloud.
Multiple agents
Large migrations may require multiple agents for throughput and redundancy.
Bandwidth controls
Migration traffic shouldn't accidentally saturate links required by production applications.
Authentication
Developer ADC credentials were acceptable for my local PoC.
Production identity should use appropriately managed service identities and credential controls.
Host architecture
I would avoid running an AMD64 agent under ARM64 emulation if I cared about meaningful performance measurements.
Metadata
UID, GID, permissions, symlinks, timestamps, and application-specific filesystem semantics need explicit validation.
Monitoring
Agent health becomes part of migration health.
If the agent stops, the migration path stops.
The infrastructure needs to be observable.
What Broke Was More Useful Than What Worked
Part 1 was cleaner.
Configure identity.
Configure permissions.
Create transfer.
Validate.
Part 2 made me deal with the layers underneath the diagram.
Credentials existed on the host but weren't available where the application expected them.
The container runtime couldn't initially authenticate to the registry.
The filesystem had to be visible inside the container namespace.
The agent had to register with the correct pool.
The image architecture didn't match the host architecture.
None of those problems were about moving files.
They were about making the environment around the transfer correct.
And that's probably the most realistic lesson from this part of the PoC.
Architecture diagrams make integrations look like arrows.
Implementation teaches you what those arrows actually cost.
Two Migration Patterns, Two Different Problems
At this point in the series, we've implemented two approaches.
Native object-storage migration
Amazon S3
↓
Storage Transfer Service
↓
GCS
Use it when the source is natively supported.
Keep the architecture small.
Private filesystem migration
POSIX / NFS / Private Storage
↓
Transfer Agents
↓
Agent Pool
↓
Storage Transfer Service
↓
GCS
Use it when something needs to run close to the source data.
Don't introduce agents when you don't need them.
Don't avoid them when the source requires them.
The source architecture should determine the migration architecture.
Not personal preference.
But Both Methods Still Make One Assumption
There's something both Part 1 and Part 2 quietly assume.
We know what data we're migrating.
We start the migration.
The dataset gets transferred.
We validate it.
But what if the source doesn't stay still?
Imagine a production application continuing to write into Amazon S3 while you're performing the migration.
At 10:00:
1,000,000 objects
You begin the transfer.
At 10:30:
1,002,000 objects
At 11:00:
1,005,000 objects
Your migration target is moving while you're trying to copy it.
Now a one-time transfer isn't the whole solution.
We need a way to capture what changes after the initial copy.
That's where the third architecture begins.
Next: Migrating a Source That Won't Stop Changing
In Part 3, I'll return to Amazon S3.
But this time, the workload remains active.
Instead of repeatedly starting batch transfers manually, I'll connect:
S3 ObjectCreated
↓
S3 Event Notification
↓
Amazon SQS
↓
Storage Transfer Service
↓
Google Cloud Storage
New S3 objects generate events.
The events enter SQS.
Storage Transfer Service listens to that event stream and automatically transfers the newly created objects.
That leads to the architecture I actually find most interesting in this series:
Bulk Migration
+
Continuous Incremental Synchronization
+
Final Delta
+
Controlled Cutover
Because once the source continues changing, the problem is no longer:
How do I copy the data?
It becomes:
How do I move the application without losing the changes happening while I'm moving it?
And that's where storage migration starts becoming a distributed-systems problem.
Part 3: Migrating a Live S3 Workload to GCP Without Waiting for Writes to Stop →