On this page
The first version of this guide, written in 2018, split large files into 100 MB chunks with split, uploaded each one, and compared md5sum values by hand. It worked. It is also unnecessary for S3 today, because the AWS CLI does all of that for you, in parallel, with integrity checks built in. This rewrite shows the modern way and keeps the manual method only for the cases where it still earns its place.
Upload large files to S3 with one command
aws s3 cp ./backup-2026-09.tar.gz s3://my-bucket/backups/That one command is a multipart upload for anything over 8 MB. The CLI cuts the file into chunks, uploads several at once, retries parts that fail, and asks S3 to assemble them at the end. You don't see any of it except a progress line, which is why an AWS S3 copy of a large file needs nothing special from you.
If you don't have the CLI yet, install AWS CLI v2 first. Version 1 from pip handles multipart too, but it lacks the faster transfer client and automatic checksums described below.
Current S3 size limits
| Limit | Value |
|---|---|
| Maximum object size | 50 TB (raised from 5 TB in December 2025) |
| Maximum single PUT upload | 5 GB |
| Parts per multipart upload | 1 to 10,000 |
| Part size | 5 MB minimum (except the last part), 5 GB maximum |
| CLI multipart threshold and part size | 8 MB by default |
The 50 TB limit is recent enough that plenty of documentation still says 5 TB; AWS announced the change at the end of 2025. With 8 MB parts and a 10,000-part cap, the defaults top out around 80 GB, and the CLI increases the part size automatically when a file needs more.
AWS large file transfer speed
Upload speed is mostly limited by your network, but the defaults are conservative. These settings live in your CLI config, per profile:
aws configure set default.s3.max_concurrent_requests 20
aws configure set default.s3.multipart_chunksize 64MB
aws configure set default.s3.preferred_transfer_client crtmax_concurrent_requests(default 10) is how many parts upload at once. More helps on fast connections and hurts on slow ones.multipart_chunksize(default 8 MB). Larger parts mean fewer requests for multi-gigabyte files.preferred_transfer_clientdefaults toauto, which already picks the C-based CRT client when conditions allow. Settingcrtexplicitly makes it the preference.
Two bigger levers. If the data is already somewhere in AWS, run the upload from an EC2 instance in the same region as the bucket; the network between them is far faster than any office connection. And for uploads across continents, S3 Transfer Acceleration routes through AWS edge locations. Enable it on the bucket, then:
aws configure set default.s3.use_accelerate_endpoint trueAcceleration costs extra per gigabyte, so test with a real file before leaving it on. The full list of transfer settings is in the AWS CLI S3 configuration reference.
Failed uploads and cleanup
aws s3 cp retries individual parts, but if the whole command dies, from a closed laptop or a dropped SSH session, it does not resume. Run it again. For many files, aws s3 sync is friendlier after an interruption, because it skips files that already arrived:
aws s3 sync ./exports s3://my-bucket/exports/Here is the part that costs money quietly. An interrupted multipart upload leaves its uploaded parts in the bucket. They don't show up in aws s3 ls, but they are billed as storage until someone aborts the upload. Check for them:
aws s3api list-multipart-uploads --bucket my-bucketAnd stop worrying about it permanently with a lifecycle rule:
{
"Rules": [
{
"ID": "abort-incomplete-multipart",
"Status": "Enabled",
"Filter": {},
"AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
}
]
}aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.jsonLong uploads over SSH
tmux or screen, or with nohup, so a dropped connection doesn't kill the command halfway through a 200 GB file.
AWS S3 file integrity monitoring and checksums
The 2018 approach compared md5sum locally and on a server. For S3 you no longer need to. Current AWS CLI versions calculate a CRC checksum for each upload and send it with the data, and S3 checks it on arrival and rejects the upload if it doesn't match. Objects uploaded without a checksum get CRC64NVME added by S3 by default, part of the default data integrity protections AWS introduced.
You can choose the algorithm and read the stored checksum back:
aws s3 cp ./backup.tar.gz s3://my-bucket/backups/ --checksum-algorithm CRC32C
aws s3api head-object --bucket my-bucket --key backups/backup.tar.gz --checksum-mode ENABLEDDon't compare your file's MD5 with the object's ETag. For multipart uploads the ETag is built from the parts' MD5s and ends in - plus the number of parts, so it never matches a whole-file MD5. That mismatch has convinced a lot of people their upload was corrupted when it wasn't.
And there is no --checksum flag on aws s3 sync, despite how often people search for one. Sync compares size and timestamp to decide what to transfer; the checksum protection applies to each individual upload.
Split and merge files by hand, when it still helps
The original split-and-reassemble method is still useful when the destination isn't S3: uploading to a server behind a proxy that caps request size, or moving a huge file over a link so unreliable that you want to verify and resend pieces by hand.
split -b 100M -d huge-file.bin huge-file.part.
sha256sum huge-file.part.* > parts.sha256
# transfer the parts and parts.sha256, then on the other side:
sha256sum -c parts.sha256
cat huge-file.part.* > huge-file.binSHA-256 replaces the old MD5 here; it is just as easy and not broken. For anything going into S3, use multipart and let the CLI and S3 do this for you. Moving virtual machine images? AWS VM Import/Export covers what to do with them once uploaded, and backing up MongoDB on AWS is a common reason to push large files to S3 on a schedule.
Frequently Asked Questions
What is the largest file I can upload to S3?
A single S3 object can be up to 50 TB since AWS raised the limit from 5 TB in December 2025. A single PUT request is still limited to 5 GB, so anything bigger must use multipart upload, which aws s3 cp does automatically.
Does aws s3 sync have a --checksum option?
No. aws s3 sync decides what to copy by comparing file size and modification time, not content checksums. Integrity of each upload is checked separately: the CLI sends CRC checksums and S3 verifies them before storing the object.
Why doesn't the S3 ETag match my file's MD5?
For multipart uploads the ETag is not the MD5 of the file. It is derived from the MD5s of the individual parts and ends with a dash and the part count. Compare S3's stored checksum from head-object with --checksum-mode ENABLED instead.
How do I make large S3 uploads faster?
Increase max_concurrent_requests, raise multipart_chunksize for very large files, let the CLI use the CRT transfer client, and run the upload from an EC2 instance in the same region when the data is already in AWS. S3 Transfer Acceleration helps for long-distance uploads at extra cost.
Do failed multipart uploads cost money?
Yes. The parts of an upload that never completed stay in the bucket, billed as storage, and they don't appear in a normal listing. Add a lifecycle rule that aborts incomplete multipart uploads after a few days.
