multipart upload fails if checksum calculation enabled and no checksum algorithm applied to multipart part uploads

[steveloughran]

Describe the bug

2.35.8

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

When uploading multipart data through an s3 client whose checksum calculation has been set to requestChecksumCalculation(WHEN_SUPPORTED) then it is impossible to upload data blocks in a multipart upload unless a checksum algorithm is explicitly attached to the upload part request.

If you do not do this the part upload succeeds, but part complete fails with parts reported absent "The part may not have been uploaded, or the specified entity tag may not match the part's entity tag."

Current Behavior

mulipart upload parts were successfully uploaded, but on the final POST an error was reported saying the parts were missing.

org.apache.hadoop.fs.s3a.AWSBadRequestException: Completing multipart upload id l8itQB.
5u7TcWqznqbGfTjHv06mxb4IlBNcZiDWrBAS0t1EMJGkr9J1QD2UAwDM5rLUZqypJfWCoPJtySxA3QK9QqKTBdKr3LXYjYJ_r9lRcGdzBRbnIJeI8tBr8yqtS on
test/testCommitEmptyFile/empty-commit.txt:
software.amazon.awssdk.services.s3.model.S3Exception: One or more of the specified parts could not be found.
The part may not have been uploaded, or the specified entity tag may not match the part's entity tag.
(Service: S3, Status Code: 400, Request ID: AQ0J4B66H626Y3FH,
Extended Request ID: g1zo25aQCZfqFh3vOzrzOBp9RjJEWmKImRcfWhvaeFHQ2hZo1xTm3GVMD03zN+d+cFB6oNeelNc=)
(SDK Attempt Count: 1):InvalidPart: One or more of the specified parts could not be found.

Reproduction Steps

take this branch of hadoop, build a release: https://github.com/steveloughran/hadoop/tree/s3/HADOOP-19654-aws-sdk-2.32

set up with generation (i.e. calculation) enabled, but leave the algorithm to use when uploading files to "none"

fs.s3a.checksum.generation true
fs.s3a.create.checksum.algorithm none

upload a file > 64MB so the multipart upload kicks in.
Observe an error on committing the work.

If you are set up to run the hadoop integration tests then you will see failures in ITestCommitOperations and all tests doing multipart uploads.

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.35.4

JDK version used

openjdk version "1.8.0_362" OpenJDK Runtime Environment (Zulu 8.68.0.21-CA-macos-aarch64) (build 1.8.0_362-b09) OpenJDK 64-Bit Server VM (Zulu 8.68.0.21-CA-macos-aarch64) (build 25.362-b09, mixed mode)

Operating System and version

Darwin 24.6.0 Darwin Kernel Version 24.6.0: Mon Aug 11 21:16:21 PDT 2025; root:xnu-11417.140.69.701.11~1/RELEASE_ARM64_T6000 arm64

[zoewangg]

Taking a look

[zoewangg]

Hi @steveloughran I'm not able to reproduce the issue by setting requestChecksumCalculation to WHEN_SUPPORTED and not passing a checksum algorithm in multipart upload requests. The SDK should use CRC32 by default in this case. Did all multipart upload test cases fail? or just some of them?

[steveloughran]

it was tests against s3 stores where we'd set the create checksum to off and had gone with the new SDK changes of (by default) checksum calculation = when_supported. The debugger showed that the part upload classes did have checksums in them (this is for immediate completion, not the s3a committer), and I always got that error. Turn on creation checksum on uploads: all good. Turn down checksum calculation to when_required: all good. It's the combination of no create checksum and request checksum calculation leads to failures.