OADP-5777: Add automatic S3 bucket region detection and document AWS HeadBucket API behavior #1740
Conversation
openshift-ci-robot
added
the
jira/valid-reference
|
@kaovilai: This pull request references OADP-5777 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.20.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
added
the
do-not-merge/work-in-progress
|
Skipping CI for Draft Pull Request. |
openshift-ci
bot
added
the
approved
|
/test unit-test |
|
/retest |
openshift-merge-robot
added
the
needs-rebase
kaovilai
changed the title
openshift-ci
bot
removed
the
do-not-merge/work-in-progress
openshift-merge-robot
removed
the
needs-rebase
Signed-off-by: Tiger Kaovilai <tkaovila@redhat.com>
kaovilai
changed the title
|
@kaovilai: This pull request references OADP-5777 which is a valid jira issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
This commit adds automatic region detection for AWS S3 buckets in BackupStorageLocation
configurations when using actual AWS S3 (not S3-compatible storage).
Changes:
- Modified UpdateBackupStorageLocation in pkg/common/common.go to auto-detect
and set the region when:
* Provider is "aws"
* No custom s3Url is configured (meaning it's real AWS S3)
* No region is already specified in the config
* A bucket name is provided in ObjectStorage
- The implementation uses aws.GetBucketRegion() which AWS Security confirmed
works with anonymous credentials for both public and private buckets
(Engagement ID: CACenGS4Mha_KeJ=e3jBSLD6rPZ2iNtfuJUv9QJViaCOt7GVNDg)
- Added comprehensive test cases to verify:
* Region auto-detection is skipped when region is already specified
* Region auto-detection is skipped for S3-compatible storage (with s3Url)
* Region auto-detection works with real AWS bucket (tested with
openshift-velero-plugin-s3-auto-region-test-1)
Benefits:
- Prevents configuration errors from incorrect region specifications
- Reduces manual configuration requirements for AWS BSLs
- Works seamlessly with existing anonymous credential approach
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
kaovilai
changed the title
| // s3:ListBucket permissions for retrieving bucket region information. | ||
| // Reference: AWS Security response (Engagement ID: CACenGS4Mha_KeJ=e3jBSLD6rPZ2iNtfuJUv9QJViaCOt7GVNDg) | ||
| // This is expected AWS behavior, not a security vulnerability. | ||
| o.Credentials = credentials.NewStaticCredentialsProvider("anon-credentials", "anon-secret", "") |
There was a problem hiding this comment.
The reason we are using here is
- simplifies unit test.
- why not. it works for prod too.
Update DoesBSLSpecMatchesDpa function to accept that DPA spec can have an empty region while the deployed BSL has an auto-detected region. The test now properly handles the scenario where: - DPA spec doesn't specify a region - No custom s3Url is configured (real AWS S3) - The deployed BSL has an auto-detected region This ensures the E2E test "DPA CR without Region, without S3ForcePathStyle and with BackupImages false" passes with the new auto-detection feature. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
|
/test 4.19-e2e-test-aws |
|
@kaovilai: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: kaovilai, shubham-pampattiwar The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
5 participants
Why the changes were made
This PR implements automatic S3 bucket region detection for AWS BackupStorageLocations (BSLs) and documents the expected AWS behavior for the
GetBucketRegionfunction, based on AWS Security's official confirmation.Key Improvements
Background
We discovered that the AWS SDK's
GetBucketRegionfunction works with anonymous credentials on both public and private S3 buckets. AWS Security has confirmed this is expected behavior, not a security vulnerability:GetBucketRegion) does not enforces3:ListBucketpermissions for region retrievalChanges Made
1. S3 Region Detection (
pkg/storage/aws/s3.go)GetBucketRegionto use anonymous credentials viacredentials.NewStaticCredentialsProvider2. Auto-Region Detection (
pkg/common/common.go)UpdateBackupStorageLocationfor AWS BSLss3Urlis configured (indicating real AWS S3, not S3-compatible storage)3. Test Coverage (
pkg/common/common_test.go&pkg/storage/aws/s3_test.go)openshift-velero-plugin-s3-auto-region-test-1Benefits
How to test the changes made
Run the S3 region tests:
go test -v ./pkg/storage/aws/... -run TestGetBucketRegionRun the BSL update tests:
go test -v ./pkg/common/... -run TestUpdateBackupStorageLocationThe tests demonstrate that:
GetBucketRegionsuccessfully retrieves region information for both public and private buckets using anonymous credentialsManual Testing
s3Url) doesn't trigger auto-detectionFixes: https://issues.redhat.com/browse/OADP-5777
Note
Responses generated with Claude