SSECustomerKey is str, but AWS docs show using bytes

[ThisGuyCodes]

Describe the bug
the SSECustomerKey arg (for various operations) is type str, but various AWS docs very clearly show passing in a bytes. This is a binary value so I believe bytes is "more" correct.

Example docs: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#uploading-downloading-files-using-sse-customer-keys

sample from above link:

import boto3
import os

BUCKET = 'amzn-s3-demo-bucket'
KEY = os.urandom(32)
s3 = boto3.client('s3')

print("Uploading S3 object with SSE-C")
s3.put_object(Bucket=BUCKET,
              Key='encrypt-key',
              Body=b'foobar',
              SSECustomerKey=KEY,
              SSECustomerAlgorithm='AES256')
print("Done")

# Getting the object:
print("Getting S3 object...")
# Note how we're using the same ``KEY`` we
# created earlier.
response = s3.get_object(Bucket=BUCKET,
                         Key='encrypt-key',
                         SSECustomerKey=KEY,
                         SSECustomerAlgorithm='AES256')
print("Done, response body:")
print(response['Body'].read())

To Reproduce

import boto3
import os

key = os.urandom(32)

s3 = boto3.client('s3')

s3.get_object(Bucket='sombucket', Key='somekey', SSECustomerAlgorithm='AES256', SSECustomerKey=key)

key will have a red underline, Pyright gives me this:

Pyright: Argument of type "bytes" cannot be assigned to parameter "SSECustomerKey" of type "str" in function "head_object"
  "bytes" is not assignable to "str"

Additional context
MacOS: 14.6.1
Boto3: 1.35.74
types-boto3[essential]: 1.35.74

[vemel]

Hello. Thanks for the bug report!

It looks like SSECustomerKey argument is mutated by boto3. I can override its type and set it to bytes. But I need to cross-check how it works.

[vemel]

As you see, boto3 documentation also refers to it as str in docs. E.g.: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/get_object.html . This is clearly a bug, because documentation is also generated using botocore shapes, same as my type annotations.

I am pretty sure there is some code in boto3 itself that mutates this argument.

[ThisGuyCodes]

it's very willing to accept a str as well, but when I have raw bytes converting it involves making an encoding decision and idk what boto3 expects.

IMO I think overriding this to be bytes | str is probably best? You're almost certainly correct about boto3 doing mutation internally.

[vemel]

@ThisGuyCodes
Should be fixed in main.

• SSECustomerKey and CopySourceSSECustomerKey arguments accept str | bytes
• SSECustomerKeyMD5 and CopySourceSSECustomerKeyMD5 arguments are removed because boto3 calculates them internally

You can help me by testing unreleased changes:

# install uv: https://docs.astral.sh/uv/getting-started/installation/

# build types-boto3-s3 package in ./typings directory
uvx --from git+https://github.com/youtype/mypy_boto3_builder@main mypy_boto3_builder ./vendored --product types-boto3-services -s s3 --output-type wheel

# install generated package
pip install vendored/types_boto3_s3*

# run type checker on your codebase

Please test if you have time and let me know if it works as expected.

[ThisGuyCodes]

I'm not sure I did this right, but no it's still failing:

$ uvx --from git+https://github.com/youtype/mypy_boto3_builder@main mypy_boto3_builder ./vendored --product types-boto3-services -s s3 --output-type wheel
 Updated https://github.com/youtype/mypy_boto3_builder (ff90cfba)
13:03:20 mypy_boto3_builder: INFO    Generating types-boto3-services product
13:03:20 mypy_boto3_builder: WARNING boto3 is not installed, using botocore version: 1.35.74
13:03:20 mypy_boto3_builder: INFO    [1/1] Generating types-boto3-s3 1.35.74.post1
$ uv pip install vendored/types_boto3*
Resolved 1 package in 1ms
Prepared 1 package in 4ms
Uninstalled 1 package in 3ms
Installed 1 package in 1ms
 ~ types-boto3-s3==1.35.74.post1 (from file:///Users/travisjohnson/code/whelp/vendored/types_boto3_s3-1.35.74.post1-py3-none-any.whl)
$ uv run pyright main.py
/Users/travisjohnson/code/whelp/main.py
  /Users/travisjohnson/code/whelp/main.py:28:24 - error: Argument of type "bytes" cannot be assigned to parameter "SSECustomerKey" of type "str" in function "head_object"
    "bytes" is not assignable to "str" (reportArgumentType)
1 error, 0 warnings, 0 informations
$

I have no system-wide pip

[vemel]

Hmm, you did everything right. As I see, the pacakge has been successfully installed in your venv. However, I do not see that type checking for client.head_boject method has changed. It might be pyright cache, or you are also using boto3-stubs (please uninstall it if you do, use types-boto3 instead).

Also, you can check HeadObjectRequestRequestTypeDef in .venv/lib/python3.13/site-packages/types_boto3_s3/type_defs.pyi (Python version might be different) and make sure that SSECustomerKey: NotRequired[Union[str, bytes]]. Let me know if it is still SSECustomerKey: NotRequired[str] for some reason.

I tested the change locally, it should work. I am going to release updated package tomorrow.

[ThisGuyCodes]

my bad! I posted the issue for boto3, but I had changed the actual thing I wrote to use aiobotocore (I believe they both get their types from here, so this should fix both), when I did the simplified example it worked! 👍🏻

(actually I got this error first, but I just pip installed the py3 one and everything was golden)

$ uv pip install vendored/types_boto3*
error: Requirements contain conflicting URLs for package `types-boto3-s3`:
- vendored/types_boto3_s3-1.35.74.post1-py3-none-any.whl
- vendored/types_boto3_s3-1.35.76-py3-none-any.whl

[vemel]

Good! THank you for testing.

You can also generate type annotations for aiobotocore/aioboto3:

# aiobotocore
uvx --from git+https://github.com/youtype/mypy_boto3_builder@main mypy_boto3_builder ./vendored --product aiobotocore-services -s s3

pip install vendored/types_aiobotocore_s3*

Type annotatiosn for aio libraries are different, so it is not safe to use types-boto3 when working with botocore

[vemel]

Fixed in the latest builder release. Also, I released packages with the fix included:

• mypy-boto3-s3 1.35.76
• types-boto3-s3 1.35.76
• types-aiobotocore-s3 2.15.2.post1

Please let me know if it works as expected now.

[ThisGuyCodes]

beautiful! Thank you!

I'm not sure it's related, but I noticed that types-aiobotocore[full] wasn't actually giving me types-aiobotocore-s3? I had to change full to all to get all the packages?

[vemel]

Hmm, probably docs are misleading a bit.

types-aiobotocore[full] installs types-aiobotocore-full package, shich contains type annotations for all services. So, you do not need types-aiobotocore-s3 or types-aiobotocore[all] when using types-aiobotocore[full].

The bugfix for this issue is included to types-aiobotocore-full 2.15.2.post1, so you can continue using types-aiobotocore[full].

[ThisGuyCodes]

perhaps you misunderstood? With types-aiobotocore[full] I did not get the s3 service annotations, I switched to types-aiobotocore[all] and then I did have all the annotations (including s3).

[vemel]

For some reason types_aiobotocore_s3/service_resource.pyi was not included to the latest types-aiobotocore-full release. I will check CI and re-release the package.

[vemel]

Fixed in 8.6.2 and published updated types-aiobotocore-full