Python and boto3 knocking my broadband out

I knocked up a script to download a number of large zip files from s3. Super simple with boto3 and Python but it was so effective at using my 100Mb/s network connection that everything else got knocked off the network including Chrome on the same PC. Luckily boto3 1.19.5 and later now support a bandwidth limit for s3 trasnfers so after adding that I was able to keep it at a sensible level.
import boto3
from boto3.s3.transfer import TransferConfig

print('boto3 version')
print(boto3.__version__)

# Download an S3 object - max_bandwidth is in Bytes per sec so 1024**2*5 is 5 MiB/s
download_config = TransferConfig(max_bandwidth=1024**2*5)
s3 = boto3.client('s3')
s3_client = boto3.client('s3')
s3_bucket = 'your_bucket_name'
filename = 'your_file'

with open(filename, 'wb') as f:
    print(f'Downloading {filename}')
    s3_client.download_fileobj(s3_bucket, filename_file, f, Config=download_config)
See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html?highlight=max_bandwidth" for the boto3 docs on this.

Comments

Popular posts from this blog

Using Python to Automate Visio

Microsoft accounts for children - Verify that you're an adult error

The computer says no