How to handle AWS S3 paths using cloudpathlib?

You might be aware that pathlib.Path cannot properly deal with S3-like paths. Cloudpathlib is an easy to use package that does handle AWS S3 paths as well as other cloud provider paths. In this example we use cloudpathlib.CloudPath instead of pathlib.Path to instantiate a S3Path object from a S3 URL string. The interface of a S3Path is the same as for a PosixPath object. For example, we can call .parts on it to obtain all components of the S3Path. 

Github gist with code

dependencies: python3.9, cloudpathlib==0.10.0

An easy way to integration test your calls to S3 with moto

Moto is a great package to mock AWS services and I think it is an easy way to have integration tests for your - non-critical - code that interacts with AWS like e.g. boto3-related calls. In our example with have a custom method that creates a AWS S3 bucket. Using the moto.mock_s3 decorator on our test method automatically mocks all calls to S3.

Github gist with code

dependencies: python3.9, boto3 ==1.24.51, moto==3.1.18

Show tqdm progress bar while uploading file to S3 with boto3

With boto3 we can obtain a S3 client to upload a local file to AWS S3 with upload_file. For larger files it might be nice to see the progress of the upload. This can be achieved with tqdm.tqdm as a callback.

Github gist with code

dependencies: python3.9, boto3 ==1.24.51, tqdm==4.64.0