How to read field values from environment variables with pydantic?

To read values and secrets from environment variables we can use python's os module. Another option is a model class based on pydantic.BaseSettings. The nice thing with BaseSettings is that by default it tries to read field values from environment variables that are either named exactly like the field name in all caps, defined by Field.env, or defined by the model class config. In addition, we can also decide to pass the values as kwargs during instantiation of the model class which then has precedence over the values from the environment variables. If we do not have the expected environment variables set and no arguments are passed during init, pydantic throws a ValidationError.

Github gist with code

dependencies: python3.9, pydantic==1.9.2

How to protect your secrets from accidental exposure by tracebacks and logs?

When you have to deal with secrets like authorization credentials and passwords you want to make sure that those values are not accidentally exposed in error messages, tracebacks, or logs. One easy way to improve protection of sensitive values is to use SecretStr from pydantic to store these values.  SecreStr is formatted as "**********". That means when we call print() or str() on a SecretStr object no sensitive information will be exposed. To access the value of a SecreStr object we call .get_secret_value() and since this only needs to happen when we hand the secrets other to perform authentication or login we reduce the chances of exposure.

Github gist with code

dependencies: python3.9, pydantic==1.9.2

A convenient FileHandler to read text from local files and files on AWS S3

How can we build one class to handle reads from both, local files as well as files in AWS S3? cloudpathlib is a nice package that can handle S3 paths (see my post on cloudpathlib.CloudPath). cloudpathlib.AnyPath is a polymorphic class that will automatically instantiate a CloudPath or a pathlib.Path object - whatever is appropriate based on the input string. In this example, we build a pydantic.BaseModel with one field file of type AnyPath. This pydantic model is our FileHandler to read content from files - be it from a local file system or AWS S3. The nice part is that AnyPath does all the heavy lifting for us since it will instantiate the appropriate Path or S3Path objects with a common interface. That's why we can just always call .read_text() in the get_content method of our FileHandler. To test our FileHandler class we use moto.mock_s3 (see my post on using moto.mock_s3) to mock calls to AWS S3 and tempfile.NamedTemporaryFile to create a temporary local file. The simplicity of the FileHandler class is unfortunately a bit buried in the setup for testing it in this example.

Github gist with code

dependencies: python3.9, boto3 ==1.24.51, cloudpathlib==0.10.0, moto==3.1.18, pydantic==1.9.2

Instantiate a pydantic BaseModel from a dictionary object

To instantiate a pydantic.BaseModel we can call the parse_obj method of our model class and provide a dictionary with the content.

Github gitst with code

dependencies: python3.9, pydantic==1.9.2