Your Web News in One Place

An Interest In:

Web News this Week

Search Archive

Some of Our Sources

View All Sources

Help Webnuz

Referal links:

Sign up for GreenGeeks web hosting
January 1, 2022 02:15 am GMT

Doctoring your application configuration

Opinion: Code for configuration handling is ugly

If you've ever written a statement that looks something like my_setting = config.value if config.value else some_default and hated it, then I expect we can agree that handling application configuration is a miserable thing to code for. Its boring to write, ugly to read, and annoying to ensure that all user configuration options have sane default values.

Oscar the Grouch - I'll be miserable here in my trash can

The argparse library can go a long way to making things better, but if you need to set dozens of options, or if your application supports plugins with configuration requirements unknown to the main program, it becomes harder to simply add_argument our way back to sanity.

Mad scientist (black and white) - We'll see if I'm crazy or not!

In these cases, it makes more sense to opt for a configuration file. Also, it would be nice if our application code didn't care about the configuration file at all, where we had a single source of truth to count on being correct.

Getting Classy

When I've hit the point in my development process where I have the basic functionality working and I've settled on the basics of how the program will flow, I like to create a new python file constfig.py and define the class _C where the 'C' stands for both "constants" and "config" ( and that's good enough for me. Both config and constants start with C ). By convention, I lead the with an underscore to signal to the other developers on the project that the class isn't intended to be implemented directly, and at the end of the file I create an instance of _C() called C which can be imported and will contain all the information needed by the application.

For example, let's implement a simple dice rolling API endpoint using Flask which returns a JSON formatted string. For this type of application we would want to be able to easily configure the IP address and port that the service listens on, so let's define our _C class, establish the variable names, pre-populate the variables with some reasonable default values, and then create an instance of our class named C which the user is meant to import from our file.

class _C(object):    def __init__(self):        self.LISTEN_IP = '0.0.0.0'        self.LISTEN_PORT = 8080C = _C()

We also have certain values which will never change at runtime, but if our specification changes later, we don't want to have to hunt down all instances of that value in our code, so let's also add our constant values.

class _C(object):    def __init__(self):        # Constant values        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_roll'        # User configurable values        self.LISTEN_IP = '0.0.0.0'        self.LISTEN_PORT = 8080C = _C()

Now that we have our constant values and our configurable variables with sane defaults, let's import C (the instance of _C) into our main Flask application.

from constfig import C  # Our constants + config = constfigfrom random import randintfrom flask import Flask, jsonifyapp = Flask(__name__)@app.route('/rolld6')def roll_d6():    value = {C.JSON_RESPONSE_KEYWORD_D6ROLL: randint(1,6)}    return jsonify(value)if __name__ == '__main__':    app.run(host=C.LISTEN_IP, port=C.LISTEN_PORT)

We can now give it a quick run:

user@host:~ $ python3 roll.py  * Serving Flask app "roll" (lazy loading) * Environment: production   WARNING: This is a development server. Do not use it in a production deployment.   Use a production WSGI server instead. * Debug mode: offINFO:werkzeug: * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)

...and test to make sure all looks good so far.

user@host:~ $ curl -s localhost:8080/rolld6 | python3 -m json.tool{    "d6_value": 3}

Operate on your self

Now that we have our application and our _C class, we're ready for it to poke at its own guts.

Characters from the TV show Scrubs in an operating room. One of them is asking what is going on

Let's add the function load_config which will open up our configuration file config.yaml, and then use the setattr function to update our own values at startup (when Python runs from constfig import C). Python's ability to alter its own state is knows as "reflection" or "reflective programming".

import yamlimport loggingclass _C(object):    def __init__(self):        # Constant values - and a "gotcha!"        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_roll'        # User configurable values        self.LISTEN_IP = '0.0.0.0'        self.LISTEN_PORT = 8080        # Load user config (override defaults above)        self.load_config()    def load_config(self):        try:            config_file = open('config.yaml', 'r')            config_string = config_file.read()            config_file.close()            configuration = yaml.load(config_string, Loader=yaml.SafeLoader)  # Don't handle badly formatted YAML. Let the parser inform the user of the error.            if isinstance(configuration, dict):                for variable_name, value in configuration.items():                    setattr(self, variable_name, value)            else:                raise yaml.scanner.ScannerError(f'The file config.yaml should be structured as type dict, but got type {type(configuration)}')        except FileNotFoundError:            logging.warning('Configuration file config.yaml is missing. Using default values.')C = _C()

We can now create our config.yaml file containing key:value pairs where the key matches the name of our configuration items

LISTEN_IP: 127.0.0.1LISTEN_PORT: 32000

Let's fire up the service again...

user@host:~ $ python3 roll.py  * Serving Flask app "roll" (lazy loading) * Environment: production   WARNING: This is a development server. Do not use it in a production deployment.   Use a production WSGI server instead. * Debug mode: off * Running on http://127.0.0.1:32000/ (Press CTRL+C to quit)

...and this time we see that the default values have been overridden by those in the configuration file. Specifically, we're now listening on the loopback IP, and that our port number has changed as expected.

Gotta' catch that "gotcha!"

The problem with this method is that we open ourselves up to having the configuration file change values that should not be changed. For example, if we set JSON_RESPONSE_KEYWORD_D6ROLL: this is bad in config.yaml then make a request to our endpoint, we see that we've indirectly altered our application's response.

user@host:~ $ curl -s http://127.0.0.1:32000/rolld6 | python3 -m json.tool{    "this is bad": 4}

American Psycho - I know my behavior can be erratic sometimes

While the fix here isn't hard, this bad behavior underscores the importance of the order in which we set out values. Let's move our constant values to where they are being set after the values in our configuration file by adding a finally clause to our try/except block on load_config() and call our new method set_constants().

import yamlimport loggingclass _C(object):    def __init__(self):        # User configurable values        self.LISTEN_IP = '0.0.0.0'        self.LISTEN_PORT = 8080        # Load user config (override defaults above)        self.load_config()    def set_constants(self):        # Constant values        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_value'    def load_config(self):        try:            config_file = open('config.yaml', 'r')            config_string = config_file.read()            config_file.close()            configuration = yaml.load(config_string, Loader=yaml.SafeLoader)  # Don't handle badly formatted YAML. Let the parser inform the user of the error.            if isinstance(configuration, dict):                for variable_name, value in configuration.items():                    setattr(self, variable_name, value)            else:                raise yaml.scanner.ScannerError(f'The file config.yaml should be structured as type dict, but got type {type(configuration)}')        except FileNotFoundError:            logging.warning('Configuration file config.yaml is missing. Using default values.')        finally:            self.set_constants()C = _C()

...then launch again using our "bad" (which tries to set JSON_RESPONSE_KEYWORD_D6ROLL) configuration file.

user@host:~ $ python3 roll.py  * Serving Flask app "roll" (lazy loading) * Environment: production   WARNING: This is a development server. Do not use it in a production deployment.   Use a production WSGI server instead. * Debug mode: off * Running on http://127.0.0.1:32000/ (Press CTRL+C to quit)

...and of course test the result.

user@host:~ $ curl -s http://127.0.0.1:32000/rolld6 | python3 -m json.tool{    "d6_value": 5}

Tada! A single source of truth for your Python application!

The Yellow Ranger from the 90s TV show Power Rangers does a dance move and then a yellow explosion happens in the background (...and frankly, it looks like she farted). I have probably accidentally spent an hour of my life watching this gif while editing this post

You hate this, but I need validation

Yes, I hear you bemoan, "I've got a children's song about cookies stuck in my head, I'm half way through a pack of Oreos, and this seems like an abstraction that makes it harder to see how the configuration is loaded."

Yeah, this solution is probably not for everyone, but I've hidden the superpower of this method behind a mild mannered alter-ego.

Actor Christopher Reeve playing Clark Kent takes off his glasses, only to realize that he might be giving himself away as Superman

The real power is automatic validation of your configuration simply by importing C. To do this, let's define the function validate_config(), write some basic assertions to validate the configuration, and then call the function after we have successfully loaded the yaml file and set the constant values in our class.

import yamlimport loggingclass _C(object):    def __init__(self):        # Default values for user configurable items        self.LISTEN_IP = '0.0.0.0'        self.LISTEN_PORT = 8080        # Load user config (override defaults above)        self.load_config()    def set_constants(self):        # Constant values        self.JSON_RESPONSE_KEYWORD_D6ROLL = 'd6_value'    def load_config(self):        try:            config_file = open('config.yaml', 'r')            config_string = config_file.read()            config_file.close()            configuration = yaml.load(config_string, Loader=yaml.SafeLoader)  # Don't handle badly formatted YAML. Let the parser inform the user of the error.            if isinstance(configuration, dict):                for variable_name, value in configuration.items():                    setattr(self, variable_name, value)            else:                raise yaml.scanner.ScannerError(f'The file config.yaml should be structured as type dict, but got type {type(configuration)}')        except FileNotFoundError:            logging.warning('Configuration file config.yaml is missing. Using default values.')        finally:            self.set_constants()            self.validate_config()  # Validate our config file    def validate_config(self):        # Validate LISTEN_IP        assert isinstance(self.LISTEN_IP, str), 'LISTEN_IP is not a string value'        assert len(self.LISTEN_IP.split('.')) == 4, 'LISTEN_IP has an unexpected number of octets'        assert all([ip.isnumeric() for ip in self.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'        assert all([0 <= int(ip) <= 255 for ip in self.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'        # Validate LISTEN_PORT        if isinstance(self.LISTEN_PORT, str):            assert self.LISTEN_PORT.isnumeric(), 'LISTEN_PORT must be a whole number.'            self.LISTEN_PORT = int(C.LISTEN_PORT)        assert 999 < self.LISTEN_PORT < 65536, 'LISTEN_PORT is outside expected range.'C = _C()

...and just as an example, let's put a deliberate typo in our configuration file.

LISTEN_IP: 127.0.0.1x  # My fingers are fat :-(LISTEN_PORT: 32000

Now, have a couple of opportunities to validate our configuration.

For example, at runtime...

user@host:~ $ python3 roll.py Traceback (most recent call last):  File "roll.py", line 1, in <module>    from constfig import C  # Our constants + config = constfig  File "/Users/adrong/PycharmProjects/constfig/constfig.py", line 33, in <module>    validate_config()  File "/Users/adrong/PycharmProjects/constfig/constfig.py", line 14, in validate_config    assert all([a.isnumeric() for a in C.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'AssertionError: LISTEN_IP is not a valid IP address.

... or run constfig.py directly to validate your configuration in your test or deployment pipelines.

user@host:~ $ python3 constfig.py Traceback (most recent call last):  File "constfig.py", line 33, in <module>    validate_config()  File "constfig.py", line 14, in validate_config    assert all([a.isnumeric() for a in C.LISTEN_IP.split('.')]), 'LISTEN_IP is not a valid IP address.'AssertionError: LISTEN_IP is not a valid IP address.

Small, but mighty

The cartoon character Mighty Mouse eating a hole into a wheel of cheese, crawling into it, then eating it so fast it becomes a whirlwind that transforms into the hero mouse

This is a relatively small amount of code that, with some tweaks, can source configuration from a database, from environment variables, from command line arguments, and can have validation code that can reconcile configuration from any combination of those sources. This pattern for handling configuration has enabled me to quickly create configuration handling in a standardized way, across multiple tools , that members of other teams have found approachable and easy to manage.


Original Link: https://dev.to/dev_dull/doctoring-your-application-configuration-31f

Share this article:    Share on Facebook
View Full Article

Dev To

An online community for sharing and discovering great ideas, having debates, and making friends

More About this Source Visit Dev To