SLS Parameter Validation#
Parameter validation is feature essentially enables documentation and validation of params used in an SLS file. It is not at all related to params processing, as params processing only occurs during state sub-command execution - where actual param values are available. During validate phase we have no idea what params the given SLS uses. In fact, to extract those out of SLS is our goal.
Goal#
The goal of SLS parameter validation is to extract/document parameters being used in an SLS file (including any files referred using include statement) for each state defined in the SLS file. Further, an additional goal is to do this transparently without exposing the end-user to any of the idem internals.
Limitation#
When jinja processes any document it does not have any context as to what is the state that is being currently processed, since jinja is indifferent to sls syntax and only focuses on the piece of code it needs to handle. Not only that, the params may be getting used outside of any state, for initializing a jinja variables, like so:
{% set value = params.get('value') %}
state A:
state.a.present:
group: {{ value }}
state B:
state.b.present:
group: {{ value }}
The above limitations are the overbearing force behind this implementation.
Overview of the process involved#
Step 1:- Transformation#
We let jinja process the document, but instead of sending a traditional dict object as params object, we send an object of type Parameters class as defined in idem/tool/parameter.py. This class transforms orginal string (e.g. {{ params.get('rg_name').get('your_rg', 'default') }}) into another base64 encoded string, which preserves the context (e.g. ?? params.get('rg_name').get('your_rg', 'default') ?? ^^rg_name^^.~~your_rg ??). Here first portion of ?? string has the original expression and second portion helps us identify the state inside which the param is referred.
The parameters are converted to string with markers such as ?? params.get('value') ?? ^^value^^ ??. Since, the ^^value^^ if comes as complex type such as json breaks in the string format, base64 encoding is done to all the parameter values to ensure all the types fit into this marker string.
Then, the base64 encoded marker strings are decoded back to its original form during the second pass when remapping is done below in Step 4.
The above example transforms as:
state A:
state.a.present:
group: ?? params.get('value') ?? ^^value^^ ??
state B:
state.b.present:
group: ?? params.get('value') ?? ^^value^^ ??
As you can see above, the original parameter context, as well as the original params.get() string, are well preserved in the transformed YAML.
Step 2:- Extraction of parameters#
This step involves extracting the parameters out of transformed YAML using relevant regular expressions. I have tried to capture the details within the file idem/idem/validate/0001_find_params.py itself.
Step 3:- Tallying with meta section in SLS#
This step is simply giving warnings if a parameter used in any given state doesn’t have a corresponding definition in the meta section of the SLS. This is just for aiding the SLS writer so that he/she can add a meta section if it is missing. Refer: idem/idem/validate/0010_validate_meta.py
Step 4:- Remapping transformed strings original values#
This is simply mapping and base64 decoded transformed strings (base64 encoded) (e.g. ?? params.get('rg_name').get('your_rg', 'default') ?? ^^rg_name^^.~~your_rg ??) to original string (e.g. {{ params.get('rg_name').get('your_rg', 'default') }}`). Refer: `idem/idem/validate/0020_reverse_map.py
Sample Output#
For your reference, here is the output of validate sub-command on the above SLS:
{
"high": {
"state A": {
"state.a.present": {
"group": "{{ params.get('value') }}"
},
"__sls__": "test"
},
"state B": {
"state.b.present": {
"group": "{{ params.get('value') }}"
},
"__sls__": "test"
}
},
"low": [
{
"state": "state.a.present",
"name": "state A",
"__sls__": "test",
"__id__": "state A",
"fun": "group",
"order": 1
},
{
"state": "state.b.present",
"name": "state B",
"__sls__": "test",
"__id__": "state B",
"fun": "group",
"order": 1
}
],
"meta": {
"SLS": {},
"ID_DECS": {}
},
"parameters": {
"GLOBAL": {
"value": ""
},
"ID_DECS": {
"test.state A": {
"value": ""
},
"test.state B": {
"value": ""
}
}
},
"warnings": {
"GLOBAL": [],
"ID_DECS": {
"test.state A": {
"params_meta_missing": [
"value"
]
},
"test.state B": {
"params_meta_missing": [
"value"
]
}
}
}
}
You can see above two new sections are added in validate sub-command output, viz. parameters and warnings.
Some Additional Samples#
Some examples with SLS and corresponding validation output.
Iterating over list items:
{% for ruleId in params.get('ruleIds') %} {{ ruleId }}: securestate.rules_status.present: - abc: def {% endfor %}
{ "parameters": { "GLOBAL": { "ruleIds": "" }, "ID_DECS": { "resource_group.{{ params.get('ruleIds') }}": { "ruleIds": "" } } } }
Iterating over dict items:
{% set ruleIds = params.get("ruleDict") %} {% for key, value in ruleIds.items() %} {{ key }}: securestate.rules_status.present: - x: value-{{ value }} {% endfor %}
{ "parameters": { "GLOBAL": { "ruleDict": "" }, "ID_DECS": { "resource_group.key-{{ params.get('ruleDict') }}": { "ruleDict": "" } } } }
General use case of calling a function:
{% set ruleIds = params.get("ruleIdsString") %} {% for ruleId in ruleIds.split(',') %} {{ ruleId }}: securestate.rules_status.present: - x: y {% endfor %}
{ "parameters": { "GLOBAL": { "ruleIds": "" }, "ID_DECS": { "resource_group.{{ params.get('ruleIdsString') }}": { "ruleIdsString": "" } } } }