Replace Jinja2 with PyYAML

This commit is contained in:
s3lph 2021-05-30 17:52:25 +02:00
parent a3eb452786
commit 093a40fefa
31 changed files with 197 additions and 376 deletions

View file

@ -1,5 +1,5 @@
---
image: s3lph/spaceapi-server-ci:20200620-01
image: s3lph/spaceapi-server-ci:20210530-01
stages:
- test
@ -57,7 +57,7 @@ build_debian:
Section: web
Priority: optional
Architecture: all
Depends: python3 (>= 3.6), python3-jinja2, python3-bottle
Depends: python3 (>= 3.9), python3-yaml, python3-bottle
Description: Lightweight SpaceAPI endpoint server
Lightweight server for SpaceAPI endpoints. Includes support for pluggable
templating, so dynamic content, like sensor values, can be added.
@ -77,8 +77,8 @@ build_debian:
- python3 setup.py egg_info install --root=package/debian/spaceapi-server/ --prefix=/usr --optimize=1
- cd package/debian/spaceapi-server
- mkdir -p usr/lib/python3/dist-packages/
- cp -r usr/lib/python3.8/site-packages/spaceapi_server* usr/lib/python3/dist-packages/
- rm -rf usr/lib/python3.8/
- cp -r usr/lib/python3.9/site-packages/spaceapi_server* usr/lib/python3/dist-packages/
- rm -rf usr/lib/python3.9/
# Remove compiled Python files
- find usr/lib/python3/dist-packages -name __pycache__ -exec rm -r {} \; 2>/dev/null || true
- find usr/lib/python3/dist-packages -name '*.pyc' -exec rm {} \;
@ -109,7 +109,7 @@ build_archlinux:
script:
- find package/archlinux -name .gitkeep -delete
# Install dependencies
- pacman -Sy --noconfirm namcap python python-setuptools python-pip python-wheel python-jinja python-bottle base-devel
- pacman -Sy --noconfirm namcap python python-setuptools python-pip python-wheel python-yaml python-bottle base-devel
- export SPACEAPI_SERVER_VERSION=$(python -c 'import spaceapi_server; print(spaceapi_server.__version__)')
# Copy example plugin
- install -m0644 examples/plugins/example.py package/archlinux/spaceapi-server/etc/spaceapi-server/plugins/example.py

View file

@ -1,5 +1,19 @@
# SpaceAPI Server Changelog
<!-- BEGIN RELEASE v0.4 -->
## Version 0.4
Replace Jinja2+JSON with PyYAML
### Changes
<!-- BEGIN CHANGES 0.4 -->
- Switch config file format from JSON to YAML
- Remove Jinja2, replaced with YAML custom tags for plugin calls
<!-- END CHANGES 0.4 -->
<!-- END RELEASE v0.4 -->
<!-- BEGIN RELEASE v0.3.1 -->
## Version 0.3

175
README.md
View file

@ -11,7 +11,7 @@ values, can be added.
- Python 3 (>=3.6)
- [Bottle][pypi-bottle]
- [Jinja2][pypi-jinja2]
- [PyYAML][pypi-yaml]
## License
@ -24,9 +24,9 @@ SpaceAPI endpoint server. In its simplest configuration, it just
serves a plain, static, boring JSON document.
In order to provide dynamic content (e.g. whether your space is
currently open), you can replace parts of the JSON document (anything
except object keys) with [Jinja2 templates][jinja], from which you can
invoke custom plugins, which look up and return your dynamic content.
currently open), you can replace parts of the YAML document (anything
except object keys) with custom plugin invocations. These plugins
look up and return your dynamic content.
<table>
<thead>
@ -39,15 +39,19 @@ invoke custom plugins, which look up and return your dynamic content.
<tr>
<td>
```json
{
"api": "0.13",
"space": "My Hackerspace",
"state": "{{ space_state() }}",
"sensors": {
"network_connections": "{{ network_connections() }}"
}
}
```yaml
---
api: "0.13"
space: My Hackerspace
# This is a plugin invocation
# with no arguments
state: !space_state {}
sensors:
# This is a plugin invocation with
# arguments. They are passed to the
# plugin function as kwargs.
network_connections: !network_connections
networks: [ "2.4 GHz", "5 GHz" ]
```
</td>
@ -100,10 +104,10 @@ server as an OS distribution package.
The configuration of this server consists of three parts:
- The **main configuration** file, usually located at
`/etc/spaceapi-server/config.json`. This file controls all the
`/etc/spaceapi-server/config.yaml`. This file controls all the
internal settings of the server.
- The **response template** file, located at
`/etc/spaceapi-server/template.json`. This file defines the content
`/etc/spaceapi-server/template.yaml`. This file defines the content
served by your sever.
- The **plugins** directory, located at
`/etc/spaceapi-server/plugins/`. Here you can put your plugins for
@ -111,36 +115,38 @@ The configuration of this server consists of three parts:
### 2. Configure the Server
Open the file `/etc/spaceapi-server/config.json`.
Open the file `/etc/spaceapi-server/config.yaml`.
The following options are currently available:
```json
{
"address": "::1", # The address to listen on.
"port": 8000, # The TCP port to listen on.
"server": "wsgiref", # The Bottle backend server to use.
"template": "template.json", # Path to the SpaceAPI response template file.
"plugins_dir": "plugins", # Path to the directory containing your plugins.
"plugins": {
# Plugin-specific configuration should go in here, separated by plugin
"my_plugin": {
"my_option": "Hello, World!",
"my_other_option": [ 42, 1337 ]
}
}
}
```yaml
---
# The address to listen on.
address: "::1"
# The TCP port to listen on.
port: 8000
# The Bottle backend server to use.
server: wsgiref
# Path to the SpaceAPI response template file.
template: template.yaml
# Path to the directory containing your plugins.
plugins_dir: plugins
plugins:
# Plugin-specific configuration should go in here, separated by plugin
my_plugin:
my_option: "Hello, World!"
my_other_option: [ 42, 1337 ]
```
### 3. Configure a Static SpaceAPI Endpoint
Open the file `/etc/spaceapi-server/template.json`. By default it
Open the file `/etc/spaceapi-server/template.yaml`. By default it
contains a minimal example response. If you only want to serve static
content, your `template.json` should simply contain the SpaceAPI JSON
response you want to serve.
The content is served "almost-as-is" (once parsed and re-serialized).
The content is served "almost-as-is" (apart from the conversion from
YAML to JSON).
To learn about how a SpaceAPI response should look like, have a look
at [SpaceAPI: Getting Started][spaceapi-getting-started].
@ -160,6 +166,7 @@ the space as open depending on its existence.
from spaceapi_server import config, plugins
@plugins.template_function
# The plugin can be invoked by using the !space_state YAML tag
def space_state():
# Get the plugin config dict
conf = config.get_plugin_config('filestate')
@ -180,39 +187,27 @@ the space as open depending on its existence.
}
```
The `@template_function` decorator registers the function as a
callable in Jinja's globals. There's also `@template_filter`,
which registers a Jinja2 filter. For more information on the
Jinja2 templating engine, see [Jinja2][jinja].
The `@template_function` decorator registers a constructor in
PyYAML's parser with the function's name as tag
(e.g. `!space_state).
2. Call the template function in your template:
```json
{
# ...
"state": "{{ space_state() }}"
# ...
}
```yaml
# ...
state: !space_state
# ...
```
Although the value for the `state` key is a string containing the
Jinja2 template, the return value of your template function is a
complex data type, which will be inserted into the result as such.
Please be aware that only the first token of a templated string is
used in the result. This limitation is caused by the way Jinja2
is (ab-) used.
3. Configure the server:
```json
```yaml
# ...
"template": "template.json",
"plugins_dir": "plugins",
"plugins": {
"filestate": {
"filename": "/var/space_state"
}
}
template: template.yaml
plugins_dir: plugins
plugins:
filestate:
filename: /var/space_state
# ...
```
@ -230,7 +225,7 @@ e.g. through `systemctl reload`.
If you need to run the server ad-hoc, you can start it with:
```bash
python3 -m spaceapi_server /path/to/config.json
python3 -m spaceapi_server /path/to/config.yaml
```
### 6. Test the Server
@ -267,8 +262,8 @@ print(plugins.get_plugin_config('my_plugin'))
### Templating
The following decorators register a function in the Jinja2
environment, so they become usable from within templates. They are
The following decorators register a function for use as a PyYAML
constructor, so they become usable from within templates. They are
invoked when the template is rendered, which happens for each HTTP
request.
@ -277,11 +272,11 @@ plugins, or by using a caching HTTP reverse proxy.
#### `spaceapi_server.plugins.template_function`
This decorator registers a function as a **global function** in the
Jinja2 environment.
This decorator registers the function's name as a YAML tag in the parser.
The decorated function may take any arguments that can be represented
in a Jinja2 template.
The decorated function may take arguments (always passed as
`**kwargs`, so `*args`, or arguments before `*` won't work) that can
be represented in a YAML file.
The decorated function may return any value that can be serialized
into JSON. This includes objects and arrays.
@ -300,45 +295,11 @@ def lookup_sensor(query, default=None):
return default or []
```
```json
{
# ...
"state": "{{ lookup_sensor('SELECT timestamp, value FROM people_now_present LIMIT 1') }}"
# ...
}
```
#### `spaceapi_server.plugins.template_filter`
This decorator registers a function as a **filter** in the Jinja2
environment. The decorated function must take at least one argument.
The decorated function may take any arguments that can be represented
in a Jinja2 template.
The decorated function may return any value that can be serialized
into JSON. This includes objects and arrays.
Usage:
```python
from spaceapi_server import plugins
@plugins.template_filter
def lookup_sensor(query, default=None):
# Do something with the query
result = ...
# If the loo
if not result:
return default or []
```
```json
{
# ...
"state": "{{ 'SELECT timestamp, value FROM people_now_present LIMIT 1' | lookup_sensor }}"
# ...
}
```yaml
# ...
state: !lookup_sensor
query: SELECT timestamp, value FROM people_now_present LIMIT 1
# ...
```
@ -346,8 +307,8 @@ def lookup_sensor(query, default=None):
[releases]: https://gitlab.com/s3lph/spaceapi-server/-/releases
[spaceapi]: https://spaceapi.io/
[pypi-bottle]: https://pypi.org/project/bottle/
[pypi-jinja2]: https://pypi.org/project/Jinja2/
[pypi-yaml]: https://pypi.org/project/PyYAML/
[mit]: https://gitlab.com/s3lph/spaceapi-server/blob/master/LICENSE
[spaceapi-getting-started]: https://spaceapi.io/getting-started/
[jinja]: https://jinja.palletsprojects.com/
[registry]: https://gitlab.com/s3lph/spaceapi-server/container_registry
[registry]: https://gitlab.com/s3lph/spaceapi-server/container_registry

View file

@ -1,4 +1,4 @@
FROM python:3.8-buster as python
FROM python:3.9-buster as python
RUN apt update \
&& apt install -y --no-install-recommends lintian rsync sudo docker.io \

View file

@ -8,7 +8,7 @@ url="https://gitlab.com/s3lph/spaceapi-server"
license=('MIT')
groups=()
depends=('python'
'python-jinja'
'python-yaml'
'python-bottle')
makedepends=('python-setuptools')
checkdepends=()
@ -16,8 +16,8 @@ optdepends=()
provides=()
conflicts=()
replaces=()
backup=('etc/spaceapi-server/config.json'
'etc/spaceapi-server/template.json'
backup=('etc/spaceapi-server/config.yaml'
'etc/spaceapi-server/template.yaml'
'etc/spaceapi-server/plugins')
install=$pkgname.install
changelog=$pkgname.changelog

View file

@ -1,7 +0,0 @@
{
"address": "::",
"port": 8080,
"template": "/etc/spaceapi-server/template.json",
"plugins_dir": "/etc/spaceapi-server/plugins",
"plugins": {}
}

View file

@ -0,0 +1,6 @@
---
address: "::"
port: 8080
template: /etc/spaceapi-server/template.json
plugins_dir: /etc/spaceapi-server/plugins
plugins: {}

View file

@ -1,19 +0,0 @@
{
"api": "0.13",
"space": "Example Space",
"logo": "https://example.org/logo.png",
"url": "https://example.org",
"location": {
"lat": 0.0,
"lon": 0.0
},
"state": {
"open": null
},
"contact": {
"email": "example@example.org"
},
"issue_report_channels": [
"email"
]
}

View file

@ -0,0 +1,14 @@
---
api: "0.13"
space: Example
logo: https://example.org/logo.png
url: https://example.org
location:
lat: 0.0
lon: 0.0
state:
open: null
contact:
email: example@example.org
issue_report_channels:
- email

View file

@ -2,7 +2,7 @@
Description=Lightweight SpaceAPI Endpoint Server
[Service]
ExecStart=/usr/bin/python -m spaceapi_server /etc/spaceapi-server/config.json
ExecStart=/usr/bin/python -m spaceapi_server /etc/spaceapi-server/config.yaml
ExecReload=/usr/bin/kill -HUP $MAINPID
User=spaceapi-server

View file

@ -1,3 +1,3 @@
/etc/spaceapi-server/config.json
/etc/spaceapi-server/template.json
/etc/spaceapi-server/config.yaml
/etc/spaceapi-server/template.yaml
/etc/spaceapi-server/plugins/example.py

View file

@ -1,7 +0,0 @@
{
"address": "::",
"port": 8080,
"template": "/etc/spaceapi-server/template.json",
"plugins_dir": "/etc/spaceapi-server/plugins",
"plugins": {}
}

View file

@ -0,0 +1,6 @@
---
address: "::"
port: 8080
template: /etc/spaceapi-server/template.json
plugins_dir: /etc/spaceapi-server/plugins
plugins: {}

View file

@ -1,19 +0,0 @@
{
"api": "0.13",
"space": "Example Space",
"logo": "https://example.org/logo.png",
"url": "https://example.org",
"location": {
"lat": 0.0,
"lon": 0.0
},
"state": {
"open": null
},
"contact": {
"email": "example@example.org"
},
"issue_report_channels": [
"email"
]
}

View file

@ -0,0 +1,14 @@
---
api: "0.13"
space: Example
logo: https://example.org/logo.png
url: https://example.org
location:
lat: 0.0
lon: 0.0
state:
open: null
contact:
email: example@example.org
issue_report_channels:
- email

View file

@ -2,7 +2,7 @@
Description=Lightweight SpaceAPI Endpoint Server
[Service]
ExecStart=/usr/bin/python3 -m spaceapi_server /etc/spaceapi-server/config.json
ExecStart=/usr/bin/python3 -m spaceapi_server /etc/spaceapi-server/config.yaml
ExecReload=/usr/bin/kill -HUP $MAINPID
User=spaceapi-server

View file

@ -1,13 +1,13 @@
FROM python:3.8-alpine
FROM python:3.9-alpine
ADD spaceapi_server/ /spaceapi_server/
ADD setup.py /setup.py
RUN python setup.py install && mkdir -p /config/plugins && chmod 0755 -R /config
ADD package/docker/config.json /config/config.json
ADD package/docker/template.json /config/template.json
ADD package/docker/config.yaml /config/config.yaml
ADD package/docker/template.yaml /config/template.yaml
VOLUME /config
EXPOSE 8000/tcp
USER 1000
ENTRYPOINT [ "/usr/local/bin/python3.8", "-m", "spaceapi_server", "/config/config.json" ]
ENTRYPOINT [ "/usr/local/bin/python3.9", "-m", "spaceapi_server", "/config/config.yaml" ]

View file

@ -1,8 +0,0 @@
{
"address": "0.0.0.0",
"port": 8000,
"server": "wsgiref",
"template": "/config/template.json",
"plugins_dir": "/config/plugins",
"plugins": {}
}

View file

@ -0,0 +1,6 @@
---
address: "::"
port: 8080
template: /etc/spaceapi-server/template.json
plugins_dir: /etc/spaceapi-server/plugins
plugins: {}

View file

@ -1,23 +0,0 @@
{
"api": "0.13",
"space": "example.space",
"logo": "http://example.space/logo.png",
"url": "http://example.space",
"location": {
"address": "example.space, 42 Example Street, 1337 Example Town",
"lon": 42.42,
"lat": 13.37
},
"contact": {
"email": "example@example.space"
},
"issue_report_channels": [
"email"
],
"state": {
"open": null
},
"projects": [
"http://example.space/projects"
]
}

View file

@ -0,0 +1,14 @@
---
api: "0.13"
space: Example
logo: https://example.org/logo.png
url: https://example.org
location:
lat: 0.0
lon: 0.0
state:
open: null
contact:
email: example@example.org
issue_report_channels:
- email

View file

@ -19,7 +19,7 @@ setup(
python_requires='>=3.6',
install_requires=[
'bottle',
'jinja2==2.10'
'PyYAML',
],
entry_points={
'console_scripts': [
@ -31,7 +31,7 @@ setup(
'Environment :: Web Environment',
'Intended Audience :: System Administrators',
'License :: OSI Approved :: MIT License',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.9',
'Topic :: Internet :: WWW/HTTP :: WSGI :: Application'
]
)

View file

@ -1,2 +1,2 @@
__version__ = '0.3.1'
__version__ = '0.4'

View file

@ -1,5 +1,5 @@
import json
import yaml
# The parsed config object
@ -8,13 +8,13 @@ __CONFIG = {}
def load(filename: str) -> None:
"""
Load a JSON-formatted configuration file.
Load a YAML-formatted configuration file.
:param filename: The config file to load.
"""
global __CONFIG
# Open and parse the JSON config file
# Open and parse the YAML config file
with open(filename, 'r') as conf:
__CONFIG = json.load(conf)
__CONFIG = yaml.safe_load(conf)
def get() -> dict:

View file

@ -1,38 +0,0 @@
import unittest
import tempfile
from spaceapi_server.config import load, get
from spaceapi_server.plugins import get_plugin_config
class ConfigTest(unittest.TestCase):
CONFIG = '''
{
"foo": "bar",
"baz": 42,
"plugins": {
"myplugin": {
"foo": "baz"
}
}
}
'''
def setUp(self) -> None:
_, self.temp = tempfile.mkstemp('.json', 'spaceapi_server_config_', text=True)
with open(self.temp, 'w') as f:
f.write(self.CONFIG)
def test_load(self):
pre = get()
pre_plugin = get_plugin_config('myplugin')
self.assertEqual({}, pre)
self.assertEqual({}, pre_plugin)
load(self.temp)
post = get()
post_plugin = get_plugin_config('myplugin')
self.assertEqual('bar', post['foo'])
self.assertEqual(42, post['baz'])
self.assertEqual('baz', post_plugin['foo'])

View file

@ -1,4 +1,6 @@
import yaml
from spaceapi_server import config, template
@ -7,22 +9,8 @@ def template_function(fn):
Register the decorated function as a callable template function.
:param fn: The function to register.
"""
# Make sure the Jinja2 environment is initialized
env = template._env_init()
# Add the function to the environment's globals
env.globals[fn.__name__] = fn
return fn
def template_filter(fn):
"""
Register the decorated function as a template filter.
:param fn: The function to register.
"""
# Make sure the Jinja2 environment is initialized
env = template._env_init()
# Add the function to the environment's filters
env.filters[fn.__name__] = fn
yaml.SafeLoader.add_constructor(f'!{fn.__name__}', template.plugin_constructor)
template.PluginInvocation.register_plugin(fn.__name__, fn)
return fn

View file

@ -1,6 +1,7 @@
import os
import sys
import json
import yaml
import signal
import importlib
@ -51,10 +52,10 @@ def load(*args, **kwargs):
m_spec.loader.exec_module(module)
# Get the template path
template_path = conf.get('template', 'template.json')
template_path = conf.get('template', 'template.yaml')
# Load and parse the JSON template
with open(template_path, 'r') as f:
__TEMPLATE = json.load(f)
__TEMPLATE = yaml.safe_load(f)
def init():

View file

@ -1,44 +1,31 @@
import json
import jinja2
class PluginInvocation():
_plugins = {}
def __init__(self, name, data):
super().__init__()
if name not in self._plugins:
raise KeyError(f'No such plugin function: {name}')
self._name = name
self._data = data
def __call__(self):
return self._plugins[self._name](**self._data)
@classmethod
def register_plugin(cls, name, fn):
cls._plugins[name] = fn
# The Jinja2 environment
__ENV = None
def _env_init(force: bool = False):
"""
Initialize the Jinja2 environment.
:param force: If true, force reload the environment.
"""
global __ENV
if __ENV is None or force:
# Use json.dumps as finalizer in order to preserve complex data structures
__ENV = jinja2.Environment(finalize=json.dumps)
return __ENV
def render(template: str):
"""
Render the given string as a Jinja2 template.
:param template: The template string to render.
"""
# Make sure the Jinja2 environment is initialized
env = _env_init()
# Create a Jinja2 template from the input string
t = env.from_string(template)
decoder = json.JSONDecoder()
# Render the template and turn the JSON dump back into complex data structures
# Only parse the first JSON object in the string, ignore the rest
obj, i = decoder.raw_decode(t.render())
return obj
def plugin_constructor(loader, node):
data = loader.construct_mapping(node)
return PluginInvocation(node.tag[1:], data)
def render_traverse(obj):
"""
Walk through a complex, JSON-serializable data structure, and pass
string objects through the Jinja2 templating engine.
Walk through a complex, JSON-serializable data structure, and
invoke plugins if for custom tags.
:param obj: The object to traverse.
"""
if isinstance(obj, list):
@ -47,9 +34,9 @@ def render_traverse(obj):
elif isinstance(obj, dict):
# dict -> recurse into the value of each (key, value)
return {k: render_traverse(v) for k, v in obj.items()}
elif isinstance(obj, str):
# str -> template
return render(obj)
elif isinstance(obj, PluginInvocation):
# PluginTag -> invoke the plugin with the stored arguments
return obj()
else:
# anything else -> return as-is
return obj

View file

@ -1,69 +0,0 @@
import unittest
from spaceapi_server.template import _env_init, render, render_traverse
from spaceapi_server.plugins import template_function, template_filter
class TemplateTest(unittest.TestCase):
@staticmethod
@template_function
def template_test_function(value):
return f'test_{value}'
@staticmethod
@template_filter
def template_test_filter(value, other):
return f'test_{other}_{value}'
@staticmethod
@template_function
def template_test_function_nocache():
TemplateTest.counter1 += 1
return TemplateTest.counter1
def setUp(self) -> None:
TemplateTest.counter1 = 0
def test_template(self):
env = _env_init()
self.assertEqual('"not a template"', env.from_string('not a template').render())
self.assertEqual('"a template"', env.from_string('{{ "a template" }}').render())
self.assertEqual('42', env.from_string('{{ 42 }}').render())
self.assertEqual('["foo"]', env.from_string('{{ [ "foo" ] }}').render())
self.assertEqual('{"foo": ["bar"]}', env.from_string('{{ { "foo": [ "bar" ] } }}').render())
def test_render(self):
self.assertEqual('not a template', render('not a template'))
self.assertEqual('a template', render('{{ "a template" }}'))
self.assertEqual(42, render('{{ 42 }}'))
self.assertEqual(['foo'], render('{{ [ "foo" ] }}'))
self.assertEqual({'foo': ['bar']}, render('{{ { "foo": [ "bar" ] } }}'))
self.assertEqual('foo', render('foo{{ "bar" }}'))
def test_render_traverse(self):
template = {
'foo': 42,
'bar': [1, 2, 3],
'notemplate': 'foo',
'builtin': '{{ [ 1337, 42 ] | first }}',
'test_functions': {
'test_function': '{{ template_test_function("foo") }}',
'test_filter': '{{ "bar" | template_test_filter("other") }}'
}
}
rendered = render_traverse(template)
self.assertEqual(42, rendered['foo'])
self.assertEqual([1, 2, 3], rendered['bar'])
self.assertEqual('foo', rendered['notemplate'])
self.assertEqual(1337, rendered['builtin'])
self.assertEqual('test_foo', rendered['test_functions']['test_function'])
self.assertEqual('test_other_bar', rendered['test_functions']['test_filter'])
def test_no_cache(self):
template = ['{{ template_test_function_nocache() }}']
r1 = render_traverse(template)
r2 = render_traverse(template)
self.assertEqual(1, r1[0])
self.assertEqual(2, r2[0])