Virtual products

A virtual product is not data downloaded from a web service, it is a local function dynamically computing a product.

You register your function with an inventory path, and then use it as any other Speasy product: request it with speasy.get_data(), or call it by its name.

The registration can be done either directly, or with the decorator speasy.virtual_products.register_virtual_product().

For arithmetic on existing products, such as 10 * B / N, you don’t need to write a function: see Virtual expressions.

Writing the function

The user function to register takes a time range as argument (start_time, stop_time) and returns a SpeasyVariable, or None when there is no data. It receives the time range exactly as it was given to speasy.get_data():

>>> import pandas as pd
>>> import speasy as spz
>>> def hourly_ramp(start_time, stop_time):
...     index = pd.date_range(start_time, stop_time, freq=pd.Timedelta(hours=1), inclusive="left")
...     return spz.SpeasyVariable.from_dataframe(pd.DataFrame({"ramp": range(len(index))}, index=index))

Registering with a decorator

The registering path must start with virtual/, followed by the inventory path of your choice (e.g. virtual/demo/ramp).

After the decorator, the function name refers to the product, and the function is registered in the inventory tree:

>>> from speasy.virtual_products import register_virtual_product
>>> @register_virtual_product("virtual/demo/ramp")
... def ramp(start_time, stop_time):
...     return hourly_ramp(start_time, stop_time)
>>> spz.inventories.tree.virtual.demo.ramp is ramp
True

Registering directly without a decorator

Pass the function as second argument; the product is returned:

>>> slope = register_virtual_product("virtual/demo/slope", hourly_ramp)
>>> spz.inventories.tree.virtual.demo.slope is slope
True

Describing the product

Both forms accept meta= and labels=.

meta= — a dictionary that does two things. Its keys become attributes of the product in the inventory tree, and they are also injected into the SpeasyVariable.meta returned by the callback, with keys uppercased to follow the ISTP convention (e.g. "units" becomes "UNITS"). Values from the inventory always take precedence over any conflicting metadata returned by the callback:

>>> level = register_virtual_product("virtual/demo/level", hourly_ramp, meta={"units": "nT"})
>>> spz.inventories.tree.virtual.demo.level.units
'nT'
>>> spz.get_data("virtual/demo/level", "2016-10-10", "2016-10-11").unit
'nT'

labels= — a list of strings that overrides the column names of the SpeasyVariable returned by the callback:

>>> labeled = register_virtual_product("virtual/demo/labeled", hourly_ramp, labels=["value"])
>>> spz.get_data("virtual/demo/labeled", "2016-10-10", "2016-10-11").columns
['value']

Using a virtual product

By path, by the decorated name, through the inventory tree, or by a direct call:

>>> spz.get_data("virtual/demo/ramp", "2016-10-10", "2016-10-11").shape
(24, 1)
>>> spz.get_data(ramp, "2016-10-10", "2016-10-11").shape
(24, 1)
>>> spz.get_data(spz.inventories.tree.virtual.demo.ramp, "2016-10-10", "2016-10-11").shape
(24, 1)
>>> ramp("2016-10-10", "2016-10-11").shape
(24, 1)

Distributing virtual products as a plugin

A package can ship virtual products that are registered as soon as Speasy is imported. Declare a zero-argument function as a speasy.virtual_products entry point in the package’s pyproject.toml:

[project.entry-points."speasy.virtual_products"]
my_products = "my_pkg:register"

The function only has to import the module holding the decorated functions:

# my_pkg/__init__.py
def register():
    from . import products  # noqa: F401
# my_pkg/products.py
from speasy.virtual_products import register_virtual_product

@register_virtual_product("virtual/my_pkg/beta")
def beta(start_time, stop_time):
    ...

Once the package is installed, spz.inventories.tree.virtual.my_pkg.beta exists after import speasy.

  • Plugins are loaded while import speasy is running: import register_virtual_product from speasy.virtual_products, not from the top-level speasy package. The function bodies only run when the product is requested, so they can use all of Speasy.

  • A plugin that fails to load is logged as a warning and skipped; import speasy still completes.

  • A plugin can be turned off with the core.disabled_plugins config entry, by its name (my_products) or by its group-qualified name (speasy.virtual_products.my_products).

Loading virtual products from a directory

Without packaging anything, every .py file in the user virtual products directory (~/.local/share/speasy/virtual_products/ on Linux; speasy.virtual_products.registry.user_vp_dir() gives it on your system) is executed by import speasy:

# ~/.local/share/speasy/virtual_products/my_products.py
from speasy.virtual_products import register_virtual_product

@register_virtual_product("virtual/mine/beta")
def beta(start_time, stop_time):
    ...

More directories can be scanned with the core.user_vp_extra_dirs config entry, a comma-separated list (see Configuration).

  • Files are loaded after the plugins: a file registering the same path as an installed plugin replaces it, with a warning.

  • A file that fails to load is logged as a warning, with a traceback pointing to the file, and skipped; the other files and import speasy are unaffected.

Good to know

  • Registering again at the same path replaces the previous product and emits a warning, which is convenient when iterating in a notebook.

  • Requesting a path that was never registered raises UnknownVirtualProduct, a ValueError. Its message suggests the closest registered paths, if any.

  • If the function returns anything other than a SpeasyVariable or None, calling the product or speasy.get_data() raises VirtualProductTypeError, a TypeError naming the function.

  • A virtual product may request other virtual products, but never itself, not even for another time range and not even through other products. Such a request raises VirtualProductCycleError, a RecursionError, whose message shows the chain of products. To reuse a product’s computation, call its function directly.

  • This check follows the requests made in one thread. If your function starts its own threads and requests virtual products from them, a product requesting itself through them is not detected.