This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
[](https://gitter.im/Microsoft/qlib?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
> **Upgrading configuration-driven workflows?** See the
> [configuration migration guide](docs/start/config_migration.rst) for the
> unreleased source changes: per-component `trusted: true` for local `.py`
> imports, restricted expressions, and explicit extension mappings. Package
> imports remain available. Match this guide to your checkout; PR source,
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -78,6 +78,93 @@ Users can use ``Data Handler`` to build formulaic alphas `MACD` in qlib:
SZ300251 -0.001026 0.021739
SZ300315 -0.007559 0.012455
.. _expression_syntax:
Expression syntax and migration
===============================
Feature expressions are a restricted language, not general Python.
``Qlib`` interprets their syntax and calls registered operators rather than evaluating arbitrary Python code.
Standard Alpha158/Alpha360 feature definitions and registered custom operators remain supported.
For an upgrade checklist covering expressions, file imports, and extension
registries, see :ref:`config_migration`.
Supported expressions include:
.. code-block:: text
$close
Ref($close, 1) / $close - 1
Mean($close, 2 + 3)
$close / (1 + 0.01)
If(Gt($close, $open), $close, $open)
($close > $open) & ($volume > 0)
Ref(*[$close, 1])
Mean($close, **{"N": 2 + 3})
Ref($close, [1, 5, 10][0])
Ref(*[$close, 1, 99][:2])
Ref($close, 5 if (1 < 2 and not False) else 10)
You can use feature references (``$field`` and point-in-time ``$$field``), registered operator calls, arithmetic on expressions, and single comparisons.
Operator arguments can include literals, lists, tuples, dictionaries, and named arguments where the operator accepts them.
Numeric parameter arithmetic is supported, including ``+``, ``-``, ``*``, ``/``, ``//``, ``%``, and ``**``.
Constant arithmetic requires real numbers; integers are limited to 4096 bits and the absolute value of a constant exponent is limited to 4096.
String/list repetition (such as ``'x' * n`` or ``[1] * n``) and complex-valued constant arithmetic are rejected.
The final result must be a Qlib ``Expression`` object, not a standalone constant.
Safe parameter syntax
---------------------
* ``*`` expands literal lists or tuples into positional arguments or another list/tuple.
``**`` expands literal dictionaries into keyword arguments or another dictionary.
Arbitrary iterables and mapping objects are not accepted.
* Dictionary keys must be literal scalars (numbers, booleans, strings, bytes, or ``None``), not expression objects or tuples.
Keyword argument names must be strings, and duplicate keyword arguments are rejected.
Dictionary literals retain Python's last-value-wins behavior when keys are repeated.
* Literal lists, tuples, dictionaries, strings, and bytes support indexing.
Sequences also support slices with integer bounds and steps.
These containers can hold expressions, but expression objects themselves cannot be indexed: ``[$close][0]`` is supported, while ``$close[0]`` is not.
* Each list, tuple, or dictionary is limited to 4096 items, including items introduced by expansion.
A call is limited to 4096 positional and keyword arguments in total.
* Literal scalar ``<``, ``<=``, ``>``, ``>=``, ``==``, and ``!=`` comparisons, including chained comparisons, are supported.
Their truth-value tests must be built from literal scalars, optionally using constant arithmetic or literal-container lookups; they cannot contain Qlib operator calls.
For example, ``$close if True else $open`` is supported, but ``$close if $volume > 0 else $open`` is not.
All branches are checked for supported syntax and registered operator names before any operators are constructed, even if a branch will not be selected.
String arguments are preserved literally; text such as ``"$close"`` inside a string is not rewritten into a feature reference.
Dataset and disk-cache field normalization also preserve spaces inside literals and the token boundaries needed by scalar conditions.
Expression logic and unsupported Python syntax
----------------------------------------------
For elementwise logic involving Qlib expressions, use ``&`` and ``|`` with parenthesized comparisons, or ``If``.
Use ``If($volume > 0, $close, $open)`` instead of a Python conditional whose test is an expression.
Python ``and``, ``or``, ``not``, and chained comparisons involving expression objects are rejected rather than silently changing their meaning.
Attribute access, lambdas, comprehensions, and arbitrary function calls inside expression strings remain unsupported.
Move such logic into trusted Python code or a registered custom operator.
These restrictions do not affect normal Python code used to generate expression strings:
.. code-block:: Python
fields = [f"Mean($close, {window})" for window in [5, 10, 20]]
Unsupported syntax and unknown operator names raise ``qlib.data.expression_parser.ExpressionSyntaxError``, a subclass of ``ValueError``.
Operator-specific argument validation still applies.
Register custom operators before use, for example through ``qlib.init(custom_ops=[...])``; merely making a Python function importable does not make it an expression operator.
See ``tests/test_register_ops.py`` for an example.
For file-based custom operators, put ``trusted: true`` in each operator's
configuration alongside ``class`` and ``module_path``; see the working example
in :ref:`config_migration`. This permits that file import only and never
relaxes the expression grammar.
.. note::
Custom operator code must be trusted. The expression language restricts syntax, but does not sandbox registered operators or impose general resource limits on expression evaluation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Qlib supports custom models. If users are interested in customizing their own models and integrating the models into ``Qlib``, please refer to `Custom Model Integration <../start/integration.html>`_.
See :ref:`config_migration` when upgrading file-based models: each ``.py``
component needs its own top-level boolean ``trusted`` declaration, while
package imports remain available. The guide also covers trusted older
file-model pickles and registering custom built-in TRA backbones through
``qlib.contrib.model.pytorch_tra.MODEL_TYPES`` before construction, including
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -189,6 +189,9 @@ The meaning of each field is as follows:
The value of `region` should be aligned with the data stored in `provider_uri`.
File-import consent belongs to each component's configuration, not the
``qlib_init`` section. See :ref:`config_migration`.
Task Section
------------
Expand Down
Expand Up
@@ -222,14 +225,64 @@ The meaning of each field is as follows:
Type: str. The name for the model class.
- `module_path`
Type: str. The path for the model in qlib.
Type: str. An importable Python module name, or a ``.py`` file path.
- `trusted`
Type: bool, optional (default: ``False``). Set to ``True`` only to authorize
importing this component's reviewed ``.py`` file. Package imports and class
objects do not require this field. It is separate from constructor ``kwargs``.
- `kwargs`
The keywords arguments for the model. Please refer to the specific model implementation for more information: `models <https://github.com/microsoft/qlib/blob/main/qlib/contrib/model>`_.
.. note::
``Qlib`` provides a util named: ``init_instance_by_config`` to initialize any class inside ``Qlib`` with the configuration includes the fields: `class`, `module_path` and `kwargs`.
``Qlib`` provides ``init_instance_by_config`` to initialize a class from
``class``, ``module_path``, and ``kwargs``. For file modules, add ``trusted``
alongside those fields, not as a keyword argument to the factory.
.. _config_file_modules:
Custom modules and migration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
For the upgrade checklist and complete Python/custom-operator examples, see
:ref:`config_migration`. Package-based configurations such as
``module_path: qlib.contrib.model.gbdt`` or ``module_path: my_package.model``
remain available without file-import consent. To load a reviewed local model,
declare consent in that model's configuration:
.. code-block:: YAML
qlib_init:
provider_uri: "~/.qlib/qlib_data/cn_data"
region: cn
task:
model:
class: MyModel
module_path: custom_modules/model.py
trusted: true
kwargs: {}
# Keep the existing dataset and record sections here.
Every nested file-based dataset, handler, or custom operator needs its own
top-level ``trusted: true``; trust is not inherited. Only boolean values are
accepted. Without consent, file imports raise ``PermissionError``.
Relative paths use the process's current working directory, not the YAML file.
Keep these fields when saving configurations or passing them to workers.
``qrun`` forwards the ``qlib_init`` section automatically.
If you load YAML in a Python script, forward the complete section to preserve
initialization options (this does not grant file-import permission):
.. code-block:: Python
qlib.init(**config["qlib_init"])
Only run configurations and source code you trust; embedded consent does not
make untrusted YAML safe. See :ref:`config_migration` for the distinction between
import consent and constructor/artifact permissions, and recovery of trusted
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
@@ -168,6 +168,12 @@ Also, there are some optional fields. The meaning of each field is as follows:
- `tuner_module_path`
The module path, str type, absolute url is also supported, indicates the path of the implementation of tuner. The default value is `qlib.contrib.tuner.tuner`
- `trusted`
Strict bool, default ``False``. Set ``experiment.trusted: true`` only after
reviewing a file-based ``tuner_module_path``. This permits only the tuner's
module import, not other components or the workflow. Package imports remain
available without it. See :ref:`config_migration` for details.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix(security): constrain config-driven code execution #2340
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Are you sure you want to change the base?
Uh oh!
There was an error while loading. Please reload this page.
fix(security): constrain config-driven code execution #2340
Filter by extension
Viewed files
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
There are no files selected for viewing
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.