FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

fix(security): constrain config-driven code execution by XianBW · Pull Request #2340 · microsoft/qlib · GitHub

/ qlib Public
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
babb240
security: constrain config-driven code execution
XianBW Sep 2, 2026
e8c6856
fix: address security test and lint failures
XianBW Sep 2, 2026
d32d9c6
ci: restore dependency and title lint compatibility
XianBW Sep 2, 2026
fdbc939
ci: handle current MLflow and Windows OSQP changes
XianBW Sep 2, 2026
066586d
ci: use lxml release with cross-platform wheels
XianBW Sep 2, 2026
8f9defa
fix: defer optional CVXPY solver imports
XianBW Sep 2, 2026
a6ceab2
style: document intentional lazy import
XianBW Sep 2, 2026
49c1f9f
ci: skip broken released OSQP on Windows Python 3.8
XianBW Sep 2, 2026
06f9d1f
test: make MLflow client timing check robust
XianBW Sep 2, 2026
0e17df9
ci: retry nondeterministic benchmark smoke test
XianBW Sep 2, 2026
5c07604
fix: cap filelock before fork safety regression
XianBW Sep 2, 2026
ee6b451
fix: use filelock release compatible with multiprocessing
XianBW Sep 2, 2026
21862cb
Merge main to retain current CI and dependency fixes
XianBW Sep 16, 2026
1d948fd
fix(security): restrict operator lookup and validate trusted roots
XianBW Sep 16, 2026
078d9d4
fix(security): preserve numeric expressions and legacy model imports
XianBW Sep 16, 2026
98eb04a
fix: preserve TRA initialization and document config migration
XianBW Sep 17, 2026
585dbb8
fix: preserve safe expression syntax and cache semantics
XianBW Sep 21, 2026
716f4c3
fix: use explicit scoped trust for file modules
XianBW Sep 22, 2026
6f9af97
fix: satisfy CI lint without weakening trust checks
XianBW Sep 22, 2026
00b33fe
test: handle escaped file paths in module refusal errors
XianBW Sep 22, 2026
3d97079
fix: preserve expression parser error logging
XianBW Sep 22, 2026
27eb02a
test: isolate expression logger capture
XianBW Sep 22, 2026
File filter

Filter by extension

Filter by extension .md  (4) .py  (19) .rst  (11) .yaml  (8) All 4 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
19 changes: 19 additions & 0 deletions CHANGELOG.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
# Changelog

## Unreleased: configuration-driven execution

- Local `.py` imports require explicit boolean consent: top-level `trusted: true`
on each component configuration, or `trusted=True` on a direct
`get_module_by_module_path` call. Package imports and class objects are unchanged.
- Directory-root authorization from earlier PR revisions was removed. Consent is
neither global nor inherited, and it does not authorize artifact loading.
- Feature expressions use a restricted AST interpreter; custom built-in TRA
backbones and model-performance graph names use explicit mappings.

See the [configuration migration guide](docs/start/config_migration.rst) for the
upgrade checklist, Python/YAML examples, extension registration, and trusted
older file-model pickle recovery. These are PR #2340 source changes, not a
statement that `main` or a tagged/PyPI release contains them. PR #2339's artifact
permissions are a separate unreleased change.

The full release history is maintained in [CHANGES.rst](CHANGES.rst).
28 changes: 28 additions & 0 deletions CHANGES.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,34 @@ Changelog
=========
Here you can see the full list of changes between each QLib release.

Unreleased
----------

Configuration-driven execution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- **Breaking change:** loading a local ``.py`` module through configuration now
requires top-level boolean ``trusted: true`` on each file-based component;
direct ``get_module_by_module_path`` calls require ``trusted=True``.
Package imports and class objects are unchanged. Directory-root authorization
from earlier PR revisions was removed; consent is not inherited or process-wide.
See :ref:`config_migration` for the upgrade checklist, constructor/artifact
trust distinction, and recovery of trusted older file-model pickles.
- Feature expressions use a restricted AST interpreter instead of Python ``eval``.
Registered operators and standard Alpha158/Alpha360 definitions remain supported,
as do bounded literal containers, argument expansion, indexing/slicing, and scalar
conditions. Arbitrary Python execution and Python truth-value tests on expression
objects are not supported. See :ref:`expression_syntax` for supported syntax,
limits, and migration examples.
- Built-in TRA model names and model-performance graph names use explicit mappings.
Custom extensions must update the corresponding mapping rather than relying on
dynamically evaluated module globals; :ref:`config_migration` includes working
registration examples. Configuration and extension code must still be trusted;
these changes do not create a sandbox.
- These are unreleased PR #2340 source changes. PR source, ``main``, and tagged
releases can differ. PR #2339's separate artifact-consent changes are not
implied to be merged or released.

Version 0.1.0
-------------
This is the initial release of QLib library.
Expand Down
7 changes: 7 additions & 0 deletions README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,13 @@
[![License](https://img.shields.io/pypi/l/pyqlib)](LICENSE)
[![Join the chat at https://gitter.im/Microsoft/qlib](https://badges.gitter.im/Microsoft/qlib.svg)](https://gitter.im/Microsoft/qlib?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)

> **Upgrading configuration-driven workflows?** See the
> [configuration migration guide](docs/start/config_migration.rst) for the
> unreleased source changes: per-component `trusted: true` for local `.py`
> imports, restricted expressions, and explicit extension mappings. Package
> imports remain available. Match this guide to your checkout; PR source,
> `main`, and tagged/PyPI releases may differ.

## :newspaper: **What's NEW!**   :sparkling_heart:

Recent released features
Expand Down
87 changes: 87 additions & 0 deletions docs/advanced/alpha.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -78,6 +78,93 @@ Users can use ``Data Handler`` to build formulaic alphas `MACD` in qlib:
SZ300251 -0.001026 0.021739
SZ300315 -0.007559 0.012455

.. _expression_syntax:

Expression syntax and migration
===============================

Feature expressions are a restricted language, not general Python.
``Qlib`` interprets their syntax and calls registered operators rather than evaluating arbitrary Python code.
Standard Alpha158/Alpha360 feature definitions and registered custom operators remain supported.
For an upgrade checklist covering expressions, file imports, and extension
registries, see :ref:`config_migration`.

Supported expressions include:

.. code-block:: text

$close
Ref($close, 1) / $close - 1
Mean($close, 2 + 3)
$close / (1 + 0.01)
If(Gt($close, $open), $close, $open)
($close > $open) & ($volume > 0)
Ref(*[$close, 1])
Mean($close, **{"N": 2 + 3})
Ref($close, [1, 5, 10][0])
Ref(*[$close, 1, 99][:2])
Ref($close, 5 if (1 < 2 and not False) else 10)

You can use feature references (``$field`` and point-in-time ``$$field``), registered operator calls, arithmetic on expressions, and single comparisons.
Operator arguments can include literals, lists, tuples, dictionaries, and named arguments where the operator accepts them.
Numeric parameter arithmetic is supported, including ``+``, ``-``, ``*``, ``/``, ``//``, ``%``, and ``**``.
Constant arithmetic requires real numbers; integers are limited to 4096 bits and the absolute value of a constant exponent is limited to 4096.
String/list repetition (such as ``'x' * n`` or ``[1] * n``) and complex-valued constant arithmetic are rejected.
The final result must be a Qlib ``Expression`` object, not a standalone constant.

Safe parameter syntax
---------------------

* ``*`` expands literal lists or tuples into positional arguments or another list/tuple.
``**`` expands literal dictionaries into keyword arguments or another dictionary.
Arbitrary iterables and mapping objects are not accepted.
* Dictionary keys must be literal scalars (numbers, booleans, strings, bytes, or ``None``), not expression objects or tuples.
Keyword argument names must be strings, and duplicate keyword arguments are rejected.
Dictionary literals retain Python's last-value-wins behavior when keys are repeated.
* Literal lists, tuples, dictionaries, strings, and bytes support indexing.
Sequences also support slices with integer bounds and steps.
These containers can hold expressions, but expression objects themselves cannot be indexed: ``[$close][0]`` is supported, while ``$close[0]`` is not.
* Each list, tuple, or dictionary is limited to 4096 items, including items introduced by expansion.
A call is limited to 4096 positional and keyword arguments in total.
* Literal scalar ``<``, ``<=``, ``>``, ``>=``, ``==``, and ``!=`` comparisons, including chained comparisons, are supported.
Scalar ``and``, ``or``, ``not``, and conditional expressions preserve Python's short-circuit behavior.
Their truth-value tests must be built from literal scalars, optionally using constant arithmetic or literal-container lookups; they cannot contain Qlib operator calls.
For example, ``$close if True else $open`` is supported, but ``$close if $volume > 0 else $open`` is not.

All branches are checked for supported syntax and registered operator names before any operators are constructed, even if a branch will not be selected.
String arguments are preserved literally; text such as ``"$close"`` inside a string is not rewritten into a feature reference.
Dataset and disk-cache field normalization also preserve spaces inside literals and the token boundaries needed by scalar conditions.

Expression logic and unsupported Python syntax
----------------------------------------------

For elementwise logic involving Qlib expressions, use ``&`` and ``|`` with parenthesized comparisons, or ``If``.
Write ``($close > 0) & ($close < 10)`` instead of ``0 < $close < 10``.
Use ``If($volume > 0, $close, $open)`` instead of a Python conditional whose test is an expression.
Python ``and``, ``or``, ``not``, and chained comparisons involving expression objects are rejected rather than silently changing their meaning.

Attribute access, lambdas, comprehensions, and arbitrary function calls inside expression strings remain unsupported.
Move such logic into trusted Python code or a registered custom operator.
These restrictions do not affect normal Python code used to generate expression strings:

.. code-block:: Python

fields = [f"Mean($close, {window})" for window in [5, 10, 20]]

Unsupported syntax and unknown operator names raise ``qlib.data.expression_parser.ExpressionSyntaxError``, a subclass of ``ValueError``.
Operator-specific argument validation still applies.

Register custom operators before use, for example through ``qlib.init(custom_ops=[...])``; merely making a Python function importable does not make it an expression operator.
See ``tests/test_register_ops.py`` for an example.
For file-based custom operators, put ``trusted: true`` in each operator's
configuration alongside ``class`` and ``module_path``; see the working example
in :ref:`config_migration`. This permits that file import only and never
relaxes the expression grammar.

.. note::

Custom operator code must be trusted. The expression language restricts syntax, but does not sandbox registered operators or impose general resource limits on expression evaluation.

Reference
=========

Expand Down
7 changes: 7 additions & 0 deletions docs/component/model.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,13 @@ Custom Model

Qlib supports custom models. If users are interested in customizing their own models and integrating the models into ``Qlib``, please refer to `Custom Model Integration <../start/integration.html>`_.

See :ref:`config_migration` when upgrading file-based models: each ``.py``
component needs its own top-level boolean ``trusted`` declaration, while
package imports remain available. The guide also covers trusted older
file-model pickles and registering custom built-in TRA backbones through
``qlib.contrib.model.pytorch_tra.MODEL_TYPES`` before construction, including
in workers.


API
===
Expand Down
8 changes: 8 additions & 0 deletions docs/component/report.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,14 @@ Introduction
All of the accumulated profit metrics(e.g. return, max drawdown) in Qlib are calculated by summation.
This avoids the metrics or the plots being skewed exponentially over time.

.. note::

Model-performance graph names use the explicit ``GRAPH_FUNCTIONS`` mapping.
Built-in names are unchanged; custom names must be registered before use,
including in workers. See :ref:`config_migration` for a working extension
example and the upgrade checklist. File-import ``trusted`` does not bypass
this mapping.

Graphical Reports
=================

Expand Down
57 changes: 55 additions & 2 deletions docs/component/workflow.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -189,6 +189,9 @@ The meaning of each field is as follows:

The value of `region` should be aligned with the data stored in `provider_uri`.

File-import consent belongs to each component's configuration, not the
``qlib_init`` section. See :ref:`config_migration`.


Task Section
------------
Expand Down Expand Up @@ -222,14 +225,64 @@ The meaning of each field is as follows:
Type: str. The name for the model class.

- `module_path`
Type: str. The path for the model in qlib.
Type: str. An importable Python module name, or a ``.py`` file path.

- `trusted`
Type: bool, optional (default: ``False``). Set to ``True`` only to authorize
importing this component's reviewed ``.py`` file. Package imports and class
objects do not require this field. It is separate from constructor ``kwargs``.

- `kwargs`
The keywords arguments for the model. Please refer to the specific model implementation for more information: `models <https://github.com/microsoft/qlib/blob/main/qlib/contrib/model>`_.

.. note::

``Qlib`` provides a util named: ``init_instance_by_config`` to initialize any class inside ``Qlib`` with the configuration includes the fields: `class`, `module_path` and `kwargs`.
``Qlib`` provides ``init_instance_by_config`` to initialize a class from
``class``, ``module_path``, and ``kwargs``. For file modules, add ``trusted``
alongside those fields, not as a keyword argument to the factory.

.. _config_file_modules:

Custom modules and migration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

For the upgrade checklist and complete Python/custom-operator examples, see
:ref:`config_migration`. Package-based configurations such as
``module_path: qlib.contrib.model.gbdt`` or ``module_path: my_package.model``
remain available without file-import consent. To load a reviewed local model,
declare consent in that model's configuration:

.. code-block:: YAML

qlib_init:
provider_uri: "~/.qlib/qlib_data/cn_data"
region: cn
task:
model:
class: MyModel
module_path: custom_modules/model.py
trusted: true
kwargs: {}
# Keep the existing dataset and record sections here.

Every nested file-based dataset, handler, or custom operator needs its own
top-level ``trusted: true``; trust is not inherited. Only boolean values are
accepted. Without consent, file imports raise ``PermissionError``.
Relative paths use the process's current working directory, not the YAML file.
Keep these fields when saving configurations or passing them to workers.

``qrun`` forwards the ``qlib_init`` section automatically.
If you load YAML in a Python script, forward the complete section to preserve
initialization options (this does not grant file-import permission):

.. code-block:: Python

qlib.init(**config["qlib_init"])

Only run configurations and source code you trust; embedded consent does not
make untrusted YAML safe. See :ref:`config_migration` for the distinction between
import consent and constructor/artifact permissions, and recovery of trusted
older file-model pickles.

Dataset Section
~~~~~~~~~~~~~~~
Expand Down
6 changes: 6 additions & 0 deletions docs/hidden/tuner.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,12 @@ Also, there are some optional fields. The meaning of each field is as follows:
- `tuner_module_path`
The module path, str type, absolute url is also supported, indicates the path of the implementation of tuner. The default value is `qlib.contrib.tuner.tuner`

- `trusted`
Strict bool, default ``False``. Set ``experiment.trusted: true`` only after
reviewing a file-based ``tuner_module_path``. This permits only the tuner's
module import, not other components or the workflow. Package imports remain
available without it. See :ref:`config_migration` for details.

About the optimization criteria
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
1 change: 1 addition & 0 deletions docs/index.rst
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ Document Structure
:caption: FIRST STEPS:

Installation <start/installation.rst>
Configuration Migration <start/config_migration.rst>
Initialization <start/initialization.rst>
Data Retrieval <start/getdata.rst>
Custom Model Integration <start/integration.rst>
Expand Down
Loading
Loading

Back | FazBrowse Home | New Git URL