FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

gh-107265: Remove all ENTER_EXECUTOR when execute _Py_Instrument by corona10 · Pull Request #108539 · python/cpython · GitHub

/ cpython Public

gh-107265: Remove all ENTER_EXECUTOR when execute _Py_Instrument #108539

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
corona10 merged 17 commits into from
Sep 7, 2023
Conversation 29 Commits 17 Checks 0 Files changed
Merged

gh-107265: Remove all ENTER_EXECUTOR when execute _Py_Instrument #108539

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
23e1f48
Revert "gh-107265: Fix initialize/remove_tools for ENTER_EXECUTOR cas…
corona10 Aug 27, 2023
8820df8
gh-107265: Remove all ENTER_EXECUTOR when execute _Py_Instrument
corona10 Aug 27, 2023
e015f14
nit
corona10 Aug 27, 2023
7612b8e
Add assert
corona10 Aug 27, 2023
7d38c77
nit
corona10 Aug 27, 2023
a01354d
Address code review
corona10 Sep 1, 2023
9a631d7
nit
corona10 Sep 1, 2023
7e81bf0
Add test
corona10 Sep 1, 2023
162a870
Merge remote-tracking branch 'upstream/main' into gh-107265-mark
corona10 Sep 1, 2023
05773f9
Add test
corona10 Sep 1, 2023
9ddbfe7
Add space between +=
corona10 Sep 1, 2023
1199dfe
Return -1 if there is ENTER_EXECUTOR
corona10 Sep 1, 2023
1232432
Update
corona10 Sep 1, 2023
e051be1
Merge remote-tracking branch 'upstream/main' into gh-107265-mark
corona10 Sep 6, 2023
4a0be4d
fix
corona10 Sep 6, 2023
bc251b5
Fix
corona10 Sep 6, 2023
83d8829
Address Guido's review
corona10 Sep 6, 2023
File filter

Filter by extension

Filter by extension .c  (2) .h  (1) .py  (1) All 3 file types selected
Viewed files
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Unified
Split
Hide whitespace
Diff view
Unified
Split
Hide whitespace
3 changes: 3 additions & 0 deletions Include/internal/pycore_code.h
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,9 @@ extern void _PyLineTable_InitAddressRange(
extern int _PyLineTable_NextAddressRange(PyCodeAddressRange *range);
extern int _PyLineTable_PreviousAddressRange(PyCodeAddressRange *range);

/** API for executors */
extern void _PyCode_Clear_Executors(PyCodeObject *code);

#define ENABLE_SPECIALIZATION 1

/* Specialization functions */
Expand Down
28 changes: 28 additions & 0 deletions Lib/test/test_monitoring.py
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -1718,3 +1718,31 @@ def make_foo_optimized_then_set_event():
make_foo_optimized_then_set_event()
finally:
sys.monitoring.set_events(TEST_TOOL, 0)


class TestOptimizer(MonitoringTestBase, unittest.TestCase):

corona10 Sep 1, 2023 •
edited
Loading

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

@markshannon cc @gvanrossum

I added a test, but not sure what you wanted :)
Without this PR, Python/instrumentation.c#L1589 will not be able to pass the assert through this test.

gvanrossum Sep 6, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

What is this test supposed to do? When I copy this test into the main branch, it passes (in debug mode, so failing asserts would cause crashes), so I don't think this proves your fix works.

By adding some dis.dis(test_func, adaptive=True) calls to the test I think I see the difference -- in main, an ENTER_EXECUTOR opcode remains in the code object, whereas with this PR, the ENTER_EXECUTOR opcode appears after the function is called, but disappears again after the second set_local_events call.

You could check for that ENTER_EXECUTOR (there are examples in test_capi/test_misc.py) but it would be nice if you had example code that actually crashed or triggered an assert (in debug mode) in main without your fix.

corona10 Sep 6, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

Hmm,, interesting, It looks like I watched the Hallucination when I wrote the test in the first place :(
I will soon update the PR.

corona10 Sep 6, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

See: #108539 (comment)

def setUp(self):
import _testinternalcapi
self.old_opt = _testinternalcapi.get_optimizer()
opt = _testinternalcapi.get_counter_optimizer()
_testinternalcapi.set_optimizer(opt)
super(TestOptimizer, self).setUp()

def tearDown(self):
import _testinternalcapi
super(TestOptimizer, self).tearDown()
_testinternalcapi.set_optimizer(self.old_opt)

def test_for_loop(self):

gvanrossum Sep 1, 2023

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Choose a reason Spam Abuse Off Topic Outdated Duplicate Resolved Low Quality

I'd like @markshannon to review this part at least.
def test_func(x):
i = 0
while i < x:
i += 1

code = test_func.__code__
sys.monitoring.set_local_events(TEST_TOOL, code, E.PY_START)
self.assertEqual(sys.monitoring.get_local_events(TEST_TOOL, code), E.PY_START)
test_func(1000)
sys.monitoring.set_local_events(TEST_TOOL, code, 0)
self.assertEqual(sys.monitoring.get_local_events(TEST_TOOL, code), 0)
17 changes: 17 additions & 0 deletions Objects/codeobject.c
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -1479,6 +1479,23 @@ clear_executors(PyCodeObject *co)
co->co_executors = NULL;
}

void
_PyCode_Clear_Executors(PyCodeObject *code) {
int code_len = (int)Py_SIZE(code);
for (int i = 0; i < code_len; i += _PyInstruction_GetLength(code, i)) {
_Py_CODEUNIT *instr = &_PyCode_CODE(code)[i];
uint8_t opcode = instr->op.code;
uint8_t oparg = instr->op.arg;
if (opcode == ENTER_EXECUTOR) {
_PyExecutorObject *exec = code->co_executors->executors[oparg];
assert(exec->vm_data.opcode != ENTER_EXECUTOR);
instr->op.code = exec->vm_data.opcode;
instr->op.arg = exec->vm_data.oparg;
}
}
clear_executors(code);
Comment thread
gvanrossum marked this conversation as resolved.
}

static void
deopt_code(PyCodeObject *code, _Py_CODEUNIT *instructions)
{
Expand Down
43 changes: 11 additions & 32 deletions Python/instrumentation.c
Show comments
View file
Open in desktop
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Show hidden characters
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
#include "opcode_ids.h"

#include "pycore_call.h"
#include "pycore_code.h" // _PyCode_Clear_Executors()
#include "pycore_frame.h"
#include "pycore_interp.h"
#include "pycore_long.h"
Expand Down Expand Up @@ -583,13 +584,7 @@ de_instrument(PyCodeObject *code, int i, int event)
_Py_CODEUNIT *instr = &_PyCode_CODE(code)[i];
uint8_t *opcode_ptr = &instr->op.code;
int opcode = *opcode_ptr;
if (opcode == ENTER_EXECUTOR) {
int oparg = instr->op.arg;
_PyExecutorObject *exec = code->co_executors->executors[oparg];
opcode_ptr = &exec->vm_data.opcode;
opcode = *opcode_ptr;
assert(opcode != ENTER_EXECUTOR);
}
assert(opcode != ENTER_EXECUTOR);
if (opcode == INSTRUMENTED_LINE) {
opcode_ptr = &code->_co_monitoring->lines[i].original_opcode;
opcode = *opcode_ptr;
Expand Down Expand Up @@ -734,22 +729,7 @@ remove_tools(PyCodeObject * code, int offset, int event, int tools)
assert(event != PY_MONITORING_EVENT_LINE);
assert(event != PY_MONITORING_EVENT_INSTRUCTION);
assert(PY_MONITORING_IS_INSTRUMENTED_EVENT(event));
#ifndef NDEBUG
_Py_CODEUNIT co_instr = _PyCode_CODE(code)[offset];
uint8_t opcode = co_instr.op.code;
uint8_t oparg = co_instr.op.arg;
if (opcode == ENTER_EXECUTOR) {
_PyExecutorObject *exec = code->co_executors->executors[oparg];
assert(exec->vm_data.opcode != ENTER_EXECUTOR);
opcode = _PyOpcode_Deopt[exec->vm_data.opcode];
opcode = exec->vm_data.oparg;
}
else {
opcode = _Py_GetBaseOpcode(code, offset);
}
assert(opcode != ENTER_EXECUTOR);
assert(opcode_has_event(opcode));
#endif
assert(opcode_has_event(_Py_GetBaseOpcode(code, offset)));
_PyCoMonitoringData *monitoring = code->_co_monitoring;
if (monitoring && monitoring->tools) {
monitoring->tools[offset] &= ~tools;
Expand Down Expand Up @@ -1315,16 +1295,10 @@ initialize_tools(PyCodeObject *code)
for (int i = 0; i < code_len; i++) {
_Py_CODEUNIT *instr = &_PyCode_CODE(code)[i];
int opcode = instr->op.code;
int oparg = instr->op.arg;
if (opcode == ENTER_EXECUTOR) {
_PyExecutorObject *exec = code->co_executors->executors[oparg];
opcode = exec->vm_data.opcode;
oparg = exec->vm_data.oparg;
}
else if (opcode == INSTRUMENTED_LINE) {
assert(opcode != ENTER_EXECUTOR);
if (opcode == INSTRUMENTED_LINE) {
opcode = code->_co_monitoring->lines[i].original_opcode;
}
assert(opcode != ENTER_EXECUTOR);
bool instrumented = is_instrumented(opcode);
if (instrumented) {
opcode = DE_INSTRUMENT[opcode];
Expand All @@ -1335,7 +1309,7 @@ initialize_tools(PyCodeObject *code)
if (instrumented) {
int8_t event;
if (opcode == RESUME) {
event = oparg != 0;
event = instr->op.arg != 0;
}
else {
event = EVENT_FOR_OPCODE[opcode];
Expand Down Expand Up @@ -1588,6 +1562,9 @@ _Py_Instrument(PyCodeObject *code, PyInterpreterState *interp)
);
return 0;
}
if (code->co_executors != NULL) {
_PyCode_Clear_Executors(code);
Comment thread
gvanrossum marked this conversation as resolved.
}
int code_len = (int)Py_SIZE(code);
/* code->_co_firsttraceable >= code_len indicates
* that no instrumentation can be inserted.
Expand Down Expand Up @@ -1629,7 +1606,9 @@ _Py_Instrument(PyCodeObject *code, PyInterpreterState *interp)
for (int i = code->_co_firsttraceable; i < code_len; i+= _PyInstruction_GetLength(code, i)) {
_Py_CODEUNIT *instr = &_PyCode_CODE(code)[i];
CHECK(instr->op.code != 0);
assert(instr->op.code != ENTER_EXECUTOR);
int base_opcode = _Py_GetBaseOpcode(code, i);
assert(base_opcode != ENTER_EXECUTOR);
if (opcode_has_event(base_opcode)) {
int8_t event;
if (base_opcode == RESUME) {
Expand Down
Back | FazBrowse Home | New Git URL