FazBrowse GitHub Viewer | Trending |
URL:
| Home
Tools: [Download Repo ZIP]   [Original HTTPS Page]

host_env: share thread, locale, loader and BSTR (#8702) · RustPython/RustPython@3191eca · GitHub

Repository navigation

Commit 3191eca

Browse files
committed
host_env: share thread, locale, loader and BSTR (#8702)
* host_env: share thread, locale, loader and BSTR Add Windows thread name, locale-info/wcscoll, CompareStringOrdinal, LoadLibraryEx/GetProcAddress/FreeLibrary, BSTR and COM helpers, and AF_HYPERV UUID conversion. Wire _thread, locale, ctypes LoadLibrary and WMI BSTRs through those wrappers. Ensure =X: before spawnve/execve. Assisted-by: Claude * host_env: expose rustls-free ssl core on wasm Compile MemoryBIO, protocol constants, OID tables, hostname checks, and ALPN parsing without the rustls feature so wasm and other hosts can implement _ssl on the shared types. Assisted-by: Claude * stdlib: add wasm _ssl on rustls-free host_env Install _ssl on wasm32 against rustpython_host_env::ssl. MemoryBIO, constants, OID, hostname, and ALPN work. Context settings are stored; wrap/handshake and rustls-only loaders raise SSLError until a wasm TLS engine is wired. Assisted-by: Claude * host_env: address #8702 review comments Cache-miss Windows ctypes lookup uses GetProcAddress. FreeLibrary propagates OSError. start_new_thread returns GetThreadId. Export _thread._get_name and drop pyfunction doc comments. Reject embedded NULs in Windows locale.strcoll/strxfrm. Terminate wcscoll/wcsxfrm and SetThreadDescription inputs. wasm _ssl accepts wrap_bio owner and get_ca_certs(binary_form), and MemoryBIO.read(-1) returns all pending bytes. Assisted-by: Claude * host_env: gate the whole ssl module on feature ssl Compile rustpython_host_env::ssl only with the ssl feature. Rustls crates stay native-only, so enabling ssl on wasm does not pull rustls. Inside the module, rustls files follow not(wasm32) rather than a second feature flag. Assisted-by: Claude * host_env: fix Windows types and thread names Cast SysAllocStringLen to *mut u16 and RPC status to u32 so windows-gnu and MSVC agree. Export _NAME_MAXLEN, truncate names at NUL and the OS byte cap, and decode get_name with fsdecode. Assisted-by: Claude * host_env: compile rustls engine on WASI Keep rustls-free MemoryBIO/constants/OID/ALPN on all wasm targets. Compile the rustls engine on WASI, where UnixTime::now exists. Allow the rustpython ssl feature on wasm without ssl-rustls or ssl-openssl. Assisted-by: Claude * compiler: pick the earliest CPython parse override A later incompatible-prefix scan must not hide an earlier number or bracket diagnostic. Parser-driven tokenization stops at the first failure. Assisted-by: Claude * host_env: address remaining #8702 review comments Map a missing Windows export to a symbol-load error, keep LoadLibrary's Win32 code on the OSError, drop incomplete UTF-16 pairs when truncating thread names, hide set_name and _get_name off linux/macOS/Windows, and let ssl-rustls enable host_env. Assisted-by: Claude * stdlib: add wasm _socket and read-all MemoryBIO Register a rustls-free-style _socket shim on wasm32 so Lib/ssl.py can import. Treat MemoryBIO.read(-1) as read-all, matching _ssl_MemoryBIO_read_impl. Assisted-by: Claude * host_env: fix clippy, WASI ssl cfg, and wasm rustls Defer unclosed-bracket overrides until no token diagnostic exists, keep WASI rustls and wasm ssl modules exclusive, preserve lone Windows surrogates, compile the rustls engine on browser wasm, and clear clippy denials. Assisted-by: Claude * host_env: fix Windows clippy and wasm UnixTime Gate thread-name byte truncation to linux/macOS, allow GetProcAddress/FreeLibrary pointer deref, and enable rustls-pki-types web time on browser wasm. Assisted-by: Claude * host_env: lock rustls-pki-types web for wasm Assisted-by: Claude * host_env: silence Windows unused thread id and shear Assisted-by: Claude * stdlib: drop Windows locale clippy needless wraps Windows clippy failed on locale.strcoll/strxfrm needless return and _getdefaultlocale always wrapping Ok. Assisted-by: Claude * compiler: keep lexer errors over print hints print x; 0x must report an invalid hex literal. Ranking by source offset let the earlier print hint win. Treat a finished radix literal as valid so print x; 0x1 still gets the print message. Assisted-by: Claude * compiler: keep decode errors over unclosed openers Ranking by offset made ( '\\N' report an unclosed paren and wiped the unclosed-bracket flag on f'{, so compile snippets raised SyntaxError instead of incomplete input. Export wasm socket._accept for Lib/socket.py. Assisted-by: Claude * wasm: drop duplicate _socket; split locale cfgs _socket lives in rustpython-stdlib's socket_wasm. The wasm crate only re-exported that file when freeze-stdlib was off. Split strcoll, strxfrm, and getencoding into cfg'd function defs. Assisted-by: Claude * compiler: keep decode blockers over unclosed openers (print x; '\\N' must keep the print hint. A later malformed escape still suppresses the EOF opener even when print wins. Assisted-by: Claude * compiler: keep earlier f-string over later lexer errors f'{x'; ' and f'{x'; 0x must not report the trailing quote or hex literal. Decode and f-string diagnostics now compete with lexer failures by offset. Assisted-by: Claude * compiler: rank decode vs lexer vs print by class Unterminated quotes no longer hide format-spec newlines or empty f-string fields. Print is considered only after the final winner is not a lexer failure, so print x; f'{x'; 0x keeps the print hint. Line-continuation uses CPython wording. Assisted-by: Claude * compiler: keep unclosed paren over line-cont at EOF test_exceptions wants '(' was never closed at the opener when a final backslash follows. Skip escaped quotes in the bracket scan so {\'a\'} still reports the continuation error. Un-xfail test_join_daemon_thread_in_finalization. Assisted-by: Claude * test_threading: xfail daemon join finalization off Linux The test passes on Linux. macOS and Windows leave an ignored deallocator exception on stderr. Assisted-by: Claude * vm: hang daemon threads at interpreter finalize Forced-off daemons must stay not-done so join() during finalization GC raises PythonFinalizationError and Thread.is_alive() stays true. SystemExit made the handle look finished on macOS/Windows. Drop the Lib/test xfail. Assisted-by: Claude (cherry picked from commit d1c4925) Assisted-by: Codex:gpt-6
1 parent 7c4017b commit 3191eca

32 files changed

Lines changed: 1971 additions & 187 deletions

‎Cargo.lock‎

Lines changed: 2 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ freeze-stdlib = ["stdlib", "rustpython-vm/freeze-stdlib", "rustpython-pylib?/fre
2222
jit = ["rustpython-vm/jit"]
2323
threading = ["rustpython-vm/threading", "rustpython-stdlib/threading"]
2424
sqlite = ["rustpython-stdlib/sqlite"]
25-
ssl = ["host_env"]
25+
ssl = ["host_env", "rustpython-stdlib?/ssl"]
2626
ssl-rustls = ["ssl", "rustpython-stdlib/ssl-rustls"]
2727
ssl-rustls-aws-lc = ["ssl-rustls", "dep:rustls", "rustls/aws_lc_rs"]
2828
ssl-rustls-aws-lc-fips = ["ssl-rustls-aws-lc", "rustls/fips"]

‎Lib/test/test_threading.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1183,7 +1183,6 @@ def __del__(self):
11831183
self.assertEqual(out.strip(), b"OK")
11841184
self.assertIn(b"can't create new thread at interpreter shutdown", err)
11851185

1186-
@unittest.expectedFailure # TODO: RUSTPYTHON
11871186
def test_join_daemon_thread_in_finalization(self):
11881187
# gh-123940: Py_Finalize() prevents other threads from running Python
11891188
# code, so join() can not succeed unless the thread is already done.

‎crates/compiler/src/lib.rs‎

Lines changed: 207 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -208,6 +208,7 @@ impl NormalizedParseDiagnostic {
208208
/// column. These are reconstructed by re-scanning after ruff's parse has already failed, so they
209209
/// carry CPython's wording rather than a translation of ruff's own error, and they never reach
210210
/// ruff — `NormalizedParseDiagnostic` and `CompileError` are the only things that consume one.
211+
#[derive(Clone)]
211212
struct CpythonDiagnostic {
212213
message: String,
213214
range: ruff_text_size::TextRange,
@@ -243,6 +244,52 @@ impl CpythonDiagnostic {
243244
}
244245
}
245246

247+
/// Lexer-class failures outrank print hints. Decode and f-string diagnostics
248+
/// compete with lexer failures by offset, and an unterminated quote is only
249+
/// a fallback when no decode/f-string diagnostic exists. Print is considered
250+
/// against the final winner: a later 0x still beats print, but an f-string
251+
/// that replaced that 0x must not hide an earlier print. A decode diagnostic
252+
/// also suppresses an EOF unclosed opener.
253+
#[derive(Clone, Copy, PartialEq, Eq)]
254+
enum OverrideClass {
255+
Lexer,
256+
Decode,
257+
Print,
258+
}
259+
260+
struct RankedOverride {
261+
diagnostic: CpythonDiagnostic,
262+
unclosed_bracket: bool,
263+
class: OverrideClass,
264+
}
265+
266+
fn consider_override(
267+
best: &mut Option<RankedOverride>,
268+
diagnostic: CpythonDiagnostic,
269+
class: OverrideClass,
270+
) {
271+
let unclosed_bracket = diagnostic.is_unclosed_bracket;
272+
consider_ranked(best, diagnostic, unclosed_bracket, class);
273+
}
274+
275+
fn consider_ranked(
276+
best: &mut Option<RankedOverride>,
277+
diagnostic: CpythonDiagnostic,
278+
unclosed_bracket: bool,
279+
class: OverrideClass,
280+
) {
281+
if best
282+
.as_ref()
283+
.is_none_or(|current| diagnostic.range.start() < current.diagnostic.range.start())
284+
{
285+
*best = Some(RankedOverride {
286+
diagnostic,
287+
unclosed_bracket,
288+
class,
289+
});
290+
}
291+
}
292+
246293
fn cpython_parse_diagnostic_override(
247294
error: &parser::ParseError,
248295
source_file: &SourceFile,
@@ -258,18 +305,80 @@ fn cpython_parse_diagnostic_override(
258305
};
259306
}
260307

261-
source_error!(invalid_number_literal_error(source_text));
262-
source_error!(invalid_legacy_statement_error(source_text));
263-
source_error!(incompatible_string_prefix_error(source_text));
264-
source_error!(malformed_unicode_n_escape_error(source_text));
265-
source_error!(non_printable_character_error(source_text));
266-
source_error!(invalid_interpolated_string_error(source_text));
267-
source_error!(mixed_tstring_literal_error(error, source_text));
268-
269-
if let Some(bracket) = bracket_syntax_error(source_text) {
308+
let mut earliest: Option<RankedOverride> = None;
309+
if let Some(diagnostic) = invalid_number_literal_error(source_text) {
310+
consider_override(&mut earliest, diagnostic, OverrideClass::Lexer);
311+
}
312+
if let Some(diagnostic) = incompatible_string_prefix_error(source_text) {
313+
consider_override(&mut earliest, diagnostic, OverrideClass::Lexer);
314+
}
315+
if let Some(diagnostic) = non_printable_character_error(source_text) {
316+
consider_override(&mut earliest, diagnostic, OverrideClass::Lexer);
317+
}
318+
let bracket = bracket_syntax_error(source_text);
319+
if let Some(bracket) = bracket.as_ref() {
320+
// Unclosed openers are reported at the opener and only become errors
321+
// at EOF. A later token-time diagnostic (invalid number, prefix, …)
322+
// must keep winning. Mismatched closers stay in the lexer-class
323+
// positional ranking.
324+
if !bracket.unclosed {
325+
consider_ranked(
326+
&mut earliest,
327+
bracket.diagnostic.clone(),
328+
false,
329+
OverrideClass::Lexer,
330+
);
331+
}
332+
}
333+
let mut saw_decode = false;
334+
if let Some(diagnostic) = malformed_unicode_n_escape_error(source_text) {
335+
saw_decode = true;
336+
consider_override(&mut earliest, diagnostic, OverrideClass::Decode);
337+
}
338+
if let Some(diagnostic) = invalid_interpolated_string_error(source_text) {
339+
saw_decode = true;
340+
consider_override(&mut earliest, diagnostic, OverrideClass::Decode);
341+
}
342+
if let Some(diagnostic) = mixed_tstring_literal_error(error, source_text) {
343+
saw_decode = true;
344+
consider_override(&mut earliest, diagnostic, OverrideClass::Decode);
345+
}
346+
// A later ordinary unterminated quote is only a fallback. Format-spec
347+
// newlines and empty fields are decode diagnostics and must keep winning.
348+
let line_continuation = matches!(
349+
&error.error,
350+
parser::ParseErrorType::Lexical(parser::LexicalErrorType::LineContinuationError)
351+
);
352+
if !saw_decode
353+
&& !line_continuation
354+
&& let Some(diagnostic) = unterminated_string_error(source_text, mode)
355+
{
356+
consider_override(&mut earliest, diagnostic, OverrideClass::Lexer);
357+
}
358+
if earliest
359+
.as_ref()
360+
.is_none_or(|current| current.class != OverrideClass::Lexer)
361+
&& let Some(diagnostic) = invalid_legacy_statement_error(source_text)
362+
{
363+
consider_override(&mut earliest, diagnostic, OverrideClass::Print);
364+
}
365+
if !saw_decode
366+
&& earliest
367+
.as_ref()
368+
.is_none_or(|current| current.class == OverrideClass::Print)
369+
&& let Some(bracket) = bracket.filter(|bracket| bracket.unclosed)
370+
{
371+
consider_ranked(
372+
&mut earliest,
373+
bracket.diagnostic,
374+
true,
375+
OverrideClass::Lexer,
376+
);
377+
}
378+
if let Some(override_diag) = earliest {
270379
return Some(
271-
NormalizedParseDiagnostic::other(source_file, bracket.diagnostic)
272-
.with_unclosed_bracket(bracket.unclosed),
380+
NormalizedParseDiagnostic::other(source_file, override_diag.diagnostic)
381+
.with_unclosed_bracket(override_diag.unclosed_bracket),
273382
);
274383
}
275384

@@ -293,7 +402,9 @@ fn cpython_parse_diagnostic_override(
293402
}
294403
let loc = source_location(source_file, error.location.start() + TextSize::from(1));
295404
return Some(NormalizedParseDiagnostic::new(
296-
error.error.clone(),
405+
parser::ParseErrorType::OtherError(
406+
"unexpected character after line continuation character".to_owned(),
407+
),
297408
loc,
298409
loc,
299410
));
@@ -591,7 +702,11 @@ fn invalid_radix_literal_error(
591702
let mut has_digit = false;
592703
loop {
593704
let Some(&byte) = bytes.get(index) else {
594-
return Some((format!("invalid {kind} literal"), start + 1));
705+
return if has_digit {
706+
None
707+
} else {
708+
Some((format!("invalid {kind} literal"), start + 1))
709+
};
595710
};
596711
if byte == b'_' {
597712
let Some(&next) = bytes.get(index + 1) else {
@@ -5876,6 +5991,7 @@ fn expected_opening_bracket(closing: char) -> char {
58765991

58775992
/// A bracket diagnostic, and whether it is an opener that was never closed. The caller needs
58785993
/// that apart from the message because ruff reports the unclosed case as an EOF error.
5994+
#[derive(Clone)]
58795995
struct BracketError {
58805996
diagnostic: CpythonDiagnostic,
58815997
unclosed: bool,
@@ -5932,6 +6048,24 @@ fn bracket_syntax_error(source: &str) -> Option<BracketError> {
59326048
continue;
59336049
}
59346050

6051+
if ch == '\\' {
6052+
match chars.get(index + 1).map(|(_, next)| *next) {
6053+
Some('\n' | '\r') => {
6054+
escape_next = true;
6055+
index += 1;
6056+
continue;
6057+
}
6058+
Some(_) => {
6059+
index += 2;
6060+
continue;
6061+
}
6062+
None => {
6063+
index += 1;
6064+
continue;
6065+
}
6066+
}
6067+
}
6068+
59356069
if ch == '\'' || ch == '"' {
59366070
is_raw_string = false;
59376071
for look_back in 1..=2.min(index) {
@@ -7496,6 +7630,49 @@ mod tests {
74967630
("fu''", "'u' and 'f' prefixes are incompatible"),
74977631
("fb''", "'b' and 'f' prefixes are incompatible"),
74987632
("ufr''", "'u' and 'r' prefixes are incompatible"),
7633+
(
7634+
"(]\nbu'x'",
7635+
"closing parenthesis ']' does not match opening parenthesis '('",
7636+
),
7637+
("0x\nbu'x'", "invalid hexadecimal literal"),
7638+
("(0x", "invalid hexadecimal literal"),
7639+
("print x; 0x", "invalid hexadecimal literal"),
7640+
("exec x; 0x", "invalid hexadecimal literal"),
7641+
(
7642+
"print x; 0x1",
7643+
"Missing parentheses in call to 'print'. Did you mean print(...)?",
7644+
),
7645+
(
7646+
"print x; (",
7647+
"Missing parentheses in call to 'print'. Did you mean print(...)?",
7648+
),
7649+
("print x; )", "unmatched ')'"),
7650+
(
7651+
"( '\\N'",
7652+
"(unicode error) 'unicodeescape' codec can't decode bytes in position 0-1: malformed \\N character escape",
7653+
),
7654+
("(print x", "'(' was never closed"),
7655+
(
7656+
"(print x; '\\N'",
7657+
"Missing parentheses in call to 'print'. Did you mean print(...)?",
7658+
),
7659+
("f'{x'; '", "f-string: expecting '}'"),
7660+
("f'{x'; 0x", "f-string: expecting '}'"),
7661+
(
7662+
"print x; f'{x'; 0x",
7663+
"Missing parentheses in call to 'print'. Did you mean print(...)?",
7664+
),
7665+
(
7666+
concat!("f'{1:", "d\n}'"),
7667+
"f-string: newlines are not allowed in format specifiers",
7668+
),
7669+
("f'{\n}'", "f-string: valid expression required before '}'"),
7670+
(
7671+
"f'''\n{\n# only a comment\n}'''",
7672+
"f-string: valid expression required before '}'",
7673+
),
7674+
("{\\'a\\'}", "unexpected character after line continuation"),
7675+
("\"\\\n\"(1 for c in I,\\\n\\", "'(' was never closed"),
74997676
(
75007677
r"'\N'",
75017678
"(unicode error) 'unicodeescape' codec can't decode bytes in position 0-1: malformed \\N character escape",
@@ -7514,6 +7691,23 @@ mod tests {
75147691
}
75157692
}
75167693

7694+
#[test]
7695+
fn unclosed_fstring_field_keeps_the_unclosed_bracket_flag() {
7696+
let err = compile("f'{", Mode::Eval, "<interp>", CompileOpts::default())
7697+
.expect_err("should not compile");
7698+
let crate::CompileError::Parse(parse) = err else {
7699+
panic!("expected a parse error, got {err}");
7700+
};
7701+
assert!(
7702+
parse.is_unclosed_bracket,
7703+
"unclosed f-string field must stay incomplete, got {parse}"
7704+
);
7705+
assert!(
7706+
parse.to_string().contains("'{' was never closed"),
7707+
"got {parse}"
7708+
);
7709+
}
7710+
75177711
#[test]
75187712
fn interpolated_literals_do_not_take_the_escaped_quote_hint() {
75197713
// Parser/lexer/lexer.c offers "perhaps you escaped the end quote?" from its plain-string

‎crates/host_env/Cargo.toml‎

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -16,19 +16,19 @@ ssl = ["dep:rustls", "dep:rustls-pemfile", "dep:rustls-webpki", "dep:x509-parser
1616
rustpython-wtf8 = { workspace = true }
1717

1818
bitflags = { workspace = true }
19+
der = { workspace = true, optional = true }
1920
getrandom = { workspace = true }
2021
libc = { workspace = true }
2122
num-traits = { workspace = true }
2223
parking_lot = { workspace = true }
2324
paste = { workspace = true }
24-
widestring = { workspace = true }
25+
pkcs8 = { workspace = true, features = ["encryption", "pkcs5", "pem"], optional = true }
2526
rustls = { workspace = true, default-features = false, features = ["std", "tls12"], optional = true }
2627
rustls-pemfile = { workspace = true, optional = true }
2728
rustls-webpki = { workspace = true, optional = true }
28-
x509-parser = { workspace = true, optional = true }
29-
der = { workspace = true, optional = true }
30-
pkcs8 = { workspace = true, features = ["encryption", "pkcs5", "pem"], optional = true }
3129
sha2 = { workspace = true, optional = true }
30+
widestring = { workspace = true }
31+
x509-parser = { workspace = true, optional = true }
3232

3333
[target.'cfg(unix)'.dependencies]
3434
nix = { workspace = true }
@@ -39,6 +39,11 @@ rustix = { workspace = true }
3939
[target.'cfg(any(not(target_arch = "wasm32"), target_os = "wasi"))'.dependencies]
4040
num_cpus = "1.17.0"
4141

42+
# rustls-pki-types::UnixTime::now is gated off wasm32-unknown-unknown unless
43+
# the `web` feature is on. Browser rustls and Charon/LLBC both need it.
44+
[target.'cfg(all(target_arch = "wasm32", not(target_os = "wasi")))'.dependencies]
45+
rustls-pki-types = { version = "1", default-features = false, features = ["alloc", "std", "web"] }
46+
4247
[target.'cfg(not(any(target_os = "ios", target_os = "android", target_os = "windows", target_arch = "wasm32", target_os = "redox")))'.dependencies]
4348
mac_address = { workspace = true }
4449

@@ -76,9 +81,11 @@ windows-sys = { workspace = true, features = [
7681
"Win32_Security",
7782
"Win32_Security_Authorization",
7883
"Win32_Storage_FileSystem",
84+
"Win32_System_Com",
7985
"Win32_System_Console",
8086
"Win32_System_Diagnostics_Debug",
8187
"Win32_System_Environment",
88+
"Win32_System_Ole",
8289
"Win32_System_IO",
8390
"Win32_System_Ioctl",
8491
"Win32_System_JobObjects",
@@ -106,3 +113,8 @@ cc = "1"
106113

107114
[lints]
108115
workspace = true
116+
117+
[package.metadata.cargo-shear]
118+
# Feature-unification only: enables rustls-pki-types `web` so UnixTime::now
119+
# exists on wasm32-unknown-unknown. No rust source imports this crate.
120+
ignored = ["rustls-pki-types"]

0 commit comments

Comments
 (0)

Back | FazBrowse Home | New Git URL