| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
Sorry, something went wrong.
|
One other thing I'll note - the pure python implementation here is at least partially optimized for speed. There is a fairly trivial implementation that looks, more or less, like this: from datetime import datetime
_base_strptimes = {
10: '%Y-%m-%d',
13: '%Y-%m-%dT%H',
16: '%Y-%m-%dT%H:%M',
19: '%Y-%m-%dT%H:%M:%S',
23: '%Y-%m-%dT%H:%M:%S.%f',
26: '%Y-%m-%dT%H:%M:%S.%f'
}
def from_strptime(dtstr):
if not isinstance(dtstr, str):
raise TypeError('isoformat takes str')
if len(dtstr) >= 19 and dtstr[-6] in '+-':
fmt = _base_strptimes[len(dtstr) - 6] + '%z'
else:
fmt = _base_strptimes[len(dtstr)]
return datetime.strptime(dtstr, fmt)This will work, but strptime is actually fairly slow, and in my benchmarks the pure python implementation here is twice as fast as the strptime-based method. |
Sorry, something went wrong.
There was a problem hiding this comment.
I don't think "static inline" is interpreted differently from "static" by any relevant compilers. AFAIK, we don't use "static inline" elsewhere in CPython code. Please just use "static".
Sorry, something went wrong.
There was a problem hiding this comment.
Not a problem. I translated these from macros in my original code so it seemed like inline was appropriate at the time, but I don't have strong opinions in this regard.
Sorry, something went wrong.
There was a problem hiding this comment.
The name "ptr" is usually used for pointers. Please change "ptr" to "ch" here. On the other hand, I don't think to_int(ch) is clearer than ch - '-'. Call it digit_to_int or just use the expanded expression.
Sorry, something went wrong.
There was a problem hiding this comment.
Yeah, this is my mistake. As I mentioned in another comment, I translated this from a macro where it was actually manipulating the pointer directly. When I translated it into a function the variable name didn't get fixed.
Sorry, something went wrong.
There was a problem hiding this comment.
I would just use ANSI C isdigit on the character.
Sorry, something went wrong.
There was a problem hiding this comment.
I can actually just replace this with tmp <= 9, I think isdigit might be overkill. This is_digit relies on the specific way that to_int works - and I only used this particular combination because in early Cython-based tests my profiling indicated that atoi was a slow step.
Sorry, something went wrong.
|
A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated. Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request. |
Sorry, something went wrong.
There was a problem hiding this comment.
Should the colon after :MM be optional?
Sorry, something went wrong.
There was a problem hiding this comment.
Singular: constructor
Sorry, something went wrong.
There was a problem hiding this comment.
I think you need to parse the time zone without :MM:SS.mmmmmm, e.g. 09:12:32+11:00. See your datetime.datetime format.
Sorry, something went wrong.
There was a problem hiding this comment.
If it is worth testing this, maybe it is worth documenting it?
Sorry, something went wrong.
There was a problem hiding this comment.
That can be done. I considered doing it but decided against it because while this behavior is both true and tested for all the constructors in datetime, it's not documented for any of them as far as I've seen. I can document it in just this one, or I can document it in all of them, I'm not picky.
Sorry, something went wrong.
There was a problem hiding this comment.
6 × 5 × 8 × 5 is 1200 tests. Perhaps just one test for each case (6 + 5 + 8 + 5) instead?
Sorry, something went wrong.
There was a problem hiding this comment.
This is a good point. I translated these tests from hypothesis where the space is sampled sparsely. I think I can refactor out at least the separator dimension and probably reduce the dimensionality of some of the remaining tests. I'll look at it tomorrow.
Sorry, something went wrong.
There was a problem hiding this comment.
I refactored out the separator tests and reduced some of the space that is sampled. I think most of the variation in the date and time portions separately is covered by the respective date and time parser functions. I think we're down to ~160 tests.
Sorry, something went wrong.
There was a problem hiding this comment.
In retrospect, I think maybe it's logically impossible to hit this branch, since we only get here if tstr[pos:pos+1] == ':'. I'll remove this in the next iteration.
Sorry, something went wrong.
There was a problem hiding this comment.
I guess this check is redundant with the (len_str - pos) != 6 check above.
Sorry, something went wrong.
|
@abalkin @vadmium I think I've fixed all the concerns raised so far - some good catches in there. There are three documentation-related concerns:
|
Sorry, something went wrong.
There was a problem hiding this comment.
I'm not really sure why this is not getting hit by the test suite. test_fromisoformat_fails_typeerror is designed explicitly to hit this condition. Anyone have an idea why it's getting missed?
Sorry, something went wrong.
There was a problem hiding this comment.
self.theclass.fromisoformat(bad_type)
Sorry, something went wrong.
There was a problem hiding this comment.
A-ha! Thanks!
Sorry, something went wrong.
There was a problem hiding this comment.
Sorry I missed this on the first review. PEP 7 calls for "function name in column 1". See PEP 7.
Sorry, something went wrong.
There was a problem hiding this comment.
C99 - style for loops are not specifically mentioned in the "C dialect" section of PEP 7. I personally like this style, so if this compiles without warnings, we may leave it.
Sorry, something went wrong.
There was a problem hiding this comment.
Please alight the .
Sorry, something went wrong.
There was a problem hiding this comment.
No need for a ; after the label.
Sorry, something went wrong.
There was a problem hiding this comment.
Unfortunately yes, there is. It's a syntax error without the ;, because the following line is a variable declaration, not a statement, and the label is intended to be a labeled statement (Ref StackOverflow, citing section 6.8.1).
That said, there's no particular reason why those variables can't be declared at the top of the function (before the first call to PARSE_ISOFORMAT_ADVANCE_TIME_SEP). Would you prefer that?
Sorry, something went wrong.
There was a problem hiding this comment.
Please use PEP 7 code layout.
Sorry, something went wrong.
|
To fully support isoformat output you also need to parse seconds on the timezone:
|
Sorry, something went wrong.
|
@mariocj89 Thanks for the catch. The PR is updated with tests and implementation. Didn't realize that second precision was allowed in isoformat() in Python 3.7. |
Sorry, something went wrong.
There was a problem hiding this comment.
This check is actually wrong, so this branch is never reached. The correct code is:
if ( (PyTypeObject*)cls == &PyDateTime_DateType ) {
Sorry, something went wrong.
Moreover, the sub-second precision is now allowed as well: >>> from datetime import *
>>> import random
>>> datetime.now(timezone(timedelta(hours=24*random.random()))).isoformat()
'2017-12-08T11:34:30.238049+15:47:34.297540'The good news is that you should be able to reuse the time fields parser logic to parse the timezone. cc: @pganssle |
Sorry, something went wrong.
|
@abalkin Unless I'm doing something wrong, it seems that the pure python implementation of isoformat() doesn't actually support subsecond precision. import sys
sys.modules['_datetime'] = None # cause ImportError
from datetime import *
print(time(12, 30, 15, tzinfo=timezone(timedelta(microseconds=123456))).isoformat())
# 12:30:15+00:00:00
print(datetime(2017, 1, 1, 12, 30, 15,
tzinfo=timezone(timedelta(microseconds=123456))).isoformat())
# Traceback (most recent call last):
# File "<stdin>", line 2, in <module>
# File ".../cpython/Lib/datetime.py", line 1832, in isoformat
# assert not ss.microseconds
# AssertionErrorI suppose I'll fix that as part of this PR. |
Sorry, something went wrong.
|
Latest changes fix both isoformat() and fromisoformat() in pure python and C implementations for subsecond offsets. |
Sorry, something went wrong.
There was a problem hiding this comment.
Documentation needs updating to reflect second and microsecond precision.
Sorry, something went wrong.
@pganssle - It looks like you are right. Please mention bpo-5288 in the commit message when you fix this. Please also find out why this was not caught by the test suit. We should be running all tests with and without C acceleration. |
Sorry, something went wrong.
|
@abalkin It's already fixed and tests added. I didn't see any tests actually testing subsecond support for timezones in .isoformat, which is why it wasn't caught by the tests. I just pushed a rewritten history that mentions bpo-5288 in the commit fixing the pure python version. |
Sorry, something went wrong.
|
@pganssle - great job! This PR looks good to me now and I will merge it in a few days to give others a chance to review. |
Sorry, something went wrong.
|
|
||
| Return a :class:`time` corresponding to a *time_string* in one of the ISO 8601 | ||
| formats emitted by :meth:`time.isoformat`. Specifically, this function supports | ||
| strings in the format(s) ``HH[:MM[:SS[.mmm[mmm]]]]][+HH:MM[:SS[.ffffff]]]``. |
There was a problem hiding this comment.
Too many square brackets.
Sorry, something went wrong.
| .. classmethod:: datetime.fromisoformat(date_string) | ||
|
|
||
| Return a :class:`datetime` corresponding to a *date_string* in one of the | ||
| ISO 8601 formats emitted by :meth:`datetime.isoformat`. Specifically, this function |
There was a problem hiding this comment.
I would drop “ISO 8601” and just say “one of the formats emitted by isoformat”. As far as I understand, ISO 8601 doesn’t have a seconds field in time zones, but it seems you want to support this.
If you intend to support dates without the time part, maybe write “emitted by datetime.isoformat and date.isoformat”. This may also need a test case; I didn’t notice anything relevant.
Sorry, something went wrong.
There was a problem hiding this comment.
@vadmium The test cases from TestDate are actually inherited by TestDateTime, so it is indeed supported. I think it's fair to support them.
Sorry, something went wrong.
| try: | ||
| assert len(date_string) == 10 | ||
| return cls(*_parse_isoformat_date(date_string)) | ||
| except: |
There was a problem hiding this comment.
Write except Exception, to avoid catching KeyboardInterrupt or similar. Or even better, be explicit and list the exceptions you are expecting (AssertionError, ValueError, IndexError?).
Sorry, something went wrong.
There was a problem hiding this comment.
@vadmium I think catching Exception is probably the right thing to do, I didn't think about KeyboardInterrupt. I'm mainly trying to get the C implementation and the Python implementation to always raise the same exceptions, and the C implementation only ever raises ValueError (I will fuzz it some time this week to verify this), hence the catch-and-re-raise.
Sorry, something went wrong.
| @@ -1193,19 +1320,7 @@ def __hash__(self): | |||
| def _tzstr(self, sep=":"): | |||
There was a problem hiding this comment.
Good opportunity to remove the unsupported sep parameter
Sorry, something went wrong.
| @@ -1193,19 +1320,7 @@ def __hash__(self): | |||
| def _tzstr(self, sep=":"): | |||
| """Return formatted timezone offset (+xx:xx) or None.""" | |||
There was a problem hiding this comment.
or the empty string
Sorry, something went wrong.
| Return a :class:`datetime` corresponding to a *date_string* in one of the | ||
| ISO 8601 formats emitted by :meth:`datetime.isoformat`. Specifically, this function | ||
| supports strings in the format(s) ``YYYY-MM-DD[*HH[:MM[:SS[.mmm[mmm]]]]][+HH:MM[:SS[.ffffff]]]``, | ||
| where ``*`` can match any single character. |
There was a problem hiding this comment.
I think it would be less ambiguous putting the time zone inside the optional time part. Test case:
datetime(2017, 12, 18, 11, 0).isoformat(sep="+", timespec="minutes") -> "2017-12-18+11:00"
datetime.fromisoformat("2017-12-18+11:00") -> datetime(2017, 12, 18, 11, 0)
Sorry, something went wrong.
There was a problem hiding this comment.
Hmmm.. As much as I dislike the idea of timezone being part of the time component, this is fine I suppose (and in fact that is how it is currently implemented).
That said, another design I considered is one where we take sep as a keyword argument to isoparse, which would relieve this ambiguity for all separators other than - and + if we wanted to eventually allow parsing of strings of the format YYYY-MM-DD+HH:MM. That's one decision we'd have to make in this version because changing it would not be backwards compatible.
Sorry, something went wrong.
|
|
||
| Return a :class:`date` corresponding to a *date_string* in one of the ISO 8601 | ||
| formats emitted by :meth:`date.isoformat`. Specifically, this function supports | ||
| strings in the format(s) ``YYYY-MM-DD``. |
There was a problem hiding this comment.
There is only one format emitted by date.isoformat and supported by your date.fromisoformat, as far as I know.
Sorry, something went wrong.
|
|
||
| Other constructor: | ||
|
|
||
| .. classmethod:: time.fromisoformat(date_string) |
There was a problem hiding this comment.
time_string?
Sorry, something went wrong.
|
@vadmium Thanks for all the comments. I've added a test for the ambiguous cases you identified and updated the documentation (plus dropped the unused sep parameter). |
Sorry, something went wrong.
|
@abalkin @vadmium Any thoughts on the question of whether to add a sep parameter to fromisoformat (from this now-hidden comment)? The only reason to do so would be if in a future release we want to support ISO-8601 style strings that are just dates with offsets attached (e.g. 2017-01-01+12:00). I'm not sure, but I think ISO-8601 makes time zone offsets a property of time, so such dates-with-offsets are probably not actually valid ISO-8601 strings. Because 2017-01-01+12:00 could have been generated from datetime.isoformat(sep='+'), in its current form it would be impossible to get fromisoformat to interpret that +12:00 as anything except the time component. If we make the default behavior "strict", then in the future we would be able to support these "out of the box", specifying the separator as + or - would cause these style strings to be interpreted as times, and otherwise they would be offsets. Having given this a bit of thought, I think it's probably a good idea to leave out sep, and if we want to support this use case in future releases, specifying a separator other than + or - would allow these things to be parsed as intended. Having the default value ignore the separator will make it easier to handle what is probably the most frequent use cases, which will be sep=' ' and sep='T' out of the box without any sort of branching to detect which one you're in, etc. If we assume that some fraction of people will want to be strict and some people will want to be lenient, no matter which one we choose, the other behavior can easily be emulated. If we choose lenient by default, strict users can do the equivalent with: def strict_fromisoformat(dtstr, sep='T'):
if dtstr[10:11] != sep:
raise ValueError('Invalid isoformat string: %s' % dstr)
return datetime.fromisoformat(dtstr)If we choose strict by default, the "lenient by default" option is: def lenient_fromisoformat(dtstr):
# Assuming sep requires a single character be passed, for dates it can be anything
return datetime.fromisoformat(dtstr, sep=dtstr[10:11] or 'T')Anyway, this is a kind of long comment just to say, "I think we should keep the status quo", but I thought I'd at least outline my thinking. |
Sorry, something went wrong.
-1 or adding a parameter, but I would not mind having fromisoformat() not checking the separator at all. |
Sorry, something went wrong.
|
@abalkin The current behavior is to allow any single character for the separator at all, if that's what you mean. |
Sorry, something went wrong.
Yes. That's what I want. |
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
Per discussion on the python-dev mailing list, this is a C implementation of fromisoformat as alternate constructors for datetime, date and time, resolving this issue.
If it is deemed desirable, it can be extended later to cover all ISO-8601 datetime strings, but I believe the consensus so far is to have a minimum implementation that only covers the outputs of datetime.isoformat().
One thing I'd like to call attention to is that my profiling seems to indicate that in the common case (i.e. not calling this from a subclass), using the C API directly is much faster than going through PyObject_CallFunction.
Here is a profiling script:
Result on my laptop:
If there's no particularly pressing reason why all the other alternate constructors universally go through the main constructor call, I could write a small function or macro that would take the fast path if available and use it for all the alternate constructors.
CC @abalkin @mariocj89
https://bugs.python.org/issue15873