Converting Logging client->list_entries to iterator. - #2636
Conversation
Sorry, something went wrong.
|
iirc logging has some very strange behavior with listing entries, I think @waprin is pretty familiar with it. |
Sorry, something went wrong.
|
Yes specifically it uses the page tokens as a progress indicator, not just as a means of breaking results into pages. This means you can get 0 results on first page, but then 10 results on the next page, so you really always want to check the page token. This is especially true for slower queries, which are most of them without an index e.g. logName= . |
Sorry, something went wrong.
|
@waprin So what are the implications for using an Iterator? It'll just take a long time consuming empty pages until the results are available? |
Sorry, something went wrong.
|
@dhermes yeah, if anything I'm thinking this is a very good change because with the iterator people are less likely to get no results on the first page and think there are no results. |
Sorry, something went wrong.
|
@waprin is there a possibility the iterator will never finish? |
Sorry, something went wrong.
| method='GET', path=self.path, | ||
| query_params=self._get_query_params()) | ||
| params = self._get_query_params() | ||
| if self._HTTP_METHOD == 'GET': |
This comment was marked as spam.
This comment was marked as spam.
| 'data': {}, | ||
| }) | ||
|
|
||
| def test__get_next_page_bad_http_method(self): |
This comment was marked as spam.
This comment was marked as spam.
| >>> client = logging.Client() | ||
| >>> entries, token = client.list_entries() # API call | ||
| >>> for entry in entries: | ||
| >>> for entry in client.list_entries(): # API call(s) |
This comment was marked as spam.
This comment was marked as spam.
| """Detect correct entry type from resource and instantiate. | ||
|
|
||
| :type resource: dict | ||
| :param resource: one entry resource from API response |
This comment was marked as spam.
This comment was marked as spam.
This comment was marked as spam.
| elif 'protoPayload' in resource: | ||
| return ProtobufEntry.from_api_repr(resource, client, loggers) | ||
|
|
||
| raise ValueError('Cannot parse log entry resource') |
This comment was marked as spam.
This comment was marked as spam.
| entry_pb = LogEntry(log_name=self.LOG_NAME, | ||
| timestamp = datetime.datetime.utcnow().replace(tzinfo=UTC) | ||
| timestamp_pb = _datetime_to_pb_timestamp(timestamp) | ||
| entry_pb = LogEntry(log_name=self.LOG_PATH, |
This comment was marked as spam.
This comment was marked as spam.
| return self.client.connection.api_request( | ||
| method=self._HTTP_METHOD, | ||
| path=self.path, | ||
| data=params) |
This comment was marked as spam.
|
As discussed with @daspecster, going to make the docstring updates in a follow-up PR |
Sorry, something went wrong.
Converting Logging client->list_entries to iterator.
Converting Logging client->list_entries to iterator.
@waprin @jonparrott I just realized how weird the logging HTTP/JSON API is. For /entries:list, POST is used instead of GET, and the "query params" come in the payload instead of as query params.
Just as #2633 started, I'm sending this out without fixing unit tests to get it in the hands of reviewers. The unit tests may take me a non-trivial time, but I am working on them.