Repository navigation
Default value for max_retries is too short to "clear" all rate limit windows #1889
Description
Activity
Thanks for the report @swarren. Do you have a few links for some of these new endpoints and rate limit behaviors, maybe some upstream MR/issues for us to look at? :) Thanks. 10 minutes seems like quite a lot so we should maybe find some sensible approach, I like your idea in the other issue with global options.
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78364
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73069/diffs#1246552930c7d15d9888238d19f23329f673f4d6_52_52BTW, I wrote a script to continually hit this endpoint, and found that the Retry-After header isn't sent for it:-(
https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78364 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73069/diffs#1246552930c7d15d9888238d19f23329f673f4d6_52_52
BTW, I wrote a script to continually hit this endpoint, and found that the Retry-After header isn't sent for it:-(
Interesting, I guess GitLab is almost approaching something similar to GitHub's way of doing secondary rate limits and we'll have issues similar to PyGithub/PyGithub#2113. 🤔
I notice this on the user endpoint
I removed the non RateLimit headers.
{'RateLimit-Limit': '2000', 'RateLimit-Observed': '20', 'RateLimit-Remaining': '1980', 'RateLimit-Reset': '1645057248', 'RateLimit-ResetTime': 'Thu, 17 Feb 2022 00:20:48 GMT'}Looks like we need to add support for these new headers: https://docs.gitlab.com/ee/user/admin_area/settings/user_and_ip_rate_limits.html#response-headers
- added 2 commits that reference this issue on Feb 17, 2022
@JohnVillalovos I don't believe that change will solve my original issue. There are no rate-limit or retry headers at all when the rate limit is hit on /users/:id:
for i in $(seq 305); do curl --verbose --header "Authorization: Bearer xxx" "https://gitlab.com/api/v4/users/5400102"; done 2>&1 | tee log.txt
eventually yields:
HTTP/2 429 < date: Thu, 10 Mar 2022 04:08:28 GMT < content-type: application/json < content-length: 89 < cache-control: no-cache < vary: Origin < x-content-type-options: nosniff < x-frame-options: SAMEORIGIN < x-request-id: 01FXS0728WE6KDCRW14TR5RE9P < x-runtime: 0.026642 < strict-transport-security: max-age=31536000 < referrer-policy: strict-origin-when-cross-origin < gitlab-lb: fe-01-lb-gprd < gitlab-sv: api-gke-us-east1-c < cf-cache-status: DYNAMIC < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=JfaflNapv18Po%2BZPk13XOp2%2FJJE0oSpTsz4CtpzSA3f0RBuphqWYa%2BWImkWAzxE5Urk20dQtug9veDTFkRFPqyCkK6WOr9Lzk7tTT7XbqrhnZBaZMFuj4pkKYcpaE7mhuySC8V%2BkDMI%3D"}],"group":"cf-nel","max_age":604800} < nel: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800} < server: cloudflare < cf-ray: 6e99307aeccfc7b5-DEN < { [89 bytes data] 100 89 100 89 0 0 380 0 --:--:-- --:--:-- --:--:-- 378@JohnVillalovos I don't believe that change will solve my original issue. There are no rate-limit or retry headers at all when the rate limit is hit on /users/:id:
Has an issue been filed with GitLab to figure out if this is "as intended" or an oversight on their part?
I haven't filed a bug
Gitlab support did notify me that the server-side changes would affect my usage, and I mentioned to them that python-gitlab should handle this based on rate limit headers so I wouldn't need to change anything. Then when the server-side change was rolled out I notified them again that python-gitlab wasn't able to handle this due to this missing rate limit headers. They didn't reply to that last point. Thus, at least part of gitlab should be aware of this, although I haven't formally filed a bug.
Hello, I've also encountered this issue while working with gitlab.com API. It seems that the "Retry-After", etc. headers are missing, hence the default wait_time set by python-gitlab is too low.
IMO we should:- Set the default wait time should be 1 minute since gitlab.com rate limits are calculate per 1 minute.
- Open a bug with gitlab's team.
edit: opened a bug request with them
https://gitlab.com/gitlab-org/gitlab/-/issues/365728Reacted by Nejc HabjanThis issue was marked stale because it has been open 60 days with
no activity. Please remove the stale label or comment on this
issue. Otherwise, it will be closed in 15 days.As an open-source project, we rely on community contributions to
address many of the reported issues. Without a proposed fix or
active work towards a solution it is our policy to close inactive
issues. This is documented in CONTRIBUTING.rstHow to keep this issue open:
- If you are still experiencing this issue and are willing to
investigate a fix, please comment and let us know. - If you (or someone else) can propose a pull request with a
solution, that would be fantastic. - Any significant update or active discussion indicating progress
will also prevent closure.
We value your input. If you can help provide a fix, we'd be happy
to keep this issue open and support your efforts.This is documented in CONTRIBUTING.rst
https://github.com/python-gitlab/python-gitlab/blob/main/CONTRIBUTING.rst- If you are still experiencing this issue and are willing to
This issue was closed because it has been marked stale for 15 days
with no activity.This open-source project relies on community contributions, and
while we value all feedback, we have a limited capacity to address
every issue without a clear path forward.Currently, this issue hasn't received a proposed fix, and there
hasn't been recent active discussion indicating someone is planning
to work on it. To maintain a manageable backlog and focus our
efforts, we will be closing this issue for now.This doesn't mean the issue isn't valid or important. If you or
anyone else in the community is willing to investigate and propose
a solution (e.g., by submitting a pull request), please do.We believe that those who feel a bug is important enough to fix
should ideally be part of the solution. Your contributions are
highly welcome.Thank you for your understanding and potential future
contributions.This is documented in CONTRIBUTING.rst
https://github.com/python-gitlab/python-gitlab/blob/main/CONTRIBUTING.rst- locked as resolved and limited conversation to collaborators on Jul 27, 2026
Description of the problem, including code/CLI snippet
The default value of max_retries is insufficient to "clear" any rate limiting limit, for responses where the Retry-After header is missing. The default is 10, which yields a max cur_retries value of 9, which yields a maximum sleep time of ~50 seconds. At least some rate limits are computed over a 10 minute window (600 seconds) (e.g. the recently added limit on /users/:id), so ideally we'd have a default max_retries that would (cumulatively) sleep at least that long.
Expected Behavior
Default max_retries value is large enough to cause the library to sleep long enough to pass any rate limit window.
Actual Behavior
The library doesn't sleep long enough, so a exception is thrown when the rate limit is reached, despite retrying for a while.
Specifications