Repository navigation
projects.all() returns only 20 projects #93
Description
Activity
There's some inconsistency in the listing methods. The managers' list()method can return all the items with all=True, so this might do what you're looking for:
all_projects = gl.projects.list(all=True)
This is not supported on other listing method (all, search, owned) for the moment.
gl.projects.list(all=True) returns 10 projects only. gl.projects.all() returns 20 projects.
If you look at the code: projects.all() calls custom_list, which makes a single API request.
projects.list call gl.list(), which has some code here but that stops after 10 results because there's no 'next' field in the response. The returned response is:
{'last': {'url': 'https://git.cs.vt.edu/api/v3/projects?page=1&per_page=0', 'rel': 'last'}, 'first': {'url': 'https://git.cs.vt.edu/api/v3/projects?page=1&per_page=0', 'rel': 'first'}}which does not have 'next' property.
How do I iterate to return all projects in the new code base?
ps: gl.projects.all() receives this API response:
{'first': {'url': 'https://git.cs.vt.edu/api/v3/projects/all?page=1&per_page=0', 'rel': 'first'}, 'last': {'url': 'https://git.cs.vt.edu/api/v3/projects/all?page=38&per_page=0', 'rel': 'last'}, 'next': {'url': 'https://git.cs.vt.edu/api/v3/projects/all?page=2&per_page=0', 'rel': 'next'}}but then fails to process the 'next' field as far as I can see.
How is this supposed to work?
Oh, and I know why. projects.all() issues a request for /projects/all, but then doesn't process the 'next' field in the r.links. projects.list(all=True) issues a request for /projects, in which case the API response doesn't have a 'next' field.
OK, I think I get what you need. You can still iterate using the page and per_page arguments.
l = []; page = 0 while True: next_list = gl.projects.all(per_page=50, page=page) if not next_list: break l.extend(next_list) page += 1
I'll implement the all argument support in _custom_list to ease this as soon as I can.
Thanks.
Reacted by Dragonlin, Sergey and David FischerI just run into this limitation and had to use code from #93 (comment) as a workaround. So in my opinion this issue should not be closed. Sorry, I mixed things up. Actually gl.projects.all(all=True) worked ..probably as my Gitlab timeouted on this request (we have A LOT of projects) and changing it's settings didn't seem to help. So the above code was a workaround for my timeouting issue.
- added a commit that references this issue on Jul 21, 2019
- locked as resolved and limited conversation to collaborators on Nov 29, 2021
In issue #53 I had asked how to list all projects. The answer was pagination.
I am now trying the same with the current version of python-gitlab. This time, I am using gl.projects.all() as recommended in the deprecation warning I received with my old code.
However, it seems the code isn't doing any pagination (is it no longer necessary? In Issue #53, you said there's no way to avoid it?
As a result, I'm only getting 20 projects returned.
What's the way to iterate over all projects using python-gitlab?