Hey @AyanSinhaMahapatra I might need a hand here.

I've I tested a lot of approaches but I can't seem to get the tests to pass in Azure.

I cannot reproduce the error in any of the environments that I have (arm64 mac and amd64 linux container).
To pinpoint the bug I've been going line by line through the configure script doing and I'm not seeing what is happening in the pipelines.

If I understand correctly it fails to configure the environment with the dev dependencies. (basically the ./configure --clean && ./configure --dev is failing).
I have updated the requirements.txt and the requirements-dev.txt using the scripts in the etc/scripts. Nothing changes. The tests pass when I run them locally though.

I'm kinda lost. Can you help me out somehow?

TLDR click has dropped support for python3.9 even though the EOL for this version is in 6 months. See:

• drop end of life python versions pallets/click#2893
• No support for python3.9 even though this is not EOL pallets/click#2975

@piiq Thanks for looking into this intially.
I could reproduce the error locally, if you force the configure script to install using python3.9 instead of whatever is the default python3 executable.

The error is happening because https://github.com/pallets/click/blob/main/pyproject.toml#L16 where they are enforcing python3.10 and above, which is frankly problematic, since this python version is not EOL for another half a year and they are breaking core functionality simultaneously.

It took me quite some time to get to the cause of this issue since this only happens on python3.9 and the pip error message does not help at all in debugging this (opening an issue on pip for this: pypa/pip#13422):

INFO: pip is looking at multiple versions of commoncode to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install None because these package versions have conflicting dependencies.

The conflict is caused by:
    commoncode 0.0.0 depends on click>=8.2.0
    The user requested (constraint) click==8.2.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip to attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Seems like we cannot possibly enforce click >= 8.2.0 and add this to scancode-toolkit because that would mean dropping support for installing scancode in python3.9 as this would bring the same dependency conflict there.

I'm suggesting the following instead for now as an alternative:

• let's put an upper bound on click temporarily so that we only install click <= 8.1.8
• let's keep this PR open, we can merge this after python3.9 is EOL in 6 months, and we can use click >= 8.2.0 exclusively
• let the click maintainers know that this is hampering usage of click as a library

Thanks a lot for the debugging @AyanSinhaMahapatra I should have tried doing this myself to be honest. If I would have tried launching it in 3.9 I would see the error fairly early on in my process. Well - I'll treat it as a learning opportunity.

I've updated this PR to remove python 3.9 from the test matrix and created an new PR that caps click to <8.2
#87

I can't convert this one to draft or add a "do not merge" label because I don't have permissions, but let me know if you'd like me to add anything here while this PR is on hold waiting for 3.9 EOL.

No worries, @piiq this was a head scratched indeed.

Thank you for your patience with this, much appreciated.

Click maintainers have actually responded at pallets/click#2975 (comment) pointing out that a better fix might be to extend the class and add a property which we can deprecate later, which probably better than dropping support for many older click versions, since people might put constraints/depend on those. Tbh, I missed this too, so lots of learning for me too there :P

Could you modify this PR to have something like:

class CompatProgressBar(click.ProgressBar):
    # TODO Remove when dropping support for Python 3.9 or Click 8.1.
    @property
    def is_hidden(self) -> bool:
        return self.hidden

    @is_hidden.setter
    def is_hidden(self, value: bool) -> None:
        self.hidden = value

as pointed out in the issue above and revert the tests/CI and click dependency constraints accordingly?

@piiq I'm applying your commits and some modifications and adding the changes with https://github.com/aboutcode-org/commoncode/tree/bugfix/progressbar-hidden-propetry since we need this for a new scancode release.

Thanks++ for all the help here 🚀