FYI @renato2099 getting the python based providers ended up being a blocking issue for some of my work so I took a stab at implementing it. Please tell me what you think if you have some time.

Hi @timsaucer ,

I am sorry I wasn't able to complete this in time, but I had it still on my radar. I pushed my version yesterday after the holidays :) #1137 and after a quick look at your PR, it seems they are relatively similar.
One thing is that you declared a new "catalog" module whereas I introduced a rust wrapper class for the python providers ... but it is very minor difference tbh , anyway let me take a closer look after work today though and provide some more useful input here

Hi @timsaucer ,

thanks for pushing on this, I think the main difference is that you are extending the PyCatalog whereas I am introducing a new PyCatalogProvider. Both approaches have their pros/cons, imho keeping the separation between catalogs and catalog_providers might make things clearer at the interface level.

That said I don't have a feel what datafusion-python users might find better, i.e., more rigid+"clearer" APIs vs more fluid+a bit less clearer API by accepting different object types (catalogs and catalog_providers when registering providers). Maybe it is more pythonic to accept different object types?

PR looks good to me overall though, thanks again! and sorry for the delay on my side :(