Generic, higher kinded function not sharing Callable’s limitations

[flying-sheep]

Documentation

I can define the following decorator:

P = ParamSpec('P')
R = TypeVar('R', covariant=True)

def file_cache(fn: Callable[Concatenate[Path, P], R]) -> Callable[P, R]:
    path: Path = ...
    def wrapper(*args: P.args, **kw: P.kwargs) -> R:
        if path.is_file():
            return read(path)
        return fn(path, *args, **kw)
    return wrapper

But Callable is limited to kwargs-less functions, so mypy complains when I apply the decorator to a function, and try to specify the keyword argument:

@file_cache
def get_thing(path: Path, *, some_arg: int) -> X:
    x = calculate_x()
    path.write_bytes(x.serialize())
    return x

if __name__ == '__main__':
    get_thing(some_arg=1)

error: Unexpected keyword argument "some_arg" for "get_thing"

In an attempt to circumvent the problem, I tried to define the following, but mypy doesn’t seem to understand it:

P = ParamSpec('P')
R = TypeVar('R', covariant=True)


class Function(Protocol[P, R]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

And will complain that the decorated function was a Callable[[Path, DefaultNamedArg(int, 'some_arg')], X, not a Function[<nothing>, <nothing>].

If I made a mistake in defining Function, please help.

If it looks correct: a) why doesn’t it exist / why is Callable so limited? and b) mypy should understand my definition.

[erictraut]

You are correct that Callable is limited to describing positional-only parameters, so you are correct to use a callback protocol here.

Here's my attempt to create a complete stand-alone sample from your description. Is this correct?

from typing import Callable, Concatenate, ParamSpec, Protocol, TypeVar

P = ParamSpec("P")
R = TypeVar("R", covariant=True)

class Path:
    ...

class Function(Protocol[P, R]):
    def __call__(self, *args: P.args, **kwargs: P.kwargs) -> R:
        ...

def file_cache(fn: Function[Concatenate[Path, P], R]) -> Callable[P, R]:
    def wrapper(*args: P.args, **kw: P.kwargs) -> R:
        return fn(Path(), *args, **kw)

    return wrapper

@file_cache
def get_thing(path: Path, *, some_arg: int) -> int:
    return 1

For this sample, mypy emits two errors:
error: Argument 1 to "__call__" of "Function" has incompatible type "Path"; expected "[Path, **P.args]"
error: Argument 1 to "file_cache" has incompatible type "Callable[[Path, NamedArg(int, 'some_arg')], int]"; expected "Function[<nothing>, <nothing>]"

Both of these errors look like false positives — probably two separate bugs. For comparison, pyright doesn't emit any errors for this snippet.

[flying-sheep]

Here's my attempt to create a complete stand-alone sample from your description. Is this correct?

Almost, thank you! I should have made a working example from the start. I think your consolidation has one little mistake, but that actually doesn’t change anything in the errors encountered.

-def file_cache(fn: Function[Concatenate[Path, P], R]) -> Callable[P, R]:
+def file_cache(fn: Function[Concatenate[Path, P], R]) -> Function[P, R]:

ParamSpec support just seems to be really buggy and needs a lot more tests, see e.g. #12595

Also very weird that mypy chooses to express the type as Callable[[Path, NamedArg(int, 'some_arg')], int]: Seems like mypy itself is trying to get around the arbitrary limitation of Callable and trips over its own feet.