| FazBrowse GitHub Viewer | Trending | | Home |
| Tools: [Download Repo ZIP] [Original HTTPS Page] |
This is minor, I know. `tokenize` simply iterates over `texts`, so in addition to `list`, `tuple` is fine. The intended type hint for this is `Sequence`. I am not sure which version of Python this project targets, but judging from the other type hints in this file, I am going to assume `<3.9`. Otherwise, I would suggest importing `Sequence` from `collections.abc` instead of `typing`.
|
|
||
|
|
||
| def tokenize(texts: Union[str, List[str]], context_length: int = 77, truncate: bool = False) -> Union[torch.IntTensor, torch.LongTensor]: | ||
| def tokenize(texts: Union[str, Sequence[str]], context_length: int = 77, truncate: bool = False) -> Union[torch.IntTensor, torch.LongTensor]: |
There was a problem hiding this comment.
The docstring at line 203 should be modified too.
Sorry, something went wrong.
| Back | FazBrowse Home | New Git URL |
This is minor, I know. tokenize simply iterates over texts, so in addition to list, tuple is fine. The intended type hint for this is Sequence.
I am not sure which version of Python this project targets, but judging from the other type hints in this file, I am going to assume <3.9. Otherwise, I would suggest importing Sequence from collections.abc instead of typing.