Generic type annotation error due to old version of CloudPickle

[LachlanStuart]

The bundled version of CloudPickle (1.2.2) has an issue with pickling type annotations that subclass from Generic. This bug was fixed in version 1.4.0.

Reproduction code (should be run with Python 3.8 - earlier versions use a different codepath in CloudPickle and may not have the same issue):

from typing import TypeVar, Generic
from lithops import function_executor

T = TypeVar('T')
class MyType(Generic[T]):
    pass

def my_func(foo: MyType[int]):
    return True

fexec = function_executor()
future = fexec.call_async(my_func, [None])
fexec.get_result(future)

Raises exception:

  ...
  File "/action/lithops/worker/jobrunner.py", line 257, in run
  File "/action/lithops/worker/jobrunner.py", line 132, in _unpickle_function
  File "/action/lithops/libs/cloudpickle/cloudpickle.py", line 1323, in _make_skeleton_class
  File "/usr/local/lib/python3.8/typing.py", line 908, in __init_subclass__
TypeError: Cannot inherit from plain Generic

Full logs

For now I have a workaround, which is to add the following line to every file:

from __future__ import annotations

This flag (which won't be activated by default until Python 3.10) makes the Python parser store the type annotations as strings until they're requested by code, which means the above would pickle the string "Generic[T]" instead of an actual Generic[T] class reference.

[JosepSampe]

Thanks for reporting @LachlanStuart. Will try to update bundled cloudpickle lib

[JosepSampe]

@LachlanStuart The example you provided works with the current cloudpickle version and python 3.6. In addition, I executed some tests with Lithops and cloudpickle 1.6.0 and the example works with python 3.8, however it doesn't work with python 3.6.

In summary, I tested all cloudpickle versions and seems that from >=1.4.0 the example you provided stops working with python3.6 and starts working with Python3.8.

From >=1.4.0 I get this error message in python3.6:

AttributeError: module '__main__' has no attribute 'T'

[JosepSampe]

@gilv What should we prioritize? Python 3.6 or Python3.8?

[JosepSampe]

@LachlanStuart @gilv Well, In fact I can keep both cloudpickle versions and use 1.2.2 for python < 3.8 and 1.6.0 for python > 3.8. This way we won't have backwards compatibility issues

[gilv]

@JosepSampe I think 3.8..but i have hard times with Mac and Pytho 3.8.5(6)... but the best would be avoid backwards issues and keep both cloudpickle versions.. it's not just cloudpickle i guess... #401

[LachlanStuart]

@JosepSampe Based on that error, I suspect the issue is that CloudPickle assumes T is statically defined and can just be imported during unpickling. The issue is that the __main__ module never gets pickled fully pickled - CloudPickle only picks out the specific variables that it finds references to, and in this case it seems it doesn't understand the reference to T. However, I'd expect if T was defined in any other module, Lithops' module-pickling code would pickle the entire module that contains T, allowing it to be resolved during unpickling.

Could you try moving T and MyType into a separate file, e.g. the below?

my_types.py:

from typing import TypeVar, Generic

T = TypeVar('T')
class MyType(Generic[T]):
    pass

code.py (or python console):

from lithops import function_executor
from my_types import MyType

def my_func(foo: MyType[int]):
    return True

fexec = function_executor()
future = fexec.call_async(my_func, [None])
fexec.get_result(future)

It's actually an unrealistic scenario to have dynamically defined typevars in __main__, because when you're developing in the Python console or in a Jupyter Notebook, there's no benefit to using type annotations. If the above code works in Python 3.6, I'd suggest only using 1.6.0, because it's unlikely that someone would hit this problem normally.

[JosepSampe]

@LachlanStuart Yes, if the var definition is in the other file, lithops pickle the entire module that contains T and then it works

[JosepSampe]

@LachlanStuart Cloudpickle 1.6.0 added in #411