gh-140544: Always assume that thread locals are available

[ZeroIntensity]

Python has required thread local support since 3.12. By assuming that thread locals are always supported, we can improve the performance of third-party extensions by allowing them to access the attached thread and interpreter states directly.

cc @cdce8p

• Issue: Store current interpreter as a thread local variable #140544

[cdce8p]

Hmm, I'm still getting the same error I mentioned earlier.

  In file included from /.../cpython/Include/internal/pycore_object.h:12: (diff)
  In file included from /.../cpython/Include/internal/pycore_gc.h:12: (diff)
  /.../cpython/Include/internal/pycore_pystate.h:93:8: error: unknown type name '_Py_thread_local' (diff)
  extern _Py_thread_local PyThreadState *_Py_tss_tstate; (diff)
         ^ (diff)
  /.../cpython/Include/internal/pycore_pystate.h:93:38: error: expected ';' after top level declarator (diff)
  extern _Py_thread_local PyThreadState *_Py_tss_tstate; (diff)
                                       ^ (diff)
                                       ; (diff)
  /.../cpython/Include/internal/pycore_pystate.h:94:8: error: unknown type name '_Py_thread_local' (diff)
  extern _Py_thread_local PyInterpreterState *_Py_tss_interp; (diff)
         ^ (diff)
  /.../cpython/Include/internal/pycore_pystate.h:94:43: error: expected ';' after top level declarator (diff)
  extern _Py_thread_local PyInterpreterState *_Py_tss_interp; (diff)
                                            ^ (diff)
                                            ; (diff)
  /.../cpython/Include/internal/pycore_pystate.h:119:12: error: use of undeclared identifier '_Py_tss_tstate' (diff)
      return _Py_tss_tstate; (diff)
             ^ (diff)
  /.../cpython/Include/internal/pycore_pystate.h:213:12: error: use of undeclared identifier '_Py_tss_interp' (diff)
      return _Py_tss_interp; (diff)
             ^ (diff)
  6 errors generated. (diff)

Somehow the # ifdef Py_BUILD_CORE check fails, even though mypyc defines it.
https://github.com/python/mypy/blob/v1.18.2/mypyc/lib-rt/pythonsupport.h#L16-L18

[cdce8p]

Do I need to pass Py_BUILD_CORE through to my local cpython build? This is my current build setup

make clean
./configure --with-openssl=$(brew --prefix openssl@3)
make -j4 -s

[ZeroIntensity]

Hm, you're defining Py_BUILD_CORE after Python.h has been included, so pyport.h isn't seeing it. I'm not sure if that's something we intended to support.

[cdce8p]

Hm, you're defining Py_BUILD_CORE after Python.h has been included, so pyport.h isn't seeing it. I'm not sure if that's something we intended to support.

That hasn't been an issue so far. The check here works fine / if I remove Py_BUILD_CORE from mypyc I get the error as expected.

cpython/Include/internal/pycore_mimalloc.h

Lines 4 to 6 in f5394c2

	#ifndef Py_BUILD_CORE
	# error "this header requires Py_BUILD_CORE define"
	#endif

[ZeroIntensity]

I just pushed a new commit, where _Py_thread_local is moved to pycore_pystate.h. Can you test that?

[cdce8p]

I just pushed a new commit, where _Py_thread_local is moved to pycore_pystate.h. Can you test that?

This works now.

--

Hm, you're defining Py_BUILD_CORE after Python.h has been included, so pyport.h isn't seeing it. I'm not sure if that's something we intended to support.

I missed that mypyc is defining Py_BUILD_CORE before including any internal/** headers. Though that's much simpler than doing the same for Python.h which is included nearly everywhere.

[kumaraditya303]

@cdce8p Would you please check that mypyc still works after the new changes?

[cdce8p]

@cdce8p Would you please check that mypyc still works after the new changes?

Just ran the whole test suite again. Looks good.