Python 2 desupport and removal - #2
Merged
Merged
Conversation
py4j has supported Python 3+ for some time (CI tests py3.9-py3.13)
but the codebase still carried full py2/3 dual-support machinery:
- ``from __future__`` imports in every module
- ``# -*- coding: UTF-8 -*-`` declarations
- a 130-line ``py4j.compat`` module exporting py2/3 shims
- ``sys.version_info.major < 3`` conditional templates
- ``long(x)``, ``unicode(s)``, ``basestring``, ``bytearray2``,
``bytestr``, ``isbytestr``, ``ispython3bytestr``, ``isbytearray``,
``bytetoint``, ``bytetostr``, ``strtobyte``, ``iteritems``,
``CompatThread``, ``range`` (from ``compat``)
- packaging metadata (setup.py classifiers, ``setup.cfg``'s
``universal=1`` wheel marker, tox.ini's ``envlist=py27,py34,py35``
+ ``nosetests``)
- IDE config (``.pydevproject`` files specifying ``python 2.6``)
- docs referencing py2/3 behaviour differences
This change is strictly mechanical: replaces each py2/3 alias with
its py3 equivalent inline, drops the unconditional ``from __future__``
imports, removes the encoding declarations (PEP 3120 makes UTF-8 the
default in py3), removes the ``try/except ImportError`` for
``collections.abc`` (always present on py3.3+), updates packaging
metadata to py3.9+ and drops the universal-wheel marker.
Function bodies are unchanged where possible. ``smart_decode`` calls
in source code are preserved (they will continue to work — the
function itself is just slimmed to its py3 body). ``encode_float``,
``decode_bytearray``, ``encode_bytearray``, ``get_command_part``, and
``escape_new_line`` are translated alias-by-alias with no behaviour
change.
``py4j.compat`` is kept around as a soft-deprecated shim. Its py2
branch is removed and a module docstring documents the migration path
for downstream callers, but every exported name still resolves so
external code that imports from it (``from py4j.compat import
unicode``, etc.) continues to work. The module is a candidate for
removal in a future major release.
## Files changed
Source modules:
- ``__init__.py``, ``clientserver.py``, ``finalizer.py``,
``java_collections.py``, ``java_gateway.py``, ``protocol.py``,
``signals.py`` — dropped ``from __future__``, ``# -*- coding ...``,
``from py4j.compat import (...)`` for dead names; replaced
``basestring`` → ``str``, ``long`` → ``int``, ``unicode`` → ``str``,
``bytestr`` → ``bytes``, ``bytearray2`` → ``bytes``, ``CompatThread``
→ ``threading.Thread``, ``items(d)`` → ``list(d.items())``,
``iteritems(self)`` → ``self.items()``, ``isbytestr`` /
``ispython3bytestr`` / ``isbytearray`` → direct ``isinstance(...)``,
``from py4j.compat import Queue`` → ``from queue import Queue``.
- ``java_collections.py`` — collapsed the ``try/except ImportError``
for ``collections.abc``; dropped the ``sys.version_info.major < 3``
ternary on ``__EMPTY_SET`` / ``__SET_TEMPLATE``.
- ``protocol.py`` — slimmed ``smart_decode`` body to py3 types;
``encode_bytearray`` / ``decode_bytearray`` / OUTPUT_CONVERTER
``LONG_TYPE`` lambda use native ``bytes`` / ``int``;
``get_command_part`` uses ``isinstance(..., str/int/bytes/
bytearray)`` directly.
Test modules:
- 17 test files — dropped ``from __future__``, ``# -*- coding ...``,
``from py4j.compat import`` of py2-only names. ``long(N)`` → ``N``;
``bytearray2`` → ``bytes``; ``isbytearray(a) or
ispython3bytestr(a)`` → ``isinstance(a, (bytearray, bytes))``;
``unicode(i)`` → ``str(i)``.
- ``java_gateway_test.py`` — dropped two
``if sys.version_info < (3,):`` branches (py2-only sleep paths;
the py3 ``proc.wait(5)`` branch is the only one that ran on
supported Pythons).
- New ``protocol_test.py`` (30 pure-protocol unit tests, no JVM
required): round-trip ``encode_bytearray``/``decode_bytearray``
for both ``bytes`` and ``bytearray``; ``escape_new_line``
str-passthrough and round-trip with ``unescape_new_line``;
``encode_float`` precision / inf / -inf / NaN; each branch of
``get_command_part`` including JAVA_MAX_INT boundary and bool-vs-int
dispatch; ``smart_decode`` semantics on str / bytes / non-string;
``py4j.compat`` back-compat (every export name still resolves to
its py3 equivalent, including ``CompatThread is threading.Thread``
and ``Queue is queue.Queue``).
``py4j.compat``:
- Slimmed: removed the ``if version_info.major < 3:`` branch
entirely; kept the py3 form as the only definition. Module-level
docstring documents the migration path. ``hasattr2``,
``CompatThread``, ``Queue``, ``Empty`` and every alias still
resolve so downstream imports continue to work.
Packaging:
- ``setup.py``: removed ``"Programming Language :: Python :: 2"`` and
``"... :: 3.8"`` (EOL) classifiers. Added ``... :: 3 :: Only`` and
3.9-3.13. Set ``python_requires=">=3.9"`` to match CI. Removed
``distutils`` fallback and 2to3-era comments.
- ``setup.cfg``: removed — only contained ``[bdist_wheel]
universal=1``, which is a py2/3 universal-wheel marker that
produces an incorrectly-tagged wheel on a py3-only project.
- ``tox.ini``: was ``envlist=py27,py34,py35`` calling ``nosetests``.
Now ``envlist=py{39,310,311,312,313}`` calling ``pytest``, matching
what CI runs. ``nose.cfg`` deleted alongside.
IDE config:
- ``py4j-python/.pydevproject`` and ``py4j-web/.pydevproject``:
``python 2.6`` → ``python 3.9``.
Docs:
- ``py4j-web/advanced_topics.rst``: dropped the parenthetical
``(Python 2.x) or bytes (Python 3.x)`` qualifier on the byte-array
return-type description; now reads simply ``bytes``.
## Validation
Local pytest on macOS arm64 + JDK 17:
**205 passed, 104 deselected, 0 failed** (skipping TLS / perf-framework
/ memory-leak / multithread suites that need extra setup, matching
the main CI workflow's filter).
Co-authored-by: Isaac
Merging this PR will improve performance by 20.22%
Performance Changes
Tip Curious why this is faster? Comment Comparing Footnotes
|
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Python 2 desupport / removal.
py4j has supported Python 3+ for some time (CI tests py3.9 – py3.13) but the codebase still carried the full py2 / py3 dual-support machinery:
from __future__imports in every module, a 130-linepy4j.compatmodule exporting py2/3 shims,# -*- coding: UTF-8 -*-declarations,sys.version_infoconditionals, thelong(x)/unicode(s)/basestring/CompatThreadaliases, packaging metadata (setup.cfg'suniversal=1,tox.ini'senvlist=py27,py34,py35+nosetests), IDE config (.pydevprojectfiles specifyingpython 2.6), and a couple of docs references. None of that does anything on Python 3.This change is strictly mechanical — function bodies are unchanged, no perf optimizations included. Each py2 / py3 alias is replaced inline with its py3 equivalent.
Scope
from __future__,# -*- coding ..., deadfrom py4j.compat import (...)lines. Replacedbasestring→str,long→int,unicode→str,bytestr→bytes,bytearray2→bytes,CompatThread→threading.Thread,items(d)→list(d.items()),iteritems(self)→self.items(),isbytestr/ispython3bytestr/isbytearray→ directisinstance(...),from py4j.compat import Queue→from queue import Queue.java_collections.pytry/except ImportErrorforcollections.abc(always present on py3.3+). Droppedsys.version_info.major < 3ternary on__EMPTY_SET/__SET_TEMPLATE.protocol.pysmart_decodebody to py3 types (isinstance(s, str)instead ofisinstance(s, unicode), etc.).encode_bytearray/decode_bytearray/ OUTPUT_CONVERTERLONG_TYPElambda use nativebytes/int.get_command_partusesisinstance(..., str/int/bytes/bytearray)directly. Allsmart_decode(...)call sites in source are preserved — the function itself still exists and behaves identically.py4j.compatif version_info.major < 3:branch entirely; kept the py3 form as the only definition. Module-level docstring documents the migration path. Every export still resolves (unicode,basestring,long,range,bytestr,bytearray2,unichr,iteritems,items,isbytestr,ispython3bytestr,isbytearray,bytetoint,bytetostr,strtobyte,Queue,Empty,hasattr2,CompatThread,version_info,tobytestr) so downstream code that imports from it (from py4j.compat import unicode, etc.) continues to work. The module is a candidate for removal in a future major release.long(N)→N;bytearray2→bytes;isbytearray(a) or ispython3bytestr(a)→isinstance(a, (bytearray, bytes));unicode(i)→str(i). Dropped twoif sys.version_info < (3,):branches injava_gateway_test.py(py2-only sleep paths; the py3proc.wait(5)branch is the only one that ran on supported Pythons). Newprotocol_test.py(30 unit tests, no JVM required) pinning thebytes/strboundary and thepy4j.compatback-compat surface.setup.py"Programming Language :: Python :: 2"and"... :: 3.8"(EOL) classifiers. Added... :: 3 :: Onlyand 3.9 – 3.13. Setpython_requires=">=3.9"to match CI. Removeddistutilsfallback + 2to3-era comments.setup.cfg[bdist_wheel] universal=1, the py2/3 universal-wheel marker, which produces an incorrectly-tagged wheel on a py3-only project.tox.inienvlist=py27,py34,py35callingnosetests. Nowenvlist=py{39,310,311,312,313}callingpytest, matching CI.nose.cfgdeleted alongside..pydevproject(py4j-python and py4j-web)python 2.6→python 3.9.py4j-web/advanced_topics.rst(Python 2.x) or bytes (Python 3.x)qualifier on the byte-array return-type description; now reads simplybytes.Java side
No changes. The Java side speaks pure UTF-8 over the wire and has zero Python-version-specific code (confirmed by
grep -rn -i "python 2\|python2\|py2" py4j-java/src/). The clean separation between protocol-level and Python-version-level is one of py4j's good properties — the JVM side is oblivious to which Python is on the other end.Validation
Local pytest on macOS arm64 + JDK 17: 205 passed, 104 deselected, 0 failed (skipping TLS / perf-framework / memory-leak / multithread suites that need extra setup, matching the main CI workflow's filter).
The CodSpeed workflow will pick this PR up automatically (path filter matches
py4j-python/src/py4j/**/*.py) and post per-scenario perf deltas vs master. Expected deltas: near-zero, since this is mechanical alias substitution, not optimization.Co-authored-by: Isaac