*.py packaging, again

[Jacek Konieczny]

On Thu, Jul 14, 2011 at 05:44:19PM +0200, Jakub Bogusz wrote:
> And one more: why byte-compiling on every target machine when the result
> should be exactly the same for given Python major version?
> 
> This argument is still valid when considering byte-compilation on rpm
> install side.

True. For CPython packaging pre-compiled *.pyc files makes sense only
for this reason.

> > 
> > We gain:
> > - ability to use the installed noarch packages by other Python 2.7
> >   implementations (like PyPy)
> 
> Are particular PyPy versions meant to be compatible with particular
> Python 2 major version?

PyPy 1.5 is compatible with Python 2.7. As far as pure-python (noarch in
PLD) *.py modules are concerned.

BTW PyPy also generates *.pyc files, in a quite different format that
those from Python 2.7 and they are most probably architecture dependant.

This gives some ugly behaviour:
> [jajcus at lolek tt]$ echo "print('module')" > mod.py
> [jajcus at lolek tt]$ echo "import mod" > test.py
> [jajcus at lolek tt]$ python test.py
> module
> [jajcus at lolek tt]$ ls -l
> total 12
> -rw-r--r-- 1 jajcus users  16 Jul 14 18:17 mod.py
> -rw-r--r-- 1 jajcus users 117 Jul 14 18:18 mod.pyc
> -rw-r--r-- 1 jajcus users  11 Jul 14 18:18 test.py
> [jajcus at lolek tt]$ file mod.pyc
> mod.pyc: python 2.7 byte-compiled
> [jajcus at lolek tt]$ pypy test.py
> module
> [jajcus at lolek tt]$ ls -l
> total 12
> -rw-r--r-- 1 jajcus users  16 Jul 14 18:17 mod.py
> -rw-r--r-- 1 jajcus users 117 Jul 14 18:18 mod.pyc
> -rw-r--r-- 1 jajcus users  11 Jul 14 18:18 test.py
> [jajcus at lolek tt]$ file mod.pyc
> mod.pyc: data

That is why the new Python 3.2 way makes sense:

> [jajcus at lolek tt]$ python3 test.py
> module
> [jajcus at lolek tt]$ ls -l
> total 16
> drwxr-xr-x 2 jajcus users 4096 Jul 14 18:19 __pycache__
> -rw-r--r-- 1 jajcus users   16 Jul 14 18:17 mod.py
> -rw-r--r-- 1 jajcus users  117 Jul 14 18:18 mod.pyc
> -rw-r--r-- 1 jajcus users   11 Jul 14 18:18 test.py
> [jajcus at lolek tt]$ ls __pycache__/
> mod.cpython-32.pyc
 
Please note that not every piece of python code will be always
installed from an RPM (e.g. developer machine)

> Even in Python 3 times noarch packages still use versioned source
> directory.

In python 3.2 this won't be needed any more. 

> Also, the question whether to package *.py refers only to noarch
> Python 2 packages, as arch-dependent ones are bound to particular
> cpython major version anyway.

True, for the 'alternative implementations' argument.
 
> The same gain can be reached by adding -source packages.

We could even create a pypy-* (with *.pyc in /usr/lib for every *.py
which would go to /usr/share in Python 2.7?) and jython-* for every
python-*?

Adding -source packages adds extra maintenance problems. Not removing 
*.py and not patching 'broken' apps, removes some maintenance problems.

> But there are still many packages not using setuptools(?) 

setuptools are dead and ever supported Python 3 AFAIK
There are distutils (part of the man Python library), distribute (fork
of setuptools, supporting Python 3 and 2to3 conversion) and on the way
are 'packaging' and distutils2…

Anyway, that doesn't change a thing in 'packaging python 2 way' or
'packaging python 3 way'

> and thus
> packaged in Python 2 way.

Most of our python3-* packages were never updated or rebuilt for the
python3 changes. I think they won't even build, as the *.pyc files
listed in %files are missing.

> Should we enforce the new packaging method for them?
> %py3_comp/%py3_ocomp macros use the old one.

Those only start the standard python compilation functions. For Python <
3.2 they will produce X.py[co] for every X.py and for Python >= 3.2 they
will produce __pycache__/X.magic.py[co] for every X.py

What is will not work in this case are not the macros, but the %files
section. And it is hard to write a spec so both python 3.1 and 3.2 are
supported.

Additionally if %py3_postclean is run (*.py removed) then, even if
%files are 'correct', the package won't work (*.pyc from __pycache__
won't be ever loaded)

Greets,
        Jacek