From marcin.rybak at gmail.com  Wed Jul  6 13:13:56 2011
From: marcin.rybak at gmail.com (Marcin Rybak)
Date: Wed, 6 Jul 2011 13:13:56 +0200
Subject: PLD AC, bind-9.4.3.P5
Message-ID: <CAKow9E2v_82vpXKMu2khrOJi3OV6PySXaSdhzVwkcMXhy3nZeA@mail.gmail.com>

what is the reason that in PLD AC we have bind-9.4.3.P5. I've just checked,
and bind-9.8.0.P4 build and works preety good :)

br,
---
Marcin Rybak
http://marcinrybak.com


From arekm at maven.pl  Tue Jul 12 15:42:02 2011
From: arekm at maven.pl (Arkadiusz Miskiewicz)
Date: Tue, 12 Jul 2011 15:42:02 +0200
Subject: deny some files in apache by default
Message-ID: <201107121542.02249.arekm@maven.pl>


What do you think about adding to our apache default config:

[arekm at t400 ~/rpm/packages/apache]$ cvs diff -u apache-common.conf
Index: apache-common.conf
===================================================================
RCS file: /cvsroot/packages/apache/apache-common.conf,v
retrieving revision 1.9
diff -u -u -r1.9 apache-common.conf
--- apache-common.conf  9 Jan 2006 11:24:05 -0000       1.9
+++ apache-common.conf  12 Jul 2011 13:40:56 -0000
@@ -19,6 +19,12 @@
                Order deny,allow
                Deny from all
        </IfModule>
+       <IfModule mod_authz_host.c>
+               <FilesMatch "(~|#|^\.\?\?.*|\.inc)$">
+                       Order deny,allow
+                       Deny from all
+               </FilesMatch>
+       </IfModule>
 </Directory>

 #

?

-- 
Arkadiusz Mi?kiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/


From glen at delfi.ee  Wed Jul 13 01:02:06 2011
From: glen at delfi.ee (=?UTF-8?B?RWxhbiBSdXVzYW3DpGU=?=)
Date: Wed, 13 Jul 2011 02:02:06 +0300
Subject: deny some files in apache by default
In-Reply-To: <201107121542.02249.arekm@maven.pl>
References: <201107121542.02249.arekm@maven.pl>
Message-ID: <4E1CD26E.8020402@delfi.ee>

On 07/12/2011 04:42 PM, Arkadiusz Miskiewicz wrote:
>
> What do you think about adding to our apache default config:
>
> [arekm at t400 ~/rpm/packages/apache]$ cvs diff -u apache-common.conf
> Index: apache-common.conf
> ===================================================================
> RCS file: /cvsroot/packages/apache/apache-common.conf,v
> retrieving revision 1.9
> diff -u -u -r1.9 apache-common.conf
> --- apache-common.conf  9 Jan 2006 11:24:05 -0000       1.9
> +++ apache-common.conf  12 Jul 2011 13:40:56 -0000
> @@ -19,6 +19,12 @@
>                  Order deny,allow
>                  Deny from all
>          </IfModule>
> +<IfModule mod_authz_host.c>

afaik module name is wrong

> +<FilesMatch "(~|#|^\.\?\?.*|\.inc)$">
backup files ok, maybe add more, like .BAK and .bak?
".inc" file, may cause conflicts
and "^\.\?\?.*" is what? '?' needs to be esacaped and '*' not in apache?

perhaps you wanted to say just dot-files?:
<FilesMatch "(^\..*|~|#|\.inc)$">

also, perhaps add to the list files that (our) lighttpd denies (vcs 
control files):

## deny access the file-extensions
#
# ~    is for backupfiles from vi, emacs, joe, ...
# .inc is often used for code includes which should in general not be part
#      of the document-root
# *,v and *,t - CVS files
url.access-deny = ( "~", ".inc", ",v", ",t" )

# forbid access to files inside CVS or RCS dirs
$HTTP["url"] =~ "/(?:CVS|RCS)/" {
         url.access-deny = ("")
}


> +                       Order deny,allow
> +                       Deny from all
> +</FilesMatch>
> +</IfModule>
>   </Directory>
>
>   #
>
> ?
>


-- 
glen


From arekm at maven.pl  Wed Jul 13 08:06:11 2011
From: arekm at maven.pl (Arkadiusz Miskiewicz)
Date: Wed, 13 Jul 2011 08:06:11 +0200
Subject: deny some files in apache by default
In-Reply-To: <4E1CD26E.8020402@delfi.ee>
References: <201107121542.02249.arekm@maven.pl> <4E1CD26E.8020402@delfi.ee>
Message-ID: <201107130806.12182.arekm@maven.pl>

On Wednesday 13 of July 2011, Elan Ruusam?e wrote:
> On 07/12/2011 04:42 PM, Arkadiusz Miskiewicz wrote:
> > What do you think about adding to our apache default config:
> > 
> > [arekm at t400 ~/rpm/packages/apache]$ cvs diff -u apache-common.conf
> > Index: apache-common.conf
> > ===================================================================
> > RCS file: /cvsroot/packages/apache/apache-common.conf,v
> > retrieving revision 1.9
> > diff -u -u -r1.9 apache-common.conf
> > --- apache-common.conf  9 Jan 2006 11:24:05 -0000       1.9
> > +++ apache-common.conf  12 Jul 2011 13:40:56 -0000
> > @@ -19,6 +19,12 @@
> > 
> >                  Order deny,allow
> >                  Deny from all
> >          
> >          </IfModule>
> > 
> > +<IfModule mod_authz_host.c>
> 
> afaik module name is wrong

Then we have it wrong in the same config, few lines above.

> > +<FilesMatch "(~|#|^\.\?\?.*|\.inc)$">
> 
> backup files ok, maybe add more, like .BAK and .bak?
> ".inc" file, may cause conflicts
> and "^\.\?\?.*" is what? '?' needs to be esacaped and '*' not in apache?

Any file starting with .??, not sure which program makes such backup files.

> 
> perhaps you wanted to say just dot-files?:
> <FilesMatch "(^\..*|~|#|\.inc)$">
> 
> also, perhaps add to the list files that (our) lighttpd denies (vcs
> control files):

Fine for me.

-- 
Arkadiusz Mi?kiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/


From glen at delfi.ee  Wed Jul 13 11:22:02 2011
From: glen at delfi.ee (=?UTF-8?B?RWxhbiBSdXVzYW3DpGU=?=)
Date: Wed, 13 Jul 2011 12:22:02 +0300
Subject: deny some files in apache by default
In-Reply-To: <201107130806.12182.arekm@maven.pl>
References: <201107121542.02249.arekm@maven.pl> <4E1CD26E.8020402@delfi.ee>
	<201107130806.12182.arekm@maven.pl>
Message-ID: <4E1D63BA.5010302@delfi.ee>

On 07/13/2011 09:06 AM, Arkadiusz Miskiewicz wrote:
>>> +<IfModule mod_authz_host.c>
>>
>> afaik module name is wrong
>
> Then we have it wrong in the same config, few lines above.

sorry, my mad, it is correct

https://httpd.apache.org/docs/2.2/mod/mod_authz_host.html#deny


-- 
glen


From jajcus at jajcus.net  Wed Jul 13 13:05:05 2011
From: jajcus at jajcus.net (Jacek Konieczny)
Date: Wed, 13 Jul 2011 13:05:05 +0200
Subject: *.py packaging, again
Message-ID: <20110713110505.GA32005@jajo.eggsoft>

Hello,

I have just packaged pypy, another Python implementation. It is
python-2.7 compatible, so should work with the 'noarch' python
packages for python 2.7 (just ask pypy to look into
/usr/share/python2.7/site-packages too). It would work if the *.py files
were provided and not only the py2.7 pyc files..

IMHO it is another reason to start including *.py files in the packages,
making 'pypy-*' with pypy-compiled files for every 'python-*' package,
just for the few who would ever use pypy doesn't make much sense to me.

And the Python 3.2 __pycache__ directory starts to make more sense when
another python implementation comes to play?

We could have one /usr/share/python/site-packages, which contents could
be linked to compatible python implementation's site-packages dir (like
it works for browser plugins). Some packages could even work for both
python2 and python3, others just for python2 and pypy, etc.

The __pycache__ could be populated by the %post scripts. python2.7 and
pypy could be patched to use that as 3.2 does, but we could also keep
python2.7 *.py[co] files the old way.

Greets,
        Jacek


From jajcus at jajcus.net  Wed Jul 13 14:59:34 2011
From: jajcus at jajcus.net (Jacek Konieczny)
Date: Wed, 13 Jul 2011 14:59:34 +0200
Subject: ERRORS: pypy.spec
In-Reply-To: <pld-builder.1310558248.523258.8814@ymir-builder>
References: <182383f6-88fb-4388-b95f-e7e39d0a984b@pld.src.builder>
	<pld-builder.1310558248.523258.8814@ymir-builder>
Message-ID: <20110713125934.GB32005@jajo.eggsoft>

On Wed, Jul 13, 2011 at 11:57:28AM +0000, PLD th-x86_64 builder wrote:
> [translation:info] Compiling c source...
> [platform:execute] make -j 3 in /tmp/B.0f0703/usession-unknown-0/testing_1
> Killed 
> error: Bad exit status from /tmp/B.0f0703/rpm-tmp.25739 (%build)

Why, oh, why?

OOM?

>From the PyPy build documentation:

> Enjoy Mandelbrot :-) It takes on the order of half an hour to finish the
> translation, and 2 GB of RAM on a 32-bit system and 4 GB on 64-bit
> systems. (Do not start a translation on a machine with insufficient RAM!
> It will just swap forever.)

(it could be less for the (slower, made with CPython) bootstrap build, though)

Is such a monster buildable at our builders? ;)

Greets,
        Jacek


From qboosh at pld-linux.org  Wed Jul 13 15:07:07 2011
From: qboosh at pld-linux.org (Jakub Bogusz)
Date: Wed, 13 Jul 2011 15:07:07 +0200
Subject: *.py packaging, again
In-Reply-To: <20110713110505.GA32005@jajo.eggsoft>
References: <20110713110505.GA32005@jajo.eggsoft>
Message-ID: <20110713130707.GA5995@mail>

On Wed, Jul 13, 2011 at 01:05:05PM +0200, Jacek Konieczny wrote:
> Hello,
> 
> I have just packaged pypy, another Python implementation. It is
> python-2.7 compatible, so should work with the 'noarch' python
> packages for python 2.7 (just ask pypy to look into
> /usr/share/python2.7/site-packages too). It would work if the *.py files
> were provided and not only the py2.7 pyc files..
> 
> IMHO it is another reason to start including *.py files in the packages,
> making 'pypy-*' with pypy-compiled files for every 'python-*' package,
> just for the few who would ever use pypy doesn't make much sense to me.
> 
> And the Python 3.2 __pycache__ directory starts to make more sense when
> another python implementation comes to play???
> 
> We could have one /usr/share/python/site-packages, which contents could
> be linked to compatible python implementation's site-packages dir (like
> it works for browser plugins). Some packages could even work for both
> python2 and python3, others just for python2 and pypy, etc.

"some" doesn't make good base for general solution...

> The __pycache__ could be populated by the %post scripts. python2.7 and
> pypy could be patched to use that as 3.2 does, but we could also keep
> python2.7 *.py[co] files the old way.

I don't like the idea of __pycache__ not managed by rpm (or not
maintained with rpm database in case of more robust post scripts)
because of at least:
- possible inconsistences (like leftovers in case of upgrade failures;
  in such case it would be even possible that after installing another
  version of package with *.py files having older date than pycache,
  pycache won't be rebuilt)
- more work needed to find package "owning" __pycache__/* files
- one more place to hide some malicious code not detectable by rpm -Va


-- 
Jakub Bogusz    http://qboosh.pl/


From n3npq at mac.com  Wed Jul 13 15:13:06 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Wed, 13 Jul 2011 09:13:06 -0400
Subject: *.py packaging, again
In-Reply-To: <20110713130707.GA5995@mail>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
Message-ID: <E78E5E7B-C25F-4F62-BEE4-3DD41F62363D@mac.com>


On Jul 13, 2011, at 9:07 AM, Jakub Bogusz wrote:

> 
> I don't like the idea of __pycache__ not managed by rpm (or not

We likely disagree on the details, perhaps sharply, of
	__pycache__ managed by rpm
but not on the general goal.

> maintained with rpm database in case of more robust post scripts)
> because of at least:
> - possible inconsistences (like leftovers in case of upgrade failures;
>  in such case it would be even possible that after installing another
>  version of package with *.py files having older date than pycache,
>  pycache won't be rebuilt)
> - more work needed to find package "owning" __pycache__/* files
> - one more place to hide some malicious code not detectable by rpm -Va
> 

If you can state positively (the above comments are all negative problem avoidance,
not positive feature seeking) what you would like to see for RPM management
of __pycache__, I'll try to get some implementation together in RPM.

If the chosen implementation is %post scrip tie, well, so be it. Some
attempt to manage __pycache__ is better than none. I do think much
better than %post scripting needs to be attempted: the edit to
add functionality in 10's to 100's of *.spec files converges
to a solution rather slowly.

hth

73 de Jeff




From jajcus at jajcus.net  Thu Jul 14 09:44:01 2011
From: jajcus at jajcus.net (Jacek Konieczny)
Date: Thu, 14 Jul 2011 09:44:01 +0200
Subject: *.py packaging, again
In-Reply-To: <20110713130707.GA5995@mail>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
Message-ID: <20110714074401.GB6660@jajo.eggsoft>

On Wed, Jul 13, 2011 at 03:07:07PM +0200, Jakub Bogusz wrote:
> I don't like the idea of __pycache__ not managed by rpm (or not
> maintained with rpm database in case of more robust post scripts)
> because of at least:
> - possible inconsistences (like leftovers in case of upgrade failures;
>   in such case it would be even possible that after installing another
>   version of package with *.py files having older date than pycache,
>   pycache won't be rebuilt)
> - more work needed to find package "owning" __pycache__/* files
> - one more place to hide some malicious code not detectable by rpm -Va

Ok, you may be right. But what than? We don't even have an idea how
should the proper rpm solution work and no volunteers to design and code
it. At the same time Python 3.2 is waiting for being included in PLD,
Python 3.3 is on its way and PyPy could be useful too, provided
compatible packages are available.

Maybe we should just start packing the *.py files together with the
compiled files for the python version they are built for? Yes, I know I
can start the old 'we don't package sources' flame war again, but '*.py'
files are not sources, they are the executable code. '*.pyc' and '*.pyo'
are just 'optimized versions' of those. And for Python 3.2 we have no
option ? I have already started adding both *.py and __pycache__/* to
the python3 packages I have recently done.

For Python 2 (python-* packages), when including *.py:

We gain:
- ability to use the installed noarch packages by other Python 2.7
  implementations (like PyPy)
- easier debugging 
- no need to patch apps that look only for python code in '*.py' only

We lose:
- a little bit of the disk space
- some 'purity' some people see in not distributing 'sources'

For Python >= 3.2 (python3-* packages):

We gain:
- compatibility with the language

We lose:
- simplicity of the %files section (listing extra directories may make
  things a bit more complicated)
- some 'purity' some people see in not distributing 'sources'

In case some automated mechanism is provided, in the future, for keeping
the compiled python code files in the RPM database without listing them
in the spec files, we loose nothing by having the files there already
(by explicitly listing them in the specs).

Greets,
        Jacek


From patrys at pld-linux.org  Thu Jul 14 10:35:17 2011
From: patrys at pld-linux.org (Patryk Zawadzki)
Date: Thu, 14 Jul 2011 10:35:17 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714074401.GB6660@jajo.eggsoft>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
Message-ID: <CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>

On Thu, Jul 14, 2011 at 9:44 AM, Jacek Konieczny <jajcus at jajcus.net> wrote:
> On Wed, Jul 13, 2011 at 03:07:07PM +0200, Jakub Bogusz wrote:
>> I don't like the idea of __pycache__ not managed by rpm (or not
>> maintained with rpm database in case of more robust post scripts)
>> because of at least:
>> - possible inconsistences (like leftovers in case of upgrade failures;
>> ? in such case it would be even possible that after installing another
>> ? version of package with *.py files having older date than pycache,
>> ? pycache won't be rebuilt)
>> - more work needed to find package "owning" __pycache__/* files
>> - one more place to hide some malicious code not detectable by rpm -Va
> Ok, you may be right. But what than? We don't even have an idea how
> should the proper rpm solution work and no volunteers to design and code
> it. At the same time Python 3.2 is waiting for being included in PLD,
> Python 3.3 is on its way and PyPy could be useful too, provided
> compatible packages are available.

(?)

> In case some automated mechanism is provided, in the future, for keeping
> the compiled python code files in the RPM database without listing them
> in the spec files, we loose nothing by having the files there already
> (by explicitly listing them in the specs).

Can we plug into cpython's/pypy's module loader to detect when it
compiles stuff? If so, can we then launch a suid helper that injects
the newly created files into the rpm package that contained the
original .py?

-- 
Patryk Zawadzki
I solve problems.


From n3npq at mac.com  Thu Jul 14 13:11:41 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 07:11:41 -0400
Subject: *.py packaging, again
In-Reply-To: <20110714074401.GB6660@jajo.eggsoft>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
Message-ID: <8C516201-74C5-4994-B99F-D9BBDAB12294@mac.com>


On Jul 14, 2011, at 3:44 AM, Jacek Konieczny wrote:

> 
> Ok, you may be right. But what than? We don't even have an idea how
> should the proper rpm solution work and no volunteers to design and code
> it. At the same time Python 3.2 is waiting for being included in PLD,

Um, that isn't true: I did just ask for a statement of
what is desired with __pycache__ handling for the purpose
of attempting to design/code a solution in RPM.

73 de Jeff



From gotar at polanet.pl  Thu Jul 14 13:12:59 2011
From: gotar at polanet.pl (Tomasz Pala)
Date: Thu, 14 Jul 2011 13:12:59 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714074401.GB6660@jajo.eggsoft>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
Message-ID: <20110714111259.GA6996@polanet.pl>

On Thu, Jul 14, 2011 at 09:44:01 +0200, Jacek Konieczny wrote:

> For Python 2 (python-* packages), when including *.py:
> 
> We lose:
> - some 'purity' some people see in not distributing 'sources'
> 
> For Python >= 3.2 (python3-* packages):
> 
> We lose:
> - some 'purity' some people see in not distributing 'sources'

Usability is _always_ more important than some rules. So if
the *.py files bring _any_ added value to the workstation (not
development machine) they should be packaged regardless of 'purity'.
Existence of alternate runtime environment that make use of them is enough.
So the only question remaining is not 'if' to package but 'how' - I
don't use python, but it must not be overcomplicated - for the time
being other distros might have worked it out.

-- 
Tomasz Pala <gotar at pld-linux.org>


From gotar at polanet.pl  Thu Jul 14 13:17:04 2011
From: gotar at polanet.pl (Tomasz Pala)
Date: Thu, 14 Jul 2011 13:17:04 +0200
Subject: *.py packaging, again
In-Reply-To: <CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
Message-ID: <20110714111704.GB6996@polanet.pl>

On Thu, Jul 14, 2011 at 10:35:17 +0200, Patryk Zawadzki wrote:

> compiles stuff? If so, can we then launch a suid helper that injects
> the newly created files into the rpm package that contained the
> original .py?

OMG...
Please don't mess with rpm database in indeterministic time (like some
runtime) and do NOT use SUID.

This is excelent example of what I've called 'overcomplicated'.

-- 
Tomasz Pala <gotar at pld-linux.org>


From n3npq at mac.com  Thu Jul 14 13:19:34 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 07:19:34 -0400
Subject: *.py packaging, again
In-Reply-To: <CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
Message-ID: <9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>


On Jul 14, 2011, at 4:35 AM, Patryk Zawadzki wrote:
> 
>> In case some automated mechanism is provided, in the future, for keeping
>> the compiled python code files in the RPM database without listing them
>> in the spec files, we loose nothing by having the files there already
>> (by explicitly listing them in the specs).
> 
> Can we plug into cpython's/pypy's module loader to detect when it
> compiles stuff? If so, can we then launch a suid helper that injects
> the newly created files into the rpm package that contained the
> original .py?
> 

This is essentially a "push" strategy, where python compilation
is forced to undertake package management.

The design problem is that compilation is per-file, while
registered packages are containers of files. You will
end up with a package-per-file, and/or some extension
to a package container to attach compiled files.

You also need to decide whether all compilations are on
the build system or after installation: there are different
implementations.

I'd suggest that the problem needs to be solved on the install
not the build system as has been "traditionally" attempted.

73 de Jeff



From n3npq at mac.com  Thu Jul 14 13:21:35 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 07:21:35 -0400
Subject: *.py packaging, again
In-Reply-To: <20110714111704.GB6996@polanet.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<20110714111704.GB6996@polanet.pl>
Message-ID: <150C0C3D-09B4-48C2-89FE-91B44D73AA6A@mac.com>


On Jul 14, 2011, at 7:17 AM, Tomasz Pala wrote:

> On Thu, Jul 14, 2011 at 10:35:17 +0200, Patryk Zawadzki wrote:
> 
>> compiles stuff? If so, can we then launch a suid helper that injects
>> the newly created files into the rpm package that contained the
>> original .py?
> 
> OMG...
> Please don't mess with rpm database in indeterministic time (like some
> runtime) and do NOT use SUID.
> 
> This is excelent example of what I've called 'overcomplicated'.
> 

And you comments (which I agree with) are negatively phrased.

State what you want to see positively. Or prepare yourself for
the SUID audit that will inevitably be attempted.


73 de Jeff


From patrys at pld-linux.org  Thu Jul 14 13:28:38 2011
From: patrys at pld-linux.org (Patryk Zawadzki)
Date: Thu, 14 Jul 2011 13:28:38 +0200
Subject: *.py packaging, again
In-Reply-To: <9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
Message-ID: <CANw2pUGvOFRw6N-yBT4XJ1eCZb6SiGQwHwtuP+Yqh61zDLHASQ@mail.gmail.com>

On Thu, Jul 14, 2011 at 1:19 PM, Jeff Johnson <n3npq at mac.com> wrote:
> This is essentially a "push" strategy, where python compilation
> is forced to undertake package management.
>
> The design problem is that compilation is per-file, while
> registered packages are containers of files. You will
> end up with a package-per-file, and/or some extension
> to a package container to attach compiled files.

I was thinking more like "one package per egg" for packages installed
but "attach files to an existing package" for newly compiled files
that come from known packages. What I am after is rpm being able to
answe "what package does this file belong to" and "what files do I
remove when uninstalling this package".

> You also need to decide whether all compilations are on
> the build system or after installation: there are different
> implementations.

I'd vote for "on install" or "on demand" (think three versions of
python sharing the same python code but each using only a fraction of
installed packages).

-- 
Patryk Zawadzki
I solve problems.


From gotar at polanet.pl  Thu Jul 14 14:04:50 2011
From: gotar at polanet.pl (Tomasz Pala)
Date: Thu, 14 Jul 2011 14:04:50 +0200
Subject: *.py packaging, again
In-Reply-To: <150C0C3D-09B4-48C2-89FE-91B44D73AA6A@mac.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<20110714111704.GB6996@polanet.pl>
	<150C0C3D-09B4-48C2-89FE-91B44D73AA6A@mac.com>
Message-ID: <20110714120448.GA9019@polanet.pl>

On Thu, Jul 14, 2011 at 07:21:35 -0400, Jeff Johnson wrote:

>>> compiles stuff? If so, can we then launch a suid helper that injects
>>> the newly created files into the rpm package that contained the
>>> original .py?
>> 
>> OMG...
>> Please don't mess with rpm database in indeterministic time (like some
>> runtime) and do NOT use SUID.
>> 
>> This is excelent example of what I've called 'overcomplicated'.
>> 
> 
> And you comments (which I agree with) are negatively phrased.
> 
> State what you want to see positively.

1. I don't know python internals, but if these cache files are not
strictly machine-dependant (i.e. they don't differ on machines having
the same arch) they should be shipped in compiled form (otherwise we
might as well end up with %_make; %configure; ... in %build of every
package) - this is not gentoo.

2. if anything wants to alter rpm database, it must be done during
transaction or shortly after it (i.e. within installation process); it's
not acceptable to do changes there during regular system use (on
unprivilidged account) - after all it's MD5 repo and might be
hard-locked by admin.

3. install -> compile -> save to rpmdb is pointless - saved MD5 might be already
compromised by some malicious software and thus undetectable even via
verification run from outer source (i.e. rescuecd). Checksums must be
calculated in clean environment and be comparable between different
systems (so we're back to point 1).


Otherwise we might just put some
rm -f /usr/share/python/**/__pycache__/* to cron or %post of python
pkgs to ensure that no malicious/leftovers will harm our system. If the
files are to be regenerated by regular user, why not remove them once a
week to save space?

> Or prepare yourself for
> the SUID audit that will inevitably be attempted.

SUID is broken by design and obsoleted (and disabled entirely on some
machines). Are we talking about CAP_DAC_OVERRIDE?

-- 
Tomasz Pala <gotar at pld-linux.org>


From gotar at polanet.pl  Thu Jul 14 14:05:52 2011
From: gotar at polanet.pl (Tomasz Pala)
Date: Thu, 14 Jul 2011 14:05:52 +0200
Subject: *.py packaging, again
In-Reply-To: <9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
Message-ID: <20110714120552.GB9019@polanet.pl>

On Thu, Jul 14, 2011 at 07:19:34 -0400, Jeff Johnson wrote:

> You also need to decide whether all compilations are on
> the build system or after installation: there are different
> implementations.
> 
> I'd suggest that the problem needs to be solved on the install
> not the build system as has been "traditionally" attempted.

Do you want to skip them during repackage?

-- 
Tomasz Pala <gotar at pld-linux.org>


From n3npq at mac.com  Thu Jul 14 14:36:37 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 08:36:37 -0400
Subject: *.py packaging, again
In-Reply-To: <20110714120448.GA9019@polanet.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<20110714111704.GB6996@polanet.pl>
	<150C0C3D-09B4-48C2-89FE-91B44D73AA6A@mac.com>
	<20110714120448.GA9019@polanet.pl>
Message-ID: <92C9C5D9-F19C-433F-A492-A65FBC035E84@mac.com>


On Jul 14, 2011, at 8:04 AM, Tomasz Pala wrote:

> On Thu, Jul 14, 2011 at 07:21:35 -0400, Jeff Johnson wrote:
> 
>>>> compiles stuff? If so, can we then launch a suid helper that injects
>>>> the newly created files into the rpm package that contained the
>>>> original .py?
>>> 
>>> OMG...
>>> Please don't mess with rpm database in indeterministic time (like some
>>> runtime) and do NOT use SUID.
>>> 
>>> This is excelent example of what I've called 'overcomplicated'.
>>> 
>> 
>> And you comments (which I agree with) are negatively phrased.
>> 
>> State what you want to see positively.
> 
> 1. I don't know python internals, but if these cache files are not
> strictly machine-dependant (i.e. they don't differ on machines having
> the same arch) they should be shipped in compiled form (otherwise we
> might as well end up with %_make; %configure; ... in %build of every
> package) - this is not gentoo.
> 

Historically, the *.py[co] files have been packaged in *.rpm as if the
content is both per-arch and per-python-version. I don't know what
the implementation reality actually is, or whether the choices are unnecessarily
restrictive and overly paranoid "future proofing" or ?

.. but if the *.py[co] are portable (whatever that means), then
the portability should be reflected in the design of the packaging.

There are aspects of eggs, where eggs can load other eggs dynamically,
that are very different from other static content delivered in *.rpm,
and the dynamic side effects are purely on the install system and
are not easily captured on the build system. E.g. what other eggs
might be installed as a side effect likely has dynamic state associated
that cannot be reproduced within a build for all usage cases.


> 2. if anything wants to alter rpm database, it must be done during
> transaction or shortly after it (i.e. within installation process); it's
> not acceptable to do changes there during regular system use (on
> unprivilidged account) - after all it's MD5 repo and might be
> hard-locked by admin.
> 

Well that isn't necessarily true or even needed (depending on what
is attempted).

Headers can be digitally signed, and mostly do not contain any critically
important information that can't be found in the original *.rpm file.

There are a few (mostly timestamps) pieces of information that can
be assigned only during a "transaction", but that does NOT mean
"within installation process" necessarily.

It would mean that any attempt to register a "package" in an rpmdb
SHOULD
	1) carry digests or attempt signing
	2) faithfully include a "transaction id" and a "installation time"
just like an RPM installation does.

And yes -- if you choose to treat an rpmdb with concepts like
"locked" and "privileged" -- then other installers will be
constrained in what what is permitted.

Both "locked" and "privileged" can be avoided if an rpm tool
which is already "privileged" pulls information rather than having
the python JIT'er push directly into an rpmdb.

This introduces a TOCTTOU window
	http://en.wikipedia.org/wiki/Time-of-check-to-time-of-use
between when an egg is installed (and might queue a request for rpmdb registration)
and when the registration is actually handled. There are several approaches
to avoiding TOCTTOU queueing latency if you think a bit.

> 3. install -> compile -> save to rpmdb is pointless - saved MD5 might be already
> compromised by some malicious software and thus undetectable even via
> verification run from outer source (i.e. rescuecd). Checksums must be
> calculated in clean environment and be comparable between different
> systems (so we're back to point 1).
> 

And the TOCTTOU window can be eliminated by
	1) including the digests in the queued request
	2) signing the queued request
and having RPM undertake verification before servicing the rpmdb
registration request.

Other issues like "clean" and "safe" can only be determined by
examining explicit thread models or auditing an actual implementation.

> 
> Otherwise we might just put some
> rm -f /usr/share/python/**/__pycache__/* to cron or %post of python
> pkgs to ensure that no malicious/leftovers will harm our system. If the
> files are to be regenerated by regular user, why not remove them once a
> week to save space?
> 

That's one approach to caching:
	Don't do it! Disk space is expensive!

>> Or prepare yourself for
>> the SUID audit that will inevitably be attempted.
> 
> SUID is broken by design and obsoleted (and disabled entirely on some
> machines). Are we talking about CAP_DAC_OVERRIDE?
> 

CAP_DAC_OVERRIDE isn't _THAT_ much better than SUID imho. But
you can advocate whatever security scheme you wish: CAP_DAC_OVERRIDE
is certainly finer grained privilege access than SUID is.

73 de Jeff



From n3npq at mac.com  Thu Jul 14 14:57:40 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 08:57:40 -0400
Subject: *.py packaging, again
In-Reply-To: <20110714120552.GB9019@polanet.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
	<20110714120552.GB9019@polanet.pl>
Message-ID: <AC9F9D93-2C77-4AA3-8F92-0C8B533FC82B@mac.com>


On Jul 14, 2011, at 8:05 AM, Tomasz Pala wrote:

> On Thu, Jul 14, 2011 at 07:19:34 -0400, Jeff Johnson wrote:
> 
>> You also need to decide whether all compilations are on
>> the build system or after installation: there are different
>> implementations.
>> 
>> I'd suggest that the problem needs to be solved on the install
>> not the build system as has been "traditionally" attempted.
> 
> Do you want to skip them during repackage?
> 

(aside)
Do you actually use repackaged packages? For what purpose?

There are solutions here as well, depending on what is desired.

One can view side-effects like JIT'ed cached files as
either disposable and easily regenerated (in which case
the content should not be repackaged) or you can view
the JIT'ed cached files as stageful and persistent
(in which case repackaging should include the content).

All depends on POV and "expectations".

And since its highly likely that noone can state
the desired POV precisely, then
	Have it your own way!
as an engineering solution would need to be attempted.

The two obvious resolutions to cached files are
	Include them in repackaging.
and
	Don't include in repackaging.
and that's essentially an if statement somewhere.

*shrug*

73 de Jeff



From gotar at polanet.pl  Thu Jul 14 15:23:52 2011
From: gotar at polanet.pl (Tomasz Pala)
Date: Thu, 14 Jul 2011 15:23:52 +0200
Subject: *.py packaging, again
In-Reply-To: <AC9F9D93-2C77-4AA3-8F92-0C8B533FC82B@mac.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
	<20110714120552.GB9019@polanet.pl>
	<AC9F9D93-2C77-4AA3-8F92-0C8B533FC82B@mac.com>
Message-ID: <20110714132352.GA13681@polanet.pl>

On Thu, Jul 14, 2011 at 08:57:40 -0400, Jeff Johnson wrote:

> (aside)
> Do you actually use repackaged packages? For what purpose?

Unfortunately yes, for restoring working set - too often...
And during tests of course (various, like my private builds or bug
hunting) [*].

> There are solutions here as well, depending on what is desired.
> 
> One can view side-effects like JIT'ed cached files as
> either disposable and easily regenerated (in which case
> the content should not be repackaged) or you can view
> the JIT'ed cached files as stageful and persistent
> (in which case repackaging should include the content).

I ask because I see that rpm stores original file list (with sizes and
timestamps) in it's headers, regardless of actual content of cpio. It
includes both modified and nonexistent files (especially %doc when excludedocs
is set and ommitted %langs).
In this case cpio would contain more files than headers know about. What
might happen with recompilation on different timestamps during downgrade or some other
weird operations like --noscripts?


*  I had an idea once upon a time to verify content of repackaged files
   against original digest, I really miss this feature in rpm (rpm -Vp
   verifies package against filesystem not internal cpio).

-- 
Tomasz Pala <gotar at pld-linux.org>


From n3npq at mac.com  Thu Jul 14 16:43:56 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 10:43:56 -0400
Subject: *.py packaging, again
In-Reply-To: <20110714132352.GA13681@polanet.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
	<20110714120552.GB9019@polanet.pl>
	<AC9F9D93-2C77-4AA3-8F92-0C8B533FC82B@mac.com>
	<20110714132352.GA13681@polanet.pl>
Message-ID: <22BA971F-BBA8-4C13-86B9-CD98444073DF@mac.com>


On Jul 14, 2011, at 9:23 AM, Tomasz Pala wrote:

> On Thu, Jul 14, 2011 at 08:57:40 -0400, Jeff Johnson wrote:
> 
>> (aside)
>> Do you actually use repackaged packages? For what purpose?
> 
> Unfortunately yes, for restoring working set - too often...
> And during tests of course (various, like my private builds or bug
> hunting) [*].
> 

Well its *IS* you "unfortunate" choice to use --repackage (or not).
You have none but yourself to blame for your misfortunes.

Hint: Repackage packages were _DELIBERATELY_ poisoned by
adding RPMTAG_REMOVETID in order to prevent morons from
	install -> modify -> re-package -> re-publish -> blame @redhat
a *.rpm as if it were built "reproducibly" and "official".

Your "unfortunately" is likely mostly alleviated by simply
	Don't poison repackaged *.rpm by adding RPMTAG_REMOVETID.
I.e -- for most packages that are never ever modified --
you will find that repackaged *.rpm packages start to pass
	rpm -Kvv repackaged*.rpm
tests instantly.

There's little that can be done about modifications to %config
and the fundamental design flaw
	You can't repackage a file after "rm /path/to/file" has been done.

But that lack of functionality WILL give you years of RFE's and complaints:

	RPM sux because ?

>> There are solutions here as well, depending on what is desired.
>> 
>> One can view side-effects like JIT'ed cached files as
>> either disposable and easily regenerated (in which case
>> the content should not be repackaged) or you can view
>> the JIT'ed cached files as stageful and persistent
>> (in which case repackaging should include the content).
> 
> I ask because I see that rpm stores original file list (with sizes and
> timestamps) in it's headers, regardless of actual content of cpio. It
> includes both modified and nonexistent files (especially %doc when excludedocs
> is set and ommitted %langs).

Not quite, --relocate stores both the original and the relocated paths.

And RPM uses cpio headers for exactly one purpose:

	The file path in the cpio header is used to locate the
	metadata in the RPM header.

> In this case cpio would contain more files than headers know about. What

And adding more files will fail to find _ALL_ information about
the added file, and the file will be skipped (iirc, been years, I fergit)

This is/was by intent: the advantage to RPM packaging is that _EVERYTHING_
necessary to perform an install is encapsulated in a digitally signed
blob that is essentially immutable. Any/all modifications to that blob
are detectable.

> might happen with recompilation on different timestamps during downgrade or some other
> weird operations like --noscripts?
> 

recompilation != immutable

And only immutable will do: there's lots of TOCTTOU issues with
rebuilding as well, honking about time stamps is a mere fig leaf.

> 
> *  I had an idea once upon a time to verify content of repackaged files
>   against original digest, I really miss this feature in rpm (rpm -Vp
>   verifies package against filesystem not internal cpio).
> 

Remove the RPMTAG_REMOVETID poisoning and repackaging (for most packages)
becomes an exact inverse and file digests can then be verified just
like any other package.

hth

73 de Jeff



From gotar at polanet.pl  Thu Jul 14 17:04:02 2011
From: gotar at polanet.pl (Tomasz Pala)
Date: Thu, 14 Jul 2011 17:04:02 +0200
Subject: *.py packaging, again
In-Reply-To: <22BA971F-BBA8-4C13-86B9-CD98444073DF@mac.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
	<20110714120552.GB9019@polanet.pl>
	<AC9F9D93-2C77-4AA3-8F92-0C8B533FC82B@mac.com>
	<20110714132352.GA13681@polanet.pl>
	<22BA971F-BBA8-4C13-86B9-CD98444073DF@mac.com>
Message-ID: <20110714150402.GA18021@polanet.pl>

On Thu, Jul 14, 2011 at 10:43:56 -0400, Jeff Johnson wrote:

> Well its *IS* you "unfortunate" choice to use --repackage (or not).
> You have none but yourself to blame for your misfortunes.

No, there are idiots from many many software devel teams to blame
(last time Xorg xinput and Intel drivers, firefox since many years).

> Hint: Repackage packages were _DELIBERATELY_ poisoned by
> adding RPMTAG_REMOVETID in order to prevent morons from
> 	install -> modify -> re-package -> re-publish -> blame @redhat
> a *.rpm as if it were built "reproducibly" and "official".
> 
> Your "unfortunately" is likely mostly alleviated by simply
> 	Don't poison repackaged *.rpm by adding RPMTAG_REMOVETID.
> I.e -- for most packages that are never ever modified --
> you will find that repackaged *.rpm packages start to pass
> 	rpm -Kvv repackaged*.rpm

You didn't understand me - I don't have any problems with failing
digests. I don't have any problems with repackage at all.
The only problem is that I _need_ to use them, because some morons break
their software in regular way - you've asked _if_ I use it and _for what_,
that's the answer, no complaining on rpm itself.

The one feature I miss:

> This is/was by intent: the advantage to RPM packaging is that _EVERYTHING_
> necessary to perform an install is encapsulated in a digitally signed
> blob that is essentially immutable. Any/all modifications to that blob
> are detectable.

is that it only detects modification and can't point the files modified
(which could be done by comparing files stored in cpio against
informations in header).

> recompilation != immutable
> 
> And only immutable will do: there's lots of TOCTTOU issues with
> rebuilding as well, honking about time stamps is a mere fig leaf.

If you downgrade a package to repackaged version with (by any way) newer
timestamps you need to ensure, that python actually does rebuild the
cache. In short: you must ensure cached files are _removed_ everytime
package is modified (replaced, removed, whatever).

>> *  I had an idea once upon a time to verify content of repackaged files
>>   against original digest, I really miss this feature in rpm (rpm -Vp
>>   verifies package against filesystem not internal cpio).
> 
> Remove the RPMTAG_REMOVETID poisoning and repackaging (for most packages)
> becomes an exact inverse and file digests can then be verified just
> like any other package.

By rpm binary? How?

-- 
Tomasz Pala <gotar at pld-linux.org>


From gotar at polanet.pl  Thu Jul 14 17:10:09 2011
From: gotar at polanet.pl (Tomasz Pala)
Date: Thu, 14 Jul 2011 17:10:09 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714150402.GA18021@polanet.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
	<20110714120552.GB9019@polanet.pl>
	<AC9F9D93-2C77-4AA3-8F92-0C8B533FC82B@mac.com>
	<20110714132352.GA13681@polanet.pl>
	<22BA971F-BBA8-4C13-86B9-CD98444073DF@mac.com>
	<20110714150402.GA18021@polanet.pl>
Message-ID: <20110714151009.GB18021@polanet.pl>

On Thu, Jul 14, 2011 at 17:04:02 +0200, Tomasz Pala wrote:

>>> *  I had an idea once upon a time to verify content of repackaged files
>>>   against original digest, I really miss this feature in rpm (rpm -Vp
>>>   verifies package against filesystem not internal cpio).
>> 
>> Remove the RPMTAG_REMOVETID poisoning and repackaging (for most packages)
>> becomes an exact inverse and file digests can then be verified just
>> like any other package.
> 
> By rpm binary? How?

For example, I got original file (currently installed):

~:  rpm -Kvv iceweasel-3.6.17-1.i686.rpm 
D: Expected size:      6683934 = lead(96)+sigs(920)+pad(0)+data(6682918)
D:   Actual size:      6683934
iceweasel-3.6.17-1.i686.rpm:
    Header SHA1 digest: OK (b0509c227544d3d4860b484e8afde6eec28a051b)
    MD5 digest: OK (a92e0a2dc52504d42959bb67741596f7)

~:  rpm -Vp iceweasel-3.6.17-1.i686.rpm 
Unsatisfied dependencies for iceweasel-3.6.17-1.i686:
        Requires: libhunspell-1.2.so.0

S.5....T   /usr/bin/iceweasel
S.5....T   /usr/lib/iceweasel/components/nsLoginManager.js
S.5....T   /usr/lib/iceweasel/components/storage-Legacy.js
S.5....T   /usr/lib/iceweasel/components/storage-mozStorage.js


And I got some previous, repackaged file:

/var/spool/repackage/1309337365#  rpm -Kvv iceweasel-3.6.13-2.i686.rpm
D: Expected size:      7452100 = lead(96)+sigs(180)+pad(4)+data(7451820)
D:   Actual size:      7506959
iceweasel-3.6.13-2.i686.rpm:
    Header SHA1 digest: OK (9ad4a3fb6c194ba14c5b85f20e8aa567bbce3764)
    MD5 digest: BAD Expected(51df700b3bfa11a73df0d1219eae9f97) != (43b884021c55da3d87f9e76f83f5ceb4)


How can I list files within the latter archive, that were modified
before repackage (in this case I know they are exactly the same 3 as above).

-- 
Tomasz Pala <gotar at pld-linux.org>


From qboosh at pld-linux.org  Thu Jul 14 17:44:19 2011
From: qboosh at pld-linux.org (Jakub Bogusz)
Date: Thu, 14 Jul 2011 17:44:19 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714074401.GB6660@jajo.eggsoft>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
Message-ID: <20110714154419.GO5300@stranger.qboosh.pl>

On Thu, Jul 14, 2011 at 09:44:01AM +0200, Jacek Konieczny wrote:
> On Wed, Jul 13, 2011 at 03:07:07PM +0200, Jakub Bogusz wrote:
> > I don't like the idea of __pycache__ not managed by rpm (or not
> > maintained with rpm database in case of more robust post scripts)
> > because of at least:
> > - possible inconsistences (like leftovers in case of upgrade failures;
> >   in such case it would be even possible that after installing another
> >   version of package with *.py files having older date than pycache,
> >   pycache won't be rebuilt)
> > - more work needed to find package "owning" __pycache__/* files
> > - one more place to hide some malicious code not detectable by rpm -Va

And one more: why byte-compiling on every target machine when the result
should be exactly the same for given Python major version?
(modulo python bugs, but then - we want the same compiled code even more)
In big packages byte-compilation may take significant amount of time.

This argument is still valid when considering byte-compilation on rpm
install side.

> Ok, you may be right. But what than? We don't even have an idea how
> should the proper rpm solution work and no volunteers to design and code
> it. At the same time Python 3.2 is waiting for being included in PLD,
> Python 3.3 is on its way and PyPy could be useful too, provided
> compatible packages are available.

There is also jpython since some time.
I don't know how compatibility looks like (IIRC jpythons were released
with some delay after cpython version it was +/- compatible with).

> Maybe we should just start packing the *.py files together with the
> compiled files for the python version they are built for? Yes, I know I
> can start the old 'we don't package sources' flame war again, but '*.py'
> files are not sources, they are the executable code. '*.pyc' and '*.pyo'
> are just 'optimized versions' of those. And for Python 3.2 we have no
> option - I have already started adding both *.py and __pycache__/* to
> the python3 packages I have recently done.
> 
> For Python 2 (python-* packages), when including *.py:
> 
> We gain:
> - ability to use the installed noarch packages by other Python 2.7
>   implementations (like PyPy)

Are particular PyPy versions meant to be compatible with particular
Python 2 major version?

Even in Python 3 times noarch packages still use versioned source
directory.

Also, the question whether to package *.py refers only to noarch
Python 2 packages, as arch-dependent ones are bound to particular
cpython major version anyway.

The same gain can be reached by adding -source packages.

> For Python >= 3.2 (python3-* packages):
> 
> We gain:
> - compatibility with the language

For new Python 3 way of packaging it's unquestionable.

But there are still many packages not using setuptools(?) and thus
packaged in Python 2 way.
Should we enforce the new packaging method for them?
%py3_comp/%py3_ocomp macros use the old one.


-- 
Jakub Bogusz    http://qboosh.pl/


From n3npq at mac.com  Thu Jul 14 18:15:31 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 12:15:31 -0400
Subject: *.py packaging, again
In-Reply-To: <20110714151009.GB18021@polanet.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CANw2pUHsdvvQUE5wEBGDq6oo6RYwLM9UDY2o0GFd2-C7J=az0g@mail.gmail.com>
	<9EA5B5E1-3D53-48DC-93E0-1E009FD5DD73@mac.com>
	<20110714120552.GB9019@polanet.pl>
	<AC9F9D93-2C77-4AA3-8F92-0C8B533FC82B@mac.com>
	<20110714132352.GA13681@polanet.pl>
	<22BA971F-BBA8-4C13-86B9-CD98444073DF@mac.com>
	<20110714150402.GA18021@polanet.pl> <20110714151009.GB18021@polanet.pl>
Message-ID: <CA91F10E-E99B-43F4-AD62-5BD9CBC879EF@mac.com>


On Jul 14, 2011, at 11:10 AM, Tomasz Pala wrote:

> On Thu, Jul 14, 2011 at 17:04:02 +0200, Tomasz Pala wrote:
> 
>>>> *  I had an idea once upon a time to verify content of repackaged files
>>>>  against original digest, I really miss this feature in rpm (rpm -Vp
>>>>  verifies package against filesystem not internal cpio).
>>> 
>>> Remove the RPMTAG_REMOVETID poisoning and repackaging (for most packages)
>>> becomes an exact inverse and file digests can then be verified just
>>> like any other package.
>> 
>> By rpm binary? How?
> 
> For example, I got original file (currently installed):
> 
> ~:  rpm -Kvv iceweasel-3.6.17-1.i686.rpm 
> D: Expected size:      6683934 = lead(96)+sigs(920)+pad(0)+data(6682918)
> D:   Actual size:      6683934
> iceweasel-3.6.17-1.i686.rpm:
>    Header SHA1 digest: OK (b0509c227544d3d4860b484e8afde6eec28a051b)
>    MD5 digest: OK (a92e0a2dc52504d42959bb67741596f7)
> 
> ~:  rpm -Vp iceweasel-3.6.17-1.i686.rpm 
> Unsatisfied dependencies for iceweasel-3.6.17-1.i686:
>        Requires: libhunspell-1.2.so.0
> 
> S.5....T   /usr/bin/iceweasel
> S.5....T   /usr/lib/iceweasel/components/nsLoginManager.js
> S.5....T   /usr/lib/iceweasel/components/storage-Legacy.js
> S.5....T   /usr/lib/iceweasel/components/storage-mozStorage.js
> 
> 
> And I got some previous, repackaged file:
> 
> /var/spool/repackage/1309337365#  rpm -Kvv iceweasel-3.6.13-2.i686.rpm
> D: Expected size:      7452100 = lead(96)+sigs(180)+pad(4)+data(7451820)
> D:   Actual size:      7506959
> iceweasel-3.6.13-2.i686.rpm:
>    Header SHA1 digest: OK (9ad4a3fb6c194ba14c5b85f20e8aa567bbce3764)

This indicates that someone has already removed the poisoning:
	if RPMTAG_REMOVETID were present, then the header SHA1 would fail

>    MD5 digest: BAD Expected(51df700b3bfa11a73df0d1219eae9f97) != (43b884021c55da3d87f9e76f83f5ceb4)
> 

So the operation that needs doing is essentially this

	mkdir -p XXX
	cd XXX
	rpm2cpio iceweasel-3.6.13-2.i686.rpm | cpio -dim
	rpm -Vp --root `pwd` iceweasel-3.6.13-2.i686.rpm

which RPM will not do for you.

Personally I'd rather see RPM chill the shot glasses, pour the vodka, and
serve up peanuts and chips than bother chasing down which file in a repackaged
payload is actually modified. YMMV.

> 
> How can I list files within the latter archive, that were modified
> before repackage (in this case I know they are exactly the same 3 as above).
> 

See above:

Kazdy wypity kieliszek ? to gw?zdz do naszej trumny? Pijmy wiec tak by trumna sie nie rozpadla.

73 de Jeff
> -- 
> Tomasz Pala <gotar at pld-linux.org>
> _______________________________________________
> pld-devel-en mailing list
> pld-devel-en at lists.pld-linux.org
> http://lists.pld-linux.org/mailman/listinfo/pld-devel-en



From jajcus at jajcus.net  Thu Jul 14 18:34:11 2011
From: jajcus at jajcus.net (Jacek Konieczny)
Date: Thu, 14 Jul 2011 18:34:11 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714154419.GO5300@stranger.qboosh.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<20110714154419.GO5300@stranger.qboosh.pl>
Message-ID: <20110714163411.GA4099@lolek.nigdzie>

On Thu, Jul 14, 2011 at 05:44:19PM +0200, Jakub Bogusz wrote:
> And one more: why byte-compiling on every target machine when the result
> should be exactly the same for given Python major version?
> 
> This argument is still valid when considering byte-compilation on rpm
> install side.

True. For CPython packaging pre-compiled *.pyc files makes sense only
for this reason.

> > 
> > We gain:
> > - ability to use the installed noarch packages by other Python 2.7
> >   implementations (like PyPy)
> 
> Are particular PyPy versions meant to be compatible with particular
> Python 2 major version?

PyPy 1.5 is compatible with Python 2.7. As far as pure-python (noarch in
PLD) *.py modules are concerned.

BTW PyPy also generates *.pyc files, in a quite different format that
those from Python 2.7 and they are most probably architecture dependant.

This gives some ugly behaviour:
> [jajcus at lolek tt]$ echo "print('module')" > mod.py
> [jajcus at lolek tt]$ echo "import mod" > test.py
> [jajcus at lolek tt]$ python test.py
> module
> [jajcus at lolek tt]$ ls -l
> total 12
> -rw-r--r-- 1 jajcus users  16 Jul 14 18:17 mod.py
> -rw-r--r-- 1 jajcus users 117 Jul 14 18:18 mod.pyc
> -rw-r--r-- 1 jajcus users  11 Jul 14 18:18 test.py
> [jajcus at lolek tt]$ file mod.pyc
> mod.pyc: python 2.7 byte-compiled
> [jajcus at lolek tt]$ pypy test.py
> module
> [jajcus at lolek tt]$ ls -l
> total 12
> -rw-r--r-- 1 jajcus users  16 Jul 14 18:17 mod.py
> -rw-r--r-- 1 jajcus users 117 Jul 14 18:18 mod.pyc
> -rw-r--r-- 1 jajcus users  11 Jul 14 18:18 test.py
> [jajcus at lolek tt]$ file mod.pyc
> mod.pyc: data

That is why the new Python 3.2 way makes sense:

> [jajcus at lolek tt]$ python3 test.py
> module
> [jajcus at lolek tt]$ ls -l
> total 16
> drwxr-xr-x 2 jajcus users 4096 Jul 14 18:19 __pycache__
> -rw-r--r-- 1 jajcus users   16 Jul 14 18:17 mod.py
> -rw-r--r-- 1 jajcus users  117 Jul 14 18:18 mod.pyc
> -rw-r--r-- 1 jajcus users   11 Jul 14 18:18 test.py
> [jajcus at lolek tt]$ ls __pycache__/
> mod.cpython-32.pyc
 
Please note that not every piece of python code will be always
installed from an RPM (e.g. developer machine)

> Even in Python 3 times noarch packages still use versioned source
> directory.

In python 3.2 this won't be needed any more. 

> Also, the question whether to package *.py refers only to noarch
> Python 2 packages, as arch-dependent ones are bound to particular
> cpython major version anyway.

True, for the 'alternative implementations' argument.
 
> The same gain can be reached by adding -source packages.

We could even create a pypy-* (with *.pyc in /usr/lib for every *.py
which would go to /usr/share in Python 2.7?) and jython-* for every
python-*?

Adding -source packages adds extra maintenance problems. Not removing 
*.py and not patching 'broken' apps, removes some maintenance problems.

> But there are still many packages not using setuptools(?) 

setuptools are dead and ever supported Python 3 AFAIK
There are distutils (part of the man Python library), distribute (fork
of setuptools, supporting Python 3 and 2to3 conversion) and on the way
are 'packaging' and distutils2?

Anyway, that doesn't change a thing in 'packaging python 2 way' or
'packaging python 3 way'

> and thus
> packaged in Python 2 way.

Most of our python3-* packages were never updated or rebuilt for the
python3 changes. I think they won't even build, as the *.pyc files
listed in %files are missing.

> Should we enforce the new packaging method for them?
> %py3_comp/%py3_ocomp macros use the old one.

Those only start the standard python compilation functions. For Python <
3.2 they will produce X.py[co] for every X.py and for Python >= 3.2 they
will produce __pycache__/X.magic.py[co] for every X.py

What is will not work in this case are not the macros, but the %files
section. And it is hard to write a spec so both python 3.1 and 3.2 are
supported.

Additionally if %py3_postclean is run (*.py removed) then, even if
%files are 'correct', the package won't work (*.pyc from __pycache__
won't be ever loaded)

Greets,
        Jacek


From qboosh at pld-linux.org  Thu Jul 14 19:27:01 2011
From: qboosh at pld-linux.org (Jakub Bogusz)
Date: Thu, 14 Jul 2011 19:27:01 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714163411.GA4099@lolek.nigdzie>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<20110714154419.GO5300@stranger.qboosh.pl>
	<20110714163411.GA4099@lolek.nigdzie>
Message-ID: <20110714172701.GP5300@stranger.qboosh.pl>

On Thu, Jul 14, 2011 at 06:34:11PM +0200, Jacek Konieczny wrote:
> On Thu, Jul 14, 2011 at 05:44:19PM +0200, Jakub Bogusz wrote:
> > > We gain:
> > > - ability to use the installed noarch packages by other Python 2.7
> > >   implementations (like PyPy)
> > 
> > Are particular PyPy versions meant to be compatible with particular
> > Python 2 major version?
> 
> PyPy 1.5 is compatible with Python 2.7. As far as pure-python (noarch in
> PLD) *.py modules are concerned.
> 
> BTW PyPy also generates *.pyc files, in a quite different format that
> those from Python 2.7 and they are most probably architecture dependant.
> 
> This gives some ugly behaviour:

Very ugly... so PyPy can't refer to the system directory containing
(pure-)python 2.7 modules.
At most sources (*.py) could be symlinked to pypy-specific directory.

> > But there are still many packages not using setuptools(?) 
> 
> setuptools are dead and ever supported Python 3 AFAIK
> There are distutils (part of the man Python library), distribute (fork
> of setuptools, supporting Python 3 and 2to3 conversion) and on the way
> are 'packaging' and distutils2...

OK, I've seen some names, but didn't know which should be currently
used.

> Anyway, that doesn't change a thing in 'packaging python 2 way' or
> 'packaging python 3 way'
> 
> > and thus
> > packaged in Python 2 way.
> 
> Most of our python3-* packages were never updated or rebuilt for the
> python3 changes. I think they won't even build, as the *.pyc files
> listed in %files are missing.
> 
> > Should we enforce the new packaging method for them?
> > %py3_comp/%py3_ocomp macros use the old one.
> 
> Those only start the standard python compilation functions. For Python <
> 3.2 they will produce X.py[co] for every X.py and for Python >= 3.2 they
> will produce __pycache__/X.magic.py[co] for every X.py

Right. I was misleaded by some python3-* packages (namely pycairo and
pygobject) still producing *.py[co] in the same directory as *.py.

> What is will not work in this case are not the macros, but the %files
> section. And it is hard to write a spec so both python 3.1 and 3.2 are
> supported.

IMO we could assume that "python3" means ">= 3.2" and don't maintain
support for 3.0/3.1.

> Additionally if %py3_postclean is run (*.py removed) then, even if
> %files are 'correct', the package won't work (*.pyc from __pycache__
> won't be ever loaded)

Right.


-- 
Jakub Bogusz    http://qboosh.pl/


From wrobell at pld-linux.org  Thu Jul 14 20:31:28 2011
From: wrobell at pld-linux.org (Artur Wroblewski)
Date: Thu, 14 Jul 2011 19:31:28 +0100
Subject: *.py packaging, again
In-Reply-To: <20110714074401.GB6660@jajo.eggsoft>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
Message-ID: <CA+92fu_xHpXxvQ1N8fSrzSp=tVLsv+guOEKc0=eNRqjHwcGu=Q@mail.gmail.com>

On Thu, Jul 14, 2011 at 8:44 AM, Jacek Konieczny <jajcus at jajcus.net> wrote:
[...]
> We lose:
> - a little bit of the disk space
> - some 'purity' some people see in not distributing 'sources'

IMHO, it was not about purity but quite practical aspect of
file distribution (src vs. bin) - we treated byte compiled files
for Java and Python the same way, no src allowed.

Now, Python's idea about byte compiled files evolved into different
concept - cache files? Another game changer is pypy, of course.

Few months ago we agreed on *.py distribution for Python 3.2 (IMHO,
that implies we do not longer support 3.0 and 3.1 in python-*.spec). Lack
of 3.2 in Th is matter of time and resources to recompile the packages,
not __pycache__ problem.

We should treat cpython and pypy as we would treat two compilers,
which optimize code in different way. We will not build code with gcc
and Intel compilers and put two versions of packages on ftp.

IMHO, at the moment, we should settle on one Python "compiler". Obvious
choice is cpython, now. You want pypy - use its mechanism (hopefully it
exists) to put compiled bytecode in temp directory (or whatever).

[...]

Best regards,

Artur


From jajcus at jajcus.net  Thu Jul 14 21:00:14 2011
From: jajcus at jajcus.net (Jacek Konieczny)
Date: Thu, 14 Jul 2011 21:00:14 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714172701.GP5300@stranger.qboosh.pl>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<20110714154419.GO5300@stranger.qboosh.pl>
	<20110714163411.GA4099@lolek.nigdzie>
	<20110714172701.GP5300@stranger.qboosh.pl>
Message-ID: <20110714190014.GB4099@lolek.nigdzie>

On Thu, Jul 14, 2011 at 07:27:01PM +0200, Jakub Bogusz wrote:
> > What is will not work in this case are not the macros, but the %files
> > section. And it is hard to write a spec so both python 3.1 and 3.2 are
> > supported.
> 
> IMO we could assume that "python3" means ">= 3.2" and don't maintain
> support for 3.0/3.1.

I am all for it, but will other PLD repository users (Ti?) agree?

Greets,
        Jacek


From arekm at maven.pl  Thu Jul 14 21:05:34 2011
From: arekm at maven.pl (Arkadiusz Miskiewicz)
Date: Thu, 14 Jul 2011 21:05:34 +0200
Subject: *.py packaging, again
In-Reply-To: <20110714190014.GB4099@lolek.nigdzie>
References: <20110713110505.GA32005@jajo.eggsoft>
	<20110714172701.GP5300@stranger.qboosh.pl>
	<20110714190014.GB4099@lolek.nigdzie>
Message-ID: <201107142105.34715.arekm@maven.pl>

On Thursday 14 of July 2011, Jacek Konieczny wrote:
> On Thu, Jul 14, 2011 at 07:27:01PM +0200, Jakub Bogusz wrote:
> > > What is will not work in this case are not the macros, but the %files
> > > section. And it is hard to write a spec so both python 3.1 and 3.2 are
> > > supported.
> > 
> > IMO we could assume that "python3" means ">= 3.2" and don't maintain
> > support for 3.0/3.1.
> 
> I am all for it, but will other PLD repository users (Ti?) agree?

You won't have to worry about Ti - they will do as they wish regardless of 
what PLD does.

> Greets,
>         Jacek

-- 
Arkadiusz Mi?kiewicz        PLD/Linux Team
arekm / maven.pl            http://ftp.pld-linux.org/


From jajcus at jajcus.net  Thu Jul 14 21:18:02 2011
From: jajcus at jajcus.net (Jacek Konieczny)
Date: Thu, 14 Jul 2011 21:18:02 +0200
Subject: *.py packaging, again
In-Reply-To: <CA+92fu_xHpXxvQ1N8fSrzSp=tVLsv+guOEKc0=eNRqjHwcGu=Q@mail.gmail.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CA+92fu_xHpXxvQ1N8fSrzSp=tVLsv+guOEKc0=eNRqjHwcGu=Q@mail.gmail.com>
Message-ID: <20110714191802.GC4099@lolek.nigdzie>

On Thu, Jul 14, 2011 at 07:31:28PM +0100, Artur Wroblewski wrote:
> On Thu, Jul 14, 2011 at 8:44 AM, Jacek Konieczny <jajcus at jajcus.net> wrote:
> [...]
> > We lose:
> > - a little bit of the disk space
> > - some 'purity' some people see in not distributing 'sources'
> 
> IMHO, it was not about purity but quite practical aspect of
> file distribution (src vs. bin) - we treated byte compiled files
> for Java and Python the same way, no src allowed.

But it never was the same. *.py files are executable code, *.pyc is only
optimized version of these. *.java cannot be run without compilation.

> Few months ago we agreed on *.py distribution for Python 3.2 (IMHO,
> that implies we do not longer support 3.0 and 3.1 in python-*.spec).

This was not clear to me, otherwise I would have myself converted a few
packages until now.

> We should treat cpython and pypy as we would treat two compilers,
> which optimize code in different way. We will not build code with gcc
> and Intel compilers and put two versions of packages on ftp.

PyPy cannot replace CPython in every place, but can be useful for some
deployments to speed up some Python applications.

I think we should try to provide the PyPy itself and maybe a few basic
packages for it. E.g. Distribute, to have 'easy_install' working for
pypy. This way any other packages could be installed through
easy_install rather then from RPM ? not the way 'supported in PLD', but
would make PyPy from PLD useful for those who need it. Having the 'pypy'
binary handy would be good enough from PLD.

To sum up:

- there is no easy way and no point to provide all our Python 2.7
  packages for CPython and PyPy. We can still build some as pypy-*
  and put in /usr/lib/pypy-*/site-packages

- we stop supporting python 3.{0,1}, and package *.py with __pycache__/*
  for Python 3.2

- for Python 2.7 things stay the old way (though, I would prefer to stop
  the strict no *.py policy, without forcing the other way either)

- maybe, some day, RPM will manage the files in some other, smart way,
  but that is another discussion.

Greets,
        Jacek


From n3npq at mac.com  Thu Jul 14 21:26:07 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 15:26:07 -0400
Subject: *.py packaging, again
In-Reply-To: <CA+92fu_xHpXxvQ1N8fSrzSp=tVLsv+guOEKc0=eNRqjHwcGu=Q@mail.gmail.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CA+92fu_xHpXxvQ1N8fSrzSp=tVLsv+guOEKc0=eNRqjHwcGu=Q@mail.gmail.com>
Message-ID: <BD78497A-E3A6-4800-B802-A9C8977BB37F@mac.com>


On Jul 14, 2011, at 2:31 PM, Artur Wroblewski wrote:

> On Thu, Jul 14, 2011 at 8:44 AM, Jacek Konieczny <jajcus at jajcus.net> wrote:
> [...]
>> We lose:
>> - a little bit of the disk space
>> - some 'purity' some people see in not distributing 'sources'
> 
> IMHO, it was not about purity but quite practical aspect of
> file distribution (src vs. bin) - we treated byte compiled files
> for Java and Python the same way, no src allowed.
> 

All depends on what you want to emphasize in the comparison.

E.g. man pages are generated and cached and none is
asking for support from RPM for handling "ownership"
and integrity of the generated man pages. The ached aspects
of __pycache__ aren't that different (but are executable so
the audit is more complex than for data files).

73 de Jeff



From wrobell at pld-linux.org  Thu Jul 14 21:52:11 2011
From: wrobell at pld-linux.org (Artur Wroblewski)
Date: Thu, 14 Jul 2011 20:52:11 +0100
Subject: *.py packaging, again
In-Reply-To: <BD78497A-E3A6-4800-B802-A9C8977BB37F@mac.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CA+92fu_xHpXxvQ1N8fSrzSp=tVLsv+guOEKc0=eNRqjHwcGu=Q@mail.gmail.com>
	<BD78497A-E3A6-4800-B802-A9C8977BB37F@mac.com>
Message-ID: <CA+92fu9ibDkBrFDSSirte925eTchJ12doed0Oguw9_=3-M0rUw@mail.gmail.com>

On Thu, Jul 14, 2011 at 8:26 PM, Jeff Johnson <n3npq at mac.com> wrote:
>
> On Jul 14, 2011, at 2:31 PM, Artur Wroblewski wrote:
>
>> On Thu, Jul 14, 2011 at 8:44 AM, Jacek Konieczny <jajcus at jajcus.net> wrote:
>> [...]
>>> We lose:
>>> - a little bit of the disk space
>>> - some 'purity' some people see in not distributing 'sources'
>>
>> IMHO, it was not about purity but quite practical aspect of
>> file distribution (src vs. bin) - we treated byte compiled files
>> for Java and Python the same way, no src allowed.
>>
>
> All depends on what you want to emphasize in the comparison.
>
> E.g. man pages are generated and cached and none is
> asking for support from RPM for handling "ownership"
> and integrity of the generated man pages. The ached aspects
> of __pycache__ aren't that different (but are executable so
> the audit is more complex than for data files).

Sure. I just re-stated our approach we took in the past.

Best regards,

w


From n3npq at mac.com  Thu Jul 14 22:51:07 2011
From: n3npq at mac.com (Jeff Johnson)
Date: Thu, 14 Jul 2011 16:51:07 -0400
Subject: *.py packaging, again
In-Reply-To: <CA+92fu9ibDkBrFDSSirte925eTchJ12doed0Oguw9_=3-M0rUw@mail.gmail.com>
References: <20110713110505.GA32005@jajo.eggsoft> <20110713130707.GA5995@mail>
	<20110714074401.GB6660@jajo.eggsoft>
	<CA+92fu_xHpXxvQ1N8fSrzSp=tVLsv+guOEKc0=eNRqjHwcGu=Q@mail.gmail.com>
	<BD78497A-E3A6-4800-B802-A9C8977BB37F@mac.com>
	<CA+92fu9ibDkBrFDSSirte925eTchJ12doed0Oguw9_=3-M0rUw@mail.gmail.com>
Message-ID: <CF699F22-2986-43B8-A35D-A68601B93AA2@mac.com>


On Jul 14, 2011, at 3:52 PM, Artur Wroblewski wrote:

> On Thu, Jul 14, 2011 at 8:26 PM, Jeff Johnson <n3npq at mac.com> wrote:
>> 
>> On Jul 14, 2011, at 2:31 PM, Artur Wroblewski wrote:
>> 
>>> On Thu, Jul 14, 2011 at 8:44 AM, Jacek Konieczny <jajcus at jajcus.net> wrote:
>>> [...]
>>>> We lose:
>>>> - a little bit of the disk space
>>>> - some 'purity' some people see in not distributing 'sources'
>>> 
>>> IMHO, it was not about purity but quite practical aspect of
>>> file distribution (src vs. bin) - we treated byte compiled files
>>> for Java and Python the same way, no src allowed.
>>> 
>> 
>> All depends on what you want to emphasize in the comparison.
>> 
>> E.g. man pages are generated and cached and none is
>> asking for support from RPM for handling "ownership"
>> and integrity of the generated man pages. The ached aspects
>> of __pycache__ aren't that different (but are executable so
>> the audit is more complex than for data files).
> 
> Sure. I just re-stated our approach we took in the past.
> 

And no offense intended: truly there's a balance point to
handle __pycache__ registration here somehow/somwhere.

I merely wished to point out one of the extrema constraints:

	Don't bother handling __pycache__ at all, treat like
	cached man pages.

I don't know what the right/best answer is. I do trust PLD will
find a sane answer first. There's no other distro around attempting
python-3.x packaging that I've heard/seen yet.

73 de Jeff
> Best regards,
> 
> w
> _______________________________________________
> pld-devel-en mailing list
> pld-devel-en at lists.pld-linux.org
> http://lists.pld-linux.org/mailman/listinfo/pld-devel-en



From jajcus at jajcus.net  Wed Jul 20 12:30:59 2011
From: jajcus at jajcus.net (Jacek Konieczny)
Date: Wed, 20 Jul 2011 12:30:59 +0200
Subject: ftp.pld-linux.org unreachable via IPv6
Message-ID: <20110720103059.GC18503@jajo.eggsoft>


ftp.pld-linux.org resovles to IPv4 and IPv6 addresses:

# host ftp.pld-linux.org
ftp.pld-linux.org is an alias for ftp3.pld-linux.org.
ftp3.pld-linux.org is an alias for ftp.agmk.pld-linux.org.
ftp.agmk.pld-linux.org has address 91.192.224.120
ftp.agmk.pld-linux.org has IPv6 address 2001:6a0:159::120


And the IPv6 address is unreachable via a sixxs tunnel:

# mtr --report 2001:6a0:159::120
HOST: jajo                        Loss%   Snt   Rcv   Last   Avg  Best  Wrst StDev
  1.|-- XXXXX.pl                   0.0%    10    10    0.4   0.7   0.2   4.2   1.2
  2.|-- gw-640.waw-01.pl.sixxs.ne  0.0%    10    10    8.8   8.9   8.4   9.2   0.2
  3.|-- cl-36.waw-01.pl.sixxs.net  0.0%    10    10    9.7  10.0   9.4  12.3   0.9
  4.|-- ???                       100.0    10     0    0.0   0.0   0.0   0.0   0.0

And from the HE tunnel too:

# mtr --report 2001:6a0:159::120
HOST: tropek                      Loss%   Snt   Rcv   Last   Avg  Best  Wrst StDev
  1.|-- giuseppedb-1.tunnel.tserv  0.0%    10    10   44.4  45.5  43.1  52.5   2.7
  2.|-- gige-g2-4.core1.fra1.he.n  0.0%    10    10   34.6  36.0  34.3  40.8   2.3
  3.|-- as1299.gige-g2-9.core1.fr  0.0%    10    10   34.5  39.6  34.2  69.3  10.9
  4.|-- ffm-b7-v6.telia.net        0.0%    10    10   35.6  35.2  34.4  36.0   0.5
  5.|-- dante-ic-125713-mno-b1.c.  0.0%    10    10   51.9  46.1  43.5  51.9   2.9
  6.|-- as0.rt1.gen.ch.geant2.net  0.0%    10    10   47.2  47.7  46.7  49.7   0.8
  7.|-- so-4-0-0.rt1.fra.de.geant  0.0%    10    10   47.4  48.2  46.6  56.7   3.0
  8.|-- so-1-0-0.rt1.poz.pl.geant  0.0%    10    10   60.3  64.5  59.4  88.2   9.2
  9.|-- pionier-bckp-gw.rt1.poz.p  0.0%    10    10   78.2  79.2  77.7  85.4   2.3
 10.|-- 2001:b10:c000:204::2       0.0%    10    10   83.2  84.0  82.8  86.7   1.4
 11.|-- 2001:6a0:1:1::2            0.0%    10    10   85.6  83.7  83.1  85.6   0.7
 12.|-- cl-36.waw-01.pl.sixxs.net  0.0%    10    10   86.5  85.1  83.8  88.5   1.4
 13.|-- ???                       100.0    10     0    0.0   0.0   0.0   0.0   0.0


IPv6 connectivity should be fixed or the AAAA record removed, otherwise one
needs to force IPv4 connection to install packages.

I have been broken for a few days.

Greets,
        Jacek


From caleb at pld-linux.org  Sat Jul 23 11:00:58 2011
From: caleb at pld-linux.org (Caleb Maclennan)
Date: Sat, 23 Jul 2011 12:00:58 +0300
Subject: DISTFILES: pacgraph: ERRORS: keenerd-pacgraph-e495c03.tar.gz
In-Reply-To: <CAP9w9P7EUiXOJnTi7u0Nji_1wDDghZ7YoegBSg01PxCuwvkzRQ@mail.gmail.com>
References: <2918.1311410124@distfiles.pld-linux.org>
	<CAP9w9P7EUiXOJnTi7u0Nji_1wDDghZ7YoegBSg01PxCuwvkzRQ@mail.gmail.com>
Message-ID: <CAP9w9P7hUiT-LHLtR6tZGGYCiPO-yOJV1MewTKg-qT9tOxbOKg@mail.gmail.com>

On Sat, Jul 23, 2011 at 11:35, caleb <caleb at pld-linux.org> wrote:
> wget -nv --no-check-certificate --user-agent=PLD/distfiles -O ./tmp/9c4646a0-f84d-45c6-9e96-8c9a251e6297/fdf09c8cdea9c9e58c6fc2acc79a0528/keenerd-pacgraph-e495c03.tar.gz "https://download.github.com/keenerd-pacgraph-e495c03.tar.gz":
> https://download.github.com/keenerd-pacgraph-e495c03.tar.gz:
> 2011-07-23 10:35:24 ERROR 404: Not Found.

What's the proper way to specify a source tarball from github at a
specific commit? The URL in this package works, but seemingly only
after you hit the trunk download url:

https://github.com/keenerd/pacgraph/tarball/master

Caleb


From glen at pld-linux.org  Mon Jul 25 10:10:02 2011
From: glen at pld-linux.org (=?ISO-8859-1?Q?Elan_Ruusam=E4e?=)
Date: Mon, 25 Jul 2011 11:10:02 +0300
Subject: DISTFILES: pacgraph: ERRORS: keenerd-pacgraph-e495c03.tar.gz
In-Reply-To: <CAP9w9P7hUiT-LHLtR6tZGGYCiPO-yOJV1MewTKg-qT9tOxbOKg@mail.gmail.com>
References: <2918.1311410124@distfiles.pld-linux.org>	<CAP9w9P7EUiXOJnTi7u0Nji_1wDDghZ7YoegBSg01PxCuwvkzRQ@mail.gmail.com>
	<CAP9w9P7hUiT-LHLtR6tZGGYCiPO-yOJV1MewTKg-qT9tOxbOKg@mail.gmail.com>
Message-ID: <4E2D24DA.3090906@pld-linux.org>

On 23.07.2011 12:00, Caleb Maclennan wrote:
> On Sat, Jul 23, 2011 at 11:35, caleb<caleb at pld-linux.org>  wrote:
>> wget -nv --no-check-certificate --user-agent=PLD/distfiles -O ./tmp/9c4646a0-f84d-45c6-9e96-8c9a251e6297/fdf09c8cdea9c9e58c6fc2acc79a0528/keenerd-pacgraph-e495c03.tar.gz "https://download.github.com/keenerd-pacgraph-e495c03.tar.gz":
>> https://download.github.com/keenerd-pacgraph-e495c03.tar.gz:
>> 2011-07-23 10:35:24 ERROR 404: Not Found.
> What's the proper way to specify a source tarball from github at a
> specific commit? The URL in this package works, but seemingly only
> after you hit the trunk download url:
>
> https://github.com/keenerd/pacgraph/tarball/master

try to fetch the url just before commit of spec to hit file from cache

or just use our distfiles upload...

-- 
glen



From caleb at pld-linux.org  Mon Jul 25 10:32:32 2011
From: caleb at pld-linux.org (Caleb Maclennan)
Date: Mon, 25 Jul 2011 11:32:32 +0300
Subject: DISTFILES: pacgraph: ERRORS: keenerd-pacgraph-e495c03.tar.gz
In-Reply-To: <4E2D24DA.3090906@pld-linux.org>
References: <2918.1311410124@distfiles.pld-linux.org>
	<CAP9w9P7EUiXOJnTi7u0Nji_1wDDghZ7YoegBSg01PxCuwvkzRQ@mail.gmail.com>
	<CAP9w9P7hUiT-LHLtR6tZGGYCiPO-yOJV1MewTKg-qT9tOxbOKg@mail.gmail.com>
	<4E2D24DA.3090906@pld-linux.org>
Message-ID: <CAP9w9P54Q-=69Zbc41PUCpv7uNQY1VbDsic4QiBJYWiOcVYWfg@mail.gmail.com>

2011/7/25 Elan Ruusam?e <glen at pld-linux.org>:
> or just use our distfiles upload...

Would you believe I don't know how to do that?


From adamg at pld-linux.org  Mon Jul 25 13:18:45 2011
From: adamg at pld-linux.org (Adam Golebiowski)
Date: Mon, 25 Jul 2011 13:18:45 +0200
Subject: DISTFILES: pacgraph: ERRORS: keenerd-pacgraph-e495c03.tar.gz
In-Reply-To: <CAP9w9P54Q-=69Zbc41PUCpv7uNQY1VbDsic4QiBJYWiOcVYWfg@mail.gmail.com>
References: <2918.1311410124@distfiles.pld-linux.org>
	<CAP9w9P7EUiXOJnTi7u0Nji_1wDDghZ7YoegBSg01PxCuwvkzRQ@mail.gmail.com>
	<CAP9w9P7hUiT-LHLtR6tZGGYCiPO-yOJV1MewTKg-qT9tOxbOKg@mail.gmail.com>
	<4E2D24DA.3090906@pld-linux.org>
	<CAP9w9P54Q-=69Zbc41PUCpv7uNQY1VbDsic4QiBJYWiOcVYWfg@mail.gmail.com>
Message-ID: <4E2D5115.3060207@pld-linux.org>

W dniu 2011-07-25 10:32, Caleb Maclennan pisze:
> 2011/7/25 Elan Ruusam?e<glen at pld-linux.org>:
>> or just use our distfiles upload...
>
> Would you believe I don't know how to do that?

upload it to ftp://dropin.pld-linux.org/ (using your cvs 
username/password), then change sourceX from http://something/file to 
Source: file and commit. Distfiles will pick it up from dropin

-- 
adamg at pld-linux.org