From n3npq at me.com  Wed Mar  1 05:18:01 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Tue, 28 Feb 2017 23:18:01 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <20170228204233.GA12663@mail>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail>
Message-ID: <47469D41-43B2-42A4-953D-021F837F4629@me.com>


> On Feb 28, 2017, at 3:42 PM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
> 
> On Thu, Feb 23, 2017 at 07:15:22PM -0500, Jeffrey Johnson wrote:
>> 
>>> On Feb 23, 2017, at 3:36 PM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
>>> 
>>> On Thu, Feb 23, 2017 at 01:43:14PM -0500, Jeffrey Johnson wrote:
>>>> This one is left though:
>>>> 
>>>>> error: db3: header #187105280 cannot be loaded -- skipping.
>>>>> error: db3: header #4127850496 cannot be loaded -- skipping.
>>>> 
>>>> How to check what these "headers" mean?
>>>> (old, unsupported keys? some old packages with missing fields which are
>>>> now required?)
>>>> The error message is printed on a headerCopyLoad() failure.
>>>> 
>>>> The failure is usually an indication of header damage of some sort, not missing fields now required.
>>>> 
>>>> The numbers  are  primary keys into Packages printed in in decimal.
>>>> 
>>>> The values appear to be in the wrong-endian order when converted to hex
>>>> 	0xB270000
>>>> 	0xF60A0000
>>>> 
>>>> You can try finding the header by doing, say, rpm -q ???querybynumber 0xB270000
>>>> (or its reverse: its unclear what order).
>>> 
>>> Same result for bigger values, no result after swapping bytes:
>>> 
>>> -bash-4.4# rpm -q --querybynumber 0xF60A0000
>>> error: rpmdb: header #4127850496 cannot be loaded -- skipping.
>>> -bash-4.4# rpm -q --querybynumber 0x00000AF6
>>> -bash-4.4# rpm -q --querybynumber 0xB270000
>>> error: rpmdb: header #187105280 cannot be loaded -- skipping.
>>> -bash-4.4# rpm -q --querybynumber 0x0000270B
>>> -bash-4.4#
>>> 
>> 
>> So at least simply reproducible with ???querybynumber ;-)
>> 
>> If you just need a fix, db_dump will give you {KEY, VALUE} pairs in hex.
>> Find the 2 {KEY,VALUES} and delete the 2 occurrences of 2 lines of hex in a text editor.
>> Feed the result to db_load to recreate Packages.
>> rpm ???rebuild will recreate the indices.
> 
> I investigated these two entries. They doesn't seem to be invalid, but
> are not accepted after this change:
> 
> http://rpm5.org/cvs/filediff?f=rpm/rpmdb/header.c&v1=1.198.2.23&v2=1.198.2.24 <http://rpm5.org/cvs/filediff?f=rpm/rpmdb/header.c&v1=1.198.2.23&v2=1.198.2.24>
> 
> more precisely, this part:
> 
> -/*@=sizeoftype@*/
> -               rpmuint32_t * stei = (rpmuint32_t *)
> -                       memcpy(alloca(nb), dataStart + off, nb);
> +               /* XXX copy to fix alignment problems */
> +                rpmuint32_t * stei = (rpmuint32_t *)
> +                          memcpy(alloca(nb), dataStart + off, nb);
> +               if ((off + nb) > dl)
> +                   goto errxit;
>                rdl = (rpmuint32_t)-ntohl(stei[2]);     /* negative offset */
> -assert((rpmint32_t)rdl >= 0);  /* XXX insurance */
> -               ril = (rpmuint32_t)(rdl/sizeof(*pe));
> -               if (hdrchkTags(ril) || hdrchkData(rdl))
> +               if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb))
>                    goto errxit;
> 
> (nb. IMO "if ((off + nb) > dl)" check should be done before memcpy?)

Ah yes ? fixing problems with AFL (American Fuzzy Lop) was totally annoying there.
You can tell because of the remnant printf?s that do not align with generally accepted
C coding practices.

Meanwhile, we may be working at cross purposes, with packages that have already been installed
by older (or heavily patched) versions of rpm.

What SHOULD have prevented that problem is verifying (always) the header SHA1 while installing
which ensures the header was structurally intact, preventing the later failure of headerCopyLoad() that
was displayed as an error message.

But perhaps there is something better that can be done: O looked at that specific code path
for a couple of days with AFL fuzzed headers: what you see is my best effort (by _ALWAYS_
verifying the header SHA1. The risk of adding stricter tests is that previously known good
headers might fail.

If you have a better solution (other than disabling stricter tests), I?ll be happy to add to

> 1) package #0xB270000 the first entry of header is:
> 0000003f 00000007 00000280 00000010
> it points to data:
> dataStart+0280  0000003f 00000007 fffffca0 00000010
> so rdl=0x360, which is bigger than off+nb (0x280+0x10)
> but AFAICS rdl is pointer inside header part, not data?
> 

This is broken.

0x3f is a HEADER_IMMUTABLE marker whose length indicates the
part of a header that never (i.e. same in *.rpm and in rpmdb) changes, and is matched
by a trailer that duplicates the size of a the ?immutable region?.

I cannot tell how that header was produced, nor guess if there was a big in some version of RPM,
without further information.


> 2) similarly, package #0xF60A0000 has first header entry:
> 0000003f 00000007 00000378 00000010
> which points to data:
> 
> dataStart+0378 0000003f 00000007 fffffc70 00000010
> and rdl=0x390 > 0x378+0x10
> 
> The rest of packages I've got installed seem to have longer data in
> "immutable" part and pass this check.
> 

Its always possible that there is some obscure bug in some version of RPM.

If you can pin down the value of RPMTAG_RPMVBERSION (and possibly associated patches),
I can try to do the necessary forensics.

Thanks for the report: I am not infallible (but have run like 30 billion tests over several weeks with AFL
fuzzed headers trying to ensure that rpm-5.4.17 is as fault free as possible).

hah

73 de Jeff
> 
> -- 
> Jakub Bogusz    http://qboosh.pl/ <http://qboosh.pl/>
> _______________________________________________
> pld-devel-en mailing list
> pld-devel-en at lists.pld-linux.org <mailto:pld-devel-en at lists.pld-linux.org>
> http://lists.pld-linux.org/mailman/listinfo/pld-devel-en <http://lists.pld-linux.org/mailman/listinfo/pld-devel-en>


From n3npq at me.com  Thu Mar  2 18:49:20 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Thu, 02 Mar 2017 12:49:20 -0500
Subject: PLD *.src.rpm cannot be installed by rpm-4.13
Message-ID: <68CD3568-3314-47DA-BC3D-53B4A679519E@me.com>

Here is the RHBZ bug report
	https://bugzilla.redhat.com/show_bug.cgi?id=1427287

This issue has been discussed ad nauseum back in 2009.

PLD has 2 choices:

1) Retrofit RPMTAG_SOURCEPACKAGE and rpmlead boolean
into RPM5 and rebuild everything

OR

2) patch rpm.org rpm-4.13 (and perhaps older) with a couple line patch
to recognize an RPM5 *.src.rpm (which rpm.org flatly refuses to do
	pmatilai <member:pmatilai>: rpm 5 interoperability is not a goal, at all:rpm 5 interoperability is not a goal, at all

hth

73 de Jeff



From qboosh at pld-linux.org  Thu Mar  2 21:52:52 2017
From: qboosh at pld-linux.org (Jakub Bogusz)
Date: Thu, 2 Mar 2017 21:52:52 +0100
Subject: pld rpm 5.4.17
In-Reply-To: <47469D41-43B2-42A4-953D-021F837F4629@me.com>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
Message-ID: <20170302205252.GA852@mail>

On Tue, Feb 28, 2017 at 11:18:01PM -0500, Jeffrey Johnson wrote:
> 
> > On Feb 28, 2017, at 3:42 PM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
> > 
> > On Thu, Feb 23, 2017 at 07:15:22PM -0500, Jeffrey Johnson wrote:
> >> 
> >>> On Feb 23, 2017, at 3:36 PM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
> >>> 
> >>> On Thu, Feb 23, 2017 at 01:43:14PM -0500, Jeffrey Johnson wrote:
> >>>> This one is left though:
> >>>> 
> >>>>> error: db3: header #187105280 cannot be loaded -- skipping.
> >>>>> error: db3: header #4127850496 cannot be loaded -- skipping.
> >>>> 
> >>>> How to check what these "headers" mean?
> >>>> (old, unsupported keys? some old packages with missing fields which are
> >>>> now required?)
> >>>> The error message is printed on a headerCopyLoad() failure.
> >>>> 
> >>>> The failure is usually an indication of header damage of some sort, not missing fields now required.
> >>>> 
> >>>> The numbers  are  primary keys into Packages printed in in decimal.
> >>>> 
> >>>> The values appear to be in the wrong-endian order when converted to hex
> >>>> 	0xB270000
> >>>> 	0xF60A0000
> >>>> 
> >>>> You can try finding the header by doing, say, rpm -q ???querybynumber 0xB270000
> >>>> (or its reverse: its unclear what order).
> >>> 
> >>> Same result for bigger values, no result after swapping bytes:
> >>> 
> >>> -bash-4.4# rpm -q --querybynumber 0xF60A0000
> >>> error: rpmdb: header #4127850496 cannot be loaded -- skipping.
> >>> -bash-4.4# rpm -q --querybynumber 0x00000AF6
> >>> -bash-4.4# rpm -q --querybynumber 0xB270000
> >>> error: rpmdb: header #187105280 cannot be loaded -- skipping.
> >>> -bash-4.4# rpm -q --querybynumber 0x0000270B
> >>> -bash-4.4#
> >>> 
> >> 
> >> So at least simply reproducible with ???querybynumber ;-)
> >> 
> >> If you just need a fix, db_dump will give you {KEY, VALUE} pairs in hex.
> >> Find the 2 {KEY,VALUES} and delete the 2 occurrences of 2 lines of hex in a text editor.
> >> Feed the result to db_load to recreate Packages.
> >> rpm ???rebuild will recreate the indices.
> > 
> > I investigated these two entries. They doesn't seem to be invalid, but
> > are not accepted after this change:
> > 
> > http://rpm5.org/cvs/filediff?f=rpm/rpmdb/header.c&v1=1.198.2.23&v2=1.198.2.24 <http://rpm5.org/cvs/filediff?f=rpm/rpmdb/header.c&v1=1.198.2.23&v2=1.198.2.24>
> > 
> > more precisely, this part:
> > 
> > -/*@=sizeoftype@*/
> > -               rpmuint32_t * stei = (rpmuint32_t *)
> > -                       memcpy(alloca(nb), dataStart + off, nb);
> > +               /* XXX copy to fix alignment problems */
> > +                rpmuint32_t * stei = (rpmuint32_t *)
> > +                          memcpy(alloca(nb), dataStart + off, nb);
> > +               if ((off + nb) > dl)
> > +                   goto errxit;
> >                rdl = (rpmuint32_t)-ntohl(stei[2]);     /* negative offset */
> > -assert((rpmint32_t)rdl >= 0);  /* XXX insurance */
> > -               ril = (rpmuint32_t)(rdl/sizeof(*pe));
> > -               if (hdrchkTags(ril) || hdrchkData(rdl))
> > +               if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb))
> >                    goto errxit;
> > 
> > (nb. IMO "if ((off + nb) > dl)" check should be done before memcpy???)
> 
> Ah yes ??? fixing problems with AFL (American Fuzzy Lop) was totally annoying there.
> You can tell because of the remnant printf???s that do not align with generally accepted
> C coding practices.
> 
> Meanwhile, we may be working at cross purposes, with packages that have already been installed
> by older (or heavily patched) versions of rpm.
> 
> What SHOULD have prevented that problem is verifying (always) the header SHA1 while installing
> which ensures the header was structurally intact, preventing the later failure of headerCopyLoad() that
> was displayed as an error message.
> 
> But perhaps there is something better that can be done: O looked at that specific code path
> for a couple of days with AFL fuzzed headers: what you see is my best effort (by _ALWAYS_
> verifying the header SHA1. The risk of adding stricter tests is that previously known good
> headers might fail.
> 
> If you have a better solution (other than disabling stricter tests), I???ll be happy to add to

As far as I understand the code, rdl is size of immutable entry infos
part, while off is an offset in tags data part.
And when immutable tags data is short enough (shorter than entry infos
of immutable part), this check refuses to load header.

IMO the checks should be like in the attached patch.
With it, the two refused packages are accessible again.


-- 
Jakub Bogusz    http://qboosh.pl/
-------------- next part --------------
--- rpm-5.4.17/rpmdb/header.c.orig	2017-02-25 09:37:52.627550403 +0100
+++ rpm-5.4.17/rpmdb/header.c	2017-03-02 21:12:16.348808677 +0100
@@ -998,14 +998,15 @@
 	    if (off < 0)
 		goto errxit;
 	    if (off) {
+                rpmuint32_t * stei;
 		size_t nb = REGION_TAG_COUNT;
-		/* XXX copy to fix alignment problems */
-                rpmuint32_t * stei = (rpmuint32_t *)
-                          memcpy(alloca(nb), dataStart + off, nb);
 		if ((off + nb) > dl)
 		    goto errxit;
+		/* XXX copy to fix alignment problems */
+                stei = (rpmuint32_t *)
+                          memcpy(alloca(nb), dataStart + off, nb);
 		rdl = (rpmuint32_t)-ntohl(stei[2]);	/* negative offset */
-		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb))
+		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(il * REGION_TAG_COUNT))
 		    goto errxit;
 		ril = (rpmuint32_t)(rdl/sizeof(*pe));
 	    } else {

From n3npq at me.com  Thu Mar  2 22:05:49 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Thu, 02 Mar 2017 16:05:49 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <20170302205252.GA852@mail>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail>
Message-ID: <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>


> On Mar 2, 2017, at 3:52 PM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
> 
> 
> As far as I understand the code, rdl is size of immutable entry infos
> part, while off is an offset in tags data part.
> And when immutable tags data is short enough (shorter than entry infos
> of immutable part), this check refuses to load header.
> 

Yes. there is a ?immutable region? header and trailer, where the
offset field is used as a double check on the tags in the immutable region.

> IMO the checks should be like in the attached patch.
> With it, the two refused packages are accessible again.
> 

I?ve applied the patch and will do a few tests before checking in.

One item I note (just scanning the patch) is

-		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb))
+		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(il * REGION_TAG_COUNT))

The variable il is derived and may be tainted, while off and nb are de facto positioning
within the header memory blob. And yes, it may not matter.
Meanwhile the entire issue is rather obscure, and only testing will tell.
Is there any information about what header?s are failing headerCopyLoad()?
If those headers are public keys, then the real flaw is elsewhere, wrapping
a public key within an immutable region, with an appended SHA1.

Thanks for the patch.

73 de Jeff



From qboosh at pld-linux.org  Sat Mar  4 10:17:26 2017
From: qboosh at pld-linux.org (Jakub Bogusz)
Date: Sat, 4 Mar 2017 10:17:26 +0100
Subject: pld rpm 5.4.17
In-Reply-To: <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
Message-ID: <20170304091726.GA12545@mail>

On Thu, Mar 02, 2017 at 04:05:49PM -0500, Jeffrey Johnson wrote:
> 
> > On Mar 2, 2017, at 3:52 PM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
> > 
> > 
> > As far as I understand the code, rdl is size of immutable entry infos
> > part, while off is an offset in tags data part.
> > And when immutable tags data is short enough (shorter than entry infos
> > of immutable part), this check refuses to load header.
> > 
> 
> Yes. there is a ???immutable region??? header and trailer, where the
> offset field is used as a double check on the tags in the immutable region.
> 
> > IMO the checks should be like in the attached patch.
> > With it, the two refused packages are accessible again.
> > 
> 
> I???ve applied the patch and will do a few tests before checking in.
> 
> One item I note (just scanning the patch) is
> 
> -		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb))
> +		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(il * REGION_TAG_COUNT))
> 
> The variable il is derived and may be tainted, while off and nb are de facto positioning
> within the header memory blob. And yes, it may not matter.

il is already used earlier to calculate dataStart. And length of the
whole data (pvlen).

> Meanwhile the entire issue is rather obscure, and only testing will tell.
> Is there any information about what header???s are failing headerCopyLoad()?
> If those headers are public keys, then the real flaw is elsewhere, wrapping
> a public key within an immutable region, with an appended SHA1.

No, these are two packages.
I'm attaching whole db data of one of them (partially described by me
during investigation).


-- 
Jakub Bogusz    http://qboosh.pl/
-------------- next part --------------
* index 0b270000:
00000043 il
00000384 dl

0000003f 00000007 00000280 00000010	HEADER_IMMUTABLE	REGION_TAG_TYPE=RPM_BIN_TYPE
00000064 00000008 00000000 00000002	HEADER_I18NTABLE	RPM_STRING_ARRAY_TYPE
000003e8 00000006 0000000b 00000001	RPMTAG_NAME		RPM_STRING_TYPE
000003e9 00000006 00000019 00000001	RPMTAG_VERSION		RPM_STRING_TYPE
000003ea 00000006 00000020 00000001
000003eb 00000004 00000024 00000001
000003ec 00000009 00000028 00000002
000003ed 00000009 00000058 00000002
000003ee 00000004 00000090 00000001
000003ef 00000006 00000094 00000001
000003f1 00000004 000000a8 00000001
000003f2 00000006 000000ac 00000001
000003f6 00000006 000000b0 00000001
000003f7 00000006 000000b4 00000001
000003f8 00000009 000000c1 00000001
000003fc 00000006 000000d7 00000001
000003fd 00000006 000000f6 00000001
000003fe 00000006 000000fc 00000001
00000404 00000004 00000104 00000001
00000406 00000003 00000108 00000001
00000409 00000003 0000010a 00000001
0000040a 00000004 0000010c 00000001
0000040b 00000008 00000110 00000001
0000040c 00000008 00000131 00000001
0000040d 00000004 00000134 00000001
0000040f 00000008 00000138 00000001
00000410 00000008 0000013d 00000001
00000414 00000006 00000142 00000001
00000415 00000004 0000015c 00000001
00000417 00000008 00000160 00000001
00000418 00000004 00000170 00000002
00000419 00000008 00000178 00000002
0000041a 00000008 0000019b 00000002
00000428 00000006 000001ac 00000001
00000447 00000004 000001b0 00000001
00000448 00000004 000001b4 00000001
00000449 00000008 000001b8 00000001
00000458 00000004 000001bc 00000001
00000459 00000008 000001c0 00000001
0000045c 00000004 000001cc 00000001
0000045d 00000008 000001d0 00000001
0000045e 00000008 000001da 00000001
00000462 00000006 000001e8 00000001
00000464 00000006 0000022b 00000001
00000465 00000006 00000230 00000001
00000466 00000006 00000235 00000001
0000046c 00000006 00000237 00000001
00000474 00000004 00000248 00000001
00000475 00000004 0000024c 00000001
00000476 00000008 00000250 00000003
00000477 00000004 00000270 00000001
00000478 00000004 00000274 00000001
0000047b 00000008 00000278 00000001
00000499 00000004 0000027c 00000001	RPMTAG_FILEDIGESTALGOS	RPM_UINT32_TYPE
-- immutable end
00000101 00000004 00000290 00000001	RPMTAG_SIGSIZE
00000105 00000007 00000294 00000010	RPMTAG_SIGMD5
0000010d 00000006 000002a4 00000001	RPMTAG_SHA1HEADER
000003f0 00000004 000002d0 00000001	RPMTAG_INSTALLTIME
00000405 00000001 000002d4 00000001	RPMTAG_FILESTATES
00000416 00000004 000002d8 00000001	RPMTAG_ARCHIVESIZE
00000467 00000004 000002dc 00000001	RPMTAG_INSTALLCOLOR
00000468 00000004 000002e0 00000001	RPMTAG_INSTALLTID
0000048c 00000008 000002e4 00000001	RPMTAG_BLINKPKGID
0000048d 00000008 00000305 00000001	RPMTAG_BLINKHDRID
0000048e 00000008 0000032e 00000001	RPMTAG_BLINKNEVRA
00000492 00000006 0000034a 00000001	RPMTAG_PACKAGEORIGIN
000004a0 00000004 00000380 00000001	RPMTAG_PACKAGECOLOR

dataStart
d+0000	4300706c2e5554462d3800
d+000b	746f6c75612b2b2d646576656c00
d+0019	312e302e393300
d+0020	35000000
d+0024	00000000
d+0028	746f6c75612b2b206865616465722066696c657300506c696b69206e6167c582c3b3776b6f776520746f6c75612b2b00
d+0058	4865616465722066696c657320666f7220746f6c75612b2b2e00506c696b69206e6167c582c3b3776b6f776520746f6c75612b2b2e000000
d+0090	50180128
d+0094	737472616e6765722e71626f6f73682e706c0000
d+00a8	00001c20
d+00ac	504c4400
d+00b0	4d495400
d+00b4	4a616b756220426f6775737a00
d+00c1	446576656c6f706d656e742f4c696272617269657300
d+00d7	687474703a2f2f7777772e636f64656e69782e636f6d2f7e746f6c75612f00
d+00f6	6c696e757800
d+00fc	6936383600000000
d+0104	00001c20
d+0108	81a4
d+010a	0000
d+010c	50180127
d+0110	636235333237613036333131356161663335306339326438653931633261386600
d+0131	000000
d+0134	00000000
d+0138	726f6f7400
d+013d	726f6f7400
d+0142	746f6c75612b2b2d312e302e39332d352e7372632e72706d0000
d+015c	ffffffff
d+0160	746f6c75612b2b2d646576656c000000
d+0170	000000080100000a
d+0178	746f6c75612b2b2d6c6962730072706d6c6962285061796c6f616449734c7a6d612900
d+019b	312e302e39332d3500342e342e362d3100
d+01ac	342e3500
d+01b0	00000803
d+01b4	40366500
d+01b8	00000000
d+01bc	00000008
d+01c0	303a312e302e39332d350000
d+01cc	00000000
d+01d0	746f6c75612b2b2e6800
d+01da	2f7573722f696e636c7564652f00
d+01e8	2d4f32202d70697065202d666e6f2d7374726963742d616c696173696e67202d667772617076202d6d617263683d69363836202d6d74756e653d70656e7469756d3400
d+022b	6370696f00
d+0230	6c7a6d6100
d+0235	3900
d+0237	693638362d706c642d6c696e7578000000
d+0248	00000000
d+024c	00000001
d+0250	004320736f757263652c2041534349492074657874006469726563746f727900
d+0270	00000000
d+0274	00000000
d+0278	00000000
d+027c	00000001
-- immutable trailer
d+0280	0000003f 00000007 fffffca0 00000010
-- immutable end
d+0290	00000c11
d+0294	c8a786a05dc5214888b2ff2ae38a6147
d+02a4	61303336323662323336313030633035346134363334353931616161363263343436613038663430000000005018
d+02d0	01c70000
d+02d4	00000000
d+02d8	1d240000
d+02dc	00035018
d+02e0	01c33166
d+02e4	35383066316336383866306466393866353662646364363431626432333900
d+0305	3435393438653936306330623837336332666131633433633537653834346139653233346239323600
d+032e	746f6c75612b2b2d646576656c2d312e302e39332d342e6936383600
d+034a	2f686f6d652f636f6d702f72706d2f52504d532f746f6c75612b2b2d646576656c2d312e302e39332d352e693638362e72706d000000
d+0380	00000000
dataEnd=dataStart+0384

From qboosh at pld-linux.org  Sat Mar  4 10:30:26 2017
From: qboosh at pld-linux.org (Jakub Bogusz)
Date: Sat, 4 Mar 2017 10:30:26 +0100
Subject: pld rpm 5.4.17
In-Reply-To: <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
Message-ID: <20170304093026.GB12545@mail>

And one more patch, to make rpm4compat.h header usable again.
fdSize() became exported function, so stop defining static function of
the same name.


-- 
Jakub Bogusz    http://qboosh.pl/
-------------- next part --------------
fdSize is now extern function, declared in <rpmio.h>.
--- rpm-5.4.17/lib/rpm4compat.h.orig	2017-03-02 21:47:58.792117545 +0100
+++ rpm-5.4.17/lib/rpm4compat.h	2017-03-04 09:14:59.103993683 +0100
@@ -352,13 +352,6 @@
 	return rpmProblemGetDiskNeed(prob);
 }
 
-static inline off_t fdSize(FD_t fd){
-	struct stat sb;
-	Fstat(fd, &sb);
-	return sb.st_size;
-}
-
-
 static inline unsigned int rpmdbGetIteratorOffset(rpmdbMatchIterator mi) {
     return rpmmiInstance(mi);
 }

From n3npq at me.com  Sat Mar  4 19:21:44 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Sat, 04 Mar 2017 13:21:44 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
Message-ID: <5B17CB7B-D908-4714-B502-31560B364D50@me.com>


> On Mar 2, 2017, at 4:05 PM, Jeffrey Johnson <n3npq at me.com> wrote:
> 
> 
>> On Mar 2, 2017, at 3:52 PM, Jakub Bogusz <qboosh at pld-linux.org <mailto:qboosh at pld-linux.org>> wrote:
>> 
>> 
>> As far as I understand the code, rdl is size of immutable entry infos
>> part, while off is an offset in tags data part.
>> And when immutable tags data is short enough (shorter than entry infos
>> of immutable part), this check refuses to load header.
>> 
> 
> Yes. there is a ?immutable region? header and trailer, where the
> offset field is used as a double check on the tags in the immutable region.
> 
>> IMO the checks should be like in the attached patch.
>> With it, the two refused packages are accessible again.
>> 
> 
> I?ve applied the patch and will do a few tests before checking in.
> 
> One item I note (just scanning the patch) is
> 
> -		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb))
> +		if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(il * REGION_TAG_COUNT))
> 
> The variable il is derived and may be tainted, while off and nb are de facto positioning
> within the header memory blob. And yes, it may not matter.

(after spending some time resurrecting my ancient implementation memories)

The test above prevents nested ?immutable regions? (but works fine for
the only case that matters: a single ?immutable region?.

Its not so much tainting, but rather the total number of index entries, not
the number of index entries in the immutable region, that is stored in il.

The original intent was to be able to embed nested regions in order to make
appended tags (there are several) also immutable (and verifiable with digest/signature).

I can try to describe with pictures (but it may just make matters worse):
	A	= 16b index
	a	= variable length data region associated with tag A
So a (RPMv3) header looked like (white space added for readability)
	ABCD abcd

Now assume
	X,Y	= immutable region tag begin markers
	x,y	= immutable region tag end trailers

So the usual Header (for a long time now) looks like
	X ABCD abcd x
where ABCD are tags, and X is the region marker.

After installation other tags {J, K, L} are appended, so a header looks like
	X ABCD JKL abcd x jkl
permitting the original immutable region plaintext to be recreated to verify digest/signature
even after installation.

A header with nested immutable regions would then look like
	Y X ABCD QRST abcd x qrst y
where Q,R,S,T are tags associated with the outer immutable region.

I hope the above explains the very obscure implementation.

I?ll rework your patch slightly to do the argument check before doing the memcpy
but preserve the original check on {off,nb}.

I prefer sticking with ?last known good? until there is a clearer indication
that something needs to be changed.

Meanwhile ? if you have either of the 2 failing headers ? I can attempt to
do forensics on what the root cause failure for headerCopyLoad() is, and perhaps guess at what
happened.

hth
	





From n3npq at me.com  Sat Mar  4 19:34:20 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Sat, 04 Mar 2017 13:34:20 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <5B17CB7B-D908-4714-B502-31560B364D50@me.com>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
 <5B17CB7B-D908-4714-B502-31560B364D50@me.com>
Message-ID: <1D7A7110-DEDB-4C74-A235-E477A3E4F4A3@me.com>


> On Mar 4, 2017, at 1:21 PM, Jeffrey Johnson <n3npq at me.com> wrote:
> 
> 
> 
> A header with nested immutable regions would then look like
> 	Y X ABCD QRST abcd x qrst y
> where Q,R,S,T are tags associated with the outer immutable region.
> 

Grrr ? but at least a picture helped me recreate what was intended ?

This SHOULD have been
	Y QRST X ABCD abcd x qrst y
not
	Y X ABCD QRST abcd x qrst y
the difference being that offsets in the ?x? trailer do not need to be recalculated when nesting.

A ?real world? implementation would be to complete the nested immutable region implementation to
ensure that appended tags (i.e. JKL) are part of an immutable region (with canonical plaintext)
and add a second signature on the header blob while registering in an rpmdb.

hth

73 de Jeff




From n3npq at me.com  Sat Mar  4 21:59:24 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Sat, 04 Mar 2017 15:59:24 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <20170304091726.GA12545@mail>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
 <20170304091726.GA12545@mail>
Message-ID: <8A38038B-F8B3-44D3-89C2-5CC91C4E0116@me.com>


> On Mar 4, 2017, at 4:17 AM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
>> 
>> 
>> The variable il is derived and may be tainted, while off and nb are de facto positioning
>> within the header memory blob. And yes, it may not matter.
> 
> il is already used earlier to calculate dataStart. And length of the
> whole data (pvlen).
> 

Yes. Please note ?And yes it may not matter.? I?m absolutely sure your analysis is
sound, just perhaps there is more to do.

>> Meanwhile the entire issue is rather obscure, and only testing will tell.
>> Is there any information about what header???s are failing headerCopyLoad()?
>> If those headers are public keys, then the real flaw is elsewhere, wrapping
>> a public key within an immutable region, with an appended SHA1.
> 
> No, these are two packages.
> I'm attaching whole db data of one of them (partially described by me
> during investigation).
> 

Thank you.

Please be patient while I do forensics to understand where the regression/flaw
entered into 5.4.17.

For starters (after reading the dump, decoding the hex is next):

There is no appended signature tag in the dump you sent.

That basically means that those headers were not produced by any version of RPM5
in the last 5-6y, all headers are signed, and some signature tag SHOULD have been appended.

I will know more from examining RPMTAG_RPMVERSION and other build tracking tags ?

? it will take a bit of digging to find the root cause.

Meanwhile, by all means, apply your patch if it works for PLD. I?m just trying
not to flip-flop-flip-flop patches upstream until I understand fully what the problem
is and what needs to be done.

hth

73 de Jeff

From n3npq at me.com  Sat Mar  4 22:17:40 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Sat, 04 Mar 2017 16:17:40 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <20170304093026.GB12545@mail>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
 <20170304093026.GB12545@mail>
Message-ID: <27EBD31C-BC57-4EAB-95A9-AA3757468B7E@me.com>


> On Mar 4, 2017, at 4:30 AM, Jakub Bogusz <qboosh at pld-linux.org> wrote:
> 
> And one more patch, to make rpm4compat.h header usable again.
> fdSize() became exported function, so stop defining static function of
> the same name.
> 

Thanks for the patch.

Note that maintaining an API compatible rpm4compat.h include for an abandoned (by rpm5.org in 2007)
API from rpm.org (which is explicitly and overly hostile even to polite inquiries) is rather non-maintainable
in rpm5 sources.

The underlying issue is that fdSize() is being phased out: there are methods replacing the size_t value
returned by fdSize.

Both a FD_t and a Header type carry a full stat(2) structure. And network/pipe I/O has a
synthesized stat(2) with many additional values than st_size.

So fdStat() will surely disappear again, and similar for rpm4compat.h, on some long time scale.

hth

73 de Jeff

From n3npq at me.com  Sat Mar  4 22:35:23 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Sat, 04 Mar 2017 16:35:23 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <8A38038B-F8B3-44D3-89C2-5CC91C4E0116@me.com>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
 <20170304091726.GA12545@mail> <8A38038B-F8B3-44D3-89C2-5CC91C4E0116@me.com>
Message-ID: <4E1267AA-62A8-41CA-A226-0E65735D9212@me.com>


> On Mar 4, 2017, at 3:59 PM, Jeffrey Johnson <n3npq at me.com> wrote:
> 
> 
> 
> I will know more from examining RPMTAG_RPMVERSION and other build tracking tags ?
> 

OK, here is RPMTAG_RPMVERSION:

00000428 00000006 000001ac 00000001
...
d+01ac	342e3500

which is ?4.5?, and that package was built (RPMTAG_BUILDTIME)

000003ee 00000004 00000090 00000001
?
d+0090	50180128

which is decimal

1343750440

or

Tue, 31 Jul 2012 16:00:40 GMT

I have a dim memory of a compatibility patch fix for immutable region trailers
that was needed like a 10+ years ago that I deleted (because deemed unneeded).

I?ll resuscitate my memories in the next few days and dig into cvs history.

hth

73 de Jeff

From n3npq at me.com  Sat Mar  4 23:37:14 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Sat, 04 Mar 2017 17:37:14 -0500
Subject: pld rpm 5.4.17
In-Reply-To: <4E1267AA-62A8-41CA-A226-0E65735D9212@me.com>
References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com>
 <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com>
 <20170228204233.GA12663@mail> <47469D41-43B2-42A4-953D-021F837F4629@me.com>
 <20170302205252.GA852@mail> <8E5F045A-CED5-4E3C-8BCB-1D3AD6DC476F@me.com>
 <20170304091726.GA12545@mail> <8A38038B-F8B3-44D3-89C2-5CC91C4E0116@me.com>
 <4E1267AA-62A8-41CA-A226-0E65735D9212@me.com>
Message-ID: <EE6870F4-3DB8-400D-BB4B-6641205E9C71@me.com>


> On Mar 4, 2017, at 4:35 PM, Jeffrey Johnson <n3npq at me.com> wrote:
> 
> 
> I have a dim memory of a compatibility patch fix for immutable region trailers
> that was needed like a 10+ years ago that I deleted (because deemed unneeded).
> 

Good: the hack that I recall was very very ancient (like more than 10y ago).

> I?ll resuscitate my memories in the next few days and dig into cvs history.
> 

Appended is the cause of the regression, basically stronger tests on tag data validity
driven by AFL fuzzing (5.4.15 is ?-?, 5.4.18 is ?+?, in the attached patch snippet).

The hdrchkData(off) was (and is at rpm.org) a very weak test of data validity.

Now that I understand the context ? let me think about adding the rest of your patch,
its highly unlikely that nested immutable headers will ever be attempted.

hth. Thanks for patience, and thanks for persisting. Legacy compatibility is hard ...

73 de Jeff

================================================
@@ -1067,34 +995,32 @@ assert(h != NULL);

        {   rpmint32_t off = (rpmint32_t) ntohl(pe->offset);

-           if (hdrchkData(off))
+           if (off < 0)
                goto errxit;
            if (off) {
-/*@-sizeoftype@*/
+               rpmuint32_t * stei;
                size_t nb = REGION_TAG_COUNT;
-/*@=sizeoftype@*/
-               rpmuint32_t * stei = (rpmuint32_t *)
+               if ((off + nb) > dl)
+                   goto errxit;
+               /* XXX copy to fix alignment problems */
+               stei = (rpmuint32_t *)
                        memcpy(alloca(nb), dataStart + off, nb);
                rdl = (rpmuint32_t)-ntohl(stei[2]);     /* negative offset */
-assert((rpmint32_t)rdl >= 0);  /* XXX insurance */
-               ril = (rpmuint32_t)(rdl/sizeof(*pe));
-               if (hdrchkTags(ril) || hdrchkData(rdl))
+               if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb))
                    goto errxit;
+               ril = (rpmuint32_t)(rdl/sizeof(*pe));
            } else {
                ril = il;
-               /*@-sizeoftype@*/
                rdl = (rpmuint32_t)(ril * sizeof(struct entryInfo_s));
-               /*@=sizeoftype@*/
                entry->info.tag = (rpmTag) HEADER_IMAGE;
            }
        }
        entry->info.offset = (rpmint32_t) -rdl; /* negative offset */




From baggins at pld-linux.org  Sat Mar  4 23:43:13 2017
From: baggins at pld-linux.org (Jan =?utf-8?Q?R=C4=99korajski?=)
Date: Sat, 4 Mar 2017 23:43:13 +0100
Subject: PLD *.src.rpm cannot be installed by rpm-4.13
In-Reply-To: <68CD3568-3314-47DA-BC3D-53B4A679519E@me.com>
References: <68CD3568-3314-47DA-BC3D-53B4A679519E@me.com>
Message-ID: <20170304224313.GC3997@home>

On Thu, 02 Mar 2017, Jeffrey Johnson wrote:

> Here is the RHBZ bug report
> 	https://bugzilla.redhat.com/show_bug.cgi?id=1427287
> 
> This issue has been discussed ad nauseum back in 2009.
> 
> PLD has 2 choices:
> 
> 1) Retrofit RPMTAG_SOURCEPACKAGE and rpmlead boolean
> into RPM5 and rebuild everything
> 
> OR
> 
> 2) patch rpm.org rpm-4.13 (and perhaps older) with a couple line patch
> to recognize an RPM5 *.src.rpm (which rpm.org flatly refuses to do
> 	pmatilai <member:pmatilai>: rpm 5 interoperability is not a goal, at all:rpm 5 interoperability is not a goal, at all

No it's not PLD choice. It's yours.
You are rpm maintainer and it's your concern if you want rpm5 be
compatible with rpm.org.

-- 
Jan R?korajski                    | PLD/Linux
SysAdm | baggins<at>pld-linux.org | http://www.pld-linux.org/


From n3npq at me.com  Sat Mar  4 23:52:06 2017
From: n3npq at me.com (Jeffrey Johnson)
Date: Sat, 04 Mar 2017 17:52:06 -0500
Subject: PLD *.src.rpm cannot be installed by rpm-4.13
In-Reply-To: <20170304224313.GC3997@home>
References: <68CD3568-3314-47DA-BC3D-53B4A679519E@me.com>
 <20170304224313.GC3997@home>
Message-ID: <7808C9F1-A699-4DD8-B0B4-770CC1E1D906@me.com>


> On Mar 4, 2017, at 5:43 PM, Jan R?korajski <baggins at pld-linux.org> wrote:
> 
> On Thu, 02 Mar 2017, Jeffrey Johnson wrote:
> 
>> Here is the RHBZ bug report
>> 	https://bugzilla.redhat.com/show_bug.cgi?id=1427287
>> 
>> This issue has been discussed ad nauseum back in 2009.
>> 
>> PLD has 2 choices:
>> 
>> 1) Retrofit RPMTAG_SOURCEPACKAGE and rpmlead boolean
>> into RPM5 and rebuild everything
>> 
>> OR
>> 
>> 2) patch rpm.org rpm-4.13 (and perhaps older) with a couple line patch
>> to recognize an RPM5 *.src.rpm (which rpm.org flatly refuses to do
>> 	pmatilai <member:pmatilai>: rpm 5 interoperability is not a goal, at all:rpm 5 interoperability is not a goal, at all
> 
> No it's not PLD choice. It's yours.
> You are rpm maintainer and it's your concern if you want rpm5 be
> compatible with rpm.org <http://rpm.org/>.
> 

The message was basically FYI.

The code that needs changing is in rpm-4.13 (and the problem is worse than compatibility
with RPM5: there are other applications that are producing headers without RPMTAG_SOURCEPACKAGE),
so I will wait to see what rpm.org chooses to do (if anything).

There?s lots of ways to detect a SRPM (e.g. file paths are relative, not absolute).

This whole issue was hashed out in 2009 ? little has changed since.

73 de Jeff



From baggins at pld-linux.org  Sun Mar 12 13:22:28 2017
From: baggins at pld-linux.org (Jan =?utf-8?Q?R=C4=99korajski?=)
Date: Sun, 12 Mar 2017 13:22:28 +0100
Subject: Removal of openjdk8 from Th
Message-ID: <20170312122228.GA32733@tachikoma>

Now that we have icedtea8 in Th, I'm going to remove openjdk8 from ftp.

We don't need two java8 implementations and openjdk8 is known to have
problems buliding.

-- 
Jan R?korajski                    | PLD/Linux
SysAdm | baggins<at>pld-linux.org | http://www.pld-linux.org/


From baggins at pld-linux.org  Sun Mar 26 19:29:21 2017
From: baggins at pld-linux.org (Jan =?utf-8?Q?R=C4=99korajski?=)
Date: Sun, 26 Mar 2017 19:29:21 +0200
Subject: Kernel packages changes
Message-ID: <20170326172920.GB32733@tachikoma>

It's time to update the set of kernels we have in Th.

3.18 is EOL upstream and will be removed from Th
4.9 is new longterm line

This means that the only kernel with Vserver support wiil be 4.1.
If you still rely on vserer it's time to look for a replacement.

-- 
Jan R?korajski                    | PLD/Linux
SysAdm | baggins<at>pld-linux.org | http://www.pld-linux.org/