From atler at pld-linux.org  Wed Jun 10 13:27:00 2020
From: atler at pld-linux.org (Jan Palus)
Date: Wed, 10 Jun 2020 13:27:00 +0200
Subject: [packages/systemd] - this version is broken in an interesting
 way, rebooted during upgrade   and then became unable to
In-Reply-To: <bcc348b831315aef4775c2f8f9c3b349b3337d5b_refs_heads_master@pld-linux.org>
References: <6580fc0dbb1c6d22b05d11920c45980c4ccfe921_refs_heads_master@pld-linux.org>
 <bcc348b831315aef4775c2f8f9c3b349b3337d5b_refs_heads_master@pld-linux.org>
Message-ID: <20200610112700.lckiueokxrv7u2ga@kalarepa>

On 10.06.2020 09:35, baggins wrote:
> commit bcc348b831315aef4775c2f8f9c3b349b3337d5b
> Author: Jan R?korajski <baggins at pld-linux.org>
> Date:   Wed Jun 10 09:35:13 2020 +0200
> 
>     - this version is broken in an interesting way, rebooted during upgrade
>       and then became unable to boot fully

Any clue what went wrong? Worked fine for me in a VM and on real hardware (efi +
secure boot + full disk enc + grub2 + dracut).


From baggins at pld-linux.org  Wed Jun 10 22:53:08 2020
From: baggins at pld-linux.org (Jan =?utf-8?Q?R=C4=99korajski?=)
Date: Wed, 10 Jun 2020 22:53:08 +0200
Subject: [packages/systemd] - this version is broken in an interesting
 way, rebooted during upgrade   and then became unable to
In-Reply-To: <20200610112700.lckiueokxrv7u2ga@kalarepa>
References: <6580fc0dbb1c6d22b05d11920c45980c4ccfe921_refs_heads_master@pld-linux.org>
 <bcc348b831315aef4775c2f8f9c3b349b3337d5b_refs_heads_master@pld-linux.org>
 <20200610112700.lckiueokxrv7u2ga@kalarepa>
Message-ID: <20200610205308.GA2364@starbug>

On Wed, 10 Jun 2020, Jan Palus wrote:

> On 10.06.2020 09:35, baggins wrote:
> > commit bcc348b831315aef4775c2f8f9c3b349b3337d5b
> > Author: Jan R?korajski <baggins at pld-linux.org>
> > Date:   Wed Jun 10 09:35:13 2020 +0200
> > 
> >     - this version is broken in an interesting way, rebooted during upgrade
> >       and then became unable to boot fully
> 
> Any clue what went wrong? Worked fine for me in a VM and on real hardware (efi +
> secure boot + full disk enc + grub2 + dracut).

Unfortunately not. I run 'poldek -F *' expecting systemd update, went to
grab some coffee and came back to system stuck at reboot. I had to forcing
the reboot via sysrq-b, since it was abvious that it wasn't a sservice
issue. After that the workstation couldn't properly boot. No X, kust waiting
for someting. I was able to access text console, downgrade got my system back in
order.

I'll try to figure out WTF happened, time permitting.

-- 
Jan R?korajski                    | PLD/Linux
SysAdm | baggins<at>pld-linux.org | http://www.pld-linux.org/


From baggins at pld-linux.org  Fri Jun 12 19:54:55 2020
From: baggins at pld-linux.org (Jan =?utf-8?Q?R=C4=99korajski?=)
Date: Fri, 12 Jun 2020 19:54:55 +0200
Subject: [packages/systemd] - this version is broken in an interesting
 way, rebooted during upgrade   and then became unable to
In-Reply-To: <20200610205308.GA2364@starbug>
References: <6580fc0dbb1c6d22b05d11920c45980c4ccfe921_refs_heads_master@pld-linux.org>
 <bcc348b831315aef4775c2f8f9c3b349b3337d5b_refs_heads_master@pld-linux.org>
 <20200610112700.lckiueokxrv7u2ga@kalarepa>
 <20200610205308.GA2364@starbug>
Message-ID: <20200612175455.GA2454@starbug>

On Wed, 10 Jun 2020, Jan R?korajski wrote:

> On Wed, 10 Jun 2020, Jan Palus wrote:
> 
> > On 10.06.2020 09:35, baggins wrote:
> > > commit bcc348b831315aef4775c2f8f9c3b349b3337d5b
> > > Author: Jan R?korajski <baggins at pld-linux.org>
> > > Date:   Wed Jun 10 09:35:13 2020 +0200
> > > 
> > >     - this version is broken in an interesting way, rebooted during upgrade
> > >       and then became unable to boot fully
> > 
> > Any clue what went wrong? Worked fine for me in a VM and on real hardware (efi +
> > secure boot + full disk enc + grub2 + dracut).
> 
> Unfortunately not. I run 'poldek -F *' expecting systemd update, went to
> grab some coffee and came back to system stuck at reboot. I had to forcing
> the reboot via sysrq-b, since it was abvious that it wasn't a sservice
> issue. After that the workstation couldn't properly boot. No X, kust waiting
> for someting. I was able to access text console, downgrade got my system back in
> order.
> 
> I'll try to figure out WTF happened, time permitting.

I've seen you applied some fixes, thanks for all your work.

There is some progress. Once again I tried the update, and again my
system went straight to shutdown/reboot. The difference this time is
that after I did force the reboot via sysrq-b, my workstation is fully
functional now. So, progress :)

Seems to me like 'systemctl daemon-reload' is doing something bad.
Unfortunately I'm unable to catch what's going on, it just happens too
fast.

I'll try rebuilding 245.6-0.1 and updating it to check if that's a
one-time issue.

-- 
Jan R?korajski                    | PLD/Linux
SysAdm | baggins<at>pld-linux.org | http://www.pld-linux.org/


From baggins at pld-linux.org  Fri Jun 12 20:01:52 2020
From: baggins at pld-linux.org (Jan =?utf-8?Q?R=C4=99korajski?=)
Date: Fri, 12 Jun 2020 20:01:52 +0200
Subject: [packages/systemd] - this version is broken in an interesting
 way, rebooted during upgrade   and then became unable to
In-Reply-To: <20200612175455.GA2454@starbug>
References: <6580fc0dbb1c6d22b05d11920c45980c4ccfe921_refs_heads_master@pld-linux.org>
 <bcc348b831315aef4775c2f8f9c3b349b3337d5b_refs_heads_master@pld-linux.org>
 <20200610112700.lckiueokxrv7u2ga@kalarepa>
 <20200610205308.GA2364@starbug> <20200612175455.GA2454@starbug>
Message-ID: <20200612180152.GB2454@starbug>

On Fri, 12 Jun 2020, Jan R?korajski wrote:

> On Wed, 10 Jun 2020, Jan R?korajski wrote:
> 
> > On Wed, 10 Jun 2020, Jan Palus wrote:
> > 
> > > On 10.06.2020 09:35, baggins wrote:
> > > > commit bcc348b831315aef4775c2f8f9c3b349b3337d5b
> > > > Author: Jan R?korajski <baggins at pld-linux.org>
> > > > Date:   Wed Jun 10 09:35:13 2020 +0200
> > > > 
> > > >     - this version is broken in an interesting way, rebooted during upgrade
> > > >       and then became unable to boot fully
> > > 
> > > Any clue what went wrong? Worked fine for me in a VM and on real hardware (efi +
> > > secure boot + full disk enc + grub2 + dracut).
> > 
> > Unfortunately not. I run 'poldek -F *' expecting systemd update, went to
> > grab some coffee and came back to system stuck at reboot. I had to forcing
> > the reboot via sysrq-b, since it was abvious that it wasn't a sservice
> > issue. After that the workstation couldn't properly boot. No X, kust waiting
> > for someting. I was able to access text console, downgrade got my system back in
> > order.
> > 
> > I'll try to figure out WTF happened, time permitting.
> 
> I've seen you applied some fixes, thanks for all your work.
> 
> There is some progress. Once again I tried the update, and again my
> system went straight to shutdown/reboot. The difference this time is
> that after I did force the reboot via sysrq-b, my workstation is fully
> functional now. So, progress :)
> 
> Seems to me like 'systemctl daemon-reload' is doing something bad.
> Unfortunately I'm unable to catch what's going on, it just happens too
> fast.
> 
> I'll try rebuilding 245.6-0.1 and updating it to check if that's a
> one-time issue.

Yes, there must be some intricate issue when updating from 244 to 245.
Updating 245 to 245 shows no side effects, clean restart.

If you know of anything I should look at, or have any pointers where
should I dig, I'm open to suggestions.

BTW, did anyone else try to do the update? Folks?

-- 
Jan R?korajski                    | PLD/Linux
SysAdm | baggins<at>pld-linux.org | http://www.pld-linux.org/


From glen at pld-linux.org  Fri Jun 12 23:17:04 2020
From: glen at pld-linux.org (=?UTF-8?Q?Elan_Ruusam=c3=a4e?=)
Date: Sat, 13 Jun 2020 00:17:04 +0300
Subject: libc 2.31/i686: operation not permitted for preserving timestamps
In-Reply-To: <b029ddfe-70ac-fcdd-a243-7683f2df18d4@pld-linux.org>
References: <b029ddfe-70ac-fcdd-a243-7683f2df18d4@pld-linux.org>
Message-ID: <d96f441e-912a-ebce-4df9-8d9c69fc5532@pld-linux.org>

On 3/16/20 11:46 PM, Elan Ruusam?e wrote:

> i've got reports that cp -a (or just cp --preserve=timestamps) fails 
> on i686 and glibc 2.31
>

this is still not solved. and rpm (berkeleydb?) crashes misearably on 
that and unable to recover:

- https://gitlab.com/pld-linux/pld/-/jobs/593531470


i'll be disabling i686 builds over the weekend so that at least updated 
x86_64 images get built



From arekm at maven.pl  Fri Jun 12 23:29:02 2020
From: arekm at maven.pl (=?UTF-8?Q?Arkadiusz_Mi=c5=9bkiewicz?=)
Date: Fri, 12 Jun 2020 23:29:02 +0200
Subject: libc 2.31/i686: operation not permitted for preserving timestamps
In-Reply-To: <d96f441e-912a-ebce-4df9-8d9c69fc5532@pld-linux.org>
References: <b029ddfe-70ac-fcdd-a243-7683f2df18d4@pld-linux.org>
 <d96f441e-912a-ebce-4df9-8d9c69fc5532@pld-linux.org>
Message-ID: <4ff09590-7312-dbda-9eb6-2d84b743922a@maven.pl>

On 12/06/2020 23:17, Elan Ruusam?e wrote:
> On 3/16/20 11:46 PM, Elan Ruusam?e wrote:
> 
>> i've got reports that cp -a (or just cp --preserve=timestamps) fails
>> on i686 and glibc 2.31
>>
> 
> this is still not solved. and rpm (berkeleydb?) crashes misearably on
> that and unable to recover:
> 
> - https://gitlab.com/pld-linux/pld/-/jobs/593531470
> 
> 
> i'll be disabling i686 builds over the weekend so that at least updated
> x86_64 images get built

Could you try  --security-opt and the other thing I asked in april?

(didn't see response... maybe you did test that)
-- 
Arkadiusz Mi?kiewicz, arekm / ( maven.pl | pld-linux.org )


From atler at pld-linux.org  Fri Jun 12 23:46:09 2020
From: atler at pld-linux.org (Jan Palus)
Date: Fri, 12 Jun 2020 23:46:09 +0200
Subject: [packages/systemd] - this version is broken in an interesting
 way, rebooted during upgrade   and then became unable to
In-Reply-To: <20200612180152.GB2454@starbug>
References: <6580fc0dbb1c6d22b05d11920c45980c4ccfe921_refs_heads_master@pld-linux.org>
 <bcc348b831315aef4775c2f8f9c3b349b3337d5b_refs_heads_master@pld-linux.org>
 <20200610112700.lckiueokxrv7u2ga@kalarepa>
 <20200610205308.GA2364@starbug> <20200612175455.GA2454@starbug>
 <20200612180152.GB2454@starbug>
Message-ID: <20200612214609.kdjywxvetqhezogc@pine>

On 12.06.2020 20:01, Jan R?korajski wrote:
> Yes, there must be some intricate issue when updating from 244 to 245.
> Updating 245 to 245 shows no side effects, clean restart.
> 
> If you know of anything I should look at, or have any pointers where
> should I dig, I'm open to suggestions.

The only thing I could think of is journalctl output from the moment of upgrade
till reboot.

> BTW, did anyone else try to do the update? Folks?

FWIW I have upgraded my aarch64 laptop and still no reboot. It ended my user
session though -- a bug in systemd which is already fixed since yesterday.


From glen at pld-linux.org  Sat Jun 13 04:17:14 2020
From: glen at pld-linux.org (=?UTF-8?Q?Elan_Ruusam=c3=a4e?=)
Date: Sat, 13 Jun 2020 05:17:14 +0300
Subject: libc 2.31/i686: operation not permitted for preserving timestamps
In-Reply-To: <4ff09590-7312-dbda-9eb6-2d84b743922a@maven.pl>
References: <b029ddfe-70ac-fcdd-a243-7683f2df18d4@pld-linux.org>
 <d96f441e-912a-ebce-4df9-8d9c69fc5532@pld-linux.org>
 <4ff09590-7312-dbda-9eb6-2d84b743922a@maven.pl>
Message-ID: <4e141d39-ec9c-a892-8ce1-ed769bc02564@pld-linux.org>


On 6/13/20 12:29 AM, Arkadiusz Mi?kiewicz wrote:
> On 12/06/2020 23:17, Elan Ruusam?e wrote:
>> On 3/16/20 11:46 PM, Elan Ruusam?e wrote:
>>
>>> i've got reports that cp -a (or just cp --preserve=timestamps) fails
>>> on i686 and glibc 2.31
>>>
>> this is still not solved. and rpm (berkeleydb?) crashes misearably on
>> that and unable to recover:
>>
>> - https://gitlab.com/pld-linux/pld/-/jobs/593531470
>>
>>
>> i'll be disabling i686 builds over the weekend so that at least updated
>> x86_64 images get built
> Could you try  --security-opt and the other thing I asked in april?

that's not a solution!


1. it's a regression (no such problem with glibc 2.30)

2. not all cloud providers allow you override such option!


> (didn't see response... maybe you did test that)

reply is here:

- 
http://lists.pld-linux.org/mailman/pipermail/pld-devel-en/2020-April/025905.html


if you want a reproducer, then docker image that can be tested with is

- 
registry.gitlab.com/pld-linux/pld/i686 at sha256:832f004cc28798aafd758a7e6aa867681f1f860ac22b341dc8a2183a75a53e36


command that should not fail:

- cp -a /var/lib/rpm/ /tmp/rpm1


run the command before glibc upgrade (on that image glibc-2.30-3.i686),

run the command after glibc upgrade

test with and without --privileged option to docker run


i'm able to reproduce with docker4mac locally, the underlying filesystem 
is ext4 on 4.19.76-linuxkit:


```

$ docker run -v /var/lib/docker:/var/lib/docker alpine df /var/lib/docker -T
Filesystem?????????? Type?????? 1K-blocks????? Used Available Use% 
Mounted on
/dev/vda1??????????? ext4??????? 65792556?? 1159988? 61260792?? 2% 
/etc/resolv.conf

```


and docker4win-legacy, ext4, 4.14.154-boot2docker



```

$ docker run -v /var/lib/docker:/var/lib/docker alpine df /var/lib/docker -T
Filesystem?????????? Type?????? 1K-blocks????? Used Available Use% 
Mounted on
/dev/sda1??????????? ext4??????? 18714000?? 4190364? 13534508? 24% 
/etc/resolv.conf


$ docker run -v /var/lib/docker:/var/lib/docker alpine uname -r
4.14.154-boot2docker

```


and docker4linux 4.19.0-8-amd64


```

$ docker run -v /var/lib/docker:/var/lib/docker alpine sh -c 'df 
/var/lib/docker -T; uname -r'

Filesystem?????????? Type?????? 1K-blocks????? Used Available Use% 
Mounted on
/dev/vdc???????????? xfs???????? 52403200? 13322340? 39080860? 25% 
/etc/resolv.conf
4.19.0-8-amd64

```


the only place it does not fail of the 5 tests is docker4linux, btrfs, 
4.4.30-1


```

$ docker run -v /var/lib/docker:/var/lib/docker alpine sh -c 'df 
/var/lib/docker -T; uname -r'
Filesystem?????????? Type?????? 1K-blocks????? Used Available Use% 
Mounted on
/dev/mapper/sys-docker
 ???????????????????? btrfs?????? 41943040? 15572720? 19025392? 45% 
/etc/resolv.conf
4.4.30-1

```



From arekm at maven.pl  Sat Jun 13 09:08:19 2020
From: arekm at maven.pl (=?UTF-8?Q?Arkadiusz_Mi=c5=9bkiewicz?=)
Date: Sat, 13 Jun 2020 09:08:19 +0200
Subject: libc 2.31/i686: operation not permitted for preserving timestamps
In-Reply-To: <4e141d39-ec9c-a892-8ce1-ed769bc02564@pld-linux.org>
References: <b029ddfe-70ac-fcdd-a243-7683f2df18d4@pld-linux.org>
 <d96f441e-912a-ebce-4df9-8d9c69fc5532@pld-linux.org>
 <4ff09590-7312-dbda-9eb6-2d84b743922a@maven.pl>
 <4e141d39-ec9c-a892-8ce1-ed769bc02564@pld-linux.org>
Message-ID: <d5efd321-cd40-7078-505a-60e8ea73c69e@maven.pl>

On 13/06/2020 04:17, Elan Ruusam?e wrote:
> 
> On 6/13/20 12:29 AM, Arkadiusz Mi?kiewicz wrote:
>> On 12/06/2020 23:17, Elan Ruusam?e wrote:
>>> On 3/16/20 11:46 PM, Elan Ruusam?e wrote:
>>>
>>>> i've got reports that cp -a (or just cp --preserve=timestamps) fails
>>>> on i686 and glibc 2.31
>>>>
>>> this is still not solved. and rpm (berkeleydb?) crashes misearably on
>>> that and unable to recover:
>>>
>>> - https://gitlab.com/pld-linux/pld/-/jobs/593531470
>>>
>>>
>>> i'll be disabling i686 builds over the weekend so that at least updated
>>> x86_64 images get built
>> Could you try? --security-opt and the other thing I asked in april?
> 
> that's not a solution!

It's not. It's debugging. There was second question there, too (libseccomp).

Obviously run on pld host and docker "guest".

-- 
Arkadiusz Mi?kiewicz, arekm / ( maven.pl | pld-linux.org )


From baggins at pld-linux.org  Sat Jun 13 10:33:29 2020
From: baggins at pld-linux.org (Jan =?utf-8?Q?R=C4=99korajski?=)
Date: Sat, 13 Jun 2020 10:33:29 +0200
Subject: [packages/systemd] - this version is broken in an interesting
 way, rebooted during upgrade   and then became unable to
In-Reply-To: <20200612214609.kdjywxvetqhezogc@pine>
References: <6580fc0dbb1c6d22b05d11920c45980c4ccfe921_refs_heads_master@pld-linux.org>
 <bcc348b831315aef4775c2f8f9c3b349b3337d5b_refs_heads_master@pld-linux.org>
 <20200610112700.lckiueokxrv7u2ga@kalarepa>
 <20200610205308.GA2364@starbug> <20200612175455.GA2454@starbug>
 <20200612180152.GB2454@starbug>
 <20200612214609.kdjywxvetqhezogc@pine>
Message-ID: <20200613083329.GA2039@starbug>

On Fri, 12 Jun 2020, Jan Palus wrote:

> On 12.06.2020 20:01, Jan R?korajski wrote:
> > Yes, there must be some intricate issue when updating from 244 to 245.
> > Updating 245 to 245 shows no side effects, clean restart.
> > 
> > If you know of anything I should look at, or have any pointers where
> > should I dig, I'm open to suggestions.
> 
> The only thing I could think of is journalctl output from the moment of upgrade
> till reboot.
> 
> > BTW, did anyone else try to do the update? Folks?
> 
> FWIW I have upgraded my aarch64 laptop and still no reboot. It ended my user
> session though -- a bug in systemd which is already fixed since yesterday.

Thanks. I figured out WTF happend. LOL.

Updated systemd from 244 to 245.6 again, and did not reboot or crash or whatever.

systemd on daemon-reload decided to restart a bunch of units,
_including_ plymouth stuff. And this started plymouth-start.service,
switched to a text / dmesg console with the boot splash, what looked
like a crash/failed reboot.

That's all. Alt-F2 and I'm back in X and everything works.

We're good to go with systemd upgrade.

-- 
Jan R?korajski                    | PLD/Linux
SysAdm | baggins<at>pld-linux.org | http://www.pld-linux.org/