rpm 4.16 landed -> errors!

[Neal Gompa]

On Wed, Mar 17, 2021 at 2:48 PM Jan Rękorajski <baggins at pld-linux.org> wrote:
>
> On Tue, 16 Mar 2021, Neal Gompa wrote:
>
> > On Tue, Mar 16, 2021 at 1:39 PM Jan Rękorajski <baggins at pld-linux.org> wrote:
> > >
> > > On Tue, 16 Mar 2021, Elan Ruusamäe wrote:
> > >
> > > >
> > > > On 16.03.2021 16:45, Elan Ruusamäe wrote:
> > > > >
> > > > > the automatic upgrade is failing:
> > > > >
> > > > > - https://gitlab.com/pld-linux/pld/-/jobs/1094531330#L214
> > > > >
> > > > oh, and user is left without rpmdb:
> > > >
> > > >
> > > > [@46a17bd0ade9 /]# rpm -q rpm poldek
> > > > warning: Found bdb Packages database while attempting sqlite backend:
> > > > using bdb backend.
> > > > warning: Generating 6 missing index(es), please wait...
> > > > package rpm is not installed
> > > > package poldek is not installed
> > > >
> > > > [@46a17bd0ade9 /]# rpm -q rpm poldek
> > > > warning: Found bdb Packages database while attempting sqlite backend:
> > > > using bdb backend.
> > > > package rpm is not installed
> > > > package poldek is not installed
> > > > [@46a17bd0ade9 /]#
> > >
> > > First, regarding lost database.
> > > You do have a backup of the database from before the update:
> > >
> > > Backup of the rpm database has been created in /var/lib/rpm.rpmbackup-4.16.1.2-6
> > >
> > > There is also the new database, that rpm was unable to move:
> > >
> > > error: replace files in /var/lib/rpm with files from /var/lib/rpmrebuilddb.42 to recover
> > >
> > > Is /var/lib/rpm a mountpoint? That would be the explanation that comes
> > > to mind. There must be something that prevented rpm from renaming temporary
> > > directory in /var/lib to rpm.
> > >
> > > I've been (live) testing upgrade path and I assumed it does work.
> > > Never encountered problems after squashing rpm5 oddities.
> > >
> > > I've been also asking over and over for testing and no one came with any
> > > issues like this.
> > >
> >
> > This is probably an overlayfs specific issue, I've seen various
> > problems with renames (which is what rpm does for rebuilding
> > databases) in containers.
>
> Neal,
>
> Do you have any references to such issues? Like bug reports, or docs?
> I'd like to link them to the rpm migration page on our wiki.
>

Not offhand, but I know that this change to RPM[1] (and a few other
fun quirks of bdb) is what started causing things to trip up in
containers with rpmdb rebuilds. It should basically go away once the
conversion to sqlite rpmdb is done. There is information about how
OverlayFS handles directory renames in the Linux kernel
documentation[2]. This affected YUM too[3], though DNF has workarounds
built into it now[4]. For this specific issue, you can avoid this by
regenerating the container entirely from scratch instead of using an
upgrade to fix it.

More generally, I advise being careful with OverlayFS and being
mindful of the pitfalls[2]. I personally use Btrfs instead, which
neatly avoids this and is a lot more performant.

(As an aside, can someone rebase the DNF package manager stack in PLD?
It's pretty old and broken...)

[1]: https://github.com/rpm-software-management/rpm/commit/fffd652c56eaef8fc41d23190e39513639f2092d
[2]: https://www.kernel.org/doc/html/latest/filesystems/overlayfs.html#renaming-directories
[3]: https://bugzilla.redhat.com/1213602
[4]: https://bugzilla.redhat.com/1658565





--
真実はいつも一つ！/ Always, there's only one truth!