From jajcus at jajcus.net Tue Feb 14 13:32:21 2017 From: jajcus at jajcus.net (Jacek Konieczny) Date: Tue, 14 Feb 2017 13:32:21 +0100 Subject: RFC: systemd-timers to obsolete crond Message-ID: <006c8cf3-6309-7ef9-c62f-39c00346fc2b@jajcus.net> Hi, I am not happy with crond running under systemd. The same could be done with systemd timers, but we should keep compatibility with old systems. My idea is to create a new package, 'systemd-timers', which would obsolete all 'crondaemon' packages. Some other packages require 'crondaemon' as they provide their own crontabs or cron.{daily,hourly,daily,weekly,monthly} scripts. We could add .timer and .service units for those packages too and then they could work with crond OR systemd-timers. The dependency could be defined with a virtual 'Provides', but how should I call it? 'Provides: crondaemon-or-systemd-timers' does not look right. ;-) Jacek From jajcus at jajcus.net Tue Feb 14 14:10:41 2017 From: jajcus at jajcus.net (Jacek Konieczny) Date: Tue, 14 Feb 2017 14:10:41 +0100 Subject: RFC: systemd-timers to obsolete crond In-Reply-To: <006c8cf3-6309-7ef9-c62f-39c00346fc2b@jajcus.net> References: <006c8cf3-6309-7ef9-c62f-39c00346fc2b@jajcus.net> Message-ID: <9ea58a83-9261-bda3-ce3b-30b6140d3ed3@jajcus.net> On 2017-02-14 13:32, Jacek Konieczny wrote: > I am not happy with crond running under systemd. The same could be done > with systemd timers, but we should keep compatibility with old systems. I guess, I sent the mail too early, as now I have a bit better idea how it could look like. A new package: systemd-cronjobs (pseudo-spec) Provides: cronjobs Obsoletes: crondaemon %post: systemctl enable cronjobs.target %postun systemctl disable cronjobs.target %files: /lib/systemd/system/cronjobs.target /lib/systemd/system/cronjob-hourly.timer /lib/systemd/system/cronjob-hourly.service /lib/systemd/system/cronjob-daily.timer /lib/systemd/system/cronjob-daily.service /lib/systemd/system/cronjob-weekly.timer /lib/systemd/system/cronjob-weekly.service /lib/systemd/system/cronjob-monthly.timer /lib/systemd/system/cronjob-monthly.service ----------------- Updated: systemd-units: %files +/etc/systemd/system/cronjobs.target.wants ------------------ Updated: cronie (and other crondaemons) +Provides: cronjobs Provides: crondaemon Obsoletes: systemd-cronjobs ------------------- Updated: logrotate (and other packages providing their crontabs) -Requires: crondaemon +Requires: cronjobs %post +%systemd_reload %files /etc/cron.d/logrotate +/lib/systemd/system/cronjob-logrotate.timer +/lib/systemd/system/cronjob-logrotate.service The systemd unit dependencies would be: cronjobs.target WantedBy multi-user.target cronjob-*.timer WantedBy cronjobs.target How it would work: When systemd-cronjobs is not installed ? exactly as it works now. Scheduled task are run by crond. Installing systemd-cronjobs would uninstall any crond, but would be possible only if there is no package left which would Require: crondaemon (meaning: no cronjob-*.timer yet). When systemd-cronjobs is installed scheduled tasks would be run from systemd instead of crond. Why it is better: ? no crond service running ? one process less ? no unnecessary shell processes when a single executable is to be run ? no login session created (lots of PAM and systemd logs) when not necessary What I am not sure yet: ? is 'Provides: cronjobs' and 'Name: systemd-cronjobs' any good? ? do we still need to care about crond, or should we just migrate everything to systemd timers? Jacek From qboosh at pld-linux.org Tue Feb 21 21:51:13 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Tue, 21 Feb 2017 21:51:13 +0100 Subject: pld rpm 5.4.17 In-Reply-To: <587537B8.50603@pld-linux.org> References: <587537B8.50603@pld-linux.org> Message-ID: <20170221205113.GA2445@mail> On Tue, Jan 10, 2017 at 09:36:24PM +0200, Elan Ruusam?e wrote: > not cool. > > $ rpm -q rpm > BDB0056 DB->cursor: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW > require locking > error: db3copen:db3.c:1470: db->cursor(22): Invalid argument > BDB0056 DB->cursor: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW > require locking > error: db3copen:db3.c:1470: db->cursor(22): Invalid argument > BDB0630 DB_THREAD mandates memory allocation flag on primary key DBT > error: db3cpget:db3.c:1568: db->pget(22): Invalid argument > error: error(22) getting keys from Nvra index > error: error(1) getting records from Nvra index > package rpm is not installed > > luckily was able to downgrade. Which db version was used? Was that default system db (with headers in /usr/include)? I experienced such behaviour when trying to upgrade rpm 5.4.15+db 6.1.19 to 5.4.17+db 6.1.29 (while the system db is 6.2.23, maybe some file caught system db.h) rpm 5.4.17+db 6.2.23 seems to have more chances to work, but: - dbconvert utility crashes badly: -bash-4.4# ~comp/rpm/BUILD/rpm-5.4.17.db62/tools/dbconvert --rebuilddb BDB0055 illegal flag specified to DB_ENV->set_ext_file_threshold error: db_init:db3.c:1032: dbenv->set_event_notify(22): Z?y argument *** db_init: dbenv->open argument: flags: 0x2421 *** db_init: dbenv->get_open_flags: flags: 0x0 BDB1565 DB_ENV->dbremove: method not permitted before handle's open method error: db_init:db3.c:1224: dbenv->failchk(22): Z?y argument [2304927.265310] Process lt-dbconvert (pid: 19580, ti=db03a000 task=c847fb40 task.ti=db03a000) Segmentation fault - rpm --rebuilddb complains: error: db3: header #187105280 cannot be loaded -- skipping. error: db3: header #4127850496 cannot be loaded -- skipping. (once per index) -- Jakub Bogusz http://qboosh.pl/ From qboosh at pld-linux.org Wed Feb 22 22:02:17 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Wed, 22 Feb 2017 22:02:17 +0100 Subject: pld rpm 5.4.17 In-Reply-To: <20170221205113.GA2445@mail> References: <587537B8.50603@pld-linux.org> <20170221205113.GA2445@mail> Message-ID: <20170222210217.GA13257@mail> On Tue, Feb 21, 2017 at 09:51:13PM +0100, Jakub Bogusz wrote: > On Tue, Jan 10, 2017 at 09:36:24PM +0200, Elan Ruusam?e wrote: > > not cool. > > > > $ rpm -q rpm > > BDB0056 DB->cursor: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW > > require locking > > error: db3copen:db3.c:1470: db->cursor(22): Invalid argument > > BDB0056 DB->cursor: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW > > require locking > > error: db3copen:db3.c:1470: db->cursor(22): Invalid argument > > BDB0630 DB_THREAD mandates memory allocation flag on primary key DBT > > error: db3cpget:db3.c:1568: db->pget(22): Invalid argument > > error: error(22) getting keys from Nvra index > > error: error(1) getting records from Nvra index > > package rpm is not installed > > > > luckily was able to downgrade. > > Which db version was used? > Was that default system db (with headers in /usr/include)? > > I experienced such behaviour when trying to upgrade rpm 5.4.15+db 6.1.19 > to 5.4.17+db 6.1.29 (while the system db is 6.2.23, maybe some file > caught system db.h) Update: still the same with uid>0 (no write permission to /var/lib/rpm). It seems that DB_READ_COMMITTED and DB_READ_UNCOMMITTED require +w, so they should be filtered out in unprivileged mode. > rpm 5.4.17+db 6.2.23 seems to have more chances to work, but: > - dbconvert utility crashes badly: > > -bash-4.4# ~comp/rpm/BUILD/rpm-5.4.17.db62/tools/dbconvert --rebuilddb > BDB0055 illegal flag specified to DB_ENV->set_ext_file_threshold > error: db_init:db3.c:1032: dbenv->set_event_notify(22): Z?y argument > *** db_init: dbenv->open argument: > flags: 0x2421 > *** db_init: dbenv->get_open_flags: > flags: 0x0 > BDB1565 DB_ENV->dbremove: method not permitted before handle's open method > error: db_init:db3.c:1224: dbenv->failchk(22): Z?y argument > [2304927.265310] Process lt-dbconvert (pid: 19580, ti=db03a000 task=c847fb40 task.ti=db03a000) > Segmentation fault > > - rpm --rebuilddb complains: > > error: db3: header #187105280 cannot be loaded -- skipping. > error: db3: header #4127850496 cannot be loaded -- skipping. > > (once per index) And it wasn't possible to rebuild Filepaths index because of not enough locks (16384 previously). Increasing locks required Packages rebuild anyway (db_dump + db_load) and most of the db problems are gone now. This one is left though: > error: db3: header #187105280 cannot be loaded -- skipping. > error: db3: header #4127850496 cannot be loaded -- skipping. How to check what these "headers" mean? (old, unsupported keys? some old packages with missing fields which are now required?) -- Jakub Bogusz http://qboosh.pl/ From n3npq at me.com Thu Feb 23 19:43:14 2017 From: n3npq at me.com (Jeffrey Johnson) Date: Thu, 23 Feb 2017 13:43:14 -0500 Subject: pld rpm 5.4.17 Message-ID: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com> This one is left though: > error: db3: header #187105280 cannot be loaded -- skipping. > error: db3: header #4127850496 cannot be loaded -- skipping. How to check what these "headers" mean? (old, unsupported keys? some old packages with missing fields which are now required?) The error message is printed on a headerCopyLoad() failure. The failure is usually an indication of header damage of some sort, not missing fields now required. The numbers are primary keys into Packages printed in in decimal. The values appear to be in the wrong-endian order when converted to hex 0xB270000 0xF60A0000 You can try finding the header by doing, say, rpm -q ?querybynumber 0xB270000 (or its reverse: its unclear what order). hth 73 de Jeff From n3npq at me.com Thu Feb 23 20:15:12 2017 From: n3npq at me.com (Jeffrey Johnson) Date: Thu, 23 Feb 2017 14:15:12 -0500 Subject: RPM + SEMANAGE? Message-ID: I see that semanage has been enabled 3 days ago https://github.com/pld-linux/rpm/commit/ec7b8d8fb16f5789772693ff807e0a93a5c653e4 Be forewarned: the semanage code in RPM hasn?t been looked at for quite some years: at the time I implemented, semanage was just being invented. If there is a need, I can/will update the SELinux modules to latest released in rpm-5.4.18. There hasn?t been any detectable interest so far ? hth 73 de Jeff From n3npq at me.com Thu Feb 23 20:31:56 2017 From: n3npq at me.com (Jeffrey Johnson) Date: Thu, 23 Feb 2017 14:31:56 -0500 Subject: OpenMandriva upgrade 5.4.15 -> 5.4.17+ Message-ID: <326FDD98-45CD-4046-8E49-838AC7BF7983@me.com> (This is largely a FYI ? PLD includes many Mandriva patches) OpenMandriva is attempting an upgrade to 5.4.17 (or rpm-5.4.18) with 300+ patches, ~90 of which need to be rebased (or dropped): https://www.mail-archive.com/om-cooker at ml.openmandriva.org/msg09198.html I was asked to help port the patches. In order to avoid discussing 300+ patches item-by-item, the following process is being used CVS checkout on the rpm-5_4_15-release tag. Build and fix issues with newer prereqs. Manually extract all the patches (and order of application) in a script. Apply, build, and fix. Upgrade to the rpm-5_4_16-release tag. Fix merge conflicts and build. Upgrade to the rpm-5_4_17-release tag. Fix merge conflicts and build. and this process may be continued up through a rpm-5.4.18 release. The process has proceeded to a collapsed rpm-5.4.17 patch, a separate configuration tarball (almost 1/3rd of the OMA patches are to rpm configuration), and a custom 5.4.17 tarball with all patches applied. The next steps will be to either upgrade to rpm-5.4.18 (still undecided), and then to go through regression testing. Comments are welcome at https://github.com/OpenMandrivaAssociation/rpm/issues fyi 73 de Jeff From qboosh at pld-linux.org Thu Feb 23 21:33:40 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Thu, 23 Feb 2017 21:33:40 +0100 Subject: RPM + SEMANAGE? In-Reply-To: References: Message-ID: <20170223203340.GA20987@mail> On Thu, Feb 23, 2017 at 02:15:12PM -0500, Jeffrey Johnson wrote: > I see that semanage has been enabled 3 days ago > https://github.com/pld-linux/rpm/commit/ec7b8d8fb16f5789772693ff807e0a93a5c653e4 > > Be forewarned: the semanage code in RPM hasn???t been looked at for quite some years: at the > time I implemented, semanage was just being invented. > > If there is a need, I can/will update the SELinux modules to latest released in rpm-5.4.18. > There hasn???t been any detectable interest so far ??? Actually, in this commit nothing changed to semanage switch. It was enabled in 6e115b2320d8152309c7183c8b36641fbb1316b9, over 4 years ago. The patch mentioned above fixes build with semanage and no other option which uses "globalI" symbol (it seems that in standard PLD build it comes from WITH_SQLITE, enabled by dbsql switch); probably dbsql wasn't detected properly in my previous build of 5.4.15. -- Jakub Bogusz http://qboosh.pl/ From qboosh at pld-linux.org Thu Feb 23 21:36:13 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Thu, 23 Feb 2017 21:36:13 +0100 Subject: pld rpm 5.4.17 In-Reply-To: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com> References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com> Message-ID: <20170223203613.GB20987@mail> On Thu, Feb 23, 2017 at 01:43:14PM -0500, Jeffrey Johnson wrote: > This one is left though: > > > error: db3: header #187105280 cannot be loaded -- skipping. > > error: db3: header #4127850496 cannot be loaded -- skipping. > > How to check what these "headers" mean? > (old, unsupported keys? some old packages with missing fields which are > now required?) > The error message is printed on a headerCopyLoad() failure. > > The failure is usually an indication of header damage of some sort, not missing fields now required. > > The numbers are primary keys into Packages printed in in decimal. > > The values appear to be in the wrong-endian order when converted to hex > 0xB270000 > 0xF60A0000 > > You can try finding the header by doing, say, rpm -q ???querybynumber 0xB270000 > (or its reverse: its unclear what order). Same result for bigger values, no result after swapping bytes: -bash-4.4# rpm -q --querybynumber 0xF60A0000 error: rpmdb: header #4127850496 cannot be loaded -- skipping. -bash-4.4# rpm -q --querybynumber 0x00000AF6 -bash-4.4# rpm -q --querybynumber 0xB270000 error: rpmdb: header #187105280 cannot be loaded -- skipping. -bash-4.4# rpm -q --querybynumber 0x0000270B -bash-4.4# -- Jakub Bogusz http://qboosh.pl/ From n3npq at me.com Fri Feb 24 01:15:22 2017 From: n3npq at me.com (Jeffrey Johnson) Date: Thu, 23 Feb 2017 19:15:22 -0500 Subject: pld rpm 5.4.17 In-Reply-To: <20170223203613.GB20987@mail> References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com> <20170223203613.GB20987@mail> Message-ID: <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com> > On Feb 23, 2017, at 3:36 PM, Jakub Bogusz wrote: > > On Thu, Feb 23, 2017 at 01:43:14PM -0500, Jeffrey Johnson wrote: >> This one is left though: >> >>> error: db3: header #187105280 cannot be loaded -- skipping. >>> error: db3: header #4127850496 cannot be loaded -- skipping. >> >> How to check what these "headers" mean? >> (old, unsupported keys? some old packages with missing fields which are >> now required?) >> The error message is printed on a headerCopyLoad() failure. >> >> The failure is usually an indication of header damage of some sort, not missing fields now required. >> >> The numbers are primary keys into Packages printed in in decimal. >> >> The values appear to be in the wrong-endian order when converted to hex >> 0xB270000 >> 0xF60A0000 >> >> You can try finding the header by doing, say, rpm -q ???querybynumber 0xB270000 >> (or its reverse: its unclear what order). > > Same result for bigger values, no result after swapping bytes: > > -bash-4.4# rpm -q --querybynumber 0xF60A0000 > error: rpmdb: header #4127850496 cannot be loaded -- skipping. > -bash-4.4# rpm -q --querybynumber 0x00000AF6 > -bash-4.4# rpm -q --querybynumber 0xB270000 > error: rpmdb: header #187105280 cannot be loaded -- skipping. > -bash-4.4# rpm -q --querybynumber 0x0000270B > -bash-4.4# > So at least simply reproducible with ?querybynumber ;-) If you just need a fix, db_dump will give you {KEY, VALUE} pairs in hex. Find the 2 {KEY,VALUES} and delete the 2 occurrences of 2 lines of hex in a text editor. Feed the result to db_load to recreate Packages. rpm ?rebuild will recreate the indices. If you want me to take a look, tar up /var/lib/rpm/{DB_CONFIG,Packages} and give me a URL. hth 73 de Jeff From n3npq at me.com Fri Feb 24 01:26:59 2017 From: n3npq at me.com (Jeffrey Johnson) Date: Thu, 23 Feb 2017 19:26:59 -0500 Subject: RPM + SEMANAGE? In-Reply-To: <20170223203340.GA20987@mail> References: <20170223203340.GA20987@mail> Message-ID: <0D50A4E5-32BF-45AF-BF4F-A42139534E33@me.com> > On Feb 23, 2017, at 3:33 PM, Jakub Bogusz wrote: > > On Thu, Feb 23, 2017 at 02:15:12PM -0500, Jeffrey Johnson wrote: >> I see that semanage has been enabled 3 days ago >> https://github.com/pld-linux/rpm/commit/ec7b8d8fb16f5789772693ff807e0a93a5c653e4 >> >> Be forewarned: the semanage code in RPM hasn???t been looked at for quite some years: at the >> time I implemented, semanage was just being invented. >> >> If there is a need, I can/will update the SELinux modules to latest released in rpm-5.4.18. >> There hasn???t been any detectable interest so far ??? > > Actually, in this commit nothing changed to semanage switch. > It was enabled in 6e115b2320d8152309c7183c8b36641fbb1316b9, over 4 years > ago. > Sorry for the confusion. Still, expect issues with RPM+SEMANAGE (if enabled/used). > The patch mentioned above fixes build with semanage and no other option > which uses "globalI" symbol (it seems that in standard PLD build it > comes from WITH_SQLITE, enabled by dbsql switch); probably dbsql wasn't > detected properly in my previous build of 5.4.15. > Using WITH_SQLITE is tricky because there are 2 libraries with identical symbols but different implementations and you will have to ensure through linkage whether dbsql or sqlite3 is intended. The choices is exclusive ? but there is likely nothing doing configure that ensures only one. The issue will show up with embedded interpreters and loadable modules. I?ve been using the dbsql for embedded sqlite3 testing, and ignoring sqlite3. And ? if using RPM with db-6.1.26/db-6.2.23 there is a patch that is needed. I?ve tried to send that patch a couple of times: its now documented in INSTALL in cvs on the -r rpm-5_4 branch. hth 73 de Jeff > > -- > Jakub Bogusz http://qboosh.pl/ > _______________________________________________ > pld-devel-en mailing list > pld-devel-en at lists.pld-linux.org > http://lists.pld-linux.org/mailman/listinfo/pld-devel-en From qboosh at pld-linux.org Fri Feb 24 23:03:47 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Fri, 24 Feb 2017 23:03:47 +0100 Subject: pld rpm 5.4.17 In-Reply-To: <20170222210217.GA13257@mail> References: <587537B8.50603@pld-linux.org> <20170221205113.GA2445@mail> <20170222210217.GA13257@mail> Message-ID: <20170224220347.GA27803@mail> On Wed, Feb 22, 2017 at 10:02:17PM +0100, Jakub Bogusz wrote: > On Tue, Feb 21, 2017 at 09:51:13PM +0100, Jakub Bogusz wrote: > > On Tue, Jan 10, 2017 at 09:36:24PM +0200, Elan Ruusam?e wrote: > > > not cool. > > > > > > $ rpm -q rpm > > > BDB0056 DB->cursor: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW > > > require locking > > > error: db3copen:db3.c:1470: db->cursor(22): Invalid argument > > > BDB0056 DB->cursor: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW > > > require locking > > > error: db3copen:db3.c:1470: db->cursor(22): Invalid argument > > > BDB0630 DB_THREAD mandates memory allocation flag on primary key DBT > > > error: db3cpget:db3.c:1568: db->pget(22): Invalid argument > > > error: error(22) getting keys from Nvra index > > > error: error(1) getting records from Nvra index > > > package rpm is not installed > > > > > > luckily was able to downgrade. > > > > Which db version was used? > > Was that default system db (with headers in /usr/include)? > > > > I experienced such behaviour when trying to upgrade rpm 5.4.15+db 6.1.19 > > to 5.4.17+db 6.1.29 (while the system db is 6.2.23, maybe some file > > caught system db.h) > > Update: still the same with uid>0 (no write permission to /var/lib/rpm). > It seems that DB_READ_COMMITTED and DB_READ_UNCOMMITTED require +w, > so they should be filtered out in unprivileged mode. Issue explained: DB_READ_COMMITTED and DB_READ_UNCOMMITTED are cursor/get/set/txn options. DB_RDONLY and DB_READ_UNCOMMITTED are open options. But... DB_READ_COMMITTED and DB_RDONLY have the same value! So "read_committed" cannot be applied to dbi_oflags (should be in eflags?) and applied to DB->cursor operations, like it's done since rpm 5.4.17 (leaving DB_READ_COMMITTED bit from oflags to cursor causes DB_RDONLY bit to be passed as DB_READ_COMITTED, which conflicts with read-only database). -- Jakub Bogusz http://qboosh.pl/ From n3npq at me.com Sat Feb 25 01:35:46 2017 From: n3npq at me.com (Jeffrey Johnson) Date: Fri, 24 Feb 2017 19:35:46 -0500 Subject: pld rpm 5.4.17 In-Reply-To: <20170224220347.GA27803@mail> References: <587537B8.50603@pld-linux.org> <20170221205113.GA2445@mail> <20170222210217.GA13257@mail> <20170224220347.GA27803@mail> Message-ID: <4E51E180-1CFC-4269-AE7C-2F357EB058E8@me.com> > On Feb 24, 2017, at 5:03 PM, Jakub Bogusz wrote: > >>> >> >> Update: still the same with uid>0 (no write permission to /var/lib/rpm). >> It seems that DB_READ_COMMITTED and DB_READ_UNCOMMITTED require +w, >> so they should be filtered out in unprivileged mode. > > Issue explained: > > DB_READ_COMMITTED and DB_READ_UNCOMMITTED are cursor/get/set/txn options. > DB_RDONLY and DB_READ_UNCOMMITTED are open options. > > But... DB_READ_COMMITTED and DB_RDONLY have the same value! > So "read_committed" cannot be applied to dbi_oflags (should be in > eflags?) and applied to DB->cursor operations, like it's done since rpm > 5.4.17 (leaving DB_READ_COMMITTED bit from oflags to cursor causes DB_RDONLY > bit to be passed as DB_READ_COMITTED, which conflicts with read-only database). > I?m sure your analysis is correct ? ? meanwhile what was the core issue? What problem are you trying to solve? Issues relating to whether a cursor COULD (or SHOULD) read uncommitted data are quite complex, including whether the flags are inherited from the database open and applied to the cursor open. RPM code doesn?t directly set either of those flags, and likely Just Works with either setting depending if/when correctly configured. But there are almost no checks on the configuration from macros, and DB_CONFIG, not RPM macros, is the preferred means of configuring Berkeley DB because standard and documented. FWIW, patches to RPM to open the database RDONLY, and then reopen RDWRITE, are known to be racy. Attempts to ?harden? an rpmdb trigger have never been shown to be effective (in my experience), and triggers Yet Another Level Of Exclusive Locking just in case in an attempt to avoid raciness. Simplest would be to open RDONLY when querying/verifying and RDWRITE when installing/erasing, and moving the file based dbenv into /var/cache/rpm with 666 permissions or 660+setgid (or equivalent). But there?s no ability to discuss these changes rationally, so everything never changes. Opening a database RDONLY does not mean that there are no writes: in fact, shared/read locks MUST be written even if the database is opened RDONLY. What was the core issue? What problem are you trying to solve? hth 73 de Jeff From n3npq at me.com Sat Feb 25 02:10:33 2017 From: n3npq at me.com (Jeffrey Johnson) Date: Fri, 24 Feb 2017 20:10:33 -0500 Subject: pld rpm 5.4.17 In-Reply-To: <4E51E180-1CFC-4269-AE7C-2F357EB058E8@me.com> References: <587537B8.50603@pld-linux.org> <20170221205113.GA2445@mail> <20170222210217.GA13257@mail> <20170224220347.GA27803@mail> <4E51E180-1CFC-4269-AE7C-2F357EB058E8@me.com> Message-ID: > > I?m sure your analysis is correct ? > > ? meanwhile what was the core issue? What problem are you trying to solve? > > Issues relating to whether a cursor COULD (or SHOULD) read uncommitted data > are quite complex, including whether the flags are inherited from the database open > and applied to the cursor open. RPM code doesn?t directly set either of those flags, > and likely Just Works with either setting depending if/when correctly configured. > As you probably know, these 3 lines are the cause of the problem: #define FMASK (DB_READ_UNCOMMITTED|DB_READ_COMMITTED) flags |= (dbi->dbi_oflags & FMASK); #undef FMASK As near as I can recall, dbi_oflags on some older version of Berekely DB was the place to set the flags. BerkeleyDB has evolved to finer grained control of flags since then. Dropping DB_READ_COMMITTED in FMASK prevents DB_READONLY from propagating. But the then the issue of whether ?read_committed? and ?read_uncommitted? read from macros makes any sense at all. I?ll try to sort the issue before rpm-5.4.18. 73 de Jeff From qboosh at pld-linux.org Sat Feb 25 08:12:34 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Sat, 25 Feb 2017 08:12:34 +0100 Subject: pld rpm 5.4.17 In-Reply-To: References: <587537B8.50603@pld-linux.org> <20170221205113.GA2445@mail> <20170222210217.GA13257@mail> <20170224220347.GA27803@mail> <4E51E180-1CFC-4269-AE7C-2F357EB058E8@me.com> Message-ID: <20170225071234.GA28618@mail> On Fri, Feb 24, 2017 at 08:10:33PM -0500, Jeffrey Johnson wrote: > > I???m sure your analysis is correct ??? > > > > ??? meanwhile what was the core issue? What problem are you trying to solve? This one - inability to query database without write access to db environment (i.e. from plain user account). > > Issues relating to whether a cursor COULD (or SHOULD) read uncommitted data > > are quite complex, including whether the flags are inherited from the database open > > and applied to the cursor open. RPM code doesn???t directly set either of those flags, > > and likely Just Works with either setting depending if/when correctly configured. I didn't set db_read_uncommitted or db_read_committed in *dbi* macros. DB_RDONLY is set by db3open() if there is no write access to dbenv (access(dbhome, W_OK) == -1). > As you probably know, these 3 lines are the cause of the problem: > > #define FMASK (DB_READ_UNCOMMITTED|DB_READ_COMMITTED) > flags |= (dbi->dbi_oflags & FMASK); > #undef FMASK > > As near as I can recall, dbi_oflags on some older version of Berekely DB > was the place to set the flags. BerkeleyDB has evolved to finer > grained control of flags since then. > > Dropping DB_READ_COMMITTED in FMASK prevents DB_READONLY from > propagating. According to BDB error: BDB0056 DB->cursor: DB_READ_COMMITTED, DB_READ_UNCOMMITTED and DB_RMW require locking error: db3copen:db3.c:1470: db->cursor(22): Invalid argument DB_READ_COMMITTED (as well as DB_READ_UNCOMMITTED) requires locking, i.e. write access to dbenv, so it cannot be used with database open with DB_RDONLY. > But the then the issue of whether ???read_committed??? and ???read_uncommitted??? > read from macros makes any sense at all. Yes, just dropping propagating DB_RDONLY(==DB_READ_COMMITTED) from dbi_oflags to DB->cursor flags disables passing read_committed from *dbi* macros. To support this option, it must be passed in some other way, not dbi_oflags used for DB->open (nb. for DB->open it would mean... DB_RDONLY). -- Jakub Bogusz http://qboosh.pl/ From qboosh at pld-linux.org Sat Feb 25 08:18:25 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Sat, 25 Feb 2017 08:18:25 +0100 Subject: mail infrastructure issues: x86_64 builder logs, CVS commit logs Message-ID: <20170225071825.GA31658@mail> I found two issues, probably not related: - builder logs from th-x86_64 are not (always) delivered; it seems to affect all FAIL logs and some OK logs (since few weeks) - commit logs from CVS repository (e.g. PLD-update-TODO) seem not delivered to pld-commit mailing list (since few months) -- Jakub Bogusz http://qboosh.pl/ From qboosh at pld-linux.org Tue Feb 28 21:42:33 2017 From: qboosh at pld-linux.org (Jakub Bogusz) Date: Tue, 28 Feb 2017 21:42:33 +0100 Subject: pld rpm 5.4.17 In-Reply-To: <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com> References: <102CCCFA-71F1-4843-9E08-D83DA52A78E6@me.com> <20170223203613.GB20987@mail> <57B5C0F3-F6DB-4679-B8AD-C969117E0EBF@me.com> Message-ID: <20170228204233.GA12663@mail> On Thu, Feb 23, 2017 at 07:15:22PM -0500, Jeffrey Johnson wrote: > > > On Feb 23, 2017, at 3:36 PM, Jakub Bogusz wrote: > > > > On Thu, Feb 23, 2017 at 01:43:14PM -0500, Jeffrey Johnson wrote: > >> This one is left though: > >> > >>> error: db3: header #187105280 cannot be loaded -- skipping. > >>> error: db3: header #4127850496 cannot be loaded -- skipping. > >> > >> How to check what these "headers" mean? > >> (old, unsupported keys? some old packages with missing fields which are > >> now required?) > >> The error message is printed on a headerCopyLoad() failure. > >> > >> The failure is usually an indication of header damage of some sort, not missing fields now required. > >> > >> The numbers are primary keys into Packages printed in in decimal. > >> > >> The values appear to be in the wrong-endian order when converted to hex > >> 0xB270000 > >> 0xF60A0000 > >> > >> You can try finding the header by doing, say, rpm -q ???querybynumber 0xB270000 > >> (or its reverse: its unclear what order). > > > > Same result for bigger values, no result after swapping bytes: > > > > -bash-4.4# rpm -q --querybynumber 0xF60A0000 > > error: rpmdb: header #4127850496 cannot be loaded -- skipping. > > -bash-4.4# rpm -q --querybynumber 0x00000AF6 > > -bash-4.4# rpm -q --querybynumber 0xB270000 > > error: rpmdb: header #187105280 cannot be loaded -- skipping. > > -bash-4.4# rpm -q --querybynumber 0x0000270B > > -bash-4.4# > > > > So at least simply reproducible with ???querybynumber ;-) > > If you just need a fix, db_dump will give you {KEY, VALUE} pairs in hex. > Find the 2 {KEY,VALUES} and delete the 2 occurrences of 2 lines of hex in a text editor. > Feed the result to db_load to recreate Packages. > rpm ???rebuild will recreate the indices. I investigated these two entries. They doesn't seem to be invalid, but are not accepted after this change: http://rpm5.org/cvs/filediff?f=rpm/rpmdb/header.c&v1=1.198.2.23&v2=1.198.2.24 more precisely, this part: -/*@=sizeoftype@*/ - rpmuint32_t * stei = (rpmuint32_t *) - memcpy(alloca(nb), dataStart + off, nb); + /* XXX copy to fix alignment problems */ + rpmuint32_t * stei = (rpmuint32_t *) + memcpy(alloca(nb), dataStart + off, nb); + if ((off + nb) > dl) + goto errxit; rdl = (rpmuint32_t)-ntohl(stei[2]); /* negative offset */ -assert((rpmint32_t)rdl >= 0); /* XXX insurance */ - ril = (rpmuint32_t)(rdl/sizeof(*pe)); - if (hdrchkTags(ril) || hdrchkData(rdl)) + if (rdl < REGION_TAG_COUNT || rdl > (rpmuint32_t)(off+nb)) goto errxit; (nb. IMO "if ((off + nb) > dl)" check should be done before memcpy...) 1) package #0xB270000 the first entry of header is: 0000003f 00000007 00000280 00000010 it points to data: dataStart+0280 0000003f 00000007 fffffca0 00000010 so rdl=0x360, which is bigger than off+nb (0x280+0x10) but AFAICS rdl is pointer inside header part, not data... 2) similarly, package #0xF60A0000 has first header entry: 0000003f 00000007 00000378 00000010 which points to data: dataStart+0378 0000003f 00000007 fffffc70 00000010 and rdl=0x390 > 0x378+0x10 The rest of packages I've got installed seem to have longer data in "immutable" part and pass this check. -- Jakub Bogusz http://qboosh.pl/