[projects/buildlogs] Replace PHP/shell buildlog pipeline with addlog.py
arekm
arekm at pld-linux.org
Mon Apr 20 20:19:25 CEST 2026
commit f38d9ada1e969d7d96397c9b8b6840fc541a8be9
Author: Arkadiusz Miśkiewicz <arekm at maven.pl>
Date: Mon Apr 20 20:10:01 2026 +0200
Replace PHP/shell buildlog pipeline with addlog.py
.gitignore | 7 +
PRZECZYTAJ.TO | 32 --
READ.ME | 30 --
README.md | 46 ++
addlog.py | 852 ++++++++++++++++++++++++++++++++++++
buildlogs.example.ini | 18 +
buildlogs.inc | 9 -
helpers/README-pl | 16 -
helpers/buildlogs-inotify-mover.sh | 14 -
helpers/buildlogs-mover.conf | 3 -
helpers/buildlogs-mover.sh | 66 ---
index.php | 120 ++++-
init.sql | 86 ----
lib.php | 40 +-
pld-buildlogs/scripts/addlog.php | 126 ------
pld-buildlogs/scripts/migration.php | 108 -----
16 files changed, 1047 insertions(+), 526 deletions(-)
---
diff --git a/.gitignore b/.gitignore
new file mode 100644
index 0000000..aa293f9
--- /dev/null
+++ b/.gitignore
@@ -0,0 +1,7 @@
+docs/
+*~
+.playwright-mcp/
+__pycache__/
+buildlogs.ini
+addlog.lock
+addlog.log
diff --git a/PRZECZYTAJ.TO b/PRZECZYTAJ.TO
deleted file mode 100644
index 0c4dc9d..0000000
--- a/PRZECZYTAJ.TO
+++ /dev/null
@@ -1,32 +0,0 @@
-# vim:fileencoding=UTF-8
-# $Revision: 1.3 $, $Date: 2007-11-28 12:42:52 $
-
-Poprzedni silnik buildlogów był mało wydajny, szczególnie dla szukania
-zaawansowanego. Ten silnik korzysta z bazy sqlite3 (php-pdo-sqlite)
-i jest w miarę szybki.
-
-Najpierw należy zainicjalizować bazę danych przy pomocy skryptu migration.php.
-Skrypt ten wymaga php-program. Należy go uruchomić raz. Czas wykonywania
-zależy od liczby plików i filesystemu. Może to trwać kilka minut.
-
-Następnie umieścić index.php, buildlogs.inc i powpld.png na serwerze www.
-Skrypt wymaga dodatkowo php-gettext.
-
-Każdy nowy buildlog (dla nowego lub starego speca) powinien zostać
-dodany do bazy używając skryptu addlog.php (korzysta z php-cli i php-pdo-sqlite).
-W zależności od tego czy spec jest nowy czy nie, w bazie zostanie uaktualniony
-rekord lub dodany nowy. Parametrem dla addlog.php jest bezwzględna ścieżka
-do pliku loga,
-np. addlog.php /home/services/ftp/pub/pld-buildlogs/ac/i686/OK/kernel.bz2
-
-
-Skrypty index.php, addlog.php i migration.php używają zmiennej $database.
-Zmienna ta powinna być jednakowa we wszystkich trzech skryptach.
-Jej obecna wartość to 'sqlite:/home/services/ftp/buildlogs.db'.
-Należy ją ustawić na taką wartość, by plik bazy i katalog, w którym się on znajduje
-był zapisywalny przez skrypty migration.php i addlog.php oraz możliwy do odczytania przez
-index.php.
-
-Konfiguracja skryptów jest zapisana w pliku buildlogs.inc.
-Po każdej zmianie w tym pliku należy się upewnić, czy numerki się zgadzają
-i przebudować bazę uruchamiając migration.php.
diff --git a/READ.ME b/READ.ME
deleted file mode 100644
index 70c0f7c..0000000
--- a/READ.ME
+++ /dev/null
@@ -1,30 +0,0 @@
-# vim:fileencoding=UTF-8
-# $ Revision: 1.4 $, $ Date: 2008/10/20 7:55:09 p.m. $
-
-Previous buildlogs engine was inefficient, especially for advanced search. This
-engine uses the sqlite3 database (php-pdo-sqlite) and is quite fast.
-
-First you must initialize a database using a script migration.php.
-This script requires php-program. Please run it again. Execution time
-depends on the number of files and file system. This may take several minutes.
-
-Then put the index.php, and powpld.png buildlogs.inc web server.
-The script also requires php-gettext.
-
-Each new buildlog (for new or old spec) should be
-added to the database using the script addlog.php (using php-cli and php-pdo-sqlite).
-Depending on whether the spec is new or not, the database will be updated
-or added a new record. Addlog.php parameter for the absolute path
-to file logos,
-eg addlog.php /home/services/ftp/pub/pld-buildlogs/ac/i686/OK/kernel.bz2
-
-Scripts index.php migration.php addlog.php and use the variable $database.
-This variable should be identical in all three scripts.
-Its current value is 'sqlite:/home/services/ftp/buildlogs.db'.
-It must be set to a value that the database file and directory in which it is situated
-was writable by scripts and addlog.php migration.php and impossible to read!
-index.php.
-
-Configuration script is saved in a file buildlogs.inc.
-After each change in this file, make sure that passes are correct
-and rebuild the database by running migration.php.
diff --git a/README.md b/README.md
new file mode 100644
index 0000000..a3683e8
--- /dev/null
+++ b/README.md
@@ -0,0 +1,46 @@
+# pld-buildlogs
+
+SQLite-backed index + web viewer for PLD build logs.
+
+## Components
+
+- `index.php` — web frontend (PHP-CLI + `php-pdo-sqlite` + `php-gettext`).
+- `addlog.py` — long-running Python script that watches the build-log tree, moves completed logs from `.new/` into `OK/`/`FAIL/`, and maintains the SQLite index.
+- `lib.php` — PHP helpers (decompressor dispatch).
+- `buildlogs.example.ini` — config template.
+- `helpers/install-buildlogs-tree.sh` — creates the `$root/$dist/$arch/{.new,OK,FAIL,prevOK}` layout for a new host.
+
+## Dependencies
+
+**PHP:** `php-pdo-sqlite`, `php-gettext`.
+**Python 3:** `inotify_simple`, `zstandard`, `python-lzo` (install via PLD packaging).
+**Binaries:** `lbzcat`/`bzcat`, `zcat`, `xzcat`, `zstdcat`, `lzop` (for PHP viewer decompression).
+
+## Setup
+
+1. Copy `buildlogs.example.ini` → `buildlogs.ini` and edit paths.
+2. Create the log directory (`/var/log/pld-buildlogs/`) and the lock directory (`/var/run/`) with write permission for the cron user.
+3. Create the build-log tree with `helpers/install-buildlogs-tree.sh <root>`.
+4. Add a cron entry:
+ ```
+ * * * * * /path/to/checkout/addlog.py --quiet
+ ```
+ `addlog.py` creates the SQLite database and table on first run. Subsequent runs find the lock held by the still-active watcher and silently exit.
+
+## `addlog.py` modes
+
+| Invocation | Behaviour |
+|---|---|
+| `addlog.py` | Long-running: full-tree scan on start, then inotify watch. |
+| `addlog.py --cleanup` | One-shot: drop DB rows whose files are missing. |
+| `addlog.py --backfill` | One-shot: fill NULL parsed columns from existing logs. |
+| `addlog.py --debug` | Verbose logging to stderr + log file. |
+| `addlog.py --quiet` | No stderr; log file only (default for cron). |
+
+## Schema
+
+`addlog.py` manages the `logs` table: creation on first run, additive `ALTER TABLE ADD COLUMN` for columns it knows about that aren't in the live DB. Column renames, type changes, or drops are manual operator tasks (out of script scope).
+
+## Supported log compressions
+
+`bz2`, `gz`, `xz`, `zst`, `lzo`, and raw (no extension). The on-disk filename determines the compression; `addlog.py` and `index.php` probe the filesystem to find the actual suffix.
diff --git a/addlog.py b/addlog.py
new file mode 100755
index 0000000..bfd5954
--- /dev/null
+++ b/addlog.py
@@ -0,0 +1,852 @@
+#!/usr/bin/env python3
+"""addlog.py — watch build-log tree, keep SQLite index in sync.
+
+Modes:
+ (default) long-running watcher — full-scan + inotify
+ --cleanup one-shot prune of DB rows whose files are gone
+ --backfill one-shot fill of NULL parsed columns
+"""
+from __future__ import annotations
+
+import argparse
+import bz2 as _bz2
+import configparser
+import fcntl
+import gzip as _gzip
+import logging
+import lzma as _lzma
+import os
+import re
+import signal
+import sqlite3
+import sys
+import time
+from dataclasses import dataclass
+from logging.handlers import RotatingFileHandler
+from multiprocessing import Pool
+from pathlib import Path
+
+
+class ConfigError(Exception):
+ pass
+
+
+ at dataclass(frozen=True)
+class Config:
+ database: str
+ root: str
+ lock: str
+ log: str
+ workers: int
+
+
+def _default_config_path() -> str:
+ return str(Path(__file__).resolve().parent / "buildlogs.ini")
+
+
+def load_config(path: str | None = None) -> Config:
+ ini_path = path or _default_config_path()
+ if not os.path.isfile(ini_path):
+ raise ConfigError(f"config file not found: {ini_path}")
+ parser = configparser.ConfigParser()
+ parser.read(ini_path)
+ if "buildlogs" not in parser:
+ raise ConfigError(f"[buildlogs] section missing in {ini_path}")
+ sec = parser["buildlogs"]
+ if "database" not in sec:
+ raise ConfigError("required key 'database' missing in [buildlogs]")
+ if "root" not in sec:
+ raise ConfigError("required key 'root' missing in [buildlogs]")
+ script_dir = Path(__file__).resolve().parent
+ workers_raw = sec.getint("workers", fallback=0)
+ if workers_raw <= 0:
+ workers_raw = max(1, (os.cpu_count() or 2) - 1)
+ return Config(
+ database=sec["database"],
+ root=sec["root"],
+ lock=sec.get("lock", str(script_dir / "addlog.lock")),
+ log=sec.get("log", str(script_dir / "addlog.log")),
+ workers=workers_raw,
+ )
+
+
+LOG_FORMAT = "%(asctime)s %(levelname)s %(name)s: %(message)s"
+
+
+def setup_logging(log_path: str, *, quiet: bool, debug: bool) -> None:
+ level = logging.DEBUG if debug else logging.INFO
+ root = logging.getLogger()
+ root.setLevel(level)
+ # Prevent handler accumulation when called more than once (e.g. in tests).
+ for h in list(root.handlers):
+ root.removeHandler(h)
+ fmt = logging.Formatter(LOG_FORMAT)
+ os.makedirs(os.path.dirname(log_path) or ".", exist_ok=True)
+ file_h = RotatingFileHandler(log_path, maxBytes=10_000_000, backupCount=5)
+ file_h.setFormatter(fmt)
+ file_h.setLevel(level)
+ root.addHandler(file_h)
+ if not quiet:
+ stream_h = logging.StreamHandler(sys.stderr)
+ stream_h.setFormatter(fmt)
+ stream_h.setLevel(level)
+ root.addHandler(stream_h)
+
+
+_stop = False
+
+
+def _handle_signal(signum, frame):
+ global _stop
+ sig_name = signal.Signals(signum).name
+ if _stop:
+ # Second signal — graceful shutdown didn't land. Hard-exit.
+ logging.getLogger("addlog").warning("received second %s, hard-exit", sig_name)
+ signal.signal(signum, signal.SIG_DFL)
+ os.kill(os.getpid(), signum)
+ return
+ _stop = True
+ logging.getLogger("addlog").info("received %s, shutting down", sig_name)
+
+
+def install_signal_handlers() -> None:
+ signal.signal(signal.SIGTERM, _handle_signal)
+ signal.signal(signal.SIGINT, _handle_signal)
+
+
+def should_stop() -> bool:
+ return _stop
+
+
+def acquire_lock(lock_path: str) -> int | None:
+ """Return an open fd that holds an exclusive flock, or None if held elsewhere.
+ Caller is responsible for os.close() on shutdown.
+ """
+ os.makedirs(os.path.dirname(lock_path) or ".", exist_ok=True)
+ fd = os.open(lock_path, os.O_RDWR | os.O_CREAT, 0o644)
+ try:
+ fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
+ except BlockingIOError:
+ os.close(fd)
+ return None
+ os.ftruncate(fd, 0)
+ os.write(fd, f"{os.getpid()}\n".encode())
+ return fd
+
+
+def release_lock(fd: int) -> None:
+ try:
+ os.close(fd)
+ except OSError:
+ pass
+
+
+SCHEMA: list[tuple[str, str, str | None]] = [
+ # (column, type, default)
+ ("log_id", "INTEGER PRIMARY KEY", None),
+ ("dist", "TEXT", None),
+ ("arch", "TEXT", None),
+ ("ok", "INTEGER", None),
+ ("name", "TEXT", None),
+ ("size", "INTEGER", None),
+ ("mtime", "INTEGER", None),
+ ("id", "TEXT", "''"),
+ ("runtime", "INTEGER", None),
+ ("build_rpm_section", "TEXT", None),
+]
+INDEXES: list[tuple[str, str, str]] = [
+ ("dao_index", "logs", "dist, arch, ok"),
+]
+
+
+def _column_def(col: str, typ: str, default: str | None) -> str:
+ base = f"{col} {typ}"
+ if default is not None:
+ base += f" DEFAULT {default}"
+ return base
+
+
+def connect_db(path: str) -> sqlite3.Connection:
+ os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
+ conn = sqlite3.connect(path, isolation_level=None) # autocommit
+ # Classic rollback journal (not WAL) — so readers (httpd) don't need
+ # write access to the DB's directory for the -shm/-wal sidecar files.
+ conn.execute("PRAGMA journal_mode=DELETE")
+ # Wait up to 5s if a concurrent writer holds the exclusive lock.
+ conn.execute("PRAGMA busy_timeout=5000")
+ conn.row_factory = sqlite3.Row
+ return conn
+
+
+def ensure_schema(conn: sqlite3.Connection) -> None:
+ cols_sql = ", ".join(_column_def(c, t, d) for c, t, d in SCHEMA)
+ conn.execute(f"CREATE TABLE IF NOT EXISTS logs({cols_sql})")
+ for name, table, body in INDEXES:
+ conn.execute(f"CREATE INDEX IF NOT EXISTS {name} ON {table}({body})")
+ existing = {row["name"] for row in conn.execute("PRAGMA table_info(logs)").fetchall()}
+ for col, typ, default in SCHEMA:
+ if col in existing:
+ continue
+ sql = f"ALTER TABLE logs ADD COLUMN {_column_def(col, typ, default)}"
+ logging.getLogger("addlog").info("applying migration: %s", sql)
+ conn.execute(sql)
+
+
+_IDENT_RE = re.compile(r"^[A-Za-z0-9][A-Za-z0-9_.+-]*$")
+KNOWN_EXTS = ("bz2", "gz", "xz", "zst", "lzo")
+
+
+def is_valid_identifier(s: str) -> bool:
+ return bool(_IDENT_RE.match(s))
+
+
+def parse_filename(filename: str) -> tuple[str, str, str] | None:
+ """Return (name, id, ext) or None if filename is invalid.
+ ext is '' for raw (uncompressed) logs, otherwise one of KNOWN_EXTS.
+ """
+ if not filename or filename.startswith("."):
+ return None
+ ext = ""
+ stem = filename
+ for candidate in KNOWN_EXTS:
+ suffix = "." + candidate
+ if filename.endswith(suffix):
+ ext = candidate
+ stem = filename[: -len(suffix)]
+ break
+ if "," in stem:
+ name, _, id_ = stem.rpartition(",")
+ else:
+ name, id_ = stem, ""
+ if not is_valid_identifier(name):
+ return None
+ if id_ and not is_valid_identifier(id_):
+ return None
+ return (name, id_, ext)
+
+
+RT_BUILD = re.compile(rb"^Build-Time:.*\breal: ?(\d+(?:\.\d+)?)s")
+RT_DONE = re.compile(rb"^ended at: .+, done in (\d+):(\d+):(\d+)(?:\.\d+)?")
+RPM_SECTION = re.compile(rb"^Executing\(%([A-Za-z_]+)\):")
+
+
+def _open_for_read(path: str, ext: str):
+ if ext == "":
+ return open(path, "rb")
+ if ext == "bz2":
+ return _bz2.open(path, "rb")
+ if ext == "gz":
+ return _gzip.open(path, "rb")
+ if ext == "xz":
+ return _lzma.open(path, "rb")
+ if ext == "zst":
+ import io
+ import zstandard
+ return io.BufferedReader(zstandard.ZstdDecompressor().stream_reader(open(path, "rb")))
+ if ext == "lzo":
+ import io
+ import lzo
+ # python-lzo has no streaming API; logs are small, read whole file.
+ with open(path, "rb") as fh:
+ raw = fh.read()
+ return io.BytesIO(lzo.decompress(raw))
+ raise ValueError(f"unknown ext: {ext!r}")
+
+
+def parse_log(path: str, ext: str) -> dict:
+ """Return parsed fields from a log file.
+
+ Keys returned: {'runtime': int|None, 'build_rpm_section': str|None}.
+ Extend this function when new parsed columns are added to the schema.
+ `build_rpm_section` is the last section name from `Executing(%<name>):` lines
+ (the section that was running when the build ended). The caller decides
+ whether to persist it — typically only for FAIL rows.
+ """
+ build_rt = None
+ done_rt = None
+ last_section = None
+ try:
+ with _open_for_read(path, ext) as fh:
+ for line in fh:
+ m = RT_BUILD.match(line)
+ if m:
+ build_rt = int(float(m.group(1)) + 0.5)
+ continue
+ m = RT_DONE.match(line)
+ if m:
+ done_rt = int(m.group(1)) * 3600 + int(m.group(2)) * 60 + int(m.group(3))
+ continue
+ m = RPM_SECTION.match(line)
+ if m:
+ last_section = m.group(1)
+ except (OSError, EOFError, ValueError) as e:
+ logging.getLogger("addlog").warning("parse_log: error reading %s: %s", path, e)
+ return {"runtime": None, "build_rpm_section": None}
+ return {
+ "runtime": build_rt if build_rt is not None else done_rt,
+ "build_rpm_section": last_section.decode("ascii", "replace") if last_section else None,
+ }
+
+
+STATUS_TO_OK = {"OK": 1, "FAIL": 0}
+
+
+def _worker_parse(args):
+ """Run in worker process. Returns (key, parsed_fields)."""
+ key, path, ext = args
+ return (key, parse_log(path, ext))
+
+
+def _scandir_or_warn(path: str):
+ """os.scandir() that logs and returns an empty iterator on OSError."""
+ try:
+ return os.scandir(path)
+ except FileNotFoundError:
+ return iter(())
+ except OSError as e:
+ logging.getLogger("addlog").warning("skipping %s: %s", path, e)
+ return iter(())
+
+
+def _iter_fs_logs(root: str):
+ """Yield (dist, arch, ok, name, id, ext, path, size, mtime) for every valid log on disk."""
+ for dist_entry in _scandir_or_warn(root):
+ if not dist_entry.is_dir(follow_symlinks=False):
+ continue
+ if not is_valid_identifier(dist_entry.name):
+ continue
+ for arch_entry in _scandir_or_warn(dist_entry.path):
+ if not arch_entry.is_dir(follow_symlinks=False):
+ continue
+ if not is_valid_identifier(arch_entry.name):
+ continue
+ for status, ok in STATUS_TO_OK.items():
+ sdir = os.path.join(arch_entry.path, status)
+ for e in _scandir_or_warn(sdir):
+ if not e.is_file(follow_symlinks=False):
+ continue
+ parsed = parse_filename(e.name)
+ if parsed is None:
+ continue
+ name, id_, ext = parsed
+ try:
+ st = e.stat(follow_symlinks=False)
+ except OSError:
+ continue
+ yield (dist_entry.name, arch_entry.name, ok,
+ name, id_, ext, e.path, st.st_size, int(st.st_mtime))
+
+
+def full_scan(conn: sqlite3.Connection, root: str, *, workers: int) -> dict:
+ log = logging.getLogger("addlog")
+ t0 = time.monotonic()
+
+ # 1. Snapshot DB.
+ db_index: dict[tuple, tuple] = {} # (dist,arch,name,id) -> (log_id, size, mtime, ok)
+ for row in conn.execute(
+ "SELECT log_id, dist, arch, name, id, size, mtime, ok FROM logs"
+ ):
+ db_index[(row["dist"], row["arch"], row["name"], row["id"] or "")] = (
+ row["log_id"], row["size"], row["mtime"], row["ok"]
+ )
+
+ seen: set[tuple] = set()
+ to_parse: list = [] # [(key, path, ext, size, mtime, ok, action)]
+ unchanged = 0
+
+ # 2. Walk.
+ now = int(time.time())
+ for dist, arch, ok, name, id_, ext, path, size, mtime in _iter_fs_logs(root):
+ # Clamp future timestamps for storage, but compare raw mtime to DB.
+ stored_mtime = min(mtime, now)
+ key = (dist, arch, name, id_)
+ seen.add(key)
+ existing = db_index.get(key)
+ if existing is not None:
+ e_id, e_size, e_mtime, e_ok = existing
+ if e_size == size and e_mtime == mtime and e_ok == ok:
+ unchanged += 1
+ continue
+ action = ("update", e_id)
+ else:
+ action = ("insert", None)
+ to_parse.append((key, path, ext, size, stored_mtime, ok, action))
+
+ # 3. Parallel parse.
+ parsed_by_key: dict[tuple, dict] = {}
+ if to_parse:
+ args_iter = ((row[0], row[1], row[2]) for row in to_parse)
+ if workers > 1:
+ pool = Pool(processes=workers)
+ try:
+ for key, parsed in pool.imap_unordered(_worker_parse, args_iter, chunksize=32):
+ if should_stop():
+ break
+ parsed_by_key[key] = parsed
+ finally:
+ pool.terminate()
+ pool.join()
+ else:
+ for a in args_iter:
+ if should_stop():
+ break
+ key, parsed = _worker_parse(a)
+ parsed_by_key[key] = parsed
+
+ # 4. Batch writes.
+ new_count = 0
+ upd_count = 0
+ conn.execute("BEGIN")
+ for key, path, ext, size, mtime, ok, action in to_parse:
+ parsed = parsed_by_key.get(key, {"runtime": None, "build_rpm_section": None})
+ dist, arch, name, id_ = key
+ runtime = parsed.get("runtime")
+ section = parsed.get("build_rpm_section") if not ok else None
+ if action[0] == "insert":
+ conn.execute(
+ "INSERT INTO logs(dist, arch, ok, name, size, mtime, id, runtime, build_rpm_section) "
+ "VALUES(?,?,?,?,?,?,?,?,?)",
+ (dist, arch, ok, name, size, mtime, id_, runtime, section))
+ new_count += 1
+ else:
+ log_id = action[1]
+ conn.execute(
+ "UPDATE logs SET ok=?, size=?, mtime=?, runtime=?, build_rpm_section=? WHERE log_id=?",
+ (ok, size, mtime, runtime, section, log_id))
+ upd_count += 1
+ conn.execute("COMMIT")
+
+ # 5. Orphan tally.
+ orphans = [k for k in db_index if k not in seen]
+ dt = time.monotonic() - t0
+ summary = {
+ "total": len(db_index) + new_count,
+ "new": new_count,
+ "updated": upd_count,
+ "unchanged": unchanged,
+ "orphaned": len(orphans),
+ "seconds": round(dt, 2),
+ }
+ log.info(
+ "startup scan: total=%d new=%d updated=%d unchanged=%d orphaned=%d in %.2fs (workers=%d)",
+ summary["total"], summary["new"], summary["updated"],
+ summary["unchanged"], summary["orphaned"], summary["seconds"], workers,
+ )
+ return summary
+
+
+def _read_info(info_path: str) -> tuple[bool, str | None]:
+ """Return (has_end, status_word). status_word is 'OK'|'FAIL'|None."""
+ has_end = False
+ status_word = None
+ try:
+ with open(info_path, "r", errors="replace") as f:
+ for line in f:
+ s = line.rstrip("\n")
+ if s == "END":
+ has_end = True
+ elif s.startswith("Status:"):
+ val = s.split(":", 1)[1].strip()
+ if val == "OK":
+ status_word = "OK"
+ elif val.startswith("FAIL"):
+ status_word = "FAIL"
+ except OSError:
+ return (False, None)
+ return (has_end, status_word)
+
+
+def _upsert_log(conn, dist, arch, ok, name, id_, size, mtime, runtime, section) -> None:
+ row = conn.execute(
+ "SELECT log_id FROM logs WHERE dist=? AND arch=? AND name=? AND id=? LIMIT 1",
+ (dist, arch, name, id_),
+ ).fetchone()
+ if row:
+ conn.execute(
+ "UPDATE logs SET ok=?, size=?, mtime=?, runtime=?, build_rpm_section=? WHERE log_id=?",
+ (ok, size, mtime, runtime, section, row["log_id"]),
+ )
+ else:
+ conn.execute(
+ "INSERT INTO logs(dist, arch, ok, name, size, mtime, id, runtime, build_rpm_section) "
+ "VALUES(?,?,?,?,?,?,?,?,?)",
+ (dist, arch, ok, name, size, mtime, id_, runtime, section),
+ )
+
+
+def handle_info(conn: sqlite3.Connection, info_path: str, root: str) -> None:
+ """Process a single `.info` file: move the matching log, upsert DB row, unlink .info."""
+ log = logging.getLogger("addlog")
+ info_name = os.path.basename(info_path)
+ if not info_name.endswith(".info"):
+ return
+ log_name = info_name[:-len(".info")]
+
+ new_dir = os.path.dirname(info_path)
+ arch_dir = os.path.dirname(new_dir)
+ arch = os.path.basename(arch_dir)
+ dist = os.path.basename(os.path.dirname(arch_dir))
+
+ has_end, status_word = _read_info(info_path)
+ if not has_end:
+ log.debug("skip %s: no END marker yet", info_path)
+ return
+ if status_word is None:
+ log.warning("skip %s: unrecognised Status", info_path)
+ os.unlink(info_path)
+ return
+
+ parsed = parse_filename(log_name)
+ if parsed is None:
+ log.warning("skip %s: invalid log filename %s", info_path, log_name)
+ os.unlink(info_path)
+ return
+ name, id_, ext = parsed
+
+ if not (is_valid_identifier(dist) and is_valid_identifier(arch)):
+ log.warning("skip %s: invalid dist/arch %s/%s", info_path, dist, arch)
+ os.unlink(info_path)
+ return
+
+ src = os.path.join(new_dir, log_name)
+ dst_dir = os.path.join(arch_dir, status_word)
+ dst = os.path.join(dst_dir, log_name)
+
+ os.makedirs(dst_dir, exist_ok=True)
+ try:
+ os.replace(src, dst)
+ except FileNotFoundError:
+ log.warning("skip %s: log file %s missing", info_path, src)
+ os.unlink(info_path)
+ return
+
+ try:
+ st = os.stat(dst)
+ except OSError as e:
+ log.error("stat failed for %s: %s", dst, e)
+ return
+ size = st.st_size
+ mtime = min(int(st.st_mtime), int(time.time()))
+
+ parsed_fields = parse_log(dst, ext)
+ runtime = parsed_fields.get("runtime")
+ ok = STATUS_TO_OK[status_word]
+ # build_rpm_section is meaningful only for failed builds.
+ section = parsed_fields.get("build_rpm_section") if ok == 0 else None
+
+ _upsert_log(conn, dist, arch, ok, name, id_, size, mtime, runtime, section)
+ os.unlink(info_path)
+ log.info("added %s/%s/%s/%s%s%s",
+ dist, arch, status_word, name,
+ ("," + id_) if id_ else "",
+ ("." + ext) if ext else "")
+
+
+def _scan_new_dirs_for_pending_info(conn, root):
+ """Walk every $root/$dist/$arch/.new/ once and handle any .info files present."""
+ log = logging.getLogger("addlog")
+ for dist in _scandir_or_warn(root):
+ if not dist.is_dir(follow_symlinks=False) or not is_valid_identifier(dist.name):
+ continue
+ for arch in _scandir_or_warn(dist.path):
+ if not arch.is_dir(follow_symlinks=False) or not is_valid_identifier(arch.name):
+ continue
+ new_dir = os.path.join(arch.path, ".new")
+ for e in _scandir_or_warn(new_dir):
+ if e.is_file(follow_symlinks=False) and e.name.endswith(".info"):
+ try:
+ handle_info(conn, e.path, root)
+ except Exception:
+ log.exception("handle_info failed for %s", e.path)
+
+
+def _add_new_dir_watch(inotify, wd_to_path, path):
+ from inotify_simple import flags
+ try:
+ wd = inotify.add_watch(path, flags.CLOSE_WRITE | flags.MOVED_TO)
+ wd_to_path[wd] = ("new", path)
+ except FileNotFoundError:
+ pass
+
+
+def _add_container_watch(inotify, wd_to_path, path, kind):
+ from inotify_simple import flags
+ try:
+ wd = inotify.add_watch(path, flags.CREATE | flags.ONLYDIR)
+ wd_to_path[wd] = (kind, path) # kind = "root" | "dist"
+ except FileNotFoundError:
+ pass
+
+
+def _install_watches(inotify, wd_to_path, root):
+ """Install watches on $root, each $dist, and each existing .new/ dir."""
+ if os.path.isdir(root):
+ _add_container_watch(inotify, wd_to_path, root, "root")
+ for dist in _scandir_or_warn(root):
+ if not (dist.is_dir(follow_symlinks=False) and is_valid_identifier(dist.name)):
+ continue
+ _add_container_watch(inotify, wd_to_path, dist.path, "dist")
+ for arch in _scandir_or_warn(dist.path):
+ if not (arch.is_dir(follow_symlinks=False) and is_valid_identifier(arch.name)):
+ continue
+ new_dir = os.path.join(arch.path, ".new")
+ if os.path.isdir(new_dir):
+ _add_new_dir_watch(inotify, wd_to_path, new_dir)
+
+
+def _maybe_watch_newly_created(inotify, wd_to_path, kind, parent, name):
+ """Called on IN_CREATE+IN_ISDIR for container watches."""
+ if not is_valid_identifier(name):
+ return
+ created = os.path.join(parent, name)
+ if kind == "root":
+ # New $dist dir appeared. Watch it and look for .new/ subdirs beneath.
+ _add_container_watch(inotify, wd_to_path, created, "dist")
+ for arch in _scandir_or_warn(created):
+ if arch.is_dir(follow_symlinks=False) and is_valid_identifier(arch.name):
+ new_dir = os.path.join(arch.path, ".new")
+ if os.path.isdir(new_dir):
+ _add_new_dir_watch(inotify, wd_to_path, new_dir)
+ elif kind == "dist":
+ # New $arch under an existing $dist. If it has a .new/, watch it.
+ new_dir = os.path.join(created, ".new")
+ if os.path.isdir(new_dir):
+ _add_new_dir_watch(inotify, wd_to_path, new_dir)
+
+
+def run_watcher(conn, cfg):
+ from inotify_simple import INotify, flags
+ log = logging.getLogger("addlog")
+ inotify = INotify()
+ wd_to_path: dict[int, tuple[str, str]] = {}
+ _install_watches(inotify, wd_to_path, cfg.root)
+
+ # Close the race: anything that landed between end of full_scan and now.
+ _scan_new_dirs_for_pending_info(conn, cfg.root)
+
+ log.info("watcher active: %d inotify watches", len(wd_to_path))
+ try:
+ while not should_stop():
+ try:
+ events = inotify.read(timeout=1000)
+ except OSError as e:
+ log.exception("inotify read error: %s", e)
+ break
+ for ev in events:
+ info = wd_to_path.get(ev.wd)
+ if info is None:
+ continue
+ kind, path = info
+ if kind == "new":
+ if not ev.name.endswith(".info"):
+ continue
+ info_path = os.path.join(path, ev.name)
+ try:
+ handle_info(conn, info_path, cfg.root)
+ except Exception:
+ log.exception("handle_info failed for %s", info_path)
+ elif kind in ("root", "dist") and (ev.mask & flags.ISDIR):
+ _maybe_watch_newly_created(inotify, wd_to_path, kind, path, ev.name)
+ finally:
+ inotify.close()
+
+
+def run_cleanup(conn: sqlite3.Connection, cfg) -> int:
+ """Drop rows whose files are missing. Returns number removed."""
+ log = logging.getLogger("addlog")
+ to_delete: list[int] = []
+ for row in conn.execute(
+ "SELECT log_id, dist, arch, ok, name, id FROM logs"
+ ):
+ if _find_log_file(cfg.root, row) is None:
+ to_delete.append(row["log_id"])
+ if to_delete:
+ conn.execute("BEGIN")
+ conn.executemany("DELETE FROM logs WHERE log_id=?",
+ [(i,) for i in to_delete])
+ conn.execute("COMMIT")
+ log.info("cleanup: removed %d rows", len(to_delete))
+ return len(to_delete)
+
+
+PARSED_COLUMNS: tuple[str, ...] = ("runtime", "build_rpm_section")
+
+
+def _find_log_file(root: str, row) -> tuple[str, str] | None:
+ """Probe the filesystem for the log file matching this DB row.
+
+ Returns (path, ext) for the first existing file, or None if none found.
+ """
+ status = "OK" if row["ok"] else "FAIL"
+ base = row["name"] + (("," + row["id"]) if row["id"] else "")
+ dir_ = os.path.join(root, row["dist"], row["arch"], status)
+ for ext in KNOWN_EXTS:
+ candidate = os.path.join(dir_, base + "." + ext)
+ if os.path.isfile(candidate):
+ return (candidate, ext)
+ raw = os.path.join(dir_, base)
+ if os.path.isfile(raw):
+ return (raw, "")
+ return None
+
+
+def _worker_parse_for_backfill(args):
+ log_id, path, ext, ok, null_cols = args
+ parsed = parse_log(path, ext)
+ # build_rpm_section is meaningful only for FAIL rows.
+ if ok:
+ parsed["build_rpm_section"] = None
+ return (log_id, parsed, null_cols)
+
+
+PROGRESS_EVERY = 1000
+
+
+def _log_progress(log, phase, i, total, t0, last):
+ now = time.monotonic()
+ rate = (i - last["i"]) / max(now - last["t"], 1e-6)
+ eta = (total - i) / max(rate, 1e-6)
+ elapsed = now - t0
+ log.info("backfill %s: %d/%d %.0f/s elapsed=%.1fs eta=%.1fmin",
+ phase, i, total, rate, elapsed, eta / 60)
+ last["t"] = now
+ last["i"] = i
+
+
+def run_backfill(conn: sqlite3.Connection, cfg) -> int:
+ log = logging.getLogger("addlog")
+ where = " OR ".join(f"{c} IS NULL" for c in PARSED_COLUMNS)
+ null_col_list = ", ".join(PARSED_COLUMNS)
+ rows = conn.execute(
+ f"SELECT log_id, dist, arch, ok, name, id, {null_col_list} FROM logs WHERE {where}"
+ ).fetchall()
+ if not rows:
+ log.info("backfill: nothing to do")
+ return 0
+
+ log.info("backfill: %d rows to examine; resolving filesystem paths...", len(rows))
+ tasks = []
+ missing = 0
+ for row in rows:
+ hit = _find_log_file(cfg.root, row)
+ if hit is None:
+ missing += 1
+ continue
+ path, ext = hit
+ null_cols = frozenset(c for c in PARSED_COLUMNS if row[c] is None)
+ tasks.append((row["log_id"], path, ext, row["ok"], null_cols))
+ log.info("backfill: %d tasks (%d rows have no on-disk file)", len(tasks), missing)
+
+ workers = getattr(cfg, "workers", 1) or 1
+ t0 = time.monotonic()
+ last = {"t": t0, "i": 0}
+ results: list[tuple[int, dict, frozenset]] = []
+ interrupted = False
+ if workers > 1 and len(tasks) > 1:
+ log.info("backfill: parsing with %d workers", workers)
+ pool = Pool(processes=workers)
+ try:
+ for i, res in enumerate(
+ pool.imap_unordered(_worker_parse_for_backfill, tasks, chunksize=32), 1
+ ):
+ if should_stop():
+ interrupted = True
+ break
+ results.append(res)
+ if i % PROGRESS_EVERY == 0:
+ _log_progress(log, "parse", i, len(tasks), t0, last)
+ finally:
+ # terminate() kills workers immediately; avoids the slow close()+join()
+ # wait for the remaining queued work on Ctrl+C.
+ pool.terminate()
+ pool.join()
+ else:
+ log.info("backfill: parsing serially")
+ for i, t in enumerate(tasks, 1):
+ if should_stop():
+ interrupted = True
+ break
+ results.append(_worker_parse_for_backfill(t))
+ if i % PROGRESS_EVERY == 0:
+ _log_progress(log, "parse", i, len(tasks), t0, last)
+
+ if interrupted:
+ log.warning("backfill: interrupted after %d/%d; committing partial results",
+ len(results), len(tasks))
+ else:
+ log.info("backfill: parse complete, writing updates to DB...")
+
+ conn.execute("BEGIN")
+ filled = 0
+ for log_id, parsed, null_cols in results:
+ set_parts = []
+ vals = []
+ for col in PARSED_COLUMNS:
+ # Only fill columns that were NULL in the DB row.
+ if col in null_cols and parsed.get(col) is not None:
+ set_parts.append(f"{col}=?")
+ vals.append(parsed[col])
+ if not set_parts:
+ continue
+ vals.append(log_id)
+ conn.execute(f"UPDATE logs SET {', '.join(set_parts)} WHERE log_id=?", vals)
+ filled += 1
+ conn.execute("COMMIT")
+ dt = time.monotonic() - t0
+ log.info("backfill: filled %d rows (examined %d, parsed %d) in %.1fs",
+ filled, len(rows), len(tasks), dt)
+ return filled
+
+
+def build_parser() -> argparse.ArgumentParser:
+ p = argparse.ArgumentParser(prog="addlog.py", description=__doc__)
+ p.add_argument("--config", default=None, metavar="PATH",
+ help="config file (default: ./buildlogs.ini next to script)")
+ verbosity = p.add_mutually_exclusive_group()
+ verbosity.add_argument("--quiet", action="store_true",
+ help="suppress stderr output; file log only")
+ verbosity.add_argument("--debug", action="store_true",
+ help="DEBUG level to both file and stderr")
+ mode = p.add_mutually_exclusive_group()
+ mode.add_argument("--cleanup", action="store_true",
+ help="one-shot: drop DB rows whose files are missing")
+ mode.add_argument("--backfill", action="store_true",
+ help="one-shot: fill NULL parsed columns")
+ return p
+
+
+def main(argv: list[str] | None = None) -> int:
+ # Public data — files 644, dirs 755. No umask-sensitive secrets here.
+ os.umask(0o022)
+ args = build_parser().parse_args(argv)
+ cfg = load_config(args.config)
+
+ lock_fd = acquire_lock(cfg.lock)
+ if lock_fd is None:
+ # Another instance holds the lock. Silent exit as specified.
+ return 0
+
+ try:
+ setup_logging(cfg.log, quiet=args.quiet, debug=args.debug)
+ install_signal_handlers()
+ log = logging.getLogger("addlog")
+ conn = connect_db(cfg.database)
+ ensure_schema(conn)
+ if args.cleanup:
+ run_cleanup(conn, cfg)
+ return 0
+ if args.backfill:
+ run_backfill(conn, cfg)
+ return 0
+ full_scan(conn, cfg.root, workers=cfg.workers)
+ run_watcher(conn, cfg)
+ return 0
+ except Exception:
+ logging.getLogger("addlog").exception("unhandled error")
+ return 1
+ finally:
+ release_lock(lock_fd)
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/buildlogs.example.ini b/buildlogs.example.ini
new file mode 100644
index 0000000..8bf5815
--- /dev/null
+++ b/buildlogs.example.ini
@@ -0,0 +1,18 @@
+# Copy to buildlogs.ini and adjust paths for your host.
+[buildlogs]
+; required
+database = /home/services/httpd/html/db/buildlogs.db
+root = /home/services/ftp/pub/pld-buildlogs
+
+; optional -- these are the defaults
+; lock = ./addlog.lock
+; log = ./addlog.log
+; workers = 0 -- 0 means auto: cpu_count minus 1
+
+; optional -- tokens to hide from the web UI dist/arch discovery
+; surface. Space-separated. Tokens:
+; "ac" -- hide all arches under dist "ac"
+; "th/SRPMS" -- hide just that dist/arch pair
+; Direct URLs still work -- this affects only the
+; dropdown and Builders checkbox list.
+; ignore = ac th/SRPMS
diff --git a/buildlogs.inc b/buildlogs.inc
deleted file mode 100644
index 3ce4ebb..0000000
--- a/buildlogs.inc
+++ /dev/null
@@ -1,9 +0,0 @@
-<?php
-$database_file = "/home/services/httpd/html/pld-buildlogs/db/buildlogs2.db";
-$database = "sqlite:$database_file";
-$root_directory = "/home/services/ftp/pub/pld-buildlogs";
-$addr = array(
- "th" => array("SRPMS", "x32", "i686", "x86_64"),
- "ac" => array("SRPMS", "i386", "i586", "i686", "alpha", "amd64", "athlon",
- "ppc", "sparc", "sparc64")
-);
diff --git a/helpers/README-pl b/helpers/README-pl
deleted file mode 100644
index 262d17b..0000000
--- a/helpers/README-pl
+++ /dev/null
@@ -1,16 +0,0 @@
-buildlogs-mover.sh
- skrypt uruchamiany z crona, co minut�, na ho�cie z buildlogami.
- trzeba w nim ustawi� zmienn� root="...". Szuka plik�w
- $root/*/*/.new/*.info.
-
-install-buildlogs-tree.sh
- instaluje drzewo katalog�w na ho�cie z buildlogami, pld-specific.
-
-
-Wymagane katalogi:
-
- $root/$dist/$arch/{.new,OK,FAIL,prevOK}
-
-gdzie $arch to i386, ppc etc, oraz *koniecznie* SRPMS.
-
-$dist -- cokolwiek.
diff --git a/helpers/buildlogs-inotify-mover.sh b/helpers/buildlogs-inotify-mover.sh
deleted file mode 100644
index e7db89f..0000000
--- a/helpers/buildlogs-inotify-mover.sh
+++ /dev/null
@@ -1,14 +0,0 @@
-#!/bin/sh
-set -eu
-
-root="/home/services/ftpd/buildlogs"
-
-if [ -f /etc/buildlogs-mover.conf ]; then
- . /etc/buildlogs-mover.conf
-fi
-
-inotifywait -q -m -r -e move "$root"/*/*/.new | \
-while IFS= read -r path change file; do
- [ "$change" != "MOVED_TO" ] && continue
- /bin/su - ftp -s /bin/sh -c "/home/services/buildlogs/buildlogs-mover.sh"
-done
diff --git a/helpers/buildlogs-mover.conf b/helpers/buildlogs-mover.conf
deleted file mode 100644
index 39dd2ee..0000000
--- a/helpers/buildlogs-mover.conf
+++ /dev/null
@@ -1,3 +0,0 @@
-# This is location of buildlogs tree:
-
-root="/home/services/ftp/pub/pld-buildlogs"
diff --git a/helpers/buildlogs-mover.sh b/helpers/buildlogs-mover.sh
deleted file mode 100644
index 32595ea..0000000
--- a/helpers/buildlogs-mover.sh
+++ /dev/null
@@ -1,66 +0,0 @@
-#!/bin/sh
-#
-# Script to be run from crontab on buildlogs host.
-# Moves buildlogs around.
-#
-# Expected directory structure:
-#
-# root/$(dist)/$(arch)/
-# .new/
-# OK/
-# FAIL/
-#
-# Note that we look for root/*/*/.new/*.info, so don't place any additional
-# directories there.
-
-set -eu
-
-root="/home/services/ftpd/buildlogs"
-ADDLOG="/home/services/httpd/html/pld-buildlogs/scripts/addlog.php"
-
-if [ -f /etc/buildlogs-mover.conf ]; then
- . /etc/buildlogs-mover.conf
-fi
-
-# Ensure at least one .new dir exists. If the glob doesn't match anything,
-# find will return no results and the script exits gracefully.
-set +e
-first=$(find "$root" -mindepth 3 -maxdepth 3 -type d -name '.new' | head -n 1)
-set -e
-if [ -z "$first" ]; then
- echo "no .new directories found under $root" >&2
- exit 1
-fi
-
-handle_info() {
- info="$1"
- info_val=$(cat "$info" 2>/dev/null) || return 0
- echo "$info_val" | grep -q '^END$' || return 0
- status=$(echo "$info_val" | grep '^Status:' | sed -e 's/.*: *//')
- case "$status" in
- OK) s=OK ;;
- FAIL*) s=FAIL ;;
- *)
- {
- echo "bad buildlog status: $status in $info:"
- echo "#v+"
- echo "$info_val"
- echo "#v-"
- } >&2
- rm -f "$info"
- return 0
- ;;
- esac
- archdir=$(dirname "$(dirname "$info")")
- file=$(basename "$info" .info)
- if [ -f "$archdir/.new/$file" ]; then
- mv -f "$archdir/.new/$file" "$archdir/$s/$file"
- "$ADDLOG" "$archdir/$s/$file"
- rm -f "$info"
- fi
-}
-
-find "$root" -mindepth 4 -maxdepth 4 -type f -name '*.info' -path '*/.new/*' | \
-while IFS= read -r info; do
- [ -f "$info" ] && handle_info "$info"
-done
diff --git a/index.php b/index.php
index c96c0c0..8523a21 100644
--- a/index.php
+++ b/index.php
@@ -119,8 +119,49 @@ $buildlogs_server = "buildlogs.pld-linux.org";
$url = "index.php";
$fail_or_ok = ["FAIL", "OK"];
-// $database, $root_directory and others are taken from buildlogs.inc
-include('buildlogs.inc');
+$cfg = parse_ini_file(__DIR__ . '/buildlogs.ini', true);
+if ($cfg === false || !isset($cfg['buildlogs']['database'], $cfg['buildlogs']['root'])) {
+ http_response_code(500);
+ die("configuration error: buildlogs.ini missing or malformed");
+}
+$database_file = $cfg['buildlogs']['database'];
+$database = 'sqlite:' . $database_file;
+$root_directory = $cfg['buildlogs']['root'];
+// Tokens to hide from the dist/arch discovery UI. Two forms:
+// "ac" — hide every arch under dist "ac".
+// "th/SRPMS" — hide just that dist/arch pair.
+// Direct URLs (?dist=ac&arch=amd64) still work — this only affects
+// the dropdown/checkbox surfaces built by dists_and_archs().
+$ignore_display = preg_split('/\s+/', trim($cfg['buildlogs']['ignore'] ?? ''), -1, PREG_SPLIT_NO_EMPTY);
+
+function dists_and_archs(string $dsn): array {
+ global $ignore_display;
+ try {
+ $dbh = open_db($dsn);
+ } catch (PDOException $e) {
+ return [];
+ }
+ $ignore_dists = [];
+ $ignore_pairs = [];
+ foreach ($ignore_display as $tok) {
+ if (strpos($tok, '/') === false) {
+ $ignore_dists[$tok] = true;
+ } else {
+ $ignore_pairs[$tok] = true;
+ }
+ }
+ $out = [];
+ $q = $dbh->query("SELECT dist, arch FROM logs GROUP BY dist, arch ORDER BY dist, arch");
+ foreach ($q as $row) {
+ $dist = $row['dist'];
+ $arch = $row['arch'];
+ if (isset($ignore_dists[$dist])) continue;
+ if (isset($ignore_pairs["$dist/$arch"])) continue;
+ $out[$dist][] = $arch;
+ }
+ return $out;
+}
+
include('lib.php');
session_set_cookie_params([
@@ -406,6 +447,7 @@ function list_logs()
echo "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(4, _("Arch"))."</th>";
}
echo "<th bgcolor=\"#CCCCFF\" align=\"left\" valign=\"middle\" width=\"80%\">".$sl(1, _("Log File"))."</th>".
+ "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Failed rpm section")."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(6, _("Runtime"))."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\" width=\"15%\">".$sl(2, _("Size"))."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(0, _("Date"))."</th>".
@@ -422,7 +464,7 @@ function list_logs()
if ($ok !== '') $conds[] = "ok = :ok";
if ($q !== '') $conds[] = "name LIKE :q ESCAPE '\\'";
$where = empty($conds) ? "1=1" : implode(' AND ', $conds);
- $query = "SELECT log_id, dist, arch, ok, name, mtime, size, id, runtime FROM logs WHERE "
+ $query = "SELECT log_id, dist, arch, ok, name, mtime, size, id, runtime, build_rpm_section FROM logs WHERE "
. "$where ORDER BY $order LIMIT :limitnr OFFSET :offset ";
try {
@@ -502,10 +544,13 @@ function list_logs()
echo "<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">".h($row["arch"])."</td>";
}
$rt = format_runtime($row["runtime"] ?? null);
+ $section = ($row_ok == 0 && !empty($row["build_rpm_section"]))
+ ? "<code>".h($row["build_rpm_section"])."</code>" : "";
echo "<td bgcolor=\"#CCCCCC\" valign=\"middle\"><a href=\"$u\">" . h($f) . "</a> ".
"[<a href=\"$u&action=text\">"._("text")."</a> | ".
"<a href=\"$u&action=tail\">"._("tail")."</a>]".
- "</td><td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">$rt</td>".
+ "</td><td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">$section</td>".
+ "<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">$rt</td>".
"<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\">".
h((string)$s)."</td><td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">".
h($date_str)."</td><td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">$t</td></tr>\n";
@@ -569,14 +614,26 @@ function list_logs()
function file_name()
{
- global $ok, $dist, $arch, $name, $name_url, $id;
+ global $ok, $dist, $arch, $name, $name_url, $id, $root_directory;
if (isset($name) && isset($ok) && isset($arch) && isset($dist)) {
+ $base = $name . ($id != '' ? ",$id" : '');
+ $w = $ok ? "OK" : "FAIL";
+ $rel_dir = "$dist/$arch/$w";
+
+ // Probe the filesystem for a matching known compression suffix.
+ $ext_suffix = '';
+ foreach (['.bz2', '.gz', '.xz', '.zst', '.lzo', ''] as $suffix) {
+ if (is_file("$root_directory/$rel_dir/$base$suffix")) {
+ $ext_suffix = $suffix;
+ break;
+ }
+ }
+
if (isset($id) && $id != '') {
$name = $name . ",$id";
}
- $w = $ok ? "OK" : "FAIL";
- return "$dist/$arch/$w/$name.bz2";
+ return "$rel_dir/$name{$ext_suffix}";
}
}
@@ -602,7 +659,7 @@ function list_package_history($pkg_name, $cur_dist, $cur_arch, $cur_ok, $cur_id)
if ($has_ids) {
$stmt = $dbh->prepare(
- "SELECT dist, arch, ok, name, mtime, size, id, runtime FROM logs "
+ "SELECT dist, arch, ok, name, mtime, size, id, runtime, build_rpm_section FROM logs "
. "WHERE name = :name AND id IN ("
. " SELECT id FROM logs WHERE name = :name2 AND id != '' "
. " GROUP BY id ORDER BY MAX(mtime) DESC LIMIT 10"
@@ -613,7 +670,7 @@ function list_package_history($pkg_name, $cur_dist, $cur_arch, $cur_ok, $cur_id)
$stmt->bindValue(':name2', $pkg_name, PDO::PARAM_STR);
} else {
$stmt = $dbh->prepare(
- "SELECT dist, arch, ok, name, mtime, size, id, runtime FROM logs "
+ "SELECT dist, arch, ok, name, mtime, size, id, runtime, build_rpm_section FROM logs "
. "WHERE name = :name AND NOT "
. "(dist = :cd AND arch = :ca AND ok = :co AND COALESCE(id,'') = :ci) "
. "ORDER BY mtime DESC LIMIT 20"
@@ -649,7 +706,11 @@ function list_package_history($pkg_name, $cur_dist, $cur_arch, $cur_ok, $cur_id)
}
$now = time();
- echo "<h2 id=\"history\">"._("Previous builds of this package:")."</h2>\n";
+ $n_builds = count($rows);
+ echo "<h2 id=\"history\">".sprintf(
+ ngettext("Previous %d build of this package:",
+ "Previous %d builds of this package:", $n_builds),
+ $n_builds)."</h2>\n";
echo "<div align=\"center\"><table border=\"0\" cellspacing=\"1\" ".
"cellpadding=\"6\" bgcolor=\"#000000\" width=\"90%\">\n";
$first_col = $has_ids
@@ -661,6 +722,7 @@ function list_package_history($pkg_name, $cur_dist, $cur_arch, $cur_ok, $cur_id)
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Dist")."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Arch")."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"left\" valign=\"middle\">"._("Log File")."</th>".
+ "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Failed rpm section")."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Runtime")."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Size")."</th>".
"<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Date")."</th>".
@@ -708,6 +770,8 @@ function list_package_history($pkg_name, $cur_dist, $cur_arch, $cur_ok, $cur_id)
: "<font color=\"red\"><b>"._("FAIL")."</b></font>";
$rt = format_runtime($row["runtime"] ?? null);
+ $section = ($r_ok == 0 && !empty($row["build_rpm_section"]))
+ ? "<code>".h($row["build_rpm_section"])."</code>" : "";
echo "<tr>".
$first_cell.
"<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\">$status</td>".
@@ -716,6 +780,7 @@ function list_package_history($pkg_name, $cur_dist, $cur_arch, $cur_ok, $cur_id)
"<td bgcolor=\"#CCCCCC\" valign=\"middle\"><a href=\"$u\">" . h($r_name) . "</a> ".
"[<a href=\"$u&action=text\">"._("text")."</a> | ".
"<a href=\"$u&action=tail\">"._("tail")."</a>]</td>".
+ "<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">$section</td>".
"<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">$rt</td>".
"<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\">".h((string)$r_size)."</td>".
"<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">".h($date_str)."</td>".
@@ -788,11 +853,19 @@ function dump_log($tail)
"[<a href=\"#content\">"._("Content")."</a>]".
"</p>";
+ $full_path = safe_log_path($root_directory, $f);
+ $log_exists = ($full_path !== null && is_file($full_path));
+
echo "<table border=\"0\" cellpadding=\"3\" cellspacing=\"1\" bgcolor=\"#000000\" width=\"100%\">";
- one_item(_("Status"), ($ok == 1 ?
- "<font color=\"green\"><b>"._("OK")."</b></font>" :
- "<font color=\"red\"><b>"._("Failed")."</b></a>"));
+ if (!$log_exists) {
+ $status_html = "<font color=\"#b07000\"><b>"._("Log not found or not yet uploaded")."</b></font>";
+ } else {
+ $status_html = $ok == 1
+ ? "<font color=\"green\"><b>"._("OK")."</b></font>"
+ : "<font color=\"red\"><b>"._("Failed")."</b></font>";
+ }
+ one_item(_("Status"), $status_html);
$source_url = "https://" . $buildlogs_server . "/pld/" . $f;
one_item(_("Source URL"),
href_tag(h($source_url), h($source_url)));
@@ -817,8 +890,7 @@ function dump_log($tail)
} else {
one_item(_("rpm -qa of builder"), _("Not available"));
}
- $full_path = safe_log_path($root_directory, $f);
- if ($full_path !== null && is_file($full_path)) {
+ if ($log_exists) {
one_item("Date", date("Y/m/d H:i:s", filemtime($full_path)));
} else {
one_item("Date", _("unknown"));
@@ -835,9 +907,9 @@ function dump_log($tail)
# what can I say beside PHP suxx? how the fuck should I create
# bidirectional pipe? gotta use wget
- if ($full_path === null) {
- echo "<p>"._("Log not found")."</p>";
+ if (!$log_exists) {
echo "</table>";
+ list_package_history($name, $dist, $arch, $ok, $id);
return;
}
$h = open_log_stream($full_path);
@@ -1013,7 +1085,8 @@ function dump_text()
function list_archs()
{
- global $addr, $url, $cnt, $ok, $ns, $dist, $arch;
+ global $database, $url, $cnt, $ok, $ns, $dist, $arch;
+ $addr = dists_and_archs($database);
if (!isset($cnt))
$cnt = 50;
@@ -1284,7 +1357,8 @@ function dump_qa($plain)
function adv_search()
{
- global $database, $addr, $url, $off, $cnt, $ok, $ns, $dir;
+ global $database, $url, $off, $cnt, $ok, $ns, $dir;
+ $addr = dists_and_archs($database);
$req = array_merge($_POST, $_GET);
@@ -1423,7 +1497,7 @@ function adv_search()
$ns_int = ($ns === '') ? 0 : (int)$ns;
$order = SORT_ORDER_BY[$ns_int . effective_dir($ns, $dir)] ?? 'mtime DESC';
- $sql = 'SELECT log_id, dist, arch, ok, name, size, mtime, id, runtime FROM logs WHERE '
+ $sql = 'SELECT log_id, dist, arch, ok, name, size, mtime, id, runtime, build_rpm_section FROM logs WHERE '
. $where . ' ORDER BY ' . $order . ' LIMIT :limitnr OFFSET :offset';
try {
@@ -1469,6 +1543,7 @@ function adv_search()
echo "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(5, _("Dist"))."</th>";
echo "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(4, _("Arch"))."</th>";
echo "<th bgcolor=\"#CCCCFF\" align=\"left\" valign=\"middle\" width=\"60%\">".$sl(1, _("Log File"))."</th>";
+ echo "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">"._("Failed rpm section")."</th>";
echo "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(6, _("Runtime"))."</th>";
echo "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(2, _("Size"))."</th>";
echo "<th bgcolor=\"#CCCCFF\" align=\"center\" valign=\"middle\">".$sl(0, _("Date"))."</th>";
@@ -1518,6 +1593,9 @@ function adv_search()
echo "<td bgcolor=\"#CCCCCC\" valign=\"middle\"><a href=\"$u\">".h($r_name)."</a> ".
"[<a href=\"$u&action=text\">"._("text")."</a> | ".
"<a href=\"$u&action=tail\">"._("tail")."</a>]</td>";
+ $section = ($r_ok == 0 && !empty($row["build_rpm_section"]))
+ ? "<code>".h($row["build_rpm_section"])."</code>" : "";
+ echo "<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">$section</td>";
echo "<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">".format_runtime($row["runtime"] ?? null)."</td>";
echo "<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\">".h((string)$r_size)."</td>";
echo "<td bgcolor=\"#CCCCCC\" align=\"center\" valign=\"middle\" nowrap=\"nowrap\">".h($date_str)."</td>";
@@ -1584,7 +1662,7 @@ if ($action == "text") {
else if (isset($dist) || $q !== '')
list_logs();
else {
- $keys = array_keys($addr);
+ $keys = array_keys(dists_and_archs($database));
$dist = $keys[0] ?? 'th';
list_logs();
}
diff --git a/init.sql b/init.sql
deleted file mode 100644
index 83a79fe..0000000
--- a/init.sql
+++ /dev/null
@@ -1,86 +0,0 @@
--- MySQL dump 9.11
---
--- Host: localhost Database: buildlogs
--- ------------------------------------------------------
--- Server version 4.0.22
-
---
--- Table structure for table `architectures`
---
-CREATE DATABASE buildlogs;
-
-USE buildlogs;
-
-
-CREATE TABLE architectures (
- arch_id tinyint(4) NOT NULL default '0',
- name varchar(15) default NULL,
- PRIMARY KEY (arch_id)
-) TYPE=MyISAM;
-
---
--- Dumping data for table `architectures`
---
-
-INSERT INTO architectures VALUES (0,'th/SRPMS');
-INSERT INTO architectures VALUES (1,'th/i486');
-INSERT INTO architectures VALUES (2,'th/i686');
-INSERT INTO architectures VALUES (3,'th/athlon');
-INSERT INTO architectures VALUES (4,'th/x86_64');
-INSERT INTO architectures VALUES (5,'th/ia64');
-INSERT INTO architectures VALUES (6,'th/alpha');
-INSERT INTO architectures VALUES (7,'th/ppc');
-INSERT INTO architectures VALUES (8,'th/sparc');
-INSERT INTO architectures VALUES (9,'ac/SRPMS');
-INSERT INTO architectures VALUES (10,'ac/i386');
-INSERT INTO architectures VALUES (11,'ac/i586');
-INSERT INTO architectures VALUES (12,'ac/i686');
-INSERT INTO architectures VALUES (13,'ac/athlon');
-INSERT INTO architectures VALUES (14,'ac/amd64');
-INSERT INTO architectures VALUES (15,'ac/alpha');
-INSERT INTO architectures VALUES (16,'ac/ppc');
-INSERT INTO architectures VALUES (17,'ac/sparc');
-INSERT INTO architectures VALUES (18,'ac/sparc64');
-INSERT INTO architectures VALUES (19,'ti/SRPMS');
-INSERT INTO architectures VALUES (20,'ti/i586');
-INSERT INTO architectures VALUES (21,'ti/i686');
-INSERT INTO architectures VALUES (22,'ti/x86_64');
-
-
--- Table structure for table `logs`
---
-
-CREATE TABLE logs (
- log_id int(11) NOT NULL auto_increment,
- arch_id tinyint(4) default NULL,
- result tinyint(4) default NULL,
- size int(11) default NULL,
- spec_id smallint(6) default NULL,
- mtime int(11) default NULL,
- PRIMARY KEY (log_id)
-) TYPE=MyISAM;
-
-
-CREATE TABLE result (
- result_id tinyint(4) NOT NULL default '0',
- name varchar(5) default NULL,
- PRIMARY KEY (result_id)
-) TYPE=MyISAM;
-
---
--- Dumping data for table `result`
---
-
-INSERT INTO result VALUES (0,'FAIL');
-INSERT INTO result VALUES (1,'OK');
-
---
--- Table structure for table `specs`
---
-
-CREATE TABLE specs (
- spec_id smallint(6) NOT NULL auto_increment,
- spec varchar(70) default NULL,
- PRIMARY KEY (spec_id)
-) TYPE=MyISAM;
-
diff --git a/lib.php b/lib.php
index ec6d6cd..8a27b92 100644
--- a/lib.php
+++ b/lib.php
@@ -1,26 +1,36 @@
<?php
declare(strict_types=1);
+function open_db(string $dsn): PDO {
+ $dbh = new PDO($dsn);
+ $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
+ $dbh->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
+ return $dbh;
+}
+
function decompressor_for(string $path): string {
- if (preg_match('/\.bz2$/', $path)) {
- return is_executable('/usr/bin/lbzcat') ? '/usr/bin/lbzcat' : '/usr/bin/bzcat';
- }
- if (preg_match('/\.gz$/', $path)) {
- return '/usr/bin/zcat';
- }
- return '/bin/cat';
+ if (str_ends_with($path, '.bz2')) {
+ return is_executable('/usr/bin/lbzcat') ? '/usr/bin/lbzcat' : '/usr/bin/bzcat';
+ }
+ if (str_ends_with($path, '.gz')) return '/usr/bin/zcat';
+ if (str_ends_with($path, '.xz')) return '/usr/bin/xzcat';
+ if (str_ends_with($path, '.zst')) return '/usr/bin/zstdcat';
+ if (str_ends_with($path, '.lzo')) return '/usr/bin/lzop';
+ return '/bin/cat';
}
function open_log_stream(string $path) {
- $descriptors = [0 => ['pipe', 'r'], 1 => ['pipe', 'w'], 2 => ['pipe', 'w']];
- $proc = proc_open([decompressor_for($path), $path], $descriptors, $pipes);
- if (!is_resource($proc)) return null;
- fclose($pipes[0]);
- fclose($pipes[2]);
- return ['proc' => $proc, 'fd' => $pipes[1]];
+ $descriptors = [0 => ['pipe', 'r'], 1 => ['pipe', 'w'], 2 => ['pipe', 'w']];
+ $cmd = decompressor_for($path);
+ $argv = str_ends_with($cmd, '/lzop') ? [$cmd, '-dc', $path] : [$cmd, $path];
+ $proc = proc_open($argv, $descriptors, $pipes);
+ if (!is_resource($proc)) return null;
+ fclose($pipes[0]);
+ fclose($pipes[2]);
+ return ['proc' => $proc, 'fd' => $pipes[1]];
}
function close_log_stream(array $h): void {
- fclose($h['fd']);
- proc_close($h['proc']);
+ fclose($h['fd']);
+ proc_close($h['proc']);
}
diff --git a/pld-buildlogs/scripts/addlog.php b/pld-buildlogs/scripts/addlog.php
deleted file mode 100644
index e30ccac..0000000
--- a/pld-buildlogs/scripts/addlog.php
+++ /dev/null
@@ -1,126 +0,0 @@
-#!/usr/bin/php.cli
-<?php
-declare(strict_types=1);
-
-$result = ["FAIL" => 0, "OK" => 1];
-include('buildlogs.inc');
-include('lib.php');
-
-if (!isset($argv[1])) {
- fwrite(STDERR, "Usage: {$argv[0]} full_path_to_the_log\n");
- exit(1);
-}
-
-$path = $argv[1];
-
-if (!preg_match("|^" . preg_quote($root_directory, '|')
- . "/([^/]+)/([^/]+)/([^/]+)/(.+)\\.bz2$|", $path, $matches)) {
- exit(0);
-}
-
-$dist_raw = $matches[1];
-$arch_raw = $matches[2];
-$status = $matches[3];
-$basename = $matches[4];
-
-if (preg_match('/^(.*),(.*)$/', $basename, $m2)) {
- $name_raw = $m2[1];
- $id_raw = $m2[2];
-} else {
- $name_raw = $basename;
- $id_raw = '';
-}
-
-$identifier_re = '/^[A-Za-z0-9][A-Za-z0-9_.+-]*$/';
-if (!preg_match($identifier_re, $dist_raw)
- || !preg_match($identifier_re, $arch_raw)
- || !preg_match($identifier_re, $name_raw)
- || ($id_raw !== '' && !preg_match($identifier_re, $id_raw))
- || !isset($result[$status])) {
- fwrite(STDERR, "skip: invalid identifier in {$path}\n");
- exit(0);
-}
-
-$ok = $result[$status];
-$size = filesize($path);
-$mtime = filemtime($path);
-if ($size === false || $mtime === false) {
- fwrite(STDERR, "stat failed: {$path}\n");
- exit(1);
-}
-
-$now = time();
-if ($mtime > $now) {
- fwrite(STDERR, "clamp: mtime {$mtime} > now {$now} for {$path}\n");
- $mtime = $now;
-}
-
-function parse_runtime(string $path): ?int {
- $h = open_log_stream($path);
- if ($h === null) return null;
- $build_rt = null;
- $done_rt = null;
- while (($line = fgets($h['fd'], 102400)) !== false) {
- if (preg_match('/^Build-Time:.*\breal:(\d+(?:\.\d+)?)s/', $line, $m)) {
- $build_rt = (int)round((float)$m[1]);
- continue;
- }
- if (preg_match('/^ended at: .+, done in (\d+):(\d+):(\d+)(?:\.\d+)?/', $line, $m)) {
- $done_rt = (int)$m[1] * 3600 + (int)$m[2] * 60 + (int)$m[3];
- }
- }
- close_log_stream($h);
- return $build_rt ?? $done_rt;
-}
-
-$runtime = parse_runtime($path);
-
-try {
- $dbh = new PDO($database);
- $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
- $dbh->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
-} catch (PDOException $e) {
- fwrite(STDERR, "PDO connect: " . $e->getMessage() . "\n");
- exit(1);
-}
-
-$sel = $dbh->prepare(
- 'SELECT log_id FROM logs WHERE dist = :dist AND arch = :arch '
- . 'AND name = :name AND id = :id LIMIT 1'
-);
-$sel->execute([
- ':dist' => $dist_raw,
- ':arch' => $arch_raw,
- ':name' => $name_raw,
- ':id' => $id_raw,
-]);
-$row = $sel->fetch();
-
-if ($row !== false) {
- $upd = $dbh->prepare(
- 'UPDATE logs SET ok = :ok, size = :size, mtime = :mtime, runtime = :runtime '
- . 'WHERE log_id = :log_id'
- );
- $upd->execute([
- ':ok' => $ok,
- ':size' => $size,
- ':mtime' => $mtime,
- ':runtime' => $runtime,
- ':log_id' => $row['log_id'],
- ]);
-} else {
- $ins = $dbh->prepare(
- 'INSERT INTO logs(dist, arch, ok, name, size, mtime, id, runtime) '
- . 'VALUES(:dist, :arch, :ok, :name, :size, :mtime, :id, :runtime)'
- );
- $ins->execute([
- ':dist' => $dist_raw,
- ':arch' => $arch_raw,
- ':ok' => $ok,
- ':name' => $name_raw,
- ':size' => $size,
- ':mtime' => $mtime,
- ':id' => $id_raw,
- ':runtime' => $runtime,
- ]);
-}
diff --git a/pld-buildlogs/scripts/migration.php b/pld-buildlogs/scripts/migration.php
deleted file mode 100644
index 6ad0e4d..0000000
--- a/pld-buildlogs/scripts/migration.php
+++ /dev/null
@@ -1,108 +0,0 @@
-#!/usr/bin/php.cli
-<?php
-declare(strict_types=1);
-
-include('buildlogs.inc');
-
-if (file_exists($database_file)) {
- unlink($database_file);
-}
-
-try {
- $dbh = new PDO($database);
- $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
- $dbh->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE, PDO::FETCH_ASSOC);
-} catch (PDOException $e) {
- fwrite(STDERR, "PDO connect: " . $e->getMessage() . "\n");
- exit(1);
-}
-
-$dbh->query(
- "CREATE TABLE logs(log_id INTEGER PRIMARY KEY, "
- . "dist TEXT, arch TEXT, ok INTEGER, name TEXT, "
- . "size INTEGER, mtime INTEGER, id TEXT DEFAULT '', "
- . "runtime INTEGER)"
-);
-$dbh->query("CREATE INDEX IF NOT EXISTS dao_index ON logs(dist, arch, ok)");
-
-$ins = $dbh->prepare(
- 'INSERT INTO logs(dist, arch, ok, name, size, mtime, id) '
- . 'VALUES(:dist, :arch, :ok, :name, :size, :mtime, :id)'
-);
-
-$identifier_re = '/^[A-Za-z0-9][A-Za-z0-9_.+-]*$/';
-$result = ["FAIL", "OK"];
-
-$dbh->beginTransaction();
-$inserted = 0;
-$skipped = 0;
-
-$dh = opendir($root_directory);
-if ($dh === false) {
- fwrite(STDERR, "opendir $root_directory failed\n");
- exit(1);
-}
-while (($dist = readdir($dh)) !== false) {
- if ($dist[0] === '.') continue;
- if (!is_dir("$root_directory/$dist")) continue;
- if (!preg_match($identifier_re, $dist)) {
- $skipped++;
- continue;
- }
- $ah = opendir("$root_directory/$dist");
- if ($ah === false) {
- fwrite(STDERR, "opendir $dist failed\n");
- continue;
- }
- while (($arch = readdir($ah)) !== false) {
- if ($arch[0] === '.') continue;
- if (!is_dir("$root_directory/$dist/$arch")) continue;
- if (!preg_match($identifier_re, $arch)) {
- $skipped++;
- continue;
- }
- for ($ok = 0; $ok < 2; $ok++) {
- $directory = "$root_directory/$dist/$arch/" . $result[$ok];
- $sh = @opendir($directory);
- if ($sh === false) continue;
- while (($file = readdir($sh)) !== false) {
- if (!preg_match('/^(.*)\.bz2$/', $file, $match)) continue;
- if (preg_match('/^(.*),(.*)$/', $match[1], $m2)) {
- $name = $m2[1];
- $id = $m2[2];
- } else {
- $name = $match[1];
- $id = '';
- }
- if (!preg_match($identifier_re, $name)
- || ($id !== '' && !preg_match($identifier_re, $id))) {
- $skipped++;
- continue;
- }
- $f = "$directory/" . $match[0];
- $size = filesize($f);
- $mtime = filemtime($f);
- if ($size === false || $mtime === false) {
- $skipped++;
- continue;
- }
- $ins->execute([
- ':dist' => $dist,
- ':arch' => $arch,
- ':ok' => $ok,
- ':name' => $name,
- ':size' => $size,
- ':mtime' => $mtime,
- ':id' => $id,
- ]);
- $inserted++;
- }
- closedir($sh);
- }
- }
- closedir($ah);
-}
-closedir($dh);
-
-$dbh->commit();
-fwrite(STDERR, "done: inserted $inserted, skipped $skipped invalid entries\n");
================================================================
---- gitweb:
http://git.pld-linux.org/gitweb.cgi/projects/buildlogs.git/commitdiff/f38d9ada1e969d7d96397c9b8b6840fc541a8be9
More information about the pld-cvs-commit
mailing list