[projects/git-slug] Przepisanie slug.py: git pld jako passthrough do git na wielu repozytoriach
arekm
arekm at pld-linux.org
Tue May 12 15:08:00 CEST 2026
commit 988267148da83b874759c10fbc13bbd1720bb7a5
Author: Arkadiusz Miśkiewicz <arekm at maven.pl>
Date: Mon Apr 6 18:59:14 2026 +0200
Przepisanie slug.py: git pld jako passthrough do git na wielu repozytoriach
Gruntowna przebudowa narzędzia slug.py, aby polecenie "git pld <komenda>"
działało jak uruchomienie "git <komenda>" we wszystkich pasujących
repozytoriach pakietów PLD jednocześnie. Dotychczas narzędzie
reimplementowało operacje git w Pythonie (np. pull ręcznie wykonywał
rebase, checkout wymuszał -m), co uniemożliwiało przekazywanie
standardowych flag git.
Nowa architektura - dwa rodzaje poleceń:
- Passthrough (pull, fetch, checkout, status, log, diff, push, itp.):
uruchamia "git <komenda> <argumenty-użytkownika>" w każdym pasującym
repozytorium równolegle. Wszystkie argumenty po nazwie polecenia
trafiają bezpośrednio do git bez modyfikacji.
- Slug-specific (update, clone, list, init): zachowują własną logikę
wielorepozytoryjną (update korzysta z optymalizacji Refs repo,
clone wymaga konstrukcji URL, itp.).
Rozdzielanie argumentów (split_args):
Opcje slug (przed poleceniem) są oddzielone od opcji git (po poleceniu)
pozycyjnie, analogicznie do "git -C dir pull -q". Implementacja obsługuje
sklejone opcje krótkie (-j4, -qj4, -d/tmp) oraz formy --jobs=8.
Dzięki temu "git pld pull -q" przekazuje -q do git pull, a
"git pld -q pull" ustawia tryb cichy slug.
Inteligentne domyślne konfiguracje przez git -c:
Slug przekazuje domyślne ustawienia jako flagi "git -c key=value",
które wpływają tylko na wywołanie git-pld, nie zmieniając ustawień
użytkownika w gitconfig:
- pull: pull.rebase=true, pull.autostash=true
- fetch: fetch.prune=true
- status: status.short=true
Konfigurowalne przez PLD.<komenda>-config w gitconfig.
Ustawienie pustej wartości wyłącza domyślne flagi.
Routowanie wyjścia:
- stdout z git -> stdout (dane, możliwe do pipowania)
- stderr z git -> stderr (diagnostyka)
- komunikaty slug -> stderr
Tryb cichy (-q): tłumi stdout z udanych repozytoriów, zawsze pokazuje
stderr (ostrzeżenia) i pełne wyjście z repozytoriów z błędami.
Obsługa pomocy:
- "git pld --help" / "-h" -> pomoc slug (opcje globalne)
- "git pld pull --help" -> pomoc git pull (execvp, jedno wywołanie)
- "git pld update --help" -> pomoc polecenia slug update
Konfiguracja i wartości domyślne:
Zastąpienie parsowania ~/.gitconfig przez configparser wywołaniami
"git config --global --get PLD.<opcja>". Flaga --global zapewnia,
że lokalna konfiguracja repozytorium nie wpływa na zachowanie
operacji wsadowych. Priorytet: CLI > git config > wartości wbudowane.
Leniwa ewaluacja default_packagesdir() i cpu_count() - wywoływane
tylko gdy CLI i git config nie ustawiły wartości.
Poprawki istniejących błędów:
- --depth: dodano type=int (string "0" był truthy, powodując
niepotrzebne --depth=0 w fetch)
- default_options: usunięto pętlę nadpisującą wartości z CLI/gitconfig
po parse_args() (np. clone wymuszał prune=False ignorując gitconfig)
- Typy branch: znormalizowane do list ['*'] zamiast string '[*]'
(iteracja stringa dawała znaki '[', '*', ']')
- pull --all/--noall: naprawiono konflikt default= na wspólnym dest
Ustandaryzowanie wyjścia zgodnie z konwencjami git:
- Wszystkie komunikaty diagnostyczne na stderr (nie stdout)
- Prefiksy błędów: fatal:, error:, warning: (małymi literami)
- Kod wyjścia 1 dla wszystkich błędów (nie 2 dla argparse)
- SlugArgumentParser normalizuje kody wyjścia argparse do 1
- createpackage() raportuje błędy SSH zamiast cichego ignorowania
- init_command() kończy się kodem 1 gdy tworzenie pakietu się nie powiedzie
- Prune raportuje błędy usuwania zamiast cichego połykania (ignore_errors)
Opcja globalna --pattern propaguje do poleceń slug-specific:
"git pld --pattern 'perl-*' update" poprawnie filtruje repozytoria.
Wzorce pozycyjne po podkomendzie mają priorytet nad globalnym --pattern.
Optymalizacje:
- find_git_repos(): glob na pattern/.git zamiast podwójnego stat
- itertools.repeat() zamiast [options] * len(list)
Pliki zmienione:
- slug.py: gruntowna przebudowa
- git_slug/gitrepo.py: WARNING: -> warning: (konwencja git)
git_slug/gitrepo.py | 2 +-
slug.py | 823 ++++++++++++++++++++++++++++++++++++++++------------
2 files changed, 637 insertions(+), 188 deletions(-)
---
diff --git a/git_slug/gitrepo.py b/git_slug/gitrepo.py
index d9f88ee..2b46e5b 100644
--- a/git_slug/gitrepo.py
+++ b/git_slug/gitrepo.py
@@ -73,7 +73,7 @@ class GitRepo:
def init(self, remotepull, remotepush = None, remotename=REMOTE_NAME):
if os.path.isdir(self.gdir):
- print("WARNING: Directory {} already existed".format(self.gdir), file=sys.stderr)
+ print("warning: directory {} already existed".format(self.gdir), file=sys.stderr)
self.init_gitdir()
self.commandio(['remote', 'add', remotename, remotepull])
if remotepush is not None:
diff --git a/slug.py b/slug.py
index 4a089d7..8b1e7a2 100755
--- a/slug.py
+++ b/slug.py
@@ -1,30 +1,75 @@
#!/usr/bin/python3
+# git-pld: Run git commands across PLD Linux package repositories.
+#
+# Two kinds of commands:
+# - Slug-specific (update, clone, list, init): custom multi-repo orchestration
+# with no single-repo git equivalent.
+# - Git passthrough (everything else): runs 'git <command>' in each matching
+# repo with the user's arguments passed through unmodified.
+#
+# Slug options (before the command) are separated from git options (after the
+# command) positionally, just like 'git -C dir pull -q' separates git-core
+# options from git-pull options.
+
+__version__ = '0.15.1'
+
import copy
import glob
+import itertools
import sys
import os
import shutil
import subprocess
-import queue
+import shlex
import multiprocessing
import argparse
-
import signal
-import configparser
from multiprocessing import Pool as WorkerPool
-from git_slug.gitconst import GITLOGIN, GITSERVER, GIT_REPO, GIT_REPO_PUSH, REMOTE_NAME, REMOTEREFS
+from git_slug.gitconst import (GITLOGIN, GITSERVER, GIT_REPO, GIT_REPO_PUSH,
+ REMOTE_NAME, REMOTEREFS)
from git_slug.gitrepo import GitRepo, GitRepoError
from git_slug.refsdata import GitArchiveRefsData, NoMatchedRepos, RemoteRefsError
-class UnquoteConfig(configparser.ConfigParser):
- def get(self, section, option, **kwargs):
- value = super().get(section, option, **kwargs)
- return value.strip('"')
+
+# ---------------------------------------------------------------------------
+# Built-in per-command git config overrides.
+# These are passed as 'git -c key=value' so they only affect the git-pld
+# invocation, not the user's regular git settings.
+# User's CLI flags (e.g. --no-rebase) still override these.
+#
+# Developers can customize via gitconfig:
+# [PLD]
+# pull-config = pull.rebase=false # override built-in
+# fetch-config = # disable built-in (empty)
+# ---------------------------------------------------------------------------
+BUILTIN_CONFIG = {
+ 'pull': ['pull.rebase=true', 'pull.autostash=true'],
+ 'fetch': ['fetch.prune=true'],
+ 'status': ['status.short=true'],
+}
+
+
+# ---------------------------------------------------------------------------
+# Helpers
+# ---------------------------------------------------------------------------
+
+class SlugArgumentParser(argparse.ArgumentParser):
+ """ArgumentParser that exits with code 1 on all errors (matching git convention)."""
+ def exit(self, status=0, message=None):
+ if status != 0:
+ status = 1
+ super().exit(status, message)
+
class DelAppend(argparse._AppendAction):
+ """Argparse action: on first use, clears the default list, then appends.
+
+ This lets '-b master -b devel' give ['master', 'devel'] instead of
+ appending to the default, which would give ['default', 'master', 'devel'].
+ """
def __call__(self, parser, namespace, values, option_string=None):
item = copy.copy(getattr(namespace, self.dest, None)) if getattr(namespace, self.dest, None) is not None else []
try:
@@ -35,17 +80,237 @@ class DelAppend(argparse._AppendAction):
item.append(values)
setattr(namespace, self.dest, item)
+
def cpu_count():
+ """Number of CPUs, with a safe fallback."""
try:
return multiprocessing.cpu_count()
except NotImplementedError:
+ return 4
+
+
+def default_packagesdir():
+ """Default directory for local package repositories.
+
+ Tries RPM macro %_topdir first (usually ~/rpm), falls back to ~/rpm/packages.
+ """
+ try:
+ import rpm
+ return rpm.expandMacro('%_topdir')
+ except Exception:
+ return os.path.expanduser('~/rpm/packages')
+
+
+def git_config_get(key):
+ """Read a single value from global/system git config.
+
+ Uses --global so that PLD settings come from ~/.gitconfig (or
+ XDG_CONFIG_HOME/git/config), not from the cwd repo's .git/config.
+ This prevents a single repo's local config from changing batch
+ behavior across all repos.
+
+ Returns the value as a string, or None if the key is not set.
+ """
+ try:
+ result = subprocess.run(
+ ['git', 'config', '--global', '--get', key],
+ capture_output=True, text=True
+ )
+ if result.returncode == 0:
+ return result.stdout.strip()
+ except FileNotFoundError:
pass
- return 4
+ return None
+
+
+def print_prefixed(stream_data, prefix, dest):
+ """Print each line of stream_data prefixed with 'prefix: ' to dest."""
+ for line in stream_data.splitlines():
+ print("{}: {}".format(prefix, line), file=dest)
+
+
+# ---------------------------------------------------------------------------
+# Argument splitting — separates slug options from the git command + args
+# ---------------------------------------------------------------------------
+
+def split_args(argv):
+ """Split argv into (slug_args, command, git_args).
+
+ We separate slug's own options from the git command and its arguments
+ by scanning left to right. Everything before the first unrecognized
+ token is a slug option; the first unrecognized token is the git
+ command name; everything after it is passed to git untouched.
+
+ This mirrors how 'git' itself works:
+ git -C dir pull -q
+ ^^^^^^^^^^^ slug-level (git core options)
+ ^^^^ the command
+ ^^ git-pull options
+
+ We support all common short-option forms so that -j4, -qj4, -d/tmp
+ work as expected (not mistaken for a command name).
+ """
+ # Long-form slug flags that take no value
+ SLUG_LONG_FLAGS = {'--quiet', '--version', '--help'}
+ # Long-form slug options that require a value (next token or =val)
+ SLUG_LONG_WITH_VALUE = {'--packagesdir', '--jobs', '--pattern'}
+ # Single-char slug flags (no value): -q, -h
+ SHORT_FLAGS = {'q', 'h'}
+ # Single-char slug options that consume the rest of the cluster
+ # or the next token as their value: -j4, -d /tmp
+ SHORT_WITH_VALUE = {'d', 'j'}
+
+ slug_args = []
+ i = 0
+ while i < len(argv):
+ arg = argv[i]
+
+ # --- Long options ---
+ # Exact match: --quiet, --help, --version
+ # With value: --jobs 8 or --jobs=8 or --pattern 'foo'
+ if arg.startswith('--'):
+ if arg in SLUG_LONG_FLAGS:
+ slug_args.append(arg)
+ elif arg in SLUG_LONG_WITH_VALUE:
+ # Value is the next token: --jobs 8
+ slug_args.append(arg)
+ i += 1
+ if i < len(argv):
+ slug_args.append(argv[i])
+ elif '=' in arg and arg.split('=')[0] in SLUG_LONG_WITH_VALUE:
+ # Value is attached: --jobs=8
+ slug_args.append(arg)
+ else:
+ # Unknown long option — not ours, so we've reached
+ # the command area
+ break
+
+ # --- Short options ---
+ # Parse character by character to handle clusters like -qj4:
+ # -q -> slug quiet flag
+ # -j4 -> slug jobs=4 (value attached)
+ # -qj4 -> slug quiet + jobs=4
+ # -d/tmp -> slug packagesdir=/tmp (value attached)
+ elif arg.startswith('-') and len(arg) > 1:
+ chars = arg[1:] # strip the leading '-'
+ j = 0
+ recognized = True
+ while j < len(chars):
+ c = chars[j]
+ if c in SHORT_FLAGS:
+ # Simple flag, consume one character
+ slug_args.append('-{}'.format(c))
+ j += 1
+ elif c in SHORT_WITH_VALUE:
+ # Option that takes a value — everything after
+ # this character is the value (e.g. -j4 -> value "4"),
+ # or if nothing follows, the next argv token is the value
+ slug_args.append('-{}'.format(c))
+ rest = chars[j+1:]
+ if rest:
+ slug_args.append(rest) # -j4 -> -j, 4
+ else:
+ i += 1 # -j 4 -> -j, 4
+ if i < len(argv):
+ slug_args.append(argv[i])
+ j = len(chars) # consumed entire cluster
+ else:
+ # Unknown character — this isn't a slug option,
+ # so the whole token belongs to the command area
+ recognized = False
+ break
+ if not recognized:
+ break
+
+ # --- Not an option (no leading '-') ---
+ # First positional token is the git command name
+ else:
+ break
+
+ i += 1
+
+ # Everything from here on is the command + git arguments
+ command = argv[i] if i < len(argv) else None
+ git_args = argv[i+1:] if i+1 < len(argv) else []
+ return slug_args, command, git_args
+
+
+# ---------------------------------------------------------------------------
+# Config and defaults
+# ---------------------------------------------------------------------------
+
+def apply_defaults(options):
+ """Fill in missing option values from git config, then hardcoded defaults.
+
+ Priority (highest to lowest):
+ 1. CLI arguments — already set on 'options' by argparse
+ 2. Git config — PLD.packagesdir, PLD.jobs, etc.
+ 3. Hardcoded — sensible fallbacks
+
+ Because we used default=argparse.SUPPRESS in the parser, an option that
+ wasn't passed on the CLI won't exist as an attribute. We check hasattr()
+ to decide whether to fill it from a lower-priority source.
+ """
+ # Options that can come from git config (key name -> type converter)
+ CONFIG_KEYS = {
+ 'packagesdir': ('PLD.packagesdir', str),
+ 'jobs': ('PLD.jobs', int),
+ }
+
+ # Layer 2: fill from git config if CLI didn't set the value
+ for attr, (config_key, conv) in CONFIG_KEYS.items():
+ if not hasattr(options, attr):
+ val = git_config_get(config_key)
+ if val is not None:
+ setattr(options, attr, conv(val))
+
+ # Layer 3: fill anything still missing from hardcoded defaults.
+ # Callables are evaluated lazily — default_packagesdir() and cpu_count()
+ # only run if the value wasn't set by CLI or git config.
+ HARDCODED = {
+ 'packagesdir': default_packagesdir,
+ 'jobs': cpu_count,
+ 'quiet': lambda: False,
+ 'pattern': lambda: ['*'],
+ }
+ for attr, factory in HARDCODED.items():
+ if not hasattr(options, attr):
+ setattr(options, attr, factory())
+
+ # Normalize: expand ~ in paths
+ options.packagesdir = os.path.expanduser(options.packagesdir)
+
+
+def get_command_config(command):
+ """Get -c key=value pairs for a git command.
+
+ Priority: PLD.<command>-config from gitconfig (if set) > built-in defaults.
+ Setting PLD.<command>-config to empty disables all defaults for that command.
+ Values are parsed with shlex.split() to handle quoting.
+ """
+ configured = git_config_get('PLD.{}-config'.format(command))
+ if configured is not None:
+ # User explicitly set this key — use their value.
+ # Empty string means "no overrides" (disables built-in defaults).
+ return shlex.split(configured) if configured.strip() else []
+ # No user override — use built-in defaults (if any).
+ return list(BUILTIN_CONFIG.get(command, []))
+
+
+# ---------------------------------------------------------------------------
+# Worker pool
+# ---------------------------------------------------------------------------
def pool_worker_init():
+ """Ignore SIGINT in worker processes — let the parent handle Ctrl-C."""
signal.signal(signal.SIGINT, signal.SIG_IGN)
+
def run_worker(function, options, args):
+ """Run function(*arg) for each arg in args using a process pool.
+
+ Returns a list of non-None results (typically failed repo paths).
+ """
ret = []
pool = WorkerPool(options.jobs, pool_worker_init)
try:
@@ -60,77 +325,205 @@ def run_worker(function, options, args):
sys.exit(1)
return ret
-def readconfig(path):
- config = UnquoteConfig(delimiters='=', interpolation=None, strict=False)
- try:
- config.read(path)
- except UnicodeDecodeError:
- raise SystemExit("I have problems parsing {} file.\n\
-Check if it is consistent with your locale settings.".format(path))
- optionslist = {}
- for option in ('newpkgs', 'prune'):
- if config.has_option('PLD', option):
- optionslist[option] = config.getboolean('PLD', option)
- for option in ('depth', 'repopattern', 'packagesdir'):
- if config.has_option('PLD', option):
- optionslist[option] = config.get('PLD', option)
- if config.has_option('PLD','branch'):
- optionslist['branch'] = config.get('PLD', 'branch').split()
- for option in ('jobs'):
- if config.has_option('PLD', option):
- optionslist[option] = config.getint('PLD', option)
-
- for pathopt in ('packagesdir'):
- if pathopt in optionslist:
- optionslist[pathopt] = os.path.expanduser(optionslist[pathopt])
- return optionslist
-def initpackage(name, options):
- repo = GitRepo(os.path.join(options.packagesdir, name))
- remotepush = os.path.join(GIT_REPO_PUSH, name)
- repo.init(os.path.join(GIT_REPO, name), remotepush)
- return repo
+# ---------------------------------------------------------------------------
+# Git passthrough — the core of the new design
+# ---------------------------------------------------------------------------
-def createpackage(name, options):
- subprocess.Popen(['ssh', GITLOGIN + GITSERVER, 'create', name]).wait()
- initpackage(name, options)
+def find_git_repos(packagesdir, patterns):
+ """Find directories containing .git under packagesdir matching patterns.
-def create_packages(options):
- for package in options.packages:
- createpackage(package, options)
+ Globs for '<packagesdir>/<pattern>/.git' directly to avoid a separate
+ os.path.isdir() stat call per candidate.
+ """
+ repos = set()
+ for pat in patterns:
+ for gitdir in glob.iglob(os.path.join(packagesdir, pat, '.git')):
+ repos.add(os.path.dirname(gitdir))
+ return sorted(repos)
+
+
+def get_matching_repos(options):
+ """Find git repos in packagesdir matching the --pattern globs."""
+ return find_git_repos(options.packagesdir, options.pattern)
+
+
+def build_git_cmd(repo_dir, git_command, git_args, config_pairs):
+ """Assemble the full git command line.
+
+ Produces: git [-c key=val ...] -C repo_dir <command> [<args>...]
+ The -c flags inject config overrides, -C sets the working directory.
+ """
+ cmd = ['git']
+ for pair in config_pairs:
+ cmd.extend(['-c', pair])
+ cmd.extend(['-C', repo_dir, git_command])
+ cmd.extend(git_args)
+ return cmd
+
+
+def git_passthrough_worker(repo_dir, git_command, git_args, config_pairs, quiet):
+ """Run a git command in one repo, capture and prefix output.
+
+ Called by the worker pool for each repo in parallel.
+ Returns repo_dir on failure (so the caller can count failures), None on success.
+
+ Output policy:
+ - Failed repos: always print everything (stdout + stderr) regardless of -q.
+ - Successful repos, stderr: always print (may contain warnings).
+ - Successful repos, stdout: print unless -q (quiet suppresses data output).
+ """
+ directory = os.path.basename(repo_dir)
+ cmd = build_git_cmd(repo_dir, git_command, git_args, config_pairs)
+ result = subprocess.run(cmd, capture_output=True, text=True)
+
+ if result.returncode != 0:
+ # Failed — always show all output so the developer can diagnose
+ print_prefixed(result.stdout, directory, sys.stdout)
+ print_prefixed(result.stderr, directory, sys.stderr)
+ else:
+ # Succeeded — always show stderr (warnings, progress), because
+ # even a successful command might emit important diagnostics.
+ print_prefixed(result.stderr, directory, sys.stderr)
+ if not quiet:
+ # In quiet mode we suppress stdout from successful repos
+ # (e.g. "Already up to date" from hundreds of repos).
+ print_prefixed(result.stdout, directory, sys.stdout)
+
+ return repo_dir if result.returncode else None
+
+
+def passthrough_command(options, git_command, git_args):
+ """Run 'git <command>' across all matching repos in parallel.
+
+ 1. Resolve per-command -c config overrides (built-in or from gitconfig).
+ 2. Find all repos matching --pattern in --packagesdir.
+ 3. Run git in each repo via the worker pool.
+ 4. Report failures and exit non-zero if any repo failed.
+ """
+ config_pairs = get_command_config(git_command)
+
+ repos = get_matching_repos(options)
+ if not repos:
+ print("fatal: no matching repositories found", file=sys.stderr)
+ sys.exit(1)
+
+ if not options.quiet:
+ print("Running 'git {}' in {} repo(s)...".format(git_command, len(repos)),
+ file=sys.stderr)
+
+ # Build argument tuples for pool.starmap()
+ args = [(r, git_command, git_args, config_pairs, options.quiet)
+ for r in repos]
+ failed = run_worker(git_passthrough_worker, options, args)
+
+ if failed:
+ print("error: failed in {} repo(s)".format(len(failed)), file=sys.stderr)
+ sys.exit(1)
+
+
+# ---------------------------------------------------------------------------
+# Slug-specific commands — custom multi-repo orchestration
+# ---------------------------------------------------------------------------
def getrefs(*args):
+ """Fetch remote ref data from the centralized Refs repository."""
try:
refs = GitArchiveRefsData(*args)
except RemoteRefsError as e:
- print('Problem with file {} in repository {}'.format(*e.args), file=sys.stderr)
+ print('fatal: problem with file {} in repository {}'.format(*e.args),
+ file=sys.stderr)
sys.exit(1)
except NoMatchedRepos:
- print('No matching package has been found', file=sys.stderr)
- sys.exit(2)
+ print('fatal: no matching package has been found', file=sys.stderr)
+ sys.exit(1)
return refs
+
+def initpackage(name, options):
+ """Initialize a local repo for a package, pointing at the PLD server.
+
+ Returns the GitRepo on success, or None on failure. Catches GitRepoError
+ so that worker-pool callers don't abort the whole batch on a single
+ repo init failure.
+ """
+ try:
+ repo = GitRepo(os.path.join(options.packagesdir, name))
+ remotepush = os.path.join(GIT_REPO_PUSH, name)
+ repo.init(os.path.join(GIT_REPO, name), remotepush)
+ return repo
+ except GitRepoError as e:
+ print('error: failed to init {}: {}'.format(name, e),
+ file=sys.stderr)
+ return None
+
+
+def createpackage(name, options):
+ """Create a new package repository on the server and init locally.
+
+ Returns True on success, False on failure.
+ """
+ result = subprocess.run(['ssh', GITLOGIN + GITSERVER, 'create', name])
+ if result.returncode != 0:
+ print('error: failed to create {} on server'.format(name),
+ file=sys.stderr)
+ return False
+ if initpackage(name, options) is None:
+ return False
+ return True
+
+
+class _FetchError:
+ """Sentinel returned by fetch_package on failure.
+
+ Truthy so run_worker's filter(None) keeps it, but distinguishable
+ from a GitRepo so callers can count errors.
+ """
+ pass
+
+
def fetch_package(gitrepo, refs_heads, options):
+ """Fetch changed branches for a single package (worker function).
+
+ Compares local refs against the centralized Refs data to determine
+ which branches need updating, then fetches only those.
+
+ Returns gitrepo on success, _FetchError on failure, None if nothing to fetch.
+ """
ref2fetch = []
for ref in refs_heads:
if gitrepo.check_remote(ref) != refs_heads[ref]:
- ref2fetch.append('+{}:{}/{}'.format(ref, REMOTEREFS, ref[len('refs/heads/'):]))
+ ref2fetch.append('+{}:{}/{}'.format(ref, REMOTEREFS,
+ ref[len('refs/heads/'):]))
if ref2fetch:
ref2fetch.append('refs/notes/*:refs/notes/*')
else:
return
+ directory = os.path.basename(gitrepo.wtree)
try:
(stdout, stderr) = gitrepo.fetch(ref2fetch, options.depth)
if stderr != b'':
- print('------', gitrepo.gdir[:-len('.git')], '------\n' + stderr.decode('utf-8'))
+ print_prefixed(stderr.decode('utf-8'), directory, sys.stderr)
return gitrepo
except GitRepoError as e:
- print('------', gitrepo.gdir[:-len('.git')], '------\n', e)
-
+ print_prefixed(str(e), directory, sys.stderr)
+ return _FetchError()
+
+
def fetch_packages(options, return_all=False):
+ """Smart fetch: use centralized Refs repo to selectively update packages.
+
+ This is the core of the 'update' command — it checks which branches
+ changed upstream and only fetches those, which is much faster than
+ running 'git fetch' in every repo.
+
+ Sets options.had_errors = True if any init or prune operation failed,
+ so callers can exit non-zero.
+ """
refs = getrefs(options.branch, options.repopattern)
- print('Read remotes data')
+ print('Read remotes data', file=sys.stderr)
+ had_errors = False
pkgs_new = []
if options.newpkgs:
for pkgdir in sorted(refs.heads):
@@ -138,7 +531,13 @@ def fetch_packages(options, return_all=False):
if not os.path.isdir(gitdir):
pkgs_new.append(pkgdir)
- run_worker(initpackage, options, zip(pkgs_new, [options] * len(pkgs_new)))
+ # initpackage returns None on failure — count those as errors
+ results = run_worker(initpackage, options,
+ zip(pkgs_new, itertools.repeat(options)))
+ # run_worker filters out None, so results contains only successful repos.
+ # If fewer came back than we sent, some failed.
+ if len(results) < len(pkgs_new):
+ had_errors = True
args = []
for pkgdir in sorted(refs.heads):
@@ -148,161 +547,211 @@ def fetch_packages(options, return_all=False):
gitrepo = GitRepo(os.path.join(options.packagesdir, pkgdir))
args.append((gitrepo, refs.heads[pkgdir], options))
- updated_repos = run_worker(fetch_package, options, args)
+ fetch_results = run_worker(fetch_package, options, args)
+ # Separate successful fetches from errors
+ updated_repos = [r for r in fetch_results if not isinstance(r, _FetchError)]
+ if len(updated_repos) < len(fetch_results):
+ had_errors = True
if options.prune:
- refs = getrefs('*')
- for pattern in options.repopattern:
- for fulldir in glob.iglob(os.path.join(options.packagesdir, pattern)):
- pkgdir = os.path.basename(fulldir)
- if len(refs.heads[pkgdir]) == 0 and os.path.isdir(os.path.join(fulldir, '.git')):
- print('Removing', fulldir)
+ refs = getrefs(['*'])
+ for fulldir in find_git_repos(options.packagesdir, options.repopattern):
+ pkgdir = os.path.basename(fulldir)
+ if len(refs.heads[pkgdir]) == 0:
+ print('warning: removing', fulldir, file=sys.stderr)
+ try:
shutil.rmtree(fulldir)
+ except OSError as e:
+ print('error: failed to remove {}: {}'.format(
+ fulldir, e), file=sys.stderr)
+ had_errors = True
+
+ options.had_errors = had_errors
if return_all:
return refs.heads
else:
return updated_repos
-def checkout_package(repo, options):
+
+def clone_package(repo, options):
+ """Checkout master branch after initial clone (worker function).
+
+ Returns None on success, repo path string on failure (truthy,
+ so run_worker keeps it and callers can count errors).
+ """
try:
- repo.checkout(options.checkout)
+ repo.checkout('master')
except GitRepoError as e:
- print('Problem with checking branch {} in repo {}: {}'.format(options.checkout, repo.gdir, e), file=sys.stderr)
+ print('error: checking branch master in repo {}: {}'.format(
+ repo.gdir, e), file=sys.stderr)
+ return repo.gdir
+
+
+# ---------------------------------------------------------------------------
+# Slug-specific command handlers
+# ---------------------------------------------------------------------------
+
+def update_command(options, args):
+ """Slug-specific 'update': selective fetch via centralized Refs repo.
+
+ Uses the Refs repo optimization to avoid fetching unchanged repos.
+ This is faster than 'git pld fetch' which runs plain 'git fetch'
+ in every matching repo.
+ """
+ p = SlugArgumentParser(prog='git pld update',
+ description='Smart fetch using centralized Refs repo')
+ p.add_argument('--new', action='store_true', default=False,
+ help='init local repos for packages that exist upstream but not locally')
+ p.add_argument('--no-new', dest='new', action='store_false')
+ p.add_argument('-p', '--prune', action='store_true', default=False,
+ help='remove local repos that were deleted upstream')
+ p.add_argument('-b', '--branch', action=DelAppend, default=['master'],
+ help='branch pattern to fetch (repeatable)')
+ p.add_argument('--depth', type=int, default=0,
+ help='shallow fetch depth (0 = full history)')
+ p.add_argument('repopattern', nargs='*', default=None,
+ help='repo name globs (default: from global --pattern or *)')
+ uopts = p.parse_args(args)
+
+ # Map update-specific options to the names fetch_packages() expects.
+ # Positional patterns after the subcommand take priority, then global
+ # --pattern, then default '*'.
+ options.branch = uopts.branch
+ options.repopattern = uopts.repopattern or options.pattern
+ options.newpkgs = uopts.new
+ options.prune = uopts.prune
+ options.depth = uopts.depth
+ options.omitexisting = False
-def checkout_packages(options):
- if options.checkout is None:
- options.checkout = "/".join([REMOTE_NAME, options.branch[0]])
fetch_packages(options)
- refs = getrefs(options.branch, options.repopattern)
- repos = []
- for pkgdir in sorted(refs.heads):
- repos.append(GitRepo(os.path.join(options.packagesdir, pkgdir)))
+ if options.had_errors:
+ sys.exit(1)
- run_worker(checkout_package, options, zip(repos, [options] * len(repos)))
-def clone_package(repo, options):
- try:
- repo.checkout('master')
- except GitRepoError as e:
- print('Problem with checking branch master in repo {}: {}'.format(repo.gdir, e), file=sys.stderr)
+def clone_command(options, args):
+ """Slug-specific 'clone': batch clone matching packages from PLD server.
+
+ Fetches all branches and checks out master in each new repo.
+ """
+ p = SlugArgumentParser(prog='git pld clone',
+ description='Clone matching PLD package repositories')
+ p.add_argument('--depth', type=int, default=0,
+ help='shallow fetch depth (0 = full history)')
+ p.add_argument('repopattern', nargs='*', default=None,
+ help='repo name globs (default: from global --pattern or *)')
+ uopts = p.parse_args(args)
+
+ # Set up options for fetch_packages()
+ options.branch = ['*']
+ options.repopattern = uopts.repopattern or options.pattern
+ options.newpkgs = True
+ options.prune = False
+ options.depth = uopts.depth
+ options.omitexisting = True
-def clone_packages(options):
repos = fetch_packages(options)
- run_worker(clone_package, options, zip(repos, [options] * len(repos)))
+ checkout_failures = run_worker(clone_package, options,
+ zip(repos, itertools.repeat(options)))
+ if options.had_errors or checkout_failures:
+ sys.exit(1)
-def pull_package(gitrepo, options):
- directory = os.path.basename(gitrepo.wtree)
- try:
- (out, err) = gitrepo.commandexc(['rev-parse', '-q', '--verify', '@{u}'])
- sha1 = out.decode().strip()
- (out, err) = gitrepo.commandexc(['rebase', sha1])
- for line in out.decode().splitlines():
- print(directory,":",line)
- except GitRepoError as e:
- for line in e.args[0].splitlines():
- print("{}: {}".format(directory,line))
- pass
-def pull_packages(options):
- repolist = []
- if options.updateall:
- pkgs = fetch_packages(options, True)
- for directory in sorted(os.listdir(options.packagesdir)):
- if directory in pkgs:
- repolist.append(GitRepo(os.path.join(options.packagesdir, directory)))
- else:
- repolist = fetch_packages(options, False)
- print('--------Pulling------------')
- pool = WorkerPool(options.jobs, pool_worker_init)
- run_worker(pull_package, options, zip(repolist, [options] * len(repolist)))
+def list_command(options, args):
+ """List upstream repositories matching patterns."""
+ p = SlugArgumentParser(prog='git pld list',
+ description='List PLD package repositories')
+ p.add_argument('-b', '--branch', action=DelAppend, default=['*'],
+ help='only show packages with this branch (repeatable)')
+ p.add_argument('repopattern', nargs='*', default=None,
+ help='repo name globs (default: from global --pattern or *)')
+ uopts = p.parse_args(args)
-def list_packages(options):
- refs = getrefs(options.branch, options.repopattern)
+ refs = getrefs(uopts.branch, uopts.repopattern or options.pattern)
+ # Output to stdout — this is data, pipeable
for package in sorted(refs.heads):
print(package)
-def default_packagesdir():
- try:
- import rpm
- return rpm.expandMacro('%_topdir')
- except:
- return os.path.expanduser('~/rpm/packages')
+
+def init_command(options, args):
+ """Create new package repositories on the server and init locally."""
+ p = SlugArgumentParser(prog='git pld init',
+ description='Create new PLD package repositories')
+ p.add_argument('packages', nargs='+', help='package names to create')
+ uopts = p.parse_args(args)
+
+ failed = 0
+ for package in uopts.packages:
+ if not createpackage(package, options):
+ failed += 1
+ if failed:
+ sys.exit(1)
+
+
+# ---------------------------------------------------------------------------
+# Main entry point
+# ---------------------------------------------------------------------------
+
+# Commands with custom slug-specific logic (not passed through to git).
+# Everything else is treated as a git command and run in each repo.
+SLUG_COMMANDS = {
+ 'update': update_command,
+ 'clone': clone_command,
+ 'list': list_command,
+ 'init': init_command,
+}
+
+
+def main():
+ # --- Slug option parser ---
+ # Only parses slug-global options (before the command name).
+ # We use default=argparse.SUPPRESS so that unset options are simply
+ # absent from the namespace. This lets apply_defaults() distinguish
+ # "user passed -j4 on CLI" from "nobody set jobs" — and fill in
+ # git config or hardcoded values only for the latter.
+ slug_parser = SlugArgumentParser(
+ add_help=True,
+ description='Run git commands across PLD package repositories',
+ usage='%(prog)s [options] <command> [<args>...]')
+ slug_parser.add_argument('-d', '--packagesdir',
+ default=argparse.SUPPRESS,
+ help='local directory with git repositories')
+ slug_parser.add_argument('-j', '--jobs', type=int,
+ default=argparse.SUPPRESS,
+ help='number of parallel workers')
+ slug_parser.add_argument('-q', '--quiet', action='store_true',
+ default=argparse.SUPPRESS,
+ help='suppress stdout from successful repos')
+ slug_parser.add_argument('--pattern', action='append',
+ default=argparse.SUPPRESS,
+ help='repo name glob, repeatable (default: *)')
+ slug_parser.add_argument('--version', action='version',
+ version='%(prog)s ' + __version__)
+
+ # Step 1: Split argv at the command boundary.
+ # Slug options come before the command, git args come after.
+ slug_args, command, git_args = split_args(sys.argv[1:])
+
+ # Step 2: Parse slug options and apply config/defaults.
+ options = slug_parser.parse_args(slug_args)
+ apply_defaults(options)
+
+ # Step 3: No command given -> show help and exit (like 'git' does).
+ if command is None:
+ slug_parser.print_help(sys.stderr)
+ sys.exit(1)
+
+ # Step 4: Dispatch.
+ if command in SLUG_COMMANDS:
+ # Slug-specific command — it parses its own args from git_args.
+ SLUG_COMMANDS[command](options, git_args)
+ else:
+ # Git passthrough — run 'git <command>' in each matching repo.
+ # Special case: if the user asked for help (e.g. 'git pld pull --help'),
+ # show git's own help page once instead of running in every repo.
+ if '--help' in git_args or '-h' in git_args:
+ os.execvp('git', ['git', command] + git_args)
+ passthrough_command(options, command, git_args)
-common_options = argparse.ArgumentParser(add_help=False)
-common_options.add_argument('-d', '--packagesdir', help='local directory with git repositories',
- default=default_packagesdir())
-
-common_fetchoptions = argparse.ArgumentParser(add_help=False, parents=[common_options])
-common_fetchoptions.add_argument('-j', '--jobs', help='number of threads to use', default=cpu_count(), type=int)
-common_fetchoptions.add_argument('repopattern', nargs='*', default = ['*'])
-common_fetchoptions.add_argument('--depth', help='depth of fetch', default=0)
-
-default_options = {}
-parser = argparse.ArgumentParser(description='PLD tool for interaction with git repos',
- formatter_class=argparse.RawDescriptionHelpFormatter)
-parser.set_defaults(**readconfig(os.path.expanduser('~/.gitconfig')))
-
-subparsers = parser.add_subparsers(help='[-h] [options]', dest='command')
-update = subparsers.add_parser('update', help='fetch repositories', parents=[common_fetchoptions],
- formatter_class=argparse.RawDescriptionHelpFormatter)
-update.add_argument('-b', '--branch', help='branch to fetch', action=DelAppend, default=['master'])
-newpkgsopt = update.add_mutually_exclusive_group()
-newpkgsopt.add_argument('-n', '--newpkgs', help='download packages that do not exist on local side',
- action='store_true')
-newpkgsopt.add_argument('-nn', '--nonewpkgs', help='do not download new packages', dest='newpkgs', action='store_false')
-update.add_argument('-P', '--prune', help='prune git repositories that do no exist upstream',
- action='store_true')
-update.set_defaults(func=fetch_packages, omitexisting=False)
-default_options['update'] = {'omitexisting': False}
-
-init = subparsers.add_parser('init', help='init new repository', parents=[common_options],
- formatter_class=argparse.RawDescriptionHelpFormatter)
-init.add_argument('packages', nargs='+', help='list of packages to create')
-init.set_defaults(func=create_packages)
-default_options['init'] = {}
-
-clone = subparsers.add_parser('clone', help='clone repositories', parents=[common_fetchoptions],
- formatter_class=argparse.RawDescriptionHelpFormatter)
-clone.set_defaults(func=clone_packages, branch='[*]', prune=False, newpkgs=True, omitexisting=True)
-default_options['clone'] = {'branch': '[*]', 'prune': False, 'newpkgs': True, 'omitexisting': True}
-
-fetch = subparsers.add_parser('fetch', help='fetch repositories', parents=[common_fetchoptions],
- formatter_class=argparse.RawDescriptionHelpFormatter)
-fetch.set_defaults(func=fetch_packages, branch='[*]', prune=False, newpkgs=False, omitexisting=False)
-default_options['fetch'] = {'branch': '[*]', 'prune': False, 'newpkgs': False, 'omitexisting': False}
-
-pull = subparsers.add_parser('pull', help='git-pull in all existing repositories', parents=[common_fetchoptions],
- formatter_class=argparse.RawDescriptionHelpFormatter)
-pull.add_argument('--all', help='update local branches in all repositories', dest='updateall', action='store_true', default=False)
-pull.add_argument('--noall', help='update local branches only when something has been fetched', dest='updateall', action='store_false', default=True)
-newpkgsopt = pull.add_mutually_exclusive_group()
-newpkgsopt.add_argument('-n', '--newpkgs', help='download packages that do not exist on local side',
- action='store_true')
-newpkgsopt.add_argument('-nn', '--nonewpkgs', help='do not download new packages', dest='newpkgs', action='store_false')
-pull.set_defaults(func=pull_packages, branch='[*]', prune=False, newpkgs=False, omitexisting=False)
-default_options['pull'] = {'branch': ['*'], 'prune': False, 'omitexisting': False}
-
-checkout =subparsers.add_parser('checkout', help='checkout repositories', parents=[common_fetchoptions],
- formatter_class=argparse.RawDescriptionHelpFormatter)
-checkout.add_argument('-b', '--branch', help='branch to fetch', action=DelAppend, default=['master'])
-checkout.add_argument('-c', '--checkout', help='branch to fetch', default=None)
-checkout.add_argument('-P', '--prune', help='prune git repositories that do no exist upstream',
- action='store_true')
-checkout.set_defaults(func=checkout_packages, newpkgs=True, omitexisting=False)
-default_options['checkout'] = {'newpkgs': True, 'omitexisting': False}
-
-listpkgs = subparsers.add_parser('list', help='list repositories',
- formatter_class=argparse.RawDescriptionHelpFormatter)
-listpkgs.add_argument('-b', '--branch', help='show packages with given branch', action=DelAppend, default=['*'])
-listpkgs.add_argument('repopattern', nargs='*', default = ['*'])
-listpkgs.set_defaults(func=list_packages)
-default_options['list'] = {}
-
-options = parser.parse_args()
-if hasattr(options, "func"):
- for key in default_options[options.command]:
- setattr(options, key, default_options[options.command][key])
- options.func(options)
-else:
- parser.print_help()
+if __name__ == '__main__':
+ main()
================================================================
---- gitweb:
http://git.pld-linux.org/gitweb.cgi/projects/git-slug.git/commitdiff/4a7e426b8f1a3571094b5dc89412bc49b8f29666
More information about the pld-cvs-commit
mailing list