JUST-DEDUPLICATE-REPOS(1)

📦
version 1.6.5 · 3 versions
, in ubuntu stonking

💖 just-deduplicate-repos — the manual page, but make it make sense 💖

okay bestie, sit down, hydrate, and let me tell you about just-deduplicate-repos, a lil command from the General Commands Manual whose whole personality is: remove duplicate repositories from a multi-repository configuration. that's it. that's the job. and it ate. 💅

✨ NAME ✨

just-deduplicate-repos — literally just removes duplicate repositories from a multi-repository configuration. no notes. 🎀

🌸 SYNOPSIS 🌸

just-deduplicate-repos [repository name]...

🦋 DESCRIPTION 🦋

okay so here's the tea. just-deduplicate-repos reads a multi-repository configuration from stdin and prints to stdout a repository configuration where all the indistinguishable repositories get merged into one single repository. slay. 🔥

in doing so, it keeps the "main" repository, and it also keeps any repositories you passed as arguments on the command line — even if that means some indistinguishable repositories cannot be merged into one (because both of them have to be kept). yes bestie, sometimes the merge just can't happen. it's giving "we tried". 😭

🧚 RATIONALE 🧚

so like, why does this exist? buckle up, this is the lore. 📖

Since justbuild(1) is a multi-repository build system, the dependencies you import typically also contain a multi-repository setup of their own. so a project typically has three components describing the involved logical repositories. three. a trilogy. 👑

  • First — there's a description of the local repositories, i.e. the logical repositories (as "file" repositories) that reside in this physical repository in the sense of the version-control system (typically git(1)). They're described in a file often called etc/repos.template.json in just-mr-repository-config(5) format, with open names for the direct dependencies. 🌱
  • Next — there's a description of where to get the direct dependencies and which branches to follow there. This description is typically a script piping said repos.template.json through a sequence of invocations of just-import-git(1), which also adds the indirect dependencies. 🌀
  • Finally — the output of that script is the multi-repository configuration etc/repos.json for this project. 🌈

and here's the part where etc/repos.json gets its main character moment, because even though it's generated, it's still committed into the repository. why? let me count the ways:

  • First of all, it contains additional information: the precise pinned versions of (all) the dependencies, which might change over time. So committing this file allows others to build with the precise same dependencies — including for old versions of the project. no cap. 🫶
  • Moreover, having the multi-repository configuration materialized in the repository allows offline builds once the dependencies have been fetched (possibly by a different project) to just-mr(1)'s local build root (possibly by the setup for a different project that happens to have those dependencies as well).
  • To update the dependencies, the import script can be run again and the newly created etc/repos.json committed in an update commit — after verifying that this update does not break anything. pls verify first. 🙈
  • The final reason to commit a generated etc/repos.json is to close the loop and allow this project to be the dependency of another one. To keep the work done by just-import-git(1) manageable, it requires the imported repository to have all its (direct or indirect) dependencies described precisely in their multi-repository config. the circle of life. 🦋

now. here's where it gets messy. 😭

When a project has more than one direct dependency, it can happen that two of the direct dependencies have a common dependency. and if you simply chain just-import-git(1), this dependency will end up twice in the final repository configuration. dramatic. 💀

will it result in additional actions? no bestie. but it will increase the cost of the analysis. and worse — not merging indistinguishable repositories will make the resulting etc/repos.json bigger and propagate those redundant copies to other projects importing this one. and that mechanism can, over a long chain of imports, lead to exponentially many of those redundant copies. exponential. not cute. 🚩

so to avoid all that: just-deduplicate-repos can be added as a last step (before the JSON pretty printing) in the import script. periodt. 💅✨

ngl, this lil command is the de-dup queen and she deserves her flowers. 🌸

👀 See also 👀

and that's the whole page bestie!! JUST-DEDUPLICATE-REPOS(1) from the General Commands Manual — go forth, deduplicate, and keep your etc/repos.json skinny. 💖✨🎀🦋 lowkey obsessed with you for reading the whole thing. bye bestie!! 🫶👑🌈⭐

📦 where this page comes from