LIBSOLV-HISTORY(3)

💖 LIBSOLV-HISTORY 💖

okay bestie, so you know libsolv — the manual page here is LIBSOLV-HISTORY(3), and it's literally the origin story. the lore. how the libsolv library came into existence. grab a snack, we're doing a flashback episode 🦋

🌸 HISTORY 🌸

so this whole thing kicked off in May 2007, when the zypp folks decided they wanted to switch to a database to speed up installation. and mls — that's me, hi 🙋 — was NOT a big fan of databases. like, at all. so I was sitting there wondering if there'd even be any real merit in using one for solving, because the package dependencies of all packages have to be read in anyway?? make it make sense 😭

flashback further: back in 2002, I'd already researched that using a dictionary approach for storing dependencies can shrink the packages file down to 1/3 of its size. so I extended that idea a little more and decided to store all strings and relations as unique 32-bit numbers. and this has three big advantages:

• because of the unification, testing whether two strings are equal is the same as testing the equality of two numbers, thus very fast ⚡
• much space is saved, as numbers do not take up as much space as strings — the internal memory representation does not take more space on a 64-bit system where a pointer is twice the size of a 32-bit number

and that's how the solv format was created ✨ it stores a repository as a string dictionary, a relation dictionary and then all packages dependencies. tests showed that reading and merging multiple solv repositories takes just some milliseconds. iconic behavior, no notes 👑

🧪 Early solver experiments

having a new repository format was one big step, but the other area where libzypp needed improvement was the solver. Libzypp's solver was a port from the Red Carpet solver, which was written to update packages in an already installed system. using it for the complete installation progress brought it to its limits 😬 also, the added extensions like support for weak dependencies and patches made it fragile and unpredictable. not the vibe.

and since I was not very pleased with the way the solver worked, I went looking at other solver algorithms. I checked smart, yum and apt, but could not find a convincing algorithm. my own experiments also were not very convincing — they worked fine for some problems but failed miserably for other corner cases 💀

🔥 Using SAT for solving

SUSE's hack week at the end of June 2007 turned out to be a turning point for the solver. googling for solver algorithms, I stumbled over some note saying that some people are trying to use SAT algorithms to improve solving on Debian. looking at the SAT entry in Wikipedia, it was easy to see that this indeed was the missing piece: SAT algorithms are well researched and there are quite some open source implementations. so I decided to look at the minisat code, because it's one of the fastest solvers while consisting of not too many lines of code. main character research moment fr fr 🎀

of course, directly using minisat would not work, because a package solver does not need to find just one correct solution — it also has to optimize some metrics, i.e. keep as many packages installed as possible. so I needed to write my own solver, incorporating the ideas and algorithms used in minisat. this wasn't very hard, and at the end of the hack week the solver calculated the first right solutions. we ate that 💅

💼 Selling it to libzypp

with those encouraging results, I went to Klaus Kaempf, the system management architect at SUSE. we spoke about how to convince the team to make libzypp switch to the new solver. fortunately, libzypp comes with a plethora of solver test cases, so we decided to make the solver pass most of the test cases first. Klaus wrote a "deptestomatic" implementation to check the test cases. together with Stephan Kulow, who is responsible for the openSUSE distribution, we tweaked and extended the solver until most of the test cases looked good 🙌

Duncan Mac-Vicar Prett, the team lead of the YaST team, also joined development by creating Ruby bindings for the solver. later, Klaus improved the bindings and ported them to some other languages. group project energy, love that for us 🫶

📦 The attribute store

the progress with the repository format and the solver attracted another hacker to the project: Michael Matz from the compiler team. he started with improving the repository parsers so that patches and content files also generate solvables. after that, he concentrated on storing all of the other metadata of the repositories that are not used for solving, like the package summaries and descriptions. at the end of October, a first version of this "attribute store" was checked in. its design goals were:

• space efficient storage of attributes
• paging/on demand loading of data
• page compression

the first version of the attribute store used a different format for storing information — we later merged this format with the solv file format. evolution, bestie 🌈

🧩 libzypp integration

integration of the sat-solver into libzypp also started in October 2007, by Stefan Schubert and Michael Andres from the YaST team. the first versions supported both the old solver and the new one by using the old repository read functions and converting the old package data in-memory into a sat solver pool. solvers could be switched with the environment variable ZYPP_SAT_SOLVER. 👀

the final decision to move to the new solver was made in January of 2008, first just by making the new solver the default one, later by completely throwing out the old solver code. this had the advantage that the internal solvable storage could also be done by using the solver pool — something Michael Matz already played with in a proof of concept implementation showing some drastic speed gains 🔥

the last traces of the old database code were removed in February. rest in peace old database code, you will not be missed 🪦

✍️ AUTHOR

Michael Schroeder <mls@suse.de>

and that's the whole saga, bestie 💜 this was libsolv, page dated 03/02/2022, and yes — we really did all of that. go forth and solve some dependencies, you've got this 👑✨