LFIT(1)

📦 version 0.9.4, in debian bookworm, debian trixie, ubuntu jammy, ubuntu noble, ubuntu resolute, ubuntu stonking

💖 LFIT(1) 💖

okay bestie, so this is LFIT(1), and it lives in the User Commands section of the manual. buckle up, she's a whole vibe. 🎀

✨ NAME ✨

lfit — general purpose evaluation and regression analysis tool

🦋 SYNOPSIS 🦋

lfit [method of analysis] [options] <input> [-o, --output <output>]

🌸 DESCRIPTION 🌸

So lfit is a standalone, command-line-driven tool for data analysis and regression, and she works both interactively and in batch processing. Versatile queen. 💅

Basically she runs in two modes, and this is the whole plot:

  1. Regression analysis mode — lfit supports a whole roster of regression analysis methods you can use to hunt down the "best fit" parameters of model functions, so you can model your input data (which gets read from one or more input files, in tabulated form). ✨
  2. Arithmetic / evaluation mode — lfit can read input data and perform all kinds of arithmetic operations, exactly as you the user specify them. Realistically this second mode is what you use to evaluate your model functions using the parameters you presumably already got from the actual regression methods — and to complete that evaluation, you only need slight changes in your command line invocation arguments. so close to the first one, it's giving sister energy. 🫶

👑 OPTIONS 👑

🌈 General options:

  • -h, --help — gives you the general summary about the command line options.
  • --long-help, --help-long — gives a detailed list of command line options.
  • --wiki-help, --help-wiki, --mediawiki-help, --help-mediawiki — gives a detailed list of command line options, but in Mediawiki format.
  • --version, --version-short, --short-version — gives some version information about the program.
  • --functions, --list-functions, --function-list — lists the available arithmetic operations and built-in functions that the program supports.
  • --wiki-functions, --functions-wiki — lists the available arithmetic operations and built-in functions the program supports, in Mediawiki format.
  • --examples — prints some very basic examples for the program invocation.

💜 Common options for regression analysis:

  • -v, --variable, --variables <list-of-variables> — comma-separated list of regression variables. For non-linear regression analysis, all of these fit variables are expected to have some initial values (specified as <name>=<value>); otherwise, the initial values are set to be zero. Heads up: for some of the regression/analysis methods, additional parameters should be assigned to these fit/regression variables. See the section "Regression analysis methods" for the extra details. 📋
  • -c, --column, --columns <independent>[:<column index>],... — comma-separated list of independent variable names, read from the subsequent columns of the primary input data file. If your independent variables aren't in sequential order in the input file, you should define the optional column indices for each variable by separating the column index with a colon after the variable's name. And when you're dealing with multiple input files and data blocks, you should assign the individual independent variables plus the respective column names and definitions for each file (more on this later, in the section "Multiple data blocks"). 📊
  • -f, --function <model function> — the model function of the analysis, in symbolic form. This expression should contain built-in arithmetic operators, built-in functions, user-defined macros (see -x, --define) or functions provided by dynamically loaded external modules (see -d, --dynamic). The model function can depend on both the fit/regression variables (see -v, --variables) and the independent variables read from the input file (see -c, --columns). With multiple input files and data blocks, you should assign the respective model functions for each data block (see later). Note that some of the analysis methods expect the model function to be either differentiable or linear in the fit/regression variables. See "Regression analysis methods" below for more details. 🧮
  • -y, --dependent <dependent expression> — the dependent variable of the regression analysis, in the form of an arithmetic expression. This expression can depend only on the variables read from the input file (see -c, --columns). With multiple input files and data blocks, you should assign the respective dependent expressions for each data block (see later). 🎯
  • -o, --output <output file> — name of the output file into which the fit results (the values for the fit/regression variables) are written.

🧚 Common options for function evaluation:

  • -f, --function <function to evaluate>[...] — list of functions to be evaluated. You can specify more expressions by either separating the subsequent expressions with a comma, or by specifying more -f, --function options on the command line. either way, slay. ✨
Note that the two basic modes of lfit are distinguished only by the presence or the absence of the -y, --dependent command line argument. In other words, there is no explicit command line argument that specifies the mode of lfit. If the -y, --dependent command line argument is omitted, lfit runs in function evaluation mode; otherwise, the program runs in regression analysis mode. that's the whole vibe check right there. 👑
  • -o, --output <output file> — name of the output file in which the results of the function evaluation are written.

🔥 Regression analysis methods:

  • -L, --clls, --linear — the default mode of lfit, aka the classical linear least squares (CLLS) method. The model functions specified after -f, --function are expected to be both differentiable and linear with respect to the fit/regression variables. Otherwise, lfit detects the non-differentiable and non-linear property of the model function(s) and refuses the analysis. 💀 In that case you can apply other types of regression analysis methods depending on your needs — for instance the Levenberg-Marquardtalgorithm (NLLM, see -N, --nllm) or the downhill simplex minimization (DHSX, see -D, --dhsx).
  • -N, --nllm, --nonlinear — this option implies a regression involving the nonlinear Levenberg-Marquardt (NLLM) minimization algorithm. The model function(s) specified after -f, --function are expected to be differentiable with respect to the fit/regression variables. Otherwise, lfit detects the non-differentiable property and refuses the analysis. There are some fine-tune parameters of the Levenberg-Marquardt algorithm — see the section "Fine-tuning of regression analysis methods" for more details on how these additional regression parameters can be set. Note that all of the fit/regression variables should have a proper initial value, defined in the command line argument -v, --variable (see also there). 🔧
  • -U, --lmnd — Levenberg-Marquardt minimization with numerical partial derivatives (LMND). Same as the NLLM method, except that the partial derivatives of the model function(s) are calculated numerically. Because of that, the model function(s) may contain functions whose partial derivatives are not known in an analytic form. The differences used in the computations of the partial derivatives should be declared by the user — see also the command line option -q, --differences. 🧠
  • -D, --dhsx, --downhill — this option implies a regression involving the nonlinear downhill simplex (DHSX) minimization algorithm. You should specify the proper initial values and their uncertainties as <name>=<initial>:<uncertainty> — unless the "fisher" option is passed to the -P, --parameters command line argument (see later in the section "Fine-tuning of regression analysis methods"). In the first case, the initial size of the simplex is based on the uncertainties you provided; in the second case, the initial simplex is derived from the eigenvalues and eigenvectors of the Fisher covariance matrix. Note that the model functions must be differentiable in that latter case. 🏔️
  • -M, --mcmc — this option implies the method of Markov Chain Monte-Carlo (MCMC). The model function(s) can be arbitrary as far as differentiability goes. However, each of the fit/regression variables must have an initial assumption for their uncertainties, which must be specified via the command line argument -v, --variable. You should specify the proper initial values and uncertainties of these as <name>=<initial>:<uncertainty>. In the actual implementation of lfit, each variable has an uncorrelated Gaussian a priori distribution with the specified uncertainty. The MCMC algorithm has some fine-tune parameters — see the section "Fine-tuning of regression analysis methods" for more details. 🎲
  • -K, --mchi, --chi2 — with this option you can perform a "brute force" Chi^2 minimization, by evaluating the value of the merit function of Chi^2 on a grid of the fit/regression variables. In this case the grid size and resolution must be specified in a specific form after the -v, --variable command line argument. Namely, each of the fit/regression variables intended to be varied on a grid must have the format <name>=[<min>:<step>:<max>], while the other ones specified as <name>=<value> are kept fixed. The output of this analysis will be a series of lines with N+1 columns, where the values of fit/regression variables are followed by the value of the merit function. Note that all of the declared fit/regression variables are written to the output, including the ones which are fixed (so yeah, the output is somewhat redundant). 📈
  • -E, --emce — okay so this option implies the method of "refitting to synthetic data sets", or "error Monte-Carlo estimation" (EMCE). This method must have a primarily assigned minimization algorithm, and that can be any of the CLLS, NLLM or DHSX methods. First, the program searches the best fit values for the fit/regression variables involving the assigned primary minimization algorithm and reports these best fit variables. Then, additional synthetic data sets are generated around this set of best fit variables and the minimization is repeated involving the same primary method. The synthetic data sets are generated independently for each input data block, taking into account the fit residuals. The noise added to the best fit data is generated from the power spectrum of the residuals. periodt 💅
  • -X, --xmmc — bestie, this option implies an improved/extended version of the Markov Chain Monte-Carlo analysis (XMMC). The major differences between the classic MCMC and XMMC methods are the following: 1/ The transition distribution is derived from the Fisher covariance matrix. 2/ The program performs an initial minimization of the merit function involving the method of downhill simplex. 3/ Various sanity checks are performed in order to verify the convergence of the Markov chains, including the comparison of the actual and theoretical transition probabilities, the computation of the autocorrelation lengths of each fit/regression variable series and the comparison of the statistical and Fisher covariance. it's giving rigorous, no cap 🔥
  • -A, --fima — Fisher information matrix analysis (FIMA). With this analysis method one can estimate the uncertainties and correlations of the fit/regression variables involving the method of Fisher matrix analysis. This method does not minimize the merit functions by adjusting the fit/regression variables; instead, the initial values (specified after the -v, --variables option) are expected to be the "best fit" ones. main character energy fr fr ✨

✨ Fine-tuning of regression analysis methods ✨

  • -e, --error <error expression> — Expression for the uncertainties. Note that zero or negative uncertainty is equivalent to zero weight, i.e. input lines with zero or negative errors are discarded from the fit. so don't play with those error bars, bestie 😭
  • -w, --weight <weight expression> — Expression for the weights. The weight is simply the reciprocal of the uncertainty. The default error/uncertainty (and therefore the weight) is unity. Note that most of the analysis/regression methods are rather sensitive to the uncertainties since the merit function also depends on these. she's sensitive, okay? 💖
  • -P, --parameters <regression parameters> — this option is followed by a set of optional fine-tune parameters, and that set is different for each primary regression analysis method:
    • default, defaults — Use the default fine-tune parameters for the given regression method. easy, iconic, no notes 👑
    • clls, linear — Use the classic linear least squares method as the primary minimization algorithm of the EMCE method. Like in the case of the CLLS regression analysis (see -L, --clls), the model function(s) must be both differentiable and linear with respect to the fit/regression variables. math girlies rise 🧚
    • nllm, nonlinear — Use the non-linear Levenberg-Marquardt minimization algorithm as the primary minimization algorithm of the EMCE method. Like in the case of the NLLM regression analysis (see -N, --nllm), the model function(s) must be differentiable with respect to the fit/regression variables. differentiable or it's not invited 💜
    • lmnd — Use the non-linear Levenberg-Marquardt minimization algorithm as the primary minimization algorithm of the EMCE method. Like in the case of -U, --lmnd regression method, the parametric derivatives of the model function(s) are calculated by a numerical approximation (see also -U, --lmnd and -q, --differences for additional details). numerical approximation era ✨
    • dhsx, downhill — Use the downhill simplex (DHSX) minimization as the primary minimization algorithm of the EMCE method. Unless the additional 'fisher' option is specified directly, like in the default case of the DHSX regression method, the user should specify the uncertainties of the fit/regression variables that are used as an initial size of the simplex. simplex said slay 🦋
    • mc, montecarlo — Use a primitive Monte-Carlo diffusion minimization technique as the primary minimization algorithm of the EMCE method. The user should specify the uncertainties of the fit/regression variables which are then used to generate the Monte-Carlo transitions. This primary minimization technique is rather nasty (very slow), so its usage is not recommended. literally do not, bestie, I'm begging 💀
    • fisher — In the case of the DHSX regression method or in the case of the EMCE method when the primary minimization is the downhill simplex algorithm, the initial size of the simplex is derived from the Fisher covariance approximation evaluated at the point represented by the initial values of the fit/regression variables. Since the derivation of the Fisher covariance requires the knowledge of the partial derivatives of the model function(s) with respect to the fit/regression variables, the(se) model function(s) must be differentiable. On the other hand, the user do not have to specify the initial uncertainties after the -v, --variables option since these uncertainties derived automatically from the Fisher covariance. she does the math for you, queen 👑
    • skip — In the case of EMCE and XMMC method, the initial minimization is skipped. we're skipping, periodt 💅
    • lambda=<value> — Initial value for the "lambda" parameter of the Levenberg-Marquardt algorithm. lambda bestie ✨
    • multiply=<value> — Value of the "lambda multiplicator" parameter of the Levenberg-Marquardt algorithm. multiply era ✨
    • iterations=<max.iterations> — Number of iterations during the Levenberg-Marquardt algorithm. loop it, bestie 🔁
    • accepted — Count the accepted transitions in the MCMC and XMMC methods (default). accepted, we love that 🫶
    • nonaccepted — Count the total (accepted plus non-accepted) transitions in the MCMC and XMMC methods. the rejected ones still count, okay? 🙈
    • gibbs — Use the Gibbs sampler in the MCMC method. gibbs sampler said hi 🎀
    • adaptive — Use the adaptive XMMC algorithm (i.e. the Fisher covariance is re-computed after each accepted transition). adaptive queen behavior 💖
    • window=<window size> — Window size for calculating the autocorrelation lengths for the Markov chains (these autocorrelation lengths are reported only in the case of XMMC method). The default value is 20, which is fine in the most cases since the typical autocorrelation lengths are between 1 and 2 for nice convergent chains. window of opportunity, lowkey 🪟
  • -q, --difference <variablename>=<difference>[,...] — The analysis method of LMND (Levenberg-Marquardt minimization using numerical derivatives, see -U, --lmnd) requires the differences that are used during the computations of the partial derivatives of the model function(s). With this option, one can specify these differences. tiny differences, big impact 🦋
  • -k, --separate <variablename>[,...] — In the case of non-linear regression methods (for instance, DHSX or XMMC) the fit/regression variables in which the model functions are linear can be separated from the nonlinear part and therefore make the minimization process more robust and reliable. Since the set of variables in which the model functions are linear is ambiguous, the user should explicitly specify this supposedly linear subset of regression variables. (For instance, the model function "a*b*x+a*cos(x)+b*sin(x)+c*x^2" is linear in both "(a,c)" and "(b,c)" parameter vectors but it is non-linear in "(a,b,c)".) The program checks whether the specified subset of regression variables is a linear subset and reports a warning if not. Note that the subset of separated linear variables (defined here) and the subset of the fit/regression variables affected by linear constraints (see also section "Constraints") must be disjoint. disjoint or it's drama 😭
  • --perturbations <noise level>, --perturbations <key>=<noise level>[,...] — Additional white noise to be added to each EMCE synthetic data sets. Each data block (referred here by the appropriate data block keys, see also section "Multiple data blocks") may have different white noise levels. If there is only one data block, this command line argument is followed only by a single number specifying the white noise level. a lil noise never hurt nobody 🎀

✨ Additional parameters for Monte-Carlo analysis ✨

  • -s, --seed <random seed> — Seed for the random number generator. By default this seed is 0, thus all of the Monte-Carlo regression analyses (EMCE, MCMC, XMMC and the optional generator for the FIMA method) generate reproducible parameter distributions. A positive value after this option yields alternative random seeds while all negative values result in an automatic random seed (derived from various available sources, such as /dev/[u]random, system time, hardware MAC address and so), therefore distributions generated involving this kind of automatic random seed are not reproducible. reproducible by default, chaotic if you want it 🌈
  • -i, --[mcmc,emce,xmmc,fima]-iterations <iterations> — The actual number of Monte-Carlo iterations for the MCMC, EMCE, XMMC methods. Additionally, the FIMA method is capable to generate a mock Gaussian distribution of the parameter with the same covariance as derived by the Fisher analysis. The number of points in this mock distribution is also specified by this command line option. iterations go brrr 🔥

✨ Clipping outlier data points ✨

  • -r, --sigma, --rejection-level <level> — Rejection level in the units of standard deviations. bye outliers, it's not personal 💅
  • -n, --iterations <number of iterations> — Maximum number of iterations in the outlier clipping cycles. The actual number of outlier points can be traced by increasing the verbosity of the program (see -V, --verbose). clip, clip, clip again ✂️
  • --[no-]weighted-sigma — During the derivation of the standard deviation, the contribution of the data points data points can be weighted by the respective weights/error bars (see also -w, --weight or -e, --error in the section "Fine-tuning of regression analysis methods"). If no weights/error bars are associated to the data points (i.e. both -w, --weight or -e, --error options are omitted), this option will have no practical effect. weighted sigma said it depends, bestie 🙈

Quick lil heads-up before we get into the block stuff, bestie: in the actual version of lfit, only the CLLS, NLLM and LMND regression methods support that whole outlier-clipping thing we were just talking about. So don't come crying to me when the other methods ghost you on it. 💅

🧩 Multiple data blocks 🧩

Okay so you can have several data blocks going at once, and each one gets its own little keyed tag. Here's how you feed them:

  • -i<key> <input file name> — the input file name for the data block tagged with <key>. 📂
  • -c<key> <independent>[:<column index>],... — column definitions (same energy as -c, --columns) but for the data block named <key>.
  • -f<key> <model function> — the expression for the model function you're assigning to the data block named <key>.
  • -y<key> <dependent expression> — the expression of the dependent variable for the data block named <key>.
  • -e<key> <errors> — the expression of the uncertainties for the data block named <key>.
  • -w<key> <weights> — the expression of the weights for the data block named <key>. And just like with -e, --errors and -w, --weights, you should only ever specify <b>one</b> of the -e<key>, -w<key> arguments. Pick a lane, babe. 🚦

🔗 Constraints 🔗

  • -t, --constraint, --constraints <expression>{=<>}<expression>[,...] — a list of fit and domain constraints between the regression variables. ✨ Every fit constraint expression has to be linear in the fit/regression variables, and lfit will literally check that for you and throw an error if any of your constraints turn out to be non-linear. A domain constraint, on the other hand, can be any expression involving an arbitrary binary arithmetic relation — think strict greater than '>', strict less than '<', greater or equal to '>=', or less or equal to '<='. You can hand these over either as a comma-separated list after a single -t, --constraints argument, or as a whole bunch of separate -t, --constraints arguments. Your call, queen. 👑
  • -v, --variable <name>:=<value> — this is the other flavour of constraint-specifying. The variable specs after -v, --variable can also define constraints, you just write ":=" instead of "=" between the variable name and its initial value. So basically:
-v <name>:=<value>

is equivalent to

-v <name>=<value> -t <name>=<value>

Two-for-one, we love to see it. 🫶

🛠️ User-defined functions 🛠️

  • -x, --define, --macro <name>(<parameters>)=<definition expression> — this is your moment to be a main character and define your own extra functions (aka macros) stacked right on top of the built-in functions and operators, the dynamically loaded functions, and any macros you already defined earlier. But listen: each of these user-defined functions has to be stand-alone — meaning external variables (like fit/regression variables and independent variables) absolutely cannot appear in the definition expression. Only the function's own parameters, periodt. 🚫

📚 Dynamically loaded extensions and functions 📚

  • -d, --dynamic <library>:<array>[,...] — loads the dynamically linked library (aka shared object) named <library> and imports the global lfit-compatible set of functions that live in the arrays named after the library. Those arrays have to be declared with the type 'lfitfunction', exactly as it's defined in the file "lfit.h". 📎 Each record in the array is like a lil info card for one imported function, holding: the actual name of the function, flags saying whether it's differentiable and/or linear in its regression parameters, the number of regression variables and independent variables, and the actual C subroutine that implements the evaluation of the function (plus the optional computation of the partial derivatives). The module 'linear.c' and 'linear.so' is a cute simple example that implements the "line(a,b,x)=a*x+b" function — it's got two regression variables ("a" and "b") and one independent variable ("x"), and the function itself is linear in the regression variables. 🌈

🎀 More on outputs 🎀

  • -z, --columns-output <column indices> — the column indices where results get written in evaluation mode. If you skip this option entirely, the results of the function evaluation are written sequentially. Otherwise, the input file gets written to the output and the appropriate columns (the ones you named here) get replaced by the respective results of the function evaluation. So even though the default column order is sequential, there's a big difference between leaving this option out and passing "-z 1,2,...,N". In the first case, the output file contains only the results of the function evaluations. In the latter case, the first N columns of the original file get swapped out for the results. 💀 Different vibes entirely.
  • --errors, --error-line, --error-columns — prints the uncertainties of the fit/regression variables. 📏
  • -F, --format <variable name>=<format>[,...] — the printf-style format of the output for each fit/regression variable (go peek at printf(3) if you need a refresher). The default format is %12.6g (that's 6 significant figures). 🔢
  • -F, --format <format>[,...] — the format of the output in evaluation mode. Default is %12.6g (6 significant figures).
  • -C, --correlation-format <format> — the format of the correlation matrix elements. Default is %6.3f (3 significant figures).
  • -g, --derived-variable[s] <variable name>=<expression>[,...] — some of the regression and analysis methods can compute uncertainties and correlations for derived regression variables. These extra variables (which are therefore not independent) get defined with this command line option. In the definition expression you should only use the fit/regression variables (as defined by the -v, --variables command line argument). You can also set the output format for these variables with the -F, --format command line argument. 🦋
  • -u, --output-fitted <filename> — name of an output file that receives all the lines of the input that were involved in the final regression. Super useful when you're doing outlier clipping and wanna see exactly which subset of the input data actually made it into the fit (see also the -n, --iterations and -r, --sigma options). 👑
  • -j, --output-rejected <filename> — name of an output file that receives all the lines of the input that were rejected from the final regression. Also super useful for outlier clipping, so you can see which subset of the input had a dependent variable that was giving outlier-point energy (see also the -n, --iterations and -r, --sigma options). 🙈
  • -a, --output-all <filename> — a file containing the lines of the input file that were involved in the complete regression analysis. This file is literally just the original file, except the commented and empty lines get dropped.
  • -p, --output-expression <filename> — into this file, the model function gets written with the fit/regression variables replaced by their best-fit values. ✨
  • -l, --output-variables <filename> — a list of the names and values of the fit/regression variables, in the exact same format used after the -v, --variables command line argument. That means you can feed the content of this file straight into later invocations of lfit. Chain them together, bestie. 🔗
  • --delta — writes the individual differences between the independent variables and the evaluated best fit model function values, for each line, in the output files specified by the -u, --output-fitted, -j, --output-rejected and -a, --output-all command line options.
  • --delta-comment — same as --delta, except the differences are written as a comment, i.e. separated by a '##' from the original input lines. 💬
  • --residual — writes the final fit residual to the output file, right after the list of the best-fit values for the fit/regression variables.

🐛 REPORTING BUGS 🐛

Found a bug? Report it to <apal@szofi.net>, and you can also check out https://fitsh.net/. Don't be shy, they wanna know. 🫶

Copyright © 1996, 2002, 2004-2008, 2009-2020; Pal, Andras <apal@szofi.net>

And that's the whole page, bestie!! 🌸 This was lfit version 7.9, dated 2021.01.24, and the page itself is from January 2021 — you'll find it filed under LFIT(1). Go forth and fit some curves, you gorgeous genius. 💖✨🎀

📦 where this page comes from