indexamajig(1)

📦
version 0.13.0 · 2 versions
, in nixpkgs unstable

🌸 NAME 🌸

indexamajig — bulk indexing and data reduction program. And yep bestie, this is the indexamajig(1) manual page, but we're doing it in group-chat mode 💖

✨ SYNOPSIS ✨

indexamajig -i filename -o output.stream -g detector.geom --peaks=method --indexing=method ...

💅 DESCRIPTION 💅

okay so indexamajig takes a list of diffraction snapshots from crystals in random orientations and attempts to find peaks, index and integrate each one. The input is a list of diffraction image files plus some auxiliary files and parameters. The output is a long text file (a 'stream') containing the results from each image in turn ✨

For minimal basic use, you need to provide the list of diffraction patterns, the method which will be used to index, a file describing the geometry of the detector, and a file which contains the unit cell which will be used for the indexing. Here is what the minimal use might look like on the command line:

indexamajig
   -i mypatterns.lst
   -g mygeometry.geom
   --indexing=xgandalf
   --peaks=hdf5
   -p mycell.pdb
   -o test.stream

More typical use includes all the above, but might also include extra parameters to modify the behaviour. For example, you'll probably want to run more than one indexing job at a time (-j). 🫶

See man crystfel_geometry for information about how to create a file describing the detector geometry and beam characteristics.

🦋 DIFFRACTION PATTERN LIST 🦋

indexamajig requires an input file with a list of diffraction patterns to process. In its simplest form, this is just a text file containing a list of image filenames. The image files might be in some folder a long way from the current directory, so you might want to specify a full pathname to be added in front of each filename. The geometry file includes a description of the data layout within the files. indexamajig uses this description to determine the number of diffraction patterns stored in each file, and tries to process them all. You can also specify explicitly which frames you would like to process by putting a string describing the frames after the file name(s) in this list.

🎀 PEAK DETECTION 🎀

You can control the peak detection on the command line. First, you can choose the peak detection method using --peaks=method. --peaks=hdf5 or --peaks=cxi will take the peak locations from the input file. See the documentation for peak_list and peak_list_type in crystfel_geometry(5) for details.

If you use --peaks=zaef, indexamajig will use a simple gradient search after Zaefferer (2000). You can control the overall threshold and minimum squared gradient for finding a peak using --threshold and --min-squared-gradient. The threshold has arbitrary units matching the pixel values in the data, and the minimum gradient has the equivalent squared units. Peaks will be rejected if the 'foot point' is further away from the 'summit' of the peak by more than the inner integration radius (see below). They will also be rejected if the peak is closer than twice the inner integration radius from another peak.

If you instead use --peaks=peakfinder8, indexamajig will use the "peakfinder8" peak finding algorithm described in Barty et al. (2014). Pixels above a radius-dependent intensity threshold are considered as candidate peaks (although the user sets an absolute minimum threshold for candidate peaks). Peaks are then only accepted if their signal to noise level over the local background is sufficiently high. Peaks can include multiple pixels and the user can reject a peak if it includes too many or too few. The distance of a peak from the center of the detector can also be used as a filtering criterion. Note that the peakfinder8 will not report more than 2048 peaks for each panel: any additional peak is ignored. 😭

If you instead use --peaks=peakfinder9, indexamajig will use the "peakfinder9" peak finding algorithm described in the master thesis "Real-time image analysis and data compression in high throughput X-ray diffraction experiments" by Gevorkov. Unlike peakfinder8, peakfinder9 uses local background estimation based on border pixels in a specified radius (--local-bg-radius). For being fast and precise, a hierarchy of conditions is used.

  1. The first condition is only useful for speed consideration. It demands that a pixel that is the biggest pixel in a peak must be larger than every border pixel by a constant value (--min-peak-over-neighbour).
  2. The second condition ensures that the pixel passing the previous condition is the highest pixel in the peak. It assumes that peaks rise monotonically towards the biggest pixel.
  3. The third condition ensures that the biggest pixel in the peak is significantly over the noise level (--min-snr-biggest-pix) by computing the local statistics from the border pixels in a specified radius.
  4. The fourth condition sums up all pixels belonging to the peak (--min-snr-peak-pix) and demands that the whole peak must be significantly over the noise level (--min-snr).

Only if all conditions are passed, the peak is accepted. 💖

🔥 INDEXING METHODS 🔥

You can choose between a variety of indexing methods. You can choose more than one method, in which case each method will be tried in turn until one of them reports that the pattern has been successfully indexed. Choose from:

  • dirax — Invoke DirAx. See Duisenberg, J. Applied Crystallography 25 (1992) p92, https://doi.org/10.1107/S0021889891010634.
  • mosflm — Invoke Mosflm. See Powell, Acta Crystallographica D55 (1999) p1690, https://doi.org/10.1107/S0907444999009506.
  • asdf — This is an implementation of the dirax algorithm, with some very small changes such as using a 1D Fourier transform for finding the lattice repeats. This algorithm is implemented natively within CrystFEL meaning that no external software is required.
  • felix — Invoke Felix, which will use your cell parameters to find multiple crystals in each pattern. To use this option, 'Felix' must be in your shell's search path. This can be a link to the latest version of Felix. If you see the Felix version information when you run Felix on the command line, things are set up correctly. See Beyerlein et al., J. Appl. Cryst. 50, p1075-1083, https://doi.org/10.1107/s1600576717007506.
  • xds — Invoke XDS, and use its REFIDX procedure to attempt to index the pattern.
  • taketwo — Use the TakeTwo algorithm. See Ginn et al., Acta Crystallographica D72 (2016), p956, https://doi.org/10.1107/S2059798316010706.
  • xgandalf — Use the eXtended GrAdieNt Descent Algorithm for Lattice Finding. See Gevorkov et al., Acta Crystallographica A75 (2019) p694, https://doi.org/10.1107/S2053273319010593.
  • pinkIndexer — Use the pinkIndexer algorithm. See Gevorkov et al., Acta Crystallographica A76 (2020) p121, https://doi.org/10.1107/S2053273319015559.
  • file — See Re-playing old indexing and injecting external results below.
  • ffbidx — Use the GPU-based fast-feedback indexer, similar to the TORO algorithm. See Gasparotto et al., J. Applied Crystallography 57 (2024) p931, https://doi.org/10.1107/S1600576724003182.
  • smallcell — Use an implementation of the cctbx.small_cell algorithm. See Brewster et al., Acta Crystallographica D71 (2015) p357, https://doi.org/10.1107/S1399004714026145.

Most of the indexing methods require some extra software to be installed, either at the time of compiling CrystFEL or afterwards. CrystFEL is distributed with a script (scripts/install-indexers) which can help you to quickly install all the required programs. 👑

If you don't specify any indexing methods, indexamajig will try to automatically determine which indexing methods are available. You can also specify indexing method none, in which case no indexing will be done. This is useful if you just want to check that the peak detection is working properly.

Versions of CrystFEL prior to (including) 0.12.0 allowed the indexing methods to be suffixed with flags -latt, -cell, -nolatt and --nocell, to control what prior information is given to the indexing algorithm. These are no longer used. The maximum possible use of the provided prior unit cell information will always be made.

🛠️ Post-indexing stages 🛠️

The indexing results from the indexing engine will be put through a number of refinement and checking stages. See the options --no-check-cell, --no-multi, --no-retry and --no-refine below for more details.

🧚 Re-playing old indexing and injecting external results 🧚

With --indexing=file, indexamajig will read indexing results from a text file. Use --fromfile-input-file to specify the filename. Each line of the file represents one diffraction pattern in one image, and should be formatted as follows:

filename frameID asx asy asz bsx bsy bsz csx csy csz xshift yshift latt_cen

To describe multiple overlapping diffraction patterns in one frame, simply use the same filename and frameID in multiple lines.

The lattice type and centering information are contained in latt_cen. This should consist of two or three characters, e.g. tIc, aP or oP. The first letter represents the lattice type (a, m, o, t, c, h, r for, respectively, triclinic, monoclinic, orthorhombic, tetragonal, cubic, hexagonal or rhombohedral). The centering symbol (P, A, B, C, I, F, H or R) follows. For monoclinic, tetragonal or hexagonal lattice types, the unique axis should be given as the third character. Otherwise, there should be only two characters.

The vector components asx,asy,... are the reciprocal lattice basis vectors in reciprocal nanometres. The xshift and yshift values are the offsets of the detector position in millimetres. Usually, these offsets should be zero.

This method can be used to inject indexing results from an external indexing program. It can also be used to re-play previous indexing results, which is useful for changing integration parameters while using slower indexing methods (such as pinkIndexer). To generate an indexing result from a previous indexamajig run, use the script stream2sol in the CrystFEL scripts folder. 🌈

💜 REFLECTION INTEGRATION 💜

If the pattern could be successfully indexed, peaks will be predicted in the pattern and their intensities measured. You have a choice of integration methods, and you specify the method using --integration. Choose from:

  • rings — bestie this one uses three concentric rings to figure out the peak region, the buffer region, and the background estimation region ✨ The radius of the smallest circle sets the peak region. The radius of the middle circle and the outer circle describe an annulus, and that annulus is where the background gets estimated from. You set the radii with --int-radius (see below, it's right there 💅). Ngl the default behaviour with rings is to not center the peak boxes first — if you want centering, use rings-cen instead.
  • prof2d — this one integrates the peaks using 2D profile fitting with a planar background, close to the method described by Rossmann (1979) J. Appl. Cryst. 12 p225, https://doi.org/10.1107/S0021889879012218. Iconic literature moment 🎀 The default behaviour with prof2d is to center the peak first — use prof2d-nocen to skip that step. Periodt.

Okay so you can also add one or more of these modifiers to any of the integration methods above 👇

  • -cen — center the peak boxes iteratively on the actual peak locations 🎯 The opposite is -nocen, which is the default.
  • -sat — normally, reflections that contain one or more pixels above max_adu (that's defined in the detector geometry file) will not be integrated and won't be written to the stream. Using this option skips that check entirely and lets saturated reflections get passed to the later merging stages 💀 The opposite is -nosat, which is the default for all integration methods. But heads up: the saturation check will only be done if max_adu is set in the geometry file. Usually it's better to exclude saturated reflections at the merging stage instead. See the docs for max_adu in crystfel_geometry(5) 🫶
  • -grad — fit the background around the reflection using gradients in two dimensions. Fun fact this was the default until version 0.6.1 ⭐ Without the option (or with its opposite, -nograd, which is the default), the background is considered to have the same value across the entire integration box, which gives better results in most cases. Slay.

💖 BASIC OPTIONS 💖

  • -i filename, --input=filename — read the list of images to process from filename. --input=- means read from stdin 🧚 There is no default.
  • -o filename, --output=filename — write the output data stream to filename.
  • -g filename, --geometry=filename — read the detector geometry description from filename. See man crystfel_geometry for more info 🌸

Also: --zmq-input=address receives data over ZeroMQ from address. The options --input and --zmq-input are mutually exclusive — you must specify exactly one of them, no cap 🔥 Example: --zmq-input=tcp://127.0.0.1:5002. If you use this option, you should also use either --zmq-subscribe to add a ZeroMQ subscription, or --zmq-request to define how to request data.

  • --zmq-subscribe=tag — subscribe to ZeroMQ message type tag. You can use this option multiple times to add multiple subscriptions 💅 This option and --zmq-request are mutually exclusive.
  • --zmq-request=msg — request new data over ZeroMQ by sending the string msg. This will cause indexamajig's ZeroMQ socket to use REQ mode instead of SUB 🦋 This option and --zmq-subscribe are mutually exclusive.
  • --asapo-endpoint=endpoint — receive data via the specified ASAP::O endpoint. This option and --zmq-input are mutually exclusive.
  • --asapo-token=token — authentication token for ASAP::O data 🔑
  • --asapo-beamtime=beamtime — beamtime ID for ASAP::O data.
  • --asapo-source=source — data source for ASAP::O data.
  • --asapo-group=group — consumer group name for ASAP::O data. Concurrent indexamajig processes working on the same data should use the same value for this, so the data gets shared between them 👯‍♀️
  • --asapo-stream=stream — stream name for ASAP::O data.
  • --asapo-output-stream=stream — send an output stream via ASAP::O. For non-hits, a small placeholder will be sent 🎀
  • --asapo-wait-for-stream — if the ASAP::O stream does not exist, wait for it to appear ⏳ Without this option, indexamajig exits immediately if the stream isn't found. Dramatic but fair.
  • --no-data-timeout — shut down the entire indexamajig process if the specified number of seconds elapse without any data being seen 😭 This currently applies to ASAP::O data only, but might be extended to other streaming systems in future. The default is 60 seconds.
  • --asapo-consumer-timeout — set the timeout used for "get next" calls from ASAP::O, in ms. The default is 500 ms.
  • --asapo-acks — use ASAP::O acknowledgements for more reliable message delivery ✅
  • --data-format=format — specify the data format for data received over ZeroMQ or ASAP::O. Possible values in this version are msgpack, hdf5 and seedee 🌈
  • --basename — remove the directory parts of the filenames taken from the input file. If --prefix or -x is also given, the directory parts of the filename will be removed before adding the prefix ✨
  • -x prefix, --prefix=prefix — prefix the filenames from the input file with prefix. If --basename is also given, the filenames will be prefixed after removing the directory parts of the filenames 💜
  • -j n — run n analyses in parallel. Default: 1. See also --max-indexer-threads. Tip: use -j $(nproc) to use as many processes as there are available CPUs 🔥
  • --cpu-pin — pin worker processes to CPUs. Usually this is not needed or desirable, but in some cases it dramatically improves performance 👑
  • --no-check-prefix — don't attempt to correct the prefix (see --prefix) if it doesn't look correct 🙈
  • --highres=n — mark all pixels on the detector higher than n Angstroms as bad. This might be useful when you have noisy patterns and don't expect any signal above a certain resolution 🦋
  • --profile — display timing data for performance monitoring ⏱️
  • --temp-dir=path — put the temporary folder under path.
  • --wait-for-file=n — wait at most n seconds for each image file in the input list to be created before trying to process it. This is useful for some automated processing pipelines 🧚 It obviously only really works for single-frame files. If a file exists but is not readable when this option is set non-zero, a second attempt will be made after ten seconds — this is to allow for incompletely written files. A value of -1 means to wait forever. The default value is --wait-for-file=0.
  • --no-image-data — do not load the actual image data (or bad pixel masks), only the metadata. This lets you check if patterns can be indexed without high data bandwidth requirements 💀 Obviously, any feature requiring the image data — especially peak search procedures and integration — cannot be used in this case. So you'll need to get the peaks from somewhere else (see --peaks=msgpack or --peaks=hdf5) 🫶

✨ PEAK SEARCH OPTIONS ✨

  • --peaks=method — find peaks in the images using method. See the section titled PEAK DETECTION (above) for more information 👀
  • --peak-radius=inner,middle,outer — set the inner, middle and outer radii for three-ring integration during the peak search 🌸 See the section about PEAK INTEGRATION, above, for details of how to determine these. The default is to use the same values as for --int-radius.
  • --min-peaks=n — do not try to index frames with fewer than n peaks. These frames will still be described in the output stream. To exclude them, use --no-non-hits-in-stream 💅 The default is --min-peaks=0, which means that all frames will be considered hits, even if they have no peaks at all. Iconic "everyone's a winner" energy.
  • --median-filter=n — apply a median filter with box "radius" n to the image. The median of the values from a (n+1)x(n+1) square centered on the pixel will be subtracted from each pixel 🎀 This might help with peak detection if the background is high and/or noisy. The unfiltered image will be used for the final integration of the peaks. If you also use --noise-filter, the median filter will be applied first.
  • --filter-noise — apply a noise filter to the image which checks 3x3 squares of pixels and sets all of them to zero if any of the nine pixels have a negative value 😭 This filter may help with peak detection under certain circumstances. The unfiltered image will be used for the final integration of the peaks, because the filter destroys a lot of information from the pattern. If you also use --median-filter, the median filter will be applied first.
  • --threshold=thres — set the overall threshold for peak detection using --peaks=zaef or --peaks=peakfinder8 to thres, which has the same units as the detector data. The default is --threshold=800 ⭐
  • --min-squared-gradient=grad — set the square of the gradient threshold for peak detection using --peaks=zaef to grad, which has units of "squared detector units per pixel". The default is --min-squared-gradient=100000. --min-sq-gradient and --min-gradient are synonyms for this option, however the latter should not be used, to avoid confusion 🙈
  • --min-snr=snr — set the minimum I/sigma(I) for peak detection when using --peaks=zaef, --peaks=peakfinder8 or --peaks=peakfinder9. The default is --min-snr=5 💜
  • --min-snr-biggest-pix= — (peakFinder9 only) min snr of the biggest pixel in the peak, given as a factor of the standard deviation. Default is 7.0 🔥
  • --min-snr-peak-pix= — (peakFinder9 only) min snr of a peak pixel, given as a factor of the standard deviation. Should be smaller or equal to sig_fac_biggest_pix. Default is 6.0 ✨
  • --min-sig= — (peakFinder9 only) minimum standard deviation of the background. Prevents finding of peaks in erroneous or highly shadowed unmasked regions 🛡️ Default is 11.0.
  • --min-peak-over-neighbour= — (peakFinder9 only) just for speed. The biggest pixel must be n higher than the pixels at window_radius distance to be a candidate for the biggest pixel in a peak 🏃‍♀️ Should be chosen as a small positive number, a few times smaller than the weakest expected peak. The default is -INFINITY, which turns off the speedup and searches with maximum precision 👑
  • --min-pix-count=cnt — with --peaks=peakfinder8, a peak only gets accepted if it's packing more than cnt pixels. default is --min-pix-count=2. small-and-mighty peaks only, bestie 💅
  • --max-pix-count=cnt — same deal but the other direction: with --peaks=peakfinder8, a peak is only accepted if it has less than cnt pixels. default is --max-pix-count=200. this is your "no giant blobs please" dial 🫶
  • --local-bg-radius=r — the radius (in pixels) used when estimating the local background, for --peaks=peakfinder8 or --peaks=peakfinder9. default is --local-bg-radius=3. ✨
  • --min-res=px — with --peaks=peakfinder8, only accept a peak if it sits more than px pixels from the detector center. default is --min-res=0.
  • --max-res=px — with --peaks=peakfinder8, only accept a peak if it sits less than px pixels from the detector center. default is --max-res=0, which literally means the cutoff is disabled, iykyk 🦋
  • --no-use-saturated — okay so normally, peaks containing one or more pixels above max_adu (that's defined in your detector geometry file) do get used for indexing — but not for the final integration (peep the peak integration section above for that whole saga). flipping this option on makes saturated peaks get ignored completely. the opposite is --use-saturated, and that's the default. 🎀
  • --no-revalidate — when you're running --peaks=hdf5, --peaks=cxi or --peaks=msgpack, your peaks get put through some of the same checks as if you'd used --peaks=zaef. those checks throw out peaks that are too close to panel edges, are saturated (unless you're on --use-saturated), have other peaks nearby (closer than twice the inner integration radius — see --int-radius), or have any part sitting in a bad region. this option skips that whole validation step and just uses the peaks directly. no vibe check, straight in 💀
  • --no-half-pixel-shift — lore drop: CrystFEL treats every peak location as a distance from the corner of the detector panel, in pixel units, which matches how it describes detector geometry (go read man crystfel_geometry). but the software that makes the image data files — Cheetah included — might instead think of peak locations as pixel indices into the data array. so by default, peak coordinates coming from --peaks=cxi or --peaks=hdf5 get 0.5 added to them. this option turns that half-pixel offset off. 🙈
  • --check-hdf5-snr — add this alongside --peaks=hdf5 (or cxi or msgpack) and the peaks get an extra check that they satisfy the minimum SNR you set with --min-snr. extra credit behaviour fr fr ⭐
  • --peakfinder8-fast — (peakfinder8 only!) speeds things up by restricting how many sampling points get used for the background statistics calculation. fast = fewer points, you know the drill 🔥

✨ INDEXING OPTIONS ✨

  • --indexing=method — index the patterns using method. the whole INDEXING METHODS section up above is your required reading for this one. default is to auto-detect which indexing methods to use, which is very main-character of it 👑
  • -p unitcell.cell, -p unitcell.pdb, --pdb=unitcell.pdb — point at the file holding your unit cell info. PDB format or CrystFEL format, dealer's choice 💖
  • --tolerance=tol — sets the tolerances for comparing unit cells. tol looks like a,b,c,al,be,ga. a, b and c are the tolerances in percent for the respective reciprocal space axes, and al, be, ga are the tolerances in degrees for the reciprocal space angles. if the unit cell is centered, the tolerances get applied to the corresponding primitive unit cell instead. default is --tolerance=5,5,5,1.5,1.5,1.5,1.5. 🎯
  • --no-check-cell — don't check the cell parameters against the reference unit cell (the one you gave with -p). if you were around for older CrystFEL versions: yes, this replaces sticking "-raw" in the indexing method. glow-up tbh ✨
  • --no-check-peaks — skips the check that most of the peaks can actually be accounted for by the indexing solution. the thresholds for what counts as a successful result are influenced by --multi. 🦋
  • --multi — enables the "subtract and retry" method: after a successful indexing attempt, the spots that the indexing solution explains get removed, then it tries indexing again hoping to find a second lattice. and no, this has nothing to do with multi-lattice indexing algorithms like Felix — don't get it twisted. this option also adjusts the thresholds for identifying successful indexing results (see --no-check-peaks). 🌸
  • --no-retry — kills retry indexing. normally, after a failed indexing attempt, indexamajig drops the 10% weakest peaks and goes again. this turns that off, which is much faster but lowers your indexing success rate. speed vs. success, pick your fighter 💀
  • --no-refine — skips the prediction refinement step. usually that's gonna tank your result quality and let false solutions sneak through, but occasionally it might be the move you need. ⚠️
  • --mille — write detector calibration data in Millepede-II format. 🎀
  • --mille-file=filename — write the Millepede-II data into filename. default is mille-data.bin, in the current directory. 🫶
  • --max-mille-level=n — write the Millepede-II data up to a maximum hierarchy depth of n. if n is 0, only the overall detector position can be refined. bigger numbers = finer-grained refinement. set it to a big number (9 is high enough!) to disable the limit entirely — though that can make your Millepede-II files get kinda enormous. default is --max-mille-level=99, i.e. effectively no limit at all. 🧚
  • --wavelength-estimate=m, --camera-length-estimate=m — some indexing algorithms need to know the camera length or the incident radiation wavelength up front, e.g. so they can prep an internal look-up table. but if those values come from image headers, they're not available at start-up 😭 in that case you'll get prompted to add one of these options with approximate values (in metres). and if the real value differs from yours by more than 10%, you get a warning. friendly lil heads-up 💜
  • --max-indexer-threads=n — some indexing algorithms (pinkIndexer says hi) can use multiple threads to calculate faster. this is on top of the frame-based parallelism indexamajig already has (see -j). this option caps how many threads each indexing engine is allowed to grab. default: 1. ⭐
  • --taketwo-member-threshold=n — minimum number of vectors in the network before the pattern counts as indexed. default 20.
  • --taketwo-len-tolerance=n — the length tolerance, in reciprocal Angstroms. default 0.001.
  • --taketwo-angle-tolerance=n — the angle tolerance, in degrees. default 0.6.
  • --taketwo-trace-tolerance=n — the rotation matrix trace tolerance, in degrees. default 3.
  • --felix-domega=n — degree range of omega (that's mosaicity, bestie) to consider. default 2.
  • --felix-fraction-max-visits=n — cutoff for the minimum fraction of the max visits. default: 0.75.
  • --felix-max-internal-angle=n — cutoff for the maximum internal angle between observed spots and predicted spots. default: 0.25.
  • --felix-max-uniqueness=n — cutoff for the maximum fraction of found spots which are allowed to belong to other crystallites. default: 0.5.
  • --felix-min-completeness=n — cutoff for the minimum fraction of projected spots actually found in the pattern. default: 0.001.
  • --felix-min-visits=n — cutoff for the minimum number of voxel visits. default: 15.
  • --felix-num-voxels=n — number of voxels for the Rodrigues space search. default: 100.
  • --felix-sigma=n — the sigma of the 2theta, eta and omega angles. default: 0.2.
  • --felix-tthrange-max=n — maximum 2theta to consider for indexing (degrees). default: 30.
  • --felix-tthrange-min=n — minimum 2theta to consider for indexing (degrees). default: 0.
  • --xgandalf-sampling-pitch=n — picks how densely reciprocal space gets sampled. [0-4]: extremelyLoose to extremelyDense. [5-7]: standardWithSecondaryMillerIndices to extremelyDenseWithSecondaryMillerIndices. default: 6 (denseWithSecondaryMillerIndices). 🌈
  • --xgandalf-grad-desc-iterations — picks how many gradient descent iterations get performed. [0-5]: veryFew to extremelyMany. default: 4 (manyMany).
  • --xgandalf-tolerance — relative tolerance of the lattice vectors. default: 0.02.
  • --xgandalf-no-deviation-from-provided-cell — if a prior unit cell was provided and this flag is set, the found unit cell will end up exactly the same size as the one you provided. no drift, no deviation, periodt 💅
  • --xgandalf-min-lattice-vector-length — minimum and maximum possible lattice vector lengths (the unit is A). used for fitting without a prior lattice, as the starting point for gradient descent — so heads up, the final minimum lattice vector length can end up smaller/bigger than min/max. note: this applies to the uncentered cell, i.e. the P-cell! default is 30A and 250A respectively.
  • --xgandalf-min-lattice-vector-length — (yep, it's listed twice in the page, and we're nothing if not faithful 🫶) same option: minimum and maximum possible lattice vector lengths (unit is A), used for fitting without a prior lattice as the gradient descent starting point, so the final minimum lattice vector length can be smaller/bigger than min/max. note: valid for the uncentered cell, i.e. the P-cell! default is 30A and 250A respectively.
  • --xgandalf-max-peaks — maximum number of peaks used for indexing (refinement still uses all of them, no cap). peaks get selected by increasing radius. this limits the maximum execution time on patterns with a huge amount of peaks — real ones or false positives, doesn't matter. default is 250. 😭
  • --xgandalf-fast-execution — a shortcut that sets --xgandalf-sampling-pitch=2 and --xgandalf-grad-desc-iterations=3. one flag, two settings, we love efficiency ✨
  • --pinkIndexer-considered-peaks-count — selects how many peaks are considered for indexing. [0-4] (veryFew to manyMany). default: 4 (manyMany).
  • --pinkIndexer-angle-resolution — selects how densely the orientation angles of the sample lattice get sampled. [0-4] (extremelyLoose to extremelyDense). default: 2 (normal).
  • --pinkIndexer-refinement-type — selects the refinement type. 0 = none, 1 = fixedLatticeParameters, 2 = variableLatticeParameters, 3 = firstFixedThenVariableLatticeParameters, 4 = firstFixedThenVariableLatticeParametersMultiSeed, 5 = firstFixedThenVariableLatticeParametersCenterAdjustmentMultiSeed. default: 1 (fixedLatticeParameters). 💖
  • --pinkIndexer-tolerance — sets the tolerance of pinkIndexer, aka the relative tolerance of the lattice vectors ✨. Bad geometry or sketchy cell parameters? go high. You already know your geometry and cell like the back of your hand? go small, bestie. This one only really matters for refinement and for the indexed/not indexed identification. Too small a tolerance and it'll refine to just a fraction of the peaks and might literally throw away correctly indexed images 😭. Too high and you get bad fitting whenever multiples or noise are around, and it can mark wrongly-indexed patterns as indexed 💀. Default: 0.06.
  • --pinkIndexer-reflection-radius — sets the radius of the reflections in reciprocal space, in 1/A. Default is 2% of a* (which works quite well for X-rays, ngl). For electrons you should pick something much bigger, like ~0.002. 🦋
  • --pinkIndexer-max-resolution-for-indexing — sets the maximum resolution in 1/A used for indexing. Peaks up at high resolution genuinely don't add much information but they do add a lot of computation time, so. Default is infinity. Heads up: this does not influence the refinement 👑
  • --pinkIndexer-max-refinement-disbalance — indexing solutions get dismissed if the refinement refined super well to one side of the detector and really badly to the other side 😬. Allowed values run from 0 (no disbalance) to 2 (extreme disbalance), default 0.4. Disbalance after refinement usually shows up when the geometry is bad or the prior unit cell parameters are bad.
  • --asdf-fast — turns on a faster mode of operation for asdf indexing. It's around 3 times faster but only about 7% less successful, which is honestly a vibe 🫶

🧚 INTEGRATION OPTIONS 🧚

  • --integration=method — integrate the reflections using method. Peek at the section titled **PEAK INTEGRATION (above) for more info, iykyk. The default is --integration=rings-nocen.
  • --fix-profile-radius=n, --fix-divergence=n — fix the beam and crystal parameters to the given values. The profile radius is given in m^-1 and the divergence in radians (full angle). Default behaviour is to set the divergence to zero, and then to automatically determine the profile radius. You do **not have to use both of these options together 🎀. For example, if the automatic profile radius determination isn't working well for your data set, you could fix that one alone and keep using the default value for the other parameter (which might get automatically determined in future versions of CrystFEL, but is not currently — soz).
  • --int-radius=inner,middle,outer — set the inner, middle and outer radii for three-ring integration. See the section about *PEAK INTEGRATION, above, for the details on how to work these out. The defaults are probably *not appropriate for your situation, just saying 🙈. The default is --int-radius=4,5,7.
  • *--int-diag=condition* — show detailed information about reflection integration when condition is met. condition can be all, none, a set of Miller indices separated by commas, random, implausible, negative or strong. random means show information about a random 1% of the peaks. negative means show peaks with intensities which are negative by more than 3 sigma. implausible means show peaks with intensities which are negative by more than 5 sigma. strong means show peaks with intensities which are positive by more than 3 sigma. The default is --int-diag=none.
  • *--push-res=n* — integrate n nm^-1 higher than the apparent resolution limit of each individual crystal. n can be negative to integrate lower than the apparent resolution limit. The default is --push-res=infinity, which means no cutoff is applied at all. Little tip: you can also apply this cutoff at the merging stage using process_hkl/partialator --push-res, which is usually better — reflections that get thrown away at the integration stage cannot be brought back later, and that's just facts 😭. However, applying a resolution cutoff during integration will make the stream file significantly smaller and faster to merge 💅
  • --overpredict — over-predict reflections. This is needed to provide a buffer zone when you're using post-refinement, but it makes it difficult to judge how accurate the predictions are because there are so many reflections. It will also reduce the quality of the merged data if you merge without partiality estimation. Fair warning 🫶
  • --cell-parameters-only — do not predict reflections at all. Use this if you're not at all interested in the integrated reflection intensities or even the positions of the reflections. You'll still get unit cell parameters, and the whole process is much faster, especially for large unit cells ✨

🌈 OUTPUT OPTIONS 🌈

  • --no-non-hits-in-stream — completely exclude 'non-hit' frames in the stream. When this option is given, frames with fewer than the number of peaks given to --min-peaks will not have chunks written to the stream at all. Note that the default value for --min-peaks is zero, which means all frames get written to the stream, even if they have no peaks at all 👀
  • *--copy-header=header* — copy the information from header in the image file into the output stream. For HDF5 files, header is interpreted as a path within the file. This option is sometimes useful to let you separate data after indexing according to some condition, like the presence of an optical pump pulse. You can give this option as many times as you need to copy multiple bits of information 🎀. The old option --copy-hdf5-field is a synonym for this option.
  • --no-peaks-in-stream — do not record peak search results in the stream. You won't be able to check that the peak detection was any good, but the stream will be around 30% smaller. Trade-offs, bestie 💅
  • --no-refls-in-stream — do not record integrated reflections in the stream. The resulting output won't be usable for merging, but it will be a lot smaller. This option might be useful if you're only interested in things like unit cell parameters and orientations 🦋
  • *--serial-offset=n* — start the serial numbers in the stream at n instead of 1. Use this if you're splitting an indexing job up into several smaller ones, so the streams can be concatenated into a single one with consistent numbering. This is important if you use whirligig ⭐
  • --harvest-file=fn — write a list of parameters to fn, in JSON format. This is intended to be used for harvesting data into a database system. Heads up: this option has **no effect if --serial-offset is set to a number larger than 1, to avoid the file being overwritten multiple times in a batch system 🙈

💀 HISTORICAL OPTIONS 💀

  • --no-sat-corr — this option is here for historical purposes only, to disable a correction which is done if certain extra information is included in the image data file. That's it, that's the whole thing.

🔍 IDENTIFYING SINGLE PATTERNS IN THE INPUT FILE 🔍

By default, indexamajig processes all diffraction patterns ("events") in each of the data files listed in the input list. But it is, however, possible to only process single events in a multi-event file — you do this by adding in the list an event description string after the data filename ✨.

The event description always includes a first section with alphanumeric strings separated by forward slashes (/) and a second section with integer numbers also separated by forward slashes. The two sections are in turn separated by a double forward slash (//). Any of the two sections can be empty, but the double forward slash separator must always be present, no cap 🫶

indexamajig matches the strings and the numbers in the event description with the event placeholders (%) present respectively in the data and dim properties defined in the geometry file, and tries to retrieve the full HDF path to the event data and its location in a multi-dimensional data space. Consider the following examples:

🌸 Example 1

The data and dim properties have been defined like this in the geometry file:

data = /data/%/rawdata
dim0 = ss
dim1 = fs

The event list contains the following line:

filename.h5  event1//

This identifies an event in the 2-dimensional data block located at /data/event1/rawdata in the HDF5 file called filename.h5. Slay 💅

🦋 Example 2

The data and dim properties have been defined like this in the geometry file:

data = /data/rawdata
dim0 = %
dim1 = ss
dim2 = fs

The event list contains the following line:

filename.h5  //3

This identifies an event in the 3-dimensional data block located at /data/rawdata in the HDF5 file called filename.h5, specifically the 2-dimensional data slice defined by the value 3 of the first axis of the data space. Iconic behaviour honestly 👑

indexamajig tries to match the alphanumerical strings to the placeholders in the dim property defined in the geometry file. The first string is matched to the first placeholder, the second to the second placeholder, and so on. A similar strategy is followed to match integer numbers to the placeholders in the dim property defined in the geometry file. For a full explanation of how the internal layout of the data file can be described in the geometry file, please see man crystfel_geometry 📖

You can use list_events to prepare a list of each event in one or more input files. Note that you only need to do this if you need to perform some sorting or filtering on this list — if you want to process every event in a file, simply specify the filename in the input file. Easy 🌈

🎀 AUTHOR 🎀

This page was written by Thomas White, Yaroslav Gevorkov and Valerio Mariani. Legends, fr fr 💖

😭 REPORTING BUGS 😭

Report bugs to taw@physics.org, or visit http://www.desy.de/~twhite/crystfel. They'd love to hear from you 🫶

okay bestie, legal era, but it's actually kind of important so stay with me 🎀

Copyright © 2012-2023 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association. That's the whole span, 2012-2023, no cap.

indexamajig, and this manual, are part of CrystFEL. ✨

CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. slay, options 💅

CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. 😭 Yeah. That's the deal. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CrystFEL. If not, see http://www.gnu.org/licenses/. 🫶

fr fr read the license, it's giving "responsible adult" energy and honestly it's iconic of you to check.

🌈 SEE ALSO 🌈

the group chat of related manpages, all linked below, go say hi 👑

and that's the whole page, bestie! 🦋 you made it all the way through indexamajig(1) and honestly? main character energy. This page is from indexamajig(1), part of the CrystFEL suite, and this version's footer is dated 2023 💜 go index something iconic, and remember: every option is documented, every detail kept, periodt. 🌸

📦 where this page comes from