crystfel_geometry(5)

📦 version 0.13.0, in nixos 26.05, nixpkgs unstable

💅 INTRODUCTION 💅

Okay bestie, welcome to crystfel_geometry(5), the File Formats Manual page for CrystFEL geometry files. A CrystFEL “geometry file”, usually named something.geom, is used by programs in the CrystFEL suite to get information about:

  • The physical position of the detector, including all sub-detectors if applicable.
  • The layout of data in the file, especially if a “container format” such as HDF5 is used.
  • The values of parameters such as the incident radiation wavelength, or a description of where to get these values in the data files.

🌈 COORDINATE SYSTEM 🌈

It’s important to distinguish between the data coordinate system and the laboratory coordinate system. No cap.

The data coordinate system consists of the fast scan and slow scan directions. Fast scan refers to the direction whose coordinate changes most quickly as the bytes in the data file are moved through. Slow scan refers to the other dimension of the 2D data array. Arrays in the data file can have more than two dimensions — see the DATA DIMENSIONS section below.

CrystFEL’s laboratory coordinate system is defined as follows:

  • +z is the beam direction, and points along the beam (i.e. away from the source).
  • +y points towards the zenith (ceiling).
  • +x completes the right-handed coordinate system.

The CrystFEL GUI shows +x horizontally (left to right), and +y vertically (bottom to top). This means that the GUI shows images from the “into the beam” perspective. Iconic. 💖

🎀 PANELS AND GEOMETRY FILE SYNTAX 🎀

CrystFEL’s representation of a detector is broken down into one or more panels, each of which has its own position, size and various other parameters.

Lines in a CrystFEL geometry file have the following general form:

parameter = value

Many parameters, however, are specified for a specific panel, as follows:

panel_name/parameter = value

Most parameters can theoretically be specified on a per-panel level, but in practice are the same for all panels. For example, the pixel size is usually the same for all panels. In this case, specify the parameter once, without a panel name, at the top of the geometry file. The value will then be used for all panels which are first mentioned later in the file. Efficient queen behavior. ✨

The panel names can be anything of your choosing, except that the names must not start with bad, group or rigid_group. These are reserved for other purposes (see below).

You can also add comments, for example:

; This is a comment
mypanel/min_fs = 34    ; This is also a comment

🦋 SPECIFYING THE WAVELENGTH AND BEAM PARAMETERS 🦋

To specify the incident beam wavelength, you need to use one of the following forms (only one):

  • wavelength = nnn [m|A] — Specifies a wavelength directly, in meters or Angstroms according to the suffix.
  • photon_energy = nnn [eV|keV] — Specifies the energy per photon of electromagnetic radiation (e.g. X-rays). If no units suffix is given, eV (electron volts) will be assumed.
  • electron_voltage = nnn [V|kV] — Specifies the accelerating voltage in an electron microscope. This should be the accelerating voltage, not the relativistically-corrected energy of the accelerated electrons (the difference is small).

For all of these, a data location in the input files can be given instead of a literal number. For example, with data in HDF5 format:

wavelength = /data/LCLS/wavelength m

If there are multiple frames per input file, the program will do what you expect depending on the type of /data/LCLS/wavelength. For example, a scalar value in the metadata will be applied to all frames, or an array of values can be used to provide a separate wavelength for each frame. It just gets it. 💅

You can also specify the radiation bandwidth, as follows:

  • bandwidth = bw — The bandwidth of the radiation, expressed as a fraction of the wavelength. The bandwidth will be interpreted as the standard deviation of a Gaussian spectrum, and used for calculating reflection positions.

🌸 PHYSICAL PANEL LOCATIONS 🌸

For each panel, the physical location is controlled by the fs, ss, corner_x, corner_y and clen parameters, for example:

q3a15/fs = -0.0010058292x +0.9999995232y
q3a15/ss = -0.9999995232x -0.0010058292y
q3a15/corner_x = 575.475
q3a15/corner_y = -221.866
q3a15/coffset = 0.01
  • fs, ss — The vectors in the lab coordinate system of the fast and slow scan directions of the panel data, measured in pixels. Inclusion of a component in the z direction means that the panel is not perpendicular to the X-ray beam.
  • corner_x, corner_y — The position in x and y (lab coordinate system) of the corner of this panel. The corner is defined as the first point in the panel to appear in the image data. The units are pixel widths of the current panel. This should be the location of the very corner of the panel (not the center of the first pixel).
  • coffset — The offset of the panel along the z-direction from the position given by clen, in metres.

The overall detector position in the z-direction is given by clen, which can only be specified once in the geometry file (not for each panel):

  • clen = nnn [mm|m] — The overall z-position (“camera length”) for the detector, or a data file location, and units (m or mm). If no units are given, m will be assumed if the value is given as a literal number, or mm if it’s a data file header location. This discrepancy is for historical reasons, and you should always specify the units. Like when specifying the wavelength (see above), CrystFEL should do what you expect with multi-frame data files.

Getting the clen value from the image headers gives the illusion of avoiding the need to create a new geometry file every time you change the detector position. However, in practice this doesn’t work very well because the detector movement direction is usually not exactly parallel to the beam axis. That means that the beam center position varies with the camera length, and you would have to prepare a new geometry file for each position anyway. For the best results, simply make sure that the experimental geometry stays as static as possible. No cap, static geometry is the vibe. 🫶

🧚 Per-frame beam center position 🧚

You can specify an overall detector shift. The most common use of this is for serial scanning electron diffraction experiments, where the beam center moves from frame to frame. You should avoid using this feature, especially with a metadata location, because it limits CrystFEL’s ability to pre-calculate certain data structures. This feature is likely to be removed in future versions. Not me warning you, but like… warning you. 😭

  • detector_shift_[x,y] = nnn [m|mm] — These specify that the entire detector should be shifted by this amount in the x and y directions. The units should be specified as m or mm. If units are not specified, the value will be taken as metres. nnn can be a file metadata location (e.g. an HDF5 path).

🍓 PANEL DATA LOCATIONS 🍓

For each panel, you have to specify where to find the data in the file. Don’t forget that many of these parameters will be the same for each panel, so you can set them once at the top (see the section Panels and geometry file syntax).

  • data = /location/of/data — The location in the data file of the array that contains the panel’s data. The interpretation of the value depends on the file type. If you’re using HDF5 files, it will be a path such as /data/run_1/imagedata. The default value is /data/data.
  • min_fs, max_fs, min_ss, max_ss = nnn — The range of pixels in the data block that correspond to this panel. Often, multiple panels are grouped together into one “slab”. The pixel ranges are in the data coordinate system, and are specified inclusively.

🎀 Multiple frames per file 🎀

The best performance is achieved when each file on disk contains a large number of images, rather than just one. In this case, you have to additionally specify the data layout to CrystFEL.

Consider a file format where each frame has its own data array under a separate name, for example an HDF5 file with the following layout:

/data/run_1/image_1/data
/data/run_1/image_2/data
/data/run_1/image_3/data
/data/run_1/image_4/data
/data/run_1/image_5/data
...

In this case, you can use % as a placeholder in the data location. For example:

data = /data/run_1/%/data

For HDF5 files, the % must be a whole name at a certain hierarchy level, i.e. the following is not allowed:

data = /data/run_1/image_%/data  ;; This won't work

Next, consider a file format where the frames and detector panels are grouped into a four-dimensional array. The first two dimensions are the image data axes, the third dimension is the panel number, and the fourth dimension is the frame number. In this case, you can specify the data location as follows:

data = /data/run_1/image4Darray
dim0 = %
dim2 = ss
dim3 = fs
min_fs = 0
max_fs = 255
min_ss = 0
max_ss = 255

panel1/dim1 = 0
panel1/corner_x = ....
...

panel2/dim1 = 1
panel2/corner_x = ....
...

panel3/dim1 = 2
panel3/corner_x = ....
...

dim values can be a literal number, a placeholder (%), or fs or ss. Note that, in this example, the common parameter values have been placed at the top, avoiding some repetition. Main character energy: fewer repeated lines. 👑

CrystFEL assumes that the data block defined by the 'data' property has a dimensionality corresponding to the axis with the highest value of n defined by the 'dim' property. That is, if the geometry file specifies dim0, dim1 and dim2, then the data block is expected to be three-dimensional. The size of the data block along each of those axes comes from the image metadata (e.g. the array sizes in the HDF5 file).

The lowest number of n corresponds to the most slowly-changing array index as the data block is traversed. The default values are dim0=ss and dim1=fs. The value of n corresponding to fs must not be lower than the value assigned to ss, i.e. “fast scan is always fast scan”. Periodt. 💅

🌸 PEAK LISTS 🌸

Okay so bestie, you can literally drop peak lists right into the data file. If your data already went through a “hit finding” thing, there was probably a peak search in there somewhere, and reusing those results instead of making CrystFEL do a whole new peak search is honestly the move. 💅

So you need to tell it where the peak list lives in the data file, and what format it’s in.

  • peak_list = loc — where the peak list is hanging out inside your data files.
  • peak_list_type = type — the layout of the peak list. Allowed values are cxi, list3 and auto.

The possible list types are giving:

  • list3 — the peak list is a 2D array where the first dimension equals the number of peaks and the second dimension is exactly three. First two columns = fast scan and slow scan coordinates, third column = intensities. This is the correct option for “single-frame” HDF5 files as written by older versions of Cheetah. ✨
  • cxi — the peak list is an HDF5 group with four separate HDF5 datasets: nPeaks, peakXPosRaw, peakYPosRaw and peakTotalIntensity. See the specification for the CXI file format at http://www.cxidb.org/ for more details. This is the correct option for “multi-event” HDF5 files as output by recent versions of Cheetah. 🫶
  • auto — CrystFEL will decide between the above options based on the file extension.

🚨 Important note about coordinate conventions 🚨

Okay this part is lowkey important, don’t skip it. CrystFEL considers all peak locations to be distances from the corner of the detector panel, in pixel units, consistent with its description of detector geometry (see the section about corner_x above). But the software which generates the HDF5 or CXI files, including Cheetah, may instead consider the peak locations to be pixel indices in the data array. In the former case, a peak position (0,0) corresponds to the very corner of the detector panel. In the latter case, position (0,0) corresponds to the center of the first pixel, and the very corner would be (-0.5,-0.5).

To compensate for this discrepancy, CrystFEL will, by default, add 0.5 to all peak coordinates. See the indexamajig option --no-half-pixel-shift if this isn’t what you want. 💀

🎀 PIXEL SIZE 🎀

You will need to specify the size of the pixels, of course. Use the following:

  • res = pixelsPerMeter — the resolution, in pixels per metre, i.e. one divided by the pixel size in metres.

This gives the scale factor between the length of the fs,ss vectors and physical space. If the fs and ss vectors have different magnitudes, the pixels will not be square. This is allowed, but comes with a possibility of strange problems, because many algorithms assume square pixels. 😭

💜 DETECTOR GAIN 💜

CrystFEL needs to know the gain of the detector, in order to determine how many photons correspond to a particular signal level and hence calculate error estimates on the intensity values. These gain values are not used to correct the pixel values for different gains among the panels. Ngl, important distinction.

Use one of the following:

  • adu_per_photon — the number of detector intensity units which will arise from one quantum of intensity (one X-ray photon, or one electron in an electron microscope).
  • adu_per_eV — the number of detector intensity units which will arise from a 1 eV photon of electromagnetic radiation. This will be scaled by the photon energy (see photon_energy) to calculate the intensity per photon at the wavelength used by the experiment. This option should only be used for electromagnetic radiation.

🔥 DETECTOR SATURATION 🔥

You can specify the saturation value in the geometry file, which will allow indexamajig to avoid integrating saturated reflections. However, usually it’s best to include all reflections at this stage, and exclude the saturated reflections at the merging stage (see process_hkl and partialator options --max-adu). 🫶

  • max_adu — the saturation value for the panel. A warning will be displayed if you use this option, because it’s better to exclude saturated reflections at the merging stage.

Some combinations of detectors and processing methods result in the saturation level varying pixel-to-pixel. For this case, you can provide a per-pixel map of saturation values. Note that both the map values and the max_adu values will both be honoured. ✨

  • saturation_map — this specifies the location of the per-pixel saturation map in the data file.
  • saturation_map_file — specifies that the saturation map should come from the file named here, instead of the file being processed. This can be an absolute filename or relative to the working directory.

🌈 BAD REGIONS 🌈

“Bad region” refers to any set of pixels that should be completely ignored by CrystFEL. There are multiple ways to mark pixels as bad. Let’s go through them, bestie.

🌸 Marking a whole panel 🌸

To flag all pixels in one panel as bad, simply set the no_index parameter:

  • no_index — if set to true or any numerical value other than 0, indicates that the panel should be ignored. The slightly misleading name is for historical reasons.

🦋 Marking pixels at the panel edges 🦋

With many detectors, the pixels at the edge of the detector panels behave differently and should be masked out.

  • mask_edge_pixels = n — mark a border of n pixels around the edge of the panel as bad.

⭐ Marking pixels according to value ⭐

Many data files contain information about bad pixels encoded in the pixel values, for example a value of 65535 often indicates a bad pixel. 👀

  • flag_lessthan = n — mark pixels as bad if their value is less than n.
  • flag_morethan = n — mark pixels as bad if their value is more than n.
  • flag_equal = n — mark pixels as bad if their value exactly n.

Note carefully that the inequalities are strict, not inclusive: “less than”, not “less than or equal to”. Fr fr, do not forget that.

Note also that flag_equal will be difficult to use for data in floating-point format. With floating-point data, you should use flag_lessthan and flag_morethan. 💅

🍓 Marking pixels in rectangles 🍓

You can specify a range of pixels to ignore in the data coordinate system or the laboratory coordinate system.

To mask pixels in the data coordinate system, use the following syntax:

badregionB/min_fs = 128
badregionB/max_fs = 160
badregionB/min_ss = 256
badregionB/max_ss = 512
badregionB/panel = q0a1

A bad region is distinguished from a panel because it starts with bad. Apart from that, the region can use any name of your choice. ✨

The pixel ranges are specified inclusively. The panel name has to be specified, because the pixel range alone might not be unique (see section Multiple frames per file). Bad regions specified in this way therefore cannot stretch across multiple panels.

To mask pixels in the laboratory coordinate system, use the following syntax:

badregionA/min_x = -20.0
badregionA/max_x = +20.0
badregionA/min_y = -100.0
badregionA/max_y = +100.0

In this case, the panel name is not required, and the bad region can span multiple panels. However, bad regions specified in laboratory coordinates take longer to process (when loading images) than regions specified in fs/ss (image data) coordinates. You should therefore use fs/ss coordinates unless the convenience of x/y coordinates outweighs the speed reduction. 🫶

🧚 Providing a separate bad pixel mask 🧚

You can provide an array, separate to the image data array, containing information about the bad pixels. Up to 8 such masks can be provided for each detector panel. Specify the mask location using the following directives, where you should substitute N for a number between 0 and 7 inclusive:

  • maskN_data = location — the location (inside the image data file) of the mask array. Placeholders ('%') in the location will be substituted with the same values as used for the placeholders in the image data, although there may be fewer of them for the masks than for the image data.
  • maskN_file = filename — filename to use for the mask data, if not the same as the image data. The filename may be specified as an absolute filename, or relative to the working directory.
  • maskN_goodbits = bitmask — bit mask for good pixels (see below).
  • maskN_badbits = bitmask — bit mask for bad pixels (see below).

A pixel will be considered bad unless all of the bits which are set in goodbits are set. A pixel will also be considered bad if any of the bits which are set in badbits are set. In pseudocode, where & is a bitwise “and”, the algorithm is:

if (mask_value & mask_goodbits) != mask_goodbits:
    mark_pixel_as_bad

if (mask_value & mask_badbits) != 0:
    mark_pixel_as_bad

Example:

mask2_data = /data/bad_pixel_map
mask2_file = /home/myself/mybadpixels.h5
mask2_goodbits = 0x00
mask2_badbits = 0xff

There are some older mask directives which are still understood by this version of CrystFEL. They are synonyms of the new directives as follows:

mask       ----->   mask0_data
mask_file  ----->   mask0_file
mask_good  ----->   mask0_goodbits
mask_bad   ----->   mask0_badbits

💖 DETECTOR HIERARCHY 💖

okay so this is the part where the panels get to form friend groups ✨ Detector panels can be combined into groups — and certain operations, especially detector geometry refinement, use these groups to conveniently move panels around together. So think of it as panels holding hands 🫶

Groups are specified like this:

group_abc = panel1,panel2
group_def = panel3,panel4

This creates a group called abc, containing panels panel1 and panel2, and a group def containing panel3 and panel4. Slay.

Groups can themselves be combined into higher-level groups, like:

group_all = abc,def

This defines a group called all which contains both of the groups created above. It's giving nested friend circles, honestly 👑

The highest-level group should always be called all. No exceptions, bestie, that's just the vibe.

All members of a group need to be defined before defining the group. Which means: the group definitions should come after the panel definitions, and the groups should be defined from the bottom to top level of the hierarchy, with the all group coming last. Not me making you read that twice — it's just important 😭

As a convenience, CrystFEL 0.11.1 and later defer processing group lines until the rest of the geometry file has been read. That means you can put your group lines near the top of the file if you want, but the groups must still be defined from bottom to top level. And to maintain compatibility with CrystFEL 0.11.0, tools that write geometry files (e.g. align_detector) from later CrystFEL versions will put the group directives at the end of the file. Lowkey thoughtful of them 💅

If you don't define any groups, CrystFEL will automatically create the all group for you, containing all panels in a flat hierarchy. Which is great news — basic geometry refinement (level zero, see align_detector) works without any extra work from you. Free win, we love to see it 🎀

Heads up: the group system replaces the rigid_group system used in older versions of CrystFEL. If the geometry file contains any rigid_group lines, they will be ignored in this version. So don't be surprised when they do absolutely nothing, bestie 💀

🌈 EXAMPLES 🌈

For examples, look in the examples folder, which can be found online at:

https://gitlab.desy.de/thomas.white/crystfel/-/tree/master/doc/examples

Go look, they're right there, literally 💖

✨ AUTHOR ✨

This page was written by Thomas White and Valerio Mariani. Legends, both of them 👑

🦋 REPORTING BUGS 🦋

Report bugs to taw@physics.org, or visit http://www.desy.de/~twhite/crystfel. They want to hear from you, fr fr 🙈

Copyright © 2023 Deutsches Elektronen-Synchrotron DESY, a research centre of the Helmholtz Association.

CrystFEL is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

CrystFEL is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with CrystFEL. If not, see http://www.gnu.org/licenses/.

💜 SEE ALSO 💜

  • crystfel(7)
  • indexamajig(1)
  • adjust_detector(1)
  • align_detector(1)

and that's the whole page, bestie, you made it to the end 🎉 this was crystfel_geometry(5), sourced from the CrystFEL manual and dated as per the original footer — go forth and slay your geometry files 💅✨🦋

📦 where this page comes from