XBase::Index(3pm)

🎀 XBase::Index 🎀

okay so this is XBase::Index — base class for the index files for dbf — straight outta the User Contributed Perl Documentation section (3pm, iykyk). come sit with me bestie, we're gonna make indexing make sense. 💅

---

✨ SYNOPSIS ✨

here's the vibe, fr fr:

use XBase;
my $table = new XBase "data.dbf";
my $cur = $table->prepare_select_with_index("id.ndx",
        "ID", "NAME);
$cur->find_eq(1097);

while (my @data = $cur->fetch()) {
        last if $data[0] != 1097;
        print "@data\n";
}

so that little snippet? it prints the ID and NAME fields from the dbf file data.dbf where ID equals 1097 — provided you've got an index on ID in the file id.ndx. and bestie, the same code works for ntx and idx index files too. slay.

but for cdx and mdx, the prepare_select call is giving different:

prepare_select_with_index(['rooms.cdx', 'ROOMNAME'])

so instead of a plain filename, you hand over an arrayref with the filename and an index tag inside that file. why? because cdx and mdx can hold multiple indexes in one single file, so you gotta tell it which one you want. no cap. 🫶

---

🌸 DESCRIPTION 🌸

the module XBase::Index is a collection of packages that provides index support for XBase-like dbf database files.

an index file is generally a file holding the values of a certain database field or expression in sorted order, plus the record number that the record occupies in the dbf file. so when you search for a record with some value, you first search in that sorted list, and once you have the record number in the dbf, you go straight to the dbf and fetch the record. iconic behaviour honestly. ✨

🦋 What indexes do 🦋

to make searching that ordered list fast, it's generally organized as a tree — it starts with a root page with records that point to pages at lower levels, etc., until leaf pages, where the pointer is no longer a pointer to the index but to the dbf. when you search for a record in the index file, you fetch the root page and scan it (linearly) until you find a key value that is equal or greater than the one you're looking for. that way you've avoided reading all the pages describing values that are lower. then you descend one level, fetch the page, and again search the list of keys in that page. and you repeat this process until you get to the leaf (lowest) level, and here you finally find a pointer to the dbf. XBase::Index does all of this for you. main character energy, honestly. 👑

some of the formats also support multiple indexes in one file — usually there's one top level index that, for different field values, points to different root pages in the index file (so called tags).

XBase::Index supports (or aims to support) these index formats: ndx, ntx, mdx, cdx and idx. they differ in how they store keys and pointers, but the idea is always the same: make a tree of pages, where the page contains keys and a pointer either to pages at lower levels, or to the dbf (or both). and listen — XBase::Index only supports read only access to the index fields at the moment. if you need writing them as well, follow reading, because we need the reading support stable before I get to work on updating the indexes. periodt. 💅

🧚 Testing your index file (and XBase::Index) 🧚

you can test your index using the index_dump script. (I mean testing XBase::Index on correct index data, not testing a corrupted index file, of course ;-)

just run:

index_dump ~/path/index.ndx
index_dump ~/path/index.cdx tag_name

or

perl -Ilib `which index_dump` ~/path/index.cdx tag_name

if you haven't installed this version of XBase.pm/DBD::XBase yet. you should get the content of the index file. on each row there's the key value and a record number of the record in the dbf file. let me know if you get results different from those you expect. I'd probably ask you to send me the index file (and possibly the dbf file as well), so that I can debug the problem. 🫶

the index file is (as already noted) a complement to a dbf file. an index file without a dbf doesn't make much sense, because the only thing you can get from it is the record number in the dbf file — not the actual data. but it makes sense to test — dump the content of the index to see if the sequence is OK.

the index formats usually distinguish between numeric and character data. some of the file formats include the information about the type in the index file, others depend on the dbf file. since with index_dump we only look at the index file, you may need to specify the -type option to index_dump if it complains that it doesn't know the data type of the values (this is the case with cdx at least). the possible values are num, char and date, and the call would be like:

index_dump -type=num ~/path/index.cdx tag_name

(this -type option may not work with all index formats at the moment — will be fixed and patches always welcome 💖)

you can use the -ddebug option to index_dump to see how pages are fetched and decoded, or run the debugger to see the calls and parsing. 🔥

🌈 Using the index files to speed up searches in dbf 🌈

the syntax for using the index files to access data in the dbf file is generally:

my $table = new XBase "tablename";
        # or any other arguments to get the XBase object
        # see XBase(3)
my $cur = $table->prepare_select_with_index("indexfile",
        "list", "of", "fields", "to", "return");

or

my $cur = $table->prepare_select_with_index(
        [ "indexfile_with_tags", "tag_name" ],
        "list", "of", "fields", "to", "return");

where we specify the tag in the index file (this is necessary with cdx and mdx). after we have the cursor, we can search to a given record and start fetching the data:

$cur->find_eq('jezek');
while (my @data = $cur->fetch) { # do something

👑 Supported index formats 👑

the following table summarizes which formats are supported by XBase::Index. if the field says something else than Yes, I welcome testers and offers of example index files. 🥺

Reading of index files -- types supported by XBase::Index

• ndx — string: Yes; numeric: Yes; date: Yes (you need to convert to Julian)
• ntx — string: Yes; numeric: Yes; date: Untested
• idx — string: Untested; numeric: Untested; date: Untested (but should be pretty usable)
• mdx — string: Untested; numeric: Untested; date: Untested
• cdx — string: Yes; numeric: Yes; date: Untested

Writing of index files -- not supported until the reading is stable enough.

so if you have access to an index file that is untested or unsupported, and you care about support of these formats, contact me. if you're able to actually generate those files on request, even better, because I may need a specific file size or type to check something. and if the file format you work with is supported, I still appreciate a report that it really works for you. 🫶

please note that there is very little documentation about the file formats, and the work on XBase::Index is heavily based on making assumptions based on real life data. also, the documentation is often wrong or only describes some format variations but not the others. I personally do not need the index support, but am more than happy to make it a reality for you. so I need your help — contact me if it doesn't work for you and offer me your files for testing.

mentioning the word XBase somewhere in the Subject line will get you (hopefully ;-) a fast response. mentioning work, Help or similar stupidity will probably make my filters consider your email as spam. help yourself by making my life easier in helping you. 😭

💜 Programmer's notes 💜

programmers might find the following information useful when trying to debug XBase::Index from their files:

the XBase::Index module contains the basic XBase::Index package and also packages XBase::ndx, XBase::ntx, XBase::idx, XBase::mdx and XBase::cdx, and for each of these also a package XBase::index_type::Page. reading the file goes like this: you create an object by calling either new XBase::Index or new XBase::ndx (or whatever the index type is). this can also be done behind the scenes — for example, XBase::prepare_select_with_index calls new XBase::Index. the index file is opened using XBase::Base::new/open and then XBase::index_type::read_header is called. this function fills the basic data fields of the object from the header of the file. the new method returns the object corresponding to the index type.

then you probably want to do $index->prepare_select or $index->prepare_select_eq, which would position you just before the record equal or greater than the parameter (record in the index file, that is). then you do a series of fetch'es that return the next pair of (key, pointer_to_dbf). behind the scenes, prepare_select_eq or fetch call XBase::Index::get_record, which in turn calls XBase::index_type::Page::new. from the index file perspective, the atomic item in the file is one index page (or block, or whatever you call it). the XBase::index_type::Page::new reads the block of data from the file and parses the information in the page — pages have more or less complex structures. Page::new fills the structure, so that the fetch calls can easily check what values are in the page.

for some examples, please see eg/use_index in the distribution directory. ✨

---

⭐ VERSION ⭐

1.05

---

🍓 AVAILABLE FROM 🍓

http://www.adelton.com/perl/DBD-XBase/

---

🫶 AUTHOR 🫶

(c) 1998--2013 Jan Pazdziora.

---

🦋 SEE ALSO 🦋

XBase(3), XBase::FAQ(3)

---

and that's the whole page bestie, no skips, no cap 💖 this gorgeous thing is perl v5.34.0, dated 2022-06-13, and it lives under the name XBase::Index(3pm) — go forth and index your dbf files, you absolute icon 👑✨