Page 1 of 1

Large index files over 4GB length in DBFs

Posted: Fri Oct 17, 2014 4:32 pm
by Pablo César
2014-10-17 14:55 UTC+0200 Przemyslaw Czerpak (druzus/at/poczta.onet.pl)
* include/hbrddcdx.h
* src/rdd/dbfcdx/dbfcdx1.c
+ added support for large index files over 4GB length.
These are slightly modified CDX indexes which stores index page numbers
instead of index page offsets inside index file. This trick increase
maximum index files size from 2^32 (4GB) to 2^41 (2TB). This index
format is enabled automatically when DB_DBFLOCK_HB64 is used. This is
the same behavior as in DBFNTX and DBFNSX for which I added support
for large indexes (up to 4TB) few years ago.
Warning: new CDX indexes are not backward compatible and cannot be
read by other systems or older [x]Harbour versions.
If you try to open new indexes using older [x]Harbour RDDs
then RTE "DBFCDX/1012 Corruption detected" is generated.
When current Harbour *DBFCDX/SIXCDX RDD open index file
then it automatically recognize type of index file so it
will work correctly with both versions without any problem.
In short words: People using DB_DBFLOCK_HB64 should remember
that after reindexing with new Harbour applications old ones
cannot read new CDX indexes.
; In next step I plan to add support for user defined page size in CDX
index files.

* doc/xhb-diff.txt
* added information about extended CDX format to section "NATIVE RDDs"

* src/rdd/dbfcdx/dbfcdx1.c
* src/rdd/dbfnsx/dbfnsx1.c
* src/rdd/dbfntx/dbfntx1.c
* disable record readahead buffer used during indexing when only
one record can be stored inside
! generate RTE when data cannot be read into record readahead buffer
during indexing

best regards
Przemek

Re: Large index files over 4GB length in DBFs

Posted: Sat Oct 18, 2014 7:47 am
by dhaine_adp
Thanks for the info, Pablo.

Wow, DBFCDX/SIXCDX RDD becomes a real monster. I'm a bit skeptic in implementing that changes right away but the old CDX has to go away.

-Danny

Re: Large index files over 4GB length in DBFs

Posted: Sat Oct 18, 2014 8:15 am
by mol
Dbfcdx still alive!

Re: Large index files over 4GB length in DBFs

Posted: Sat Oct 18, 2014 11:14 am
by esgici
DBFNTX has already (for a longtime) this (4TB) limit for NTX files.

Which means DBFNTX is already (from beginning) is a monster :lol:

Re: Large index files over 4GB length in DBFs

Posted: Tue Oct 28, 2014 3:01 pm
by Agil Abdullah
Hi Pablo,

4GB is large enough. But, People outhere still questioning the integrity of DBF in relation to "corrupted index". This happends when we encounter electricity failure while indexing process underway, for example.

What your answer in this matter?

Salam Hangat dari Jakarta

Large index files over 4GB length in DBFs

Posted: Tue Oct 28, 2014 4:48 pm
by Pablo César
IMHO DBFs is for small to medium data store. For big dbf I do not recommend it.

In my last Clipper app I have used a timer for re-index NTX files.

When a crash happend in DBF, the only care to be taken is:

- continuosly backup
- repair structure dbf
- restore data from backup

There is no garanties for data preservation in some cases.

What I have also used, is to have at least one dbf for each client/registered to store private account. I.E. In one Video store, each client has a exclusive dbf. This Become faster use and save many problems when a crash or corruption happen in one main dbf, for example.