Tag Archives: DBFNTX

Index fundemantals

Posted on by vivaclipper

Question : I just would like to understand a bit what is the difference between DBFNSX and SIXCDX, please. Are not  they the same thing?

Answer : I described it few times in the past.

Lets forget about SIXCDX – it’s slighlt modified DBFCDX not worth to talk about.

DBT, FMT, SPT are different MEMO formats. All of them are supported by Harbour and automatically recognized when DBF file is open. NTX, CDX and NSX are different index formats. They can be used in any combinations of MEMO formats, i.e. DBFCDX perfectly well works with DBT memo files just like with FPT and SMT ones.

In Harbour all core RDDs using above index formats (DBFNSX, DBFCDX, DBFNSX) have nearly the same functionality which cover nearly all index feautres known in xbase world and many of them are unique to [x]Harbour only so they are not supported by other drivers.

With all above RDDs user can use all ord*(), db*(), sx_*(), hsx_*() , … functions, can create multitag indexes (many orders in single file, also for NTX format), autooreder, autoopen, production indexes, etc. so for programmer used RDD should not create any difference. It’s also possible to enable disable some features using RDDI_* interface, i.e. this code change default DBFNTX behavior so it behaves just like DBFCDX and even uses “.cdx” as default file extnesion (of course internally it’s still NTX format with Harbour extenssions – we support CTX format from CLIP)

 // default index extenssion 
 rddInfo( RDDI_ORDBAGEXT, ".cdx", "DBFNTX" ) 
 //support multi tag in single index file 
 rddInfo( RDDI_MULTITAG , .t. , "DBFNTX" ) 
 // structural indexes support 
 rddInfo( RDDI_STRUCTORD, .t. , "DBFNTX" ) 
 // record number is hidden trailing part of key duirng sorted 
 rddInfo( RDDI_SORTRECNO, .t. , "DBFNTX" )

There are only few minor excpetions rather unimportant for most of users unique to given RDDs. The most important are two:

1. I implemented dynamic unique indexes only in DBFCDX It means that ordUnique( ,, .t. ) -> can enable/disable unique mode only in DBFCDX (and SIXCDX)

2. Only in DBFNTX and DBFNSX I implement special mode which allows to use page numbers instead of file offsets in index pages. In this mode indexes are not binary compatible with other languages but their maximal size has been greatly extended and for NTX and NSX files is 2^32 * index_page_size what gives 2^42 for default 1024 pages in this formats – 2^42 is 4TB I haven’t implemented it in DBFCDX so far and for this format maximum index size is still 4GB. Maybe in some spare time I’ll do that and also add support for different page sizes. BTW in ADS .adi indexes are slightly modified CDX files where page numbers are used instead of offsets and index page size can be changed.

Of course there are very serious differences in low level implementation and structures used by these formats.

NTX is simple BTREE without any compression. The operation are extremely fast but indexes are much bigger then in CDX or NSX format so performance is usually storngly reduced by cost of IO operations. Anyhow theoretically having very strong server application with a lot of RAM so all data are accessed from memory not from harddisks this is the best choice.

CDX and NSX compress leaf nodes so total size of index files is much smaller then in NTX format. NSX uses simple BTREE when CDX uses three of the most significant keys. It means that update in CDX files can be more expensive then in NSX format especially if we are adding keys which should be sorted as last (it’s the most common situation, i.e. when we are adding records with current date) because all nodes from leaf to root have to be updated. All keys are repeated in leaf nodes and there are internal bindings for nodes on the same level (BTW some RDDs like SIX3 do not update them correctly for interior nodes) so CDX format is also a little bit redundant. Anyhow all keys are repeated in leafs nodes and we have internal bindings between all leaves so skipping can be a little bit faster, etc.

Page size in CDX is smaller then in NSX so for very long keys, i.e. over 100 bytes NSX format should be much more efficient. It also uses different compression method which should be better for keys having long space substrings inside, i.e. due to concatenation of few longer fields like: FNAME[40] + LNAME[40]

In general this is to big subject to describe it here in few words.

In include/hbrddnsx.h I’ve made some small description of NSX format when I was implementing it.

best regards,
Przemek
Source : https://groups.google.com/d/msg/harbour-devel/9nT9lZmtztk/Q3X-s81UpYYJ

C5DG-7 DBFNTX Driver

Posted on by vivaclipper

Clipper 5.x – Drivers Guide

Chapter 7

DBFNTX Driver Installation and Usage

DBFNTX is the default RDD for Clipper. This new database driver replaces the DBFNTX database driver supplied with earlier versions of Clipper and adds a number of new indexing features. With DBFNTX, you can:

. Create conditional indexes by specifying a FOR condition

. Create indexes using a record scope or WHILE condition, allowing you to INDEX based on the order of another index

. Create both ascending and descending order indexes

. Specify an expression that is evaluated periodically during indexing in order to display an index progress indicator

In This Chapter 

This chapter explains how to install DBFNTX and how to use it in your applications. The following major topics are discussed:

. Overview of the DBFNTX RDD

. New Locking Scheme

. Conditional Indexing

. Installing DBFNTX Driver Files

. Linking the DBFNTX Driver

. Using the DBFNTX Driver

. Compatibility with dBASE III

Overview of the DBFNTX RDD

As an update of the default database driver, DBFNTX is linked into and used automatically by your application unless you compile using the /R option.

New Features

The replaceable driver lets you create and maintain (.ntx) files using features above and beyond those supplied with the previous DBFNTX driver. The new indexing features are supplied in the form of several syntactical additions to the INDEX and REINDEX commands. Specifically you can:

. Specify full record scoping and conditional filtering using the standard ALL, FOR, WHILE, NEXT, REST, and RECORD clauses

. Create an index while another controlling index is still active

. Monitor indexing as each record (or a specified record number interval) is processed using the EVAL and EVERY clauses

. Eliminate separate coding for descending order keys using the DESCENDING clause

Compatibility

Index files (.ntx) created with the original DBFNTX driver are compatible with DBFNTX and can be used in new applications without reindexing. Index files (.ntx) created with this version of DBFNTX will also work with previous Clipper applications provided that you use no FOR, WHILE, <scope>, or DESCENDING clauses.

Important! Indexes produced with DBFNTX using FOR or DESCENDING are incompatible with earlier version (.ntx) files. If you attempt to access them with the original DBFNTX database driver or programs compiled with versions earlier than Clipper 5.2, you will get an unrecoverable runtime error. In Clipper, this generates an “index corrupted” error message, causing the application to terminate.

New Locking Scheme

The DBFNTX database driver implements a new locking scheme to resolve several problems identified in previous versions of Clipper and to prevent potential problems that might arise when running Clipper applications in a network environment. This section discusses these changes and their implications, including compatibility issues.

Lock Time-outs

Problem: Index locking in previous versions of Clipper was handled automatically by the database driver, and had no time-out provision. This created the potential for problems in network environments if a workstation died while holding a lock. If this situation occurred all other workstations waiting for an index lock would appear to freeze while waiting to obtain their lock. This could also happen if a user placed a Clipper application in the background on a multitasking system without sufficient processing time allocated to it. Eventually, most network operation systems would clear a connection that had no activity for a specified period of time. This would free the lock and everything would resume as normal, but frustrated users may have rebooted their machines possibly causing file corruption.

Solution: In Clipper 5.2 the NTX driver will generate a recoverable runtime error if it fails to lock the index after a predetermined number of retries. The default error handler for this system simply returns (.T.) to retry the operation. This emulates the behavior of previous Clipper versions.

Error Handling

Time out handling: The handling of this error is problematic because the lock is issued from various internal index routines. Therefore the only safe recoveries are to retry or quit. Choosing to default from the error or issuing a break will more than likely leave the index in a corrupted state. If either of the options is employed, the application should immediately recreate the index. The preferred way to handle a time out such as this is to alert the user of the situation so they don’t think their machine has hung, and then have the network administrator determine what workstation is causing the problem. When the problem workstation is cleared, the users that have timed out can select retry and continue processing.

NTXERR.PRG: The file NTXERR.PRG contains the source code for the default error handler INIT procedure. This error handler can be modified to allow user-defined error handling for index lock time-outs. Care should be exercised when modifying the error handler as detailed above.

Compatibility: The lock time-out capability when used in conjunction with the default error handler is totally compatibility with previous versions of Clipper. No changes are made to the NTX file structure and no action is required by the developer to activate the time-out functionality.

New Lock Offset

Problem: Index locking, which is transparent to the developer, uses a single-byte semaphore locking system. This semaphore was placed at a virtual offset (beyond the physical end of file) in the index file. In previous versions of Clipper, this offset was located at one billion (1,000,000,000) which was adequate at the time. But many systems today are capable of producing indexes that are large enough to cause the actual data present at the lock offset to become physically locked. This leads to problems when trying to read or write to the data at that offset.

Solution: The solution is to move the offset where locking occurs to a location at a greater offset. We have chosen FFFFFFFF hex, which is the largest offset possible under the DOS operating system. The problem with this solution is that new applications using the index will be locking this new byte while old applications using the same index will lock the old position. Clearly this would cause both applications to fail because each could have a lock on the file at the same time.

To avoid this, the signature of the index (in the index header) is modified to prevent pre-Clipper 5.2 applications from being able to open the index. Clipper 5.2 applications can detect the correct offset to use by the flag in the header and will automatically use the correct one. In Figure 7-1 below, each bit represents a flag:

BIT  7 6 5 4 3 2 1 0
FLAG R R R O P I I C
R Reserved
I Index type - both bits set (NTX)
C Index created with a Condition, condition in header
T Created as a Temporary index
O New Offset for exclusive (semaphore) lock
Figure 7-1: Bit Field for the Signature Byte of a -Clipper 5.2 NTX File

Activation

If Clipper 5.2 automatically modified the signature in the header when it created indexes, programs with automatic reindexing routines would be creating indexes that appeared corrupt to pre-Clipper 5.2 applications. This has an obvious problem with backward compatibility. Therefore, in order to create indexes with the new signature, the developer must link in the module NTXLOCK2.OBJ with the full knowledge that this will create indexes that older applications will not be able to access.

Header Changes

The signature byte of a .NTX file is 6 for an unenhanced NTX index. The inclusion of the NTXLOCK2.OBJ will cause the signature to become 26 hex. (6 hex ORed with 20 hex). See Figure 7-1 for an illustration of all the possible values for the signature byte.

Error Handling

Clipper 5.2 applications will automatically recognize the signature byte of the header, and depending on the signature value, will use the correct index lock location. Applications built with previous versions of Clipper, however, do not have the capability to detect the optional new information in the signature byte. Therefore, when an order application tries to open a file that has been created with the NTXLOCK2.OBJ linked in it will produce a Corruption Detected error.

Compatibility

The new locking location, if used, is not backward compatible with applications compiled with previous versions of Clipper.

Indexes created by applications built with a previous version of Clipper can be used by Clipper 5.2 using the new location and will not be modified unless the index is recreated in application.

Since older applications have no knowledge of the new index locking scheme nor of the significance of the header signature, these applications will assume the index is corrupt and will produce an Index Corrupted error.

Conditional Indexing

Conditional indexes are a feature of the DBFNTX driver. This section discusses this feature of the DBFNTX driver in some detail, giving you specific information about the implementation of conditional indexes. Compatibility issues are also discussed.

Conditional Indexes

Conditional indexes are produced by using a FOR condition in the index creation process. These indexes are made fully maintainable by storing the FOR condition in the index header. This condition is subsequently retrieved and compiled each time the index in opened. During updates, items are added to the index only if they meet the criteria of the condition.

Since older applications do not have the ability to recognize and use the condition stored in the header, they must be prevented from opening the index since they corrupt the index. This is accomplished by modifying the signature of the index (in the index header) preventing pre-Clipper 5.2 applications from being able to open the index. Clipper 5.2 applications can detect the flag in the header and will automatically use the stored FOR condition correctly.

Temporary Indexes

Temporary indexes are produced by using any scoping clause other than the FOR condition in the index creation process. These indexes are not automatically maintainable because the condition is not stored for later use. These indexes can be made maintainable if the condition can be expressed as a FOR condition and is added using the FOR clause. But the main use of temporary indexes is for fast creation of indexes for read- only browses or reports that operate on a subset of the database.

Since older applications would not operate properly with indexes that do not contain all the keys in a given database, they must be prevented from using them. This is accomplished by modifying the index signature to prevent pre-Clipper 5.2 applications from being able to open the index.

Activation

Conditional Indexes

The developer need only specify the FOR condition when creating the index. In doing so he must be fully aware the index will no longer be accessible to pre–Clipper 5.2 applications.

Temporary Indexes

The developer need only specify a scope other than FOR when creating the index. In doing so he must be fully aware the index will no longer be accessible to pre-Clipper 5.2 applications and that the index created is not maintainable.

Header Changes

The signature byte of a .NTX file is 6 for a unenhanced NTX index. If the index is created as a conditional index it will have a signature of 7 hex (6 hex ORed with 1 hex). If the index is created as a temporary index it will have a signature of E hex. (6 hex ORed with 8 hex). See Figure 7-1 for an illustration of all the possible values for the signature byte.

Error Handling

Corruption Detected

Since older applications have no knowledge of the new index features nor how to interpret the additional flags in the header signature, these applications will assume the index is corrupt and will produce an Index Corrupted error.

EOF()

If an index is created with a FOR condition and an attempt is made to update the index with a key that does not match the condition, the update is suppressed and the index is placed at EOF(). This is consistent with the current behavior for indexes created with the unique flag when an update is attempted with a non-unique key.

Also if a navigational action is attempted (SKIP) and the current record is not found in the index, the index will place the record pointer at EOF(). This is true for both conditional and temporary indexes.

Compatibility

Backward Compatibility

If the conditional or temporary indexing features are used the index produced will not be backward compatible with applications compiled with previous versions of Clipper. Indexes that do not use the features, however, will be 100% compatible.

Forward Compatibility

Indexes created by applications built with a previous version of Clipper can be used by Clipper 5.2 and will not be modified unless the index is recreated using either the conditional or temporary index features.

Error Message Produced by Old Applications

Since older applications have no knowledge of the new index locking scheme nor of the significance of the header signature, these applications will assume the index is corrupt and will produce an Index Corrupted error.

Installing DBFNTX Driver Files

DBFNTX is supplied as the file DBFNTX.LIB.

The Clipper installation program installs this driver as the default in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.

Important! Before installing Clipper, you may want to rename the DBFNTX.LIB that currently resides in your \CLIPPER5\LIB directory to DBFNTX.001. The new version, when installed, will overwrite DBFNTX.LIB. If you do not rename or otherwise protect the old version of DBFNTX.LIB, you will lose it.

Linking the DBFNTX Database Driver

Since DBFNTX is the default database driver for Clipper, there are no special instructions for linking. Unless you specify the /R option when you compile, the new driver will be linked into each program automatically if you specify a USE command or DBUSEAREA() function without an explicit request for another database driver. The driver is also linked if you specify an INDEX or REINDEX command with any of the new features.

Using the DBFNTX Database Driver

In applications written for the new DBFNTX driver, you can use the INDEX and REINDEX commands exactly as you have used them in the past. The index files (.ntx) you create and maintain in this way are completely compatible with those created using previous versions of the driver.

Changes to existing code are necessary only if you use the new indexing features. The (.ntx) files you create using the new features will have a slightly different header file and cannot be used by programs linked with a previous version of the driver.

Using (.ntx) and (.ndx) Files Concurrently

You can use (.ntx) and (.ndx) files concurrently in a Clipper program like this:

// (.ntx) file using default DBFNTX driver

USE File1 INDEX File1 NEW

// (.ndx) files using DBFNDX driver

USE File2 VIA "DBFNDX" INDEX File2 NEW

Note, however, that you cannot use (.ntx) and (.ndx) files in the same work area. For example, the following does not work:

USE File1 VIA "DBFNDX" INDEX File1.ntx, File2.ndx

Compatibility with dBASE III PLUS

The default DBFNTX driver makes Clipper programs behave differently than traditional dBASE programs. Some of these differences are discussed below.

Supported Data Types

The DBFNTX database driver supports the following dBASE III PLUS- compatible data types for key expressions:

. Character

. Numeric

. Date

. Logical

Supported Key Expressions

When you create (.ntx) files using the DBFNTX driver, you can use all Clipper or user-defined functions compatible with dBASE III PLUS as well as other functions accepted by the extended Clipper functionality.

Error Handling

The indexing behavior of DBFNTX and DBFNDX in a Clipper application is identical unless otherwise noted. With the default DBFNTX driver, you can handle most errors using BEGIN SEQUENCE…END SEQUENCE as illustrated in the next section.

FIND vs SEEK

In Clipper, you can use the FIND command only to locate keys in indexes where the index key expression is character data type. This differs from dBASE III PLUS where FIND supports character and numeric key values.

Note: In Clipper programs, always use the SEEK command or the DBSEEK() function to search an index for a key value.

The DBFNTX driver lets you recover from data type errors raised during a FIND or SEEK. However, since Error:canDefault, Error:canRetry or Error:canSubstitute are set to false (.F.), you should use BEGIN SEQUENCE…END to handle such SEEK or FIND data type errors. Within the error block for the current operation, issue a BREAK() using the error object that the DBFNTX database driver generates, like this:

bOld := ERRORBLOCK({|oError| BREAK(oError)})
 .
 .
 .
 BEGIN SEQUENCE
     SEEK xVar
 RECOVER USING oError
     // Recovery code END
 .
 .
 .
 ERRORBLOCK(bOld)

There is an extensive discussion of the effective use of the Clipper error system in the Error Handling Strategies chapter of the Programming and Utilities guide.

Sharing Data on a Network

The DBFNTX driver provides file and record locking schemes that are different from dBASE III PLUS schemes. This means that if the same database and index files are open in Clipper and in dBASE III PLUS, Clipper program locks are not visible to dBASE III PLUS and vice versa.

Warning! Database integrity is not guaranteed and index corruption will occur if Clipper and dBASE III PLUS programs attempt to write to a database or index file at the same time. Therefore, concurrent use of the same database (.dbf) and index (.ndx) files by dBASE III PLUS and Clipper programs is strongly discouraged and not supported by Computer Associates.

Summary

In this chapter, you were given an overview of the new features of the default DBFNTX RDD. You learned how to this driver is automatically linked and how to use it in your applications, and were given an overview of the compatiblity issues.

C5DG-4 DBFCDX Driver

Posted on by vivaclipper

Clipper 5.x – Drivers Guide

Chapter 4

DBFCDX Driver Installation and Usage

DBFCDX is the FoxPro 2 compatible RDD for Clipper. As such, it connects to the low-level database management subsystem in the Clipper architecture. When you use the DBFCDX RDD, you add a number of new features including:

. FoxPro 2 file format compatibility

. Compact indexes

. Compound indexes

. Conditional indexes

. Memo files smaller than DBFNTX format

In This Chapter

This chapter explains how to install DBFCDX and how to use it in your applications. The following major topics are discussed:

. Overview of the DBFCDX RDD

. Installing DBFCDX Driver Files

. Linking the DBFCDX Driver

. Using the DBFCDX Driver

Overview of the DBFCDX RDD

The DBFCDX driver lets you create and maintain (.cdx) and (.idx) files with features different from those supplied with the original DBFNTX driver and is compatible with files created under FoxPro 2. The new features are supplied in the form of several syntactical additions to the INDEX and REINDEX commands. Specifically, you can:

. Create indexes smaller than those created with the DBFNTX
driver. The key data is stored in a compressed format that
substantially reduces the size of the index file.

. Create a compound index file that contains multiple indexes
(TAGs), making it possible to open several indexes under one file
handle. A single (.cdx) file may contain up to 99 index keys.

. Create conditional indexes (FOR / WHILE / REST / NEXT).

. Create files with FoxPro 2 file format compatibility.

Compact Indexes

Like FoxPro 2, The DBFCDX driver creates compact indexes. This means that the key data is stored in a compressed format, resulting in a substantial size reduction in the index file. Compact indexes store only the actual data for the index keys. Trailing blanks and duplicate bytes between keys are stored in one or two bytes. This allows considerable space savings in indexes with much empty space and similar keys. Since the amount of compression is dependent on many variables, including the number of unique keys in an index, the exact amount of compression is impossible to predetermine.

Compound Indexes

A compound index is an index file that contains multiple indexes (called tags). Compound indexes (.cdx)’s make several indexes available to your application while only using one file handle. Therefore, you can overcome the Clipper index file limit of 15. A compound index can have as many as 99 tags, but the practical limit is around 50. Once you open a compound index, all the tags in the file are automatically updated as the records are changed.

Once you open a compound index, all the tags contained in the file are automatically updated as the records are changed. A tag in a compound index is essentially identical to an individual index (.idx) and supports all the same features. The first tag (in order of creation) in the compound index is, by default, the controlling index.

Conditional Indexes

The DBFCDX driver can create indexes with a built-in FOR clause. These are conditional indexes in which the condition can be any expression, including a user-defined function. As the database is updated, only records that match the index condition are added to the index, and records that satisfied the condition before, but don’t any longer, are automatically removed.

Expanded control over conditional indexing is supported with the revised INDEX and REINDEX command options as in the new DBFNTX driver.

Installing DBFCDX Driver Files

The DBFCDX driver is supplied as the file, DBFCDX.LIB.

The Clipper installation program installs this driver in the \CLIPPER5\LIB subdirectory on the drive that you specify, so you need not install the driver manually.

Linking the DBFCDX Database Driver

To link the DBFCDX database driver into an application program, you must specify DBFCDX.LIB to the linker in addition to your application object files (.OBJ).

1. To link with .RTLink using positional syntax:

C>RTLINK <appObjectList> ,,,DBFCDX

2. To link with .RTLink using freeformat syntax:

C>RTLINK FI <appObjectList> LIB DBFCDX

Note: These link commands all assume the LIB, OBJ, and PLL environment variables are set to the standard locations. They also assume that the Clipper programs were compiled without the /R option.

Using the DBFCDX Database Driver

To use FoxPro 2 files in a Clipper program:

1. Place REQUEST DBFCDX at the beginning of your application or at the top of the first program file       (.prg) that opens a database file using the DBFCDX driver.

2. Specify the VIA “DBFCDX” clause if you open the database file with the USE command.

    -OR-

3. Specify “DBFCDX” for the <cDriver> argument if you open the database file with the DBUSEAREA()       function.

   -OR-

4. Use ( “DBFCDX” ) to set the default driver to DBFCDX.

    Except in the case of REQUEST, the RDD name must be a literal character string or a variable. In all       cases it is important that the driver name be spelled correctly.

The following program fragments illustrate:

  REQUEST DBFCDX
  .
  .
  .
  USE Customers INDEX Name, Address NEW VIA "DBFCDX"

     -OR-

  REQUEST DBFCDX
  RDDSETDEFAULT( "DBFCDX" ) .
  . 
  .
  USE Customers INDEX Name, Address NEW

Using (.idx) and (.ntx) Files Concurrently

You can use both (.idx) and (.ntx) files concurrently in a Clipper program like this:

// (.ntx) file using default DBFNTX driver
 USE File1 INDEX File1 NEW
// (.idx) files using DBFCDX driver
 USE File2 VIA "DBFCDX" INDEX File2 NEW

Note, however, that you cannot use (.idx) and (.ntx) files in the same work area. For example, the following does not work:

USE File1 VIA "DBFNTX" INDEX File1.ntx, File2.idx

Using (.cdx) and (.idx) Files Concurrently

You may use (.cdx) with (.idx) files concurrently (even in the same work area); however, in most cases it is easier to use a single (.cdx) index for each database file or separate (.idx) files. When using both types of index at the same time, attempting to select an Order based on its Order Number can be confusing and will become difficult to maintain.

File Maintenance under DBFCDX

When an existing tag in a compound index (.cdx) is rebuilt using INDEX ON…TAG… the space used by the original tag is not automatically reclaimed. Instead, the new tag is added to the end of the file, increasing file size.

You can use the REINDEX command to “pack” the index file. REINDEX rebuilds each tag, eliminating any unused space in the file.

If you rebuild your indexes on a regular basis, you should either delete your (.cdx) files before rebuilding the tags or use the REINDEX command to rebuild them instead.

DBFCDX and Memo Files

The DBFCDX driver uses FoxPro compatible memo (.fpt) files to store data for memo fields. These memo files have a default block size of 64 bytes rather than the 512 byte default for (.dbt) files.

DBFCDX memo files can store any type of data. While (.dbt) files use an end of file marker (ASCII 26) at the end of a memo entry, (.fpt) files store the length of the entry. This not only eliminates the problems normally encountered with storing binary data in a memo field but also speeds up memo field access since the data need not be scanned to determine the length.

Tips For Using DBFCDX

1. Make sure index extensions aren’t hard-coded in your application. The default extension for DBFCDX indexes is (.idx), not (.ntx). You can still use (.ntx) as the extension as long as you specify the extension when you create your indexes. The best way to determine index extensions in an application is to call ORDBAGEXT().

For example, if you currently use the following code to determine the existence of an index file:

IF .NOT. FILE("index.ntx")
    INDEX ON field TO index
ENDIF

Change the code to include the INDEXEXT() function, as follows:

IF .NOT. FILE("index"+ORDBAGEXT())
   INDEX ON field TO index
ENDIF

2. If your application uses memo fields, you should convert your (.dbt) files to (.fpt) files.

There are some good reasons for using (.fpt) files. Most important is the smaller block size (64 bytes). Clipper’s (.dbt) files use a fixed block size of 512 bytes which means that every time you store even 1 byte in a memo field Clipper uses 512 bytes to store it. If the data in a memo field grows to 513 bytes, then two blocks are required.

When creating (.fpt) files, the block size is set at 64 bytes to optimize it for your needs. A simple conversion from (.dbt) files to (.fpt) files will generally shrink your memo files by approximately 30%.

3. Add DBFCDX.LIB as a library to your link command or link script.

Summary

In this chapter, you were given an overview of the features and benefits of the DBFCDX RDD. You also learned how to link this driver and how to use it in your applications.

C5DG-2 RDD Architecture

Posted on by vivaclipper

Clipper 5.x – Drivers Guide

Chapter 2

Replaceable Database Driver Architecture

Clipper supports a driver architecture that allows Clipper applications to use Replaceable Database Drivers (RDDs). The RDD system makes Clipper applications data-format independent. Such applications can, therefore, access the data formats of other database systems, including the dBASE IV (.mdx), FoxPro (.cdx), and Paradox (.db) formats on a variety of equipment. This driver architecture can even support database drivers that are not file-based, although all of the drivers supplied with Clipper 5.x are file-based.

The concept of replaceable drivers is not new to this version of Clipper. In previous versions, the use of the default database driver (DBFNTX.LIB) was hidden by the fact that it was automatically linked into your application. In fact, this is still the case. The DBFNTX driver has been replaceable since it was first introduced in version 5.0. Before this version, the DBFNTX driver was the only RDD supplied as part of the system.

In This Chapter

With the introduction of the new RDDs, Clipper provides many new and enhanced commands and functions that access and manipulate databases. These language elements can enable your applications to access data regardless of the RDD under which it is ordered. There are also commands and functions that give you specific information about the RDDs in use.

The Language Implementation section of this chapter includes tables that summarize these new and enhanced language elements. This chapter also covers basic terminology, implementation principals, and general concepts of the Order Management System.

The following major topics are discussed:

. RDD Basics

. Basic Terminology

. The Language Implementation

. Order Management System

RDD Basics

The cornerstone of the replaceable database driver system is the Clipper work area. All Clipper database commands and functions operate in a work area through a database driver that actually performs the access to the stored database information. The layering of the system looks like this:

                      +———————————+

                      | Database Commands and Functions |
                      ----------------------------------|
                      |          RDD Interface          |
                      |---------------------------------|
                      |         Database driver         |
                      |---------------------------------|
                      |           Stored Data           |
                      +---------------------------------+

 In this system, each work area is associated with a single database driver. Each database driver, in turn, is supplied as a separate library file (.LIB) you link into your application programs. Within an application, you specify the name of the database driver when you open or access a database file or table with the USE command or DBUSEAREA() function. If you specify no database driver at the time a file is opened, the default driver is used. You may select which driver will be used as the default driver.

Once you open a database in a work area, the RDD used for that work area is automatically used for all operations on that database (except commands and functions that create a new table). Any command or function that creates a new table (i.e., SORT, CREATE FROM, DBCREATE(), etc.) uses the default RDD. Most of the new commands and functions let you specify a driver other than the default driver.

The normal default database driver, DBFNTX (which supports the traditional (.dbf), (.ntx), and (.dbt) files) is installed into your \CLIPPER5\LIB directory. This driver is linked into each program automatically to provide backwards compatibility.

To use any of the other supplied drivers, either as an additional driver or an alternate driver, you must use the REQUEST command to assure that the driver will be linked in. You must also include the appropriate library on the link line.

All Clipper applications will automatically include code generated by RDDSYS.PRG from the \CLIPPER5\SOURCE\SYS subdirectory. If you wish to automatically load another RDD, you must modify and compile RDDSYS.PRG and link the resulting object file into your application. The content of the default RDDSYS.PRG is shown below. Only the portion in bold should be modified

 
     //  Current RDDSYS.PRG
     #include "rddsys.ch"

     ANNOUNCE RDDSYS                     // This line must not change
     INIT PROCEDURE RddInit
        REQUEST DBFNTX                   // Force link for DBFNTX RDD
        RDDSETDEFAULT( "DBFNTX" )        // Set up DBFNTX as default
                                         // driver

        RETURN

     // eof: rddsys.prg

To change the default to a new automatically-loading driver, modify the bold lines in RDDSYS.PRG to include the name of the new driver. For example:

     //  Revised RDDSYS.PRG
     #include "rddsys.ch"

     ANNOUNCE RDDSYS                     // This line must not change
     INIT PROCEDURE RddInit
        REQUEST DBFCDX                   // Force link for DBFCDX RDD
        RDDSETDEFAULT( "DBFCDX" )        // Set up DBFCDX as default
                                         // driver

        RETURN

     // eof: rddsys.prg

If you change this file, all Clipper applications in which it is linked will automatically include the new RDD.

To use any RDD other than the default, you must explicitly identify it through use of the VIA clause of the USE command.

You need not disable the automatic DBFNTX loading to use other RDDs in your applications, but if your application will not use any DBFNTX functionality, you can save its code overhead by disabling it.

To completely disable the automatic loading of a default RDD, remove the two lines shown above in bold. For example:

     //  New Revised RDDSYS.PRG
     //  disables auto-loading
     #include "rddsys.ch"

     ANNOUNCE RDDSYS                     // This line must not change
     INIT PROCEDURE RddInit

        RETURN
     // eof: rddsys.prg

Basic Terminology

The RDD architecture introduces several new terms and concepts that are key to the design and usage of RDDs. You should familiarize yourself with these concepts and terms as you begin to use the RDD functionality. The meaning of some earlier terminology is also further defined. The following RDD functional glossary defines the terminology for all RDDs.

. Key Expression : A valid Clipper expression that creates a key value from a single record.

. Key Value : A value that is based on value(s) contained within database fields, associated with a particular record in a database.

. Identity : A unique value guaranteed by the structure of the data file to reference a specific record in a database even if the record is empty. In the Xbase file (.dbf), the identity is the record number; but it could be the value of a unique primary key or even the offset of an array in memory.

. Keyed-Pair : A pair consisting of a key value and an identity.

. Identity Order : Describes a database arranged by identity. In Xbase, this refers to the physical arrangement of the records in the database in the order in which they were entered (natural order).

. Tag : A set of keyed-pairs that provides ordered access to the table based on a key value. Usually, an Order in a multiple-Order index (Order). An Order.

. Order : A named mechanism (index) that provides logical access to a database according to the keyed-pairs. This term encompasses both single indexes and the Tags in multiple-Tag indexes.

Orders are not, themselves, data files. They provide access to data that gives the appearance of an ordering of the data in a specific way. This ordering is defined by the relationships between keyed- pairs. An Order does not change the physical (the natural or entry) order of data in a database.

. Controlling Order : The active Order (index) for a particular work area. Only one Order may control a work area at any time, and it controls the order in which the database is accessed during paging and searching.

. Order List : A list of all the Orders available to the database in the specified work area.

. Order Bag : A container that holds zero or more Orders. Normally a disk or memory file. A traditional index like (.ntx) is an Order Bag that holds only one Order. A multiple-Tag index (.mdx or .cdx) is an Order Bag that holds zero or more Orders. Though Order Bags may be a memory or disk file, Clipper 5.x only supports Order Bags as disk files.

. Record : A record in the traditional database paradigm is a row of one or more related columns (fields) of data. In the expanded architecture of Clipper, a record could be data that does not exactly fit this definition.

A record is, in this expanded context, data associated with a single identity. In an Xbase data structure, this corresponds to a row (fields associated with a record number); in other data structures, this may not be the case.

In this document we use “record” in the traditional sense, but you should be aware that Clipper permits expansion of the meaning of record.

. single-Order Bag : An Order Bag that can contain only one Order. The (.ntx) and (.ndx) files are examples of single-Order Bags.

. multiple-Order Bag : An Order Bag that can contain any number of Orders; a multiple-Tag index. The (.cdx) and (.mdx) files are examples of multiple-Order Bags.

. maintainable scoped Orders : Scoped (filtered) Orders created using the FOR clause. The FOR condition is stored in the index header. Orders of this type are correctly updated using the expression to reflect record updates, deletions and additions.

. non-maintainable/temporary Orders : Orders created using the WHILE or NEXT clauses. These Orders are useful because they can be created quickly. However, the conditions in these clauses are not stored in the index header. Therefore, Orders of this type are not correctly updated to reflect record updates, deletions and additions. They are only for temporary use.

. Lock List : A list of the records that are currently locked in the work area.

The Language Implementation

To support the RDD architecture and let you design applications that are independent of the data format you are using, many existing Clipper commands and functions have been enhanced, and several new language elements have been added. The following tables summarize these changes and additions. See the Reference chapter of this guide for more detailed information on a particular item.

     Enhanced Commands and Functions
     ------------------------------------------------------------------------
     Command/Function  Changes
     ------------------------------------------------------------------------
     APPEND FROM       VIA clause
     COPY TO           VIA clause
     DBAPPEND()        Terminology
     GO                Terminology
     DBAPPEND()        Terminology
     INDEX             ALL, EVAL, EVERY, NEXT, RECORD, REST, TAG, and
                       UNIQUE clauses
     SEEK              SOFTSEEK option
     SET INDEX         ADDITIVE clause
     SET ORDER         IN, TAG clauses
     DBSETINDEX()      Terminology
     RECNO()           Terminology
     ------------------------------------------------------------------------

     New Commands and Functions
     ------------------------------------------------------------------------
     Command/Function    Description
     ------------------------------------------------------------------------
     DELETE TAG          Delete a Tag (Order)
     DBGOTO()            Position record pointer to a specific identity
     DBRLOCK()           Lock the record at the current or specified identity
     DBRLOCKLIST()       Return an array of the currently locked records
     DBRUNLOCK           Release all or specified record locks
     ORDBAGEXT()         Return the Order Bag file extension
     ORDBAGNAME()        Return the Order Bag name of a specific Order
     ORDCREATE()         Create an Order in an Order Bag
     ORDDESTROY()        Remove a specified Order from an Order Bag
     ORDFOR()            Return the FOR expression of an Order
     ORDKEY()            Return the Key expression of an Order
     ORDLISTADD()        Add Order Bag contents or single Order to the Order
                         List
     ORDLISTCLEAR()      Clear the current Order List
     ORDLISTREBUILD()    Rebuild all Orders in the Order List of the current
                         work area
     ORDNAME()           Return the name of an Order in the work area
     ORDNUMBER()         Return the position of an Order in the current Order
                         List
     ORDSETFOCUS()       Set focus to an Order in an Order List
     RDDLIST()           Return an array of the available Replaceable
                         Database Drivers
     RDDNAME()           Return the name of the RDD active in the current or
                         specified work area
     RDDSETDEFAULT()     Set or return the default RDD for the application
     ------------------------------------------------------------------------

User Interface Levels

We want to make it easy for you to quickly take advantage of the added functionality provided in Clipper 5.x. In order to effectively use the RDDs, you should read the following discussions. They are provided as a means of identifying the degree of programming knowledge or Clipper experience that will let you effectively use the RDD features.

For this purpose the RDD feature set is arbitrarily divided into levels A and B. Tables listing the commands or functions that comprise these access levels are also supplied. In addition, an RDD Features Summary is provided in table form which outlines the features available in each driver. The commands and functions in both of these levels of access are described in the Reference chapter of this guide.

Level A – Command-Level Interface

Level A. a simple command-level interface very similar to those found in other languages (e.g., dBASE IV, FoxPro). This is the primary access for new Clipper users who may or may not be familiar with other languages.

The following table lists the commands and functions accessible by the Clipper programmer with background in languages such as dBASE or FoxPro. The commands and functions in this table provide access to the additional features without requiring an advanced knowledge of Clipper or other programming concepts.

     Basic Commands and Functions
     ------------------------------------------------------------------------
     Command/Function  Changes
     ------------------------------------------------------------------------
     DELETE TAG        Delete a Tag
     GOTO              Move the pointer to the specified identity
     INDEX             Create an index file
     SEEK              Search an Order for a specified key value
     SET INDEX         Open one or more Order Bags in the current work area
     SET ORDER         Select the controlling Order
     DBAPPEND()        Append a new record to the current Lock List
     DBRLOCK()         Lock the record at the current or specified identity
     DBRLOCKLIST()     Return an array of the current Lock List
     DBRUNLOCK         Release all or specified record locks
     ------------------------------------------------------------------------

Level B – Function-Level Interface

Level B. Clipper also adds a function level interface that not only allows access to the enhanced functionality of the drivers, but permits the building of higher-level functions using these composing behaviors. This level is meant for more experienced Clipper users who need to take advantage of the full power of the driver and Order Management System.

The following table lists the DML and Order Management functions recommended to the intermediate to advanced Clipper programmer. These functions provide the greatest flexibility in accessing the extended features of these drivers

     Advanced Functions (including Order Management)
     ------------------------------------------------------------------------
     Command/Function    Description
     ------------------------------------------------------------------------
     DBAPPEND()          Append a new record to the current Lock List
     DBRLOCK()           Lock the record at the current or specified identity
     DBRLOCKLIST()       Return an array of the current Lock List
     DBRUNLOCK()         Release all or specified record locks
     ORDBAGEXT()         Return the default Order Bag RDD extension
     ORDBAGNAME()        Return the Order Bag name of a specific Order
     ORDCREATE()         Create an Order in an Order Bag
     ORDDESTROY()        Remove a specified Order from an Order Bag
     ORDFOR()            Return the FOR expression of an Order
     ORDKEY()            Return the Key expression of an Order
     ORDLISTADD()        Add Order Bag contents or single Order to the Order
                         List
     ORDLISTCLEAR()      Clear the current Order List
     ORDLISTREBUILD()    Rebuild all Orders in the Order List of the current
                         work area
     ORDNAME()           Return the name of an Order in the work area
     ORDNUMBER()         Return the position of an Order in the current Order
                         List
     ORDSETFOCUS()       Set focus to an Order in an Order List
     RDDLIST()           Return an array of the available Replaceable
                         Database Drivers
     RDDNAME()           Return the name of the RDD active in the current or
                         specified work area
     RDDSETDEFAULT()     Set or return the default RDD for the application
     ------------------------------------------------------------------------

RDD Features

The following decision table summarizes the availability of key features across RDDs. It lists the features available in each RDD so you can use it as an aid in correct RDD implementation and data access.

     RDD Features Summary
     ------------------------------------------------------------------------
     Item                                NTX   NDX   MDX   CDX  DBPX
     ------------------------------------------------------------------------
     Implicit record unlocking in        Yes   Yes   Yes   Yes  Yes
     single lock mode
     Multiple Record Locks               Yes   Yes   Yes   Yes  No
     Number of Concurrent Record Locks   *1    *1    *1    *1   1
     Order Management (Tag support)      Yes   Yes   Yes   Yes  No
     Orders (Tags) per Order Bag (File)  1     1     47    50   N/A
     Number of Order Bags (Files)        15    15    15    15   N/A
     per work area
     Conditional Indexes (FOR clause)    Yes   No    Yes   Yes  No
     Temporary (Partial) Indexes         Yes   No    No    Yes  No
     (WHILE, ... )
     Descending via DESCENDING clause    Yes   No    Yes   Yes  No
     Unique via the UNIQUE clause        Yes   Yes   Yes   Yes  No
     EVAL and EVERY clause support       Yes   No    No    Yes  No
     Production/Structural Indexes       No    No    Yes   Yes  No
     Maximum Key Expression length       256   256   220   255  N/A
     (bytes)
     Maximum FOR Condition length        256   N/A   261   255  N/A
     (bytes)
     ------------------------------------------------------------------------

     *1 determined by available memory.

Clipper 5.x Order Management

Clipper includes a new Order Management System which provides a more effective and flexible way of indexing data. The main objective of the new Order Management implementation is to raise the Xbase indexing paradigm from a low level of abstraction (Xbase database specific) to a higher, more robust, level. This higher level of abstraction allows the user to build new commands and functions.

Low level abstraction refers to manipulation of discrete elements in the database architecture (i.e., field names and sizes, methods of handling controlling indexes, etc.).

High level abstraction refers to manipulation of general elements in a data source. It lets us, for example, set a controlling Order without explicitly addressing the character of the data file structure. This higher level of abstraction was achieved by reviewing all the processes that indexes have in common.

The Order Management function set was generically named (i.e. non-dbf specific) to provide a semantic that could encompass future RDD implementations that may not be file-bound. For example, an RDD could easily be created that orders (indexes) on a memory array, or other data structure, instead of a database. Therefore, all Order Management functions simply begin with ORD (for Order). You will find the function names to be self-explanatory (e.g., ORDCREATE() creates an Order, and ORDDESTROY() destroys an Order).

Concept

An Order is a set of keyed-pairs that provides a logical ordering of the records in an associated database file. Each key in an Order (index) is associated with a particular identity (record number) in the data set (database file). The records can be processed sequentially in key order, and any record can be located by performing a SEEK operation with the associated key value. An Order never physically changes the data that it’s applied against, but creates a different view of that data.

There are at least four basic types of processes that you can perform with an Order:

1. Ordering: Changes the sequence in which you view the data records.

2. Scoping: Constrains the visibility of data to specified upper and lower bounds. Determines the range of data items included, through a scoping rule, like the WHILE clause.

3. Filtration: Visibility of data is subject to conditional evaluation. Filtration determines which items of data are included, through a filter rule, like the FOR clause.

4. Translation: Values in underlying data source are translated (or converted) in some form based on a selection criteria. For example:

INDEX ON IIF(CUSTID > 1000, "NEW", "OLD")

The difference between scope and condition as it applies to FOR and WHILE is that the WHILE clause provides scope, but not filtering, but a FOR clause can provide both.

There are three primary elements in Order Management:

. Order: An Order is a set that has two elements in it: an Order Name, which is a logical name that can be referenced, and an Order Expression which supplies the view of the data. The Order Name provides logical access to the expression and the Order Expression provides a way of viewing the underlying data source. Data ordering can also be modified to ascending or descending sequence.

– Order Name: An Order Name is a symbolic name, that you use to manipulate an Order, like a file’s alias. The difference between an Order Name and the Order Number with which you would normally access indexes (Orders), is that the Order Name is stored in the index file. It is available each time you run the program, and is maintained by the system. The Order Number is generated each time the Order is added to an Order List and may change from one program execution to another. This makes Order Name the preferred means of referencing Orders.

– Order Expression: Is any valid Clipper expression. This is an index expression such as:

CUSTLIST->CUSTID

This expression produces the ordered view of the data. The values derived from this expression are sorted, and it is the relationship of these values to one another that provides the actual ordering.

. Order Number: An Order Number is provided by the Order List. An Order Number is only valid as long as the work area to which it belongs is open.

– Order Numbers provide one of the services performed by Order Names, allowing you to access a specific Order. In general, you should avoid accessing Orders by number.

– The ORDNUMBER() function returns the ordinal position of the specified <orderName> within the specified <orderList>.

. Order Bag: Unsorted collection of Orders. Each Order contains two elements (Order Name and Order Expression). Each Order Bag may have zero to n Orders. The maximum is determined by the RDD driver being used. Order Bags are similar to multiple-index files in that there’s no guarantee of any specific order within the container or Bag. Within an Order Bag you can access specific Orders by referencing a particular Order Name. Order Bags have persistence between activations of the program.

. Order List: An Order List orders the collection of Orders that are associated with and active in the current work area. It provides an access to the Orders active within a given work area. Each work area has an Order List, and there is only one Order List per work area. An Order List is created when a new work area is opened, and exists only as long as that work area is active. Once you close a work area, the Order List ceases to exist.

When you SET INDEX TO, the contents of the Order Bag are emptied into the Order List. At this point, the Orders in the Order List are active in the work area, where they will be updated as the data associated with the work area is modified. You may access an Order in the list by its Order Number or by its Order Name. You should access an Order by its name rather than a hard-coded ordinal position. You can make any Order in the Order List the controlling Order by giving it focus, as explained below.

. Order List Focus: Order List Focus is, essentially, a pointer to the Order that is used to change the view of the data. It is synonymous with controlling Order or controlling index, and defines the active index order. The SET ORDER TO command does not modify the Order List in any way. It does not clear the active indexes. It only changes the Order List Focus (the controlling order in the Order List).

Notes

The following list contains specific information regarding Order Bag usage and limitations with DBFNDX and DBFNTX index files:

. Single-Order Bags: With DBFNDX and DBFNTX you can explicitly assign the Order Name within the Order creation syntax. You can then use the Order Name in any command or function that accepts an Order Name (Tag) as a parameter.

. Single-Order Bag with INDEX ON: Single-Order Bags may retain the Order Name between activations. During creation, DBFNTX stores an optionally supplied Order Name in the file’s header for subsequent use. Therefore, the Order Name is not necessarily the same as that of the file. By contrast, DBFNDX cannot store an Order Name since this would prevent dBASE from accessing the file. By default DBFNDX Orders inherit the name of their index file.

Summary

This chapter has introduced you to the RDD concept, giving you specific information on the architecture that implements RDDs in Clipper. The basic terminology of RDDs has also been defined.

Finally, you have seen an overview of the language enhancements designed to make using RDDs straightforward and to let you build applications that do not depend on the RDD in use. The next chapter elaborates on these language enhancements, discussing syntax and usage in detail.

C5DG-1 Introduction

Posted on by vivaclipper

Clipper 5.x – Drivers Guide

Chapter 1

Introduction

Important!

Some of the topics in this guide are intended for advanced Clipper developers. Much of this information is presented at a fairly high level and requires programming knowledge beyond the Clipper language. Other parts are useful to users of all levels. Refer to the “User Interface Levels” section of the Replaceable Database Driver Architecture chapter to determine which part of the language is appropriate to your level of expertise.

Based on your own experience level with the Clipper language, you should decide whether you wish to take advantage of these new and advanced features. The Reference guide contains existing Clipper command and function syntax and descriptions. This guide addresses new extensions to Clipper. Understanding this information should enable you to increase the power and effectiveness of your applications.

Clipper supports a driver architecture that allows Clipper- compiled applications to use replaceable database and terminal drivers. This Drivers guide contains all the information you need to use the replaceable drivers provided as part of the Clipper Development System.

Overview of RDD System

RDD is an abbreviation for Replaceable Database Driver, and it is used to describe an interface that controls how your application accesses and manipulates database and ancillary files.
Clipper provides several RDDs to give you access to the database, memo, and index file formats of many popular database software products. By simply linking the proper RDD with your application, you get automatic, easy access to files created by other database engines.

Moreover, Clipper gives you new and enhanced commands and functions designed to make your applications independent of the RDD in use. Using RDDs, you can give end users more flexibility in choosing to migrate to your Clipper applications without losing data and to easily move their data back and forth between applications if they prefer.

Overview of Alternate Terminal Drivers

An Alternate Terminal Driver is a library file (.LIB) that controls how your application addresses the screen output device. Clipper provides several Alternate Terminal Drivers to allow your applications to run in a wider variety of environments.

Note: To perform normal information presented in screen input/output in a Clipper application, you do not need the Drivers guide. The default database and terminal drivers are automatically linked and the commands and functions used for these purposes are discussed in the Reference guide. For several categorized lists of these commands and functions, refer to Appendix G: Categorized Language Tables in the Error Messages and Appendices guide.

In This Guide

This guide consists of nine chapters including this Introduction chapter.

For an online version of this guide accessible while operating your program editor or any other development utility, use The Guide To Clipper.

Chapter 2: Replaceable Database Driver Architecture

The Clipper database system supports a driver architecture that makes Clipper-compiled applications data format independent. Such applications can, therefore, access the data formats of other database systems, including the dBASE IV (.mdx), FoxPro (.cdx), and Paradox (.db) formats on a variety of equipment. This chapter discusses how RDDs fit into the overall Clipper architecture, defines the basic terminology you will need to understand subsequent chapters, and summarizes new and enhanced commands and functions designed to support the RDD architecture.

Chapter 3: Reference

Clipper 5.x provides many new and enhanced commands and functions that you will use to access and manipulate databases and to get specific information about the RDD in use. This chapter contains entries for all new Clipper commands and functions designed for use with RDDs. As for existing language elements, only those that were changed to accommodate the new RDD architecture are documented here. Other language elements that are specifically designed for database manipulation and have not been altered (e.g., USE) are documented in the Reference guide.

Chapter 4: DBFCDX Driver Installation and Usage

The DBFCDX database driver provides access to FoxPro 2 (.cdx) and (.idx) file formats. This chapter explains how to install DBFCDX and how to use it in your applications.

Chapter 5: DBFMDX Driver Installation and Usage

The DBFMDX database driver provides access to dBASE IV (.dbf), (.mdx), and (.dbt) file formats. The driver also supports dBASE IV-compatible file and record locking schemes, allowing shared access between Clipper and dBASE IV programs. This chapter explains how to install DBFMDX and how to use it in your applications.

Chapter 6: DBFNDX Driver Installation and Usage

The DBFNDX database driver uses the Clipper driver architecture to access dBASE III PLUS compatible index files within a Clipper program, allowing you to create, access, and update dBASE III and dBASE III PLUS compatible index (.ndx) files. This chapter explains how to install DBFNDX and how to use it in your applications.

Chapter 7: DBFNTX Driver Installation and Usage

DBFNTX is the default database driver for Clipper that lets you create and maintain (.ntx) files with features above and beyond those supplied with previous versions of DBFNTX. This chapter details these new features and explains how to install and use DBFNTX in your applications.

Chapter 8: DBPX Driver Installation and Usage

The DBPX database driver provides access to Paradox 3.5 (.db), (.px), (.x) and (.y) file formats. This chapter explains how to install DBPX and how to use it in your applications.

Chapter 9: Alternate Terminal Drivers

Clipper provides several Alternate Terminal Drivers to allow your applications to run in a wider variety of environments. This chapter discusses how Alternate Terminal Drivers fit into the overall Clipper architecture, as well as how to install and use each of the supplied terminal drivers: ANSITERM, NOVTERM, and PCBIOS.

What is Harbour ?

Posted on by vivaclipper

Harbour is a modern computer programming language. It is a Clipper-compatible compiler which is cross-platform, running on many operating systems (DOS, Microsoft Windows, Linux, Unix variants, several BSD descendants, Mac OS X, MINIX 3, Windows CE, Pocket PC, Symbian, iPhone, QNX, VxWorks, OS/2/eComStation, BeOS/Haiku, AIX) using the same source code and databases.

Although it is a powerful general-purpose programming language, it was primarily used to create database/business programs. Harbour have been actively maintained looking for diversity keeping backward-compatible with Clipper style. It has undergone many changes and revisions and regain widely popularity amongst programmers in 1980s and 1990s

The open source Harbour license is similar to the GNU General Public License, with an exception supporting proprietary applications, so proprietary applications can be produced with Harbour and distributed..

Paradigm(s) multi-paradigm: imperative, functional,object-oriented, reflective
Appeared in 1999
Designed by Antonio Linares
Developer Viktor Szakáts and community
Stable release 3.0.0 (17 July 2011)
Preview release 3.1.x available from SVN
Typing discipline optionally duck, dynamic, safe, partially strong
Dialects Clipper, Xbase ++, Flagship, FoxPro, xHarbour
Influenced by dBase, Clipper
Influenced xHarbour
OS Cross-platform
License Open source GPL Compatible
Usual filename extensions .prg, .ch, .hbs, .dbf
Website http://www.harbour-project.org/

History

The idea of a free software Clipper compiler has been floating around for a long time and the subject has often cropped up in discussion on comp.lang.clipper. Antonio Linares founded the Harbour project and the implementation was started.

Sailing the Clipper ship to a Harbour port. Clipper is a type of ship. Harbour is a synonym to port (where ship docks) Harbour is out port to the Clipper language.

On 2009 Harbour had a huge make over on its design promoted mainly by Viktor Szakáts and Przemyslaw Czerpak

 Database support

Harbour extends the Clipper Replaceable Database Drivers (RDD) approach. It offers multiple RDDs such as DBF, DBFNTX, DBFCDX, DBFDBT and DBFFPT. In Harbour multiple RDDs can be used in a single application, and new logical RDDs can be defined from combination of other RDDs. The RDD architecture allows for inheritance, so that a given RDD may extend the functionality of other existing RDD(s). Third-party RDDs, like RDDSQL, RDDSIX, RMDBFCDX, Advantage Database Server, and Mediator exemplify some of the RDD architecture features. DBFNTX implementation has almost same functionality of DBFCDX and RDDSIX. NETIO and LetoDB provide remote access over TCP protocol Harbour also offers ODBC support by means of an OOP syntax, and ADO support by means of OLE. MySQL, PostgreSQL, SQLite, Firebird, Oracle are examples of databases which Harbour can connect.

xBase technologies often is confused with a RDBMS software. Although this is true, xBase is more than a simple database system as the same time xBase languages using purely DBF can not provide full concept of a real RDBMS

Programming philosophy

Unlike Java which is intended to be write once, run anywhere, Harbour aims to be write once, compile anywhere. As the same compiler is available for all of the above operating systems, there is no need for recoding to produce identical products for different platforms, except when operating system dependent features are used. Cross-compiling is supported with MinGW32. Under Microsoft Windows, Harbour is more stable but less well-documented than Clipper, but has multi-platform capability and is more transparent, customizable and can run from a USB flash drive.

Under Linux and Windows Mobile, Clipper source code can be compiled with Harbour with very little adaptation. Most software originally written to run on Xbase ++, Flagship, FoxPro, xHarbour and others dialects can be compiled with Harbor with some adaptation. As 2010 many efforts have been made to turn the transition from other xBase dialects easier.

Harbour can use the following C compilers, amongothers: GCC, MinGW, Clang, ICC, Microsoft Visual C++ (6.0+), Borland C++, Watcom C, Pelles C and Sun Studio.

Harbour can make use of multiple Graphic Terminal emulations, including console drivers, and Hybrid Console/GUIs, such as GTWvt, and GTWvg.

Harbour supports external GUIs, free (e.g. HWGui, MiniGUI and Qt) and commercial (e.g. FiveWin, Xailer). HBQt is a library provinding bindings to Qt. HBIDE application included in official distribution and SVN repository is a sample of HBQt potencial.

Harbour is 100% Clipper-compatible and supports many language syntax extensions including greatly extended run-time libraries such as OLE, Blat, OpenSSL, FreeImage,GD, TIP, Tpathy, PCRE, HbZip (zlib and bzip2), cURL, Cairo, its own implementation of CA-Tools and NanFor libraries and many others. Harbour has an active development community and extensive third party support.

Any xBase language provides a very productive way to build business and data intensive applications. Harbour is not an exception.

Macro Operator (runtime compiler)

One of the most powerful features of xBase languages is the Macro Operator ‘&’. Harbour’s implementation of the Macro Operator allows for runtime compilation of any valid Harbour expression. Such a compiled expression may be used as a VALUE, i.e. the right side of an assignment (rvalue), but more interestingly, such a compiled expression may be used to resolve the left side (lvalue) of an assignment, i.e. PRIVATE, or PUBLIC variables, or a database FIELD.

Additionally, the Macro Operator may compile and execute function calls, complete assignments, or even list of arguments, and the result of the macro may be used to resolve any of the above contexts in the compiled application. In other words, any Harbour application may be extended and modified at runtime to compile and execute additional code on-demand.

Latest Macro compiler can compile any valid Harbour code including code to pre-process before compile.

Syntax:

 &( ... )

The text value of the expression ‘…’ will be compiled, and the value resulting from the execution of the compiled code is the result.

 &SomeId

is the short form for &( SomeId ).

 &SomeId.postfix

is the short form of &( SomeId + “postfix” ).

Object Oriented Programming

Programming in an OOP style is a broader issue than a specific library or a specific interface, but OOP programming is something many Clipper programmers have come to expect. CA-Clipper 5.2 and especially 5.3 added a number of base classes, and a matching OOP syntax. Libraries such as Class(y), Fivewin, Clip4Win, and TopClass provide additional OOP functionality.

Harbour has OOP extensions with full support for classes including inheritance, based on Class(y) syntax. OOP syntax in Harbour is very similar to that of earlier Clipper class libraries so it should be possible to maintain legacy Clipper code with minimal changes.

Syntax and semantics

Harbour as every xBase language is case insensitive and can optionally accept keywords written just by first four characters

Built-in data types

Harbour has 6 scalar types : Nil, String, Date, Logical, Number, Pointer, and 4 complex types: Array, Object, CodeBlock, and Hash. A scalar holds a single value, such as a string, number, or reference to any other type. Arrays are ordered lists of scalars or complex types, indexed by number, starting at 1. Hashes, or associative arrays, are unordered collections of any type values indexed by their associated key, which may be of any scalar or complex type.

Literal (static) representation of scalar types:

  • Nil: NIL
  • String: “hello”, ‘hello’, [hello]
  • Date: 0d20100405
  • Logical: .T., .F.
  • Number: 1, 1.1, −1, 0xFF

Complex Types may also be represent as literal values:

  • Array: { “String””, 1, { “Nested Array” }, .T., FunctionCall(), @FunctionPointer() }
  • CodeBlock: { |Arg1, ArgN| Arg1 := ArgN + OuterVar + FunctionCall() }
  • Hash: { “Name” => “John”, 1 => “Numeric key”, { “Nested” => “Hash” } }

Hashes may use any type including other Hashes as the Key for any element. Hashes and Arrays may contain any type as the Value of any member, including nesting arrays, and Hashes.

Codeblocks may have references to Variables of the Procedure/Function>method in which it was defined. Such Codeblocks may be returned as a value, or by means of an argument passed BY REFERENCE, in such case the Codeblock will “outlive” the routine in which it was defined, and any variables it references, will be a DETACHED variable.

Detached variables will maintain their value for as long as a Codeblock referencing them still exists. Such values will be shared with any other Codeblock which may have access to those same variables. If the Codeblock did not outlive its containing routine, and will be evaluated within the lifetime of the routine in which it is defined, changes to its Detached Variables(s) by means of its evaluation, will be reflected back at its parent routine.

Codeblocks can be evaluated any number of times, by means of the Eval( BlockExp ) function.

Variables

All types can be assigned to named variables. Named variable identifiers are 1 to 63 characters long, start with [A-Z|_] and further consist of the characters [A-Z|0–9|_] up to a maximum of 63 characters. Named variables are not case sensitive.

Variables have one of the following scopes:

  • LOCAL: Visible only within the routine which declared it. Value is lost upon exit of the routine.
  • STATIC: Visible only within the routine which declared it. Value is preserved for subsequent invocations of the routine. If a STATIC variable is declared before any Procedure/Function/Method is defined, it has a MODULE scope, and is visible within any routine defined within that same source file, it will maintain its life for the duration of the application lifetime.
  • PRIVATE: Visible within the routine which declared it, and all routines called by that routine.
  • PUBLIC: Visible by all routines in the same application.

LOCAL and STATIC are resolved at compile time, and thus are much faster than PRIVATE and PUBLIC variables which are dynamic entities accessed by means of a runtime Symbol table. For this same reason, LOCAL and STATIC variables are not exposed to the Macro compiler, and any macro code which attempts to reference them will generate a runtime error.

Due to the dynamic nature of PRIVATE and PUBLIC variables, they can be created and destroyed at runtime, can be accessed and modified by means of runtime macros, and can be accessed and modified by Codeblocks created on the fly.

Control structures

The basic control structures include all of the standard dBase, and Clipper control structures as well as additional ones inspired by the C or Java programming languages:

Loops

[DO] WHILE ConditionExp
   ...
   [LOOP]
   [EXIT]
END[DO]

FOR Var := InitExp TO EndExp [STEP StepExp]
   ...
   [LOOP]
   [EXIT]
NEXT
FOR EACH Var IN CollectionExp
   ...
   [HB_EnumIndex()]
   [LOOP]
   [EXIT]
NEXT
  • The … is a sequence of one of more Harbour statements, and square bracketes [] denote optional syntax.
  • The HB_EnumIndex() may be optionally used to retrieve the current iteration index (1 based).
  • The LOOP statement restarts the current iteration of the enclosing loop structure, and if the enclosing loop is a FOR or FOR EACH loop, it increases the iterator, moving to the next iteration of the loop.
  • The EXIT statement immediately terminates execution of the enclosing loop structure.
  • The NEXT statement closes the control structure and moves to the next iteration of loop structure.

In the FOR statement, the assignment expression is evaluated prior to the first loop iteration. The TO expression is evaluated and compared against the value of the control variable, prior to each iteration, and the loop is terminated if it evaluates to a numeric value greater than the numeric value of the control variable. The optional STEP expression is evaluated after each iteration, prior to deciding whether to perform the next iteration.

In FOR EACH, the Var variable will have the value (scalar, or complex) of the respective element in the collection value. The collection expression, may be an Array (of any type or combinations of types), an Hash Table, or an Object type.

IF statements

IF CondExp
   ...
[ELSEIF] CondExp
   ...
[ELSE]
   ...
END[IF]

… represents 0 or more statement(s). The condition expression(s) has to evaluate to a LOGICAL value.

SWITCH statements

Harbour supports a SWITCH construct inspired by the C implementation of switch().

SWITCH SwitchExp
   CASE LiteralExp
      ...
      [EXIT]
   [CASE LiteralExp]
      ...
      [EXIT]

   [DEFAULT]
      ...
END
  • The LiteralExp must be a compiled time resolvable numeric expression, and may involve operators, as long as such operators involve compile time static value.
  • The EXIT optional statement is the equivalent of the C statement break, and if present, execution of the SWITCH structure will end when the EXIT statement is reached, otherwise it will continue with the first statement below the next CASE statement (fall through).

BEGIN SEQUENCE statements

BEGIN SEQUENCE
   ...
   [BREAK]
   [Break([Exp])]
RECOVER [USING Var]
   ...
END[SEQUENCE]

or:

BEGIN SEQUENCE
   ...
   [BREAK]
   [Break()]
END[SEQUENCE]

The BEGIN SEQUENCE structure allows for a well behaved abortion of any sequence, even when crossing nested procedures/functions. This means that a called procedure/function, may issue a BREAK statement, or a Break() expression, to force unfolding of any nested procedure/functions, all the way back to the first outer BEGIN SEQUENCE structure, either after its respective END statement, or a RECOVER clause if present. The Break statement may optionally pass any type of expression, which may be accepted by the RECOVER statement to allow further recovery handing.

Additionally the Harbour Error Object supports canDefault, canRetry and canSubstitute properties, which allows error handlers to perform some preparations, and then request a Retry Operation, a Resume, or return a Value to replace the expression triggering the error condition.

Alternatively TRY [CATCH] [FINALLY] statements are available on xHB library working like the SEQUENCE construct.

Procedures/Functions

[STATIC] PROCEDURE SomeProcedureName
[STATIC] PROCEDURE SomeProcedureName()
[STATIC] PROCEDURE SomeProcedureName( Param1' [, ParamsN] )
INIT PROCEDURE SomeProcedureName
EXIT PROCEDURE SomeProcedureName
[STATIC] FUNCTION SomeProcedureName
[STATIC] FUNCTION SomeProcedureName()
[STATIC] FUNCTION SomeProcedureName( Param1' [, ParamsN] )

Procedures/Functions in Harbour can be specified with the keywords PROCEDURE, or FUNCTION. Naming rules are same as those for Variables (up to 63 characters non case sensitive). Both Procedures and Functions may be qualified by the scope qualifier STATIC to restrict their usage to the scope of the module where defined.

The INIT or EXIT optional qualifiers, will flag the procedure to be automatically invoked just before calling the application startup procedure, or just after quitting the application, respectively. Parameters passed to a procedure/function appear in the subroutine as local variables, and may accept any type, including references.

Changes to argument variables are not reflected in respective variables passed by the calling procedure/function/method unless explicitly passed BY REFERENCE using the@ prefix.

PROCEDURE have no return value, and if used in an Expression context will produce a NIL value.

FUNCTION may return any type by means of the RETURN statement, anywhere in the body of its definition.

An example procedure definition and a function call follows:

 x := Cube( 2 )

 FUNCTION Cube( n )
 RETURN n ** 3

 Sample code The typical “hello world” program would be:

 
  ? "Hello, world!"

Or:

  QOut( "Hello, world!" )

Or:

  Alert( "Hello, world!" )

Or, enclosed in an explicit procedure:

 PROCEDURE Main()

    ? "Hello, world!"

 RETURN

 

OOP examples

 #include "hbclass.ch"

 PROCEDURE Main()

    LOCAL oPerson := Person( "Dave" )

    oPerson:Eyes := "Invalid"

    oPerson:Eyes := "Blue"

    Alert( oPerson:Describe() )
 RETURN

 CLASS Person
    DATA Name INIT ""

    METHOD New() CONSTRUCTOR

    ACCESS Eyes INLINE ::pvtEyes
    ASSIGN Eyes( x ) INLINE IIF( ValType( x ) == 'C' .AND. ;
                 x IN "Blue,Brown,Green", ::pvtEyes := x,; 
                 Alert( "Invalid value" ) )

    // Sample of IN-LINE Method definition
    INLINE METHOD Describe()
       LOCAL cDescription

       IF Empty( ::Name )
          cDescription := "I have no name yet."
       ELSE
          cDescription := "My name is: " + ::Name + ";"
       ENDIF

       IF ! Empty( ::Eyes )
          cDescription += "my eyes' color is: " + ::Eyes
       ENDIF
    ENDMETHOD

    PRIVATE:
       DATA pvtEyes
 ENDCLASS

 // Sample of normal Method definition.
 METHOD New( cName ) CLASS Person

   ::Name := cName

 RETURN Self

 

Tools

  • HBMK2 – Powerful build tool like make
  • HBDoc2 and  HBExtern – Creates documentation for Harbour
  • HPPP – Pre-processor, a powerful tool which avoids typical problems found on C language pre-processor
  • HBFormat – Formats source code written on Harbour or another dialect according defined rules
  • HBi18n – Tools to localizing text on applications
  • HBRun – Shell interpreter for Harbour. Macro compiling allows to run any valid Harbour code as it’s being compiled
  • HBIDE – Integrated Development Environment to help Harbour development and various xBase dialects

All tools are multiplatform.

Development

Today Habour development is leading by Viktor Szakáts with huge collaborations and leading many components of core and contribs by Przemysław Czerpak. HBIDE and some components, specially HBQt, are developed by Pritpal Bedi. Others members send minor changes to the Sourceforge SVN repository.  As 2010 Harbour development is keeping vibrant activity

Popularity

Although there is no way to measure popularity of Harbour or xBase, the TIOBE Programming Community Index As of June 2006 ranked Microsoft Visual FoxPro, a high profile dialect of xBase, on 12th position on programming languages popularity ranking. FoxPro/xBase ranked on 25th position As of August 2010. As of September 2010, the Clipper Usenet newsgroupscomp.lang.clipper is still active. As of August 2010 Harbour figured on 16th position on weekly downloads in compiler category and 132th position on global rank.

xHarbour comparison

xHarbour is a fork of the earlier Harbour project. xHarbour takes a more aggressive approach to implementing new features in the language, while Harbour is more conservative in its approach, aiming first of all for an exact replication of Clipper behaviour and then implementing new features and extensions as a secondary consideration. It should also be noted that Harbour is supported on a wide variety of operating systems while xHarbour only really supports MS Windows and Linux 32-bit.

The Harbour developers have attempted to document all hidden behaviour in the Clipper language and test Harbour-compiled code alongside the same code compiled with Clipper to maintain compatibility.

The Harbour developers explicitly reject extensions to the language where those extensions would break Clipper compatibility. These rejections were soften recently since the new Harbour architecture allow extensions out of the core compiler.

A detailed comparison between extensions implemented in Harbour and xHarbour can be found in SVN repository of the project on SourceForge.

As of 2009–2010, Harbour has seen a huge increase in its adoption while xHarbour decline as can be seen on his mailing list

See also

References

  1. ^ Harbour license
  2. ^ LetoDB
  3. ^ Official Harbour page
  4. ^ http://sourceforge.net/projects/harbour-project/
  5. ^ TIOBE Programming Community Index
  6. ^ SourceForge
  7. ^ About xHarbour
  8. ^ xhb-diff.txt
  9. ^ Harbour developers’ mailing list statistics
  10. ^ xHarbour developers’ mailing list statistics
  11. ^ ohloh.net Activity comparison

External links

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Note : This post borrowed by curtesy of Vikipedia from here : http://en.wikipedia.org/wiki/Harbour_(software)