on the grid and virtualgrid

[mrduck]

In Qt there are two kinds of "grids": QTableView and QTableWidget. (1)

The difference is in how they handle the "cells".

Simplifying a bit, we may say that QTableView asks the program what and how to print each cell using a callback mechanism.

QTableWidget, instead, has to be loaded with data by the program and then handles everything itself.

This basic difference has quite big consequences.

QTableView is the basis of VIRTUALGRID and BROWSE (2), QTableWidget is the basis for GRID.

VIRTUALGRID handles both arrays and DBFs. For arrays, in the program you create the array, set grid to use it, and the grid will start to do the callbacks... how many rows ? which is the back/forecolor of cell at position 3,6 ? What text should I print at cell 2,1 ?

VIRTUALGRID answers all these queries in abstractGrid:onQuerydata(). For example the row count for arrays is quite easy:

(look in the source code, the forum program doesn't allow me to post the code....)

With VIRTUALGRID you work directly with the underlaying data. If you change a value in the array, the array is directly updated, the same for DBF fields. Of course, this is both good and bad. But for quick data entry it may be good.

As I said, QTableView asks the program for the data to display. For arrays it is quite simple, the grid rows matches the array row -1 (grid is 0-based, array is 1-based).
For DBFs with filters, index, ordscope, set deleted on the situation is a bit more complex. Infact if we can use with a simple dbGoto in case of none of the cited "filters" are set, it is a condition very rarely found in common use.
If we have index and optionally a ordscope, we may use ord* functions (not implemented now)
But if a filter is set, we must use "skip" to move between records.

Let's go with an example: we have to browse a DBF table with a controlling index on LastName. We have a VIRTUALGRID.

Qt QTableView asks for the number of rows in the data, and it gets back 200.
Then it starts to ask for the data for each row. Let's say there are 21 rows that can fit the screen, so it asks data for rows 0,1,2,3,4,5,6,7 up to 20. We move between records using skip n, with n calculated remembering the position we are and where we want to go.

Now click on first row: Qt asks again for data in row 0 (row 0 is a gotop()). Now row 0 is selected. Click on row 20. Qt asks for data for row 0 to draw the cells without the selection colors and then asks for data for row 20 to display it as selected. If we exit focus from the grid, row 20 is displayed with selected-disabled color, and data requested yet another time. (3)

As you may understand, everytime Qt needs to repaint the screen it asks the program the data... isn't it wonderful ?

Yes, it really is.

But in this DBF access we must consider SHARED dbf access. What happens if someone changed LastName of row 20 in another program instance and we click on row 20 ? If the change modified the order in the index of record 20, clicking on it would change what is shown on screen, since the skip would move to a different record!
But if I used a cache system to remember the recno() of row 20 and use dbGoto(), I may go to a record that is no more in the correct order, and since I may have some mixed cached and not cached records, it may become a mess.

QTableView in several occasions asks for a new record count, refreshes the rows shown on the screen asking for data, for example when it regains focus, when the grid is partially covered by other windows... at that moment the screen is refreshed and new rows and changed data are shown... (4)

and so we are at the crucial point: Qt needs info and ask for them, and it decides when to do... and it changes recno() pointer... it changes it several times... we may also try to reposition the pointer back at every request but it may be dangerous !

How can we trust the data in a shared DBF ? how can we be sure to not UPDATE the wrong record ?
Can Qt asks for data in the middle of an UPDATE ALL ? or a loop calculating some totals ???

(end of part 1.... is anybody interested in a part 2????)

(1) QTableWidget inherits from QTableView providing all the missing bits that the programmer must supply
(2) obsoleted
(3) it may be possible to add a cache layer to not use skip...
(4) I performed limitate testing and I don't know what happens when only some fields are changed

[concentra]

and it changes recno() pointer

I got this situation and this isn't what Clipper do !
So we must take this in mind while coding !

(end of part 1.... is anybody interested in a part 2????)

Very, please go ahead !

[Roberto Lopez]

Just some random ideas that could help to get a better one

HMG/WIN32 browse works as follows:

1. I've used a standard ListView control.

2. It is ALWAYS populated only with the records needed to complete the visible row count.

3. When the user scrolls (ie: 1 row down) the control is emptied and populated again, beginning at the second table record, until the visible row count be completed again.

4. Trapping cursor up, cursor down, page up, page down, ctrl-pgup, ctrl-pgdown and doing required record pointer movements (top, bottom, skip(+-n)) you'll cover all requirements of keyboard navigation. Regardless of how many records the table has, the navigation is nearly instantaneous, since the record movements are very fast always. This approach works fine and fast in all situations, including filtered tables.

5. The only thing that you must care about is the recno() of the first current row shown, since it will be your reference point to repopulate the control. Besides that, you must consider the situation when this record become out of the current filter, in such case, you must navigate the table until the next 'visible' record appears.

6. You always are working with a copy of the data that is updated only when the cursor is moved out of the visible-rows window or when a manual refresh is done.

7. You don't need to know the total record count at all.

8. IMHO, this is a wonderful solution, excepting the 'vertical scrollbar problem'

Yep. As you can imagine, populating the control with the visible-row count only, the vertical scrollbar will never appear.

So, the vertical scrollbar is optional in Browse control and you must enable if you want it.

I've solved that, adding a separate scrollbar control to the right of the ListView.

Coding it to be fast, is very problematic, since you need to know the total record count to make it work correctly and this is a nightmare (a slow one) for filtered tables, since, of course, the only way to know it is counting the records.

If you use ADS (even local) there is no problem, since OrdKeyCount() retrieves the 'filtered' record count for tables using filters only (no index required). Sadly, the standard Harbour RDDs doesn't.

Across time, I've developed various 'little tricks' to make it faster and it has a good performance, but in the case that those not be enough, you can simply disable vertical scrollbar.

Another approach (not used in HMG) could be automatically disable the vertical scrollbar if a filter is used. In such situation, the user deserves it, because not using conditional indexes instead

9. Regarding the record pointer, it is saved before refreshing, and later restored, so there is no problem for the user. Optionally, you can do a SET BROWSESYNC ON and the record pointer will go to user selected record.

I hope that some of this be useful for you.

[mrduck]

Thank you Roberto for your message.

You don't use vertical scrollbars, Pritpal in hbpbrowse uses ordkeyCount and doesn't care of eventually filtered-out records.

I have rarely used "browse" systems in my programs, the few times I did I created a temporary file with the filtered records or it was not possible to apply filters...

So, actually, does anybody use browse on a heavily filtered DBF ??? With heavily I mean a filter that results in 100 records out of a 100.000 records dbf...

Just some random ideas that could help to get a better one

HMG/WIN32 browse works as follows:

1. I've used a standard ListView control.

2. It is ALWAYS populated only with the records needed to complete the visible row count.

VIRTUALGRID is not "populated" since it uses the callback mechanism, and it is only requested the records that have to be displayed.

3. When the user scrolls (ie: 1 row down) the control is emptied and populated again, beginning at the second table record, until the visible row count be completed again.

4. Trapping cursor up, cursor down, page up, page down, ctrl-pgup, ctrl-pgdown and doing required record pointer movements (top, bottom, skip(+-n)) you'll cover all requirements of keyboard navigation. Regardless of how many records the table has, the navigation is nearly instantaneous, since the record movements are very fast always. This approach works fine and fast in all situations, including filtered tables.

5. The only thing that you must care about is the recno() of the first current row shown, since it will be your reference point to repopulate the control. Besides that, you must consider the situation when this record become out of the current filter, in such case, you must navigate the table until the next 'visible' record appears.

The nice thing is that QTableView takes care of all this "low-level" stuff... and life is too short to write yet another browse system...
Anyway I will probably use some ideas from you and Pritpal to update the code, but I don't want to rewrite everything, since it may not be used anyway (see introduction text of this message)

6. You always are working with a copy of the data that is updated only when the cursor is moved out of the visible-rows window or when a manual refresh is done.

I noticed this interesting part in the code, you have a transaction mechanism in place... you maty commit or rollback... I didn't notice if this supports SHARED mode dbf.

7. You don't need to know the total record count at all.

8. IMHO, this is a wonderful solution, excepting the 'vertical scrollbar problem'

Yep. As you can imagine, populating the control with the visible-row count only, the vertical scrollbar will never appear.

So, the vertical scrollbar is optional in Browse control and you must enable if you want it.

I've solved that, adding a separate scrollbar control to the right of the ListView.

A solution similar to Pritpal uses of ord Key Co..t(), that has the benefit o giving the scrollbar cursor appropriate dimension.

Coding it to be fast, is very problematic, since you need to know the total record count to make it work correctly and this is a nightmare (a slow one) for filtered tables, since, of course, the only way to know it is counting the records.

...

Some sort of caching may be introduced but it may give more problems than solutions...

9. Regarding the record pointer, it is saved before refreshing, and later restored, so there is no problem for the user. Optionally, you can do a SET BROWSESYNC ON and the record pointer will go to user selected record.

Can you please tell me more about SET BROWSESYNC ON ?




Anyway, the problem that I'm trying to identify, if it is really a problem or not, is due to the "async" nature of Qt. Imagine your nice grid on screen, with a button you start a long update procedure. The procedure does seeks, skips, replace (or simply adding fields to calculate a value), something that takes minutes to run.
Waiting for the result you start opening the browser and from time to time you flip from the browser to the application... well, when you come back to the application the grid is refreshed and the pointer is moved... during the calculation !!!!!! So the results can't be trusted at all !!!

There were two solutions applied so far:
concentra used shared mode, two indipendent recno(), one for the grid, one for the application. It is nice solution, but you have to know the shared mode pitfall (record locking/unlocking, a bit slower, etc)
ricci instead disabled the grid - I don't know which command he used exactly (I think at a couple of different solutions) but he forced the grid not to be updated: in this way he has complete control on the record pointer...


Both you and Pritpal take care of record locking, the original author of ViRTUALGRID did not.... I will add it later...


Last difference between your and Pitpal implementation versus VIRTUALGRID: if I have a empty DBF I have no cell shown... no empty cells, just a empty grid. This has a side-effect, for example a HMG3 program I could work on used double-clicking on the grid to activate a function... but without a grid it is impossible to double-click !!!

Thanks for your infos.

[Ricci]

ricci instead disabled the grid - I don't know which command he used exactly (I think at a couple of different solutions) but he forced the grid not to be updated: in this way he has complete control on the record pointer...

I tried to disable or freeze the grid, but i´m not sure if it was enough. A simple :enabled := .F. prevents the user to use the control with mouse or keyboard but what did the grid, if it gets an event to redraw from the operating system?

A command to freeze the grid would be great but i didn´t found anything in the Qt-docs.

[Roberto Lopez]

Thank you Roberto for your message.

You don't use vertical scrollbars, Pritpal in hbpbrowse uses ordkeyCount and doesn't care of eventually filtered-out records.

I've don't reviewed/tested it.

I have rarely used "browse" systems in my programs, the few times I did I created a temporary file with the filtered records or it was not possible to apply filters...

So, actually, does anybody use browse on a heavily filtered DBF ??? With heavily I mean a filter that results in 100 records out of a 100.000 records dbf...

I almost never use 'live' data. I always open the tables, make a 'query', then populate the grid and close the table, as if working with a remote data server (excepting small tables).

VIRTUALGRID is not "populated" since it uses the callback mechanism, and it is only requested the records that have to be displayed.

I know.

The nice thing is that QTableView takes care of all this "low-level" stuff... and life is too short to write yet another browse system...
Anyway I will probably use some ideas from you and Pritpal to update the code, but I don't want to rewrite everything, since it may not be used anyway (see introduction text of this message)

HMG data-bound grid, works the same (via callbacks). The only safe way to work is disabling the updates (it can be manually done via a method) when you must to be sure that the grid will be not manipulating the table.

I noticed this interesting part in the code, you have a transaction mechanism in place... you maty commit or rollback... I didn't notice if this supports SHARED mode dbf.

It supports shared mode.

A solution similar to Pritpal uses of ord Key Co..t(), that has the benefit o giving the scrollbar cursor appropriate dimension.

Sadly, not when filters are active. In such case, the vertical scrollbar will not be accurate (note that I've not reviewed the Pritpal code).

Can you please tell me more about SET BROWSESYNC ON ?

In standard browse, any time that the record pointer must be moved, its position is saved and when the operation is finished, it is restored, so, from an user POV, the browse operation does not alter record pointer position.

BrowseSync ON, simply forces to the record pointer to be moved to the position of the on-screen, interactively highlighted record (the same as old Clipper does).

Anyway, the problem that I'm trying to identify, if it is really a problem or not, is due to the "async" nature of Qt. Imagine your nice grid on screen, with a button you start a long update procedure. The procedure does seeks, skips, replace (or simply adding fields to calculate a value), something that takes minutes to run.
Waiting for the result you start opening the browser and from time to time you flip from the browser to the application... well, when you come back to the application the grid is refreshed and the pointer is moved... during the calculation !!!!!! So the results can't be trusted at all !!!

There were two solutions applied so far:
concentra used shared mode, two indipendent recno(), one for the grid, one for the application. It is nice solution, but you have to know the shared mode pitfall (record locking/unlocking, a bit slower, etc)
ricci instead disabled the grid - I don't know which command he used exactly (I think at a couple of different solutions) but he forced the grid not to be updated: in this way he has complete control on the record pointer...

I don't know anything about low-level RDD handling, so this is only a question:

Is there any way to delay user operations while the grid (the callbacks) are being processed?

Callbacks usually processing very fast, so, the user operations should be not affected.

Another question:

Can we use another workarea, specially/automatically created for the grid operation?

That workarea (of course) should be created with the exact indexes, filters, etc. to mimic the original.

In this case, the only operations on our original area, will be done when the user invokes the 'refresh' method.

Regarding the updates, a simple comparison will tell us if the original table was changed by other user from the last refresh (it could be easily implemented with a function).

All the changes on the grid will affect only to the alternate workarea, until a method ('save'/'commit') be called by the user.

Am I missing something?

Yes... how expensive is this in terms of time, memory, disk-space... I don't know... that is the reason because I've talked about 'experiment'

[Roberto Lopez]

Francesco,

Another approach:

- No 'workarea' property.

- You will need the following three instead:

- TableName
- OrderExpression
- FilterExpression

- Then, grid will create its own workarea with required index, without interfering with the user ones.

- Regarding updates, IMHO, the best way is to make them on a buffer and manually commit (as HMG.3 data-bound grid does).

To make it work the buffer properly, you only must to create an array of modified cells and 'feed' the repaint callback procedure with buffered data, instead 'live' data when the required cell is in the buffer. It is pretty simple.

At commit time, since it is manually done, all things are under control.

[mrduck]

Hi Roberto.

I will try to address a couple of points. First of all, I want to make clear that Qt is a very big beast... and I daily discover new capabilities and I'm always surprised that the things are done really well in the implementation.

Today I did some checks and discovered that, for example, when a cell content is changed, it forces a total refresh of the screen. This is ok for the secondary fields, but when you change the field of a primary key changing its ordkeyno() the screen is correctly refreshed but the cursor is not on the row that moved...




To create a parallel workarea, I don't see big problems, except that you need to open them in SHARED mode, and the application code must take care of this....

Qt, as said, is a big beast... hmg4 msgInfo creates a QMessageBox, fills the data and calls :exec(). QMessageBox is modal, but if you call :exec() the event queue of the parent window still runs... so, you think that the modal message box stops the recno() movements and instead it doesn't stop.... it may happen in other cases...