Fix confusion around reporting operations in progress. #658
Conversation
The SQLite refactor meant rewriting all the SourceTableArray loops to use a query result iterator instead. Well there is a place for a SourceTableArray loop still, and that's the pgcopydb list progress thing.
There was a problem hiding this comment.
It still has a problem majorly here,
pgcopydb/src/bin/pgcopydb/catalog.c
Lines 6910 to 6930 in ad8f20e
The sum functions are ran over all the rows and resulting a single row. Instead we should run it over groups by table oid or don't run sum at all if we are showing all parts.
The query was meant to report actual duration and transmitted bytes for each table in-progress, but the join and sum( 8000 ) operations are creating problems in the SQLite query and the rest of the progress.c code was not prepared to benefit from the extra information anyway. The SQLite problem is that the sum(s.duration), sum(s.bytes) causes a single row output when the dataset is empty, and the other columns are all set to NULL. That might be because of the lack of a GROUP BY clause. Simplify that aspect by removing the extra information for now.
Good finding! That might be why the query always returns a single row even when there is no table copy currently in progress. The solution looks indeed like adding a GROUP BY clause to the query, but here we want to track COPY progress, which is per-partition, not per-table. Looking at the consuming part of the code in |
The SQLite refactor meant rewriting all the SourceTableArray loops to use a query result iterator instead. Well there is a place for a SourceTableArray loop still, and that's the pgcopydb list progress thing.