Errors during WP Statistics import

Hi @koobm, thanks for the bug report! We’ll create a fix that should be released relatively soon (days at the latest).

It took a bit longer than expected, but the issues with the WP Statistics importer should be fixed in version 5.1.6.

Hey there,

Thanks, highly appreciate the quick response and your efforts! But I’m afraid that it didn’t work out as expected.
I reinstalled Matomo on my last site which hasn’t been migrated yet and ran the importer. This is the full log (DB name replaced) – after the first day it tried to import, it’s frozen:

[info] Starting wp-statistics import
[notice] Importing data for date 2023-11-16...
[info] Importing data for the PagesImporter plugin.
[debug] Import 0 search keywords...
[debug] Import 10 global pages...
[debug] Import 20 page titles...
[info] Importing data for the ReferrersImporter plugin.
[debug] Import 2 referrers...
WordPress database error Table '<DATABASE_NAME>.wp_statistics_search' doesn't exist for query select engine, count(visitor) AS nb from wp_statistics_search where last_counter = '2023-11-16' group by engine order by engine; made by include('phar:///usr/bin/wp-cli/php/boot-phar.php'), include('phar:///usr/bin/wp-cli/vendor/wp-cli/wp-cli/php/wp-cli.php'), WP_CLI\bootstrap, WP_CLI\Bootstrap\LaunchRunner->process, WP_CLI\Runner->start, WP_CLI\Runner->run_command_and_exit, WP_CLI\Runner->run_command, WP_CLI\Dispatcher\Subcommand->invoke, call_user_func, WP_CLI\Dispatcher\CommandFactory::WP_CLI\Dispatcher\{closure}, call_user_func, WpMatomo\Commands\MatomoCommands->importWpStatistics, WpMatomo\WpStatistics\Importer->import, WpMatomo\WpStatistics\Importer->import_day, WpMatomo\WpStatistics\Importers\Actions\ReferrersImporter->import_records, WpMatomo\WpStatistics\Importers\Actions\ReferrersImporter->import_search_engines, WpMatomo\WpStatistics\Importers\Actions\ReferrersImporter->get_keywords_and_search_engine_records
[debug] Import 0 search engines...
[info] Importing data for the UserCountryImporter plugin.
[debug] Import 2 countries...
[debug] Import 6 regions...
[debug] Import 9 cities...
[info] Importing data for the DeviceDetectionImporter plugin.
[debug] Import 5 browsers...
[debug] Import 4 platforms...
[info] Importing data for the VisitorsImporter plugin.
[debug] Import 12 visits...
[info] Importing data for the VisitsTimeImporter plugin.

I found this commit in WP Statistics where they removed the whole DB table. Seems like that change was delivered with version 14.11 (the one in which they also announced the “Pro” version of their product), which according to their Github was rolled out 2 weeks ago.

I actually don’t care much about the referrer history, but the importer being frozen after day 1 is a real problem.

@koobm apologies for the continued inconvenience, we’ll try to take care of this soon (today hopefully).

Hi @koobm, would you be willing to test a pre-release version of the plugin to ensure the importer will work for you?

Yes, I could do that.

Hi @koobm, here is a pre-release with the import fixes: https://github.com/matomo-org/matomo-for-wordpress/releases/download/5.1.7-b1/matomo.zip . Does it solve the import problems for you?

Hi @dizzyatinnocraft, I can confirm this version fixed the freezing issue after import of the first day. All relevant days (a full year) were imported within roughly 30-45 minutes, no error messages occurred anymore. The importer looks good now from my perspective. Thanks a lot!

Unfortunately my SSH connection was lost and the process cancelled during the report generation phase (somewhere around 60-90 minutes in). Not sure if this is an actual problem, the daily/weekly/monthly visitor info is all there. Only some special reports (e.g. time of day; devices) look incomplete. Not too much of an issue, I could live with that. Just out of curiosity, I found that in Matomo onPremise the report generation can be re-initiated. Is there also a way to do so in the WordPress version?

Thanks and all the best!

Update: Now the log periodically shows errors from the cron run.

archive_main	2024-11-21 15:52:40	1 total errors during this script execution, please investigate and try and fix these errors. => CronArchive.php:473; CronArchive.php:468; CronArchive.php:226; Access.php:567; CronArchive.php:221; ScheduledTasks.php:378; class-wp-hook.php:322; class-wp-hook.php:348; plugin.php:565; wp-cron.php:191;
archive_errors	2024-11-21 15:52:40	'Got invalid response from API request: ?module=API&method=CoreAdminHome.archiveReports&idSite=1&period=year&date=2024-01-01&format=json&trigger=archivephp. Response was \'{"result":"error","message":"Label column not found in the table to add in addDataTable(). Row: array ( \'label\' => NULL, \'nb_visits\' => 1, \'nb_uniq_visitors\' => 1, ) - in plugin Actions."}\'' '1 total errors during this script execution, please investigate and try and fix these errors.' => ScheduledTasks.php:392; class-wp-hook.php:322; class-wp-hook.php:348; plugin.php:565; wp-cron.php:191;

Now the log periodically shows errors from the cron run.

Ah, those errors point to another bug in the import (again, apologies). You’ll have to re-run the import with another pre-release that fixes this issue (this should be the last problem I think).

Unfortunately my SSH connection was lost and the process cancelled during the report generation phase (somewhere around 60-90 minutes in).

If the process was cut off, it would just get re-started by wp-cron eventually so there’s no issue there. It might be a good idea to run the import command through nohup next time if possible to make sure it keeps running even if your connection is lost.

Only some special reports (e.g. time of day; devices) look incomplete.

Would you be able to explain how they look incomplete? We can try and fix this in a future version.

Just out of curiosity, I found that in Matomo onPremise the report generation can be re-initiated. Is there also a way to do so in the WordPress version?

Triggering archiving manually can be done via the Troubleshooting tab in Diagnostics. But if you want to completely re-process reports, you’d need to invalidate the periods you want to re-process first. To do this, you can use the “Invalidate Reports” plugin (available through the Matomo Marketplace).

We’ll try and get a new pre-release up soon, thanks for giving your time to test this version, it’s very much appreciated!

Hi @koobm, one more version to test if you are able to: https://github.com/matomo-org/matomo-for-wordpress/releases/download/5.1.7-b2/matomo.zip . After running the import there shouldn’t be any archiving errors. Thanks again for your help.