Tainacan: importing csv (possible bug)

(@batelada)

In “Media files, an infinity of repeated thumbnails are created, as well as copies of the csv files like “filename-7.csv”. I read something, somewhere, about exceding a certain time, while uploading. Do you think this could be the reason for the issue?

Thread Starter

(@batelada)

The problem definitely is not a problem in the csv. Separating the csv in small csvs makes the update work but it os not a solution when you have a csv with 1000+ items. Big images are a problem but when you have a photo catalogue it is inevitable to have them (and I don’t mean huge images). I suppose the problem is a kind of time limitation in the uploading which generates the loop that repeats the items in the collection and fills the media with the same files (photos and csvs) endlessly when you update the files.

Plugin Author

(@vnmedeiros)

Hi @batelada,
We have a problem with duplicating items in the import mapped here: https://github.com/tainacan/tainacan/issues/383 in the next version we should release a correction for this.
Basically this is because the WP cron calls the importer’s process again before the item is fully included in the database, in most cases the csv files being imported have many columns.
One way to mitigate this problem is to increase the time between the cron calls so that the inclusion of the item is completed.

Thread Starter

(@batelada)

Thank you very much for your kind help. Um abra?o muito agradecido.

Thread Starter

(@batelada)

Perfect. In a Library of 1000+ items I would be able to import 2 items and a third item would fall in the drafts as an incomplete item. Then, I would have the duplication of items. I couldn’t find if the problem is a wp-cron call, as the issue happens in just a few seconds (1 or 2). Before this I changed the cron job of the theme, I disabled the heartbeat… and finally, after knowing that this could be a wp-cron issue, I disabled the WP-Cron, by adding the line define(‘DISABLE_WP_CRON’, true); in wp-config.php, just before the line “That’s all, stop editing! Happy blogging.” It didn’t work.

I am still working in uploading my csv… and I couldn’t find a way to do it. I made many attempts and I already have 100 items in my collection (!!!). How? Trying to upload the csv, stopping the process after 2 seconds (after realizing it created the draft and will create duplicated items), deleting the uploaded items from the original csv and restarting the upload (also deleting the many duplicate imagens in the media library. If I made 400 attempts more I would have my csv uploaded… I can not do it.

I wonder how other institutions could upload their collections. Reducing the number of columns in a second collection actually made it possible to upload 3 items on each attempt. So, in just 10 attempts I could upload 30 items… of course I did it just for testing different changes in the csvs, in cron jobs and so on and it can not be used as a method! Any other suggestions on how to mitigate or contour the issue?

It would not solve the problem but it would be very useful, handy, if it was possible to save preferences when importing the csv files (delimiter, file encoding, repeated item preferences…) in stead of turning back to the defaults each time.

Plugin Author

(@vnmedeiros)

hi,
There is really something wrong in this case that we have not realized yet, the CSV importer should work with large amounts of data, by increasing the time of calls made by CRON the problem of duplicate items should be solved.
Could you send a part of the import log when an item is duplicated?

Thread Starter

(@batelada)

I upload the csv file and the 2 items are correctly uploaded. The third item falls into a draft folder: uncomplete. Duplicate copies of those items are created then until I stop the process.
This is the copy of the log, untill I stopped the process:

New Request
—————————
Starting processing new item
2021-02-09 19:56:47
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 47 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/Gram_tica_Elementar_de_Manipula__o_para_Bonecos_de_Luva_0.jpg
Setting item thumbnail: 33001
—————————
Starting processing new item
2021-02-09 19:56:49
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 48 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/O_Grotesco_no_Teatro_de_Bonecos_0.jpg
Setting item thumbnail: 33023
New Request
—————————
Starting processing new item
2021-02-09 20:01:56
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 47 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/Gram_tica_Elementar_de_Manipula__o_para_Bonecos_de_Luva_0.jpg
Setting item thumbnail: 33033
—————————
Starting processing new item
2021-02-09 20:01:57
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 48 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/O_Grotesco_no_Teatro_de_Bonecos_0.jpg
Setting item thumbnail: 33054
New Request
—————————
Starting processing new item
2021-02-09 20:02:59
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 47 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/Gram_tica_Elementar_de_Manipula__o_para_Bonecos_de_Luva_0.jpg
Setting item thumbnail: 33068
—————————
Starting processing new item
2021-02-09 20:03:00
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 48 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/O_Grotesco_no_Teatro_de_Bonecos_0.jpg
Setting item thumbnail: 33089

Anyway, the problem is not in the csv file. Each time I delete the uploaded items, I can upload 2 new items (the next goes, again, to the draft folder and the process is repeated). The duplicated images and the duplicated copies of the csv files are stored in the media library. In a fraction of a second the 1st items are upload, then in a slow loop the uploading is repeated… I disabled the wp-cron manually and then I installed a plugin named WP Crontrol disabling “running wp-cron”, changing Tainacan recurrences to “once hourly” and apparently nothing changed.

I am uploading here an image of the image duplication until I abort the upload process, just in case:
https://batelada.com/tainacan-issues/tainacan1a.jpg
https://batelada.com/tainacan-issues/tainacan1b.jpg
https://batelada.com/tainacan-issues/tainacan1c1.jpg
https://batelada.com/tainacan-issues/tainacan1c2.jpg

Thanks for the support and kindness!!!
PS: Should we write in Portuguese?

Plugin Author

(@vnmedeiros)

(sim, vamos escrever em português)
bom dia,
é definitivamente isso n?o é um problema com o CRON, o tempo de execu??o entre cada item indicado no log é bem curto e para gerar uma nova requisi??o ao processo (New Request) teve um intervalo de quase 1 minuto.

Pelo que vejo o processo de importa??o está perdendo os dados da última requisi??o (a cada nova requisi??o o “Processing item” é zerado). Se você deixar a importa??o executar novas requisi??es em algum momento o primeiro valor no log para o “Processing item” após uma linha “New Request” difere de zero?

Algumas quest?es para ajudar:
é uma instala??o simples ou multisite?
O usuário que inicia o processo é o administrador da instala??o?
A tabela wp_tnc_bg_process tá liberada para fazer updates?

Thread Starter

(@batelada)

1) Instala??o simples.
2) O usuário que inicia o processo é, sim, o administrador da instala??o!
3) N?o entendo se vc se refere a permiss?es dos arquivos wordpress (drwxr-xr-x) ou a modifica??es na “estrutura” da tabela wp_tnc_bg_process (neste caso n?o sei o que alterar ou verificar) no phpmyadmin

Mando print da tela do phpmyadmin na aba estrutura da tabela wp_tnc_bg_process:
https://www.batelada.com/tainacan-issues/tainacan issue2.jpg

4) Passo ao teste de “novas requisi??es”.

Plugin Author

(@vnmedeiros)

Bom dia,
tentei reproduzir o erro localmente aqui e sem sucesso.
Você tem acesso aos arquivos da instala??o do WP? Se sim, pode mostrar o que é exibido no log de erros do servidor? De preferência com essas duas op??es ativas no wp-config.php:

define('TAINACAN_DEBUG_BG_PROCESS', false);
define('WP_DEBUG', true);

att!

Thread Starter

(@batelada)

Tenho acesso aos arquivos.
Editarei o wp-config.php. e farei o teste. esse é o error-log ref ao dia 09-fev.

Este é o error-log sob wp-admin (parte, ref. ao dia 09-fev):
[09-Feb-2021 20:03:10 UTC] WordPress database error Commands out of sync; you can’t run this command now for query UPDATE wp_options SET option_value = ‘1612901050’ WHERE option_name = ‘action_scheduler_lock_async-request-runner’ made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_OptionLock->set, update_option
[09-Feb-2021 20:03:10 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT COUNT(DISTINCT claim_id) FROM wp_actionscheduler_actions WHERE claim_id != 0 AND status IN ( ‘pending’, ‘in-progress’) made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Abstract_QueueRunner->has_maximum_concurrent_batches, ActionScheduler_DBStore->get_claim_count
[09-Feb-2021 20:03:10 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT a.action_id FROM wp_actionscheduler_actions a WHERE 1=1 AND a.status=’pending’ AND a.scheduled_date_gmt <= ‘2021-02-09 20:03:10’ ORDER BY a.scheduled_date_gmt ASC LIMIT 0, 5 made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Store->has_pending_actions_due, ActionScheduler_DBStore->query_actions

Farei a altera??o do wp-config.pho e farei novo teste.

Thread Starter

(@batelada)

Em wp-config.php, Acrescentei as linhas logo antes de “That’s all!”.

Novamente foram importados 2 itens do arquivo csv e o terceiro caiu, incompleto na pasta draft, em uma fra??o de segundos. Pouco depois, o mesmo recome?a e itens duplicados s?o acrescentados e caem no draft, da mesma forma. Como anteriormente. O processo só é interrompido por mim, manualmente

OBS: uso a op??o ignore, em lugar de update na importa??o do csv no Tainacan.

Aqui o log do erro (error-log sob wp-admin):
[12-Feb-2021 21:16:54 UTC] WordPress database error Commands out of sync; you can’t run this command now for query UPDATE wp_options SET option_value = ‘1613164674’ WHERE option_name = ‘action_scheduler_lock_async-request-runner’ made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_OptionLock->set, update_option
[12-Feb-2021 21:16:54 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT COUNT(DISTINCT claim_id) FROM wp_actionscheduler_actions WHERE claim_id != 0 AND status IN ( ‘pending’, ‘in-progress’) made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Abstract_QueueRunner->has_maximum_concurrent_batches, ActionScheduler_DBStore->get_claim_count
[12-Feb-2021 21:16:54 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT a.action_id FROM wp_actionscheduler_actions a WHERE 1=1 AND a.status=’pending’ AND a.scheduled_date_gmt <= ‘2021-02-12 21:16:54’ ORDER BY a.scheduled_date_gmt ASC LIMIT 0, 5 made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Store->has_pending_actions_due, ActionScheduler_DBStore->query_actions
[12-Feb-2021 21:22:00 UTC] WordPress database error Commands out of sync; you can’t run this command now for query UPDATE wp_options SET option_value = ‘1613164980’ WHERE option_name = ‘action_scheduler_lock_async-request-runner’ made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_OptionLock->set, update_option
[12-Feb-2021 21:22:00 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT COUNT(DISTINCT claim_id) FROM wp_actionscheduler_actions WHERE claim_id != 0 AND status IN ( ‘pending’, ‘in-progress’) made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Abstract_QueueRunner->has_maximum_concurrent_batches, ActionScheduler_DBStore->get_claim_count
[12-Feb-2021 21:22:00 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT a.action_id FROM wp_actionscheduler_actions a WHERE 1=1 AND a.status=’pending’ AND a.scheduled_date_gmt <= ‘2021-02-12 21:22:00’ ORDER BY a.scheduled_date_gmt ASC LIMIT 0, 5 made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Store->has_pending_actions_due, ActionScheduler_DBStore->query_actions
[12-Feb-2021 21:27:01 UTC] WordPress database error Commands out of sync; you can’t run this command now for query UPDATE wp_options SET option_value = ‘1613165281’ WHERE option_name = ‘action_scheduler_lock_async-request-runner’ made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_OptionLock->set, update_option
[12-Feb-2021 21:27:01 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT COUNT(DISTINCT claim_id) FROM wp_actionscheduler_actions WHERE claim_id != 0 AND status IN ( ‘pending’, ‘in-progress’) made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Abstract_QueueRunner->has_maximum_concurrent_batches, ActionScheduler_DBStore->get_claim_count
[12-Feb-2021 21:27:01 UTC] WordPress database error Commands out of sync; you can’t run this command now for query SELECT a.action_id FROM wp_actionscheduler_actions a WHERE 1=1 AND a.status=’pending’ AND a.scheduled_date_gmt <= ‘2021-02-12 21:27:01’ ORDER BY a.scheduled_date_gmt ASC LIMIT 0, 5 made by shutdown_action_hook, do_action(‘shutdown’), WP_Hook->do_action, WP_Hook->apply_filters, ActionScheduler_QueueRunner->maybe_dispatch_async_request, ActionScheduler_AsyncRequest_QueueRunner->maybe_dispatch, ActionScheduler_AsyncRequest_QueueRunner->allow, ActionScheduler_Store->has_pending_actions_due, ActionScheduler_DBStore->query_actions

Aqui, o log do erro, após a interrup??o do processo em Tainacan>repository>activities>processes:
New Request
—————————
Starting processing new item
2021-02-12 21:16:44
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 49 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/A_Dan_a_e_a_Forma__o_Corporal_do_Ator-Manipulador_0.jpg
Setting item thumbnail: 33121
—————————
Starting processing new item
2021-02-12 21:16:45
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 50 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/I_Semana_de_Estudos_sobre_o_Teatro_de_Anima__o_0.jpg
Setting item thumbnail: 33146
New Request
—————————
Starting processing new item
2021-02-12 21:21:49
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 49 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/A_Dan_a_e_a_Forma__o_Corporal_do_Ator-Manipulador_0.jpg
Setting item thumbnail: 33161
—————————
Starting processing new item
2021-02-12 21:21:50
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 50 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/I_Semana_de_Estudos_sobre_o_Teatro_de_Anima__o_0.jpg
Setting item thumbnail: 33185
New Request
—————————
Starting processing new item
2021-02-12 21:26:50
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 49 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/A_Dan_a_e_a_Forma__o_Corporal_do_Ator-Manipulador_0.jpg
Setting item thumbnail: 33198
—————————
Starting processing new item
2021-02-12 21:26:51
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 50 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/I_Semana_de_Estudos_sobre_o_Teatro_de_Anima__o_0.jpg
Setting item thumbnail: 33222
New Request
—————————
Starting processing new item
2021-02-12 21:31:51
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 49 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/A_Dan_a_e_a_Forma__o_Corporal_do_Ator-Manipulador_0.jpg
Setting item thumbnail: 33236
—————————
Starting processing new item
2021-02-12 21:31:52
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 50 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/I_Semana_de_Estudos_sobre_o_Teatro_de_Anima__o_0.jpg
Setting item thumbnail: 33260
New Request
—————————
Starting processing new item
2021-02-12 21:36:52
User in process: 1
Processing item 0
Processing item on line 2
Target collection: 22831
Success processing index: 0
Inserting item 0
item with ID 49 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/A_Dan_a_e_a_Forma__o_Corporal_do_Ator-Manipulador_0.jpg
Setting item thumbnail: 33271
—————————
Starting processing new item
2021-02-12 21:36:53
User in process: 1
Processing item 1
Processing item on line 3
Target collection: 22831
Success processing index: 1
Inserting item 1
item with ID 50 not found. Unable to update. Creating a new one.
Document file URL imported from https://biblio.batelada.com/I_Semana_de_Estudos_sobre_o_Teatro_de_Anima__o_0.jpg
Setting item thumbnail: 33295

Ainda n?o sei como agradecer tanto suporte e tanta aten??o ao problema que venho enfrentando… Mas estou imensamente agradecido.

Plugin Author

(@vnmedeiros)

Boa tarde @batelada,
fizemos algumas atualiza??es no plugin que devem ajudar a entender melhor como resolver esse erro, se possível pode atualizar e fazer o seguinte:

atualizar o Tainacan para a vers?o 0.17.4
limpar todos os transient do seu banco, isso vai for?ar ele atualizar o cache do WP (pode ser a causa do erro “out of sync”), SQL para fazer isso:
DELETE FROM wp_options WHERE option_name LIKE ('%\_transient\_%');

ativar os logs da importa??o adicionado no wp-config.php:

define('TAINACAN_DEBUG_BG_PROCESS', true);
define('WP_DEBUG', true);

Fazer uma importa??o de um CSV simples e verificar os Logs gerados

* Recomendo desativar outros plugins que estejam ativos, pelo log enviado parece que existem outras atividades de outros plugins causando o erro “out of sync”.

This reply was modified 4 years, 1 month ago by vnmedeiros.
This reply was modified 4 years, 1 month ago by vnmedeiros.
This reply was modified 4 years, 1 month ago by vnmedeiros.

Thread Starter