Expected data must be a string or stringable while importing large CSV

  • Resolved Manoa

    (@manoaratefy)


    10 months, 1 week ago

    Hello,

    While importing relatively big CSV on lists (big is relative to server’s performance and internet connection performance), at some point the import stop with error “Expected data must be a string or stringable”.

    While checking at browser requests, we see that these errors happens randomly on the wp-json/fluent-crm/v2/import/csv-import HTTP request.

    After digging, we understood the problem. Your cron action “fluentcrm_scheduled_hourly_tasks” is calling FluentCrm\App\Hooks\Handlers\Scheduler@maybeCleanupCsvFiles which delete the CSV file before the end of importation.

    We currently just changed the CSV to be deleted after 1 day of creation time, instead of 15 minutes (wp-content/plugins/fluent-crm/app/Hooks/Handlers/Scheduler.php line 120).

    Could you please fix this by implementing a proper way to know if a CSV file is still used for importation or not?

    Best regards.

Viewing 2 replies - 1 through 2 (of 2 total)
  • Plugin Author Shahjahan Jewel

    (@techjewel)

    We have fixed the issue in the lastest version.

    Thread Starter Manoa

    (@manoaratefy)

    Hello sir,

    I would like to thank you for your update. However, this is still not enough.

    I’ve seen that you updated it from 15 minutes to 30 minutes. However, 30 minutes is not enough for importing a very large database (mine is likely at millions of contacts to import).

    You didn’t provide also way to import more than 500 contacts per AJAX query (at least, this is not directly customizable from the plugin user interface). This is slowing the process of importation, my server is able to handle more than 500 per run (tweaked memory_limit and max_execution_time).

    With very simple maths, if each AJAX request would be accomplished in 1s (average, as it depends on server performance and internet connection performance), 30 minutes will allow only 30 minutes x 60 seconds x 500 contacts = 900 000 contacts max. per CSV. We still not computing time loose between requests.

    I’m still thinking you should implement a proper way to let know your plugin if a CSV file is used or not by any running import process, and only delete those who are not used anymore.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Expected data must be a string or stringable while importing large CSV’ is closed to new replies.