CSV Import Speed

Hi @jimario,

Sorry that you ran into issues here!

The CSV import speed is not strictly a network issue. Because we’re a custom post type, each record in the CSV must have the following done:
– Create a BD listing (custom post type, essentially creating a record in WP)
– Confirm the category exists and associate it if it does. Create it if it does not
– Process any images you have associated with it and upload them through the media API

The WP API is quite frankly, not so fast with these operations. It would be faster if we could just jam data into custom tables but because WP does processing on things, their API calls are a bit more pokey than we’d like. But we’re stuck with them, unfortunately.

Other factors that can slow you down:
– Shared hosting
– Slow database query connection
– Lack of CPU resources

Does that make more sense?

Thank you for responding to my question. Yes, what you say does make sense. However, I did get an error and by the time I got to 12%. The error message said:

A fatal error occurred during the import. If connection wasn’t lost during the import, please make sure that you have enough free disk space and memory available to PHP. Check your error logs for details.

It looks like I was able to get 2,028 records imported out of 21,523. This was one of my smallest csv files. The average size of my csv files is 60-90mb. Typically, how fast does the import process take even if database query connection is quick & cpu resources are adequate?

Hi @jimario,

If your CSV files are 60-90MB, part of the issue is simply the upload aspect of it–that’s a lot of data to pull up and process in one chunk with PHP threads.

Try breaking up the import into smaller files–maybe no more than 10MB per file. That may help make this go more smoothly.

There’s no blanket answer on CPU there–there are other constraints that typically factor in that are more important (RAM being first, and by RAM I mean the RAM allocated to a PHP thread for processing. It’s not the same as the RAM you have for your web server instance…this is a setting in php.ini and usually controlled by the host). The database query timeout is another and that is the one that usually gets a lot of people, the error you’re seeing could be from that.