[Plugin: CSV Importer] Import Large File Issue

Update: I setup WP 2.9.1 on my GoDaddy hosting account and ran several tests with different CSV files and all tests were successful. My largest test had 100 rows with 8 custom field columns.

Not sure why the plugin works with GoDaddy and not with WAMP but it does WORK!

Thanks so much for this plugin. The possibilities are endless.

joshstauffer: You’re welcome. Unfortunately, I don’t have a WAMP environment where I can test the plugin, but I’ll check to see if the plugin uses any Unix-specific functions or file name conventions before the next release.

I run WP 2.9.1 on a WAMP windows server. Csv Import plugin works fine.

however, i’m on a Mac , WP 2.9.1 MAMP (latest) and it worked 2 weeks ago,
has now stopped working, entirely. no errors… it just does not import.

I believe we did a fresh install of WP 2.9.1 on this machine….. and then installed plugins and imported all posts.
Upon initial setup, we deleted all posts, and when all items were ready, were getting ready to import again.

I found that NO importer works… csvpig, import csv, csv2post now.
Something is broken in the WP install itself I guess.
I cleard the sql databases of all traces of past csv options, still no go.

Does anyone know what might cause this type of problem.

on the windows, Wamp setup, the csv importer works fine. For now we import there, then export as wordpress export, then import into our main Mac server.

Help?

I have taken the sample.csv file and copied and pasted the 5 rows to make 100 rows. When I do the import, only 39 records were imported and there is no success or failure message after clicking import.

I think I’ve discovered the answer to my own question. I’m assuming only 39 records were imported because the script timed out so what you can do to import 100’s of posts is mess the PHP’s max execution time. I think there are a few ways of changing this value but the simplest approach I’ve found is to add this line to the .htaccess file:

# Sets max execution time to 2 minutes
php_value max_execution_time 120

Hope someone finds this useful!

joshstauffer: Thanks for the info! I did some testing and it looks like some of WordPress’ functions became significantly slower in 2.9. That would explain recent bug reports where people don’t get the confirmation message and not all of their data is imported.

I imported 505-line CSV file 3 times using the latest version of the plugin, 0.2.2, to WordPress 2.8.2 and WordPress 2.9.1. Here are the numbers:

WP 2.8.2 – 42 seconds (first import), 116 seconds (third import)
WP 2.9.1 – 103 seconds (first import), 184 seconds (third import)

Each successive import was slower than the last one. Testing was done on my laptop – I suspect the times will be higher on a shared server. I wouldn’t call 42 seconds fast, but it’s sort of acceptable for 500+ rows. 100+ seconds is pretty damn slow, so I’ll see what I can do to improve the plugin’s speed.

Hi guys: I can only load around 355 rows at once. I’ve tried multiple tests…I can’t load 400 rows at once. I’m using WP 2.9.1 and plugin, 0.2.2.

I’m using WP-DBManager (for backing up) so my .htaccess is in wordpress/wp-content/backup-db/ (is this ok?).

And, it only has this…

<Files ~ “.*\..*”>
order allow,deny
deny from all
</Files>

I added this…but, it’s still not working…any ideas? Did I do this right? Thanks!

<Files ~ “.*\..*”>
order allow,deny
deny from all
</Files>
# Sets max execution time to 2 minutes
php_value max_execution_time 120

Regards,
Mitch

First of all, I think this is a great plugin. However, I’m having the EXACT same problem as Mitch above. I can import about 356 posts, and then I get a 500 internal server error. Strangely, if I import a CSV with only 83 posts, the plugin works just fine. I’m on GoDaddy Deluxe Linux hosting with WP 2.9.2 (also had the same problem on 2.9.1) and plugin version 0.2.2. I tried adding “php_value max_execution_time 120” to my .htaccess file in the site root, but it didn’t help. I also created a php5.ini file (per the GoDaddy instructions) and added these values:

max_execution_time = 600
max_input_time = 30
memory_limit = 256M

But it still doesn’t work. Any assistance would be greatly appreciated. Thanks in advance!

Just Noticed 506 Post Revisions:
I’ve been using the Bulk Delete plugin to delete Posts so I can keep testing the Importer and I noticed on this latest delete screen it was showing:

“Select the posts which you want to delete”
“All Revisions (506 Revisons)”

I don’t import as Drafts…so this was not there before because I’ve been testing for several days. I wonder if the Importer bringing Posts in as Revisions somehow contributes to the problem. -Mitch

It seems changing max_execution_time in the .htaccess or php5.ini isn’t making a difference with GoDaddy. When I posted that solution I was testing with WAMP. I ran several tests with GoDaddy and no matter what I set the max_execution_time to, the script stopped around the same time (after 60-80 seconds). This makes me wonder if WordPress has a built in script time limit somewhere.

I’m not done trying to figure this out but I’ve temporarily moved on. I am importing close to 2,500 posts and the only way I can get them all imported is to split the CSV file. I searched for a CSV splitter and found one here. I don’t like the idea of splitting up my CSV file but whatever it takes to get my posts imported. I guess the alternative would be to put them in manually. Heck no! ??

Ugh, still haven’t had the time to look into this more carefully. Did find a nice PHP profiler to figure out what exactly is causing the plugin to run so slow. Please bear with me.

On the other hand, no matter how slow the plugin runs, you should be able to adjust script execution time on your server. Setups differ from hosting to hosting, so make sure you’re adjusting PHP settings where you’re supposed to adjust them (ask your hosting).

joshstauffer: I would try and set max_execution_time to a small value – 10 or 15 seconds – and see if it makes any difference. If it has no influence on script execution time, then you’re probably doing something wrong.

quick fix. just add the following script around line 179 of cvs_importer.php:

set_time_limit ( 120 ) ;

Set the 120 num to your desired run time in seconds.

Great plugin btw! I’ve been a .NET developer for years, but I’m new to WP & PHP. Glad to be taking advantage of such a community driven technology.

How can I add html formatting, hyperlinking, and images to the content column of my cvs?

As it stands, I can only import text posts.

Please help! (oh, great tool BTW)

mapmedia: I’m able to import posts with HTML markup without any issues. Make sure that quotes inside HTML tags are properly escaped in your CSV file and there are no WordPress settings that strip HTML from posts.

Please don’t hijack old threads. HTML formatting has nothing to do with importing large files.