[Plugin: CSV Importer] [Plugin: CVS Importer] Large CSV files taking 100x longer than normal

I did some more digging and noticed mysql was always taking up 128MB of system memory when it was working which led me to thinking that maybe this wasn’t enough to handle larger files. So I increased my key_buffer_size from the default of 128M up with 512M in the my.cnf and importing 800KB files worked like a charm. gonna see how big I can go now ??

So far so good..

Here are some benchmarks / stress tests
I’ve imported a 1.6MB file with 800 posts in 64 seconds.
I’ve imported a 2.5MB file with 1200 posts in 103 seconds
I’ve imported a 3MB file with 1500 posts 133.03 seconds.
I’ve imported a 5MB file with 2500 posts 203.54 seconds.
I’ve imported an 8.4MB file with 4000 posts in 324.42 seconds.
I’ve imported a 10MB file with 5000 posts in 410.57 seconds.

I stopped after the last one as it seems that with 512 MB of key cache available this thing can handle just about anything

Although the weird thing is that after it’s finished it reports 1599 posts imported even though there’s only 800 new posts on the blogm or 4999 when I import 2000, etc etc

With all the larger tests my Proc size stays around 469MB and my Resident Mem Size around 100MB

I’ve scaled it down and can happily report that with 384M for my key_buffer size I’m able to import 6000 posts at once as long as they are going straight to drafts.. If they aren’t going to drafts the whole thing hangs.

netflow: Thanks for the report. If you take a look at the plugin’s source, you’ll see that the plugin handles drafts exactly the same way it handles published posts. I think this means one of two things: either you have other plugins that cause WordPress to hang on import or there’s some logic in WordPress itself that causes it to hang.

It would be nice if you could zip up and upload the files you’re using for testing somewhere – I’d be interested in running these tests on my machine, too.