HOW TO: Exporting A LOT of Data (Out of Memory Issue)

Hi Eli,

This looks like a pretty complete solution to the memory issue.

The plugin did already feature a export limiter – which would allow you to break the export into smaller batches, and control the rows exported using an offset option.

If there is a wider need for this solution, I’ll review the code locally and consider inclusion in a future release.

Cheers

Q

Thanks for posting this, I was running into this problem and this helps. I _think_ I never saw it before Buddypress 2.0 came out. The release notes say they changed some things about how caching works with that release. Anyway, nice fix.

@grexican,

In your switch case in the return_bytes method, no matter the entered value the output is multiplied by 1024 ( $val *= 1024 ) – am I missing something?

Also, the first ob_end_flush() – please explain why that is required?

Cheers

Q

It’s a fall-through. So in the case of ‘g’ (gigabytes) the value (let’s say 1g) gets multiplied by 1024 * 1024 * 1024. So 1 * 1024 * 1024 * 1024 = # of bytes in a gig. 1 * 1024 = # of megabytes in a gig; again * 1024 = # of kb in a gig; again * 1024 = # of bytes. We need it in bytes because that’s what memory_get_usage() returns.

ob_end_flush() is for good measure. On low-memory systems, you can fill up the output buffer when you echo things. ob_end_flush() says that each echo is written to the output stream immediately instead of buffering. This takes the weight off the memory limits and also starts dumping data to be downloaded as soon as it’s ready. It’s also slightly slower than buffered streams because of the context switching between the PHP call to write and the actual writing, but the tradeoffs are worth it for file downloading, imho.

@grexican – all makes good sense – thanks, I’m reading over all the new options, tidying up and testing things – I might push an update onto git later today – if you’d all be up for testing from there?

Cheers

Q

Sure, I’ll definitely test it for you, but I unfortunately won’t be able to until next week. I’m about to head out for a few days for a business conference and won’t get back until Saturday.

@grexican – No problem, better if I take a bit of time and do more testing.. I’ll push a few changes to Git and you can grab it when you like ( https://github.com/qstudio/export-user-data ) – it’s well out of date there now..

Sounds like a plan. I’ve starred the project. Let me know when you’ve updated the repo and I’ll take it for a test run with my larger-than-life export.

Ok – first push is up:

https://github.com/qstudio/export-user-data

There are a few bits missing and a few changes, mostly to the saving of previous export data:

– I moved this from the wp_options table to the wp_usermeta – this will allow several users to store data on the same install
– I moved some of the saving and retrieving options around in the layout and reformatted the methods a little, to match the rest of the plugin
– I had to comment out the array data sanitizing lines for now, noted with an @todo – @cwjordan, please can you take a look at this, as it’s including the saved array as a key within itself?

I tested a 1500 member export and all the saving and loading functions, but most testing is required before this goes up on WP.

Any questions, just ask.

Q

Hi Q,

I’m back from my trip. I’ll try out your plugin changes sometime this week.

Also, since you’re adding save changes, I think adding a “select all” feature would be very much appreciated.

In my case, when I wrote this little JS snippet, it didn’t save changes and I didn’t need the fancy selector. So I just used JS to hide it and add a “select all” feature with jQuery. Adding this feature natively would be a big help I think. We have hundreds of fields.

Here’s my jquery. you should be able to easily adapt it with a static element and then call and “update” to the field picker UI to refresh based on the hidden field’s data:

setTimeout(function()
		{
			$('#bp_fields').css('position', 'static !important').css('left', '');
			$('#ms-bp_fields').hide();

			var $bpAll = $('<span> <a href="#" onclick="return false;">Select All</a></span>').on('click', function()
			{
				$('#bp_fields option').prop('selected', true);
			});

			$('#bp_fields').after($bpAll);

		}, 50);

@grexican – I’ve added a simple “All” or “None” select option for each list – this does not play nicely with the filter yet ( be nice if you could filter, then select all – but perhaps this will never get used anyhow.. ).

I’ve pushed this change back up to github

Q

@grexican – I’ve got a question, how could I contact you?

See here for reference: https://wordpress.stackexchange.com/questions/167111/wp-user-query-and-non-unique-usermeta-data

Cheers

Q

eli dot gassert
at toad-software dot com

Thanks!