improve WP memory usage

I know WP has some issues they are addressing post 2.5. I want to bring up another big issue though that I don’t see getting much attention in the forums.

I really think its time wordpress redoes its options table. I’ve discovered some things about memory usage and wordpress. When WP starts, one of the first things it does is load the options table into a GLOBAL variable. Maybe a long time ago this seemed like a good idea. However, this technique combined with how plugin authors have been told to use the options table has lead to a really bad WP memory issue.

WP recommends update_option and get_option to plugin authors. Unfortunately, a lot of plugin writers haven’t thought about the implications of this. For example, the cforms plugin uses the options table to store every detail of form data. Everything from all the form vars, to email message template.

Proof Test
This test will work on any existing blog. For best proof tho, try this on a fresh wp 2.5 install. No plugins, no articles. Just a default setup.

open index.php and add this line.
echo "memory used before WP init: " . memory_get_usage() . "<br />";

now. create a plugin. here is code. Once activated, this plugin will grab the memory being used by WP after init. I echo this in the footer of the page. For kicks, i’m printing $GLOBALS just so you can see how out of control the $GLOBALS var is.

<?php
/*
Plugin Name: Mem Usage
*/

function mem_init() {
  $GLOBALS["memused"] = "<br />memory usage after wordpress init: " . memory_get_usage() . " Bytes<br />";
}

function mem_foot() {
  echo $GLOBALS["memused"];
  echo "<br />memory usage after wordpress is done: " . memory_get_usage() . " Bytes<br />";
  print_r($GLOBALS);
}

add_action("init", "mem_init");
add_action("wp_footer", "mem_foot");
?>

Ok. now for some analysis. On my test setup, I get the following from viewing the blog front page.
memory used before WP init: 11936
memory usage after wordpress init: 6211472

So before WP loads, php is using a mere 11Kb. After it is using 6.2MB. Why is this bad? An understanding of how apache and php work is necessary. Apache’s has its own memory footprint. It creates threads to answer http requests. Each thread stays alive for a certain amount of time or requests filled. (my server, an apache process stays open for around 1000 requests) the way apache gives memory to each thread is where we run into trouble. It will give a thread the most memory it needs. It allows this memory to grow as needed. However, it does not release this memory. It holds onto the memory for the life of the thread. This saves reallocating memory costs.

So now for php’s play into this. say a fresh apache thread is 20M of memory. a php load that does nothing more than output text will use 11-12 KB of memory. so my fresh apache thread is now 20.12M with wordpress however, this shoots to 6.2M. so my apache thread is now 26-27M. Ok, this doesn’t break the bank on most sites. However, add a few plugins and add some traffic and you will find your site very quickly slow to a crawl as system memory is exhausted.

Lets now test this by adding a popular plugin. Try adding cforms. don’t even bother setting it up. just activate it. Now view the site.

On my test site, memory usage on the blog front page is now
memory used before WP init: 11936
memory usage after wordpress init: 6701800 Bytes

Whoa. We just shot up by 500K. Pre WP 2.5, having several cforms would jump memory a significant amount. the cforms author has fixed that though in his latest version. His plugin is still using too much memory though and its because WP encouraged folks to use the options table. (or we could say WP isn’t teaching people to use the options responsibly.)

Now on most systems, the default max memory php is allowed is 8M. A default WP install + cforms is already approaching this limit. And this is with no articles and only in the init stage. WP grabs more memory as it does its work looking for articles. Add a few plugins and its likely you will see memory usage shoot passed 10M.

So we come to my recommendation. The big problem is that WP loads the entire options table on init into $GLOBALS. WP itself has misused the options table recently with the all those vars that start with rss_ (i think related to the dashboard. i’ve seen forum posts on this. hopefully they are fixing it) as of this writing, my options table is 489K in size. I think 450K or so of it is those rss_ lines. I think WP needs 2 options tables. A table for WP to use for its options that it needs to use for WP to work. Then a table for all extras. I think the get_option function for this extra options table should not store the option in a global. I think it should just do the sql lookup each time. I know this could be asking a little more out of mysql, but mysql caches these query types anyway. We are doing more harm storing all these options in memory than we would by letting mysql do its job. That is just my opinion. All I really care about is that the WP authors and the WP plugin community wisen up to the effects on memory we are having. There is a responsibility of WP to fix this issue. There is also a responsibility for better programming practices by WP plugin authors.

A note. this message follows a lot of work I did recently on a site with server issues. On this site, I still see apache threads using upwards of 50M of memory. Most threads avg 35M of usage. His 50k users per day really makes this memory issue really obvious. I’ve got to imagine even wordpress.com would benefit greatly from addressing this issue.

another big memory issue. this is prolly better saved for another topic. but in summary.. wordpress’s sql functions. The get_results method loads every single sql result into an array. This is awful. Why are we not looping thru using mysql_fetch per row. doing something with that row, then get the next row. WP would vastly improve performance by simply improving its usage of sql get functionality in the displaying of posts and comments.