Scan Time Limit Exceeded

Hi @gmariani405,

Thanks for reaching out and checking through those initial debugging steps! Your scan time limit isn’t being exceeded, but instead, the scan fails to complete part way through. Please stop any currently running scans and, with debug mode still enabled, start a new scan. Once this fails, please use Email Activity Log to email a copy of the scan activity log to wftest @ wordfence . com.

Can you also send a diagnostic report to wftest @ wordfence . com? You can find the link to do so at the top of the Wordfence > Tools > Diagnostics page. Then click on “Send Report by Email”. Please add your forum username where indicated and respond here after you have sent it.

NOTE: It should look as follows – Screenshot of Tools > Diagnostic > Send by Email

Thanks,
Margaret

@wfmargaret I’ve gone ahead and done as you requested. Both emails should have been sent. Let me know what the next steps are (if any). Thanks!

Hi @gmariani405,

Thanks for sending those! From your diagnostics, you have overdue cron jobs. They’re only overdue by a day or so, so I imagine you have these configured to run daily from the server, but if not please double-check that these are running as intended.

Please try lowering the max_execution_time to 60 instead of 600. Your WP_MEMORY_LIMIT is currently set to 40M. Please increase this to 256M or higher. In Wordfence > Scan > Scan Options and Scheduling > Advanced Scan Options, please enable Use only IPv4 to start scans. Make sure to Save Changes after.

Once those settings have been adjusted, please try running a new scan. Let me know how it goes and if there are still any issues please send a new activity log to wftest @ wordfence . com and let me know here once you’ve sent it.

If scans are completing normally, make sure to disable Enable Debugging Mode.

Thanks,
Margaret

@wfmargaret So I wanted to test this a bit methodically to see what worked because what you suggested doesn’t make too much sense but I guess that really depends on how Wordfence works (which maybe you can explain). I tried on three sites i knew were failing to run reliably.

Site 1: Changed it to just use IPv4 only and it worked.
[Sep 09 14:01:25] Wordfence used 34.71 MB of memory for scan. Server peak memory usage was: 52.71 MB
[Sep 09 14:01:25] Scan Complete. Scanned 18070 files, 20 plugins, 2 themes, 32 posts, 0 comments and 13404 URLs in 6 minutes 13 seconds.

Site 2: Changed it to just use IPv4 only and it worked.
[Sep 09 14:08:16] Wordfence used 32.32 MB of memory for scan. Server peak memory usage was: 138.32 MB
[Sep 09 14:08:16] Scan Complete. Scanned 19903 files, 17 plugins, 2 themes, 43 posts, 0 comments and 24211 URLs in 5 minutes 39 seconds.

Site 3: Changed it to just use IPv4 only and it still failed. Bumped up WP_MEMORY_LIMIT to 64M, still failed. Bumped it up to 256M, and it worked.
[Sep 09 14:31:23] Wordfence used 32.7 MB of memory for scan. Server peak memory usage was: 144.7 MB
[Sep 09 14:31:23] Scan Complete. Scanned 20305 files, 21 plugins, 2 themes, 9 posts, 0 comments and 21327 URLs in 7 minutes 23 seconds.

Now, WP_MEMORY_LIMIT is per script, for front-end users. So if many scripts are running at once, they have the possibility to use up more ram than is actually available. So this SHOULD ideally be kept low as it would be cumulative. The WP_MAX_MEMORY_LIMIT constant specifically defines the maximum memory limit available when in the administration back-end. The default is 256M (256 megabytes of memory) or the original memory_limit php.ini value if this is higher.

So in theory, shouldn’t it have already been running at 256M? The WP_MAX_MEMORY_LIMIT is set to 2048M already. All three sites report Wordfence using less than 40M. On the sites I didn’t increase memory, it used well above 40M which would indicate to me that they weren’t limited to 40M.

I’m going to test changing the max_execution_time next and see if that’s a factor. Can you explain why WP_MEMORY_LIMIT would be a factor if this is running on the back-end in the meantime?

@wfmargaret So on Site 3 I ran a few different variations and they all failed after that one successful run.

256M, no IPv4, 600 timeout: worked once, failed second attempt
256M, only IPv4, 600 timeout: failed
256M, only IPv4, 60 timeout: failed
40M, only IPv4, 600 timeout: failed
40M, only IPv4, 60 timeout: failed
40M, no IPv4, 600 timeout: failed
40M, no IPv4, 60 timeout: didn’t test

I sent an email of the diagnostic data for the last failed run where i had all three suggestions enabled. There are two things I wanted to note though:

1. There are 46 cron jobs overdue, so maybe this backlog is affecting things?
2. The crons are handled via cron jobs which are set to run every 5minutes:
   */5 * * * * /usr/sbin/relax php -f /chroot/home/USER/DOMAIN.nxcli.io/html/wp-cron.php (Every 5 Minutes)

Let me know if any of this helps, thanks!

Hi @gmariani405,

Thanks for following up and I’m sorry about any confusion! You are correct in that if WP_MAX_MEMORY_LIMIT is set, then there’s no need to adjust WP_MEMORY_LIMIT specifically. Your current WP_MAX_MEMORY_LIMIT setting is more than high enough to accommodate the scan.

The scan has its own max execution time, so ideally max_execution_time wouldn’t need to be adjusted, but we have seen hosts where a high number can lead to issues in the scan.

Enabling Use only IPv4 helps in situations where the site may connect back to itself over IPv6 (such as when using Cloudflare), but the server it’s hosted on doesn’t support IPv6.

It’s possible the backlog of cron jobs could be negatively impacting the scans on the third site. Please try to ensure those are running and up-to-date. It may help to run the cron job command manually to ensure it’s not running into any errors. We also recommend using wget or cURL for the cron job command, as we have seen cases where using php has caused issues during automatic scans due to differences in cURL versions.

Once the overdue cron jobs have run, please try running a new scan. If it’s still failing, please enable debugging, run a new scan, and send me the activity log of the scan once it fails.

Thanks,
Margaret

I went to try again on Site 3, and when i looked it only have 6 cron jobcs backed up. I ran the scan and it worked fine. I updated plugins, ran the scan again it worked fine. I looked at the backed up cron jobs and it was empty. I had changed nothing since my tests yesterday. So maybe the backed up cron jobs may be a factor here. I’ll investigate some more.

Hi @gmariani405,

Thanks for the update! I’m glad to hear the scan and cron jobs are working on all three sites now. Please let me know if you start to run into any further issues!

Thanks,
Margaret

Tried on another site (Site 4) with 25 cron jobs pending, first scan failed. I manually triggered the cron and cleared out the pending cron jobs and it continued to fail on the second scan.

Site 3 i tried running it a few more times and it succeeded each time. Again I did nothing to change it since yesterday. One thing to note is the peak memory yesterday was almost 145MB when running. Now it barely cracks 50MB.

[Sep 10 10:42:19] Wordfence used 32.7 MB of memory for scan. Server peak memory usage was: 50.7 MB
[Sep 10 10:42:19] Scan Complete. Scanned 20306 files, 21 plugins, 2 themes, 9 posts, 0 comments and 21331 URLs in 7 minutes 27 seconds.

[Sep 10 11:03:08] Wordfence used 30.06 MB of memory for scan. Server peak memory usage was: 50.06 MB
[Sep 10 11:03:08] Scan Complete. Scanned 20299 files, 20 plugins, 2 themes, 9 posts, 0 comments and 24719 URLs in 8 minutes 15 seconds.

[Sep 10 11:31:14] Wordfence used 36.71 MB of memory for scan. Server peak memory usage was: 52.71 MB
[Sep 10 11:31:14] Scan Complete. Scanned 18071 files, 20 plugins, 2 themes, 32 posts, 0 comments and 13412 URLs in 6 minutes 10 seconds.

@wfmargaret Do you want me to send an activity log on Site 4 where it’s still failing?

Hi @gmariani405,

Absolutely, can you send me both the activity log with debugging enabled, as well as the diagnostics for the fourth site? Once you’ve sent those, please let me know here!

Thanks,
Margaret

@wfmargaret

Site 1: Succeeded, submitted diagnostics report and activity log (thorn…)
Site 2: Succeeded, submitted diagnostics report and activity log (bade…)
Site 3: Failed, submitted diagnostics report and activity log (mari…)
Site 4: Failed, submitted diagnostics report and activity log (mark…)

Hope this helps!

As a side note, I was working on another client site and saw this in the error log. Same host (nexcess), so this error might be related?

[18-Sep-2024 09:04:58 UTC] Cron unschedule event error for hook: wordfence_start_scheduled_scan, Error code: could_not_set, Error message: The cron event list could not be saved., Data: {"schedule":false,"args":[1726370400]}

Hi @gmariani405,

The log message you sent was added to?wp-cron?by WordPress recently, but isn’t related to a PHP error or a sign that cron is broken in some way. WordPress can sometimes start two copies of wp-cron to run the same jobs, triggered by multiple hits on the site starting close together. As wp-cron’s locking is imperfect, the jobs may run twice. When one finishes, it updates the list of cron jobs, and then when the other finishes, it’s unable to make the same update, because it’s already done. There was previously no log to report it, but now it’s intentionally logged in WordPress.

Regarding Site 3, you have High Sensitivity mode enabled and the most recent scan activity log you sent is of the scan timing out. Please try running a Standard scan with debugging disabled, which will allow it to complete faster. If that doesn’t help, please set the Time limit that a scan can run in seconds to 21600 (or 6 hours) to give the scan time to complete, especially if the site is larger.

Regarding Site 4, there are some overdue cron jobs (by a bit over 24 hours). Please make sure those are up-to-date and then try running a Standard scan rather than a High Sensitivity scan. I also recommend setting Maximum execution time for each scan stage to 20. From the activity log, the scan was running normally until one of the scan processes ended after forking. If the scan fails again, please send me a new activity log, as well as the server access logs for Site 4 from around that time. I’d like to check what response the scan process is seeing from the webserver.

Thanks,
Margaret

Sorry for the delayed reply, I don’t always have time to troubleshoot WordFence right away. I tested your recommendations on about 8 sites. Here were my results:

Site 1: High Sensitivity. Changed to Standard Scan and it scanned without issue
Site 2: Standard Scan. Scanned without issue
Site 3: High Sensitivity. Changed to Standard Scan and it scanned without issue
Site 4: High Sensitivity. Changed to Standard Scan and it failed. Changed max scan time from 0 -> 20, it failed.

Site 5: High Sensitivity. Changed to Standard Scan and it failed. Changed max scan time from 0 -> 20, it failed.
Site 6: High Sensitivity. Changed to Standard Scan and it scanned without issue
Site 7: Standard Scan. Failed. Changed max scan time from 0 -> 20, it failed.
Site 8: Standard Scan. Scanned without issue
Site 9: High Sensitivity. Changed to Standard Scan and it scanned without issue

Regarding Site 4, i sent the activity log, and diagnostic report and sent an email with the server logs to the wftest@ address.

Hi @gmariani405,

Thanks for following up! I wanted to make sure you’re aware, we typically stop monitoring topics if they haven’t been responded to in over 10 days. I noticed you responded here, so I’m happy to continue working with you here, but if you ever need to open a new topic with us in the future, please keep that in mind. To make sure we respond as soon as possible, you can always start a new topic and link the old one if you need to.

Thank you for sending us so much data for site 4. I don’t see any relevant errors in the error log, but there is a discrepancy in your access logs. I can see older scan forks in the access log, all of which received a 200 response, but in the most recent fork that failed (and the couple directly before it), there’s no matching entry in the access logs. For example, the access log entry for this scan fork is missing, even though I can see entries in the access log from before and after it:

Mon, 30 Sep 24 15:29:39 +0000::1727724579.3857:4:info::Starting cron with normal ajax at URL https://[domain.com]/wp-admin/admin-ajax.php?action=wordfence_doScan&isFork=0&scanMode=standard&cronKey=[cronkey]&signature=[signature]

Can you reach out to your host to see if there’s something that may have blocked this URL access before it reached the access logs or if there’s another reason why that entry is missing from the access logs? For NGINX servers, a copy of the NGINX access logs might have the relevant access log entry instead.

Thanks,
Margaret