Handling large volumes of rrequests via REST-API


  • The scenario is a third party notifies via WebHooks and in certain circumstances can send many hundred requests per second from a very powerful server. There is no control over the third party so cannot be changed

    When I built a REST-API end point to process the WebHook the database very quickly becomes overwhelmed hitting max_user_connections. Whist a solution would be to use a server that could be tuned, as a plugin developer I try to write solutions that don’t involve the majority of WP site owners upgrading their hosting.
    Optimized as best I can, light weight processing queuing the ‘real’ processing with Action Scheduler, I still can’t handle the spike pushes.

    It seems to me that as WP always creates a DB connection then whatever I do REST API will hit this issue. Is there something I am missing.

    The only solution I have at the moment is to write a completely ‘off WP’ end point that uses the file system to ‘cache’ the request body into files in a folder and then have an ‘on WP’ batch process process the events.

    This can handle the volume comfortably, but I’m not that happy having non WP software in a a WP plugin ( albeit not a WP.org plugin ) especially as ‘securing’ the point is harder. Thoughts?

Viewing 2 replies - 1 through 2 (of 2 total)
  • Moderator bcworkz

    (@bcworkz)

    Hi Alan,
    While an external app to act as a webhook request cache may be at odds with the WP paradigm, I don’t see that you have a choice. The process of invoking the entire WP environment for every request is simply too resource intensive for that level of traffic on typical servers.

    So you write requests to a file as a method of caching them. That’s fine if it works, but I wonder if independently connecting to a DB and keeping cached requests there would be more efficient. I’ve never personally compared the two for performance, but my understanding is DB writing is faster than file writing. But the overhead to connect may skew in file writing’s favor.

    But if it’s not broken, don’t fix it? ??

    At the risk of wandering even farther from the WP paradigm, would servicing the webhook requests be fast enough if you did what needs to be done directly without involving WP at all? Via mysqli_query() and its brethren for example. While clearly not WordPressy at all, it’d be much simpler than a caching scheme if it’d be fast enough.

    Thread Starter Alan Fuller

    (@alanfuller)

    Thanks for your input. I think that the DB max_user_connections would still be an issue with using the direct db as I am shipping as part of an existing plugin I would still be constrained to the WP database credentials. Adding a second db or using something like Golang/Bolt would not be feasible from a user base perspective, a large proportion don’t know where to add additional CSS let alone set up a second DB or install a non PHP app

    In terms of not using WP at all for the process, that might be an idea, but the max_user_connections on the WP db may still be the issue ( unless there is some way I can keep the connection open between requests without some server side object cache – but there is a thought ).

    Just in terms of context, the webhooks indicate something has changed externally so the process investigates a range of transients and destroy ones relevant to the change, thus ‘forcing’ a refresh. So still need to interact with the WP db.

Viewing 2 replies - 1 through 2 (of 2 total)
  • The topic ‘Handling large volumes of rrequests via REST-API’ is closed to new replies.