Batch adding images via API causing 504 errors

  • Resolved eclev91

    (@eclev91)


    4 years, 10 months ago

    Hey there,

    Having an issue with a REST API call that’s resulting in a 504 timeout. Body of the request:

    [
    {
        "update": [
            {
                "id": "6664",
                "images": [
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/95de7e34a9d3bc1e6e3c25034fecf019/mockup-3742648c.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/892a1e5f35b6499431e806a7be174869/mockup-b35cfad3.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/343fb301214a9020cf842ea83c98bb55/mockup-2499228f.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/8b844f3b64d5986836154767191afeed/mockup-f734d049.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/2af6618cefeb4afcc8fea28f01331839/mockup-936cd386.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/21e8162750e4490ce4cd16d8c3474f4d/mockup-b74d772f.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/b8eff9131d4cdbd4130742966552a980/mockup-5abcca8d.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/85f038e6c42160542eeff9c019e19bd3/mockup-5916aa32.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/836290cda873b582549c470bc80d08bd/mockup-baab5de8.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/68ed3bf9f2d71b85b64107e8c167e0bd/mockup-d1ee4991.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/b35e22234698d283fc024016ee3573e5/mockup-3bddec98.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/abc99536e9c1524487db0b201242010a/mockup-9817ed45.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/fb9bfb2b4e7df516481e7c88df4605be/mockup-2cc948cd.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/7a9833b04c2bd3ea6c3c306ad1440936/mockup-c4153202.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/96822fe8f9a134f72464889b622eddc6/mockup-6005de80.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/16f74e4e129356dafdd606bb9443898b/mockup-8ffa7963.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/d82398f41d6b3d2d4992fd46e14a5ea9/mockup-bf9386fe.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/f86cad7fe7dec505e04102a3662b23c3/mockup-cb82976e.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/ddaae64e33a03325b36753290e94791c/mockup-8d19cdcd.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/f61700160f90c4a64f0883907bc72ac9/mockup-d82f97eb.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/ffafa73c550f1f1dd2b6775c8ebbb68b/mockup-d124d717.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/3accacb03f8367eb6a3496ab9187c8bd/mockup-18d22b13.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/fd79c155db0367aa4d4fd8f19ffc8132/mockup-fccadd9d.jpg"
                    },
                    {
                        "src": "https:/printful-upload.s3.amazonaws.com/tmp/5c48802c0029c8e667e5347a30a7b648/mockup-9df023b2.jpg"
                    }
                ]
            }
        ]
    }
]

    I’m getting a 504 error, which implies there is some processing of these images that’s taking too long. It gets through about 17 of them. This hasn’t been an issue in the past and only started recently-ish. Not totally sure why, though it could certainly be something on the third-party’s end, if they’ve added a ton of new mockups.

    This may not even be a WooCommerce-specific issue, but that WordPress itself is not designed to handle requests to upload 20+ images at a time. But if you have any feedback, it’d be appreciated.

    Only plugin activated I thought could be related is the TinyPNG one you’ll see below, but I tested with that disabled as well. I know they moved to asynchronous processing of uploads quite some time ago. Guess it could also be related to the WP Offload Media plugin but I imagine it operates asynchronously as well. Can’t swear to that.

    System log:

    
### WordPress Environment ###

WordPress address (URL): https://spanishcuentos.com/wp
Site address (URL): https://spanishcuentos.com
WC Version: 3.8.1
REST API Version: ? 1.0.2
Log Directory Writable: ?
WP Version: 5.3.2
WP Multisite: –
WP Memory Limit: 512 MB
WP Debug Mode: –
WP Cron: ?
Language: en_US
External object cache: ?

### Server Environment ###

Server Info: nginx/1.15.8
PHP Version: 7.3.9-1+ubuntu18.04.1+deb.sury.org+1
PHP Post Max Size: 16 MB
PHP Time Limit: 30
PHP Max Input Vars: 1000
cURL Version: 7.58.0
OpenSSL/1.1.1b

SUHOSIN Installed: –
MySQL Version: 5.5.5-10.3.14-MariaDB-1:10.3.14+maria~xenial-log
Max Upload Size: 16 MB
Default Timezone is UTC: ?
fsockopen/cURL: ?
SoapClient: ?
DOMDocument: ?
GZip: ?
Multibyte String: ?
Remote Post: ?
Remote Get: ?

### Database ###

WC Database Version: 3.8.1
WC Database Prefix: wp_
MaxMind GeoIP Database: ?
Total Database Size: 55.98MB
Database Data Size: 35.14MB
Database Index Size: 20.84MB
wp_woocommerce_sessions: Data: 3.36MB + Index: 0.11MB + Engine MyISAM
wp_woocommerce_api_keys: Data: 0.00MB + Index: 0.01MB + Engine MyISAM
wp_woocommerce_attribute_taxonomies: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_woocommerce_downloadable_product_permissions: Data: 0.19MB + Index: 0.38MB + Engine InnoDB
wp_woocommerce_order_items: Data: 0.28MB + Index: 0.11MB + Engine InnoDB
wp_woocommerce_order_itemmeta: Data: 2.52MB + Index: 3.02MB + Engine InnoDB
wp_woocommerce_tax_rates: Data: 0.02MB + Index: 0.06MB + Engine InnoDB
wp_woocommerce_tax_rate_locations: Data: 0.02MB + Index: 0.05MB + Engine InnoDB
wp_woocommerce_shipping_zones: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_woocommerce_shipping_zone_locations: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_woocommerce_shipping_zone_methods: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_woocommerce_payment_tokens: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_woocommerce_payment_tokenmeta: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_woocommerce_log: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_aryo_activity_log: Data: 0.22MB + Index: 0.00MB + Engine InnoDB
wp_as3cf_items: Data: 0.50MB + Index: 0.75MB + Engine InnoDB
wp_cf7dbplugin_st: Data: 0.05MB + Index: 0.00MB + Engine InnoDB
wp_cf7dbplugin_submits: Data: 0.78MB + Index: 0.06MB + Engine MyISAM
wp_cf7_data: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_cf7_data_entry: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_commentmeta: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_comments: Data: 1.52MB + Index: 0.88MB + Engine InnoDB
wp_iwp_backup_status: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_iwp_file_list: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_iwp_processed_iterator: Data: 0.02MB + Index: 0.00MB + Engine InnoDB
wp_links: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_options: Data: 2.42MB + Index: 0.30MB + Engine InnoDB
wp_postmeta: Data: 14.52MB + Index: 9.98MB + Engine InnoDB
wp_posts: Data: 3.48MB + Index: 1.02MB + Engine InnoDB
wp_redirection_404: Data: 2.16MB + Index: 0.61MB + Engine MyISAM
wp_redirection_groups: Data: 0.00MB + Index: 0.00MB + Engine MyISAM
wp_redirection_items: Data: 0.00MB + Index: 0.02MB + Engine MyISAM
wp_redirection_logs: Data: 0.00MB + Index: 0.01MB + Engine MyISAM
wp_termmeta: Data: 0.00MB + Index: 0.01MB + Engine MyISAM
wp_terms: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_term_relationships: Data: 0.08MB + Index: 0.05MB + Engine InnoDB
wp_term_taxonomy: Data: 0.02MB + Index: 0.03MB + Engine InnoDB
wp_toolset_post_guid_id: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_usermeta: Data: 2.52MB + Index: 2.98MB + Engine InnoDB
wp_users: Data: 0.11MB + Index: 0.08MB + Engine InnoDB
wp_wc_download_log: Data: 0.06MB + Index: 0.03MB + Engine InnoDB
wp_wc_product_meta_lookup: Data: 0.05MB + Index: 0.09MB + Engine InnoDB
wp_wc_tax_rate_classes: Data: 0.02MB + Index: 0.02MB + Engine InnoDB
wp_wc_webhooks: Data: 0.02MB + Index: 0.02MB + Engine InnoDB

### Security ###

Secure connection (HTTPS): ?
Hide errors from visitors: ?

### Active Plugins (25) ###

InfiniteWP - Client: by Revmakx – 1.9.4.5
Advanced Custom Fields PRO: by Elliot Condon – 5.8.7
All In One SEO Pack: by Michael Torbert – 3.3.4
WP Offload Media: by Delicious Brains – 2.3.2
Activity Log: by Activity Log Team – 2.5.2
Honeypot for Contact Form 7: by Nocean – 1.14.1
Contact Form 7: by Takayuki Miyoshi – 5.1.6
Flamingo: by Takayuki Miyoshi – 2.1
Flexible Shipping: by WP Desk – 3.9.4
Mailgun: by Mailgun – 1.7.1
Printful Integration for WooCommerce: by Printful – 2.1.8
Redirection: by John Godley – 4.6.1
Redis Object Cache: by Till Krüss – 1.5.5
s2Member End of Term Notifications: by Ethan Clevenger – 1.0.0
s2Member Framework: by WP Sharks – 191022
Spanish Cuentos: by  –
Compress JPEG & PNG images: by TinyPNG – 3.2.1
User Switching: by John Blackbourn & contributors – 1.5.3
WooCommerce Dynamic Pricing: by Lucas Stark – 3.1.15 – Not tested with the active version of WooCommerce
WooCommerce Google Analytics Integration: by WooCommerce – 1.4.15
WooCommerce: by Automattic – 3.8.1
WP fail2ban: by Charles Lecklider – 4.2.7.1
WP Migrate DB Pro CLI: by Delicious Brains – 1.3.5
WP Migrate DB Pro Media Files: by Delicious Brains – 1.4.14
WP Migrate DB Pro: by Delicious Brains – 1.9.8

### Inactive Plugins (0) ###

### Dropin Plugins (1) ###

object-cache.php: Redis Object Cache Drop-In

### Must Use Plugins (5) ###

Bedrock Autoloader: by Roots – 1.0.0
Disallow Indexing: by Roots – 1.0.0
InfiniteWP - Client Loader: by Revmakx –
Register Theme Directory: by Roots – 1.0.0
WP Migrate DB Pro Compatibility: by Delicious Brains – 1.2

### Settings ###

API Enabled: ?
Force SSL: –
Currency: USD ($)
Currency Position: left
Thousand Separator: ,
Decimal Separator: .
Number of Decimals: 2
Taxonomies: Product Types: external (external)
grouped (grouped)
simple (simple)
variable (variable)

Taxonomies: Product Visibility: exclude-from-catalog (exclude-from-catalog)
exclude-from-search (exclude-from-search)
featured (featured)
outofstock (outofstock)
rated-1 (rated-1)
rated-2 (rated-2)
rated-3 (rated-3)
rated-4 (rated-4)
rated-5 (rated-5)

Connected to WooCommerce.com: ?

### WC Pages ###

Shop base: #522 - /shop/
Cart: #523 - /cart/
Checkout: #524 - /checkout/
My account: #525 - /my-account/
Terms and conditions: ? Page not set

### Theme ###

Name: HTML5 Blank
Version: 1.4.2
Author URL: https://toddmotto.com
Child Theme: ? – If you are modifying WooCommerce on a parent theme that you did not build personally we recommend using a child theme. See: How to create a child theme
WooCommerce Support: ?

### Templates ###

Overrides: spanish-cuentos/woocommerce/archive-product.php

### Action Scheduler ###

Complete: 230
Oldest: 2019-12-26 13:03:44 +0000
Newest: 2020-01-21 16:27:55 +0000

Pending: 0
Oldest: –
Newest: –

Canceled: 0
Oldest: –
Newest: –

In-progress: 0
Oldest: –
Newest: –

Failed: 0
Oldest: –
Newest: –

    The page I need help with: [log in to see the link]

Viewing 9 replies - 1 through 9 (of 9 total)

  • HI!

    Thanks for reaching out to us. I see that the images are being uploaded to the site from AWS, correct?

    `
    “src”: “https:/printful-upload.s3.amazonaws.com/tmp/95de7e34a9d3bc1e6e3c25034fecf019/mockup-3742648c.jpg”
    `

    According to AWS support documentation, this could be related to the site’s firewall not allowing AWS and the CloudFront CDN.

    https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/http-504-gateway-timeout.html

    Please take a look at this and see if it helps in any way. Whether it does or does not, please also come back and let us know how things are working out. Thank you for using WooCommerce!

    Thread Starter eclev91

    (@eclev91)

    Like I said, the images are being uploaded just fine. I can look in the media library and see it made it through several of them and successfully uploaded them. Obviously it just isn’t moving very fast.

    Thread Starter eclev91

    (@eclev91)

    @phillipwoo wanted to check in and see if your team had any suggestions beyond sending requests with fewer images at a time. 20-some-odd images is probably likely to take a while no matter what unless this triggered some kind of asynchronous process.

    Hello again,

    Sorry for the delay in my reply. This is just a shot in the dark, but try increasing the PHP Post Max Size setting to see if it allows the AWS server to do what it needs to do. Here is the current setting for your site, per the Status Report:

    `
    PHP Post Max Size: 16 MB
    `

    I also see that Log Directory Writable: ?, can you check the logs for this to see if they can shed any light on what is occurring?

    Thanks~

    Thread Starter eclev91

    (@eclev91)

    The request is not too large, and the logs show a timeout (ultimately, HTTP 504). It’s taking too long because it’s trying to process 20+ images. Like I said, in the media library after running this request, I can see that several of the images get uploaded, but eventually it exceeds 30 seconds (the server timeout limit) and quits.

    To give you context, my client is using a Printful integration to sell products with custom designs. When a product from Printful is selected to be imported, it also sends along a bunch of images. Some products have a few images, some have many. The reason this issue just recently showed up is my client just recently tried to do phone cases, which come with tons of images (as you see in the above request). One for each phone size, I assume, versus stuff like pillows and t-shirts which may only have a couple photos.

    I’ve recommended my client uncheck several variation photos to prevent this from happening, but an actual solution for REST API product imports with a large number of photos would be to do some kind of background processing of photo uploads. That may be a task for WordPress core, however.

    Not sure how this is handled in the block editor, which also uses the REST API for photo uploads. What happens when I drag 20+ photos in? Off the cuff, I’m assuming it sends a separate request for each one, which would get around any potential timeouts.

    Hi!

    > It’s taking too long because it’s trying to process 20+ images.

    When you are using any API and trying to access data from an external server. The other server might take some time to respond to your data query. If this response time is larger than the maximum execution time, you will see a timeout error. This isn’t your server’s timeout error, it is Amazon’s (AWS, correct?). It may be a good idea to ensure the Printful integration is up to date as well as reviewing their WooCommerce Integration FAQ.

    Another place to check is the site logs at **WooCommerce → Status → Logs tab**. If there are any logs that look like the sample below, it means the WP Memory Limit isn’t large enough for the site to process the incoming images:

    `
    <b>Fatal error</b>: Allowed memory size of XXX bytes exhausted (tried to allocate XXX bytes) in <b>/home/superbadger/public_html/yourwebsite.com/wp-content/plugins/woocommerce/includes/api/v2/class-wc-api-json-handler.php</b> on line <b>76</b><br />
    `

    This happens at times on image-heavy sites so the current setting may simply not be enough in this case. Increasing the WP Memory Limit might help with that. It may be the other server is timing out because the store is taking too long to process the images. The increased Memory Limit would allow for more images to be processed at once, making the requests faster, and (hopefully) no more time-out messages.

    Can you give this a try and see if it helps, please? If you do, let us know how this works out.

    All the best

    Hi @eclev91,

    Did you try to increase the memory limit, and did it work? Please let us know!

    I am going to go ahead and close this thread, however if you reply it will re-open for us to see. I look forward to hearing back from you!

    Thread Starter eclev91

    (@eclev91)

    Hey Mike – the issue is not a memory limit, but a PHP timeout. I could increase the timeout, but that only pushes the issue down the road. If someone tried posting 100 images via the REST API, a timeout of 2 minutes or more would probably still fail.

    If I try and post a large number of variations via the REST API, the endpoint will return an error. It’s configured to only accept so many, I assume because it knows adding a large number of variations will either be too time-consuming, or will have overall adverse affects on the site, or perhaps there’s a top limit in WooCommerce core. But the limit is there to prevent abuse of the REST API. It seems to me that trying to post 50 images at once would similarly be an abuse of the REST API. Probably why the core REST API doesn’t have _any_ batch media upload endpoint.

    Seems to me that batch media uploads to products via the REST API should be similarly limited to prevent abuse. I can increase my timeout, but for consistent results, people utilizing the REST API should, by default, be prevented from abusing it in this way and putting significant strain on the server.

    We are running into similar issues with the product batch endpoint. We are using offload media which is passing the image uploads to S3 on call. As a point of reference, we have around 20k users and 2k products. Whenever we use the REST API, the database CPU spikes up with only few connections and if not throttled, will eventually lead to 504s. I’ve been trying to optimize tables as a means to try and speed it up but to date I have yet to see any changes I’ve made actually make a significant impact in reducing the occurrence of this issue. We are also using w3 total cache but the wp-admin and rest api are not cached while the front end is redis cached.

Viewing 9 replies - 1 through 9 (of 9 total)
  • The topic ‘Batch adding images via API causing 504 errors’ is closed to new replies.