How to disable Amazon bots by Robot.txt?

Hello @alexliii,

Thank you for contacting Rank Math support, and sorry for any inconvenience that might have been caused due to that.

If there’s no physical robots.txt file in your root directory, please clear your website and server’s cache for initial troubleshooting.

If the issue persists, then your robots.txt must be generated or controlled by one of your plugins or active theme.

You may follow our guide here to determine that: https://rankmath.com/kb/check-plugin-conflicts/

Here is a video you can follow as well: https://www.youtube.com/watch?v=FcIWRhcUP2c

Only the site admin would know and your users won’t be affected by this troubleshooting.

Let us know how this goes.

Hello,

Here is out test:

https://prnt.sc/8dSGUongrBuk

and only activate Rankmath:

here is screenshot to show only one plugin is activated:

https://prnt.sc/Yzlrbtu-4FRM

However, I tried the following setting:

User-agent: *
Disallow: /wp-admin/
Disallow: */woo-feed/
Disallow: */feed/
Allow: /wp-admin/admin-ajax.php

User-agent: Amazonbot
Disallow: /

But it always show:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

and here is screenshot:
https://prnt.sc/-3sgvSSMWP_c

On our site, there is no physical robots.txt:
https://prnt.sc/PySXT8p7NA_w

Could you please let us how to fix it ?

Great thanks

Here is further test:

If put the physical robots.txt in the directory www/oursite.com/htdocs/:
https://prnt.sc/NZ8UIlnOzU0V

It works fine, so, could you please what might cause the issue please?

`
### rank-math ###

version: 1.0.219
database_version: 1
plugin_plan: Free
active_modules: link-counter, seo-analysis, sitemap, rich-snippet, woocommerce, local-seo, image-seo, redirections, analytics, role-manager, 404-monitor
refresh_token: Token exists
permissions: 
	Search Console: Given
rank_math_404_logs: 32 KB
rank_math_redirections: 32 KB
rank_math_redirections_cache: 32 KB
rank_math_internal_links: 2 MB
rank_math_internal_meta: 192 KB
rank_math_analytics_gsc: 8 MB
rank_math_analytics_objects: 2 MB
rank_math_analytics_inspections: 35 MB

### wp-core ###

version: 6.5.3
site_language: en_US
user_language: en_US
timezone: -08:00
permalink: /%postname%/
https_status: true
multisite: true
user_registration: false
blog_public: 1
default_comment_status: open
environment_type: production
site_id: 1
site_count: 1
network_count: 1
user_count: 2057
dotorg_communication: true

### wp-active-theme ###

name: Savoy (savoy)
version: 2.9.4
author: NordicMade
author_website: https://www.nordicmade.com
parent_theme: none
theme_features: core-block-patterns, widgets-block-editor, post-thumbnails, editor-styles, align-wide, editor-style, title-tag, automatic-feed-links, woocommerce, wc-product-gallery-slider, wc-product-gallery-lightbox, widgets, menus
theme_path: /var/www/outsite.com/htdocs/wp-content/themes/savoy
auto_update: Disabled

### wp-mu-plugins (1) ###

Health Check Troubleshooting Mode: author: (undefined), version: 1.9.1

### wp-plugins-active (37) ###

ALD - Aliexpress Dropshipping and Fulfillment for WooCommerce: version: 2.0.3, author: VillaTheme(villatheme.com), Auto-updates disabled
Autoptimize: version: 3.1.11, author: Frank Goossens (futtta), Auto-updates disabled
Auto Tag Creator: version: 1.0.2, author: Ecreate Infotech, Auto-updates disabled
Classic Editor: version: 1.6.3, author: WordPress Contributors, Auto-updates disabled
Code Snippets: version: 3.6.4, author: Code Snippets Pro, Auto-updates disabled
CTX Feed Pro: version: 7.3.17, author: WebAppick Author, Auto-updates disabled
Easy Table of Contents: version: 2.0.66.1, author: Magazine3, Auto-updates disabled
Elementor: version: 3.21.4, author: Elementor.com (latest version: 3.21.5), Auto-updates disabled
Elementor Pro: version: 3.21.2, author: Elementor.com, Auto-updates disabled
Forminator: version: 1.30.2, author: WPMU DEV, Auto-updates disabled
GTM4WP - A Google Tag Manager (GTM) plugin for WordPress: version: 1.20.2, author: Thomas Geiger, Auto-updates disabled
GTranslate: version: 3.0.6, author: Translate AI Multilingual Solutions, Auto-updates disabled
Heartbeat Control by WP Rocket: version: 2.0.1, author: WP Rocket, Auto-updates disabled
Instant Indexing: version: 1.1.18, author: Rank Math, Auto-updates disabled
Media File Renamer: Rename Files (Manual, Auto, AI) (Pro): version: 5.8.3, author: Jordy Meow (latest version: 5.9.1), Auto-updates disabled
Product Size Chart for WooCommerce: version: 1.2.3, author: VillaTheme, Auto-updates disabled
Rank Math SEO with AI Best SEO Tools: version: 1.0.219, author: Rank Math, Auto-updates disabled
Savoy Theme - Content Elements: version: 1.6.7, author: NordicMade, Auto-updates disabled
Savoy Theme - Settings Panel: version: 1.2.3, author: NordicMade, Auto-updates disabled
Ship Estimate for WooCommerce: version: 2.0.11, author: RLDD, Auto-updates disabled
SVG Support: version: 2.5.5, author: Benbodhi, Auto-updates disabled
Tumult Hype Animations: author: (undefined), version: 1.9.14, Auto-updates disabled
Variation Swatches for WooCommerce: version: 2.0.31, author: Emran Ahmed, Auto-updates disabled
Variation Swatches for WooCommerce - Pro: version: 2.0.31, author: Emran Ahmed, Auto-updates disabled
WooCommerce: version: 8.8.3, author: Automattic, Auto-updates disabled
WooCommerce Coupon Box Premium: version: 2.1.1, author: VillaTheme, Auto-updates enabled
WooCommerce Email Template Customizer Premium: version: 1.2.4, author: VillaTheme, Auto-updates disabled
WooCommerce Free Shipping Bar Premium: version: 1.2.3, author: VillaTheme, Auto-updates disabled
WooCommerce Orders Tracking Premium: version: 1.1.10, author: VillaTheme, Auto-updates disabled
WooCommerce PayPal Payments: version: 2.7.0, author: WooCommerce, Auto-updates disabled
WooCommerce Photo Reviews Premium: version: 1.3.12, author: VillaTheme, Auto-updates disabled
WooCommerce Support Ticket System: version: 17.1, author: Lagudi Domenico, Auto-updates disabled
Wordfence Security: version: 7.11.5, author: Wordfence, Auto-updates disabled
WPCode Lite: version: 2.1.12, author: WPCode, Auto-updates disabled
WPC Product Tabs for WooCommerce: version: 4.1.1, author: WPClever, Auto-updates disabled
WP Mail SMTP Pro: version: 3.11.1, author: WP Mail SMTP, Auto-updates disabled
WP Offload Media: version: 3.2.7, author: Delicious Brains, Auto-updates disabled

### wp-server ###

server_architecture: Linux 5.15.0-107-generic x86_64
httpd_software: nginx/1.24.0
php_version: 8.1.28 64bit
php_sapi: fpm-fcgi
max_input_variables: 20000
time_limit: 300000
memory_limit: 1024M
admin_memory_limit: 2048M
max_input_time: 300000
upload_max_filesize: 100M
php_post_max_size: 100M
curl_version: 7.81.0 OpenSSL/3.0.2
suhosin: false
imagick_availability: true
pretty_permalinks: true
current: 2024-05-20T02:34:13+00:00
utc-time: Monday, 20-May-24 02:34:13 UTC
server-time: 2024-05-19T18:34:11-08:00

### wp-database ###

extension: mysqli
server_version: 10.6.17-MariaDB-1:10.6.17+maria~ubu2204-log
client_version: mysqlnd 8.1.28
max_allowed_packet: 67108864
max_connections: 100

### wp-constants ###

WP_HOME: undefined
WP_SITEURL: undefined
WP_CONTENT_DIR: /var/www/outsite.com/htdocs/wp-content
WP_PLUGIN_DIR: /var/www/outsite.com/htdocs/wp-content/plugins
WP_MEMORY_LIMIT: 1024M
WP_MAX_MEMORY_LIMIT: 2048M
WP_DEBUG: false
WP_DEBUG_DISPLAY: true
WP_DEBUG_LOG: false
SCRIPT_DEBUG: false
WP_CACHE: false
CONCATENATE_SCRIPTS: false
COMPRESS_SCRIPTS: undefined
COMPRESS_CSS: undefined
WP_ENVIRONMENT_TYPE: Undefined
WP_DEVELOPMENT_MODE: undefined
DB_CHARSET: utf8
DB_COLLATE: undefined

### wp-filesystem ###

wordpress: writable
wp-content: writable
uploads: writable
plugins: writable
themes: writable
mu-plugins: writable
0: Writable

`

Thanks

• This reply was modified 6 months ago by alexliii.

Hello @alexliii,
?
Please share the site URL so we can check and assist you further.
?
Looking forward to helping you.