Sites checking for cookies shown as broken links

  • Resolved b-rad

    (@b-rad)


    3 years, 11 months ago

    Many times I’ll link to scientific articles using a DOI (https://www.doi.org/) and while the links are valid, the sites that they resolve to check for cookies, and the link checker is reporting them as broken.

    For example, this link https://dx.doi.org/10.1029/2007GB003176 should resolve to https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2007GB003176, but the link checker gets hung up on https://onlinelibrary.wiley.com/action/cookieAbsent and reports that it’s broken.

    Can the plugin be updated to follow the redirect fully and/or handle the cookie situation better?

    The full log message is as follows:

    Log: === HTTP code : 403 ===

Response headers
================
HTTP/1.1 301 Moved Permanently
NEL: {"report_to":"cf-nel","max_age":604800}
Via: 1.1 137.75.68.149 (McAfee Web Gateway 7.8.2.23.0.33846)
Date: Sat, 10 Apr 2021 13:13:47 GMT
CF-RAY: 63dc3c0a58b059f8-IAD
Server: cloudflare
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
Expires: Sat, 10 Apr 2021 14:13:47 GMT
X-Cache: MISS from 137.75.68.149
Location: https://dx.doi.org/10.1029/2007GB003176
Report-To: {"max_age":604800,"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=YEqIzMJ7BkjgnQpHRP2phOVaGN0uEwDNP4QKgbJ4ybn1LW6VlaTrruF%2FMpPb33NR2ayvh29ol484wc%2BlyK23Ofey3w1RdriH43h3"}],"group":"cf-nel"}
Connection: close
Cache-Control: max-age=3600
cf-request-id: 095d83da7b000059f87395d000000001
Transfer-Encoding: chunked

HTTP/1.1 302
Date: Sat, 10 Apr 2021 13:13:47 GMT
Content-Type: text/html;charset=utf-8
Content-Length: 167
Connection: close
Set-Cookie: __cfduid=dda7d71ba91f33d3b7b50fbe7edc94c111618060427; expires=Mon, 10-May-21 13:13:47 GMT; path=/; domain=.doi.org; HttpOnly; SameSite=Lax; Secure
Vary: Accept
Location: https://doi.wiley.com/10.1029/2007GB003176
Expires: Sat, 10 Apr 2021 13:22:01 GMT
CF-Cache-Status: DYNAMIC
cf-request-id: 095d83dab700005a0f393b3000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Report-To: {"max_age":604800,"group":"cf-nel","endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report?s=K9lKOdVYCfZEs4g4tcTfLqtvpgO67CtqdASN4FpXDJ1Z3MIgcm0ebwwDAETEZ2E8rkVsfwfqykKtoQYZhRzeY446RfbXhcGNb2Js"}]}
NEL: {"report_to":"cf-nel","max_age":604800}
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Server: cloudflare
CF-RAY: 63dc3c0abc9a5a0f-IAD
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

HTTP/1.1 302 Found
Via: 1.1 137.75.68.138 (McAfee Web Gateway 7.8.2.23.0.33846)
Date: Sat, 10 Apr 2021 13:13:48 GMT
Server: Apache/2.4.46 (Unix) OpenSSL/1.0.2k-fips
X-Cache: MISS from 137.75.68.138
Location: https://onlinelibrary.wiley.com/resolve/doi?DOI=10.1029/2007GB003176
Connection: close
Content-Type: text/html; charset=iso-8859-1
Content-Length: 252

HTTP/1.1 301 Moved Permanently
Date: Sat, 10 Apr 2021 13:13:48 GMT
Content-Length: 0
Connection: close
Set-Cookie: __cfduid=d923017ab93bddceab078edf145aca9ad1618060428; expires=Mon, 10-May-21 13:13:48 GMT; path=/; domain=.onlinelibrary.wiley.com; HttpOnly; SameSite=Lax; Secure
Location: https://onlinelibrary.wiley.com/doi/abs/10.1029/2007GB003176
CF-Ray: 63dc3c0bace65de1-IAD
Cache-Control: private
Strict-Transport-Security: max-age=16070400
Vary: Accept-Encoding
CF-Cache-Status: HIT
cf-request-id: 095d83db4a00005de1f48b0000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Set-Cookie: __cf_bm=e84f6a745fb50922d95526bfd87ef71853f9375f-1618060428-1800-AYQXt3UkU7UjhMOh6WY7mJh8P7Z4IhdbuX315Fx/HJFY8Fj8YOJLehd7Cik7HyJ8H+3RkKdFk3bsPbPS18SB9Gg8z6Cg7aJMJJnLX6tcvX1d; path=/; expires=Sat, 10-Apr-21 13:43:48 GMT; domain=.onlinelibrary.wiley.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

HTTP/1.1 302 Found
Date: Sat, 10 Apr 2021 13:13:48 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Set-Cookie: __cfduid=dd6633e62a7a99da27d38aea4e9ff3d901618060428; expires=Mon, 10-May-21 13:13:48 GMT; path=/; domain=.onlinelibrary.wiley.com; HttpOnly; SameSite=Lax; Secure
Location: https://onlinelibrary.wiley.com/doi/abs/10.1029/2007GB003176?cookieSet=1
CF-Ray: 63dc3c0cbf645ba6-IAD
Cache-Control: private
Strict-Transport-Security: max-age=16070400
CF-Cache-Status: DYNAMIC
cf-request-id: 095d83dbef00005ba6a58c8000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
P3P: CP="NOI DSP ADM OUR IND OTC"
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Set-Cookie: I2KBRCK=1; domain=.onlinelibrary.wiley.com; path=/; secure; expires=Sun, 10-Apr-2022 13:13:48 GMT
Set-Cookie: __cf_bm=6f0384136220b902c11a727e6131dde4519c79f6-1618060428-1800-AVXFT85mqf4eE9eYN6ZXRQVixzLe76cWSGKT0x8BnFvirZxApukOiam1EJUjrasHCrmIL5dA59WcI7JdsiBgtzq7oNj39H/NjmAjdJN4Y98f; path=/; expires=Sat, 10-Apr-21 13:43:48 GMT; domain=.onlinelibrary.wiley.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

HTTP/1.1 302 Found
Date: Sat, 10 Apr 2021 13:13:48 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Set-Cookie: __cfduid=d26549fcbc9a916d86cd5b335a75a85bd1618060428; expires=Mon, 10-May-21 13:13:48 GMT; path=/; domain=.onlinelibrary.wiley.com; HttpOnly; SameSite=Lax; Secure
Location: https://onlinelibrary.wiley.com/action/cookieAbsent
CF-Ray: 63dc3c0dbe3a5992-IAD
Cache-Control: private
Strict-Transport-Security: max-age=16070400
CF-Cache-Status: DYNAMIC
cf-request-id: 095d83dc90000059922da6f000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Set-Cookie: __cf_bm=5dc2850d432e2d732ef34b4fcfe2daef643d9239-1618060428-1800-AXjxuOXnMnc0FTX8jrB9bVObRa44KvqZ2wi/hVBqIhZOjUVoNKfpLhCoAdkkSOYWD4o44ezNVEIQRnNu7sRXdNuCwFMeTAHndaVXagWMtt8g; path=/; expires=Sat, 10-Apr-21 13:43:48 GMT; domain=.onlinelibrary.wiley.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

HTTP/1.1 403 Forbidden
Date: Sat, 10 Apr 2021 13:13:48 GMT
Content-Type: text/plain
Content-Length: 265
Connection: close
Set-Cookie: __cfduid=d0261453c2b517cc021f94adde262438f1618060428; expires=Mon, 10-May-21 13:13:48 GMT; path=/; domain=.onlinelibrary.wiley.com; HttpOnly; SameSite=Lax; Secure
Cache-Control: no-store
cf-request-id: 095d83ddfb00005a4539128000000001
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Set-Cookie: __cf_bm=dfa4e743db8477131944d90eccd40a3ccd3a0b92-1618060428-1800-AcTDg4I6wGFAl0/rdwwNfbMRhzQgmJHMIw6+strXsELxf624wdPDkgaEGJmG1oomsy/NDwjU0voJiZLJ3ce8q72dVyjNKTKw+rlj1JIUhfLt; path=/; expires=Sat, 10-Apr-21 13:43:48 GMT; domain=.onlinelibrary.wiley.com; HttpOnly; Secure; SameSite=None
Server: cloudflare
CF-RAY: 63dc3c0ff9ea5a45-IAD
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400

Request headers
================
GET /action/cookieAbsent HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36
Host: onlinelibrary.wiley.com
Accept: */*
Referer: https://www.example.com
Connection: close
Range: bytes=0-2048

Response HTML
================
Wiley Online Library requires cookies for authentication and use of other site features; therefore, cookies must be enabled to browse the site. Detailed information on how Wiley uses cookies can be found in our Privacy Policy [https://www.wiley.com/en-us/privacy].

Link is broken.
Viewing 4 replies - 1 through 4 (of 4 total)
  • Plugin Support Imran – WPMU DEV Support

    (@wpmudev-support9)

    Hi @b-rad,

    Thank you for contacting us.

    Based on the forbidden error, it’s likely possible the server is blocking the response and hence the issue.

    Could you please double-check with your hosting provider whether the user-agent listed in the above is getting blocked by your host or not? Please do share the above details with your hosting provider too?

    That could also be one of the cases for such an issue to occur. You can also try clearing all server cache and reinstalling the plugin. Please do check and let us know how that goes.

    Best,
    Jonathan S

    Thread Starter b-rad

    (@b-rad)

    Hi Jonathan. Reinstalling didn’t fix it and our host isn’t blocking anything. Is this plugin able to handle the cookie requirement that the target site has?

    Plugin Support Patrick – WPMU DEV Support

    (@wpmudevsupport12)

    Hi @b-rad

    I was able to replicate the issue.

    The main problem is that if we run the cURL command on the terminal we will get the same issue as the website forces to accept the cookies.

    We tasked the thread to our developers to verify if anything can be done on the plugin side to bypass this situation but as the website that returns the header status, it will be hard to bypass this.

    I am afraid we can’t give an estimated time to release an improved version, in case those are false positives I suggest marking it as not broken https://monosnap.com/file/AdplyfV7iSImXqtXvMKJExK5wXBYuf

    Best Regards
    Patrick Freitas

    Plugin Support Patrick – WPMU DEV Support

    (@wpmudevsupport12)

    Hi @b-rad

    I hope you are doing well.

    As I let you know in the previous reply, we were able to replicate the issue and we reported it to our developers.

    We can’t give an estimated time to release a fixed version or if we will be able to bypass this cookie error as this is the website response to the request.

    I am marking this as resolved for now, but feel free to ping us any time you need.

    Best Regards
    Patrick Freitas

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Sites checking for cookies shown as broken links’ is closed to new replies.