URLs totally messed up

For some reason, wordpress is returning a valid page when it most definitely should not. This is screwing up my google listings in a very big way.

For example, let’s say:

https://www.mysite.com/somecategory/blue-widgets/

… is a valid url.

WordPress is also returning any and all of these as valid:

https://www.mysite.com/blahblah/blue-widgets/
https://www.mysite.com/x82342kjasdfasd/blue-widgets/
https://www.mysite.com/82348q384q823/doodoodoo/blue-widgets/

I have 432 pages that are now indexed under the wrong url, tripping duplicate content filters and have lost all pagerank as a result.

I have no idea why this is happening or where it started … I assume with the last upgrade? Any ideas why this is happening? Could it be related to supercache?