I have also been struggling with the same exact problem. I have had it for a while. I turned off all plugins which didn’t solve the issue. I then changed to twentytwentythree and the issue was gone. It’s obviously something in my theme code, but I cannot pinpoint the problem. It’s happening on both custom block patterns that I have created as well as core block patterns. I’ll keep trying to debug, but I’m also subscribing to this issue if anyone else has some insight.