Oof. Stupid mistake. I was wondering why my collisions were off. The order of the points I used to define the edges of the rectangles and find intersections was wrong so I was getting bow tie shapes instead of rectangles.
Picture is worth a thousand words.
Some of the collision points were fine. Some were a little off. And some didn’t show up at all so I was pretty confused until I saw the bow ties.