Thanks @jgbrear, I think that’s it exactly! Here’s the solar load with the 2 orientations.
For the unit with long axis running EW:
During the summer, the big south facing window is more shaded because the sun is higher in the sky.
For the unit with long axis running NS:
The low morning/afternoon sun through the windows on the EW sides of the building really add to the solar load, even in the summer.
