Though this is about daylight it explains why it happens. The AnnualIrradiance recipe does the same 3 steps. The figure also explains why it can happen when your ambient bounces are > 0, eg., if step 2 is larger than step 1.
Regarding parameters for rfluxmtx/rcontrib based recipes (such as AnnualIrradiance and AnnualDaylight) the most important parameters tend to be -ad and -lw (controlled by -ad).