@Ilay_s_p It is generally not a good advice to have large jumps in the sizes of neighbouring cells. This is the reason we kind of moved from the idea of refinement region to that of gradient mesh (you can use the wind tunnel gradient component to get an even better result than what you have with a fraction of total cells).
If I undestand your case correctly this is what happens here. The refinement region at 2 levels is creating a big jump in cell size and that brings you that big jump in gradients. I would either reduce it to one (or use a distance refinement mode) or simply use the wind tunnel gradient componet (although for the case of 1 building it might simply wrap around your geometry).