It is quite possible that your bass drivers (the bass speakers other than the subwoofer) has a low-end frequency overlap with the sub. So when you have the subwoofer in opposite phase with the bass ...