Break-even on the fees
Start with the monthly payment difference. If the new loan saves $40 a month and the fees are $500, simple division gives 13 payments before the fee is repaid. That is the point where the decision stops being a bet on staying in the loan.
The lifetime number matters just as much: total payments on the old loan against total payments on the new one, plus the fees. A refinance that saves less over its whole life than it costs in fees is not a refinance, it is a longer payment plan.