It was too late; the man had made up his mind and there was no turning back. Cowardice is a challenging thing to swallow, it's a tough diet and is hard to digest, especially when you're the one who instigated the fight.
There's time to argue and time to accept the harsh reality of a situation. Wilder had learned years ago its best to accept, because in the end, it is what it is and can't be changed through force. It is only when one tries to force their hand that problems occur.