The server records the triplet of sender IP, sender address and recipient and answers the first attempt with a 4xx code. A proper mail server retries after a few minutes and gets through. Many simple spam bots never retry.
The cost is delay: the first message from a new sender arrives minutes or hours late. That hurts for one-time codes, password resets and login confirmations. Large providers also send from changing IPs, which can extend the delay.
Look in the log for a sender being refused repeatedly with 450 or 451. A whitelist for known senders, or matching on the IP network instead of the exact IP, usually solves it.
Example
451 4.7.1 <user@example.com>: Recipient address rejected: Greylisted, try again later