Because TCP is designed to provide reliable transmission at reasonable bandwidth across medium-large amounts of data, it is incredibly bad at low-latency relay of small amounts of data. It is generally tuned to send packets (each just under 1500 bytes) once and to only discover that some packets were lost after getting a response from the other side. Only then will the sender retransmit the lost packets, allowing the receiver to (potentially) reconstruct the original transmission.

I have seen packet loss over long-haul links on the internet average 1%, though if you purchase bandwidth directly from global carriers, you can expect < 0.1%. At 1%, the probability of transmitting a full, uncompressed, block without needing to wait for extra round-trips is roughly 0.99^(1000000/1500) = 0.1% (or 0.999^(1000000/1500) = 51%, if you're willing to pay for it). Worse still, packet loss starts to go up as you send more data. Even transmitting just 15KB (around 10 packets) has a probability of success at only 90% for an average hosting provider. When we're talking about 1Gbps or 100Mbps links with round-trip latencies of 100-200 milliseconds, a single round-trip takes significantly longer than sending any reasonable amount of data could possibly take.

Thus, in order to have minimal latency block transmission, we must avoid the need for retransmissions at all costs. In order to do so, we must transmit enough extra data that the receiving peer can reconstruct the entire block even though some packets were lost on the way. Luckily this is a well-researched field, largely thanks to video conferencing having similar requirements. The common solution is UDP-based transmission with some relatively simple linear algebra to send data which can fill in gaps of lost packets efficiently (this is called Forward Error Correction, or FEC).

Even with an elegant solution to the packet-loss issue, the time to transmit 1MB over a 1Gbps link is still several milliseconds, which is more than doubled with the extra overhead from both the construction and transmission time of the FEC data. Thus, the Compact Blocks scheme in Bitcoin Core is critical to performance.

Because the Compact Block design was being done simultaneously with the work on FIBRE, the cmpctblock message format was designed to ensure it fits neatly into a UDP-FEC-based relay mechanism. The only difference is that we send it over UDP with FEC, and instead of relying on a round-trip to request any missing transactions in our mempool, we send the block's transactions immediately, again with additional FEC.