> Why are they so rarely used despite an almost free performance benefit?

Try measuring the performance benefit. You'll probably find there isn't much/any.

It may have been a boost around 2000, but modern NICs have all sorts offloads now.

The extra complexity (best practice is to run a separate vlan/subnet where clients have an extra NIC with high MTU) isn't worth it as the only real drawback is on the wire header overhead (NICs have LRO/LSO and other offload mechanisms).

They are used often in storage networks. Back when I worked at a vmware shop, we also implemented them on the vmotion network, and it cut down considerably moving very large DB servers to other hosts.

It is pretty common for networks used for local NFS storage, or other uses where the packets are somewhat guaranteed to not leave the facility, like between Oracle RAC nodes, etc.

in DC Networks, 9k frames and pathmtu is very common. the issue however is that the MTU across the internet is 1500, and you cannot selectively use mtu's depending on routes, so you end up with 9k MTU at edge routers which need to split that into 1500 MTU packets.

this fragmentation is absolutely brutal to performance of edge routers.

> you cannot selectively use mtu's depending on routes

You absolutely can. Assume a single network port on the server.

On BSD, it looks something like this:

ifconfig foo 10.x.y.z/24 mtu 9000

route add default 10.x.y.1 -mtu 1500

route add 10.0.0.0/8 10.x.y.1

Linux has a mss attribute on routes which should work similarly.

Roll your own IPV6 version, but radvd can sort of maybe offer network routes I think, so it might be slightly easier to config it that way.

Doing this requires fragmenting at the IP level, which is notoriously unreliable over the internet. The other option is to send PMTU exceeded messages telling the sender to drop their MTU for the path, adding significant overhead for every new connection (assuming the sender even listens to such messages).

How so? In my example, the default route has a MTU of 1500 and the localnet route has a MTU of whatever the interface is (9000 in this example)

When you send a packet, the destination route is looked up to determine the MTU. If you send to the internet, you'll get small packets (not IP fragments), and 10. will get large packets.

That works fine for locally generated traffic which has knowledge of the routes. For hosts going through a router, they have no idea what the outbound link MTU is, requiring PMTU discovery or messing with TCP MSS.

The claim I replied to was that you couldn't run one MTU for some routes and one for other routes.

You absolutely can. But running your default route at 9000 is ill-advised as you mention.

If your datacenter routes aren't simply rfc1918 addresses, it would be harder to set the end host routes, and the effort to get larger packets on the wire might not be worth the rather small benefits.

Would it not be possible to have another network device before the edge router that encapsulates the packet to 1500? And vice versa for inbound packets?

I have a rack in a DC, I've always wanted to configure jumbo frames between the host and VMs. However I haven't because I am not sure of the recourse when serving traffic.

ofcourse this would be possible (with just another router) but this would still result in inefficienties.

IP Fragmentation is something which you do not want (it was removed from IPv6 for a good reason). TCP does not like it when IP packets get fragmented, as this usually results in lost PDU's when there is even the slightest of packet loss. Which causes TCP to resend the PDU.