For years the most common use for “serial” would be dialup internet and computer mice (among other peripherals), neither of which would have been primarily ASCII.

Not sure what you have in mind by “usually” ASCII, but it’s not the layer that serial links operate at, they’re just a dumb pipe.