Storage size of the index itself, insert/update performance. Also there may be very little gain from adding an index to a column with only few distinct values or a very skewed distribution (partitions may be a better approach in those cases).

you have to build it and store, so the footprint of this data structure has now increased.

If the array of numbers is mutable, the index also needs to be a modifiable structure (like a tree).