You owe tax on the difference between contributed dollars and converted dollars. So if you contributed $100 and it historically had grown to $150, you would owe tax on $50 to convert.

If that $150 in value is suddenly $120 instead, your tax bill to convert is significantly cheaper.