A bitterent that can cause allergic reactions (and most substances can!) is on a completely different scale to something that is an actual poison and consuming a small amount can cause permanent damage/kill you, regardless of your immune system's sensitivity.