Self-confidence based on characteristics or abilities you have can have the dangerous effect of preventing growth on those abilities. For example, basing your self-confidence on being knowledgable can create an incentive to shrink away from situations in which you would be proven wrong.
“Philosophers that give themselves airs of superior wisdom and sufficiency have a hard task, when they encounter persons of inquisitive dispositions, who push them from every corner, to which they retreat, and who are sure at last to bring them to some dangerous dilemna.”
- Hume, An Enquiry Concerning Human Understanding.
If my self-confidence stemmed from my comparative abilities, it would be hard to spend most of time with people who I consider to be more skilled than me, which is what I try to make the habit of doing. Instead, I don’t actually think in terms of self-confidence; I find it more fruitful to focus on self-compassion. Self-compassion avoids this issue, and also has the benefit of tending to make you more compassionate towards others as well.