Are summoners really that bad?

[Calianeri]

So would you guys recommend leveling as something else first just to get the avatars, then going full blown summoner?

You basically have to. Leveling SMN without the avatars is pointless IMHO. Carbuncle alone is not so hot, and the elemental spirits apparently don't do much. The Prime fights are, by all indications, not easy even for a party of level 60s. For main job up to that point, WHM would be a good choice, since it is the most common subjob for SMN. And you'll know if you like playing a caster class. You'll also need high fame in all the cities to be offered the avatar quests, that takes considerable time and at least a mid-leveled job.