Leggo My Ego

I've been lucky so far in life, in that I haven't had a whole lot of exposure to the sort of grimy and inapproriate sleazebags that are the bane of so many working women's existences.  (And men's too, perhaps.)
A conversation I had today reminded me of this and I got to thinking about what sort of inappropriate behavior I have endured, and just how bad it was compared to the sort of unbelievable scumbaggery that apparently actually happens out there.  To real people.
I haven't experienced anything in the way of sexual harrassment.  Nor have I been exposed to anger or hostility in the workplace, or a feeling of discomfort that I would feel strongly enough to report.  I guess I've been pretty lucky. 
What I have experienced is a general sort of inappropriateness that I am not even sure how to categorize.  For example, about a year ago, I announced that I was taking a new job within my organization and would be vacating my current position.  When asked, by a certain colleague, why I had made that choice, I explained that I wanted to do more for the organization and thought I would be better positioned to accomplish that via the new role.  The response I got?  "Wow, what an ego on you!"  I was sort of taken-aback.  The sort of taken-aback that is not sure how to respond and so, in the absence of the biting comeback you wish for, you stand mute trying to figure out what is the best way to handle the situation.  I think I stammered something about not having an ego and that wanting to do more for your organization doesn't count as having an oversized ego, but I'm sure it came out more like this:  "Um, well, uh, I don't really think... uh"  So it's kind of like I didn't say anything at all.

So here is the real issue.  Is it a bad thing to have a big ego?   If ego means the opinion you have of yourself and your self-importance, is it wrong to have a big one? 
The most assholey people I know are the ones who have low self-esteem, because they feel like they have to compensate for their shortcomings by being "toppers" and throwing everyone else under the bus all the time to make themselves look better.  The self-confident ones are the ones who don't mind giving other people the credit and pursuing the greater good rather than self-gain.  If that is the side effect of having a big ego - than I say serve me up another platter of self-importance.   What is so wrong with wanting to find a place for yourself where you feel like you're doing some good rather than just sitting around waiting for someone to give you work?  So I felt under utilized... I could have either sat around for the next 30 years being under utilized and taking full advantage of it, or I could admit that I can do more and actively seek out a place for myself where more would be asked of me.  In what fucked up universe is the former the preferable option just because having a "big ego" might get me a place at the unpopular table?