Based on my experience, what irritates me are those who talk behind your back,but very nice in front of you,spreading rumors that can affect your reputation at work. They just can't accept the fact that you are doing great in your job than they are.You can't avoid this kind of people, BUT ,i prefer to talk to them, to make everything CLEAR and just say- mind your own business!
Its always irritates me when people talk behind you and do all kind of office politics. its also irritates when you are trying to give your100% and the credit goes to others without any recognition to you.