I have experienced this too. I have eye contact anxiety...not sure how much to give or if I am giving too much and thus making the other person feel uncomfortable. I think if others look away they are just thinking about what they are going to say, but we wrongly feel they are doing it because they are u ncomfortable by us (ANTS at work!). The other theory I have about this is that the person who looks away is also suffering from social anxiety. I can usually tell is someone else has social anxiety...takes one to know one I guess. When I encounter others who seem to have eye contact anxiety it really heightens my eye contact anxiety, as I then feel like my eye contact is incorrect and making them feel uncomfortable.