My favourite SF stories concern aliens so different from ourselves that they are incomprehensible - not least because this will probably be true. The gold standard here is Stanislaw Lem in his books Solaris and His Master's Voice.
I'm interested in how we could understand the nature of such alien incomprehensibility. The standard model of personality is the five-factor model with the acronym OCEAN (*). This model has been applied with some success to animals and although it's somewhat atheoretic and definitely not rooted in evolutionary psychology, one can sort of see why a social animal of our sort (not as social as ants, not as selfish as, say, polar bears) would have its personality space structured along those five dimensions.
One way to design the alien would be to move their personality somewhere else in the OCEAN space, somewhat removed from the locus of human occupancy. So the aliens might be very 'Open': smart and enquiring; intellectual, cool and detached but still kind of benevolent. Or we could work on their 'Agreeableness': they can be super-friendly and trusting, or super-aggressive and suspicious.
But I'm more interested in the really alien aliens, ones so different that they don't classify in the OCEAN space at all. These aliens are going to come out of a radically non-terran ecology.
One example: there are chess programs that play perfect endgames based on storing optimal-play trees built via exhaustive search. In endgames of the order of 50-100 moves, there's no theory underlying the program's play - it's just empirically optimal.
Playing such a program is profoundly unsettling. The program evades or subverts any attempt to assign it plans or subgoals as it has none (more accurately, the structure of said endgames does not lend itself to such a decomposition); the program moves perversely yet with ruthless efficiency. It is devoid of intentionality: truly alien.
Another route which we sometimes encounter here on Earth: our opponent has objectives we profoundly don't understand. We keep ascribing goals, beliefs, plans and intentions based on our hypothesising as to what it's after: we're always wrong, always wrong-footed, always mystified. But here the alienness is not in personality but in motivation. Not the same at all.
Read Part 2.
* OCEAN: Open, Conscientious, Extravert, Agreeable, Neurotic .. are the five dimensions.