meaning - Prefix im- is for opposite or asserting

It should be noted that the prefix im- is really just the prefix in- with the consonant adjusted because of the consonant that follows.

As for passive and impassive, it's a quirk of the history of the meaning that they've ended up with similar meanings. Originally passive meant feeling pain and impassive meant not feeling pain. The former came to mean not active while the latter came to me without emotions.

But you're right, as other cases show, in- is a particularly difficult prefix to discern meaning from for words you haven't encountered before.