grammar - Is "switched" always used as a verb?

The participle of just about any verb can be used attributively. A short-cut way of saying that is saying it acts as an adjective.
Instead of mentioning for every verb that its participle may function like an adjective, most dictionaries do not include that.

A shirt can be washed, worn, bought, sold, switched, torn, lost, made, found, painted, burnt, etc. That does not mean you will find all those participles in a dictionary as an adjective — they are not.

Just about any word that can be used to act as an adjective can also be used as nouns, although that is not extremely common. That means I can do that with participles as well; I can certainly use the punished to refer to the punished prisoner, or the conquered to refer to the conquered people.

Again, if you look up punish or conquer, I doubt many dictionaries will feel the need to explicitly mention that you could use the past participle as a noun.

Referring to the switched shirts as the switched would be, I think, at best poetic.

In your sentence

The shirts are switched.

Switched is certainly not a noun, by the way. It is again, an attributively used participle, in the same way you could use any adjective:

The shirts are blue.

The only noun there is shirts, are is a copula linking the adjective blue to the shirts. Blue would only be a noun in a sentence like:

The reds and the blues are switched.