ct.category theory - Between abstract and concrete: What's the right way to think of specific categories?

Regarding the vivid discussion in the comments after the question (and hopefully, also of some interest for the question itself): I think that a "metacategory" is a definition by axioms, using only first order language, while "interpretation" means: an interpretation as in logic (say, as in p. 29 of Ebbinghaus-Flum-Thomas).

So such an interpretation (a category) is a set, or for convenience, several sets: A set of "objects," a set of "arrows" two function (that is, two more sets) "dom, cod" from the set of arrows to the set of objects, a function "1" from the objects to the arrows, a function "$circ$" on the pairs of composable arrows, etc., that satisfy the first order axioms of a metacategory.

In summary, I agree with the comment of Qiaochu Yuan: set theory is involved, but not because the objects should somehow be "sets with structure."