Set theory and alternative foundations

This was going to be a comment to Joel David Hamkins's answer on geometry, but it didn't fit.

+1 This is one of the most clear-minded things I have read on MO. It does not make a mockery of Foundations and still says something non-obvious.

I'm very skeptical of all this business with category theory being a foundation for mathematics. First, whenever anyone talks about it, it always seems to be somebody else's work. It's become something of a meme that "Bill Lawvere has proposals to provide foundations for math through category theory", but we don't ever see details provided.

Second, are we really to understand that we are going to add small integers with arrows and diagrams? Draw circles and lines, and say that the latter meet at most once? I think people work in trans-Euclidean hyperschemes of infinite type so much, they forget that math includes these things.

As Wittgenstein remarks in the Investigations, just because you can express A in terms of B, it does not mean that B actually underlies A.