The core question of topology

As others have noted, it's hopeless to try to answer this question for general topological spaces. However, there are a few positive results if you assume, say, that X and Y are both simply connected closed manifolds of a given dimension. For example, Freedman showed that if X and Y are oriented and have dimension four, then to check whether they're homeomorphic you just need to compute (i) the bilinear "intersection" forms on H^2(X;Z) and H^2(Y;Z) induced by the cup product; and (ii) a Z/2-valued invariant called the Kirby-Siebenmann invariant. The invariant in (ii) obstructs the existence of a smooth structure, so if you happened to know that both X and Y were smooth manifolds (hence that their Kirby-Siebenmann invariants vanished) you'd just have to look at their intersection forms to determine whether they're homeomorphic (however a great many examples show that this wouldn't suffice to show that they're diffeomorphic).

In higher dimensions, Smale's h-cobordism theorem shows that two simply connected smooth manifolds are diffeomorphic as soon as there is a cobordism between them for which the inclusion of both manifolds is a homotopy equivalence. Checking this criterion can still be subtle, but work of Wall and Barden shows that in the simply-connected 5-dimensional case it suffices to check that there's an isomorphism on second homology H2 which preserves both (i) the second Stiefel-Whitney classes, and (ii) a certain "linking form" on the torsion subgroup of H2.

If you drop the simply-connected assumption, things get rather harder--indeed if n>3 then any finitely presented group is the fundamental group of a closed n-manifold (which can be constructed in a canonical way given a presentation), and Markov (son of the probabilist) showed that the impossibility of algorithmically distinguishing whether two presentations yield the same group translates to the impossibility of algorithmically classifying manifolds. Even assuming you already knew the fundamental groups were isomorphic, there are still complications beyond what happens in the simply-connected case, but these can sometimes be overcome with the s-cobordism theorem.

In a somewhat different direction, in dimension 3 one can represent manifolds by link diagrams, and Kirby showed that two such manifolds are diffeomorphic (which in dimension 3 is equivalent to homeomorphic) iff you can get from one diagram to the other by a sequence of moves of a certain kind. (see Kirby calculus in Wikipedia; similar statements exist in dimension 4). I suppose that one could argue that this isn't an example of what you were looking for, since if one felt like it one could extract diffeomorphisms from the moves in a fairly explicit way, and one can't (AFAIK) just directly extract some invariants from the diagrams which completely determine whether the moves exist.