ct.category theory - Internal hom of sheaves

Well, here is a partial answer. The category of abelian group-valued sheaves is not a topos, the category of set-valued sheaves is. And I think you should look at set-valued presheaves, at least $hom(-,U)$ is one:

When the site comes from a topological space, you can see as follows that your two definitions coincide: When you insert into the $hom$ in your second expression an open set $V$, you either get $hom(V,U)=$the one-element-set containing only the inclusion, if $V subseteq U$, or $hom(V,U)=$empty set, if not, so taking a product with the set-valued presheaf $hom(-,U)$ either leaves X as it is or "deletes" it (i.e. transforms it into the (presheaf with value the) empty set).

So a natural transformation in $hom_{PreShv}(X times hom(-,U), Y)$ is given just by its components for all $V subseteq U$ and for other $V$ it extends to the unique map from the initial object into Y. This is the same as a natural transformation of restricted presheaves.

Edit: It now occurred to me that you were probably actually speaking of the sheaves of group homomorphisms, not just any natural transformations. You get that, when you apply the "free group"-functor to the set-valued sheaf "hom(-,U)" and take tensor product instead of product. The free group over the empty set ist the trivial group, thus the initial object in groups and tensoring with it trivializes $X$. Tensoring with the free group over one element doesn't change anything, so the same things happen as in the set case...

For more general sites, the expression $X| _U$ probably means that you look at $X$ as a functor defined on the slice category given by maps on your site into its object $U$. But you can also see $X$ as a sheaf not just on the site, but on all of its ambient sheaf category (as a hom-functor, and where you give to topos an appropriate topology - you sometimes do this in topos theory). Then looking at restricted sheaves is the same as passing to the slice category $Shv / hom(-,U)$, and $X times hom(-,U)$ is the image of $X$ under the canonical functor $Shv rightarrow Shv/hom(-,U)$ (which is given by taking product with $hom(-,U)$), so it actually $is$ $X| _U$. This still doesn't make it clear to me why the two functors in question coincide, but I guess it may be a way to look at it to find it out...