ag.algebraic geometry - Extending group actions on varieties

Well, not always, e.g. take $X=mathbb P^2$ with homogeneous co-ordinates $x,y,z$, $U$ the locus defined by $xyzne 0$ and $sigma$ the involution of $U$ defined by $(x,y,z)mapsto (yz,xz,xy)$ (the standard quadratic Cremona transformation of $X$). Then $sigma$ is of order $2$ and does not extend to $X$. On the other hand some sufficient conditions, in characteristic zero, are: there are no rational curves in $Xsetminus U$; $X$ has ample canonical class and is smooth (this can be weakened to having canonical singularities).

[Update: Charles Siegel asked for references, and I have none to hand, although this is all well known.]

For the Cremona example, recall that every automorphism of $mathbb P^2$ is linear, and $sigma$ clearly isn't. More geometrically, $sigma$ blows up the vertices of the triangle $xyz =0$ and collapses its sides.

No rational curves in $Xsetminus U$: fix $gin G$ and think of it as a rational map $g:Xto X$. By Hironaka, there is a minimal composite $p:Zto X$ of blow-ups such that $gcirc p$ is regular. If $Xsetminus U$ has no rational curves, then the rational curves in the exceptional divisor of the last blow-up are contracted, so the last blow-up was redundant, contradicting minimality.

Ample canonical class $omega_X$: then $X=Proj(R(X,omega_X))$, with $R(X)= oplus_{nge 0}H^0(X,omega_X^{otimes n})$, the canonical ring. This is a birational invariant of $X$, so $G$ acting on $U$ acts on $R(X)$, so on $X$.