ag.algebraic geometry - How to compute the ring of invariants of SO_3(k) acting on a polynomial ring

This is addressed by the classical invariant theory, but the answer is more complicated than for general linear or orthogonal groups (in particular, not all minimal generators are quadratic). Let $k$ be a field of characteristic 0. The group $G=SO_m$ acts on $mtimes n$ matrices by the left multiplication and this induces a $G$-action on $A=k[X_{ij}].$ Let us view the variables as the entries of the $mtimes n$ generic matrix over $k.$ Then the algebra of invariants $A^G$ is generated by:

1 Scalar products of the columns of the matrix $X.$

2 Order $m$ minors of the matrix $X.$

This is the First Fundamental Theorem (FFT) of classical invariant theory for $SO_m.$ In fact, the elements of the first type generate $O_m$-invariants and the elements of the second type generate $SL_m$-invariants ($SO_m=O_mcap SL_m$).

Moreover, all relations between these generators are also known (the Second Fundamental Theorem, SFT) and there is a good description of a standard monomial basis of $A^G.$ If I am not mistaken, the last part is due to Laskshmibai and coauthors. A comprehensive modern reference is

Laskshmibai and Raghavan, Standard monomial theory. Invariant theoretic approach. Encyclopaedia of Mathematical Sciences, vol 137 (Invariant Theory and Algebraic Transformation Groups VIII), Springer.