nt.number theory - about Hilbert and Siegel modular varieties (forms)

It seems to me that Hilbert modular varieties (forms) are generalization from Q to totally real fields. While Siegel modular varieties (forms) are generalization from 1 dimensional to higher dimensional abelian varieties. But they should both be some kind of Shimura variety (automorphic forms), right?

According to Milne's note of Shimura varieties, Siegel modular varieties are Shimura varieties coming from the Shimura datum (G, X) where G is the symplectic similitude group of a symplectic space (V, phi). So what is the corresponding Shimura datum for the Hilbert modular variety? Or am I asking a wrong question?

Also, in the definition of Shimura varieties G(Q)G(A_f)X/K, why we only consider Q and its adele group? why not general number fields and their adeles? (again, maybe a wrong question)

Thank you.