Which RPG is being played in Gamers: Dorkness Rising?

You are correct in your assumption that they are playing DnD 3.5.

More specifically, they are playing the module called The Mask of Death, which is a real game.

From the Mask of Death link:

This module is a special limited edition produced exclusively for Gen Con Indy 2005. It presents chapter one of an adventure that chronicles the hilarious escapades of the hit film The Gamers: Dorkness Rising.

More info. can be found on the wiki page.