The exact mechanism is not known:
Modafinil has wake-promoting actions similar to sympathomimetic agents like amphetamine and methylphenidate, although the pharmacologic profile is not identical to that of sympathomimetic amines. [...] Modafinil-induced wakefulness can be attenuated by the α1-adrenergic receptor antagonist prazosin; however, Modafinil is inactive in other in vitro assay systems known to be responsive to α-adrenergic agonists, such as the rat vas deferens preparation. [...] Modafinil is not a direct- or indirect-acting dopamine receptor agonist. However, in vitro, Modafinil binds to the dopamine transporter and inhibits dopamine reuptake. This activity has been associated in vivo with increased extracellular dopamine levels in some brain regions of animals [1].
A 2010 study has concluded that:
Recent evidence suggests that modafinil may block the dopamine transporter (DAT) and that the dopamine D1 receptor (D1R) may contribute to modafinil effects [2].
Another one confirms modafinil's dopaminergic activity:
These findings indicate that modafinil has a long duration of action, with alerting properties arising predominantly from dopaminergic activity [3].
It seems to be dependent on catecholaminergic signaling:
It binds competitively to the cell-membrane dopamine (DA) transporter and is dependent on catecholaminergic (dopaminergic and adrenergic) signaling for its wake-promoting effects. [...] Modafinil is an exceptionally weak, but apparently very selective, DA transporter inhibitor. [...] The conformational constraints on the interaction of modafinil with the DA transporter - and probably, as a consequence, its effects on trace amine receptor signaling in the catecholaminergic cell - are unique among catecholaminergic agents [4].
References:
- Drugs.com. Modafinil. Clinical Pharmacology. Available from http://www.drugs.com/pro/modafinil.html (accessed 29.07.2014)
- Young JW, Geyer MA. Action of modafinil--increased motivation via the dopamine transporter inhibition and D1 receptors? Biol. Psychiatry. 2010 Apr 15;67(8):784-7. doi: 10.1016/j.biopsych.2009.12.015. PubMed PMID: 20132929.
- Turner C, Belyavin A, Nicholson A. Duration of activity and mode of action of modafinil: Studies on sleep and wakefulness in humans. J. Psychopharmacol. (Oxford). 2013 Dec 3;28(7):643-654. doi: 10.1177/0269881113508173. PubMed PMID: 24306135.
- Wisor J. Modafinil as a catecholaminergic agent: empirical evidence and unanswered questions. Front Neurol. 2013 Oct 7;4:139. doi: 10.3389/fneur.2013.00139. PubMed PMID: 24109471.