Is dust extinction/reddening caused by Rayleigh scattering or some other physics phenomenon (or both)?

In an astronomical context, Rayleigh scattering does not take a huge place.
The extinction is mostly due to absorption and scattering, from gas and dust. Its effect is to diminish the flux from the source (that is, a change in magnitude). Please, take care this depends on the observed wavelength!

For simplicity, we can assume the extinction factor $Q_{ext}$ as:

$Q_{ext} = Q_{scattering} + Q_{absorption}$

Where $Q_{scattering}$ is in turn composed of many elements.

As the OP above said, blue light generally suffers of higher extinction than other frequencies, so causing reddened observed light. This is due to the cross-section dependence on the frequency: if the light has a wavelength that is comparable to the size of the grain dusts (high frequency), it is more effectively absorbed, while in the opposite case it can pass through undisturbed.
This is the reason why obscured fields are observed in radio (in our galaxy, especially) or in infrared (extragalactic).

So, the answer to you question, in space is: Rayleigh scattering is not the most responsible of the extinction/reddening, you have to invoke absorption/scattering by dust and gas.
For comparison, here some explanatory numbers about the extinction efficiency in different contests:

$Extinction simlambda^{-1}$

$Thomson simlambda^0$

$Rayleigh simlambda^{-4}$

You basically would need impressive amount of gas to account for the extinction just by Rayleigh scattering.
Instead, Mie scattering is more important in this context. Here a figure to quickly grab the concept (from here):

And here a very popular plot about extinction versus frequency from here:

Of course, if you observe from ground, you should take care of Rayleigh scattering as well.

Other references:
Lecture 1
Lecture 2