How was the nuclear reactor failure simulated?

In short, it isn't simulated, it is a real, but small leak.

In the book, someone makes a small radiation leak by drilling a small hole through a thread of a connector in some part of the cooling system, a part that is under not enough pressure to leak when the reactor is not fully powered up to run the engines. It is unsaid, but the reader is meant to presume that this has been done by the chief engineer so that they do not show the leak. The fixed radiation detectors near the reactor have also been disabled. The Engineer has also tampered with radiation dose badges that all crew members wear, by exposing to an X-Ray machine in the doctor's surgery.

A few days into the plot, the doctor who is not part of the conspiracy routinely collects the badges and photographically develops them - and discovers that they show that a leak has occurred. This is part of the plan to unsettle the crew, and provide an excuse for unexpected behavior from the captain. There really is a leak, but the conspirers have made it appear bigger than it really is.

Initially the officers send some middle ranking engineers to find the leak with a hand-held detector, and they cannot because the reactor is powered down and the steam from the cooling system isn't under enough pressure to leak - so the non-conspirators do not know whether there is a leak or not, perhaps the dose badges are faulty or someone has deliberately exposed the badges to radiation. So they power up the reactor and continue the journey to Cuba.

It is only later in the plot that the leak gets bad enough to trigger a detector further forward in the submarine, that the chief engineer joins in the search for the leak and 'discovers' how the leak was made - by this time the crew is very unsettled by the leak even though the levels of radiation are low - so the leak has performed its purpose by the time it is fixed, promotes rumors of sabotage, and sets the scene for them using the reactor as an excuse to do whatever is needed later in the conspiracy.

I think it was important to the conspirators to have this take a few days to work through, as this kept the crew busy and worried, but that it was eventually found and fixed. An uncontrolled leak would have given the conspirators only few possible plausible courses of action, and at this stage they do not know how events will work out. So they manage to engineer a situation where they can use the reactor as an excuse if necessary, but only if necessary.

The book is quite long and includes all this detail - a lot of this is left out of the movie obviously, but the principle of the plot is still the same - the officers engineer a deliberate leak.