orbital mechanics - When sending a probe to Mars, how is the optimal travel path calculated?

To define "optimal" you need an objective function that you are maximizing or minimizing. What is your objective function?

For real Mars missions, the objective function can be quite involved, since many factors are considered. Let's assume an impulsive trajectory (i.e. very close to on target to Mars immediately after launch). Then there is the mass that can be delivered to the target, which is a function of the launch energy and departure declination. There is the arrival velocity, which determines the orbit insertion propellent for a lander, or the heat shield capability and aspects of the entry trajectory for a lander. There is the approach declination, which determines what orbits you can get into with one burn, or what landing site latitudes you can access. There is the visibility of the insertion burn or entry from Earth for telecommunications during critical events, so that you have data if something goes wrong. There is Mars relay orbiter coverage constraints for the entry and landing event, again to get more data in case something goes wrong. You will need to define a few weeks of available launch days (usually three weeks) to allow for weather, range, launch vehicle, or spacecraft delays. Over that launch period, you will need to satisfy all the other constraints. You may want to have the arrival day be the same for all launch days, in order to simplify planning. You may want to not have that arrival day be on Super Bowl Sunday, so as to get better press coverage and to not annoy the crew. (I seriously took that into account once.)

I could go on. Does this answer your question?

Update:

To address the comment on optimizing for a specific parameter, e.g., $C_3$, the process for an impulsive trajectory is to make a porkchop plot. Like this:

For a given departure date from Earth and arrival date at Mars, there is one short, prograde orbit that connects them (see Lambert's problem). You then make a contour plot over a range of departures and arrivals of the parameters of interest. In the plot above, the blue contours are the injection energy. You can see two local minima, both around a $C_3$ of 16, for the 2005 opportunity. Though as noted, the $C_3$ doesn't tell the whole story. The departure declination, if greater than the launch site latitude, will reduce the injected mass at the same $C_3$.