planet - How is the mass of solar system objects calculated?

Planets are satellites of the Sun. Most planets have satellites.
Whenever you find a parent body with satellites, you get first good results about their masses and distances by applying Kepler's laws.
A second approach is using parallax measurements for distance estimates.
The distance to some of the planets can be measured very accurately by radar, e.g. for Venus.
The known planets of our solar system have been visited by probes. Using signal travel times, you can calculate the distance of the probe to Earth. One of the methods to determine the distance between probe and the planet is by photogrammetry.
Apply the measured data to the according system of equations. Usually more data than needed are available. Then you get an overdetermined system. A best solution can be found by minimizing appropriate squares.

Subtle deviations from Kepler's law can be used to infer masses of planets which are hard to measure in a more direct way.
Once you know the distance of a planet, use its apparent diameter(s) to calculate the actual diameter(s), and then the volume of a sphere or a spheroid.
With mass and volume you know the mean density of the planet.

Within our solar system, for very distant objects the apparent size of which cannot be determined you may estimate the surface albedo together with distance (by Kepler after knowing the mass of the Sun) and apparent magnitude to obtain a rough estimate of the size of the object.

Better size estimates for objects of small apparent size can be obtained by stellar occultations, meaning measuring the time a star is occulted by the object. Together with the angular velocity of the object you get data about its angular size.