Why won't the Milky Way colliding with Andromeda affect the Solar System?

I've always found that somewhat strange when they say no stars will hit out of billions, but I also trust the scientists that they know their stuff.

Lets look closer at your numbers. If the Oort Cloud has a diameter of 2 light years (I think it's a bit less than that, but numbers vary), that's a radius of 1 light year, and the Solar System to the Oort Cloud has a volume of $frac{4}{3} pi R^3$, or $frac{4}{3} pi$ cubic light-years.

With 5 light-years average distance from stars, that's an area 125 times larger. So our Sun fills up (to its Oort Cloud) 1/125th of its region of space, and 5 light-years might be low too. The average closest star might be a bit more than that.

But not all stars are as big as our Sun; many are smaller and if two Oort Clouds pass through each other, not much happens. A star has to pass within our Oort Cloud for the gravitational changes to be interesting, and that doesn't happen very often.

Another thing to consider is that the Earth is in a spiral arm of our galaxy, and both the Milky Way and Andromeda are flat disk-shaped galaxies. Are we crashing head on or at an angle where our spiral arms won't be directly hit?

I don't think we'll care much 4 billion years from now. I like to think we'll have traveled beyond the Solar System by then, but the danger to the Sun from Andromeda probably isn't huge. That said, I'd guess that some stars will pass near other stars, even if the experts say direct hits are unlikely (from the article)

If the Sun were a ping-pong ball, Proxima Centauri would be a pea about 1,100 km (680 mi) away, and the Milky Way would be about 30 million km (19 million mi) wide. Although stars are more common near the centres of each galaxy, the average distance between stars is still 160 billion (1.6×1011) km (100 billion mi). That is analogous to one ping-pong ball every 3.2 km (2.0 mi). Thus, it is extremely unlikely that any two stars from the merging galaxies would collide

If stars are ping pong balls every two miles and we use our Solar System as an example (it's really not, because the ping-pong-ball-every-two-miles example is in the center of the Solar System), but if the Sun is a ping pong ball, the Earth orbits at a radius of 4.3 meters and Neptune at a bit over 120 meters and the outer edge of the Kuiper Belt about 200 meters. That's still small compared to the average distance between stars, even in the galactic center, but not so small that interaction would never happen.

The ping pong balls might miss each other but I think it's a safe bet that there will be the occasional star-to-outer planet close encounter. From a local perspective these will be rare, but from an entire galaxy perspective, I'd think there would be quite a few of what could be called, planetary fly-bys/near misses.