the lord of the rings - Why didn't Gollum ever go to the Shire to search for Bilbo?

The Fellowship of the Ring contains a description from Gandalf about his thoughts about what happens after Bilbo takes the ring.

Gollum is afraid of the light after so many years living under the misty mountains so it takes some time before he leaves them to search for the ring. Gandalf then surmises that, because of his very long association with the ring, he is drawn by Sauron's growing power and makes his way south to Mordor and is eventually captured and tortured.

All Gollum knows is that the ring was taken by 'Bagginses' from 'The Shire', but from his actions clearly doesn't know where that is.