history - Acknowledging differentiation of species, in historical times

The branch of science you are looking for is taxonomy, that is the science of identifying and naming species, and arranging them into a classification.

Modern taxonomy was born from the studies of the Swedish zoologist Carl Linnæus (1707-1778), who first introduced, in his books Systema Naturae (Systems of Nature) and Species Plantarum (Plants Species) the now common binomial nomenclature where each different species is given a Latin name composed by two parts: one identifying the genus and one identifying the species.

For instance, various species of mice are in the genus Mus: the common house mouse is Mus musculus, but in the West Mediterranean you have another type of mouse, called Mus spretus.

Although this rigorous type of classification is quite recent, taxonomy existed much earlier.

Shennong, Emperor of China somewhere around 4000BC apparently tasted hundred of plants to test their curative properties. He wrote his observations in a book called the Shennong Ben Cao Jing.

On a similar note the Ebers Papyrus, dating ~1550 BC contains description of the properties of many plants.

To more "recent" times, around 300BC Aristotle was the first who actually classified animals (e.g. vertebrates and invertebrates) and his student Theophrastus wrote a classification of plants in his Historia Plantarum (Hystory of the Plants).

Some 400 years later Plinius in the Naturalis Historia (Natural History) enumerated many plants and animals and gave some of the first binomial names to certain species.

As to the point of how did they distinguish species: well, with their eyes and ears, of course!
You can distinguish a mouse from a vole because it is skinnier and has a longer tail.
Even more similar species can be easily distinguished without needing special equipment.
A good birdwatcher can distinguish a chiffchaff from a willow warbler by listening to their songs, looking at how they behave how they fly, the subtly different tones of their feathers etc.
We can do that now, without any special equipment, so they could before the Renaissance!