soft question - Too old for advanced mathematics?

Kind of an odd question, perhaps, so I apologize in advance if it is inappropriate for this forum. I've never taken a mathematics course since high school, and didn't complete college. However, several years ago I was affected by a serious illness and ended up temporarily disabled. I worked in the music business, and to help pass the time during my convalescence I picked up a book on musical acoustics.

That book reintroduced me to calculus with which I'd had a fleeting encounter with during high school, so to understand what I was reading I figured I needed to brush up, so I picked up a copy of Stewart's "Calculus". Eventually I spent more time working through that book than on the original text. I also got a copy of "Differential Equations" by Edwards and Penny after I had learned enough calculus to understand that. I've also been learning linear algebra - MIT's lectures and problem sets have really helped in this area. I'm fascinated with the mathematics of the Fourier transform, particularly its application to music in the form of the DFT and DSP - I've enjoyed the lectures that Stanford has available on the topic immensely. I just picked up a little book called "Introduction To Bessel Functions" by Frank Bowman that I'm looking forward to reading.

The difficulty is, I'm 30 years old, and I can tell that I'm a lot slower at this than I would have been if I had studied it at age 18. I'm also starting to feel that I'm getting into material that is going to be very difficult to learn without structure or some kind of instruction - like I've picked all the low-hanging fruit and that I'm at the point of diminishing returns. I am fortunate though, that after a lot of time and some great MDs my illness is mostly under control and I now have to decide what to do with "what comes after."

I feel a great deal of regret, though, that I didn't discover that I enjoyed this discipline until it was probably too late to make any difference. I am able, however, to return to college now if I so choose.

The questions I'd like opinions on are these: is returning to school at my age for science or mathematics possible? Is it worth it? I've had a lot of difficulty finding any examples of people who have gotten their first degrees in science or mathematics at my age. Do such people exist? Or is this avenue essentially forever closed beyond a certain point? If anyone is familiar with older first-time students in mathematics or science - how do they fare?