I would suggest that if the choice is between physics and mathematics in a college that has no astronomy department, that you go with physics. If you do very well in physics, mostly A's and a few B's, then you stand a good chance of getting into a reasonably good graduate school in astronomy. The competition is very rough so you better learn everything you can in physics. Don't worry about the mathematics. In a physics major, you will be required to take all of the necessary math you need. This will probably NOT include any advanced topics in mathematics such as topology. If you are at a college with no astronomy department, you better talk to a few profs about your interest in astronomy. Make certain you ask them how many students in the past majored in astronomy, and above all, what happened to them after they got their BA in physics!