Anything is possible, but you had better make the change quickly and make certain you are proficient in the required mathematics and physics before you make the jump. You will be competing with students in graduate school who have had a head start on you and taken the 'right' courses from the start. I suggest you speak to someone in your physics department about your realistic prospects for making the switch.