The short answer is that you can ask, but expect for the judge to deny your request. Under the Sixth Amendment, you have the right to an appointed attorney--but not the right to choose which appointed attorney you get!
Public defenders are like anything else: Some are great, some are lousy, and many are good. But all public defenders are overloaded with cases. I remember preparing cases for trial during the day, interviewing witnesses and drafting motions, and then listening to my voicemails at 6pm to make a list of people to call back. Sometimes it would take me an hour just to listen to all of them!
If you're not getting a call back--or a response to a letter or email--it's difficult to know if your attorney is busy or a bad attorney. See if you can schedule a face-to-face meeting with your public defender. If you can't get a hold of him to do that, call the main office line and see if an administrative assistant or supervisor can schedule a meeting for you. If leaving voicemails isn't working, write emails and letters.
But you also need realistic expectations. A public defender's time is extremely limited, and he or she is focusing on what's happening in court next week--not next month. You deserve to have an attorney who is zealously representing you in court, so don't give up trying to communicate with your public defender.