You cannot delete a question from a survey that has gone live. What is going 'live' you may ask? It is defined by data being associated with the survey , i.e. surveys being completed for that particular survey. What you CAN do is add questions to a live survey or add responses to question's responses.
I hope this helps!