Is it illegal for a guy to threatening to beat up another guy?

Technically, the threat to do harm is an ault. However, many things, including his masturbation could be interpreted as an ault too, so your response ( a punch) could be considered an acceptable measured response.