When it comes to unfair accusations in the workplace, the rule of the jungle, unfortunately, is "perception is reality."
If you boss perceives that you acted badly toward this temp, you have to refute it as simply as possible and move forward from there. Refuting it with vehemence will only persuade your boss that it's true.
The important thing is your ongoing relationship with your boss. Don't make this about the temp and his/her accusations. That's history. Focus on your job performance and your relationship with your boss.