Sword slashing logic on Gear VR using controller?

Description

I have a Gear VR game that has a sword fight level and I want to inflict damage to enemies when the player swayed their sword and hits the enemy. Figured I can do this with collision but there are certain problems that I need to address.

Background of the problem

I replaced the Gear VR model to a sword model basically and I can sway the sword with no problem but I want to only damage the enemy if it has been swayed/slash before hit.

Main Question

The sword should not damage an enemy if the user did not sway the sword hard and fast. The problem is if the sword collided with an enemy (getting near) it will inflict damage because of the OnCollisionEnter. How can I do this the smarter way? I need to check if it has been slashed. (Check velocity of Gear VR  controller upon hit perhaps?)

For now, I am still testing possible solution to my problem and will be posting here as answer if I discovered any. Meanwhile, any ideas or answers in regard with this problem is welcome. Thank you for your help.