Set Open Mic Threshold to higher than 8%.
Set different keys (and not the same as the Steam Voice one). Under Dota 2 Audio settings, uncheck both Open Mic. Set Recording and Receive volume to half. Under Steam Voice Settings, re-test your microphone. Check again if you disabled all other devices under Recording panel, leave only your mic, and set it both as default device and default communication device.
