I think the usage of "refuse" and "fight" might be the culprit in the comms-breakdown here.
Cue, let's just use a hypothetical:
You get an exam date from the VA. Its an exam you don't feel is necessary. You called the WH hotline to "fight" the necessity of the exam, NOT to refuse an exam. I think we can all agree on that, right?
Now assume this exam is a short-notice exam, taking place in the coming days. Hotline says they'll forward it along to have your concern addressed.
The date of the exam arrives and no word from hotline or anyone. Do you attend the unnecessary exam? Or do you just not attend pending hearing back from WH hotline?
I feel situation like this or others I probably have not experienced myself, could be what others are driving at when the say refusing an exam. While you're not outright refusing anything, the VA will ultimately view it that way of you do not attend.
So I believe the distinction between refusing and fighting the necessity of the exam should be more the forefront of this??