I certainly have sympathies for that view. And Malcolm Fraser certainly took it upon himself to accept refugees from Vietnam in the aftermath of the Vietnam war, wearing the responsibility for Australia's involvement there. I suppose there was the extra element too, that most of the refugees from Vietnam had been associated with the South Vietnamese regime, so were allied to us and were facing persecution and/or being marched off to reeducation labour camps by the incoming Communist regime as a result.
Ironically though, it was Whitlam - the demigod of the compassionate left in Australia - who was furious that Australia would take these boat people. Afterall, they were anti-communist South Vietnamese and wouldn't vote for him. While some may point out that the reverse is also true - it suited M.Fraser to take anti-communist refugees - I give him the benefit of the doubt, that he did it because he thought it the right thing to do. That said, in the hypothetical scenario that the Southern regime had prevailed and people associated with the Communists were fleeing persecution, would Fraser have been so keen to take them? That would have been the test.
Anyway, just goes to show the cynical politicking over this issue is sadly, something that has gone on for many years.