Giving evidence during the trial of the MP, who is accused of groping a teenage boy in 2008, the man said that while working with Khan more than a decade ago in Pakistan, he was assaulted by him after a night of drinking.
The man, who cannot be named for legal reasons, said he had been with Khan in Peshawar in 2010.
He told Southwark Crown Court that Khan knew politicians and Army generals in the country and was "well placed and quite influential".
He said he knew Khan was gay - that it was an "open secret" - and that he would often become flirtatious after becoming drunk, tickling the man or putting his feet across him, saying he thought the behaviour was "inappropriate".
He said that while staying in a government guest house in the northern city, they had been out drinking one evening, had returned to the house and was offered a sleeping pill by Khan.
After taking the pill, the man - who told the court he was not gay - said he fell asleep but woke a short time later to find Khan performing oral sex on him.
He said he asked Khan: "What the f*ck are you doing?" and pushed him away.
He told the court he was unhappy and upset with what had happened, left the next morning and quit his job. He said he told the British High Commission, his family and reported it to the Foreign Office.
When cross examined by Khan's barrister, Ms Gudrun Young QC, she asked the man if his memory had been impaired by a combination of drink, the sleeping pill and cannabis that he had smoked earlier in the night.
He replied: "The memory of waking up and seeing what he was doing to me is very vivid."
She suggested the man had been a willing participant, adding: "What took place between you in that moment was something you were awake for and consented."
The man denied this, and said he was angry and upset at the incident.
He said that he did not go to the police in Pakistan because he thought Khan could "manipulate" the situation because of his influence.
The man said: "I was worried professionally and legally about what he could say to people."
Khan is facing an allegation of sexually assaulting a boy after a house party in Staffordshire in January, 2008.
Described as a 'friend of a friend' to the host family, he was permitted to stay in a spare bed in the boy's room.
It was alleged that Khan had been drunk and groped the boy in his bunkbed after the lights had been switched off, moving from his leg up towards his groin, before the boy ran out.
But Khan told officers that he "categorically denied" the assault took place.
He said that he had not tried to force the boy to drink alcohol, as suggested by the youngster, but did admit to talking to him about watching pornography.
But he claims that was linked to an earlier conversation they had about sexuality, and that watching pornography would simply "indicate where one's attractions lay". He denied it was suggestive or flirtatious.
The trial continues.