The Qatar-owned club were leading the quarter-final, second leg 2-1 as the game went into stoppage time before Dortmund, who were held to a 0-0 draw in last week's first leg in Spain, equalised and then grabbed a 93rd-minute winner.
Replays showed several players from the Bundesliga team in offside positions in the build-up to the third goal and the normally restrained Malaga coach Manuel Pellegrini slammed the match officials in his post-match news conference.
The club's director general, Vicente Casado, said on Wednesday before flying back to Spain with the squad that Malaga would file a written complaint with European governing body UEFA about the refereeing.
"On this occasion we could not, or they did not want us to, get through," Pellegrini said. "After we went 2-1 ahead there was no refereeing," the Chilean added, although Eliseu's 82nd minute goal for the Spaniards also appeared to be offside.
"They forced us back with elbows and shoves. There were two sendings-off that were not given, a double offside in the third goal which should not have counted.
"They said this was the best team in Europe and by the end they were hoofing long balls forward. We are leaving with very bitter feelings."
Casado said the performance of Scottish referee Craig Thomson and his officials had not been up to the level of a Champions League quarter-final.
"We are outraged," he said."Not only Malaga lost but football too. Our complaint is not directed at UEFA but at the refereeing that was absolutely not at the right level."
Winger Joaquin, who put Malaga ahead in the 25th minute, and club owner Sheikh Abdullah Al Thani, a member of the Qatar royal family, were even more outspoken.
In a series of tweets on his official Twitter feed soon after the final whistle, Al Thani blamed the defeat on "racism" and called on UEFA to launch an inquiry.
UEFA general secretary Gianni Infantino on Wednesday dismissed Al Thani's claim, saying: "UEFA is not taking any racist action against any club.
"Now depending on what he has said or not said... I can understand if you lose a match in the 93rd minute then the emotions come up and you say things you have not really thought through and should not say," Infantino said at the Soccerex European forum in Manchester.
"I don't know how there could be any racist abuse from UEFA. How can he blame UEFA for racism when they (Malaga) lost the game from two late goals?"
Infantino added: "We will analyse his comments and the UEFA disciplinary inspectors will act if they have to act depending on what has been said."
Joaquin, who scored Malaga's first goal, suggested UEFA president Michel Platini may have had something to do with the defeat.
Platini is deeply unpopular in Malaga after UEFA slapped a ban on the club from competing in continental competition from next season due to delays in payments to creditors.
"We suspect Platini and all the rest of them who are involved," Joaquin told Spanish radio. "Because we are Malaga and not Real Madrid it's easier to do this to us."
Whatever conclusions are drawn about Tuesday's game, Malaga performed above expectations on their first foray into Europe's elite club competition but now need to put their disappointment behind them and focus on qualifying again for next season.
They have appealed against the UEFA ban at the Court of Arbitration for Sport. Malaga are sixth in La Liga, four points behind Real Sociedad who occupy Spain's fourth Champions League qualification berth.
"We all made a super-human effort and going out of the competition because of a mistake like that is painful," their Paraguayan forward Roque Santa Cruz told Spanish television.
"We were four minutes away from the semi-finals and it was snatched away from us. Now we have to lift our spirits and tackle the rest of the season in the best possible way."
- Sports & Recreation