Local resident Didier Bregardes was standing by the team buses at the start with an A4-sized sign which simply read 'Sky - Go Home'.
Luke Rowe was identified by onlookers inside the paddock as the Sky rider in question but denied any knowledge of the incident.
Bregardes said: "The rider came over quickly, grabbed my placard and threw it down."
Bregardes told reporters his banner was intended to protest against the way Team Sky handled the Salbutamol case involving Chris Froome, following the test he returned with high levels of the asthma drug during La Vuelta last year.
That investigation - now closed with Froome cleared of wrongdoing - took place in the public gaze following a leak, and Froome angered fans by choosing to continue to race as it rumbled on, winning the Giro d'Italia in May to hold all three Grand Tour crowns at once.
"When Froome was caught in the act at the Vuelta, the UCI and WADA didn't react," Bregardes said. "I've nothing against Froome or the riders, but it's the way the manager of Sky, (Sir Dave) Brailsford, dealt with it."
Rowe was asked if he took the sign but said: "No, I don't know what you mean. It wasn't me."
The Welshman said the reception the team had received at the race had been positive following the significant booing which greeted Froome at the team presentiation last week.
"I think we got quite a negative reception going to the stage for the team presentation but apart from that it's been awesome," he said.
Rowe confronted an anti-Team Sky fan (EPA)
"I've been impressed with the French crowds. They've been pretty complimentary and positive towards us, which is nice. Of course, for every thousand cheers you might get one negative person, but you just take it in your stride.
"We've got pretty broad shoulders - it's in one ear and out the other."