Ken Fisher’s allegedly vulgar comments about women were enough to cause more than $3 billion in withdrawals from his firm, Fisher Investments, by pension plans — but retail investors were far less outraged, pulling out a mere fraction of that, according to a recent report.

Mercer, an advisor to institutional investors, found that individual investors withdrew $20 million in the week following Fisher’s comments at the Tiburon CEO Summit, CNBC.com writes.

That information came out on a conference call between Fisher executives and Mercer held Oct. 14, according to a note on the call obtained by CNBC.

Overall, private and retail clients were “largely unconcerned,” Jill Hitchcock, senior executive vice president of the private client group at Fisher, said in the Mercer note, according to the TV news channel’s website.

Only around 1% of such clients contacted raised concerns, according to Mercer, CNBC.com writes.

Mercer declined further comment to CNBC.com.

Fisher allegedly made references to convicted sex offender Jeffrey Epstein, talked about drug use and compared getting new clients to “trying to get into a girl’s pants,” according to a Twitter post by Alex Chalekian, founder of Pasadena, Calif.-based RIA Lake Avenue Financial.

Ken Fisher

Institutional investors, led by the Bureau of Investments of Michigan’s Department of Treasury and quickly followed by Philadelphia’s board of pensions, the Iowa Public Employees’ Retirement System, the City of Boston, New Hampshire, Fidelity Investments, the Employees Retirement System of Texas and Goldman Sachs, have disclosed divestments of close to $3.3 billion from Fisher Investments.

Nonetheless, that represents a small fraction in Fisher’s assets under management — the firm had $115 billion as of Oct. 31, John Dillard, senior vice president and director of global public relations at Fisher Investments, tells CNBC.com.

Meanwhile, roughly $60 billion of the $94 billion in total assets reported in the firm’s 2018 Form ADV were from retail investors, according to the website.