Ben Carson was removed from the Southern Poverty Law Center's "Extremist Watch List" on Wednesday after Fox News host Bill O'Reilly and conservative news websites raised public criticism.

The "Extremist Files" is mostly made up of hate groups and white supremacists. But in October, the civil rights advocacy group decided to include the potential Republican presidential candidate for allegedly being "anti-gay," Breitbart reported.

Despite the list being up for months, Carson's inclusion was first spotted by the blog Legal Insurrection on Friday, which then began to be noticed and reported by several other conservative news sites.

"In October 2014, we posted an 'Extremist File' of Dr. Ben Carson," SPLC wrote on its website. "This week, as we've come under intense criticism for doing so, we've reviewed our profile and have concluded that it did not meet our standards, so we have taken it down and apologize to Dr. Carson for having posted it."

Initially, the SPLC justified including Carson by listing his appearance in a March 26, 2013 interview on Fox News' "Hannity," Fox News reported

"Marriage is between a man and a woman. It's a well-established pillar of society and no group, be they gays, be they NAMBLA [North American Man/Boy Love Association], be they people who believe in bestiality -- it doesn't matter what they are, they don't get to change the definition," Carson said in the interview.

On other occasions, the pediatric neurosurgeon made similar remarks to "The Baltimore Sun" and in his book, "America the Beautiful," where he linked redefining marriage based on societal pressures to the "dramatic fall of the Roman Empire."

So, the 63-year-old was designated as an anti-gay "extremist" because of "his very extreme rhetoric about the evils of gay people," Mark Potok, a senior fellow at SPLC, told TPM.

However, even though SPLC have issued an apology after removing Carson from the list, the group maintains that he has "made a number of statements that express views that we believe most people would conclude are extreme."

"We laud Dr. Carson for his many contributions to medicine and his philanthropic work, and we, like so many others, are inspired by his personal story," the statement read. "Nevertheless, particularly because Dr. Carson is such a prominent person, we believe that his views should be closely examined."

Carson "has, in fact, made a number of statements that express views that we believe most people would conclude are extreme," including arguing that Obamacare was worse than the 9/11 terror attacks and implying that gun registration was merely a tactic to make it easier for the federal government to impose martial law.

Meanwhile the 63-year-old appeared on "The O'Reilly Factor" on Wednesday night and thanked "all the people who have spoken up on my behalf."

"It's a freedom of speech deal and they can't be doing that," O'Reilly said.