The "Duck Dynasty" clan could very well be done with reality television.

According to E! News, sources from the A&E network and the Robertson clan state the family has become "very serious" about leaving their show behind.

The news comes just two days after the network put patriarch Phil Robertson on an "indefinite" hiatus from filming. The 67-year-old's suspension comes after comments he made during his interview with GQ magazine, in which he placed homosexuality in the same category as bestiality as deviant behavior.

"They're an extremely tight-knit family and they're not going to let this get in the way," a source connected to the family, told E! News. "[Phil] is the reason for their success, they're not going to abandon him. They're also not about to let anyone threaten their religious beliefs."

In addition to his comments about homosexuality, the reality star spoke about race and living in Louisiana before the civil rights era, in which he commented on never seeing "the mistreatment of any black person," who worked on a farm.

According to an insider, the family and people who work on the show, "feel like it's a big pissing match and there is no way that anyone can win." 

"Phil might be diminished but there's no way to cut him out altogether," a source said.

E! News reported the network owns the "Duck Dynasty" series and the property behind it, for about least one more season

"The family could do appear on another network once their exclusivity is up with A&E," the source said. "Under a normal contract, that usually means anywhere from six months to a year after the final episode has aired. However, if the family breaks their agreement with the network, the network could hold them for longer."

"Duck Dynasty" follows a Louisiana family that has made a fortune in the duck industry. The fourth season premiered in August, with nearly 12 million viewers, making it the highest-rated show of all time and the No. 1 non-fiction series telecast in cable history.

A source who works on the series said the Robertson controversy "can't end well," stating that the most likely scenario is that the show will be cancelled.

"If the network backs down and they bring Phil back, they look weak," one insider said. "If they stand their ground, the family probably won't move forward and A&E loses their highest-rated show. No one can really imagine the show going forward without Phil. It would be too weird."