Transportation Secretary Pete Buttigieg’s husband, Chasten, said on ABC’s “The View” Thursday that he had not heard from former Vice President Mike Pence since the latter joked at the Gridiron Dinner that Pete Buttigieg took “maternity leave” after the couple adopted newborn twins.

Pence had said Saturday, “Pete is the only person in human history to have a child and everyone else gets postpartum depression.” Chasten said he spoke up because he felt it was his responsibility to hold people accountable for their words, especially when those words are homophobic or misogynistic.

Pence, speaking at a Republican Party dinner in New Hampshire on Thursday night, defended his comments, saying the Gridiron Dinner is a roast, during which he made jokes directed to and from people in both parties.

“The only thing I can figure is Pete Buttigieg not only can’t do his job, but he can’t take a joke,” he said.

Recently appearing on “The View”, Chasten Buttigieg said he did not expect to hear from Pence, but will continue to hold him accountable for his words. He added that such remarks are part of a much bigger trend of attacking families.