(CNN) -

On ABC's "This Week" on Sunday, U.S. Attorney General Eric Holder, who happens to be black, suggested that opposition to him and President Barack Obama in part is due to racial animus. For that, he has been vilified.

He has been accused of trying to scare people to the polls (as if listening to the evening news wouldn't already do that), stoking racial divisions and throwing down the race card. And he is asked rhetorically "is it possible to oppose Obama and not be a racist?"

The response to the attorney general's words has been very heated, and there is an obvious explanation.

"This Week" allowed many whites to hear what a black man is thinking about race. Most whites don't know because most whites don't have interracial conversations about race. To be fair, neither do most blacks. It's just that we know what other blacks are thinking because, well, we talk to other blacks about race.

"This Week" allowed a window into the thinking about race from a black person's perspective. That is unusual, given the dearth of guests of color on shows such as "This Week."

If anything, this points out the need for white America to hear more voices from people of color. But if you look at Holder's actual words, there really isn't anything surprising, or especially bold, in what he said.

He said: "There's a certain racial component to this for some people. I don't think this is the thing that is a main driver, but for some, there's a racial animus."

Let's review: He said "I don't think this is the thing that is a main driver," but it is as if when certain whites hear the word race (or a derivative) as the explanation for any part of an event, they go situationally deaf.

Poll after poll shows how blacks and whites view our country through different lenses. Consider a recent Gallup poll that asked whether the justice system was biased against blacks. While 69% of whites answered no, 68% of blacks answered yes.

How does this happen?

Because by and large when talking about race in America, whites have conversations with other whites and blacks have conversations with other blacks. What Holder said is very different from what whites who don't talk to blacks generally hear. As I tell my students when I teach about race and the law, I note that important breakthroughs on race can only come when you have a conversation with someone who doesn't look like you and doesn't think like you.

But this is not a level playing field. Blacks and whites can look at the same event and come up with completely different perceptions.

When a black person says this is about race, many whites say we're playing the race card. But when a white person says this has nothing to do with race, rarely does he or she ever get called on it. In this case, Holder, a black man, very gently said race could ... maybe... possibly have something to do with a piece of it, and look at what happened.

Let's take a few examples that to me show unprecedented hostility directed toward the President, in my opinion, because he is black.

First, the utter nonsense about the President's birth certificate; it refuses to die -- it is actively kept alive -- despite all the evidence to the contrary. Second, the number of pictures or posters at rallies with the President's face on a monkey's body. Third, Congressman Joe Wilson yells "you lie" during the President's State of the Union Address.

If anything, Holder was too tepid in his analysis, but I'm sure that's a function of his knowing that whenever he says anything about race, he gets pummeled.

The real lesson for me is the paucity of guests with the same views as Holder who are given a national platform the way that he was.

If week after week, on issue after issue, blacks were invited on as guests and allowed to inform the discussion, after a while, there would be fewer and fewer instances of situational deafness.

Whites would come to understand that there is more than one way of looking at something -- especially when it comes to race -- and when it comes to race, would not ignore the expertise that lies in black voices.

Read CNNOpinion's new Flipboard magazine.