This was a little before my time, but do keep in mind some context: the Vietnam and other riots, Malcolm X, Panthers, et. al., cities burning... No one knew what was coming next. It's a lot easier to look back and judge than it is to maneuver in the moment. Long afterward there is plenty more information, and time to pick through it.
If you didn't see other examples of your father as a racist, hang onto that.
Not excusing anything, it must have been terrifying for a kid...