"I don't think right now is the time or the place. So I just took a pass," he told reporters when asked if he'd spoken to Sterling on Monday.
"I haven't given him the opportunity to explain himself and quite honestly right now I don't want him to," he added.
Rivers forced a move to the Clippers last summer after expressing his desire to leave the rebuilding Boston Celtics. Now he is in the unprecedented position of having to motivate a team of largely black players to win a championship for an owner who allegedly made racist and anti-black statements.
His team came out flat in Game 4 - which tipped off less than 48 hours after the Sterling tape went public - letting up 42 points in the first quarter.
At his press conference on Sunday, Rivers said he didn't know if he'd be back with the team next year.