Chris Ratcliffe/Getty Images
His comments come on the day that former mayor Ken Livingstone was suspended for claiming Hitler supported Zionism.
Election front-runner Khan was speaking to Sky News on Thursday after he and dozens more Labour MPs, including Yvette Cooper and Liz Kendall, expressed their outrage at Livingstone's remark before he was officially suspended.
Speaking to Sky News, Khan said: "There's been too many incidents where the Labour Party leadership appears to be taking no action against comments which are clearly anti-Semitic. The Labour Party leadership needs to take action."
He added: "What I think is really important is whether you are part of the leadership, or part of the national executive committee, you understand how serious these comments are. There should be no hierarchy when it comes to racism. There should be zero tolerance of these views in the Labour Party."
Livingstone - who was an advisor to Labour leader Jeremy Corbyn and co-covened the party's International Policy Commission - sparked outrage when he told BBC Radio London that he had "never heard anyone (in the party) say anything anti-Semitic" and went on to say that Hitler supported Zionism.
His comments came in the same week that Bradford West MP Naz Shah was also suspended for comments she made on Facebook about the state of Israel. Shah was accused of sharing a post in 2014 which read: "Solution for Israel-Palestine Conflict - Relocate Israel into United States", with the comment: "Problem solved."
Khan's closest rival Conservative candidate Zac Goldsmith told Business Insider on Thursday that Livingstone's remarks were a symptom of the anti-Semitism which he believes runs "right the way through the Labour Party" and said it's an issue the party has "got to deal with."
Some people on the right argue that Labour and the broader left has a problem with anti-Semitism, pointing to Livingstone and Shah's comments as the latest in a series of arguably racist remarks from those on the left against Jewish people.