Eli Gau was eating dinner at Applebee's with his partner, Lillian, and their sons Ryan, 3, and Ethan, 1, when his oldest boy wandered away from the table, he told local station KHOU.
Gau got up to retrieve Ryan when a manager approached him and said his children were making too much noise, KHOU reports.
The manager called authorities and asked the family to leave. On their way out, Gau was hit with a warning for criminal trespassing.
Gau said the experience left him "heartbroken."
An Applebee's spokesperson told KHOU that management had "mishandled the situation."
The restaurant chain also offered the family free dinner passes which Gau declined.