Warmer weekend weather has stoked fears that Britons could congregate in parks and open spaces, potentially increasing transmission, as the country goes into a third week of lockdown.
A major south London park was closed on Sunday after concern about high numbers of sunbathers on Saturday.
Hancock, who himself tested positive for COVID-19, told Sky News television sunbathing was "against the rules" and anyone doing so put lives at risk.
He later told the BBC most people were following guidance only to leave their homes to shop for essential supplies and medicine, and to exercise once a day.But he said it was "quite unbelievable" that a minority were not.
"The truth is the more people go out from home, the more the virus spreads," he added.
"I don't want to have to take away exercise as a reason to leave home because people are not following the rules...
"If the result of that is that too many people go out and flout the rules I'm afraid we'll have to take action." Meanwhile, police in Dorset, southwest England, said they charged a man with a public order offence for allegedly wiping his saliva on items in a supermarket.
Hancock's warning came after another record day-on-day rise in reported deaths from COVID-19 that took the overall toll in Britain to 4,313, and confirmed hospital cases to 41,903.
The death toll includes a five-year-old child, as well as seven healthcare workers.
Prime Minister Boris Johnson remains in self-isolation with a high temperature after testing positive for the disease, which is expected to peak in Britain in the next week to 10 days.
His pregnant partner, Carrie Symonds, said she has also shown symptoms but not been tested.
Queen Elizabeth II will make a rare special address at 1900 GMT on Sunday in which she is expected to praise people's response to the outbreak.
Her eldest son and heir, Prince Charles is recovering from mild symptoms of COVID-19. (AFP) NSA