The survey, commissioned by the firm's consultancy arm, demonstrated how food safety faces ever-growing scrutiny in the US, with highly-publicized food recalls and safety scares from China preying on consumers' minds.
Concern about safety was found to be particularly high in regards to meat products as well as food imported from outside of the States.
"These findings underscore how urgent it is for food manufacturers to do all they can to address the problem of food recalls head-on," said Pat Conroy, Deloitte LLP's vice chairman.
"Over half of consumers say they may drop your product if they believe you are not doing what it takes to protect them and their families."
The survey is particularly timely, given last week's multistate outbreak of the foodborne disease salmonella from red raw tomatoes, which incited the US Food and Drug Administration (FDA) to issue a warning to consumers.
Other recent food scares and recalls include spinach grown in California contaminated with E coli in 2006, salmonella-tainted peanut butter caused many people to be hospitalised in February 2007 and in March 2007 contaminated pet food lead to the deaths of a number of animals, probably from contaminants in vegetable proteins imported into the United States from China.
Also last week, Michael Doyle, a US microbiologist, said the trend towards importing more food from developing countries is opening up Western countries to greater food safety risks because of variations in sanitary standards.
He said the responsibility for ensuring that food imported into the country where it is to be consumed should lie with food industry, and not just with regulatory bodies.
But despite this, he stressed many food companies are already conducting checks on their overseas customers, so as to ensure they are safe. Thus, consumers should not automatically assume that just because a food comes from a particular country, it is unsafe.
The survey polled a nationally representative sample of 1,110 consumers.
It found that 57 percent of respondents stopped consuming a certain product following a food scare, and nearly three-quarters believe the number of food-related recalls has increase in the past year.
Even more (76 percent) said they are more concerned about the foods they eat than they were five years ago.
The product that most worried consumers is beef, with 78 percent of respondents expressing concern about the meat. Chicken recalls have also prompted concern, with 67 percent saying they felt unease about it.
The survey found that 53 percent were cautious about fruit and vegetables because of recalls, and the same number remained apprehensive about dairy products.
Continuing food recalls and decreasing consumer confidence has prompted a proposed revamp of the US food safety system.
In November last year, the US government announced wide sweeping plans to improve the safety of the nation's food supply, with measures including more stringent inspections, stronger penalties and mandatory recalls.
These include the FDA's Food Protection Plan, which is built around prevention, intervention and response.
Respondents were found to have reservations about foods produced outside the US, with 56 percent considering imported foods as "not at all" or "somewhat" safe. In contrast, 80 percent thought domestically produced foods are safe.