Why have we fallen out of love with organic food?

Lunch with Helen Browning, chief executive of the Soil Association, is in many ways a vindication of the organic movement she leads. We meet at Canteen, on Bristol's Gloucester Road, a very socially mixed neighbourhood where people didn't just sign petitions against the opening of a Tesco Express, some planned to firebomb it, a plot that led to riots when police raided the squat where they believed it was being hatched. It is a sign that organic ideals are not a middle-class preserve that the food here is not only delicious but seasonal, from small local farms with high animal welfare and environmental standards. What's more, Browning's pea risotto and my sustainable mackerel cost less than many takeaways. But one word is conspicuous by its absence on Canteen's daily changing chalkboard menu: organic.

It's a trend that can be seen at many farmers' markets, farm shops, restaurant and cafes that trade on their sustainable, ethical ethos. As many of the broad principles that organics advocates have become more mainstream, the prestige of actually being certified organic has diminished. Last year, the value of sales of organic foods in the UK went down by 3.7% and the number of producers and processors fell by 4% to 7,287.

Sales of organic products have been falling since the credit crunch first bit in late 2008. But thrift alone does not seem to be enough to explain what is now a medium-term trend, since Fairtrade, another ethical certification with a price premium, has not suffered the same reverse, with sales rising by an estimated 12% last year. What seems to be the case is that customers who used to use the organic label as a kind of proxy for good, sustainable produce now look to the specific virtues that most concern them: seasonality, locality, fair trade or animal welfare. Indeed, sometimes their other ethical concerns trump the desire for organics, such as when they choose home-grown peas over air-freighted organic alternatives, or value the Fairtrade label more than Soil Association accreditation. For many, organic is no longer the top priority for good, ethical food.

Indeed, even the Soil Association's own president, Monty Don, acknowledged as much when he took on the role in 2008: "I would much rather someone bought food that was local and sustainable but not organic than bought organic food that had to be shipped across the world."

So the question I wanted to chew over with Browning as we enter the associations's Organic September is whether "organic" as we have known it has had its day. For all the good the movement has done in challenging the most egregious practices of modern industrial farming, take any key issue on food and farming today and you will find that it's never simply a case of conventional bad, organics good, or even better.

Take, for example, the issue of animal welfare. One reason for buying organic is that the standards required for welfare are pretty high. That, at least, is the intention, and as Dr Becky Whay, senior lecturer in animal welfare and behaviour at the University of Bristol, says: "intention is worth quite a lot within the industry". But the research that has been done on whether organic standards are all really better for animals "is not very clear cut" and she tells me several times that "there are no easy answers".

Indeed, there may be some cases where organic animals are worse off. "I would never raise livestock organically," wrote Susan, a US sheep and goat producer and college-educated scientist on her baalands blog. She believes "organic standards do not allow you to treat a sick animal with anything that is scientifically proven to be effective. You can't use antibiotics, anthelmintics, anti-inflammatories, coccidiostats, steroids, hormones, feed additives, or many other conventional therapies."

It's actually more complicated than this, which is another reason why "organic" has become an inadequate guarantor of quality: people think "organic" has a clear, single meaning but standards vary between and sometimes within countries. There is a UK certification for Scottish Salmon, for example, but no EU standard for this or any other farmed fish. The EU sets the minimum standard for other organic produce, but it can be certified by one of 10 different bodies, all of which have their own set of rules. Most of the differences are quite small, but some are not: in America antibiotics are completely banned, in the UK their limited use is allowed. Because of the restrictions, however, Whay says that "there are certainly areas where we know that the organic sector is struggling to deal with, such as digital dermatitis, which is an infectious foot disease of cattle, where we know mass antibiotic treatment is very effective".

The rules do state that the animal's welfare must come first, even if that means it loses its organic status by being treated. But this creates potentially bad incentives. As Roger Longman of the non-organic White Lake Cheeses puts it: "A lot of farmers go: 'Well, she's not really, really sick, I won't treat her, I'll just hope she gets better,' and that's wrong, to my mind."

Soil Association Standards also state that farmers "must use complementary therapies and trace elements" with the proviso "provided that their healing effect works for the species and the condition you are treating". This is not very reassuring to those who believe in evidence-based medicine when read in the light of standard 10.10.21, which says you should "use effective homeopathy".

The Soil Association's recent strategy document, The Road to 2020, stated that organics must be "rooted in robust science and supported by an extensive evidence base". But unless you cherry-pick – which both defenders and detractors often do – the evidence appears to be very mixed. Washington State University professor John P Reganold is no apologist for big farming, having led the US's first and still only undergraduate major in organic agricultural systems. But writing in Nature earlier this year, he noted that the most recent meta-analysis showed "organic farming systems in developed countries produce yields that are 20% lower than their conventional counterparts". Even Browning doesn't deny that the evidence doesn't all stack up on the side of organics: "On biodiversity, organic clearly has a lot to offer. On climate change, on the greenhouse gas side of things, it does in some areas and not others."

But perhaps the biggest battle organics faces is that it has been robbed of one of its biggest selling points. Research by the Soil Association and Sustain suggests that the belief that organic food is "healthier for me and my family" is one of the main reasons why people buy organics, with 52% of customers citing it as a motivation – more than high animal welfare standards (34%) or it being more ethical (33%).

But bit by bit the case for the health benefits of organic food has been undermined. Just this week another review of the evidence has been published in the journal Annals of Internal Medicine, which concluded that "there isn't much difference between organic and conventional foods, if you are an adult and making a decision solely on your health". This came as no surprise to Professor Alan Dangour of the London School of Hygiene and Tropical Medicine, whose own 2010 review found no significant differences in the nutrient content or health benefits of organic food.

All this simply reinforces the most comprehensive study to date, an independent 2009 report commissioned by the Food Standards Agency that surveyed over 50 years of evidence and concluded that "organically and conventionally produced crops and livestock products are broadly comparable in their nutrient content." Nor was there any evidence of improved health from the more limited exposure to pesticide residues. Some farming practices can result in lower nutritional content and potential health risks, but it is never a simple case of all non-organic methods being worse than organic ones. For example, grass-fed cattle have been shown to produce more nutritionally rich milk, but these herds may be organic or conventional. This is probably why the only significant difference this week's study found was a higher level of omega-3 fatty acid in organic milk.

The case for the health benefits of organics is so flimsy that the Soil Association has effectively been banned from making any claims for them. "Anything we say, we ASA-proof it now, absolutely," says Browning, referring to the need to avoid sanction by the Advertising Standards Authority.

So where does that leave us? I put it to Browning that the organic movement will probably continue to have an important role to play in the debate about the future of farming but right now it would be a distraction to make organics the centre of our discussion.

"It would be a distraction if we spent all our time debating whether certified organic or non-organic is the way forward," she replies. But she maintains it is "absolutely critical" that the organic movement "both continues to make progress and continues to really inform the whole of agriculture. Organic principles need to be at the heart of where we farm everywhere."

But what are those principles? The Soil Association identifies four: health, ecology, fairness and care. The trouble is that many farmers and consumers would affirm these principles without also agreeing that you have to farm organically to follow them. As Dominic Coyte of Borough Cheese Company tells me, although organics is "a totally laudable reaction to the obsessive use of chemicals, it's not in itself a green light for quality and best practice". The organic movement owns a very particular set of rules, not the fundamental principles that define good farming.

Browning argues that the Association needs to "let go of some of this in order that the rest of the world can take some ownership of it". This is the direction the association started to indicate in The Road To 2020, which advocated "increasing the number of non-organic farmers we work with", maintaining that its "future will be built on partnerships with other organisations and individuals who share our vision for the future of food, farming and land use".

Browning seems to accept that this will mean placing less emphasis on the importance of being organic per se. "Whether it's 5%, 2% or 50% in terms of certified organic product is not the issue. The issue is that you've got a bunch of people striving to do more with less and that is what organic is really about."

The organic movement has arguably achieved a great deal. On the biggest issues, it has really won the debate. Most people now accept that you can't just spray and inject your way to sustainable food production, that there is much to be gained from more integrated farming practices that deploy resources from the earth rather than from chemical factories. But as conventional farming adapts, the best way ahead becomes much less clear. If we want food that is good for humans, animals and the environment, the priority now is not to praise organics or to bury it, but to accept we must look beyond it.