The team from the Cleveland Clinic Foundation in Ohio tested the sperm of 364 men who were being treated at fertility clinics in Mumbai, India, with their partners.
It was found that the heaviest users, those who used their phones for more than four hours a day had the lowest average sperm counts, at 50 million per millilitre (ml) and the least healthy sperm.
Men who used their phones for between two and four hours a day averaged sperm counts of 69 million per ml and had moderately healthy sperm.
Those who said they did not use mobile phones at all had the highest average sperm counts, of 86 million per ml, and their sperm was of the highest quality seen.
'Used without thinking twice'
Dr Ashok Agarwal, who led the research, told the New Orleans conference the study did not prove mobiles damaged fertility, but said it showed more research was warranted.
"There was a significant decrease in the most important measures of sperm health and that should definitely be reflected in a decrease in fertility, which is seen worldwide.
"People use mobile phones without thinking twice what the consequences might be.
"It is just like using a toothbrush, but mobiles could be having a devastating effect on fertility.
"It still has to be proved, but it could be having a huge impact because mobiles are so much part of lives."
He suggested radiation from mobile phones might harm sperm by damaging DNA, affecting the cells in the testes which produce testosterone or the tubes where sperm is produced.
But a British expert cast doubt on the suggested link between mobile phone use and infertility in the men studied.
Dr Allan Pacey, senior lecturer in andrology at the University of Sheffield, said: "This is a good study, but I don't think it tackles the issue.
"If you're using your phone for four hours a day, presumably it is out of your pocket for longer.
"That raises a big question: how is it that testicular damage is supposed to occur?"
Dr Pacey, who is honorary secretary of the British Fertility Society, added: "If you are holding it up to your head to speak a lot, it makes no sense that it is having a direct effect on your testes."
He added that people who use phones for longer might be more sedentary, more stressed or eat more junk food, which might be more likely explanations for the link found in the study.