The chief inspector of prisons said the unit at Long Lartin, in Worcestershire, needed a better balance between security and humane care.
The unit holds seven men suspected but not convicted of terrorism, while they await deportation or extradition.
Officials said they were working to maintain the right balance in the unit.
The Long Lartin detainee unit is separate from the main prison and the men, all accused of links to Islamist extremism, are held in maximum-security conditions.
Two foreign men have been held for more than 11 years, while one British citizen, Babar Ahmad, has been held for seven years, while he contests extradition to the United States.
A senior judge recently said he hoped Mr Ahmad’s “ordeal” would come to an end, one way or another.
Nick Hardwick, the chief inspector of prisons, said: “We have previously raised concerns about holding a small number of detainees, who already inhabit a kind of legal limbo, in a severely restricted environment for a potentially indefinite period.
“We were therefore concerned to find that the detainees were no longer able to mix with the wider prison population. These restrictions had apparently been made on security grounds, although the rationale appeared obscure as sentenced terrorists faced no such restriction in the main prison and not all detainees posed the same level of risk.”
The restrictions on the men now include a ban on them celebrating the Islamic festival of Eid alongside other Muslim prisoners.
Mr Hardwick said that given the “isolated nature” of the secure unit, the detainees did not have enough time out of their cells.
“The risks to the mental and physical health of detainees of such lengthy, ill-defined and isolated confinement are significant,” he said.
“It was therefore appropriate that health services had improved, although there were still gaps in mental health provision.
“The Long Lartin detainee unit holds individuals considered a serious threat to national security and it is inevitable that they will face rigorous controls and restrictions.
“Nonetheless, while detainees’ treatment and conditions were satisfactory in some respects, too little attention was paid to their uniquely isolated and uncertain position.”
Michael Spurr, head of the National Offender Management Service (NOMS), said: “The detainee unit at Long Lartin is challenging for staff because of the small population, the lengthy stays the majority are subject to and inevitable regime restrictions.
“It is therefore pleasing that the chief inspector has recognised the positive approach of staff and acknowledges that detainees feel safe in the unit.
“There is an inevitable balance to be struck between legitimate security concerns and integration with the general population at HMP Long Lartin. The governor will continue to monitor the situation to ensure that the right balance is maintained.”