In Wales, there are around 16,000 adults with a severe learning disability, but only 4,000 of them are living in supported accommodation.
It is thought the remaining 12,000 – 75% of the total – still live at home.
“What you always get blown away by is when families say to you ‘I hope they die before I do’,” explained Wayne Crocker from Mencap.
“That’s an awful place to be as a parent.”
Mr Crocker said the housing situation for those with a severe learning disability was “complex”.
But ultimately, almost 20 years since the last institution closed in Wales, there was still too little housing stock – and too much expectation on families.
“Lots of people live with their parents until sadly they pass,” he said.
“For many it means there’s the shock and the sadness of losing their mum or dad, and then suddenly they’re having to find emergency accommodation too.”
The Welsh government said it is “committed to improving accessibility in social housing” and has invested in building new homes for this purpose.
It added it provided grants to local authorities and housing associations to make homes suitable.