Charity House, refugees