He and his wife Nabila, 25, moved into Clifton Hall - which dates back to the Norman conquest - with their daughters, aged seven, five and three, and 18-month-old son.
Mr Rashid said: "I fell for its beauty but behind the facade it is haunted...
"The ghosts didn't want us to be there and we could not fight them because we couldn't see them."
He said the paranormal experiences began on the day they moved in, ranging from tapping on the wall and unexplained voices to ghostly presences taking the forms of their children.
Mr Rashid added: "When we found red blood spots on the baby's quilt, that was the day my wife said she'd had enough. We didn't even stay that night."
Even paranormal experts were unable to solve the problem.
The family left in August 2007 and Mr Rashid, who now lives in Wollaton, stopped paying the mortgage in January. The Yorkshire Bank finally reclaimed the hall on Thursday.
He said: "When people used to tell me about ghosts, I would never believe them and would say 'whatever'.
"But I would have to tell any new owner that it was haunted having experienced it."
An interesting story but we must remember that we have tough times economically at the moment and it wouldn't be the first time a ghost had been used as a convenient excuse to cut loses.