The name of the castle comes from a legend. It is said that, about two centuries ago, a skilled doctor from Palermo, a certain Gaetano Palmieri, saved the son of the Prince of Palagonia, Gravina-Crujllas, from a very serious illness, and that the latter, grateful, gave him a plot of his fief located near to the Fiumefreddo river.
Palmieri wanted to build a fortified villa to live there for long periods of the year also because that place was much appreciated by his beautiful wife Rosalia, who was flirting, however, with a certain Nello Corvaja from Taormina.
One day some Turkish pirates landed, who ransacked and, having arrived at the castle, kidnapped the two owners; but, as they were about to get to the beach to escape, they were joined by some armed young men, led by Corvaja, who had seen the sad prisons landing from the top of Taormina. The pirates were thus killed or were put to flight and the Palmieri freed.
To thank the Eternal Father, a small church was erected next to the castle, dedicated to Our Lady of the Sacred Letter, and the loggia was built in which the statues of two Turks were placed, who seem to look anxiously towards the sea, as if waiting to be freed from their mates. And it is thanks to these two statues of Moors, also called 'slaves' in Sicilian, that legend has it that the Castle was called "degli Schiavi".