In 1364, the first occupant of the house took up residence. That was Floirken Roeffart, a weapon boy. Above the entrance gate hangs an alliance coat of arms, which dates from 1626. This alliance coat of arms stands for the Lords of Kessel. All castles in Baarlo used to be owned by the Lords of Kessel.
After 1700, various owners lived in Huize Roffaert. The east and west wing were rebuilt in 1937 after being destroyed. In 1996, the building was restored for the last time.