Today only remnants of the town fortifications with a bastion have been preserved. The town walls were built after the year 1389 on an order by Olomouc Bishop Mikuláš.
The walls had three gates – the Upper Gate at the level of Hrnčířská Street, the Lower Gate near the museum, and the Central Gate behind the Old Town Hall and at the site of today’s main entrance to the town square arcade. During the economic crisis in the middle of the 19th century the unemployed residents of Svitavy filled in the town’s former defensive moats and gradually tore down the town walls. A tree-lined alley was planted in place of the walls. The former 10 metre high and up to 2 metres thick walls had been taken apart since the Thirty Years’ War and used as a source of inexpensive building stone. In 1849 the glory of the town walls, which no longer had even one gate, was definitively extinguished. The last mention of additional work tearing down the town walls dates to the 1960s. Parts of the walls by the town bastion were also razed.