The Quinebaug Valley Council for the Arts and Humanities or more commonly known as the Arts Center was constructed by Ebenezer Ammidown sometime between the years 1832-1836; its magnificence resulted in that it was the largest single residence in the town at that time which gave it the nickname "Ammidown Castle." Ebenezer D. Ammidown was born in 1796 and lived here until his death in 1865, when the property passed among his heirs for the next 30 years. The Ammidown family of Southbridge was large and quite successful in agriculture and commercial enterprises.

Ammidown's nephew, Chester A. Dresser, lived with him for a while, and, perhaps for that reason, the house was eventually purchased by Dresser's widowed daughter-in-law. The home remained in the Dresser Family, and is thus generally known as the "Dresser House" or "The Pines" after the hemlock trees that were planted around the property. The House remained in the Dresser family until 1977 when it was sold to Mrs. Ruth D. Wells, wife of Mr George B. Wells who then donated the property to the Quinebaug Valley Council for the Arts and Humanities.

The building has been considerably modernized since the 1830s. In particular, the penthouse-like structure on the roof is from the 1870s. Notable features include the great arched demi-lune in the gable end; characteristic early 19th-century webbing between the chimneys; and inside brick panels at the ends. This was a building of grand scope, certainly the grandest house of its time in Southbridge.