Settlers first came in 1798, making it one of the first municipalities in the Holland Purchase. For context, the Holland Purchase refers to December 1792, and again in February and July 1793, when Robert Morris sold more land to the Holland Land Company, an unincorporated group of companies created by Wilheim Willink and thirteen other Dutch bankers. Because it was illegal for them to own the property directly, the bankers chose American trustees in the United States to take title to the land.
Stafford was founded in 1820 by combining parts of Batavia and Le Roy. In 1842, a portion of Stafford was transferred to create the town of Pavilion. The Stafford Village Four Corners Historic District, located in the hamlet of Stafford, was added on the National Register of Historic Places in 1985.
The town has a total area of 31.2 square miles, of which 31.1 square miles is land and 0.1 square miles is water, according to the US Census Bureau.