The Sylvester Manor house is in fact a collection of several building and remodeling efforts spanning several hundred years, responding to the changing tastes and needs of its many descendant residents. The original c. 1652 house, to be built of “six or seven convenient rooms” served the corporate needs of the four sugar partners and provided a home for Nathaniel and Grizzell Sylvester, their eleven children and most likely several slaves and indentured servants. Aside from a brief reference to a second floor porch chamber mentioned in Grizzell’s 1685 will, there are no known descriptions of the 17th century plantation-era residence.
As far as we know, the original house remained intact until c. 1737, when Nathaniel’s grandson, Brinley Sylvester, who waged a long legal battle to claim his inheritance, built a new residence close to the original home site. Though the new construction repurposed several beams, doors and other architectural elements from the original dwelling, Brinley’s was a new, fashionable Georgian residence that cast off the family’s hybrid Dutch, new-arrival roots in favor of a style that firmly reflected the sophistication and prosperity of the English leisure class in North America.
Visitors to the Manor will still see in the south-facing front elevation of the current house the symmetrically placed “six-over-six” windows, central doorway, hipped roof and dormers and, inside, the interior features of the Georgian-period residence. Brinley repeated the rigid geometrical design of his house in the 2-acre adjoining garden, where the orchard, axial boxwood paths and formal rectangular garden beds of herbs, vegetables and flowers were laid out in a combination of kitchen and commercial use and aesthetic delight.
The house remained relatively unchanged until the mid-1830s, when proprietor Samuel Smith Gardiner and his wife, Sylvester descendant Mary Catherine L’Hommedieu Gardiner, made several modifications. Gardiner, a lawyer, adapted the house to serve the needs of his busy law practice: a first floor bedroom was remade into a study, with the addition of an eastern exterior side door and inner vestibule to accommodate waiting clients. The kitchen, currently the dining room, was expanded in an “El” to the north, and a cream-colored paint was applied over the Prussian blue and white lead pigment first applied by Brinley Sylvester in c. 1737. To this day, these are the only two coats of paint to cover the Manor’s wainscot paneling in the parlor and the paneled upstairs bedroom.
Two Gardiner daughters, first Mary, then Phoebe, married Harvard professor Eben Norton Horsford and the house, called Abbey Manor by the family, became a summer residence. Life in Cambridge placed the family in the inner circle of the American cultural and intellectual elite, many of whom were frequent Manor guests. During the late-19th century, Shelter Island became a summer salon of sorts as H.W. Longfellow, Asa Gray, Sara Orne Jewitt, John Whittier Greenleaf and other friends spent summer vacations with the family.
Cornelia Horsford, daughter of Eben and Phoebe, had been mistress of the Manor’s gardens for many years before inheriting the property in 1903. Like Brinley Sylvester before her, Cornelia wanted a home that confirmed her place in society and her times; the Colonial Revival movement, a celebration of traditional American roots and values, suited the pedigree-conscious Cornelia perfectly. In 1908, with resources left from her father’s successful food patents, she hired Henry Bacon, architect of the Lincoln Memorial, in Washington, to design a large Colonial Revival addition to the existing house and adapt its older interiors. Bacon widened the central hall and main staircase, installed a new kitchen in the basement, added symmetrical covered side-porches to the east and west, and more than doubled the size of the house with a significant addition to the back. In a nod to the Sylvester’s Dutch origins, Bacon included picturesque blue and brown Delft tiles in his fireplace redesign scheme.
The last resident “Lord of the Manor” was Andrew Fiske, Cornelia’s nephew. Inheriting the Manor from his aunt in 1944, He began a program of modernization that brought the house fully into the twentieth century. Cornelia’s first floor butler’s pantry was replaced with a modern kitchen, with its basement predecessor reinvented as a flower room for his wife, Alice Hench Fiske. The antique coal-burning furnace was replaced by a contemporary gas burner, though the last Sylvester Manor coal delivery, made over a half-century ago, still rests where it was delivered in the southwest corner of the basement. And in the early 1950s, the Fiskes installed a swimming pool near the garden, where it is flanked by the same boxwoods planted by Brinley centuries ago.