Capt. Joseph Vesey married his fourth wife, Maria Blair, a wealthy Charleston widow. He moved back into town and purchased a house at 41 Anson Street, which was destroyed by the 1838 fire. His man servant, Denmark, lived in the house with the couple.
Currently the location is a private home, constructed in 1840.