The island took on Fisher's namesake, but he didn't hold on to it for long. In the mid-1920s, Fisher met William K. Vanderbilt II, one of America's wealthiest residents, and proposed a trade: seven acres of Fisher Island for Vanderbilt's 250-foot yacht.
The railroad baron obliged and drew up plans for Fisher Island's first residence, "Alva Base," a Mediterranean-style compound with guest houses, tennis courts, and pools.
Vanderbilt completed construction on his winter retreat in 1941 and spent a few years entertaining and hosting wealthy friends until his death in 1944. The property was sold and resold over the next few years.
By the mid-1970s, total ownership of Fisher Island belonged to an investment group helmed by Charles Rebozo, a Florida banker and businessman who rose to prominence as the close friend and confidant of former US President Richard Nixon.
Rebozo wanted the island filled with luxury real estate and began selling off plots of land to investors and developers. In 1987, Fisher Island Club seized ownership and slowly turned it into the exclusive retreat it's recognized as today.
In 2014, Fisher Island Club completed a $60 million revitalization project that included the addition of a new golf course, tennis courts, a spa and salon, restaurants, a beach club and marina, and a hotel.
"Rosemary's Cottage" was originally built as an art studio for Vanderbilt's stepdaughter. Now part of the island's hotel, it rents for $4,000 a night.
Though Fisher Island is billed as a world-class travel destination, about 650 families live in private residences on the island. More than 80% of them are seasonal residents. The current median listing price of homes for sale is $3.2 million, according to Zillow.
To use the island's resort-style amenities, residents must pay membership dues, which includes an initial contribution of $250,000 and an annual fee of $20,330. Frequent visitors can become members for a one-time initiation fee of $42,800 and an annual payment of $27,306.
Access to the golf course costs an extra $4,226 to $4,975 a year for both guests and property owners.
Fisher Island maintains more than 100 boat slips for yacht-owning residents. Yacht and 24/7 public ferry are the only way to access the island.