I spent a day on a secluded island that's a 5-minute ferry ride from NYC's ritziest vacation spot. It was immediately clear why the area is known as the 'un-Hamptons.'

Shelter Island is a secluded, 8,000-acre island on the eastern end of Long Island in New York.

It's only about a five-minute ride from the Hamptons, the ritziest vacation destination of New York City's wealthy elite.

The island is about a three-hour drive from New York City, depending on traffic.

It's only accessible by ferry.

Shelter Island sits right in between the North Fork and the town of Greenport to the north, and the South Fork, which includes most of the Hamptons, to the south.

The green section of Shelter Island that can be seen on the map is a 2,100-acre nature conservancy, the Mashomack Preserve.

On a recent spring day, I drove out to Shelter Island from New York City.

The drive to Greenport, where I boarded the ferry, took about three hours. Once I got to the ferry, it was only about a five-minute ride to the island. A round-trip ticket for a car and one driver costs $18.

On the side of the island closest to the North Ferry, the main street is aptly named North Ferry Road.

What seemed to be the busiest part of the road had a gas station, a liquor store, a hardware store, a café, and a bar.

The hardware store doubles as a toy store.

Along with hardware and toys, the shop sells fishing equipment and beach gear.

There's also Marie Eiffel, a neighborhood café recommended to me by a local.

I went inside for lunch and found that the café offers a variety of seafood, burgers, sandwiches, and pastries, including vegan options.

People were sitting outside enjoying the June sunshine and water views.

Everyone was very casually dressed and seemed in no hurry to get anywhere.

I started driving around the island with no particular destination.

It wasn't a problem to drive slowly and stop for photos, as there were hardly any other cars. The thought of traffic on the island seemed laughable.

Shelter Island is known for its bed and breakfast and boutique hotel culture. The Chequit, which made Fodor's Travel's list of 15 best hotels in the US, is a three-minute drive from the North Ferry station.

Built in 1872, the hotel was recently bought and renovated by Salt Hotels.

"It has the feeling of a bed-and-breakfast stay, but a dreamy one (perhaps in the movie version of your life) with cleaner, more contemporary style," reads the Fodor's review of The Chequit. "Likewise, its location on Shelter Island, one of the quietest, most 'untouched' places in the area, creates what a Hamptons getaway looks like in our imaginations (and more rarely reality): exclusive yet understated, rustic yet luxurious, and far from beach traffic or a crowded train car."

Rates start at Chequit range from $495 to $856 during the summer months.

Driving through Shelter Island felt like driving through a quiet, lazy suburb.

Compared to New York City, it felt like another world: calm and bucolic.

The homes on Shelter Island range in size and style. I didn't see many sprawling mansions like those that can be found in parts of the Hamptons.

All the homes I saw seemed to be very well-maintained.

And although it may not be the Hamptons, Shelter Island residents are still willing to pay for their idyllic seclusion.

The median price of homes currently listed in Shelter Island is about $1.5 million.

It's rare to find a home selling for less than $800,000 or $900,000, Humphrey told me.

Shelter Island has no regulations for the style of new homes being built, Humphrey told me, which has led to an eclectic mix of styles.

One home was being built in an ultra-modern style, a stark contrast from many homes on the island.

Shelter Island has its own schools, police department, and fire department.

There's no hospital, however, which could be problematic for elderly residents.

Shelter Island has no shortage of access to water.

The island has four marinas, five public beaches, and several ponds and creeks.

On the northwest edge of the island is the Sunset Beach Hotel, which has 20 ocean-view rooms.

In the summertime, rooms at the beachfront hotel start at about $450 per night.

The hotel bar and restaurant serves "healthy and delicious summertime food of the French Riviera and coastal Italy," according to its website.

Unfortunately, it wasn't open during my visit, but I was told the area can get quite lively in the summertime.

Just down the waterfront from the Sunset Beach Hotel is the Pridwin, an all-wood resort hotel that's been open since 1927.

A standard room starts at about $244 per night, with a two-night minimum.

Like many of the hotels on Shelter Island, the Pridwin is only open from May through Labor Day.

When late afternoon rolled around, it was time to head back on the ferry for the drive to New York City, but I found myself reluctant to leave the island.

While its Hamptons neighbors to the south have become overcrowded, pretentious, and wildly expensive, Shelter Island is quiet, laid-back, and it isn't trying too hard. While it may not be cheap, it's still a bargain compared to the Hamptons, where median home prices reach nearly $11 million in some parts.

The "un-Hamptons" describes Shelter Island perfectly, and that's exactly what makes it so appealing.