Big Sur is quieter than I thought it would be – for most of the year a collapsed bridge has kept the tourists at bay. Eventually normal service will resume, but for now, it feels a little eerie.

I knew 24 hours wouldn't be enough time, but I came anyway, because what if I never got another chance? And there was much pleasure in the planning: re-reading Miller, and Jack Kerouac, drifting in the virtual sea of "Your Big Sur Weekend" listicles.

In that dreamspace where I had unlimited budget, Big Sur was like this: a serene drive along the unspoilt coast, a pause for a selfie by the Bixby Bridge. Maybe I'd stay at the New Camaldoli Hermitage, where you get a great vegan spread but must refrain from talking; or at the $9000-a-night Wild Bird, the not-so-humble A-frame described in Time Magazine as "the most beautiful house on the most beautiful site in the US".

My ideal activities were similarly lofty: dinner and dancing at Nepenthe (colourful caftans optional); and a gentle hike to Esalen's thermal hot springs (which would be miraculously free of naked strangers).

With travel there is always a gap between the dream and the reality. For some the trip never gets better than the moment before they board the plane. Months and air-miles later I think about how my Big Sur experience was just a taste. And how maybe with a place like that, you don't go for a taste – you stay for the whole meal.

There are no trains to Big Sur and only local buses. Drive there in the daytime because it is dazzling and dizzying. The sea stretches forever, wildflowers crowd the mountainsides. After the gritty post-city grunge the air tastes supernaturally sweet. Seabirds wheel in balletic arcs. You might start to feel unshackled, you might start to feel like leaving your real life for good, running into that wilderness, growing a ladybeard and drinking from creeks.

Aside from the more eccentric accommodation options, there's a handful of hotels ranging from modernist posh to cabin rustique. You can pitch a tent in the State Forest or at the highly instagrammable Treebones Resort, where you'll find yurts, a hot-tub and a "human nest".

You must go to Nepenthe. The restaurant (and gift shop and cultural hub) was built around a 1920s cabin that was once Orson Welles' and Rita Hayworth's love-nest. Nepenthe means "to forget sorrow", which is easy to do when sitting on the terrace looking out to the edge of the world. The day I was there I had an Irish Coffee, stared into the mist, saw a condor, admired the fireplace, and fondly remembered Elizabeth Taylor's pom pom earrings.

The Henry Miller Memorial library ("where nothing happens", according to the website) used to be the residence of Emil White. White was Henry Miller's "Man Friday" and an artist in his own right. He founded the library after Miller's death, because he missed his friend. Lawrence Ferlinghetti (no Big Sur slouch) wrote "that cult of sex and anarchy" (as Harper's describes it) really only consisted of Henry Miller and Emil…" but I didn't see any debauchery at the library, just books (for sale), and cats and coffee. It should be noted that the library's idea of "nothing" includes film festivals, live music, swing-dancing and stargazing.

While the rain prevented more adventurous hiking, I did get to see Pfeiffer Beach with its famous keyhole arch and purple sands. I went just before sunset, the beach was empty. I filled my pockets with shells even though I knew I'd never get them through customs.

In The Oranges of the Millennium, Miller wrote: "One's destination is never a place but rather a new way of looking at things." I thought about how I didn't want to just pass through places; I wanted to get stranded. I wanted to get lost. How could I have gotten myself all the way here just to have to leave the next day?

At the end of The Sandpiper, Laura and the Reverend say their goodbyes; he's leaving, she's staying, her son's going back to school. For the longest time, I thought when I came to Big Sur I would find her shack, but it turns out it was built on site. The shack – and its glorious Vincente Minnelli aesthetic (driftwood assemblage, California Ikebana) – was dismantled after the shoot and now sits abandoned in a forest near Inverness. If teenage me is bereft to learn it wasn't really real, at least Big Sur is, and it's really something.