Earlier this year, before we went into lockdown, my wife and I set off on our final, farewell trip to Rye. I may go again, one day, but I know she never will. This quaint, archaic seaside town where we’d spent so many happy holidays had become a painful place for her. She was glad to say goodbye. I wanted to make a weekend of it, like we always used to, but she didn’t want to stick around. Her dad had died and her mom was in a nursing home. We’d come to clear out their house before the new owners moved in.

It was her parents who had introduced me to Rye, 24 years ago. They’d just retired and needed a new adventure. The National Trust needed some new tenants for Lamb House, a grand old house in Rye where Henry James used to live. My wife’s parents got the job. They had to look after the house and welcome visitors. In return, they got to live there. After six years they’d had enough, but by then they’d fallen in love with Rye and so had I. They bought a more modest house on the edge of town and lived there for 18 years. For our two children, it was home away from home, a place of perfect summers.

This story would probably be of scant interest to anyone outside my family, were it not for the fact that loads of other people seem to feel this way about Rye. What makes it so special? Why do so many people dream of living there? Because, more than anywhere else I know, it encapsulates a quintessential sort of Englishness: creative yet conservative, cultural yet parochial — a town where conformists and eccentrics feel equally at home.

It was this quintessential Englishness which lured Henry James here at the end of the 19th century. ‘James was an American boy who had reacted against America and steeped his sensitive psychology in everything that seemed most antiquatedly and aristocratically English,’ observed G.K. Chesterton after a visit to Lamb House. ‘In his search for the finest shades among the shadows of the past, one might have guessed that he would pick out that town of all towns, and that house from all houses.’ Chesterton guessed right.

Henry James entertained lots of other writers at Lamb House. Rudyard Kipling drove here in his smart new motor car. H.G. Wells, Joseph Conrad and Ford Madox Ford all lived nearby. If you want to know more, check out Miranda Seymour’s A Ring of Conspirators, a fascinating group biography about James and the writers of the Rye district. Subsequent tenants of Lamb House included E.F. Benson, who set his Mapp & Lucia stories here, and the novelist Rumer Godden, author of Black Narcissus. Yet James’s peevish ghost still haunts this handsome townhouse, a century since he vacated it. For such a small man, he takes up a lot of space.

In 1913, to celebrate his 70th birthday, James’s friends and fans clubbed together and commissioned a portrait of him by John Singer Sargent, which was to hang in Lamb House. But by then James had begun to tire of Rye. He left the town in 1914: ‘I can no longer stand the solitude and confinement, the immobilization, of that contracted corner.’ Rye was ever thus: it was James who had changed. He died in London in 1916. His ashes were buried, at his request, in Cambridge, Massachusetts.

I’d love to say my visits to Lamb House gave me a deep appreciation of the books James wrote here, but I’d be lying. Despite several valiant attempts, I’ve always found his wordy novels virtually unreadable. Oh well, who cares? I’ve got my own memories of his former home — above all the wonderful New Year’s Eve party my in-laws threw here in 1999, which straddled several generations. The eminent literary agent Graham Watson (who also used to live here) was the oldest guest, aged 86. The youngest was my son, aged six weeks.

Even if you’re not interested in Henry James, Lamb House is well worth a visit. The building is beautiful and was a local landmark long before James moved in. It was built in 1722 by a wealthy wine merchant called James Lamb. In 1726 King George I spent the night here, after his ship had run aground on Camber Sands. That night Mrs Lamb gave birth to a baby boy and named him George. The garden, delightful at any time of year, was designed by Alfred Parsons, a member of the Anglo-American artists’ set at Broadway in the Cotswolds, which included John Singer Sargent and Edwin Austen Abbey. Don’t miss the dog cemetery, where James buried his canine companions.

Rye was one of the medieval Cinque Ports, the ‘five ports’ given tax privileges in return for providing a naval base when needed. Buoyed by a vigorous smuggling trade — ‘Brandy for the Parson, ’Baccy for the Clerk/ Laces for a lady; letters for a spy,’ as Kipling wrote — Rye remained a thriving seaport until Georgian times. But then the sea retreated, leaving it high and dry like an abandoned sandcastle at low tide. The town became a useless curio, a crumbling citadel of mottled brick surrounded by soggy marshland.

Yet this vanishing shoreline preserved the town’s medieval character. The skyline that Van Dyck drew 400 years ago still looks much the same. Shorn of its maritime importance, Rye attracted artists instead of businessmen. Paul Nash is the most famous, but I prefer John Ryan, creator of the English cartoon pirate Captain Pugwash. The town is full of quirky bookshops and twee tea shops — two things that go very well together. It’s an ideal place in which to waste a small amount of money and a large amount of time.

When my in-laws arrived in 1996, Rye was fairly scruffy. Now the new high-speed rail link has brought more day-trippers and weekenders from London. Tourism is often a destructive force, corrupting what it covets, but, speaking as one of those weekenders, it seems to me that the recent changes have all been for the better. The George has been transformed from a stuffy old coaching inn into one of the best hotels and restaurants in the county. The new Kino is a wonderful arts cinema — not just somewhere to see a movie, but somewhere to hang out. Some of my favorite haunts have vanished, but a fair few have survived, like Grammar School Records, a 17th-century schoolhouse crammed with obscure LPs, some of them by a local resident named Paul McCartney. Rye is not too posh, but posh enough. ‘High Class Junk’ reads the sign outside the antique shop. The Kettle O’ Fish, by the harbor, does delicious fish ’n’ chips.

If you fancy a dip, the coast is a brisk walk away, past the harbormaster’s house. A few fishing boats still scratch out a living here, and you can buy fresh fish on the quayside. Camber Sands is one of the nicest beaches on the south coast (though the riptides can be treacherous). We used to take our children there, on their first outings to the seaside. Now new families are coming here — playing in the dunes, splashing in the shallows. I’m glad we were part of it, one link in a long chain of holiday-makers past and present.

We packed up the van, locked up the house and drove into town to get some gas. ‘It’s time for someone else to make some memories here,’ said my wife as she shut the front door for the last time. While she ran a few errands I went to take a look at Lamb House. From the outside it still looked just the same. I considered knocking on the door but then thought better of it. I walked back down the hill, past the ancient parish church, St Mary’s. I used to come here on Sunday mornings, but I liked it better out of hours. There’s a biblical quotation above the doorway that I’d seen before but never really thought about: ‘For our time is a very shadow that passeth away.’

It was a long drive back to London. It was dark when we got home. There wasn’t much to say. After my wife had gone to bed I looked up the rest of that quotation. It was from the Wisdom of Solomon: ‘And after our end there is no returning, for it is fast sealed so that no man shall come again.’