The late Princess of Wales used to “dread these royal family Christmases,” royal biographer Ingrid Seward said recently

Princess Diana during her first Christmas with the royal family. Photo: Tim Graham / Contributor / Getty Images

As Princess Diana herself put it, the royal family’s traditional Christmas celebrations at Sandringham were “highly fraught,” once calling the experience “terrifying and so disappointing.”

The late Princess of Wales relayed these sentiments to biographer Andrew Morton, and Diana hated Christmases at Sandringham so much that she even ducked out early on occasion.

Princess Diana and the rest of the royal family at Sandringham. Tim Graham Photo Library/Getty Images

In the latest episode of The Sun’s Royal Exclusive show, royal biographer and editor-in-chief of Majesty Magazine Ingrid Seward said that “Diana used to always leave after lunch. When things were really not going at all well, she was to dread these royal family Christmases.”

“And she would always — and sometimes she escaped even before lunch and just did the church,” Seward added of the royal, who died at age 36 in 1997, 16 years after marrying Prince Charles in 1981.

Renowned royal photographer Arthur Edwards concurred, saying on the program, “Not always, but a few times I remember yes — I remember her passing me on the motorway coming back once.”

Queen Camilla, King Charles’ second wife, also used to leave early, Edwards added: “For many years, our Queen Camilla — she would leave after lunch and go and spend time with her own family down in Wiltshire,” he said of Camilla after she married into the royal family in 2005.

Christmas Day at Sandringham — where the royal family have long spent the holiday — includes a morning walk to church at St. Mary Magdalene on the estate’s grounds; lunch then follows ahead of the monarch’s annual speech at 3 p.m. U.K. time.

Sandringham House. Indigo/Getty Images

Diana’s former butler Paul Burrell told Marie Claire that the former Princess of Wales found Christmas at Sandringham to be “like a pressure cooker,” but that “she had to grin and bear it” since she “knew that was her duty.”

But as soon as possible, Burrell added, she was out of there: “She would escape it as soon as she could, but there were huge personalities in there she couldn’t cope with,” he told the outlet.

Christmas at Sandringham provided the setting for Spencer, the 2021 film starring Kristen Stewart as Princess Diana. Though the film is a dramatization, Diana’s former bodyguard Ken Wharfe told PEOPLE of Sandringham that “It was purgatory for her.”

Princess Diana.

Can’t get enough of PEOPLE’s Royals coverage? Sign up for our free Royals newsletter to get the latest updates on Kate Middleton, Meghan Markle and more!

Wharfe added she “confined herself to spending time in the kitchen with the chef or with people like me in the hope that time would while away and she could get back to London.”