It's twilight in Seeb, an old fishing town on the outskirts of Muscat, and the souk is back in business. Evening prayer has ended, and men in dishdashas and kuma caps gather and stroll among the stalls. Pyramids of dried anchovies and prawns and trays of glistening cuttlefish vie for attention with bags of dried limes and fragrant cardamom, jars of golden ghee and masses of local dates. We're offered a cluster of pale fresh dates still on the stem. They're crunchy, astringent, with no more than a hint of sweetness.
There'll be plenty more to try during our adventure in the Sultanate of Oman, from the capital hugging the serene shores of the Arabian Sea to the jagged peaks of the Al Hajar mountains, and then south, following bone-dry frankincense trade routes, to the incongruously lush and tropical coast of Dhofar. Dates are not just a staple in Oman, used in cooking and turned into vinegar and syrup for marinades and curries; they're the flavour of Omani hospitality, offered as an essential accompaniment to conversation and spiced tea or coffee during almost every encounter, no matter how casual.
Oman is an exceedingly hospitable place, but it wasn't always so. The fabled home of the Queen of Sheba existed in isolation until relatively recently, largely undeveloped. Until 1970, the city gates of Muscat were closed at dusk and a curfew imposed. That was before the palace coup, when Sultan Qaboos bin Said, the current monarch, overthrew his father and ushered in a new era, spending the spoils of oil discovered in the mid-1960s on infrastructure and opening the nation to the world.
The oil rush could have turned ugly, but instead the Omanis have managed to embrace modernity and maintain a strong sense of their heritage. The magical Chedi Muscat, set on the beachfront overlooking the Gulf of Oman, beautifully embodies this synthesis of old and new. The engaging doormen wear formal traditional dress: white, ankle-length dishdasha, turban and a khanjar, the short, hook-shaped ceremonial dagger worn tucked under the belt. And while the design is contemporary, the resort has a bewitching air of the exotic. The almost blindingly white suites are scattered through eight and a half hectares of tranquil gardens and decorative pools, where domed pagodas are likely to be occupied by dishdasha-clad guests tapping on laptops. Inside, the suites have arabesque touches in metal fretwork lamps and sculptural sunken stone baths, and that famed hospitality comes in the form of decanters of gin, vodka and whisky, along with plates of fresh fruit, jars of nuts – and dates, of course.
On a Saturday night our guides, Issa and Ahmed, take us downtown to eat like locals. Muscat is a lowslung city bound by the sea on one side and mountains on the other, its white and sand-coloured buildings no higher than eight storeys by royal decree. Where Oman's flashy neighbours Dubai and Abu Dhabi are forests of high-rises, here the Islamic identity is maintained, most buildings bearing Arabic flourishes and the broad streets lined with curlicued streetlights.
Beside a small mosque in the Ministries District, a café called simply Tea House is the go-to for Oman's crêpe-like flatbread, khubz rakhal. It's folded over savoury fillings such as cheese and egg, or a combination of the two with chips, then grilled, and served along with frothy karak tea, enriched with condensed milk and saffron. The tables are full, inside and out, and waiters ferry takeaway orders on trays to a constant procession of cars that pull up outside.
Closer to the corniche we try mishkak, the popular grilled skewers of various meats and seafood sold on the streets. Trucks and vans congregate on roadsides all around the city at dusk, and the drivers set up makeshift grills, illuminated by humming generator-powered lights. We juggle sticks of grilled beef and lamb, doused with spicy tamarind or chilli sauce, and watch the parade of pimped-up cars cruise by. It's Saturday night, after all.
Like many Omanis, Issa is proud of what his country has achieved in less than 50 years. In 1994, he tells us, Omani women became the first in the Gulf region to be given the right to vote and to stand in parliamentary elections. There are currently seven women ministers in the government – "and the Saudis only just allowed women to drive cars", he hoots.
There are more figures relayed to us with pride at Muscat's Grand Mosque, a majestic marble complex of courtyards and arched walkways surrounding a prayer hall for 6,500 worshippers. They kneel upon the second largest hand-woven carpet in the world, Ahmed says. It's a 21-tonne masterpiece that took 600 women four years to weave. The colossal crystal chandelier above it measures fourteen metres by eight; a cherry-picker is required to clean and change its 1,122 bulbs.
The branch of Islam practised in Oman is called Ibadi, a liberal form of the religion that preaches tolerance of race and religion with no discrimination. The irrepressible Naima Ali, a volunteer at the Islamic Cultural Centre in the surrounding manicured grounds, greets us with tea scented with cardamom and the obligatory dish of dates. The centre opened in the wake of the 9/11 attacks in the US, she says, with the aim of helping visitors learn more about Islam. "People were confused," she says. "We're closing the gaps. Some of the fog has lifted when they leave. I'm here with a small hammer, breaking down barriers."
The souks are as central to Omani daily life as the mosques. About 20 kilometres from downtown Muscat is the old port of Muttrah, curled around a harbour in which traditional fishing boats are dwarfed by the royal yacht. A tall arched gate marks the entrance to the Muttrah Souk, one of the oldest in Oman, perhaps the Arab world. It's a pleasurably confusing maze of narrow alleys lined with shops arranged roughly by wares: silver and gold, pashminas and handicrafts, frankincense and myrrh. This must be one of the few places in the world where you can find the gifts of the three wise men under one roof. Ground turmeric, tamarind pods, cardamom, saffron, dried roses, lemons and limes are amassed in kaleidoscopic displays. Bartering is de rigueur, though not in the case of gold, and it pays to have someone do it for you. The obliging nature of the Omanis extends even to the touts, who show nothing but good humour when we pass them by.
We have lunch overlooking the port at Bait Al Luban, on the third floor of a 140-year-old former guesthouse. Its décor is traditional – wooden fretwork screens, brightly patterned cushions, a jalsa floorseating area – and so is the food. Paplou soup, made with locally caught longface emperor, a type of bream, is bright with turmeric. A salad of white onion and tomato is peppered with strips of salted shark, a staple. Shuwa is a specialty – lamb marinated in oil and spices, wrapped in palm fronds, and roasted in a fire pit for at least six hours. It's served with rice cooked in a meaty broth with chickpeas and peppercorns, and a lemon-garlic sauce. Then come the sweets, gently spiced and fragrant – first luqaimat dumplings steeped in date syrup and honey and scented with saffron, and then a final round, served on a silver cake stand, of bite-sized treats made with coconut and saffron, date and sesame, and caramelised condensed milk. The experience is complete when guests' hands are doused in rosewater as they leave.
The next day we drive two hours south-west to Nizwa, the old capital city. The highway is flanked by the Al Hajar mountain range to the west, in the centre of which lies our destination for the evening, Jabal Akhdar mountain. To the east, large stone houses, mostly two-storeyed to accommodate extended families as is the Omani custom, cluster in sun-baked towns on the flat.
Nizwa is near deserted in the midday heat, rising well into the 40s. One of the few people to be seen is a Bedouin sitting cross-legged in the shade of his van, smoking as he awaits customers for his dried shark. The souk is likewise quiet, but the Abu Eyad Al Manthri date shop is doing a lively trade. A dozen varieties of date are on offer, ranging in colour from creamy caramel to rich dark brown. Some are coated with sesame seeds or filled with tahini. Dates feature in myriad sweets, alongside various iterations of Omani halwa flavoured with saffron, rosewater, dates or nuts, including a pungent garlic-infused version eaten as a morning tonic. Proprietor Ali Al Manthri, offering the customary dates and coffee, tells us he can have up to 40 varieties on sale, from 250 or so varieties indigenous to Oman.
The ascent of Jabal Akhdar is restricted to four-wheel drives, a rule enforced at a checkpoint at the foot of the mountain. It's a wide, sealed road, with plenty of laybys where we can admire the view over the ranges, but it's steep and tortuous. As we near the top there's a loud crack like a gunshot. "Chips!" shouts Ahmed over the engine. A packet in his snack supply in the back has succumbed to the altitude. It's like an exclamation mark for the 2,000-metre sign we just passed.
Jabal Akhdar means green mountain, which seems a misnomer when we reach the grey-brown plateau at the top. But the region is renowned for its roses, from which rosewater is distilled, and for peaches, grapes and pomegranates; limbs heavy with the rosy fruits can be glimpsed hanging over garden walls everywhere.
Anantara Al Jabal Al Akhdar Resort seems to emerge from the rocky landscape as its terracotta-hued buildings of local stone heave into view. It sits spectacularly on the brink of a canyon surrounded by craggy peaks and overlooking precipitous terraces of greenery. The welcome here is as fragrant as it is exotic. Frankincense billows through the lobby, which opens to a grand courtyard with seating around a central fireplace, and a café specialising in tea. The signature blend is infused with the famed local damask rose, while the minted Moroccan tea is refreshing in the heat, which hasn't dropped noticeably despite the elevation. From here a watercourse, echoing ancient falaj irrigation channels, bisects a sprawling garden of native shrubs, pomegranates and roses. Dramatically lit at night, it leads the way to the cliff's edge.
It's clear that French-Moroccan architect Lotfi Sidirahal drew inspiration from traditional forts in designing the resort, notably in the cone-like tower that houses the signature restaurant, Al Qalaa. But the dining option to beat here is a private dinner set on a platform that stretches to the brink of the canyon, named Diana's Point. The princess apparently visited the site briefly back in 1986 (less splendidly, the resort's Bella Vista restaurant has named a burger in her honour). In this dramatic setting a chef prepares a Lebanese spread of meze and grills, served to the single pampered table by a dedicated maître d'.
A hike with one of the resort's "mountain gurus" to the three largely abandoned villages above the terraces is the chance to see the life coaxed from stony ground. Much of the walk follows the falaj channels that deliver water to groves of pomegranate, pear and walnuts before reaching the terraces themselves, where banks of rosebushes seem to cling to the escarpment. The mountain's name is not such a misnomer after all.
The children who once lived here would reach the school in the valley below, incredibly, by bounding down the terraces – a feat infinitely more suited to the goats that wander tightrope fashion along the stone walls in the villages in search of water and low-hanging leaves.
Water – although in this case an abundance of it – is also the defining feature of the Dhofar region in Oman's south. Having flown for the best part of two hours over desert- like plains often starker than the Australian outback, it's a surprise to see the landscape turn green as we descend to Salalah, the capital of the region. From May to early September the region is blessed with the khareef, or monsoon. The balmy weather rejuvenates the nearby ranges and attracts holidaymakers from all over the Arabian Peninsula, seeking to escape the 50-degree heat at home.
It also accounts for the huge walled plantations of banana and coconut palms that ring the city and line the approach to Al Baleed Resort Salalah by Anantara. Reminiscent of a whitewashed village, albeit an uncommonly luxe one, the resort has at its heart a long infinity pool that stretches to a private beach. The pool is flanked by 30 guestrooms, two of three restaurants, and rows of villas set among graceful coconut palms. Behind their high walls each villa has a courtyard with a four-poster cabana beside a garden-fringed pool.
The resort takes its name from the neighbouring Al Baleed World Heritage site, the remains of an ancient port and trading post for frankincense. While its fragrance wafts through hotel lobbies, shops, souks and homes throughout Oman, most of the country's frankincense, and the most prized, comes from the Dhofar region. This is where you come to follow the frankincense trail.
Flocks of camels lope beside the road as we head inland to see the source of this legendary resin. Our guide, Hussain, stops for us to marvel at the camels, and it turns out they eat dates, too, as we discover when one pokes her long-lashed head through the open window of our car. Save the date!
The small, rather scraggy frankincense trees grow wild in harsh, stony soil, and most notably in the protected Wadi Dawkah, about 40 kilometres from Salalah. As Hussain shows us, the resin is harvested by scraping off a swatch of the papery bark and making a small cut. The sap that instantly bleeds out is left for a few days to harden before it's collected, destined to be graded and sold for incense or distilled into essential oil, attributed with all manner of mystical and medicinal benefits.
East of Salalah lie the ruins of the ancient port of Khor Rori, also known as Sumhuram, and the fabled palace of the Queen of Sheba. Overlooking the beach and a natural harbour, it was once a trading post for the then-swashbuckling nation and a landmark on the frankincense trail. This was the destination of laden caravanserai and the departure point for their cargo on sailing ships bound for the Mediterranean, India, and other Eastern kingdoms. Signs in the maze of crumbling stone walls, now only a few metres high, pinpoint the sites of homes, workshops, a temple and a "monumental building" inside what were once imposing city walls. Inscriptions in old Arabic mark the founding of the port, dating back to the 4th century BC. Sadly, they make no mention of Sheba, immortalised in the Qur'an and the Bible – and her presence here remains the stuff of legend.
Later that evening, Salalah's Al Hafah Souk is buzzing and the frankincense stalls are the busiest. The vendors drop little nubs of the resin onto charcoal in terracotta burners and prospective buyers wave the fragrant smoke towards their noses to appraise its quality. Haggling and joking ensue. Many then wander to Lialy Hadrmout, a Yemenite restaurant that serves grilled skewers and flatbread that's slapped against the inside of a cauldron-like fire pit to cook.
Back at the resort, after a meal overlooking the beach, groups of men gather on the terrace to smoke shisha. Once they've chosen a flavour from the menu, perhaps grape and mint, the waiter disappears to prepare the tall hookah pipes and re-emerges with them locked and loaded. The sound of bubbling and banter makes a soothing soundtrack to a moonlit view of the Arabian Sea. It's a kind of peace in the Middle East.
The next day at the airport before our departure, we head to a shop to buy camel-milk chocolate. Unable to decipher the flavour on a pack, we ask for help from the shop assistant. "Dates," she laughs. "Surprise!"