Whisk cream, crème fraîche, spices and vanilla seeds in an electric mixer until soft peaks form (2-3 minutes), transfer to a bowl and refrigerate until required.
Place Christmas pudding on a tray, drizzle with 20ml brandy and set aside.
Whisk yolks, sugar and remaining brandy in a heatproof bowl over a saucepan of simmering water until thick and pale (4-5 minutes), transfer to an electric mixer and whisk until cool (2-3 minutes). Fold in cream mixture, then fold in soaked Christmas pudding and glacé orange, transfer to a 2½ litre-capacity container. Freeze until firm (6 hours-overnight).
Meanwhile, for Christmas pudding crumbs, preheat oven to 200C. Combine ingredients in a bowl, tossing lightly to coat. Spread on a baking tray lined with baking paper and bake until crisp (10-12 minutes). Cool completely. Serve scoops of semifreddo scattered with Christmas pudding crumbs.