In a large stainless mixing bowl, whisk together the eggnog, cornstarch, condensed milk and egg yolks.
Place mixing bowl on top of a double boiler (simmering water in the pan below, not so high as to touch the bowl) and cook over low heat, whisking constantly, until mixture is like thick pudding. About 20 minutes.
Remove from heat and stir in vanilla and nutmeg.
Slice bananas (I do this at the last minute).
Place enough of the custard in the bottom of the pie shell to coat the bottom and sides.
Fill with one cup (half) of the banana slices, then top with half of the remaining custard.
Add the rest of the bananas and pat down, the top with the rest of the custard, using a spatula to even it out all the way to the edges of the crust to keep as much air out as possible.
Cover pie with a piece of plastic wrap to keep a skin from forming on top, and refrigerate several hours or overnight until cold and set.
Top with whipped cream and a sprinkling of extra nutmeg before serving.