Cube the brown and serve rolls and arrange on a parchment lined baking sheet. Toast in the oven under the broiler for 5 minutes or until lightly golden. Set aside to cool and leave to dry.
To assemble, liberally brush a 10-inch cast iron skillet or similar with butter. Arrange toasted bread cubes in the skillet.
In a large mixing bowl, whisk together the half and half, eggs, sugar, brown sugar, vanilla, cinnamon and nutmeg until fully combined. Pour evenly over the bread. Cover with plastic wrap and chill overnight.
To bake: Uncover casserole and allow to sit on counter for 15 minutes. Preheat the oven to 350°F.
Bake for 40-50 minutes until puffed and top is crusty and golden.
Meanwhile in a small mixing bowl whisk together the powdered sugar, cream and cream cheese until smooth.
Remove from oven and rest for 5 minutes to allow casserole to settle, then frost. Slice into wedges and serve.