Using a tablespoon or large melon baller, scoop out about a golf-ball-size portion of cake from the top of each cupcake. Soften the ice cream slightly and spoon a scoop into each cupcake cavity.
Place the frosting in a piping bag with a star tip and pipe the frosting over the tops of the cupcakes. Freeze for 1 hour. Can be made up to 4 hours in advance.