Preheat oven to 425 degrees. Wash and pat dry the plums. Cut plums in half, remove pit and cut plums into quarters. Do this until you have 2 cups of plums (about 10-12 small plums). Add the panko crumbs and stir. Spray a cookie sheet and spread the plums out on the sheet evenly. Place the plums in the oven and bake for 4-6 minutes, or until they are slightly golden and their juices have released. Remove the plums from the oven and change the oven temperature to 350 degrees. In a mixing bowl, combine the cake ingredients and whisk until smooth. Fold in the plums and mix just until incorporated but the batter has streaks of pink plums. Grease a 9x13 pan and and pour the cake batter into the pan.
Cut and slice the peaches into thin strips. Arrange the peaches on the cake so that the cake is covered in peaches. Sprinkle the sugar over the tops of the peaches. Bake the cake at 350 for 50 minutes, or until the center of the cake comes out crumbly. Serve cake with ice cream or whipped topping