Place a rack in the center of your oven and preheat the oven to 350 degrees F. Lightly grease a 9-inch round cake pan with nonstick spray.
For the topping, melt the butter over medium-low in a small saucepan. Add the coconut sugar and ginger and stir until smooth and the butter is incorporated with the sugar, about 2 to 3 minutes. The mixture will be very thick, like melted caramel.
Remove from heat. If the butter separates from the sugar as it cools, simply stir it back together (a little separation is OK). Spoon the mixture into the prepared pan, then spread it in a thin, even layer, stretching all the way to the edges of the pan. I found it worked best to use the back of a spoon, then once the caramel was cool enough, gently press it with my fingers.
Arrange the peaches and raspberries in a single layer on top of the caramel.
To make the cake, beat the butter and sugar on medium speed in a large mixing bowl until smooth and creamy.
Add in the honey, egg, salt, baking powder, baking soda, and vanilla and stir until completely combined. Stop to scrape down the bowl as needed.
With the mixer running on medium speed, gradually mix in the whole wheat pastry flour, then the almond milk, then the all-purpose flour, incorporating completely between additions and stopping to scrape down the bowl as needed. Once the last of the flour is added, mix very briefly, just until smooth. Do not overbeat. The batter will be very thick.
Transfer the batter to the top of the fruit in the pan, spreading it to the pan’s edges. The cake will seem thin but will rise as it bakes.
Bake for 25 to 30 minutes, until deep golden brown and a toothpick inserted into the center comes out clean. The top of the cake will spring back lightly when touched. Place the pan on a wire rack and let the cake cool in the pan for 10 minutes.
Run a sharp knife along the cake’s edge to loosen it from the pan, then invert the cake onto a serving plate. Let the pan rest upside down on the plate for 1 minute, then gently lift the pan away. Let cool on the plate for 10 additional minutes. Enjoy warm or at room temperature.
TO STORE: Store leftovers in an airtight container in the refrigerator for up to 5 days. Enjoy chilled or let come to room temperature before serving.
TO FREEZE: When wrapped tightly in plastic and a layer of aluminum foil. Peach upside down cake may be kept in the freezer for up to 2 months. Let thaw overnight in the refrigerator.