Preheat oven to 400 degrees F. Lightly grease a 1 1/2 to 2-quart casserole dish with baking spray. Set aside.
With a mandoline or sharp chef's knife, slice the potatoes into very thin slices, 1/8-inch-thick or less. Place the potatoes in a large bowl, drizzle with olive oil, then sprinkle with the minced garlic, salt, and pepper. Toss to coat the slices as evenly as you can.
Spread 1/3 of the potato slices in the bottom of the prepared dish. Crumble half of the goat cheese over the top. Repeat with the next 1/3 of the potatoes, then the remaining goat cheese, then finish by layering on the final third of the potatoes. The potatoes may discard some liquid as they rest in the bowl. If this happens, simply leave the liquid in the bottom bowl and shake the potato slices gently in your hands to remove excess liquid before layering them in the dish.
Pour the milk evenly over the top of the dish, then sprinkle with the Parmesan cheese. Cover the dish with foil, bake for 30 minutes, then uncover and bake for 15 additional minutes, until the top has browned. Scatter the rosemary over the top. Serve hot.
*I do not recommend russet potatoes for this recipe, as they have less flavor, a dryer texture, and thicker skin than Yukon golds.
TO MAKE AHEAD: The dish can be prepared through Step 3 one day in advance (wait to add the milk/cheese until just before you bake). Store covered in the refrigerator, let come to room temperature, then continue with the recipe as directed.
TO MAKE FOR A CROWD: This recipe can be doubled and baked in a 3-quart casserole dish. You may need to adjust the baking time.
TO STORE: Place cooked and cooled leftovers in an airtight storage container in the refrigerator for up to 4 days.
TO REHEAT: Gently reheat leftovers in an oven-safe baking dish in the oven at 350 degrees F until warmed through, adding a splash of broth as needed. You can also reheat this dish in the microwave until hot.
TO FREEZE: Store cooked and cooled leftovers in a freezer-safe storage container in the freezer for up to 3 months. Let thaw in the refrigerator overnight before reheating.