Preheat your oven to 350°F (175°C) and grease a 9x13-inch baking pan.

In a large mixing bowl, combine the mashed sweet potatoes, granulated sugar, brown sugar, vegetable oil, eggs, buttermilk, and vanilla extract. Whisk until smooth and creamy.

In a separate bowl, sift together the all-purpose flour, baking powder, baking soda, salt, ground cinnamon, nutmeg, and ginger.

Gradually add the dry mixture into the wet mixture, folding them together until just combined.

In a small bowl, mix the granulated sugar and ground cinnamon for the cinnamon swirl.

Pour half of the batter into the prepared pan, sprinkle half of the cinnamon mixture, then add the remaining batter and top with the rest of the cinnamon mixture.

Bake for 35-40 minutes, or until a toothpick comes out clean.

Cool the cake in the pan on a wire rack.

Whisk together the powdered sugar, milk, and vanilla for the glaze until smooth.

Drizzle the glaze over the cooled cake and serve.