If they are on ice, whirlpools, just like letters, belong to the player whose turn it is, not the person who played them. So the "Going Swimming" talent is still doing it's thing, it's just that the whirlpool doesn't belong to you anymore. This is specifically spelled out in the Armory entry for Ice under "The Freezing Effect of Ice", after already having discussed Snowflakes, Shields and Frost Bombs...