There is no direct relationship between how long a skill takes to train and the current level of the skill.
There are a 2 things that usually make higher level pops indirectly take longer: 1) players that get those pops are likely older and so training is slower, 2) the elastically related skill(s) is more likely to be lower than the skill you're training, making training slower.
But, yes, with everything else stripped out a 20yo 6'4" player will take the same time to pop from 4 to 5 as from 14-15.