handling and passing are offensive skill from the opponent, so they shouldn't affect his defence so you don'tneed to care about it when you set your offense.
A lack of handling and passing could be used for your advanage with an full court press, but for this you should have outside defence on every position.
But when i'm setting my defence i will look more about his strength and his most likely offense(if he often plays run and gun, i like to play 3-2 Zone for example), in the offense it was the same if he better defence outside i try to attack under the basket.