You don't need xcode, you need QtCreator, you need some compiler (Clang probably) and the libraries.
That's why I wrote to Install offline version (Qt 5.4.0 for Mac (599 MB)), there should be less options. On my Linux setup, in the page with components, I always unselect Qt WebEngine and Qt Extras (Qt Canvas & Qt WebView). No idea how it looks in Mac setup.
By the way, once you compile the program, you can just copy few needed libraries,, and remove the whole SDK. That's like 20 MB only.