Given quadratic equation,

$$3{x^2} + (\alpha - 6)x + (\alpha + 3) = 0$$

Let, a and b are the roots of the equation,

$$\therefore$$ $$a + b = - {{\alpha - 6} \over 3}$$

and $$ab = {{\alpha + 3} \over 3}$$

For real roots,

$$D \ge 0$$

$$ \Rightarrow {(\alpha - 6)^2} - 4\,.\,3\,.\,(\alpha + 9) \ge 0$$

$$ \Rightarrow {\alpha ^2} - 12\alpha + 36 - 12\alpha - 36 \ge 0$$

$$ \Rightarrow {\alpha ^2} - 24\alpha \ge 0$$

$$ \Rightarrow \alpha (\alpha - 24) \ge 0$$

$$\therefore$$ $$\alpha > 24$$ or $$\alpha < 0$$

$$\therefore$$ Real roots of the equation possible for $$\alpha > 24$$ or $$\alpha < 0$$.

Now, sum of square of roots

$$ = {a^2} + {b^2}$$

$$ = {(a + b)^2} - 2ab$$

$$ = {{{{(\alpha - 6)}^2}} \over 9} - 2\,.\,{{(\alpha + 3)} \over 3}$$

$$ = {{{\alpha ^2} - 12\alpha + 36 - 6\alpha - 18} \over 9}$$

$$ = {{{\alpha ^2} - 18\alpha + 18} \over 9} = f(x)$$

$$\therefore$$ Sum of square of roots are minimum when $${a^2} + {b^2} = $$ minimum.

$$\therefore$$ $$f{(\alpha )_{\min }} = {{{\alpha ^2} - 18\alpha + 18} \over 9}$$

Value of quadratic equation $${\alpha ^2} - 18\alpha + 18$$ is minimum at $$\alpha = - {b \over {2a}} = - {{( - 18)} \over {2\,.\,1}} = 9$$

But for real roots $$\alpha$$ should be less than 0 or greater than 24.

So, there is no value of $$\alpha$$ in the range $$\alpha > 24 \cup \alpha < 0$$ where sum of squares of two real roots is minimum.

$$\therefore$$ S is an empty set.