First, we write $\overrightarrow{b}$ as a linear combination of $\overrightarrow{a}$ and $\overrightarrow{j}$ since $\overrightarrow{b}$ is a rotation of $\overrightarrow{a}$ about the y-axis.

$\vec{b}=\lambda \vec{a}+\mu \hat{j}=\lambda(-\hat{i}+2 \hat{j}+\hat{k})+\mu \hat{j}=-\lambda \hat{i}+(2 \lambda+\mu \hat{j})+\lambda \hat{k}$

$\overrightarrow{b}$ is orthogonal to $\overrightarrow{a}$ due to the right angle rotation, so $\overrightarrow{b} \cdot \overrightarrow{a} = 0$.

This implies :

$\begin{aligned} & (-\hat{i}+2 \hat{j}+\hat{k}) \cdot(-\lambda \hat{i}+(2 \lambda+\mu) \hat{j}+\lambda \hat{k})=0 \\\\ & \lambda+2(2 \lambda+\mu)+\lambda=0 \Rightarrow 6 \lambda+2 \mu=0 \Rightarrow \mu+3 \lambda=0\end{aligned}$

$$ \therefore $$ $\vec{b}=\lambda \vec{a}-3 \lambda \hat{j}=\lambda(-\hat{i}+2 \hat{j}+\hat{k})-3 \lambda \hat{j}=\lambda(-\hat{i}-\hat{j}+\hat{k})$

The magnitude of $\overrightarrow{b}$ is the same as the magnitude of $\overrightarrow{a}$ because a rotation doesn't change the magnitude of a vector. This gives us :

$$ |\vec{b}|=\sqrt{3}|\lambda|=\sqrt{6}[\because|a|=\sqrt{6}] \Rightarrow|\lambda|=\sqrt{2} \Rightarrow \lambda \neq \sqrt{2} $$

as for this value of $\lambda$ angle between $b$ and $y$-axis is not acute.

Therefore $\lambda=-\sqrt{2}$

Thus, we have :

$\overrightarrow{b} = -\sqrt{2}(-\hat{i} - \hat{j} + \hat{k}) = \sqrt{2}\hat{i} + \sqrt{2}\hat{j} - \sqrt{2}\hat{k}$

Then, we find the vector $3\overrightarrow{a} + \sqrt{2}\overrightarrow{b}$ :

$3\overrightarrow{a} + \sqrt{2}\overrightarrow{b} = 3(-\hat{i} + 2\hat{j} + \hat{k}) + \sqrt{2}(\sqrt{2}\hat{i} + \sqrt{2}\hat{j} - \sqrt{2}\hat{k}) $

$= -3\hat{i} + 6\hat{j} + 3\hat{k} + 2\hat{i} + 2\hat{j} - 2\hat{k} = -\hat{i} + 8\hat{j} + \hat{k}$

The projection of this vector onto $\overrightarrow{c}$ is given by the dot product divided by the magnitude of $\overrightarrow{c}$ :

$\frac{(-\hat{i} + 8\hat{j} + \hat{k}) \cdot (5\hat{i} + 4\hat{j} + 3\hat{k})}{\sqrt{5^2 + 4^2 + 3^2}} = \frac{-5 + 32 + 3}{\sqrt{50}} = \frac{30}{5\sqrt{2}} = 3\sqrt{2}$

So the projection of $3\overrightarrow{a} + \sqrt{2}\overrightarrow{b}$ onto $\overrightarrow{c}$ is $3\sqrt{2}$ which is option D.