$$\begin{aligned} & f_1(x)=\log _5\left(18 x-x^2-77\right) \\ & \therefore 18 x-x^2-77>0 \\ & \quad x^2-18 x+77<0 \\ & \quad x \in(7,11) \alpha=7, \beta=11 \\ & f_2(x)=\log _{(x-1)}\left(\frac{2 x^2+3 x-2}{x^2-3 x-4}\right) \\ & \therefore \quad x-1>0, x-1 \neq 1, \frac{2 x^2+3 x-2}{x^2-3 x-4}>0 \\ & \quad x>1, x \neq 2, \frac{(2 x-1)(x+2)}{(x-4)(x+1)}>0 \\ & \quad x>1, x \neq 2, \end{aligned}$$

$$\begin{aligned} & \therefore \quad x \in(4, \infty) \\ & \therefore \gamma=4 \\ & \therefore \alpha^2+\beta^2+\gamma^2=49+121+16 \\ & =186 \end{aligned}$$