General term of $\left(a x^2+\frac{1}{2 b x}\right)^{11}$ is

$$ T_{r+1}={ }^{11} C_r\left(a x^2\right)^{11-r}\left(\frac{1}{2 b x}\right)^r={ }^{11} C_r(a)^{11-r}\left(\frac{1}{2 b}\right)^r x^{22-3 r} $$

$$ \begin{array}{rlrl} &\text { Now, } 22-3 r =7 \\\\ &\Rightarrow 15 =3 r \\\\ &\Rightarrow r =5 \end{array} $$

and general term of $\left(a x-\frac{1}{3 b x^2}\right)^{11}$ is

$$ \begin{aligned} T_{r+1} & ={ }^{11} C_r(a x)^{11-r}\left(-\frac{1}{3 b x^2}\right)^r \\\\ & ={ }^{11} C_r a^{11-r}\left(-\frac{1}{3 b}\right)^r x^{11-3 r} \end{aligned} $$

Now, $11-3 r=-7$

$$ \Rightarrow 18=3 r \Rightarrow r=6 $$

$\begin{aligned} & \text { Since, coefficient of } x^7 \text { in }\left(a x^2+\frac{1}{2 b x}\right)^{11} \\\\ & =\text { Coefficient of } x^{-7} \text { in }\left(a x-\frac{1}{3 b x^2}\right)^{11} \\\\ & \Rightarrow{ }^{11} C_5(a)^6\left(\frac{1}{2 b}\right)^5={ }^{11} C_6(a)^5\left(-\frac{1}{3 b}\right)^6 \\\\ & \Rightarrow \frac{a}{32 b^5}=\frac{1}{729 b^6} \Rightarrow 729 a b=32\end{aligned}$