This line is spoken by Baroka, the Bale (village head). He uses this mock outrage as part of his elaborate scheme to seduce Sidi. By pretending he has no one to guard his privacy, he creates an opportunity to lure Sidi into his bedroom under the guise that he is a vulnerable, "impotent" old man.