Yes, I think you are within your rights to let them know that the job, for whatever reason, is not a good fit and that you wonder if there is another possibility. You can offer to stay until the agency can find a replacement for you.
Now, don't do that too often or they will begin to think that you are the problem!
"I am open, and I am willing,
For to be hopeless would seem so strange.
It dishonors those who go before us,
So lift me up to the light of change."
Holly Near