The answer to this question depends upon the nature of harems. The female occupants of harems were typically not free. Whether they were slaves or the daughters of the nobility, the nature of harems is that their occupants are typically not free to leave, whether permanently or temporarily, and neither were they typically admitted to the harem voluntarily.
I have no doubt that whatever the origin of the women in a harem, they would be quite capable and probably quite willing to serve as scribes, if only to relieve the boredom of being isolated from society.
However, the fact that the women in the harem are not free should give any wise ruler pause. History has shown that harem women are quite capable of ambition for themselves and/or their offspring, whether they originated as slaves or the daughters of the nobility. They are similarly capable of resentment for their loss of freedom. These two facts alone would mean that no matter how eagerly or reluctantly a harem woman agrees to act as a scribe or an administrator, their work cannot be trusted. Additionally, the longer the custom of using harem women as scribes and administrators goes on, the greater the chance that not only may their work be compromised by their loyalties or lack thereof, but also that the security of the secrets passing through the harem may be compromised.
I have no doubt that some harem women may be perfectly loyal, capable, enthusiastic and trustworthy administrators and scribes, however they may very well be living side by side with women who are merely pretending to possess those virtues for their own ends or the ends of those who arranged for them to enter the harem.
If the owner of a harem is concerned with 'idle hands making the devil's work' (a major problem with harems historically), there are better things for the women of the harem to do than to be exposed to the functions of government. Things such as art, which may be profitable and fulfilling yet unlikely to be able to compromise the functioning of the government.
On the other hand, if the occupants of the harem are volunteers (and weren't volunteered), who compete for entry into the harem, and who have the freedom to leave temporarily or permanently, and are paid for their labours, their trustworthiness would be little more in doubt than that of any other bureaucrat.