gr.group theory - Is an invertible biset necessarily a bitorsor?

Question

Let $G$ be a group, and let $X$ be a $G$-biset that is (weakly) invertible with respect to the contracted product. Is $X$ necessarily a bitorsor?

Background

By $G$-biset, I mean a set equipped with commuting left and right $G$-actions. There is a standard tensor product on the category of $G$-bisets called the contracted product; it is defined by $X times_G Y = X times Y / (x cdot g, y) sim (x, g cdot y)$, where $G$ acts on the left by its left action on $X$, and on the right by its right action on $Y$. The unit object is the group $G$, where $G$ acts by left and right multiplication on itself.

A left $G$-torsor is a left $G$-set $X$ such that the map $G times X to X times X$, $(g, x) mapsto (g cdot x, x)$ is a bijection. A right $G$-torsor is defined analogously. A $G$-bitorsor is a $G$-biset that is both a left and a right $G$-torsor. A $G$-bitorsor $X$ is necessarily invertible with respect to the contracted product; its inverse is the opposite $G$-bitorsor $X^{operatorname{op}}$. This bitorsor has the same objects as $X$, but $g in G$ acts on the left (resp. the right) by the right (resp. left) action of $g^{-1}$.

It follows from a simple counting argument that when $G$ is a finite group, any invertible $G$-biset is a $G$-bitorsor. Is this true for arbitrary groups (and more generally, in an arbitrary topos)? What about if we replace "invertible" by "right- (or left-) invertible"?

I can show, at least in the punctual topos (and I think it's true in general), that if $X^{operatorname{op}}$ is an inverse to $X$, then $X$ must be a $G$-bitorsor. So the question is whether a $G$-biset can have an inverse not of this form.

The reason I'm interested in this question is that I want to understand how to generalize bitorsors to higher categorical settings. A possible generalization would be an invertible profunctor, but this is only a good definition if the answer to my question is affirmative.