Let
\(H \subseteq G\) be nonempty. Assume
\(H\) is a subgroup. By assumption,
\(H\) is a group under the restriction of the binary operation to
\(H\text{.}\) This means
\(H\) is closed under the group operation and closed under inverses. Hence for all
\(a,b \in H\text{,}\) \(b^{-1} \in H\) and thus
\(ab^{-1} \in H\text{.}\)
Conversely, assume for all \(a,b \in H\text{,}\) \(ab^{-1} \in H\text{.}\) Since \(H\) is nonempty, there exists an element \(a \in H\text{.}\) Hence \(1 = aa^{-1} \in H\text{.}\) Now we see that \(H\) is closed under inverses because for all \(a \in H\text{,}\)
\begin{equation*}
a^{-1} = 1a^{-1} \in H\text{.}
\end{equation*}
Finally, we conclude that \(H\) is closed under the binary operation on \(G\) because for all \(a,b \in H\text{,}\) \(b^{-1} \in H\) implies
\begin{equation*}
ab = a\left(b^{-1}\right)^{-1} \in H\text{.}
\end{equation*}
Therefore \(H\) is a subgroup of \(G\) if and only if for all \(a,b \in H\text{,}\) \(ab^{-1} \in H\text{.}\)