The periodic solutions of a strongly cubic nonlinear oscillator whose motion is described with the generalized Rayleigh equation are studied. Approximate analytic solving methods are introduced. A new method based on homotopy and averaging is developed to determine the limit cycle motion. The obtained analytical solutions are compared with those calculated by the elliptic harmonic balance method with generalized Fourier series and Jacobian elliptic functions. Three types of cubic nonlinearity are considered: the coefficients of the linear and cubic terms are positive, the coefficient of the linear term is positive and that of the cubic term is negative and the opposite case. Comparisons of the analytical solution and numerical solution, obtained by using the Runge–Kutta method, are illustrated with examples.