The nonlinear analysis of the Rayleigh-Taylor instability of two immiscible, viscous magnetic fluids in porous media, is performed for two layers, each has a finite depth. The system is subjected to both vertical vibrations and normal magnetic fields. The influence of both surface tension and gravity force is taken into account. Although the motions are assumed to be irrotational in each fluid for small perturbations, weak viscous effects are included in the boundary condition of the normal stress balance. The method of multiple scale expansion is used for the investigation. The evolution of the amplitude is governed by a nonlinear Ginzburg-Landau equation which gives the criterion for modulational instability. When the viscosity and Darcy’s coefficients are neglected, the cubic nonlinear
Schr¨odinger equation is obtained. Further, it is shown that, near the marginal state, a nonlinear diffusion equation is obtained in the presence of both viscosity and Darcy’s coefficients. Stability analysis and numerical simulations are used to describe linear and nonlinear stages of the interface evolution and then the stability diagrams are obtained. Regions of stability and instability are identified.