This paper presents a modern power system protection philosophy for Out-Of-Step (OOS) detection that can be used in a proposed smart protection relays in smart grid applications. The protection philosophy depends on a combinational/hybrid algorithm between online energy function and online rotor angle/angular speed characteristics. The hybrid algorithm increases power system security due to accurate OOS detection. The hybrid algorithm overcomes the demerits of conventional OOS detection algorithms employed in conventional relays in current grids or smart grids. The hybrid algorithm is tested using Single Machine Infinite Bus (SMIB) to ensure the validity, effectiveness, reliability, and security upon the operation of the power system grids or/and smart grids.