To improve the performance and increase the lifetime of oil-filled transformers, the thermal and dielectric properties of the transformer oil should be enhanced. Recently, nanotechnology was used as an effective science in the field of transformer oil development. In this study, barium titanate (BT) nanoparticles were inserted into the base transformer oil by a concentration of 0.005 g/L as an individual nanofluid sample (INFS). This insertion enhances the heat transfer coefficient by 33% but the breakdown voltage (BDV) was decreased by >10%. To overcome this problem of dielectric properties degradation, other three hybrid nanofluid samples (HNFS) were prepared using three different types of metal oxide (MO) nanoparticles; titania, alumina, and silica. These samples were prepared by adding a concentration 0.01 g/L of MO nanoparticles together with 0.005 g/L of BT nanoparticles into the oil. The thermal and dielectric properties of HNFS were measured to study the behaviour of nanoparticles hybridisation on transformer oil properties. HNFS using titania nanoparticles provided the best composition regarding either BDV or heat transfer coefficient. Dynamic light scattering (DLS) technique was used to evaluate the particle size distribution of hybrid nanoparticles and to clarify the corresponding physical mechanisms behind the obtained enhancement.