Developing epoxy composites with high thermal conductivity and excellent mechanical properties becomes imperative in electronic and aerospace industries. This study investigates and compares the effect of adding boron nitride (BN) sheets and graphene platelets (GnPs) on the mechanical properties and thermal conductivity of epoxy resin. The study shows that incorporation of BN or GnPs into epoxy matrix significantly enhanced both mechanical properties and thermal conductivity of epoxy composites. At fractions ranging 1–4 wt%, GnPs/epoxy composites provide higher Young's modulus, fracture toughness (K1c) and critical stress energy release rate (G1c) compared to BN/epoxy composites. The thermal conductivity of the epoxy composites is up to the maximum of 0.33 Wm−1K−1 at 4 wt% of GnP loading, which is much higher than that of the composites filled with the same loading of BN (0.23 Wm−1K−1). The study emphasizes the importance of adding thin nanosheets (thickness 3–5 nm) at low loadings in developing epoxy composites to achieve desired mechanical and thermal properties.