This article proposes a gender and geographical origin recognition system for Arabic speakers based on the dialect and accent characteristics. We demonstrate that the speaker gender and nationality can be determined from colloquial Arabic speech and recommend that this system can be integrated to more complex biometric applications. The acoustic features of our proposed dataset used to identify the speaker's dialect and accent, are extracted using Mel Frequency Cepstral Coefficients (MFCC) and Relative Spectral Analysis (RASTA) techniques. We compare results of classification based on Gaussian Mixture Model with Universal Background Model (GMM-UBM) and Identity Vector (I-vector) classifiers implemented using the MSR Identity Toolbox, which is a MATLAB toolbox for speaker-recognition research from Microsoft. The results show a significant decrease of equal error rate (EER) when recognizing dialect or accent based on gender. In addition, feature fusion of RASTA and MFCC is used to enhance the EER. Results show a 9.8% enhancement in EER over using the RASTA features only.