Artificial Intelligence (AI) and Machine Learning (ML) are set to revolutionize all industries, and the Intelligent Transportation Systems (ITS) field is no exception. While ML, especially Deep Learning models, achieve great performance in terms of accuracy, the outcomes provided are not amenable to human scrutiny and can hardly be explained. This can be very problematic, especially for systems of a safety-critical nature such as transportation systems. Explainable AI (XAI) methods have been proposed to tackle this issue by producing human interpretable representations of machine learning models while maintaining performance. These methods hold the potential to increase public acceptance and trust in AI-based ITS. Artificial Intelligence (AI), particularly Machine and Deep Learning, has been significantly advancing Intelligent Transportation Systems (ITS) research and industry. Due to their ability to recognize and to classify patterns in large datasets, AI algorithms have been successfully applied to address the major problems and challenges associated with traffic management and autonomous driving, e.g., sensing, perception, prediction, detection, and decision-making. However, in their current incarnation, AI models, especially Deep Neural Networks (DNN), suffer from the lack of interpretability.