Artificial Intelligence (AI) and Machine Learning (ML) are set to revolutionize all industries, and the Intelligent Transportation Systems (ITS) field is no exception. While ML, especially deep learning models, achieve great performance in terms of accuracy, the outcomes provided are not amenable to human scrutiny and can hardly be explained. This can be very problematic, especially for systems of a safety-critical nature such as transportation systems. Explainable AI (XAI) methods have been proposed to tackle this issue by producing human interpretable representations of machine learning models while maintaining performance. These methods hold the potential to increase public acceptance and trust in AI-based ITS.