The evaluation of the lexicon of an MT system is a pivotal issue for the process of evaluating MT systems. In this paper, the authors introduce a new methodology that was devised to enable developers and users of MT Systems to evaluate their lexicons semi-automatically. This new methodology is based on the idea of the importance of a specific word or word sense, to a given application domain. This weight determines how the presence of such a word in the lexicon affects the MT system's lexical quality. The method, which adopts a black-box approach to evaluation, was implemented and applied to evaluating the lexicons of three commercial English–Arabic MT systems. Once this database was built, test suites were presented to each of the MT systems under evaluation and their output rated by a human operator as either correct or incorrect. Based on this rating, an overall automated evaluation of the lexicons of the systems was deduced.