Test case prioritization is an essential part of test execution systems for large organizations developing software systems in the context that their software versions are released very frequently. They must be tested on a variety of compatible hardware with different configurations to ensure correct functioning of a software version on a compatible hardware. In practice, test case execution must not only execute cost-effective test cases in an optimal order, but also optimally allocate required test resources, in order to deliver high quality software releases. To optimize the current test execution system for testing software releases developed for Videoconferencing Systems (VCSs) at Cisco, Norway, in this paper, we propose a resource-aware multi-objective optimization solution with a fitness function defined based on four cost-effectiveness measures. In this context, a set of software releases must be tested on a set of compatible VCS hardware (test resources) by executing a set of cost-effective test cases in an optimal order within a given test cycle constrained by maximum allowed time budget and maximum available test resources. We empirically evaluated seven search algorithms regarding their performance and scalability by comparing with the current practice (random ordering (RO)). The results show that the proposed solution with the best search algorithm (i.e., Random-Weighted Genetic Algorithm) improved the current practice by reducing on average 40.6% of time for test resource allocation and test case execution, improved test resource usage on average by 37.9% and fault detection on average by 60%.