When the performance of different evolutionary multiobjective optimization (EMO) algorithms is compared, the same population size is usually used for all EMO algorithms in computer simulations. This setting is to obtain a solution set of the same size from a different algorithm. However, in general, each algorithm may have its own best parameter specifications for each test problem. Thus, it may be difficult to appropriately specify the same population size for all algorithms for their fair comparison. A different algorithm may be evaluated as being the best for a different specification of the population size. An alternative setting is to allow each algorithm to use its own best population size. In this setting, a solution set of different size is obtained from each algorithm. It may be difficult to perform fair comparison using solution sets of different size. In this paper, we discuss the difficulty in comparing EMO algorithms under these two settings of the population size: the same specification for all algorithms and a different specification for each algorithm. First, we discuss the effect of the number of non-dominated solutions on some performance indicators. Next we show the difficulty in the first setting: Performance of each algorithm depends on the population size. Then we discuss the difficulty in the second setting: The size of a solution set obtained by each algorithm is not the same. In this setting, we examine the use of solution selection as a post-processing procedure to choose the same number of solutions from each solution set of different size. The selected solutions are used for performance comparison.