In this research, the optimization of ND-Co3O4 aqueous nanofluid is examined by using the modified non-dominated sorting genetic algorithm (NSGA-lI) and also response surface method. The average size of nanodiamond is 4-5 nm with purity of 99%. For the optimization issue, design variables of temperature and solid volume fraction are employed with the purpose of reducing the viscosity and increasing the thermal conductivity of nanofluids. To approximate the viscosity and thermal conductivity functions, the response surface method is implemented. At first, the Design Expert software is applied for optimization. Then the modified NSGA-II is used to achieve optimal responses more accurately, and optimization outcomes of both methods are finally compared and presented. Optimal outcomes have ascertained that the most optimal state of viscosity and thermal conductivity functions happens at the maximum temperature.