A geothermal heat exchanger (GHE) uses geothermal energy for heating or cooling residential places during winter or summer. Two different designs of GHEs, the straight pipe and coiled pipe designs, are evaluated in this study, and the effect of nanofluids as the working fluid is investigated. For this purpose, a mathematical model is developed, validated, and used to predict the temperature gain, heat gain, exergy gain, and pressure loss of the working fluid for different concentrations of additive ceramic nanoparticles of aluminum oxide (Al2O3) and magnesium oxide (MgO) in the working fluid. It is shown that the coiled pipe design has a better performance compared to the straight pipe design for GHEs. It is also shown how the temperature, heat gain, and exergy gain change with increasing the additive nanoparticles into the base fluid, which is water, while the pressure loss does not change significantly. The temperature gain increases about 60% when the volume fraction of nanoparticles in the base fluid reaches 2%. This also helps to improve the natural circulation of working fluid and the GHE may not need a circulating pump to run at low flowrates. It is also shown that the additive MgO nanoparticles are more effective than Al2O3 nanoparticles to improve the GHE performance.