Numerical investigation has been carried out to compare the heat transfer performance and fluid flow behavior of microchannel heat sinks with circular and rhombus pin fins which are arranged in an in-line manner. Diameter and sides are 1 mm for circular and rhombus fins. Three-dimensional (3D) computational domain has been simulated using two types of cooling medium, i.e., water and Al2O3–H2O nanofluid. A comprehensive comparative analysis has been presented considering the coolants and pin fin profiles as variable parameters. Two operating variables, i.e., heat flux (q) and Reynolds number (Re), are varied in the range of q = 100–400 kW/m2 and Re = 100–400. A total of 64 cases have been simulated to identify the promising features of both the pin fins attributed to improved heat transfer and overall thermal performance. Comparison has also been made between the coolant medium to find out their heat dissipation potential and flow characteristics in the heat sink. Results obtained in terms of average bottom wall temperature, heat transfer coefficient, Nusselt number (Nu), and pressure drop demonstrate that heat sink with rhombus pin fins dissipates more heat compared to its counterpart. It is attributed to the shape and geometry of rhombus fins that facilitate distinct fluid flow behavior; nevertheless, the pressure drop is less in the circular fin heat sink. Moreover, for constant value of Re, nanofluid extracts more heat compared to water in both configurations of the heat sink.