Phononic crystals are periodically engineered structures with special acoustic properties that natural materials cannot have. One typical feature of phononic crystals is the emergence of band gaps wherein the wave propagation is prohibited due to the spatial periodicity of constituents. This paper presents a generalized plane wave expansion method (GPWEM) and a voxel-based discretization technique to calculate the band structures of given three-dimensional phononic crystals. Integrated with the adaptive genetic algorithm (AGA), the proposed method is used to perform topological optimization of constituent distribution to achieve maximized band gap width. Numerical results yielded from the optimization of a three-dimensional cubic phononic crystal verify the effectiveness of the proposed method. Eigenmodes of the phononic crystal with the optimized topology are investigated for a better understanding of the mechanism of band gap broadening.