Genome evolution is predicted to be rapid following the establishment of new (neo) sex chromosomes, but it is not known if neo-sex chromosome evolution plays an important role in speciation. Here we combine extensive crossing experiments with population and functional genomic data to examine neo-XY chromosome evolution and incipient speciation in the mountain pine beetle. We find a broad continuum of intrinsic incompatibilities in hybrid males that increase in strength with geographic distance between reproductively isolated populations. This striking progression of reproductive isolation is coupled with extensive gene specialization, natural selection, and elevated genetic differentiation on both sex chromosomes. Closely related populations isolated by hybrid male sterility also show fixation of alternative neo-Y haplotypes that differ in structure and male-specific gene content. Our results suggest that neo-sex chromosome evolution can drive rapid functional divergence between closely related populations irrespective of ecological drivers of divergence.