Since the heterogeneous Internet Service Provider (ISP) router policies prevent the quality-of-service (QoS) multimedia applications which require IP layer multicasting from being widely deployed on the Internet, the mechanism of implementing such applications by application layer broadcasting through organizing the multicast group in a peer-to-peer overlay network is proposed. In this paper, we study the bandwidth-delay-constrained minimum spanning tree problem in an overlay network, which is NP-complete. We propose a novel genetic algorithm for resolving this problem and compare it with a state-of-the-art method. Simulation results demonstrate that the proposed algorithm is effective and efficient.