In quantum key distribution (QKD), the bit error rate is used to estimate the information leakage and hence determines the amount of privacy amplification—making the final key private by shortening the key. In general, there exists a threshold of the error rate for each scheme, above which no secure key can be generated. This threshold puts a restriction on the environment noises. For example, a widely used QKD protocol, the Bennett-Brassard protocol, cannot tolerate error rates beyond 25%. A new protocol, roundrobin differential phase-shifted (RRDPS) QKD, essentially removes this restriction and can in principle tolerate more environment disturbance. Here, we propose and experimentally demonstrate a passive RRDPS QKD scheme. In particular, our 500 MHz passive RRDPS QKD system is able to generate a secure key over 50 km with a bit error rate as high as 29%. This scheme should find its applications in noisy environment conditions.