We don't. There are about 400 cosmic background photons per cubic centimeter of space at any one instant in time. It is these local photons that the detectors are in fact picking up. BUT, these photons are not arriving at our location in completely random directions. Instead, the vast majority are traveling along our line of sight into space, no mater in what direction we point our instruments. It is true that some of these photons are passing our location, but there are always more immediately behind them so we will always see a continuous stream of radiation coming our way. The point is that these photons have not interacted with each other, or with other matter since the time the universe was over 2000 K hot. This happened about 300,000 years after the Big Bang, which means that the photons we are intercepting have been traveling through space since 15 billion years ago minus 300,000 years. They have traveled just under 15 billion light years to get here. The ones we do not 'catch' stream through our galaxy and our solar system and are available for creatures on some other planet to intercept if they know what they are looking for.