The record holder for a quasar goes to PC 1247+3406 discovered in 1991 by astronomers Donald Schneider, Maarten Schmidt and James Gunn. The discovery was published in the Astronomical Journal, vol. 102, page 837. This quasar has a redshift of z = 4.897 and is located at RA(1950) = 12h 47m 17.8s and Dec(1950) = +34d 6' 12". For a Hubble constant of 50 km/sec/mpc and q_0 = 0.5, the light left this quasar when the universe was 7 % of its current age or less than 1 billion years old.
The most distant known radio galaxy was recently discovered by Steve Rawlings and his colleagues. See Nature October 10, 1996 for more details. Its redshift is z = 4.41 and this object seems to be a massive elliptical galaxy with a radio-loud supermassive black hole at its core.
In both instances, we are seeing that the universe formed very active galaxy- sized objects within 1 billion years after the Big Bang which for many variants on Big Bang cosmology, is quite a rapid evolutionary feat. Only 'Cold Dark Matter' versions of Big Bang cosmology seem able to create such objects so quickly, and the search for the earliest formation era for galaxies is now the holy grail of observational cosmology.