Venus is brighter than Mercury because it is much larger, because it has a reflective atmosphere, and because it makes close approaches to Earth.
First, Mercury is about 3,100 miles in diameter, compared with about 7,700 miles for Venus.
Second, there is no atmosphere surrounding Mercury, while Venus has a thick atmosphere, composed chiefly of clouds of carbon dioxide. These clouds have a very high reflectivity, causing Venus to appear to shine with great brilliance.
Third, Venus can approach the Earth much more closely than Mercury. In fact, Venus periodically comes closer to Earth than any other known celestial body, except for the Moon, as close as 25 million miles. All of these things allow Venus to shine more brightly than not just Mercury, but than any of the other planets in our solar system.
Because Mercury and Venus are closer to the Sun than Earth is, they can show phases like those of the Moon. When Mercury shines at its brightest, it appears through a telescope as a gibbous or nearly full phase.
When Venus appears to shine at its most brilliant, it appears in telescopes not as a nearly full phase, but as a crescent.