Andy & Me was started in the summer of 2002 by guitarist and lead vocalist Aaron Guzik and guitarist/vocalist Andy Schulz (Baby vs. Giant, Forget Flying). A year later, the band recorded its first full length, backed by bassist Ethan Margolis (Forget Flying) and drummer Geoff Guillard. Where the Sun Don't Shine was released later that autumn. Around this time, the band frequently performed with Guzik, Schulz, Guillard, and Guillard's older brother, Jeremy, on bass.