Joskun, I hope you don't mind--I want you to add to this too, and definitely still put up that thread about how to find those harder songs. This is just a list of sites that I personally use when I'm looking for songs.
Please refer to these sites before requesting, because I can almost assure you that one of these sites has it.
As I find more useful sites or as people tell me about more, I'll add to this list.
If you still can't find what you're looking for, go ahead and make you're request in my request thread or check my Track List.
Anime and Video Game sites
Anime Yume - Not the biggest assembly of anime songs, but this has a lot of variety. Lots of anime titles. A good place to find opening and ending themes to popular anime.
Starless Night - A site dedicated to Japanese Singers. There's a lot of nice J-Pop artists here. Big thanks to Arinayed for informing everyone of this site. ^^
KHInsider - My favorite. This site's practically got it all--from video game music to anime music.
KHInsider seems to have issues with downloads now... Lame.
AnimeOST - An amazing site packed full of anime OSTs and the like. Compliments of Joskun.
Did this site shut down or does it just no longer work on my computer?
Blue Laguna - A lot of Video game albums. Great for Final Fantasy, the 'Tales of' series, and Kingdom Hearts, among many other games.
Gendou's Anime Music - It doesn't have everything, but it does have some pretty hard to find songs.
Anime Suki Forums (Vocaloid 2 Thread) - Do you like Vocaloid? Then just root through the 70+ pages of this thread of a forum (no registration required). There are 100's of links to Vocaloid songs posted by people here.
Torrent Sites (For Japanese and Anime things)
Tokyo Toshokan - A torrent site with LOTS of Japanese albums. If you need a torrent extractor, I personally like uTorrent but of course there's hundreds of others.
Nyaa Torrents - A Japanese themed torrent site. I downloaded the famous 'Vocaloid' program from this site. I'm sure it has other awesome things as well.
Youtube audio/video ripping and Conversion sites
flv2mp3 - This site can extract audio from Youtube videos and allow you to download it. Of all the Youtube online audio capture sites, this one has always been the most reliable for me.
Vixy - An alternative to flv2mp3. Also allows you to download the Youtube video itself in a handful of different formats.
All Around Downloading Sites
mp3dl - A general mp3 downloading site... It's had a lot of random Japanese songs that I've searched for before. I use it a lot for J-Pop and such.
4Shared - A general downloading site. I find a lot of J-pop/J-rock albums here in .rar or .zip form. It has other things too, but it's really a great place to find music.
Beemp3 - This site has a bit of everything, music-wise. I found a lot of J-Pop here.
eSnips - You have to make an account (it's free) but this site has quite a number of different things. I found all my Higurashi songs here. Not all people who upload songs have a download link though. :[
Flash Flash Revolution - No, really! In the Simfile database, you can search for specific songs, then, download them to get the audio. Flash Flash has saved my butt while looking for songs a number of times. It really does have a lot of good songs, 'specially those faster, upbeat, techno-y ones.