Two of the better known sites are Grant's Tomb at 122nd street and Riverside Drive and Hamilton Grange which was Alexander Hamilton's home located at 141st between Convent Ave. and St. Nicholas Ave. You also have the Hispanic Society at 155th and B'way, a small, interesting museum on Spanish arts and culture. It's in what would be considered Spanish Harlem rather than Harlem proper. The house that was Langston Hughes' residence is at 20 East 127th Street. It's a landmark but is a private residence and you can't go in. There's a great outdoor statue of Duke Ellington at 110th st. and 5th avenue worth taking a look at. I'm sure there are several historic churches as well and Bill Clinton's office is on 125th st. between Lenox and 5th avenues but it's just a large office building and again no admittance to his specific office. That's about all I can think of. Harlem is largely residential so there aren't a lot of actual historic sites per se.
There are several organizations that offer cultural and historic tours of Harlem including Harlem Heritage. I don't have any feedback on whether their tours are any good.
http://www.harlemheritage.com/ Harlem has also gotten a lot safer in the past 10 years but I would still exercise some caution while walking around. Have fun.
