Federal Plaza is really just a governmental building across Lafayette from two little parks. It went up in the 60's and was named after a famous New York Senator (Javits). This building houses the US International Trade Court and the INS. Neither of those mean anything to me but this building means a lot to many of my friends. And, none of them have particularly fond thoughts of the ole federal plaza. Basically, if you aren't an American Citizen and you're living in New York, you'll probably have to spend a lot of time with these people -- begging, crying, arguing etc (if you're anything like my friends) because they are the people who grant you Visas, Naturalization, etc. And, apparently, they are a really tough bunch.
There's nothing really to this area but the building as the parks are considered Lafayette street. I guess it's pretty important, though, if you're moving here.