See the Time Magazine article by Massimo Calabresi:
Colin Powell: Planning for an Exit.

...he has privately grown more frustrated, and now, sources close to Powell tell Time, he has a firm plan for his exit: he will step down at the end of President Bush’s current term. “He will have done a yeoman’s job of contributing over the four years,” says a close aide. “But that’s enough.” The aide says Powell’s view of the matter is, “I did what my heart told me to do. I got (Bush) here and set him up. I did the best I could do.” If Bush wins a second term, only the imminence of a major diplomatic victory—in the Middle East, for example—could induce him to stay a short while longer. By the same token, the aide stresses that Powell is determined to serve out the entire term—even if the U.S. launches an invasion of Iraq, which Powell has fought to delay or derail.