She was asked to leave her job as Georgia's director of rural development in July after comments she made in March were misconstrued as racist. She has since received numerous apologies from the administration, including from President Barack Obama himself, and Vilsack asked her to return. But she said at the news conference with a clearly disappointed Vilsack that she did not think she could say yes to a job "at this point, with all that has happened."
Vilsack said she may work with the department in a consulting capacity in the future to help improve outreach to minorities.
"I look forward to some type of relationship with the department in the future," said Sherrod, who is black. "We do need to work on the issues of discrimination and race in this country."
Vilsack had asked her to become the deputy director of the Office of Advocacy and Outreach, a new position designed to bolster the department's shaky record on civil rights. He had also given her a chance to return to her former job. Both of them said Tuesday Sherrod may return to the department as a consultant once an ongoing review of the department's efforts on race issues is completed.
"I think I can be helpful to him and the department if I just take a little break and look at how I can be more helpful in the future," Sherrod said.
The two appeared friendly as Vilsack expressed his regret that Sherrod wouldn't return to USDA. He put his arm around her at the news conference and said he leaned on her hard to return and help the department with racial issues.
"I did my best, I think it's fair to say," he said. "There's no one in the country better suited to assist us in that effort than Shirley."
He said a consulting job may work better for Sherrod, who was concerned about assuming administrative duties like budgeting. She said she was reluctant to be responsible for the weighty duties of the position she was offered.
"There are new processes is in place, and I hope that works," she said. "I don't want to be the one to test it."
Sherrod was forced to resign after conservative blogger Andrew Breitbart posted an edited version of a March speech in which said she was initially reluctant to help a white farmer save his farm more than two decades ago, long before she worked for USDA. Vilsack and others, including the NAACP, condemned the remarks before grasping the full context of her speech, which was meant as a lesson in racial healing.
The incident proved embarrassing for the Obama administration, and Obama called her personally to express his regret. The NAACP also apologized for its reaction.
Sherrod, who said she has gotten thousands of pieces of mail supporting her, repeated Tuesday that she plans to sue Breitbart. But she declined further questions on the subject.
As he had in the past, Vilsack said he took complete responsibility for Sherrod's ouster. Though the department had conversations with the White House at the time and Sherrod said she was told it was the White House who wanted her gone, Vilsack has said the decision was his.
"I know that I disappointed the president, I disappointed this administration, I disappointed the country, I disappointed Shirley," he said. "I have to live with that ... Maybe, just maybe, this is an opportunity for the country to have the kind of conversation Shirley thinks we ought to have."
Vilsack said he talked to Sherrod for an hour and a half Tuesday morning. The two discussed a settlement for black farmers who have been victims of racism pending in the Senate and other civil rights issues facing the department.
The USDA has a long history of discrimination of black farmers who sought out loans and other aid, and the government this year settled a second round of damages stemming from a class-action lawsuit originally settled in 1999.
The department also released a list of recommendations stemming from an internal investigation into the Sherrod controversy on Tuesday. Vilsack blogged on the USDA website that "we need to improve protocols for internal communications at the department, and create a set of safeguards to avoid the sort of hasty action which led to the mishandling of the matter with Mrs. Sherrod."