In the aftermath of the shootings in Charleston, S.C., controversy arose over the Confederate flag. The racist shooter had posed with one, and to many African-Americans, it is a painful reminder of slavery. It seemed inappropriate for it to be flying from the statehouse of South Carolina.

Although I am proud of my ancestors who fought for the Confederacy, I have no desire to fly it, either. Give me the Stars and Stripes, not the Stars and Bars.

But while I agree with removing the flag from the South Carolina statehouse, we would be going way too far if we started removing all monuments and memorials to Confederate soldiers.

Most Confederate soldiers were not even slave owners, and we should strongly object to men like Robert E. Lee and Stonewall Jackson being labeled as racists. No one who truly understands these men would want to do that.

Fact: Neither man was a racist.

Robert E. Lee, in a letter dated Dec. 27, 1856, referred to slavery as “a moral and political evil.” Like most Confederate soldiers, Lee did not see himself as fighting to defend slavery.

In the same 1856 letter, Lee went on to talk about the evil that slavery did not only to the slaves themselves but also to the whites who owned them.

One Sunday after the war, as the congregation gathered at St. Paul’s Episcopal Church in Richmond, a black man came forward for communion. People did not know what to do, until a white man got up and knelt beside him. That man was Robert E. Lee.

Before the war, Stonewall Jackson was a professor at Virginia Military Institute, and everyone in Lexington knew that Jackson was no racist. In fact, he had taken a great deal of criticism from whites for starting and teaching a very popular Sunday School class for black people.

In this large and well-attended class on Sunday afternoons, Jackson shared the message of Christ’s love and also taught people to read, because he wanted them to be able to read the Bible. He was risking arrest by doing this.

In 1905, Pastor L.L. Downing, an African-American man, had a memorial stained glass window in honor of Jackson installed in his Roanoke church. Why? Because Stonewall Jackson had taught his parents to read and write.

Fact: Neither man was an advocate of secession or war.

Both were graduates of West Point, and both had bravely fought for the United States in the Mexican War. As combat veterans, they knew what battle was all about, and both were terribly upset by the move toward secession and war.

As war fever gripped the South and many celebrated, Jackson said, “It is painful to discover with what unconcern they speak of war and threaten it. They do not know its horrors. I have seen enough of it to make me look upon it as the sum of all evils.”

Lee said this of secession: “I can see no greater calamity for the country than the dissolution of the Union.”

Of course, the war came. Jackson died during the conflict. Lee survived and did everything he could to facilitate healing and unity in our country.

Let’s seek to do the same. That happens as we learn to love one another on a personal level, regardless of color.

Dr. Thurman R. Hayes is senior pastor of First Baptist Church of Suffolk. Follow him on Twitter at @ThurmanHayesJr.