At the Battle of Port Royal later that year, troops under his command at Fort Beauregard and Fort Walker came under attack by ships of the Union Navy, including the USS Pocahontas, commanded by his brother, Percival Drayton. Thomas Drayton's son, Lieutenant William Drayton, also fought with the Confederates in defense of the forts. After a lengthy bombardment, both forts fell to the Union attackers, who subsequently occupied much of the region. They gained their first deepwater port in coastal Carolina.
In 1862, Drayton was assigned command of an infantry brigade composed of the 15th South Carolina Infantry, the 3d Battalion S.C. Inf. and three Georgia infantry regiments, the 50th and 51st and Phillips' Georgia Legion, which became part of the Right Wing of the Army of Northern Virginia under Lt. Gen. James Longstreet. Drayton's Brigade fought at the Second Battle of Manassas. Defending Fox's Gap at the Battle of South Mountain, Drayton suffered high casualties. His much-depleted brigade also saw considerable action at Sharpsburg.
His tactical abilities were at times questioned by his superiors, and he was finally removed from command. He was transferred to the Western Theater to command a brigade in Sterling Price's army in August 1863. During the final two years of the war, he mainly performed administrative duties in the Trans-Mississippi Theater, although he did briefly command a division in early 1864.