When Kiryuu shows up at the Doujou and led to duel its star pupil, Ueda Ryouhei 植田, Touji says, "Let's see some of that skill you used in the Sekigahara." This is a hint that he knows that Kiryuu is really Musashi, so they talk about it outside.
It turns out that Touji does, indeed, know Musashi's true identity. He was there, that night, apparently, when Musashi was tricked into killing Tokugawa Ieyasu's 徳川家康 eldest living son, Yuuki Hideyasu 結城秀康, but he mentions that he knew something was "fishy" and fled before the operation. This implies that the other soldiers, who were just supposed to distract the troops at the entrance to Yuuki's residence, were killed, because otherwise, he would not have to explain how he survived that night.
When the rumored duel approaches, Yoshioka Seijuurou says that he refuses to fight Musashi. This could partly be because dueling was actually against shogunal law, and it could also be that Yoshioka feels a criminal like Musashi isn't worth his time.
However, Touji knows that Yoshioka has agreed to meet with Musashi at a place named earlier for the duel. So, he tells Musashi to go there, and that Musashi will find out who the impostor is.
When Musashi reaches the spot, he finds only Seijuurou. "So, you really are Miyamoto Musashi," Seijuurou tells him. It seems Touji manufactured the plot and has set up the duel--there was no impostor. It was all a plot so that Seijuurou's unskilled brother would be left alone, and so that someone would betray the Yoshioka family.
Touji's plan was that Seijuurou would die by Musashi's hands, and then he could kill Denshichirou, and he would become head of the Yoshioka Doujou. Ueda and others seem to be following Touji's lead, and try to hold Musashi back after the duel.
But Touji's plans are foiled, of course, by Musashi stepping up and taking his place in history. He ends up defeated at Musashi's hands.
Whether or not Gion Touji is a real person is questionable. He appears in Yoshikawa Eiji's book Miyamoto Musashi (simply "Musashi" in the U.S.), but not too much in actual historical accounts. It has been said that he was a great student of the Yoshikawa school who used the famous "tsubamegaeshi" attack.
The manga "Vagabond" also features Touji as a fierce, ambitious warrior."