You're right, there are definitely a lot of towers going on.
I don't think the Tower of Ghenjei is a portal to Seanchan, though. I think it's a portal to the world of the Aelfinn and Eelfinn which is in a different dimension/an alternate reality (such as in the Great Hunt)?
Well, I thought it had to do with the Malkier towers as well, but the Glossary and the Companion very specifically state that the Towers of Midnight are in Seanchan.
Welp, was gone for the weekend hoping someone would have some brilliant answer for me when I came back, but no luck.
All I know is it's mentioned in the Prophecies of the Shadow:
In that day, when the One-Eyed Fool travels the halls of mourning [this is obvs. Mat], and the First Among Vermin...