Aaah, I thought that option only applied to how you get notified, not what happens in a thread. I changed it to daily notification because that's what I wanted, but with notifications enabled I apparently get subscribed to everything.
Those really should be two separate buttons :/
Thanks.