One of the problems of libertarianism is that its principles, stretched beyond proportion and common sense, can twist around and defeat the philosophy's stated objectives. A belief in tolerating any and all consensual relationships, for example, can shade into a willingness to tolerate nonconsensual ones. And I believe Jacob Sullum is treading into some pretty treacherous territory in shrugging off a woman's complaint that her BDSM relationship turned into actual slavery. These things can happen, after all.
An even more egregious example of libertarianism turned against itself (and an argument Sullum does not make) is the view that people have a right to sell themselves into slavery (that is, real slavery, which they can't get out of by saying so). There would be less freedom in a world that tolerates slavery than in one that prohibits people from making that particular choice. Just as there would be a distinct loss of freedom in a world where your neighbors have a property right to wall you up on your property and starve you to death.