Here's how libertarian philosopher Jan Narveson explains it:

According to Contractarianism, the principles of morals are a sort of agreement, or "in a sense" an agreement. ...

Its idea is that the principles of morals are a kind of grand agreement. In theoretical principle, what makes it an "agreement" is that its rules are, at least implicitly, "iffy": each of us is to treat each of the others in certain ways provided that they do likewise. If they don't, the deal is off. And if it's off, the idea is, then we are both worse off than if it were on. Mutuality, reciprocity, is the byword. ...

The contractarian view enters the picture by proposing that the right set of principles to play this role is the set such that everyone, looking at those proposed principles from his or her point of view ex ante, can see that he or she will do better if everyone, including himself or herself complies with those principles than if there are none or some other set.

The kicker is "everyone including himself or herself." ... Now, at the point when it does overrule you, it looks as though morals is disadvantageous to you. On the other hand, though, when it overrules other people, it becomes quite advantageous to you. Morals is to the advantage of people other than the agent, typically; but of course, every single one of us is a person "other than the agent," all the time...

Let's take the example of one of the strongest and most fundamental of all moral rules - the rule against killing innocent people. ... Now and again, perhaps, it would be useful to you to kill somebody else. Suppose that morals says you cannot do this. On that occasion, it deprives you of a possible benefit. Meanwhile, however, suppose it was to somebody else's advantage to kill you. Morals deprives that person of the benefit of killing you; but of course, that means it provides you with the benefit of not being killed. It is not too much to say that it provides you with life. - Jan Narveson, The Contractarian Theory of Morals: Frequently Asked Questions (FAQ)