It means that people have the right to do what they want in their own home, and that other people or the state have no right to interfere in people's private lives.
Sir Edward Coke, in The Institutes of the Laws of England, wrote in 1628: “For a man’s house is his castle, et domus sua cuique est tutissimum refugium [and each man’s home is his safest refuge].”