The House of the Graffito is a Hadrianic medianum-apartment, forming part of the Garden Houses. It faces a small square in front of the House of the Painted Vaults (III,V,1), and is back-to-back with the House of the Yellow Walls (III,IX,12). The building is smaller than its neighbour however. It is entered from the south, through a door flanked by brick pilasters. Next to the vestibule is an inner staircase. The medianum is flanked by a small and a large representative room. The east wall of these three rooms is pierced by many windows. To the west of the medianum are two small rooms. The ceiling was at a height of 3.95 metres. There is a threshold with two pivot-holes between the vestibule and the medianum. This opens up the possibility that from some point in time the staircase led to independent upper apartments. The thickness of the walls is 60 centimetres, allowing for several (perhaps three) upper floors.

There are some remains of red and yellow paint and of black-and-white mosaics. It is not known from which graffito the building received its name. Apparently it was taken to the storage rooms.