How the rates are calculated vary. Some charge per pin or per net (or a combination thereof). Others charge per hour. Still some simply quote a fixed price for a whole job. I will say this, however. If it were my job and a layout person wanted to charge per unit area per layer, I would find another designer. Such a fee structure would be fine for the fab shop, but I don't want the designer to have any incentive to make the board any bigger (or with more layers) than is necessary. To do so will only increase the fabrication costs.