Why are soldermasks not allowed to be negative?

I cannot answer your question directly but I suspect that the assumption is that solder mask on pads is not a good thing. I'm surprised that manufacturers are recommending this. I'm curious why they would not just recommend a smaller pad since that is effectively what happens when you soldermask over a pad. @JP, you committed the change that prevents negative solder masks. Care to comment as to why you did this?

There are good reasons for having negative solder mask enlargement. In
the industry it is known as 'solder mask defined pad' (usually we use
'copper defined pad's) and it can be used obviously only on
high-performance direct print boards (otherwise the typical 0.2mm mask
registration would be a problem). Also it's only for bottom terminated
parts (BGAs and to a less extent QFNs). If
you specify 0 solder mask (assuming perfect registration) the copper
wall tends to interfere with the mask aperture; also in ultra fine pitch
assembly with the 'usual' enlargement the solder mask would disappear
raising the issue of solder bridging. Reducing the pads would leave too
few copper for adhesion (we are talking about 0.4mm pitches, here, not
coarse boards!). In other words you can avoid solder mask ganging and
having other interesting effects (like putting a 0.1mm trace between the
pads!).

Another way more common use is for heatsinking pads (like in PLCC LEDs)
where you want to specify in the footprint some copper area which
shouldn't be soldered on (later you may want to uncover some of it
anyway since no mask is better for dissipation :D)

Both of these situation can be usefully 'simulated' in pcbnew using
different 'pads' for copper, mask and paste (the same technique for
doing channeled paste areas)

As for the restriction I have no idea. It wouldn't be the first time a
check is added to kicad simply because it made sense to the programmer
(like some 'arbitrary' size limits in the past) but has no technical
need to exist (like some need in the DRC or filling issues or such)

Negative solder mask clearance sounds a bit strange to me. I'll be happy to see some reference.

But the major reason it is not allowed is (AFAIK) the fact the negative solder mask clearance was never tested (although negative clearance is allowed and even mandatory for solder paste), at least not by me.

There was no test for non acceptable values (like negative clearance bigger than pad size) in dialog.

Accepting negative solder mask clearance means also to be sure it works (for any pad shape), and some tests made to avoid non acceptable values.

Well, it is important that input data be validated and that testing be consistent. However, this is removing a functional capability as it had been working OK for several years. It may be an accidental feature for you, but a common one used often for BGA or most modern circuit layouts.

I would assume that the negative pad validation error would involve a bit more math ie. -(smallest identifiable pad side/2). I believe DipTrace does this as its check. Is there also a maximum solder mask swell in place?

I would also assume the paste should have an identical maximum shrink using the same math as above. ie. -(smallest identifiable pad side/2). Those would be good checks. Anything complicated/special outside the circle/rectangle pad design system would need to be drawn anyway.

Kicad has worked well for most of my soldermask defined footprints before this change and the results were correct enough. There is still a workaround as the global footprint swell can be set negative so it sorta still works for new designs.

NSMD(Non-Solder Mask Defined) Examples:
See Drawing RGT0016C - http://www.ti.com/product/TPS62130/datasheet/mechanical_packaging_and_orderable_information#impnote
Section 2.1 and others in PowerPad guidelines - http://www.ti.com/lit/an/sloa120/sloa120.pdf
Again starting as Section 2.1 in general QFN guidelines - http://www.ti.com/lit/an/sloa122/sloa122.pdf

Search NSMD in Design and Manufacturing of https://en.wikipedia.org/wiki/Quad_Flat_No-leads_package

I am pretty sure if I asked nicely, I could get a manufacturer FAE to provide a write up on this as an industry standard. I know I could get an FAE from Arrow, Future, or another distributor at a minimum. It is very common to increase thermal capabilities of a pad while narrowing or centering the solder alignment of a smaller foot. QFN has probably the most examples, but some BGA manufacturers will recommend it for thermal capabilities and/or small trace routing improvements as mentioned in other comments.

Thanks jean-pierre charras, that solved my question.