First thing is to write some D3D9 code that allows you to switch from the old Fixed Pipeline and your own Shaders, using the same parameters. Doing so, you will easily find any behavior differences in light calculations. You can read more about how D3D9 Fixed Pipeline calculates lighting in this page.
When writing shaders, people tend to calculate the diffuse contribution like:
Out.Color = (materialAmbient * lightAmbient) + (materialDiffuse * lightDiffuse * dot(Normal, L));Where L is the vector from the vertex position (in world coordinates) to the light.
Apart from not doing any specular or emissive calculations (which could not be necessary in many cases, depending on your scenario), there are several mistakes in that approach:
1.- You don’t want the dot to return negative values, because it will black out colors wrongly. So, you need to clamp it to the 0..1 range, using the saturate operator: saturate(dot(Normal, L))
2.- In order to get the same results as the Fixed Pipeline, you should include Attenuation calculations, because they modify the intensity of light with the distance between the point being lit and the light source. Attenuation (as opposed to what its name suggests), not only attenuates light, but also can increase intensity in some circumstances. (See below how to properly calculate attenuation factors)
3.- Once you are calculating attenuation, you should remove the materialDiffuse factor from the previous equation, as you don’t want it to be attenuated too. You will apply it later, when the entire lighting contribution is properly calculated and attenuated.
Keeping those 3 things in mind, the final calculation in a vertex shader would be:
ComparisonFirst image is the Programmable version. You can slightly tell it by the reflections on the windows.
Second image is the Fixed Pipeline version (no real time reflections on windows):