Your primary error is that the date string needs to have double quotes around it, like this:

if date paid ="" or date paid >@d("05/10/2012") and current dues paid <170 then late fee =5 else late fee =0  

The only time you can omit those is if the date string is in the same format that Sesame uses internally, which is YYYY/MM/DD. So, this will also work:

if date paid ="" or date paid >@d(2012/05/10) and current dues paid <170 then late fee =5 else late fee =0 

But, since you have 3 conditions mixed with "or" and "and", you should also be grouping two of them inside parentheses to ensure you are telling Sesame exactly what you want. So, depending on exactly what you intend, you may want to use something like the this:

if (date paid ="" or date paid >@d("05/10/2012")) and current dues paid <170 then late fee =5 else late fee =0