Am I having an affair?

I have been friends a man for four years now.When we first met we were both married.I am now divorced.Our relationship grew from friends to more last year when we began kissing but no further.I moved out of the area and now have returned.We got in contact again and were right back where we left it.He is still married .We arent ually active but talk everyday several times and I have no idea if we are real good friends or more could come from this.He has said he is stuck and this whole thing is outside the box for him.He loves being around me and always misses me and I miss him.Is this a full blown affair ive somehow created all on my own here?I love him but would hate to be the one to make a good man tainted by my temptations or is it vice versa?