Eggplant?!

This is our second year trying to grow eggplant and our second year of failing.  I don't have any idea why - I haven't really researched it or anything.  The plant looks fine, but we haven't ever gotten any fruit (I just found out that eggplant is technically a berry!).

I was taking out our tomato plants yesterday, and I thought I might as well take out the eggplant too.  Then I saw that little baby eggplants were popping up.  I think I saw four.  Should I keep the plant in?  Shouldn't this have happened in the summer?  I'm still such a novice at gardening.  For now, I'm just going to leave it and see what happens.