Adding Image Caption to Image Field or Wysiwyg Inserted Image for Drupal 7

I recently was asked to add an image caption to image field on a content type.  I added a field image caption and included it with Alt and Title field.  However, the image field was working fine with Wysiwyg inserted image but not working with the image field.
I started to investigate the problem and I came to conclusion.  But to understand the problem we have to start from the beginning.