In visual cryptography (VC) for grayscale image, size reduction leads to bad perceptual quality to the reconstructed secret image. To improve the quality, the current efforts are limited to the design of VC algorithm for binary image, and measuring the quality with metrics that are not directly related to how the human visual system (HVS) perceives halftone images. We propose an analysis-by-synthesis (AbS) framework to integrate the halftoning process and the VC encoding: the secret pixel/block is reconstructed from the shares in the encoder and the error between the reconstructed secret and the original secret images is fed back and compensated concurrently by the error diffusion process. In doing so, the error between the reconstructed secret and original secret is pushed to high frequency band, thus producing visually pleasing reconstructed secret image. This framework is simple and flexible in that it can be combined with many existing size-invariant VC algorithms, including probabilistic VC, random grid VC and vector/block VC. More importantly, it is proved that this AbS framework is as secure as the traditional VC algorithms. Experimental results demonstrate the effectiveness of the proposed AbS framework.