If your monitor/video card can display 1024x768 in Windoze, then you should also be able to display this resolution in Linux. Is it possible that some of the configuration data you have entered is not correct? For example, here is the monitor section for my Samsung LCD monitor (very low-end by today's standards ...):
Section "Monitor"
Identifier "Samsung SyncMaster 570S TFT"
# HorizSync is in kHz unless units are specified.
# HorizSync may be a comma separated list of discrete values, or a
# comma separated list of ranges of values.
# NOTE: THE VALUES HERE ARE EXAMPLES ONLY. REFER TO YOUR MONITOR'S
# USER MANUAL FOR THE CORRECT NUMBERS.
HorizSync 30-61
# HorizSync 30-64 # multisync
# HorizSync 31.5, 35.2 # multiple fixed sync frequencies
# HorizSync 15-25, 30-50 # multiple ranges of sync frequencies
# VertRefresh is in Hz unless units are specified.
# VertRefresh may be a comma separated list of discrete values, or a
# comma separated list of ranges of values.
# NOTE: THE VALUES HERE ARE EXAMPLES ONLY. REFER TO YOUR MONITOR'S
# USER MANUAL FOR THE CORRECT NUMBERS.
VertRefresh 50-75
EndSection
You might want to double-check your entries against the monitor manual. You may also be able to get some information about the characteristics of your video card/monitor from the device manager in Windoze. Again, specific information about your monitor would be helpful (manufacturer/model number).
Hope this helps.