Windows and doors are a common source of temperature and energy loss depending on what type you have. The newest, double pane, low emission, argon vinyl windows are very good. Be sure that they are clean and locked tight. They will leak if they aren't locked!
Older wood or metal frame systems are not as efficient and you need to be sure they are closed and latched firmly. Storm windows over double hung or sash weighted frames are likely to be a source of energy loss. Again make sure everything is clean and closes all the way. If you can see daylight you have a problem, often there is something stuck or out of line. Any of these window systems benefit from being exercised, raise and lower each one and look carefully to be sure they close all the way. Old fashion sash weighted windows leak around the pulley. There are caps available that snap over the hole and seal them up. Check the caulking on the storm windows for gaps and voids.
Weather stripping on exterior doors is another source of "voluntary" air supply. The seals should be clean, flexible and contact the door all the way around.
Another common pathway for air leaking into or out of your home is utility penetrations. Inspect your outside walls for gaps around the electric, HVAC, cable TV and water services where they go through into your home. A bead of caulking or some closed cell spray foam will seal these up. Look for penetrations up through your floor from the basement or crawl-space, if you have one and down through the ceiling from the attic. Remember that attics and crawl-spaces are usually colder or hotter than your living space and you don't want that air in your house. The electrical wires, water lines and duct boot penetrations are generally a pathway for leaks and energy loss. Ceiling can lights that stick through the ceiling into your attic are notorious leakers.
It is important to understand that as you air seal you are reducing the amount of "voluntary" air changes in your home. You are slowing down unwanted air infiltration and becoming more efficient. This effects the "V" or ventilation part of HVAC that is often neglected. We call it ACH (Air Changes per Hour) and the term refers to the amount of fresh air you get circulating in your living space. This is the "Too Tight" part. The idea is to control the air flow rather than get your air supply through energy robbing leaks. Once the house is tight you will want to install mechanical ventilation. This is a filtered source of fresh air that connects to your current system or can be a stand alone supply. It can be as simple as an air duct with a filter connected to your HVAC system's return or as sophisticated as an Energy Recovery Ventilator that operates on its own control sensor. Both are adequate for ACH, the ERV is more efficient.