Women have a limited number of follicles in their ovaries, and eventually all of their follicles are either used to release an egg or die off in the process of trying to ovulate. We're born with millions, but hundreds try to mature every month, and the majority will die off. Egg quality also declines with age and relative health.
Men produce sperm continually, but male fertility also begins to decline in their 30s. Although men don't have a clear cut-off for when their bodies stop producing sperm, sperm count and quality declines with both age and relative health. A 45 year old man takes much longer to successfully conceive on average than a 25 year old man does, and is much more likely to have developed some form of male-factor infertility.