You've made a thoughtful point--this is exactly how physicists introduce the so-called "Twin paradox" when they're teaching relativity. But it's not really a paradox.
The short answer is this: In order for the two observers to compare the elapsed times on their clocks, ONE of them has to turn around and come back to the starting point. That means HE has to CHANGE his inertial reference frame, while the other observer does not. There are therefore THREE inertial reference frames involved. One of them is experienced by observer "A", and the other TWO are experienced by observer "B" (the one who turned around). This breaks the symmetry, and ends up causing observer "B" to experience less total time between the instant they separate and the instant they meet again.
And interestingly enough EACH observer (in any of the 3 reference frames) STILL concludes that the OTHER observer's clock is running too slow. Let's do a thought experiment with real numbers to see how that works.
Imagine that an astronaut (observer "B") is leaving his twin on earth (observer "A") at 99% of lightspeed. At that speed, the time dilation factor is approximately 7.
Now, let's further say that each observer has agreed to transmit radio pulses to the other during the whole trip, at intervals of 1 second (each by his own personal clock). Let's calculate how often each observer RECEIVES a pulse from the other.
While the observers are separating, even if there were no time dilation, you'd still expect each observer to receive the pulses at intervals longer than 1 second, because each pulse has farther to travel than the one before. As an analogy, imagine that a pitching machine is throwing baseballs at you, one per second, at 100 mph; but at the same time, you're driving AWAY from the pitching machine at 99 mph. With a little algebra, you can see that the baseballs would hit your car at a rate of (about) 1 every 2 seconds.
In the same way, if "A" and "B" are separating at .99c (99% of lightspeed), you would expect each to receive a radio pulse about every 2 seconds, IF there were no time dilation.
But in fact, each receives a radio pulse about every FOURTEEN seconds. Each observer concludes that the other person's clock must be ticking too slow by a factor of about 7.
Conversely, when "A" and "B" are APPROACHING each other, each should receive the pulses at a much higher rate, since they're traveling "into" the stream of approaching pulses. The "baseball" analogy suggests that the pulses should arrive about every 0.01 seconds (100 pulses per second); but instead they arrive slower than that; about every 0.07 seconds (about 14 per second). Thus, each observer still concludes that the OTHER guys clock is still running 7 times too slow.
But the crucial thing is, the guy who turns around receives more TOTAL pulses than the guy who stays put. Let's analyze that.
Let's say the astronaut has agreed to go to a star that's 7 light years away, and then will "instantly" turn around and come back (so that at any given instant he's travelling at 0.99c, in one direction or the other.
In the astronaut's time frame, the first leg of the trip takes only about 1 year. (He explains that by saying the star is "really" only 1 light year away--this is a consequence of relativistic length contraction.)
During this leg, the astronaut is receive pulses about every 14 seconds, so after 1 year he has received about 2.25 million pulses.
Now he reverses direction, and is suddenly receiving the pulses every 0.07 seconds. So during his return trip (which also feels like 1 year), he receives about 451 million pulses.
So by the time he lands back on earth, he's received 453 million pulses from earth. Since those were "sent" at 1-second intervals, that means about 453 million seconds (14 years) have elapsed on earth.
If you do a similar analysis for the pulses received on earth, you find something different. The earthlings receive 32 million "slow" pulses (which takes about 14 years) and then 32 million "fast" pulses (which takes about 25 days). The astronaut transmitted 64 million (2 years' worth) of pulses in 14 earth years. So the trip only took 2 years in the astronaut's view.
Note that BOTH observers conclude (from measuring the rates of both the "fast" pulses and the "slow" pulses) that the OTHER GUY'S CLOCK IS 7 TIMES TOO SLOW AT ALL TIMES. But the number of "fast" pulses vs. "slow" pulses received is greatly imbalanced between one observer and the other; so that the total number of pulses received by each is very different; thus the two report very different elapsed times when they meet again.