Excellent question and nicely set up but in the very setting up of the question you have highlighted why it is so hard for most people to make sense of special relativity.
Have you ever wondered why all these thought experiments have explosions? It is because they are instantaneous. So think about this; After the explosion has happened, where did it take place? Bear in mind that space has no preferred reference frame.
So the deal is; I am in empty space, I am inertial and I think I am stationary and I saw an explosion 1 km away in some direction with respect to my personal reference frame.
You are inertial and moving at half the speed of light with respect to me but you consider yourself to be stationary. As you pass me, we match up our direction reference frames just as we see the flash.
One minute later, where will you say the flash occurred? Remember, you think you are stationary and when you saw the flash, it was 1km away in a certain direction that we both agreed on. So you will point in that direction and claim it was 1km in that direction.
But clearly, 1 minute later we are millions of miles apart and yet we both think the flash happened 1 km away from our own current location. You see, when they said there was no reference frame in space, they really meant it. There is no true location where the flash occurred.
And thus you can't move relative to an instantaneous event. You can only move with respect to a persistent object.
I'm not going through the full answer to your question, my goal here is just to impress on you that "space has no reference frame" is a far more tricky idea than you realised. And part of the reason is that when you visualise an explosion or draw a cross on a page, you have imposed a personal reference frame. And it is easy to forget that the page (mental or paper) on which you place your explosion is not "The Truth", it is your observation of events in your frame.
Here's a thing; Imagine in space there is a long straight railway track and a train riding along the rails at a constant velocity, all inertial. Is the train travelling along a stationary track or is the track sliding under a stationary train?
Answer: Either. It depends on your reference frame. There is no preferred answer. To grasp Special Relativity you have to be able to see it both ways and see that each is equally valid at the same time
The front wheel of the train crushes a bug and leaves a mark on the line, 1 minute later, where in space is the place the bug got crushed? Is it the bottom of the front wheel or the mark on the line?
Answer: Either. There is no right answer unless you give a reference frame, then it could be the wheel or the mark on the line, or indeed some other point with respect to a third inertial observer.