I have done some research. No, you can't say he wasn't born this time of year. We just don't know. Early Christians didn't even like celebrating birthdays and while they speculated on the date of His birth they didn't celebrate it. The earliest date proffered was in the Spring around the time of Passover. Some link it to the Fall at the time of the Fall Feasts in the Bible. Because of the belief in a thing called Integral age many believed he was born and died on the same date which is also believed for Moses and King David. We know his death was at Passover so that must be the date he was born. Then it was posited that his conception was in the Spring so his birth would have been in late December. The earliest mention I have seen of this was by Julius Africanus who believed his conception was around Passover and wrote this in the early 200's.
There is no record of celebrating the birth until the time of Constantine. This just means early Christians speculated on the date of his birth but really didn't put a lot of import into setting aside a date to celebrate. I believe it is total nonsense to think this date was picked to correlate with the celebration of Sol Invictus, the sun god. Certainly Aurelian mentions December 25 as a date to celebrate Sol but it was not the main date for its celebrations and early Christians had picked the date far earlier.
Also, later in the 5th and 6th century we find the church coopting pagan celebrations to help accelerate the assimilation of different cultures into Christianity. However, in the first 3 or 4 centuries of the church Christians would not have touched anything even faintly resembling paganism. When many Christians left the faith to save their lives under the persecution of Diocletian one group would not allow them to repent and return. These people were called Donatists after their leader Donatus. The Donatists celebrated Christmas on December 25. If the celebration were tied to Mithras or Sol Invictus there is no way the Donatists would have endorsed it.
All these words to say "celebrate Christmas the way you like." If you don't celebrate at all, fine. Don't be certain of everything you have heard and remember for most things they can be used for good or for ill. I can't stand all the elves and Santa as I think it detracts from celebrating the Messiah's nativity. Remember you don't have to do what you think is wrong but condemning others is not apt to help either. Have a wonderful day.