I know what you mean. Interesting that all of the best places to eat in the area are outside of Swindon rather than in it. I think it's partly to do with the odd demographic in the Swindon area where most of the really expensive amenities (incl houses, private schools etc.) are missing from the town itself but can be found nearby. There's a similar situation in Basingstoke.
When I was living in Old Town about 10 or 12 years ago I do remember a French restaurant opening at the top (Old Town end) of Victoria Road, but it closed soon after through lack of interest. (Might have been called Papillon?)
Swindon always has been a working class town...in the 60's going into the early 70's there were 2 places where you could get a steak, if you slightly aspirational..viz The Crown at Stratton and The Bell (restaurant was then downstairs) there was also The Khyber.
Blunsdon House, could do you a meal, but the restaurant was the size of a large living room.
Subsequently any new players have had to cater for a still an essentially working class population...