Possibly because of my driving style being less aggressive than lots of other Speedster Owners the only time I really noticed a significant difference between the IRS Speedsters I've had (several of them) and the swing axle Speedsters I've had (lots of them) was when I added a camber compensator to one of the swing axle cars and then drove with Robert and Teby. Otherwise, for me and my driving style, there wasn't that much of a difference.
The vast majority of the 3,000 plus Vintage Speedsters out there are swing axle and, it's just my guess, but the majority of them are probably satisfied with how they handle.
If you're just a pleasure cruiser and never plan to try to keep up with Robert or Teby, just get a swing axle car and spend about $100 bucks on a camber compensator. Since you are getting a Pat Downs motor, you're probably not just going to be a cruiser,so get the IRS.