There’s nothing like a good review to help make a decision is there? Whether it’s an aircraft you want or a location to fly to, a good review helps sort out the wheat from the chaff, the gold from the lead, the good from the bad. That’s the theory anyway. Do you ever wonder how it all works though? How exactly a review is born? Well tough, I’m telling you anyway, and along the way, I’ll highlight the pitfalls of having an opinion and writing it down. First of all, let’s start with a bit of background.

Behind the curtain.

Here we are, in our cafe, meeting for the first time. So my name is Jessica, and I’m a journalist who writes for PC Pilot magazine and Airliner World. I make money from my words. It’s a cool job, and the pay is ok. I write reviews both for PC Pilot and in my spare time, for Mutley's Hangar and here. What can I say? I like what I do. Based on this information, you shouldn’t be surprised to discover that I have a passion for aviation and flying. If I had lots of money, I’d be doing more flying, but such is life. Flight experience wise, I did my glider training at 16, soloing in a glider with a broken wheel brake. That was a long roll out. Powered flight wise, well I fly when I can with an instructor, often in Piper PA28’s. I like the PA28. So I’ve got some real world experience. I know which way to point the nose and what the wings do.  I started flight simming from a young age, with the ZX Spectrum. From there I graduated to the Commodore Amiga and various Microprose combat sims, then the PC and FS 4 and so on. Basically if it was a sim in the 90’, it was on my PC, including X-Plane 6, Sierra's ProPilot and the excellent Flight Unlimited 2. Currently, I’ve more sims than I know what to do with , that comes with the job I guess, and in the corner of my living room is a full size A320 sim that’s currently down for maintenance and rewiring.

Reviewing wise, I started that 10 years ago for a website called the Flight Sim Network. It was a cool social media style site that I joined. They were looking for reviewers, I wrote an unapologetic review of a Wilco 737, calling it where it needed to be called and sent it in as a test piece. They loved it and that was that.  Within 18 months I got a job with PC Pilot and the rest is history. So yeah, that’s my background.

Playing by the rules.

‍

Before a single word gets written, I need to discuss a few rules that I follow when reviewing. You can’t review anything without being impartial. These simple rules keep me on the straight and narrow.

1.Be constructive, not cruel.

This is the most important rule there is. If a product has bugs, it can be tempting to tear into it, especially if the developer has form for this. Pointing out bugs is good, destroying a product because of bugs just makes you an asshole. Chances are the bugs will be in the process of being fixed. Unless a bug is so bad that it stops the product or sim from working, point out the bugs alongside any positives. Be as impartial as you can. Being a real hard ass will play well to a certain section of the community, but developers will quickly tire of your bullshit.

2. Review what’s in front of you, not what you want it to be.

Anything you review should be reviewed in isolation. I don’t mean you need to shut away the world. It you have a new aircraft on the screen, concentrate on that and that alone. Keep comparisons to other products to a minimum. I’m not saying you can’t compare at all, but things need to be in context. If the new aircraft is a step forward from the developers previous offering, you can say so, but keep it reasonable. Don’t compare products that shouldn’t be compared. For example don’t compare the default 737 to the IXEG 737. Comparing aircraft that can’t be compared is pointless. There are exceptions, notably on price. If a product is the same price as the IXEG 737 but not as advanced systems wise, you can use that as a fair comparison. Finally, don’t compare between sim platforms. Pricing and system detail on a P3D product matters not when it comes to an XP11 add on.

3. Take your time and never review a beta product.

Reviewing a beta product is a pointless exercise. It’ll be buggy, annoying and unfinished. Programs change massively from beta to version 1. Conversely, if you review and item and a service pack comes out just after you’ve finished the review, do not adjust the review to reference this. Products change all the time. A review is a snapshot, not a portfolio.

The other big no no and something I see a lot of in the XP community is that when a product is released, a nice shiny review comes out either immediately or very shortly afterwards. These are often paid for sales pieces written alongside the developer using beta software. Weirdly these reviews are often positive. Not bad for a product that’s twelve hours old. Review only after it’s released and take the time to test the aircraft throughly. For the record, I’ve done this twice with pre release software, but I was honest with what you got, and the reviews weren’t that positive.  

4. Be Entertaining.

The world of reviewing is a world filled with facts and figures. It’s as boring as hell. So for the love of God, try and be a little entertaining in what you write. There’s nothing worse than a review that reads like a specifications table. You should always be impartial, but never dry. The truth is, if you find yourself enjoying a particular add on, it’ll come through in you’re writing. There are ways to keep things entertaining. For example, I did one review by using Frank Sinatra song titles for paragraph headings. An AN-2 review I used Russian themed headings. It’s all in how you present the information.

How a review works.

So my background sorted, here’s how a typical review works. For PC Pilot the first thing that happens is I get a commission letter. In that letter will be the details of the current issues jobs. There’ll be a word-count required for each article and an associated amount of pictures required as well. Typically, this is 1500 words + 12 HQ Pictures of the product. There’ll also be a deadline date, typically 4 weeks from the date I received the letter. Now you’re on the clock.

With the clock ticking, the first thing I need to do is to contact the publisher and get a copy of the software. This isn’t always as easy as it sounds. Almost all publishers will happily sort you out with a copy of the product, and the great ones will give you a list of features to look out for as you review. Only ever so occasionally do publishers not get back to you. This is usually a bigger issue if you’re writing outside of PC Pilot. However, again most publishers will happily help you out. 

Now that the product is downloaded and installed, my next contact with the publisher will be to let them know the publication date of the magazine. Unless I have serious issues with the product, I leave the publisher out of the loop. The review is my opinion, and when that contradicts the publishers opinion, things can get awkward.

In the next four weeks, it’s time to get to know the product. Keep in mind that during this time, life still goes on, and other articles for the same issue also need to be worked on. That a four week window is shared. However, now you can get to grips with it. You start by reading the manual and other documentation supplied. A poor manual can start a downward spiral for a product if you’re not careful. I’m not looking for spelling mistakes. I’m looking for things that don’t work. A checklist that doesn’t flow correctly or a system that isn’t there. Worse case scenarios are things that are shown in the manual that aren’t actually in the finished product. For example, I was reviewing Just Flight’s BAC 1-11 and a switch marked ‘Nav/GPS’ was listed in the manual, but wasn’t shown in the aircraft. Obviously the manual had been prepared using an earlier build of the aircraft. The result was slightly frustrating.

Conversely, too much of a manual can be a huge issue as well. Take for example the JAR Design A320. When I reviewed that, the manual provided was a huge 1500 page PDF of the actual A320 FCOM. Leaving aside the possible legality of the poorly scanned document, it didn’t help new users get the aircraft flying in the sim. A tutorial and a cockpit overview would have been better than the terrifying 1500 pages you had. Manuals are important.

Switching to the sim, you look at the install process. These days, installers are great and for the most part, work correctly. Sometimes they do screw up though, so keep an eye on things. Once you’ve installed the product, it’s time for the all important ‘first impressions.’ First impressions do count. If things start out difficult, no matter how great the product turns out to be, that first impression will stay with me and it will result in a lower score. Mostly, things work out ok.

Next up it’s all about the looks. How are the textures? If it’s scenery, how realistic is the area? Do the buildings have Ambient occlusion burnt into the textures? For Aircraft, how well is the cockpit rendered? How nice are the liveries? Nice stuff. Looks are important. Bland low res textures will earn you a mark down. That said, textures that look great but bring your FPS to single figures will also cause issues. Again Jar Designs A320 is a great example of ‘getting it slightly wrong’. The cockpit is textured beautifully with HD photo textures. But they’re not really optimised and when I was flying the aircraft on my three screen set up, the textures would regularly flood my VRAM, often exceeding it by 1gb or more on a 4gb card. Result, FPS in the toilet.

Then comes the all important testing. Scenery means flying a variety of aircraft into the airport in a variety of weather conditions and at various times of day. The big test will come with heavy overcast, rain or snow, at dusk or dawn in an aircraft that’s proven to be a bit of system killer. Give me good, usable FPS here and you’re golden.

For aircraft, things are more complex. First, how does handle on the ground? Then what’s it like to fly? Does it fly ‘by the numbers’ in general terms. That last bit is always tricky. Simply put, you don’t have time to run though the performance tables and check the speeds at every weight and in every condition. Instead you need to check the simple ‘base stall speeds’ and so on. No aircraft hits it at 100% book speeds. In fairness this is true for most real aircraft to, as subtle differences and personalities mean that identical aircraft can fly slightly differently. Given this aircraft is flying around in a $60 Sim, a ball park figure is often the best you can hope for. The smaller that ball park though, the more accurate the flight modelling is likely to be, and the more likely that a higher score will be awarded.

The flight characteristics done, you move on to the systems. This is very tricky. The big study level aircraft will have plenty to deal with, and often have things simulated you’ll never even see in regular use. This is where that lovely list the publisher hopefully gave you, detailing the systems, will help. The truth is though, unless you fly that aircraft in real life, or you’re an engineer on the thing, you have to take the word of the developer. If you do have time and a copy of an FCOM, you can run through a few tests to confirm.

Lastly, you need to live with your new aircraft for a while. Fly it on its typical mission. Several in fact. This is your new best friend until deadline day. For short haul aircraft, this is a fun time. You can criss cross Europe in a few days, logging 10 plus flights is three days. Before you know it, you’ve 30 hours flight time and a better understanding of how she flies. For long haul, things are different. A 747 can fly for hours. Sure you can fly her on a two hour short haul route, but there are some things that you can’t look for unless you’re on a 10 hour plus flight. The same 30 hours flight time will leave with just 3 flights done and you won’t feel comfortable to comment on the take off and landing phases. Besides, sitting monitoring a flight for 10 hours is difficult if you have other things to do. Basically you’re weekends are gone. Beer helps though.

In your own words.

With one week to go before deadline day, you have to write up the review. This isn’t as easy as it sounds. Suddenly you’re aware of that 1500 limit. Now if all’s gone well you have plenty to write about. If it’s gone wrong, you’ll have plenty to write about to. If it’s good, then you’ll find that you need to cut and condense to hit that 1500 word limit. This is often difficult. To cover the manuals, the textures and modelling, the flight dynamics and any bugs you’ve found, plus a ‘summing up’ in 1500 is no mean feat. As a rule, if there’s to much to write about, keep the intro short, deal with as much as you can and sum up in no more than 200 words to finish. This is the same for a review of a bad product. Keep things concise.

The worst thing that can happen is you have a product that doesn’t do enough to hit that 1500 word review. Then it’s a matter for digging for any scrap of information you can find. These are the worst reviews you can have. Fortunately they’re rare.

The final thing you need to do now is to collect the screenshots and caption them appropriately. It’s actually my least favourite part of the job. Screenshots have to show off the product in all its glory, warts and all. There needs to be a least one shot that can be used for the main profile shot. Nailing that can be difficult. Then there are restrictions on how dark a shot is. Dark screenshots get darker when you go to print. So lighter shots are preferred to preserve details.

Finally you need to score the product, give a small pros and cons list an sort out the publisher details, price and system requirements. Three weeks flying and a week to write it up and you’re free until the next review.

Fallout.

Here’s where things can interesting. You’ve done you’re best and the review is now wild and free. Now I pride myself on being brutally honest but fair. If a product is good, I’ll say so. If it’s not so good I’ll say so. Sometimes this doesn’t go down to well with certain developers. When I Reviewed the Flight 1 DC-9 a few years ago, I found the whole damn thing to be massively disappointing. It was incredibly buggy, with hot fixes not helping prior to the review release. I was still flying the aircraft the day before deadline, and it was still suffering issues. So the review reflected that. It was a cracking aircraft to fly, when you could get it to fly. At the same time, my colleague, Pete Wright aka Froogle was also flying the aircraft for his YouTube channel and he loved it. He wasn’t seeing the same kinds Of issues I was, and alongside my review, he’d prepped a video that would go onto our cover disc. My review torpedoed that and the video was scrapped.  

Another issue was with a mission pack for the then new FSX-SE. Called Dangerous Approaches, the mission pack proved to the worst thing I’d ever reviewed, and not solely to do with the product. The product was produced by then Deputy Editor Jane Whittaker, and Jane was a good friend. For seven straight days from the moment I got the software, Jane messaged me, enquiringly how it was going, filling me in on all the details I didn’t need to know and Jane was certain I’d love it. I didn’t. With a score of 70% and a review that included the word ‘dull’, the product was simply not good. I won’t go into details, but the aftermath was bloody and left me without a friend.

Not every review ends as badly. Some developers simply dislike a bad review. After reviewing the Flight Factor A350, Flight Factor wrote me an email demanding a change to the claim that the Pro version would be a free upgrade for users. I checked back though various forums and groups, confirming that at the time of release, a member of FF stated it would be a free upgrade. The review got changed to reflect the updated information, but FF are no longer willing to work with me due to the A350’s poor review score.

One criticism I’ve seen lately is that I can’t be objective because I write for PC Pilot. Because we have companies like Orbx pay to advertise in the magazine we have to score them high or loose that money. It’s bullshit. In 10 years I’ve changed a review score once. My editor asked if I’d been overly harsh in marking down a pc system,  I agreed and adjusted the score by 5%. That’s it. I am impartial. As an example, I worked with Dovetail with regard to their FSW Sim. I’m also good friends with Dovetails product manager. She’s stayed at my home and we chat regularly. I still pointed out every error with the release of the sim. It was firm but fair.

For every bad review story I have though, there are plenty of good ones to follow. Whilst I was reviewing the ProSim 737 cockpit builders suite, i was lucky enough to get some time in a full motion 737 Sim that was running ProSim. My favourite experience was when I reviewed the Sibwings AN-2. The AN-2 is a complicated aircraft. Things work backwards in the cockpit. As luck would have it, I’d recently read a flight test of the real thing, and the aircraft was based a few hours from my home. I contacted the club and they invited me down for an engine run. That day I got to experience the full start up and and run up of the mighty AN-2’s engine, and all from the cockpit. The noise, smoke and rumble were something else, and that experience was translated into the review. I certainly didn’t need to go that far for the review, but damn I’m glad I did. It’s an experience that came in handy just this month with a review of the new Aerosoft AN-2.

‍

Finishing up.

That's pretty much it. I'll finish off by saying that these are the rules I follow, and how I do my job. Others will find their own way, but in the end, to do the job right, you have to be objective and fair. I hope I am and I hope I've oven you all an idea of what its like to do this stuff professionally. It certainly isn't the dark and murky underworld some think.