So a new research paper has landed on your desk or in your inbox? Or perhaps you have a whole set of papers to read at the start of a new project? How do you know which ones are any good, and which are less reliable? It’s impossible to be certain – good research does sometimes come from surprising sources - but here are some quick indicators.
Check the source
Do you know the journal? If so, is it reputable? Are the papers refereed before being published?
If it is from a conference, is the conference a major international conference – where there has been competition for selection – or a minor local conference without such selection? If it is published in the proceedings, do you know if the proceedings contain all conference papers or just selected papers?
Check the institution and the author. Of course unknown authors do publish excellent research from time to time; how else would new researchers start their careers? But a paper on the link between food choices and health by a senior professor from a well-known teaching hospital is more likely to be reliable than one by an associate professor of dietology at Hicksville Community College.
Check the conclusions
What does the research actually say, if anything? Can you summarise the research in two or three sentences? Or is it just a long-winded ramble about experiments in progress? Sometimes institutional pressure to publish leads to papers being written on work that hasn’t yet reached significant conclusions.
Does the paper add anything to what you already know or would reasonably have guessed? Research showing that the average horse-owner is wealthier than the average ferret-owner may not add much to human knowledge. Conversely, research reported recently in the UK press showing that the average Cypriot is wealthier than the average Berliner surprised many people, and does add new information to discussions on the Euro crisis.
At conferences a former colleague of the author often asked ‘Was there anything you found surprising about your results?’. The papers where the speaker said there were surprising results were considered much more worthy of further discussion.
Check the research methods
How was the research carried out? If by questionnaire, what were the sample size and the response rate? In the author’s experience, business and management research in particular is susceptible to ‘death by spreadsheet’ analysis, along the lines of ‘we sent out 3000 questionnaires to senior managers, received 25 back, analysed every factor in the questionnaire against every other factor and concluded that…’.
Do please be aware that the 25 ‘senior managers’ who replied may not be a representative sample, perhaps invalidating any conclusions.
Questionnaire-based research may miss underlying issues. Some years ago it became fashionable for surveys to ask ‘Does your institution have a policy on the exploitation of intellectual property? Please tick the relevant box – yes or no’. The answers were normally ‘yes’, leading some to conclude that everything was well under control. The real answers all too often were along the lines of ‘actually several departments have various policies, many of which are ignored in any case’, a situation which can only be properly investigate through detailed interviews rather than by simple questionnaires.
Detailed interviews aren’t the answer to all social science research problems; interviews take time and money to carry out with large numbers of subjects, and questionnaires are useful with large sample sizes if a significant response rate is achieved. But do be cautious of research papers that have used questionnaires, or interviews, in isolation from other methods.
Check for possible bias
In an ideal world all research would be free from undue influence. But that doesn’t always happen. So when research comes up with strong conclusions and policy recommendations, do check who sponsored the research. The days of tobacco-companies encouraging people to produce results showing no clear causal link between smoking and cancer, are, one hopes, in the past.
Yet it is always worth checking for bias – perhaps a small business group had sponsored research which just happens to show that business rates should be cut, or a housing charity has produced research showing that the government should invest more in housing. The conclusions may indeed by be valid, but do treat research sponsored by ‘interested parties’ with added skepticism.