University rankings: What it means to be No.1 – and how it’s calculated

Since the first international university rankings were established nearly 20 years ago, they’ve exerted a powerful influence on students’ choices.

Based on what the rankings say, students make important, life-changing decisions – and not just about which course they’ll enrol in or which university they’ll apply to. In many cases they’re committing to the cities and countries in which they’ll study, forge their professional networks, build their careers and live their lives.

But for all that, not many people understand what the major rankings tables actually measure.

With Monash recently named No.1 in the world for pharmacy and pharmacology, we thought the time was right to take a peek under the hood and, using that ranking as a worked example, explain exactly the kinds of things that contribute to a No.1 subject ranking.

University rankings: a brief introduction

The QS Rankings (or, to give them their full title, the Quacquarelli Symonds World University Rankings) is one of three major international university rankings systems.

The other two are the Academic Ranking of World Universities (often referred to as the “ARWU” or the “Shanghai Jiao Tong”, after the university that conducts them), and the Times Higher Education.

Each set of rankings has a slightly different focus, and as a result their methodologies differ substantially.

The ARWU, for example, is concerned entirely with a university’s research output, whereas both of the other two rankings systems attempt to address the notoriously difficult-to-quantify question of educational quality.

To further complicate the picture, each organisation releases several ranking tables every year, including regional rankings, overall university rankings, and rankings that measure performance within a discipline or group of related disciplines.

The QS Rankings by Subject fall into the final category.

QS Rankings by Subject

The QS World University Rankings by Subject showcases more than 1500 institutions across 51 different subject areas.

It scores universities across five broad domains:

• Academic reputation
• Employer reputation
• Citations per paper
• H-index
• International Research Network (IRN)

In recognition of the fact that there are intrinsic and environmental differences between different discipline areas, the weight accorded to each domain varies.

For example, in arts and humanities, academic reputation contributes 60% of the final score; in life sciences and medicine (the umbrella under which pharmacy and pharmacology sits), it contributes 40%.

In social sciences and management, employer reputation accounts for 30% of the score. In life sciences and medicine it’s worth 10%.

Let’s examine each in turn.

Academic reputation

Across all subject areas, academic reputation is the domain accorded the most weight. It’s assessed via a survey completed by several thousand academics each year.

At the beginning of the survey, academics state their field speciality and their regional familiarity. The answers to this then guide the range of answers they can give in the remainder of the survey.

The Dean of Monash’s Faculty of Pharmacy and Pharmaceutical Sciences, Arthur Christopoulos, has some suspicions as to the factors that may have driven Monash’s favourable showing in academic reputation.

“To some degree, you’re going to get an echo of the other metrics that the subject ratings take into account, like h-index and citations. That’s unavoidable – a researcher’s publication record is fundamental to their reputation, and a faculty or university’s reputation is the sum of its research and education staff’s reputation,” he says.

“However, the nice thing about academic reputation is that it captures a broader range of factors. For us, it is possible that a string of recent high profile research with tangible real world impact, such as our development of Australia’s first mRNA vaccine for COVID-19, has played a part, as has our leading role in the new Neuromedicines Discovery Centre.

"But successes like this do not happen overnight, or by accident. Our university has been steadily climbing the global rankings due to many years of deliberate and strategic restructuring to focus on areas where we can really differentiate ourselves, particularly in drug discovery, development, formulation, manufacturing and medicine use and safety.”

As an example, Professor Christopoulos points to other streams of research conducted at the faculty in the past couple of years, including:

• the development of an inhaled form of oxytocin for postpartum haemorrhage in the developing world
• the development of the first oral form of the neurosteroid allopregnanolone for treatment of postpartum depression 
• development of a next-generation novel antibiotic targeting drug-resistant Gram-negative “superbugs” as likely influential.

According to Professor Christopoulos, academic reputation is also where the social and economic impact of a university’s work is felt.

“Our faculty has successfully partnered 35 compounds into clinical development to date," he says. "We’ve also helped generate multiple successful start-ups, licensing and commercialisation deals, including a major, 10-year, $40 million collaboration with the French pharmaceutical company, Servier. Earlier this year, a start-up based on our work, Septerna Inc., generated US$100  Series A financing

“Very little of that activity would be picked up in conventional metrics, but it all feeds into academic reputation."

Employer reputation

This is the metric that comes closest to assessing the quality of education a university offers in a subject, using as a proxy the employability of its graduates.

Like the academic reputation indicator, employer reputation is assessed by a survey that goes out to several thousand major employers every year. Employers are asked to nominate up to 10 domestic and 30 international institutions that they rate as being the best for producing graduates.

Professor Christopoulos acknowledges that it’s impossible to trace direct causation between any particular educational strategy and his faculty’s performance in this area, but believes the result probably reflects more than a decade of educational innovation.

 “In the past 10 years we’ve completely overhauled our course offerings and reimagined the way we teach both pharmacy and pharmaceutical sciences from the ground up,” he says, "with significant focus on active and experiential learning, skills coaching and employability."

He points in particular to the faculty’s flagship offering, the integrated Bachelor of Pharmacy (Honours)/ Master of Pharmacy, which graduates students with a higher degree than is traditional in the pharmacy profession, and is the first of its kind in Australia.

“We were getting feedback that, although our students’ scientific knowledge was unimpeachable, their clinical skills could be improved. So we redesigned the course with an explicit focus on skills – communication, teamwork, integrity etcetera.

“We nearly doubled the amount of time students spend on placement to ensure they were able to integrate those skills into their practice through experiential learning, and were much more advanced in terms of their professional development by the time they graduated.”.

Citations per paper and h-index

These two measures, worth 20% of the final score each, are both ways of assessing the impact and quality of the scientific work done by institutions, on average per publication.

The two measures draw on the same data sets, but are the results of different analytical approaches. According to QS:

“Despite being built on the same underlying data as the citations measure, the H indicator returns some different results [to Citations per Paper], these differences are central to the value of h-index. In a large institution producing a lot of research, a research group that is cutting-edge can be lost in a citations per paper approach, whereas in h-index analysis, it is the unimportant research that gets overlooked. A small, focused institution is unlikely to compete with a world-leading large institution, but can still hold their own.”

Professor Christopoulos is at pains to point out that although large streams of the faculty’s research take place outside the boundaries of pharmacy and pharmacology, those streams still make an indirect contribution to the ranking.

“By deliberately and strategically structuring itself along the drug discovery, development, formulation, manufacturing and medicine use pipeline, our faculty is the only academic institution in the country that operates in a manner akin to best-practice in the pharmaceutical industry,” he says.

“That full-service approach is incredibly valuable to all of our researchers, because it makes collaboration with high-quality partners easy. When they need a molecule to help answer some pharmacological question, our pharmacologists can walk down a flight of stairs and work with some of the finest medicinal chemists in the world to design that molecule.

 ‘Similarly, if we need to further optimise that molecule to turn it into a medicine for eventual clinical trials, we have the best drug optimisation and formulation scientists in the country in the same building”.

International Research Network

According to QS:

“[The] IRN index reflects the ability of institutions to diversify the geography of their international research network by establishing sustainable research partnerships with other higher education institutions. It also reflects the efficiency of this as we look at the diversity of partner locations against the efforts needed to achieve such a diversity.”

“In virtually all instances, the only truly novel thing that any biomedical researcher can realistically claim”, says Professor Christopoulos “is simply that start of a good idea."

"Everything else comes from partnership and collaboration. A smart institution will acknowledge that there are areas that they lead, but there are also areas where they need to partner well, early, and strategically.  And many of these partnerships are often international. As a faculty, we have had a long track record of major, sustained international partnerships, with industry and other leading academic institutions.

“One of our proudest and most important initiatives in this space has been the formation of the PharmAlliance; a research, professional practice and education partnership between the schools of pharmacy at Monash University, University College London (UCL) and the University of North Carolina at Chapel Hill (UNC). Each school is ranked in the global top 10 and it is the first cross-continent model of this type in the world, which was grown organically through a shared sense of vision and purpose. 

“I suspect, although I do not know for certain, that the new IRN Index may be capturing these types of successful international networks, which any good institution should be developing anyway.

"But it must always be for the right reason! Rankings will come and go, but scientific rigor, educational credibility and improving the human condition are not negotiable.”