Does the future of higher education look more like a mentoring pyramid scheme?

In response to my recent post about the future of higher education and one-on-one mentoring, one of my students from last year, Yunze, got in touch via email to offer a potential solution:

...I wonder whether it is possible to set a clear academic threshold within each discipline. If students who reach this threshold could mentor upper‑middle‑level students, while professors spend only a small amount of time supervising the overall direction, the system might become more sustainable. However, I suspect this could harm the interests of the top students, since they might otherwise use that time to further advance their own academic achievements, and If [sic] they fail to successfully train students with real research ability, it would likely damage both the university’s reputation and the professor’s own reputation.

You know, I think Yunze is right on the money here. Consider the problems I outlined in the earlier post: (1) the signalling value of education is falling due to generative AI; (2) a one-on-one mentoring approach may be a solution; but (3) one-on-one mentoring doesn't scale due to limited faculty time. If one-on-one mentoring is not conducted between faculty and students, but works more like a pyramid mentoring model, then this might actually work, not just for students, but for faculty and for universities as well.

So, let's think it through. But first, remember that the mentoring model I introduced in the earlier post is not simply a model of small classes, where senior students perform limited teaching roles, such as tutoring. This is a model of genuine mentoring, where the mentor encourages the mentee to become a builder, in the words of Auren Hoffman. A builder creates things, and it is the act of building, and the learning alongside that, which will be a durable signal to future employers. In relation to mentoring, I said in that post that mentors should do the following for their mentees:

Teach them to be builders. Encourage them to create things. Work with them and chart a path forward for their success.

If faculty provide one-on-one mentoring to a small number of senior students, then that makes better use of faculty time than them mentoring hundreds of first-year students. The senior students can then each mentor several second-year students, who in turn can then mentor several first-year students. [*] In this model, faculty time is targeted at the senior students, where the impact of faculty on student employment outcomes may be greatest.

Students benefit from helping junior colleagues to become builders, where the signalling value may remain even in the face of generative AI. Even better, mentoring provides student mentors with an opportunity to build - they may be able to point employers to the success of their mentees as an example of their building, talking also about what went wrong in the mentoring relationship, and what they learned from the experience.

In this mentoring pyramid model, universities retain a key role, but that role becomes very different. Universities essentially become a platform, connecting students with mentors - first-year students with second-year mentors, second-year students with senior student mentors, and senior students with faculty mentors. In the terms of my earlier post, the university runs their own OnlyStudents platform.

Of course, this platform role creates a new problem for universities. If mentoring works mainly as a way of matching students with mentors, then the market may not need eight OnlyStudents platforms in New Zealand, or thousands of OnlyStudents platforms worldwide. A small number of large platforms could have a big advantage in that case - more students attract more mentors, more mentors improve the quality of matching, and better matching attracts still more students. Those network effects could create a winner-take-all dynamic, in which universities would struggle to differentiate themselves simply by running their own mentoring platforms, and where a single surviving OnlyStudents platform might be the ultimate outcome. However, that conclusion depends on the strength of the network effects. If effective mentoring also depends on institutional trust, disciplinary reputation, local employer connections, pastoral care, or an in-person community, then universities may retain some defensible advantages. Geography alone probably won’t be enough, especially if online mentoring is close to being as effective as in-person mentoring, but local connections might still matter. So the question for universities is not just whether they can build their own OnlyStudents, but whether they can attach that platform to something that a larger, more generic OnlyStudents cannot easily replicate.

Universities may also retain a role in the initial and ongoing training of mentors. Since each student, and each faculty member, will need to be a mentor to one or more others lower down in the pyramid, they will need to understand how to mentor. That means universities would not simply be matching students with mentors. They would also need to train mentors, monitor the quality of mentoring relationships, and intervene when mentor-mentee relationships are not working well. Moreover, the adoption of a mentoring pyramid model is likely going to change who the most successful students (and faculty members) are. The top students do not necessarily make the best mentors (or the best tutors, as I have learnt across years of coordinating tutors in my first-year papers). Good mentoring requires a specific skill set, but it is those skills that may also demonstrate the quality of the student as a builder - a signal of high quality for employers.

A further point about the pyramid mentoring model is that it likely requires a strong filtering effect to be financially viable. Since each faculty member can only mentor a limited number of senior students, and each of those senior students can only mentor a limited number of second-year students, who in turn can only mentor a limited number of first-year students, each level of the pyramid probably needs to be somewhat wider than the levels above it. To achieve that, student progression needs a strong filter, limiting the number of students who progress from first-year to second-year, and from second-year to senior.

Let's consider some simple numerical examples that illustrate why filtering is needed. If each faculty member mentors five senior students, and each senior student mentors five second-year students, who each mentor five first-year students, then the pyramid contains 155 students per faculty member - five senior students, 25 second-year students, and 125 first-year students. A model where each faculty member’s salary is covered by fees from 155 students, setting aside any contribution to central university costs, seems likely to be financially viable to me. However, in this model only one-fifth of students could be allowed to progress each year. That means the model would also need some form of orderly exit for students who are filtered out - perhaps an exit qualification, or a pathway into a non-mentored track. The problem is that both options may provide negative signals about the student who is filtered out.

If all students were to progress, then that would require each student to mentor at most one student at the level below. Keeping five senior students mentored by faculty, then the pyramid would contain 15 students per faculty member - five senior students, five second-year students, and five first-year students. That system would be much less likely to cover the cost of faculty time. So, it's unlikely that the pyramid mentoring model would be viable to run without some form of filtering - perhaps not as extreme as only one-fifth of students progressing each year, but clearly not all students could progress every year.

So, to return to my conclusion from the previous post, the current mass higher education model still looks increasingly fragile, but perhaps one or a few universities might be able to navigate their way through. However, the survivors are likely to be first-movers or fast followers in developing a platform market strategy that leverages a pyramid mentoring model. This model is still going to cost students a lot, and the filtering effect would make higher education more elitist as well.

And thanks to Yunze for inspiring this post with his perceptive email comments.

*****

[*] For simplicity, I'm assuming a three-year higher education degree structure, as we have in New Zealand. For a four-year degree structure, you would of course need to add an additional level.

Read more:

• Why all students need to understand adverse selection and signalling
• Signalling theory suggests that we may need two categories of university degree
• Does the future of higher education look more like one-on-one mentoring?

Does the future of higher education look more like one-on-one mentoring?

It almost seems a cliche to say that generative AI is both the greatest threat, and the greatest opportunity, for higher education. That doesn't make the statement any less true. And there are many commentators who are trying to work out what happens next for higher education. I think one of the best is Hollis Robbins, whose Substack Anecdotal Value is well worth subscribing to.

Robbins was recently interviewed by Jay Caspian Kang of The New Yorker (paywalled, but you can find an ungated version here) about the future of higher education. The whole interview is worth reading, but I want to highlight this bit in particular, where Robbins says:

I was in Austin, Texas, a couple of times in March with a bunch of twenty-five-year-old billionaires. This is what they’re looking at. Instead of having the credential from the institution, why not have the credential from the professor? If you have a Hollis Robbins education, what would that signal? What would that credential mean as opposed to a degree from a university? There was some conversation about what that would look like, and one guy at the end of the dinner said, “Instead of OnlyFans, it’s like OnlyProfessors.”

The correct analogy here would be that it would be OnlyStudents, not OnlyProfessors (since the name contains the audience, not the performer). However, Robbins makes a good point. Higher education is, in part, an exercise in signalling (as I've noted before here and here). In fact, Bryan Caplan argued in his book The Case Against Education (which I reviewed here) that one third of the benefit of higher education is signalling (the other two-thirds is made up of genuine learning, socialisation, and transferable skills). 

The diploma that a student receives at the end of their higher education journey is a signal to employers of the student's quality as a future employee. The signal is credible to the employer because it is costly for the student to obtain, and costly in such a way that low-quality students wouldn't attempt the signal. University reputation matters here, because it is an assurance of the second of those conditions - low-quality students wouldn't attempt the signal of a Harvard degree, firstly because they wouldn't gain admission to Harvard in the first place. So, part of the signal comes from getting into Harvard. Second, low-quality students wouldn't attempt the signal of a Harvard degree because passing courses at Harvard is hard (or, at least, harder than at many other universities). The quality of the education at Harvard has traditionally been higher than elsewhere.

In the interview, Robbins makes a further important point, which is that we've spent the last several decades making higher education a commodity. Students studying a degree in a particular subject learn the same things, often using the same teaching materials, the same textbook, and the same style of assessment, regardless of which university they go to. That means that the quality of signal that arises from the education itself has reduced over time, meaning that most of the value of the signal arises from the admission process. Once a student has been admitted to Harvard, their signal is in place, and the further signalling from their education is lower than for comparable students in years past.

Robbins argues that generative AI is accelerating and expanding this commodification of education. Since generative AI has access, through its training corpus, to a large store of human knowledge, to add value over and above generative AI a professor must be a true expert in a very specific subfield. For students, learning a subject from a true expert still retains value, because generative AI cannot as easily replicate the learning that would occur from the expert. At that point, the university becomes less important as a mediator of education.

In other words, Robbins is arguing that the university could be disintermediated, with individual professors issuing credentials instead of universities. Who needs a Harvard degree, when you could have a Hollis Robbins degree? And since Robbins is an expert (in African American sonnet tradition, she says), the signal retains high value. However, that is true only to the extent that employers find the Hollis Robbins degree a credible signal.

That brings me to this Substack post by Auren Hoffman. Hoffman also argues that generative AI has diminished the value of the higher education signal. However, Hoffman argues for a different solution, this time from the perspective of the graduating student:

you have to show you can add value. that is it. that is the only thing.

the test the smart hiring manager applies in 2026 is simple. can you learn something on your own? can you finish what you started? can you do what you said you would do? these were always the skills that mattered. the difference is they are now the ONLY skills that matter, because the credential stopped doing the screening.

if you have a few years of experience, your resume can show you have these skills. if you are a new grad, you have to show what you have created and built.

Hoffman is overstating things a little, as university degrees are unlikely to disappear overnight. However, his critique is still important, as is the implication he draws. Hoffman argues that graduates (or young people, generally, since the solution doesn't depend on a student attending a university or completing a degree) need to become builders:

the most valuable thing a 22 year old can do in 2026 is create something. an app. a screenplay. a side business. an internal tool you wrote for a club you were in. a dinner series. a script that automates something annoying. a website. a chrome extension. a sculpture. a dance party. a discord bot. a substack with 32 readers and a real point of view. anything that moved from idea to working.

will the thing make money? probably not. that is not the point.

the point is that you taught yourself something. you finished it. you can describe what you learned, what broke, what you fixed, why you made the calls you made. that story is the new resume.

every hiring manager would rather interview a 22 year old with a launched app and a github full of weird side projects than a 22 year old with a 3.9 GPA from a top 50 school. it is not close. when one candidate has tangible evidence of what they can ship and the other has a transcript, the transcript will lose every time.

According to Hoffman, the best signal for young people to be sending in future is that they can build. Our students will ultimately be more successful if they can show off their skills (both technical and transferable skills) by building something. That is a signal that is costly, and costly in such a way that low-quality students will not attempt it. The signal is credible, and for the most part it retains currency even in the face of generative AI. Indeed, if the student builds something while effectively leveraging generative AI, then the signalling value to employers may be even greater. The key distinction is between the student using generative AI as a tool and the student using it as a substitute for doing the work. A student who can explain what they built, what broke, what they learned, and why they made the choices they made, is still sending a costly signal. A student who simply lets generative AI build for them is not. Employers would likely see through the latter pretty quickly.

Hoffman's idea isn't exactly new. When I think about some of my best students over the past two decades, they tend to be those that built something either while they were studying, or immediately after. For some, this was their own small business or entrepreneurial activity. For others, it was writing and publishing a research paper. Those were challenging tasks that set them apart from other students - a clear signal of quality. What Hoffman is essentially saying is that, with the signal from higher education itself being removed, the only remaining signal of quality is the signal from being a builder.

Where does that leave higher education staff? I think we can combine Robbins's and Hoffman's ideas, and chart a path forward. We don't need to start an OnlyStudents, and issue our own degrees, but we do need to cultivate closer relationships with our best students. Teach them to be builders. Encourage them to create things. Work with them and chart a path forward for their success. In other words, be a mentor.

Universities are absolutely going to hate this. Mentoring is not an activity that can be offered at scale. For example, there are simply not enough hours in the day for me to individually mentor all of the 350-plus students in my first-year economics class. Nor is mentoring easy to timetable, measure, standardise, or reward under current academic workload models. Universities are built around papers, credit points, learning outcomes, assessment rubrics, and student evaluations, all of which can be offered at scale. One-on-one mentoring doesn't work so well in that system. For example, postgraduate supervision already sits awkwardly within the system, with it being unclear whether it counts as teaching, or research. Nevertheless, mentoring may be one of the few ways that higher education can continue to offer something that is both valuable and difficult for generative AI to replicate.

If generative AI significantly reduces the signalling value of university education, and if students increasingly use it to avoid genuine learning, then the current mass higher education model looks increasingly fragile. Moreover, what remains is going to cost students a lot more. If having a high-quality mentor who can encourage a student to build is the path to their future employment, then it may be worth it. After all, high-quality signals are costly. That aspect of education, at least, won't have changed.

[HT: Marginal Revolution, for both the Hollis Robbins interview and the Auren Hoffman post]

Read more:

• Why all students need to understand adverse selection and signalling
• Signalling theory suggests that we may need two categories of university degree

What we learn from Freelancer.com about labour market signalling in the age of generative AI

In yesterday's post, I outlined my case for why generative AI reduces the quality of signalling in education. That is, how good education (qualification, or grades) is as a signal to employers of an applicant's ability. There is evidence to support this case, from two recent papers.

The first paper is this pre-print by Jingyi Cui, Gabriel Dias, and Justin Ye (all Yale University), which looks at the signalling benefit in cover letters. Specifically, they study:

...the introduction of a generative AI cover letter writing tool on Freelancer.com, one of the world’s largest online labor platforms. Freelancer connects international workers and employers to collaborate on short-term, skilled, and mostly remote jobs. On April 19, 2023, Freelancer introduced the “AI Bid Writer,” a tool that automatically generates cover letters tailored to employers’ job descriptions that workers can use or edit. The tool was available to a large subset of workers depending on their membership plans.

Cui et al. use eight months of data on two skill categories (PHP, and Internet Marketing), which covers over five million cover letters submitted to over 100,000 job opportunities. They observe who had access to the tool, as well as who used the tool to generate a cover letter, and how much time they spent refining the AI-generated cover letter.

Cui et al. look at the impact of the availability of the generative AI tool on callback rates, using a difference-in-differences research design. This effectively involves comparing differences in callback rates between applicants with and without access to the tool, before and after the tool was made available. Cui et al. find that:

...access to the generative AI writing tool increased cover-letter tailoring by 0.16 standard deviations, while actual usage raised tailoring by 1.36 standard deviations. Applying the same design to callbacks as the outcome, we find that access to the generative AI tool increased the probability of receiving a callback by 0.43 percentage points, and usage raised it by 3.56 percentage points. The latter represents a 51% increase relative to the pre-rollout average callback rate of 7.02%.

All good so far. Job applicants are made significantly better off (in terms of receiving a callback) by using the tool. However:

Our second finding is that AI substitutes for, rather than complements, workers’ pre-AI cover letter tailoring skills... We find that workers who previously wrote more tailored cover letters experienced smaller gains in cover letter tailoring—indeed, the best writers... experienced 27% smaller gains than the weakest ones. By enabling less skilled writers to produce more tailored cover letters, AI narrows the gap between workers with different initial abilities.

In other words, employers are now less able to distinguish the quality of the worker by using the quality of the writing in the cover letter. The consequence of this is that:

The correlation between cover-letter tailoring and receiving a callback fell by 51% after the launch of the AI tool, and the correlation with receiving an offer fell by 79%. Instead, employers shifted toward other signals less susceptible to AI influence, such as workers’ past work experience. The correlation between callbacks and workers’ review scores—the platform’s proprietary metric summarizing past work experiences on the platform and determining the default ranking of applications—rose by 5%. These patterns suggest that as AI adoption increases, employers substitute away from easily manipulated signals like cover letters toward harder-to-fake indicators of quality.

The total number of interviews and job offers were unchanged during this period. Cui et al. don't directly report whether the number of callbacks changed, but if we infer that from there being no aggregate change in the number of interviews, then this is consistent with the idea that the key difference is in the distribution of who received the jobs (and callbacks). Workers with a strong alternative signal (other than a well-written cover letter) received more callbacks, meaning that workers who lack an alternative signal received fewer callbacks. That has an important distributional consequence. New workers typically lack past review scores, so as employers lean more heavily on reviews, workers who are new to Freelancer.com will be disadvantaged and will find it more difficult to get a callback. Overall, in this case, the impact of the generative AI tool on the quality of signalling is negative.

The second paper is this job market paper by Anaïs Galdin (Dartmouth College) and Jesse Silbert (Princeton), who also use data from Freelancer.com. The difference is that they carefully evaluate employers' willingness-to-pay for workers, using the bid data. They also look at customisation of the text of the whole proposal, not just the cover letter. Another difference is that Galdin and Silbert look at a different job type, coding. Their data covers 2.7 million applications to 61,000 job openings, by 212,000 job applicants. Although Galdin and Silbert's paper is far more technical than the Cui et al. paper, Galdin and Silbert's results are somewhat similar (in terms of what they tell us about signalling):

First, we show that before the mass adoption of LLMs, employers had a significantly higher willingness to pay for workers who sent more customized proposals. Estimating a reduced-form multinomial logit model of employer demand using our measure of signal, we find that, all else equal, workers with a one standard deviation higher signal have the same increased chance of being hired as workers with a $26 lower bid... Second, we provide evidence that before the adoption of LLMs, employers valued workers’ signals because signals were predictive of workers’ effort, which in turn predicted workers’ ability to complete the posted job successfully. Third, we find, however, that after the mass adoption of LLMs, these patterns weaken significantly or disappear completely: employer willingness to pay for workers sending higher signals falls sharply, proposals written with the platform’s native AI-writing tool exhibit a negative correlation between effort and signal, and signals no longer predict successful job completion conditional on being hired.

This is strong evidence that, in this context at least, the introduction of the generative AI tool substantially reduces the quality of the job application signal. Galdin and Silbert then build an economic model calibrated based on their empirical results, and using that model they find that:

Compared to the status quo pre-LLM equilibrium with signaling, our no-signaling counterfactual equilibrium is far less meritocratic. Workers in the bottom quintile of the ability distribution are hired 14% more often, while workers in the top quintile are hired 19% less often.

This suggests an even worse outcome than what Cui et al. find. Galdin and Silbert's results suggest that the distributional changes in who gets offered work make high-quality workers worse off, and low-quality workers better off. That is what we would expect when the quality of signalling is reduced. Galdin and Silbert go on to say that:

These effects are driven by three mechanisms. First, employers previously relied on signals to make hiring decisions, so losing access to them impinges on their ability to discern worker ability. Second, more indirectly, the significant positive correlation between a worker’s ability and cost implies that, when employers lose access to signals and workers are forced to compete more intensely on wages, the prevailing workers with lower bids tend to have lower abilities. Third, since workers’ observable characteristics are poor predictors of their ability, employers have little to no information to distinguish between high and low-ability workers.

These changes to hiring patterns lead to a 5% reduction in average wages, a 1.5% reduction in overall hiring rate per posted job, a 4% reduction in worker surplus, and a small, less than 1%, increase in employer surplus.

The overall takeaway from both papers is that generative AI reduces the quality of signals to employers. They don't speak directly to the quality of education signalling, but we can infer that if the quality of other signals of worker quality are reduced by generative AI, then the quality of the education signal likely is as well. That's because proposals and cover letters on Freelancer.com play much the same signalling role as degrees and grades. In both cases, employers can’t observe ability directly, so they rely on an observable, costly signal. On Freelancer.com, that is the proposal or cover letter, and for education, that is the degree or grade. Generative AI makes it much easier for almost anyone to produce a polished proposal or assessment, so the observable output becomes less tightly linked to ability, weakening the value of both kinds of signal.

Read more:

• The impact of large language models on job interviews, from both sides
• David Deming on generative AI and commitment to learning, and the impact of generative AI on signalling in education

David Deming on generative AI and commitment to learning, and the impact of generative AI on signalling in education

When I was writing yesterday's post on generative AI and the economics major, I really wished I had read this post by David Deming on generative AI and learning, and then I could have linked the two together. Instead, I'll use this post to draw on Deming's ideas and flesh out why I think that generative AI makes signalling in education harder, and why that is a problem (in contrast with Matthew Kahn, who as noted in yesterday's post thinks that generative AI reduces problems of information asymmetry).

First, Deming writes about the tension in education between students' desire to learn, and their desire to make life easier (the 'divided self', drawing on the example of Odysseus:

A vivid illustration of our divided self comes from a famous behavioral economics paper called “Tying Odysseus to the Mast: Evidence from a Commitment Savings Product in the Philippines”. They found that customers flocked to and greatly benefited from a bank product that prevented them from accessing their own savings in the future. Just like when Odysseus tied himself to the mast of his ship so that he would not be tempted by the alluring song of the Sirens...

The Sirens offer Odysseus the promise of unlimited knowledge and wisdom without effort. He survives not by resisting his curiosity, but by restricting its scope and constraining his own ability to operate. The Sirens possess all the knowledge that Odysseus seeks, but he realizes he must earn it. There are no shortcuts. This is the perfect metaphor for learning in the age of superintelligence.

The analogy to generative AI is obvious. Generative AI is a tool that offers unlimited knowledge without effort, but using that tool means that the effort necessary for genuine learning is not expended. As Deming concludes:

Learning is hard work. And there is now lots of evidence that people will offload it if given the chance, even if it isn’t in their long-run interest. After nearly two decades of teaching, I’ve realized that my classroom is more than just a place where knowledge is transmitted. It’s also a community where we tie ourselves to the mast together to overcome the suffering of learning hard things.

How does this relate to the quality of signalling? It is worth reviewing the role of signalling in education, as I discussed in this post:

On the other hand, education provides a signal to employers about the quality of the job applicant. Signalling is necessary because there is an adverse selection problem in the labour market. Job applicants know whether they are high quality or not, but employers do not know. The 'quality' of a job applicant is private information. High-quality (intelligent, hard-working, etc.) job applicants want to reveal to employers that they are hard-working. To do this, they need a signal - a way of credibly revealing their quality to prospective employers.

In order for a signal to be effective, it must be costly (otherwise everyone, even those who are lower quality job applicants, would provide the signal), and it must be costly in a way that makes it unattractive for the lower quality job applicants to attempt (such as being more costly for them to engage in).

Qualifications (degrees, diplomas, etc.) provide an effective signal (they are costly, and more costly for lower quality applicants who may have to attempt papers multiple times in order to pass, or work much harder in order to pass). So by engaging in university-level study, students are providing a signal of their quality to future employers. The qualification signals to the employer that the student is high quality, since a low-quality applicant wouldn't have put in the hard work required to get the qualification.

What does generative AI like ChatGPT do to this signalling? When students can outsource much of the effort required to complete assessments, then not-so-good students no longer need to spend more time or effort to complete their qualification than do good students. Take-home assignments, essays, or written reports might be completed to a passing standard with little effort from the student at all. Completing a qualification is no longer costly in a way that makes it unattractive for lower quality job applicants to attempt. That means that employers would no longer be able to infer a job applicant's quality from whether they completed a qualification or not.

A solution suggested by Deming's post is for students to find some way of committing themselves to not using generative AI in assessment. For this to solve the signalling problem, the commitment has to be credible (believable), such as being verifiable by potential employers later. While students could commit themselves to not using generative AI, and maintaining effortful learning, it is difficult to see how students who do so could credibly reveal that they have done so. They require some way of ensuring that potential employers could verify that the student didn't use generative AI. This is where universities could step in. If universities can certify that particular qualifications were 'AI-resistant', such as where assessment includes substantial supervised, in-person components (for example, tests or examinations), then that would help maintain the quality of the education signal. There are other options of course, including oral examinations, group or individual presentations, or supervised practice assessments that make learning harder to fake. However, anything that falls short of being AI-resistant in the eyes of employers is unlikely to work. However, limiting assessment styles in order to certify effortful learning doesn't come without a trade-off. AI-resistant assessment is likely to be less accessible, less flexible, less authentic, and potentially more likely to promote anxiety in students.

Kahn suggested in his post that "AI-proctored assessments and virtual tutors suddenly make effort and mastery visible in real time". That could work. However, AI proctoring by itself is not a solution. In order to retain its status as a signal of quality for students, assessments need to require more effort to complete well for not-so-good students than for good students. Having an assessment where an AI proctors while a student uses a generative AI avatar to make an AI-generated presentation is not going to work. I'm sure that's not what Kahn was envisaging. Proctoring of online assessment (either by humans or by AI) is not as easy as it sounds. Last year I was part of a group tasked with evaluating online proctoring tools, to be rolled out for our new graduate medical school, and I was left thoroughly underwhelmed. All of the tools that we evaluated seemed to have simple workarounds that moderately tech-savvy students could easily employ. The solution that was offered (when the demonstrators could even offer a solution) was to have students complete assessments on-site, which more or less defeats the purpose of online proctoring.

Anyway, the point is that generative AI reduces the signalling value of education. There are solutions where that signalling value can be retained, but that requires students to commit to effortful learning, and universities to certify that effort in a way that students who don’t expend it cannot mimic.

[HT: Marginal Revolution]

Read more:

• Some notes on generative AI in higher education
• Some notes on generative AI and assessment (in higher education)
• More notes on generative AI and teaching (in higher education)
• Universities' (and teachers') cheap talk on generative AI and assessment
• Simas Kucinskas on AI, university education, and the 'mushy middle'
• The wicked problem of generative AI and assessment
• Matthew Kahn on generative AI and the economics major

Conflicting signals from attractiveness and education may reduce the chance of getting a job

There is fairly robust evidence of a beauty premium in the labour market (see the links at the end of this post for some examples). More attractive people earn more than less attractive people, ceteris paribus (holding everything else equal). One of the arguments for why this correlation may be causal (that is, attractiveness causes better labour market outcomes) is that attractiveness is a signal that conveys that the person will perform better in a particular job. [*] So, employers might employ more attractive people, thinking that they will be better workers. Another signal of good job performance is education. Employers employ more highly educated people, thinking that they will be better workers. But what happens if these two signal conflict?

That is the question that is addressed in this new article by Christopher Marquis (University of Cambridge), András Tilcsik (University of Toronto), and Ying Zhang (Singularity Academy), published in the American Journal of Sociology (ungated earlier version here). They hypothesise that:

HYPOTHESIS 1.—For higher-status positions, employers will favor applicants with both physically attractive looks and elite educational credentials over applicants who lack either or both of those valued status dimensions.

HYPOTHESIS 2.—For lower-status positions, employers will favor applicants without physically attractive looks and elite educational credentials over applicants who possess either or both of those valued status dimensions.

And for those whose education and attractiveness do not send the same signal, Marquis et al. note that:

...perhaps most interesting is the situation of applicants with discrepant status characteristics—those who are more attractive but from a less elite institution and those who possess elite educational credentials but are less attractive. In contrast to expectations from prior research, our theoretical framework implies that status-inconsistent applicants elicit ambiguous expectations and are thus less likely to be seen as a clear fit for either higher-status or lower-status positions.

Marquis et al. test their hypotheses using two studies: (1) a CV audit study in China, where they sent fictitious applications for various jobs and noted the callback rate for fictitious applicants with different characteristics (physical attractiveness, and education); and (2) a survey experiment in the US, where they asked research participants whether a fictitious applicant should be recommended for an interview for a particular job.

In the China study, they applied for jobs in the "sales" category across six cities on the job website Zhaopin.com. They applied once for each job, randomly varying the picture on the CV (since Chinese CVs typically include a headshot photo), and the education of the applicant (distinguishing between local universities ranked 10-39 on Project 985, and those ranked 40-116). Focusing on callback rates, Marquis et al. find that:

Overall, status-consistent combinations led to more callbacks than status-inconsistent combinations at a rate of approximately 1.7 to 1 (11.7% vs. 6.8%). This difference is significant statistically (P < .001) and substantively.

When looking at jobs of different status levels, Marquis et al. then find that:

...when restricting the sample to jobs with above-average salaries... the high-high combination of university status and attractiveness was more likely to lead to an interview invitation than the low-low combination (P < .10), the high-low combination (P < .05), and the low-high combination (P < .01). These differences were substantively significant as well; for example, the callback rate for the high-high combination in this subsample (24.4%) was about 1.7 times as high as for the low-low combination (14.1%)...

...for jobs with below-average salaries, the low-low combination led to more callbacks than any of the other combinations and that callback rates for the other three combinations were not significantly different from one another. The low-low combination had a substantial advantage in this subsample, generating 1.8 times as many callbacks (22.8%) as the high-high combination (12.9%).

This was consistent with their hypotheses. However, is this peculiar to the Chinese context? To test that, Marquis et al. then conducted the survey experiment in the US. They use a sample of 2020 people 'with hiring experience', recruited through the online panel Prolific. They showed each research participant a profile photo (as would be found on a professional social media site like LinkedIn), and the CV, and distinguished education based on three levels:

Using the 2022 National University Rankings of U.S. News and World Report, we selected universities from California, New York state, North Carolina, and Pennsylvania. From each state, we included (a) one institution ranked among the top 25 national universities, (b) one ranked in the 25–75 range, and (c) one not ranked among the top 75 national universities.

Marquis et al. then:

...asked participants to indicate the likelihood that they would interview the candidate for the position and the likelihood that they would give the candidate an offer, both on 1 (low) to 7 (high) Likert-type scales. We averaged these two items into an overall rating of the candidate for the job in question...

Similar to the Chinese results, Marquis et al. find that:

...for the higher-status position, the mean rating for the combination of elite university status and high physical attractiveness was significantly higher than the average of the five other combinations (5.92 vs. 5.53, P < .0003) and significantly higher than the mean of the second most favored combination (5.92 vs. 5.67, P < .04).

And for the lower-status position:

...the overall ratings for the two combinations of nonelite university status and lower physical attractiveness were significantly higher than the average of the other four combinations (6.04 vs. 5.72, P < .001; 6.01 vs. 5.72, P < .003). The ratings of the two combinations of nonelite university status and lower physical attractiveness were also higher than the mean of the next highest combination (6.04 vs. 5.77, P < .014; 6.01 vs. 5.77, P < .027).

Marquis et al. then use the US sample to look into the mechanism driving the results. Unsurprisingly:

...candidates with status-inconsistent combinations might fall between two stools: they are not perceived to be the most qualified candidates when they target higher-status jobs, and they are seen as lacking organizational commitment when they apply for lower-status jobs. Moreover, for both types of positions, these candidates generate greater evaluative uncertainty than other applicants.

When the two signals conflict, evaluators have less certainty about the applicant and rate them lower. This should be a cautionary tale for job applicants: make sure that all of your signals are conveying the same information about your quality as a job applicant, or you may find your application falling to the bottom of the pile.

*****

[*] I'm not sure that I find this argument very convincing, but Marquis et al. rely on a citation to this 1983 article (ungated version here) from a sociology journal to support it. In economics, for a signal to be effective, it must be costly, and costly in such a way that those with lower-quality attributes wouldn't want to attempt the signal. Being attractive is costly (or, at least, personal grooming is). But is it costly in such a way that people with low job performance wouldn't be able (or willing) to send the signal? I have my doubts. Anyway, I've put this argument aside for the moment, because the more interesting point in this article is what happens when two signals are conflicting.

Read more:

• The beauty premium in undergraduate study is small, and more attractive women major in economics
• New results questioning the beauty premium should be treated with caution
• The beauty premium in the LPGA
• The beauty and height premiums in the labour market
• Blonds have more fun (or rather, they get paid more)
• The beauty premium for economists
• More evidence on the blond wage premium
• The beauty premium at the intersection of race and gender
• Narrowing down on the source of the beauty premium
• The beauty premium and student grades for in-person and online education
• Is there a premium for very unattractive workers?

What the persuasiveness of economics experts and signalling tell us about the gender gap in economics

Are the opinions of expert economists persuasive? The answer to that question will no doubt depend on who you ask. And at least some would answer 'yes, but I wish they weren't'. Does the gender of the economist matter for how persuasive their opinion is? That is a much more difficult question to answer. However, this recent article by Hans Sievertsen and Sarah Smith (both University of Bristol), published in the Journal of Economic Behavior and Organization (open access, but see this paywalled FT article as well), provides us with a starting point. Sievertsen and Smith:

...use an information provision experiment... to test whether the opinion about a topical policy issue expressed by a senior female economist is more, or less, persuasive than the same opinion expressed by a senior male economist. We run the same experiment twice – the first time, members of the public are shown credentials of expertise, the second time they are not.

Specifically, Sievertsen and Smith survey people in the US, asking their opinion on a range of issues (rated on a five-point scale from 'strongly disagree' to 'strongly agree'). Alongside a statement of the issue, each research participant was provided with the opinion of an economist, drawn from the US Economic Experts Panel run out of the University of Chicago. By comparing the results from the survey with those of an earlier survey of people who didn't get to see an economist's opinion, the persuasiveness of the economist whose opinion is provided (on each issue) can be evaluated.

The ten issues are based on the following statements:

1. Use of artificial intelligence over the next ten years will lead to a substantial increase in the growth rates of real per capita income in the US and Western Europe over the subsequent two decades.

2. There needs to be more government regulation around Twitter’s content moderation and personal data protection.

3. It would serve the US economy well to make it unlawful for companies with revenues over $1 billion to offer goods or services for sale at an excessive price during an exceptional market shock. (Price Gouging)

4. Efforts to achieve the goal of reaching net-zero emissions of greenhouse gases by 2050 will be a major drag on global economic growth.

5. Given the centrality of semiconductors to the manufacturing of many products, securing reliable supplies should be a key strategic objective of national policy.

6. A significant factor behind today’s higher US inflation is dominant corporations in uncompetitive markets taking advantage of their market power to raise prices. (Greedflation)

7. Financial regulators in the US and Europe lack the tools and authority to deter runs on banks by uninsured depositors.

8. When economic policy-makers are unable to commit credibly in advance to a specific decision rule, they will often follow a poor policy trajectory.

9. A windfall tax on the profits of large oil companies ‚ with the revenue rebated to households‚ would provide an efficient means to protect the average US household.

10. A ban on advertising junk foods (those that are high in sugar, salt, and fat) would be an effective policy to reduce child obesity.

The economists' actual views on each of the statements is not important for the research question (although the article does tell you, and you can probably guess for some of them what the average opinion of the economists is). Sievertsen and Smith first evaluate how persuasive economists are in general, finding that:

On average, a one-point change on the Likert scale in expert opinion is associated with a 0.17 point change in public opinion... Expert opinions have no effect on public opinions about Greedflation, while there are stronger effects for Price Gouging, Financial Regulation and Economic Policy. There is some support for the argument... that persuaders are more effective when receivers are less certain: The degree of persuasiveness is weaker on issues where baseline public opinion is more certain... The degree of persuasiveness is also stronger on issues where there is less distance between sub-panel expert opinion and baseline public opinion... this suggests that experts may be perceived as less credible when their views are further out of line with those of the general public.

So, the opinions of economic experts are convincing (somewhat). The last point is interesting though - people are most convinced when the economist's views are similar to those of the general public. This suggests some confirmation bias - people believe the experts more when the experts agree with them! It is also interesting who is persuaded most:

The degree of persuasiveness is greater for men [p = 0.002] and for non-whites [p = 0.000]. It is also greater for those with a degree [p = 0.000] and for those with higher self-reported economics knowledge [p = 0.000]. Those who identify as Republicans are also more persuaded by economists’ opinions than Democrats/Independents [p = 0.030].

Those who are more educated, and who claim to have more economics knowledge, are more persuaded by expert economists. More educated people likely give more credence to the views of other educated people, while those who claim to know more economics are more likely to modify their views to fit with those of economics experts.

What about gender differences in persuasion? Sievertsen and Smith switch the analysis to evaluating whether the opinion of the research participants exactly matches that of the expert whose opinion they are provided, and find that:

...members of the public are 1.1 percentage points more likely to match with the opinion of a female expert than with the same opinion expressed by a male economist.

Sievertsen and Smith find similar results when evaluating the distance between the opinion of the research participant and the expert (the distance is smaller for female experts). The effect (1.1 percentage points, against a match rate for male experts of 33.5 percent) seems quite small, but Sievertsen and Smith note that some matches would happen purely by chance, and after accounting for that:

...the degree of persuasiveness of female expert opinions is around 20 per cent higher {than male experts]...

Why are female economics experts so persuasive? Here's where things get interesting. Sievertsen and Smith run their survey a second time. The first time around, research participants were told the name and institutional affiliation of the expert economist (as well as shown their profile photo). In the second survey, research participants were only told the name of the expert economist (and shown the photo). In that second survey:

The overall effect of

female expert on the probability of matching opinion drops from 0.011 in the main experiment to 0.0002 in the follow-up.

The extra persuasiveness of female experts (over and above male experts) disappears! Sievertsen and Smith conclude that:

...in the first experiment, credentials provided an information signal that favoured senior female experts. Removing that signal in the follow-up experiment eliminates the gender difference.

It is worth explaining that last result in a bit more detail, because what it really shows is that the general public recognises the gender bias in top economics institutions.

The quality of a purported economics expert is private information. it is known to the expert themselves, but not known to the public. This is a case of asymmetric information. The expert is the informed party, and the general public is the uninformed party. Since the general public cannot tell high-quality and low-quality experts apart, they might assume that all experts are low quality. How can a high-quality expert instead convince the general public of their high quality? They must credibly reveal their quality to the public - this is called signalling.

For a signal to be effective, it must meet two conditions: (1) it must be costly; and (2) it must be costly in such a way that those with low quality attributes would not want to attempt the signal. Getting a tenured position is signal of high quality for experts. It is costly to get such a position, and it is costly in a way that low-quality experts wouldn't want to attempt it (because they would be unsuccessful in getting tenure anyway). So, a position at a top university is a signal of quality.

Now, why would this signal be even more effective for female economists? If getting a tenured position at a top institution is even more costly for female economists than for male economists, then the quality of the signal is higher for women than for men. And therefore, the general public would be even more believing of the signal for female economists than for male economists. The gender gap in economics is pervasive (see this post, and the posts linked at the bottom of that post). What is interesting is that this gender gap is so well established that the general public is acting on it!

Sievertsen and Smith finish by pointing out a bit of a paradox:

...if senior female economists have greater credibility in the eyes of the public, then why are they less confident in giving their opinion. This remains an open question.

Indeed. Hopefully, at the margin, this research will convince senior female economists to use their persuasiveness for the good of all (where 'good' is defined as persuaded more people to believe in economists' opinions).

Junk papers are poisoning the well for meta-analyses

Regular readers of this blog will know that I am quite fond of meta-analyses. In fact, one of my PhD students will soon publish her first working paper from her thesis, which is a very ambitious (perhaps too ambitious) meta-analysis (more on that in a future post). Anyway, a meta-analysis is where a researcher combines the results of many studies to estimate an overall effect. So, while individual studies may have various biases (and that is one reason why you shouldn't over-emphasise the importance of any one study on its own), a meta-analysis is much less subject to bias.

At least, that was the case until recently. As Holly Else reported in Science last November:

Aquarius, a neuroscientist who specializes in reviewing preclinical animal research, and Wever, a metascienist, both at Radboud University Medical Center, are some of a growing number of systematic review authors who have lost faith in the evidence base they depend on. Their group put its project on hold to quantify the problem. “There is a real danger of systematic reviews losing the power they have,” Aquarius says.

The junk papers are likely the products of paper mills—businesses that produce fake science to order. The size of the problem is not clear, but a manuscript posted to the Center for Open Science’s OSF preprint server in September suggests up to one in seven published papers are fabricated or falsified...

This creates real problems for researchers who are trying to conduct credible meta-analyses. It means taking a much higher level of care with what research papers are included. However, filtering out the junk papers may also lead to 'real' papers also being filtered out, reducing the accuracy of the resulting meta-analysis.

The problem is only going to get worse. Else's article doesn't even consider the impact that generative AI is having and will continue to have on the publication process. In future, it will likely become more difficult to verify the credibility of research papers. Some will be 'real', some will have real data and analysis but be largely written by generative AI (for example, see here), while others will be the direct product of generative AI with made-up data and analyses.

Else does identify some potential solutions:

In 2023, Cochrane, an international network promoting evidence-based medicine, issued draft guidelines to help these researchers filter out junk science. The REAPPRAISED checklist, an effort by another group of research integrity specialists published in 2020, also helps researchers assess papers’ soundness.

The problem is that the guidelines don't seem well-designed to filter out AI-generated studies that appear to be of high quality. Moreover, having set guidelines just invites bad researchers to set up their junk papers to tick the right boxes to pass review.

In my view, we may need to rely on the open science movement in order to thwart the potential flood of junk papers. In particular, making data and statistical code available may become an important signal of paper quality in the future. [*] Authors would only be willing to make the data and code available if their paper is 'real'. If their paper is not 'real', then they would not be willing to do so, as they run the risk of being found out.

So, while junk papers may be poisoning the well of science right now, we might have an antidote. It is not time to despair, but for the time being researchers conducting meta-analyses will have to take a lot of extra care.

*****

[*] For a signal to be effective, it must meet two conditions: (1) it must be costly; and (2) it must be costly in such a way that those with low quality attributes would not want to attempt the signal. Making data and statistical code available seems to me to meet both of those conditions. It is costly (in terms of time and effort) to clean up and label a dataset and statistical code to be understood or used by others. And if your paper is low-quality, you wouldn't want to expend the additional time and effort necessary to make your data and code available.

Are restaurant awards a negative signal?

I've been thinking about this for some time. If you are at a restaurant, and you spy a bunch of award certificates on the wall, should this give you confidence that this is a good restaurant? Many people would say 'yes, of course'. It is an award-winning restaurant, so of course it is a good restaurant. I'm not so sure, and in fact I think that in many cases a bunch of restaurant awards on the wall might be a negative signal, or no indicator at all. Let me explain why.

First, let's consider why restaurants need to signal at all. Restaurant quality is what economists call private information. The restaurant operators know whether their restaurant is high quality or not, but the patrons do not know (at least, not until they have eaten there). Restaurant meals are an experience good - the quality of the restaurant (an experience characteristic) is only known to the patron after they have experienced the restaurant meal. This information asymmetry between the restaurant operators and restaurant patrons creates an adverse selection problem. If high-quality restaurants can't distinguish themselves from low-quality restaurants, then the restaurant patron will assume that every restaurant is a low-quality restaurant (this is a pooling equilibrium). If this was to happen, then high-quality restaurants would probably start to close down (because they won't be able to get enough additional patrons to pay for their higher costs), and we'd be left with only low-quality restaurants left.

Of course, that scenario doesn't play out in most cities. Restaurants have found ways to make their quality known to patrons. When the informed party (the party that doesn't know the private information) tries to credibly reveal the private information, this is what economists call signalling. [*] In order for a signal to be effective, it needs to meet two conditions. First, it must be costly. And second, it must be costly in such a way that the low-quality restaurants wouldn't want to attempt the signal.

Now, let's consider the signalling value of award certificates. Is an award certificate costly? Yes, getting an award certificate is costly for a restaurant. The restaurant operators would have to work hard on quality of food and ingredients, and the quality of service, in order to win a certificate. [**] Is an award certificate costly in such a way that the low-quality restaurants wouldn't want to attempt the signal? A low-quality restaurant is low quality because of poorer quality food and service (and whatever other criteria are considered in the awarding of the certificate). If the low-quality restaurant operators were willing to pay the costs of higher quality food and ingredients and higher quality service, then they would do so (and they would be a high-quality restaurant, not a low-quality restaurant). So yes, the low-quality restaurant operators wouldn't attempt the signal, and so it does appear that award certificates are a good signal of quality.

However, not every high-quality restaurant can win an award certificate every year. So, what if the certificates on the wall are five or ten years old? Is that still a good signal of quality? I look at those ancient certificates (more than five years old), and I see a restaurant that is past its peak. It's on a downward slide into oblivion. It's not signalling high quality to me today, it's signalling that the restaurant used to be high quality, but not any longer.

There are two reasons why a restaurant would keep ancient award certificates on the wall. First, they are genuinely on the downward slide, but they want patrons to think that they are still high quality. But, if they were genuinely still high quality, they would still be winning award certificates. Given that they are not winning, then they are really signalling to patrons that they haven't maintained their previous quality.

Second, maybe the award certificates are given out in such a way that multiple winners are excluded. Call this the 'give everyone else a chance' effect. In that case, the award certificates are not such a great signal of quality, since they are not awarded to the highest quality restaurants, but to the highest quality restaurants that haven't been excluded for winning too many times. The quality of the signal is lessened, and patrons shouldn't pay as much attention to the award certificates.

So, a real high-quality restaurant, that hasn't won an award for several years, would probably be better off taking down their ancient award certificates, if they don't have any recent ones, then leaving those ancient certificates on the wall. Ancient restaurant award certificates are a negative signal of quality.

Changing tack now, I have a bunch of ancient award certificates myself - my teaching awards (six of them in total). I've been thinking about hanging them on my office wall. However, I haven't won an ward since 2020. Now that I've thought about restaurant signalling, I'm reconsidering whether I should hang those teaching award certificates on my office wall. Would they just signal that I'm on a downward slide? On the other hand, I continue to be nominated every year, and by multiple students every year. Maybe my lack of award winning is just the 'give everyone else a chance' effect at play?

[HT: Inspired by this Tyler Cowen post from December last year]

*****

[*] The patrons (the uninformed party) can also try to reveal the private information about restaurant quality. When the uninformed party (the party that doesn't know the private information) tries to credibly reveal the private information, this is what economists call screening. 

[**] If that wasn't the case, then the certificate provides no signalling value at all.

Information asymmetry, adverse selection, and large language models

In an article in The Conversation earlier this week, Nicholas Davis (University of Technology Sydney) wrote about information asymmetry as it relates to large language models (like ChatGPT):

A lack of skills and experience among decision-makers is undoubtedly part of the problem. But the rapid pace of innovation in AI is supercharging another challenge: information asymmetry.

Information asymmetry is a simple, Nobel prize-winning economic concept with serious implications for everyone. And it’s a particularly pernicious challenge when it comes to AI...

AI creates information asymmetries in spades. AI models are technical and complex, they are often embedded and hidden inside other systems, and they are increasingly being used to make important choices.

Balancing out these asymmetries should deeply concern all of us. Boards, executives and shareholders want AI investments to pay off. Consumers want systems that work in their interests. And we all want to enjoy the benefits of economic expansion while avoiding the very real harms AI systems can inflict if they fail, or if they are used maliciously or deployed inappropriately.

Davis is generally talking about information in relation to AI safety, but I want to take this post in a slightly different direction first, about AI quality. I will return to AI safety at the end.

When we consider AI quality, information asymmetry will cause a problem of adverse selection, ultimately causing the market for high-quality large language models to fail. So, let's consider how this market fails. There is information asymmetry in relation to the quality of large language models. AI firms know whether their large language model (LLM) is high quality or not, but users do not know. The 'quality' of a large language model is private information. Because of this information asymmetry, a LLM user should assume that any LLM is low quality. This creates a pooling equilibrium, where the user is only willing to pay however much they value access to a low-quality LLM. AI firms with high-quality LLMs, which are more expensive to train and run, will not be willing to accept users who are only willing to pay a small amount. These AI firms with high-quality LLMs drop out of the market. The only AI firms that are left will be those with low-quality LLMs. The market for high-quality LLMs fails.

A key part of the problem here is that LLMs may be experience goods. Experience goods are goods where the consumer doesn't know the quality (or some other important attribute) until after they have consumed it (they are called experience goods, because consumers don't find out what they have really bought until they have experienced it). The quality of a LLM isn't revealed to the user until after they have used it, and seen what the LLM outputs in response to a given prompt. However, users are not experts, and they might not even know the quality of the output after they receive it. That would make LLMs not experience goods, but credence goods (goods where some of the characteristics, the credence characteristics, are not known even after the good has been consumed). It seems likely that for most non-expert LLM users, LLMs will be a credence good.

So, how can the adverse selection problem be solved for LLMs? In other words, how can we move from the pooling equilibrium to a separating equilibrium (where LLM users can separate the high-quality LLMs from the low-quality LLMs)? There are two options: (1) screening; and (2) signalling. Screening involves the uninformed party (the LLM user) trying to reveal the private information (the quality of the LLM). This is unlikely to work well, because the LLM user typically isn't an expert. The only way that they can reveal the quality of the LLM themselves is to try it out, but if LLMs are actually a credence good, then even trying them out won't reveal their quality.

That leaves signalling, which involves the informed party (the AI firm) credibly revealing the private information. Only an AI firm with a high-quality LLM would want to attempt signalling, to try and reveal that their LLM is high quality (and AI firm with a low-quality LLM is unlikely to want to reveal that their LLM is low quality). In order for a signal to be effective, it needs to meet two conditions. First, it must be costly. And second, it must be costly in such a way that the AI firms with low-quality LLMs wouldn't want to attempt the signal.

What might be good signals for an AI firm with a high-quality LLM? Firms that sell experience goods have come up with a number of options, including free trials, testimonials from other customers, branding or advertising. In this context, a free trial might not work. If LLMs are a credence good, then a free trial won't reveal the private information (and we would expect to see AI firms with low-quality LLMs also offering free trials). So, while free trials are costly (they come with an opportunity cost of foregone income for the AI firm), they don't appear to be costly in a way that AI firms with low-quality LLMs wouldn't want to attempt them.

Testimonials from other customers might be an option, but only if the LLM user can trust those other customers. So, testimonials from acknowledged experts in AI might be worthwhile, but testimonials from other non-expert users might not. Are testimonials costly to the firm? Probably not greatly, but they will be more costly for an AI firm with a low-quality LLM. They might have to pay for a testimonial, whereas an AI firm with a high-quality LLM might receive praise from an expert without payment. The problem here, though, is that a non-expert user may not be able to tell the 'experts' giving testimonials apart, and so may not trust any testimonial.

What about branding or advertising? These are costly to a firm, but it's unlikely that they are costly in a way that would prevent AI firms with low-quality LLMs from attempting them.

All of this is bad news for the LLM user. They can't tell high-quality LLMs and low-quality LLMs apart, they probably can't use screening to help tell them apart, and the AI firms with high-quality LLMs probably can't use signalling to reveal that their LLM is high quality. We're left in a situation where the LLM user just has to assume that any given LLM is low quality (the pooling equilibrium).

So, what's left? Davis suggests that:

Well-designed guardrails will improve technology and make us all better off. On this front, the government should accelerate law reform efforts to clarify existing rules and improve both transparency and accountability in the market.

Of course, this is intended as a solution to information asymmetry about AI safety, not a solution to the adverse selection problem related to AI quality. However, now that we've considered the market failing in relation to AI quality because quality is a credence characteristic, it might be worth considering: is AI safety also a credence characteristic?

If AI safety is a credence characteristic, then safety isn't revealed even after the LLM has been used. Then we end up in a similar situation for AI safety as we do for AI quality, in that there is little that users can do to separate the safe LLMs from the unsafe LLMs. We would have a AI safety pooling equilibrium, where all LLM users would have to treat LLMs as potentially unsafe. Since LLM users would not be willing to pay more for LLMs that are safe (since they couldn't be sure about safety), and because safety presumably costs AI firms more, the AI firms with safe LLMs will start to drop out of the market. That would leave only the unsafe (but regulated) LLMs available, which seems like a suboptimal outcome. This isn't an argument against AI safety regulation, but hopefully provides some food for thought about its possible consequences.

Consumer preferences may explain the pricing puzzle that is diamond engagement rings

On the Marginal Revolution blog last month, Alex Tabarrok presented a bit of a puzzle:

DeBeers also produces lab-grown diamonds and they have a very strange pricing strategy:

De Beers started selling its own lab-grown diamonds in 2018 at a steep discount to the going price, in an attempt to differentiate between the two categories. The company expects lab-grown prices to continue to tumble, in what it sees as a tsunami of more supply coming on to the market, Rowley said. That should create an even bigger delta in prices between natural diamonds and lab grown, helping differentiate the two products, he said.

What? Ordinarily, the bigger the price between a competitor and its substitute the greater pressure on the competitor to lower prices! Yet DeBeers is gambling that the bigger the difference in price between natural and lab grown diamonds the bigger the demand for natural diamonds! Strange. The only way I see this working is if the fiancée knows the price of the ring, which maybe they do! In that case, the buyer still has to spend 10k and doesn’t care whether it’s 10k on synthetic diamonds or 10k on natural grown diamonds. But 10k on synthetic diamonds will get you more carats so we need an equilibrium in which a smaller diamond signals more expensive. But that runs against hundreds of years of expectations! And remember natural and lab grown diamonds are indistinguishable by the naked eye. It’s one thing for the fiancée to know the price of the diamond but surely her friends judge by what they can see, namely the size of the ring. Which signal is the most important to send?

The quote embedded above is from this Bloomberg article. In producing its own lab-grown diamonds, De Beers is competing with itself (as I noted in this post in 2018). In that post, I noted that De Beers was engaged in rent-seeking behaviour:

Synthetic diamonds are a close substitute for natural diamonds, and natural diamonds represent a highly profitable business for De Beers. If some of De Beers' competitors (like Chatham or Diamond Foundry) start producing high-quality synthetic diamonds and selling them relatively cheaply, then some diamond consumers will be induced to switch from natural diamonds to synthetic diamonds, and De Beers will lose profits. By selling synthetic diamonds itself at a very low price, clearly De Beers' goal is to make it unprofitable for the synthetic diamond producers to operate, by seriously undercutting their prices - a tactic known as predatory pricing. The synthetic diamond competitors won't be able to compete with De Beers for long at the low prices, and will soon go out of business (at least, that is what De Beers is hoping). At that point, De Beers can quietly exit the synthetic diamond industry and resume claiming high profits from natural diamonds (having taken a bit of a hit to its profits from both synthetic and natural diamonds in the meantime).

I noted that the problem with that strategy for De Beers is that there needs to be some barrier to entry to keep competitors out once De Beers had exited the synthetic diamond industry and wanted to resume high profits in the natural diamond industry. Tabarrok presents a different problem, and potential puzzle: if consumers can't tell a lab-grown (synthetic) diamond and a natural diamond apart, why would lab-grown diamonds sell at a discount?

I thought that the solution related to signalling, as Tabarrok suggested. That is, until I started writing this post, and then ended up back at the puzzle. But I think I have a solution. So, let's work it through. As I noted in this 2015 post, spending on engagement rings (and weddings, as I noted here) is a signal. In fact, there are two potential signals. But first, let's take a step back and think about why signalling is necessary.

Signalling is a solution to an adverse selection problem. Adverse selection arises when one of the parties to an agreement has private information that is relevant to the agreement, and they use that information to their own advantage and to the disadvantage of the other party. Adverse selection is a problem of 'pre-contractual opportunism', and can cause markets to fail. Signalling is one way that markets have adapted to deal with adverse selection problems. With signalling, the informed party finds a way to credibly reveal the private information to the uninformed party (a signal). There are two important conditions for a signal to be effective: (1) it needs to be costly; and (2) it needs to be costly in such a way that those with lower quality attributes would not want to attempt the signal (and one way this criteria is fulfilled is if it is more costly for those with lower quality attributes).

In the case of engagement rings, there are two adverse selection problems that the engagement ring signal might solve. First, the person proposing marriage has private information about their quality as a future partner. The person receiving the proposal may not know this information. If the proposer is a low-quality partner, they have an incentive to hide this information until it is too late and they are already married! The engagement ring provides an effective signal, that reveals the quality of the proposer. An engagement ring is an effective signal because it is costly, and because it is costly in a way that a low-quality partner would not want to attempt. That's because the proposer would lose their investment in the ring if they are low quality and the relationship breaks up. Going one step further, the more valuable the ring (relative to the income of the proposer), the higher the quality of the signal.

The second adverse selection problem arises because the couple have private information about their quality as a couple (as I noted in this post about the cost of weddings). The people attending their wedding likely do not know this information. The couple receives gifts at their wedding, and wedding attendees are more likely to want to provide gifts to couples who will be successful. So, low-quality couples have an incentive to hide from wedding attendees that they are low quality. Is an engagement ring a good signal here? I'm not so sure. Yes, the engagement ring is costly. But is it costly in such a way that a low-quality couple wouldn't want to attempt? A low-quality couple could buy a wedding ring, receive the wedding gifts, then divorce and sell the engagement ring to re-coup their initial outlay. It seems to me that this signal is not nearly as effective as the overall cost of a wedding (which is not recoverable).

Ok, so I think we have established that an engagement ring is an effective signal from one partner to another, but not necessarily from a couple to others. A more valuable ring provides a higher quality signal. Does whether the diamond is natural or lab-grown matter? In theory, it shouldn't - they are indistinguishable to the untrained eye. However, in practice they may be very different. Natural diamonds are scarce. Lab-grown diamonds are less scarce. Perhaps, in the minds of consumers, a natural diamond is a more desirable product, even if they are indistinguishable? After all, a motivated consumer can find out (through an expert appraisal) whether a diamond is natural or lab-grown. In that case, the perceived quality of the diamond really matters - a smaller natural diamond would provide the same quality of signal as a larger lab-grown diamond. And for two diamonds of the same size, a natural diamond would be priced higher than a lab-grown diamond. In other words, consumer preferences (for natural rather than lab-grown diamonds) would explain this pricing puzzle.

Certification as a barrier to entry for doggy daycare centres

This week, my ECONS102 class covered monopolies. Monopolies arise because of barriers to entry - there is something that stops other firms from getting into the market and competing with the monopoly. Not all markets with barriers to entry result in a single firm operating (a 'pure' monopoly). However, all markets with barriers to entry convey some market power on the firms - the firms can set a price that is above their marginal cost, and make a profit.

One way that barriers to entry arise is when the government grants an exclusive right to produce and/or sell some good or service, or where being a seller requires a license. Patents are an example of an exclusive right granted by the government, while occupational licensing (like the licenses required to practice medicine, or to be a teacher or taxi driver) is an example of the latter.

A slightly weaker (but similar) form of barrier to entry to occupational licensing is created by a certification regime. Certification doesn't keep sellers out of the market, but it does convey some information to consumers. It is a form of signal of the quality of the seller. For signals to be effective, they must be costly (and a certification involves a cost to the seller who wants to be certified), and costly in a way that low-quality sellers would not want to attempt. Certification may impose restrictions on the practices of sellers, like requiring them to follow a code of practice, having a third party audit the operations of certified sellers, and/or having a public register of certified sellers, as well as a record of sellers who have lost their certification. Any of these characteristics of the certification regime would make becoming certified costly for low-quality sellers in such a way that they wouldn't want to become certified in the first place. And because the certification limits the number of willing sellers, it creates a barrier to entry.

Which brings me to this article from the New Zealand Herald from February:

New Zealand’s one and only SPCA-certified doggy daycare in Whangārei is urging other daycare centres to sign up so owners can start to “expect the best” for their pets.

The Grooming Lounge & Daycare is the first doggy daycare to join the SPCA-certified programme, which lets dog owners know of businesses willing to be independently audited to maintain high animal welfare standards.

Owner Rebekah Thompson said she signed up after a futile attempt to find daycare standards and guidelines when starting her business in 2020.

“There’s no guidelines or certification for doggy daycares, it’s completely unregulated as an industry...”

Currently in New Zealand, anyone can set up a doggy daycare facility with no minimum experience or qualifications.

SPCA developed a set of voluntary standards so businesses could raise the bar and help pet owners choose centres that put animal welfare first.

It costs $800 a year to get certified and businesses get audited twice a year. One of the visits is scheduled and the other is unscheduled.

“I went with it to show my customers I have nothing to hide, and that a third party is validating what we’re doing,” Thompson said.

Notice that the certification, as described in the article, corresponds almost exactly to the case I laid out at the start of the post. It provides a signal that "lets dog owners know of businesses willing to be independently audited to maintain high animal welfare standards". The signal is costly ($800 a year), and the twice-yearly audits will make the certification unattractive for low-quality doggy daycare centres, since it would reveal the real quality of the centres to dog owners.

What happens next will be interesting though. This could be just the start of a creeping regime of regulatory capture. The SPCA and certified doggy daycare operators could start lobbying government for mandatory regulation of the sector - note the scare quote in the article: “There’s no guidelines or certification for doggy daycares, it’s completely unregulated as an industry...” The government might introduce licenses for doggy daycare centres. That would make the barrier to entry into the doggy daycare market a bit higher. Who better to manage the licensing regime than the SPCA? Then, the requirements for getting a licence can be gradually raised, in order to "put animal welfare first". Again, the barrier to entry gets a bit higher.

Will we end up with high barriers to entry into the doggy daycare sector? If it was up to the SPCA and the already-certified operators, then yes. Watch this space.

Shareholders of dating apps should be freaking out right now

My ECONS102 class has been covered the economics of information this week, including asymmetric information and signalling. So, it is timely to revisit the topic of online dating (which I have discussed earlier this year, here and here), particularly given the rise of artificial intelligence. As the Financial Times reported last month (paywalled):

Terrible news from the world of online dating. As if a parade of dubious romantic prospects and dead-end chats wasn’t bad enough, artificial intelligence has dipped its toe into the dating pool.

Eleven years ago, Tinder helped to turn dating into a series of quick-fire interactions on the internet. But for some jaded users, even writing “Hi” to a romantic prospect is now too much effort. Tech start-ups such as Rizz and YourMove AI are gaining a foothold in the sector by offering AI assistance in creating witty opening lines and appealing profiles.

Meeting strangers on the internet is by nature a random affair. Artificial chat is at least a less sinister way to help that process along than asking users to swab the insides of their mouth, as DNA-dating app Pheramor once did. But a proliferation of AI-assisted conversations suggests that eventually dating apps will simply be full of computers trying to woo other computers.

One start-up even offers the chance to watch this exact premise unfold. Teaser AI asks users questions about themselves and their personalities and then crafts AI-generated chat that is designed to mimic them. When individuals match, they can sit back and watch as their chatbots try to chat each other up.

I'm sure I wasn't the first to point out that we would eventually find ourselves in a world where one instance of ChatGPT was chatting up another instance of ChatGPT, with no humans involved. On top of being a somewhat amusing situation, there is a downside in this development. It will simply make it more difficult for someone genuinely looking for love to find a match online.

To see why, we need to discuss adverse selection. Adverse selection arises when one of the parties to an agreement (the informed party) has private information that is relevant to the agreement, and they use that private information to their own advantage at the expense of the uninformed party. In the case of online dating, the private information is the quality of each potential date. They know if they are high quality or not, but no one else knows. That could lead to a pooling equilibrium, where every online dating subscriber assumes that everyone they are matched with is low quality (because they can't tell the high quality and low quality dates apart, and assuming that every match is high quality is a recipe for disaster). High quality dates don't want to be treated as if they are low quality, so they drop out of the online dating market. Eventually, the online dating market only has low quality dates. The 'market' for high quality dates fails.

Of course, the market typically hasn't failed. People find love in online dating, so clearly the online dating apps are doing something to reduce the adverse selection problem. One thing that they do is offer the ability for subscribers to chat with the people they are matched with. Chatting allows the subscriber to work out (albeit imperfectly) who is high quality and who is low quality. When applied by the uninformed party, this is referred to as screening.

It also allows the high-quality dates to signal that they are high quality. To be effective, a signal must meet two criteria. First, the signal must be costly (otherwise everyone, even those who are lower quality dates, would provide the signal). And second, it must be costly in a way that makes it unattractive for the lower quality dates to attempt (such as being more costly for them to engage in). Engaging in witty and flirty conversation (or whatever type of conversation a subscriber is looking for, if not witty and flirty) on a dating app is costly - it takes time and effort. It will also be less costly for high-quality dates, for whom the desired type of conversation may come more naturally. It is much easier to be yourself in conversation than it is to pretend to be someone you are not. So, conversation on dating apps is an effective signal, albeit an imperfect one.

The problem with AI is that it eliminates the signalling value of conversation on the app. An uninformed subscriber should no longer find the conversation on the app to be a credible signal of the quality of the match they are engaging with. They can't use the conversation for screening either. After matching, the best option is likely to be, as I suggested earlier, in-person meet-ups.

The shareholders of online dating services should be freaking out right now. Their business model, which is built on providing high-quality matches, is under threat by AI. Once the novelty of watching their AI wingman chatting up their match's AI wingman wears off, subscribers will realise that paying for the dating app has become a waste of time. In fact, here's the share price performance over the last six months of the three biggest dating app companies, Match Group (MTCH) in blue, Bumble (BMBL) in black, and Grindr (GRND) in green (original data taken from here):

While Match is up 10.4 percent over that time, Grinder is down 19.5 percent, and Bumble is down 25.3 percent. It's not looking good.

Read more:

• Who loses status, and who gains status, with ChatGPT
• AI may be the killer app, as well as the killer of (dating) apps