04 - Collusion Without Deterrence

Pricing algorithms that learn on their own can end up charging high prices, but in our experiments we find little evidence that this happens because they threaten each other.

More and more prices are set by software. Gas stations, airlines and online retailers hand the decision to algorithms that adjust prices continuously, and the field evidence shows this changes how firms compete (Brown and MacKay, 2023; Assad et al., 2024). The worry that follows is old in economics and new in form: what if these algorithms learn to collude, keeping prices high without anyone ever agreeing to it?

This note presents a paper written with Rodrigo De Losso, Algorithmic Collusion Without Deterrence: Auditing the Punishment Mechanism in Q-Learning Pricing Agents, accepted at the 7th ACM International Conference on AI in Finance (ICAIF 2026, Milan). The full paper will follow once the final version is closed. Here the goal is the line of the argument, readable without the math.

Two programs, one market

The starting point is an experiment by Calvano et al. (2020), an influential result in this debate. Two firms sell similar products. Each is run by a program that chooses a price every period, out of 15 possible prices. The program sees only last period’s prices, its own and its rival’s, and the profit it made.

The programs use Q-learning, the textbook trial-and-error method of reinforcement learning (covered in the previous article). Each one keeps a table with an entry for every situation and every price it could charge, scoring how good that price has turned out to be in that situation. Early on it experiments a lot, trying prices at random. Over time it experiments less and sticks to what has worked.

The experiment has two stages. In the training stage, both programs play against each other for millions of periods and keep learning. Once their tables stop changing, learning is switched off and the test stage begins: the frozen programs play, and we measure what they do.

Figure 1. The standard design. The pair of programs that learns together (training, learning on) is the same pair that is measured afterwards (test, learning off).

Two reference prices make the result readable. The competitive price, about 1.47 in this market, is where neither firm can gain by changing its price alone; economists call it the Nash price. The monopoly price, about 1.93, is what a single firm owning both products would charge. A collusion index places the observed price on that scale: 0 means competition, 1 means full monopoly. The programs in Calvano et al. land well above competition, and in our own rebuild of their experiment the average index across 50 runs is about 0.62. None of this was programmed in.

High prices are not yet a cartel

High prices alone could just be a side effect of how the programs learn. What makes economists say “cartel” is something more specific: a threat.

Think of two gas stations on the same corner, both charging high prices. Either one could cut its price and steal the other’s customers for a day. Neither does, because each knows the other would respond by cutting too, and then both would earn less for a long time. The threat of retaliation is what holds the high price in place. Game theory formalized this decades ago, in the trigger strategies of Friedman (1971) and the more refined punishment schemes of Abreu (1988). A punishment scheme has two defining features. It reacts to anyone leaving the agreed path, and it makes leaving unprofitable.

Calvano et al. went beyond showing high prices. They showed something that looks exactly like this threat, and that is what made their paper a landmark for competition policy.

The experiment that looked like retaliation

Their test is simple. Once the two trained programs have settled on their high prices, force one of them to cut its price for a single period, then let both play freely again and watch.

The rival’s price drops right after the cut, as if punishing it. A few periods later, both prices climb back to where they were before. Calvano et al. read this drop-and-return as a “reward-punishment scheme”: cut your price and your rival retaliates, so cutting never pays.

We rebuilt their environment and ran the same test, and the pattern shows up in our replication too. In 16 of 18 trained pairs, both prices return exactly to their previous level within about five periods.

Figure 2. What happens after a forced price change, averaged over 18 trained pairs. Orange: the program whose price was forced. Blue: its rival. Dashed line: competitive price. Dotted line: monopoly price. Top: a one-period price cut. Bottom: a one-period price raise, discussed in the next section.

So the pattern is robust in this environment. Our question is how to interpret it. If the programs carry a threat, we would expect it to show up in other ways too, so we ran four checks.

Four checks on the threat

Raise the price instead of cutting it. A threat is aimed at undercutting. If one program is forced to raise its price toward the monopoly level, which is a move in the cartel’s favor, there is nothing to punish. Yet the rival reacts the same way: its price drops and then returns (Figure 2, bottom). The reaction does not seem to distinguish a betrayal from a gift, which is hard to reconcile with a threat. It looks more like a system pushed away from its resting point and sliding back.

Offer a real deal. Put each trained program against a scripted opponent that plays a textbook cartel strategy, called grim trigger: it charges a high price as long as the other side does, and drops to the competitive price forever the moment it is undercut. A program that understood threats would presumably see a partner ready to keep prices high and would play along. Instead, 22 of 24 programs undercut this partner within four periods, triggering the permanent price war, and do not return to cooperation over the 200 periods we observe.

Figure 3. Six trained programs against three scripted opponents: one that always charges the competitive price (blue), one that always charges the monopoly price (green) and a grim-trigger partner that starts cooperative (dashed red). The vertical line marks the period in which the program’s own price cut triggers the grim partner’s permanent price war.

Read the rulebook. A program of this kind can be read directly. For every combination of last period’s prices, its table says what price it will charge next. If it encoded a threat, the map would show structure: high prices when the rival behaved, low prices when it did not. What we see looks mostly like noise. Outside the single point where the pair normally sits, the choices show little relation to what a sensible firm would do.

Figure 4. The full rulebook of two trained programs. Each square is a situation (own price and rival’s price last period), and its color is the price the program chooses next. The star marks where the pair normally sits.

The noise does not seem to be a limitation of the format. We hand-built threat strategies in the same table format, and they fit without trouble, with clean visible thresholds. The format can hold a threat; in these runs, training did not produce one.

Compare with random programs. For us this was the most informative check. In this setup, two frozen programs always react the same way to the same prices, and there are only 225 possible situations. Mathematically, any such pair of programs, trained or not, eventually falls into a repeating pattern, so “returning to where it was” is something they are likely to do anyway. We tested this directly by filling the tables with random numbers and running the same forced-cut experiment. Random pairs return to their starting price 67.8% of the time. Trained pairs return 66.7% of the time. In this setting, the return by itself says little about whether the programs learned a threat.

Figure 5. How many periods it takes to return to the original price, for trained pairs (blue) and pairs with random tables (gray). Both return about as often. Trained pairs return faster.

The one thing training adds is speed: trained pairs get back in about 5 periods, random ones in about 13. Training builds a well-worn groove around the pair’s usual prices. A groove brings you back quickly, but it does not know why you left it.

Change the partner

If the threat is not there, what keeps prices high? One way to approach the question is a test we had not seen applied in this literature.

In the standard design, the pair of programs that trains together is the pair that gets tested. Experimental economists have a name for this, the partner treatment, and a well-known alternative, the stranger treatment, in which people are rematched with someone new (Andreoni, 1988). The comparison tells apart two very different things. A personal strategy travels with you to a new partner. A habit built with one specific partner does not.

So we trained many pairs separately, under identical conditions, froze them, and then matched a program from one pair with a program from another. They had never met.

Figure 6. The stranger test. Training is exactly the same. Only the test changes: each program faces a partner from a different training run.

If the programs had each learned to collude, we would expect collusion to survive the swap. In our experiments it mostly does not. The collusion index falls from 0.674 with the original partner to 0.126 with a stranger, a drop of about 81%. In a larger set of swaps, almost a third of the new pairs end up below the competitive price.

Figure 7. Collusion with the original partner (blue) and with a stranger (orange). Left: each of 12 swaps. Right: the averages.

The forced price cut, which produced the celebrated drop-and-return with the original partner, disappears into the noise with a stranger, since prices are already close to the competitive level.

Figure 8. The same forced-change experiment as Figure 2, now between programs from different training runs. Same vertical scale as Figure 2. Prices hover near the competitive level and the shock leaves no lasting trace.

Each program learned to handle the situations its own partner creates, and nothing else. In machine-learning terms, the pair seems to have overfit to itself, a problem studied in multi-agent learning (Lanctot et al., 2017) that appears relevant here. In our setting, the high price looks less like a strategy each program has and more like a habit the pair keeps.

The deviation that pays

There is still one sense in which the high price looks protected. Calvano et al. checked that the one-period price cut does not pay: the gain from undercutting once is smaller than the losses that follow. On our pairs this holds in 18 of 18 cases.

But a one-period cut followed by a return to business as usual is a naive way to deviate. A smart deviator would plan ahead, anticipating how the rival reacts. Because the rival’s rulebook is frozen and fully readable, we can compute exactly the best possible deviation against it. On average it earns 14.8% more than staying at the high price. Only one program out of 38 is already doing the best it can.

Figure 9. Profit each program leaves on the table against its own training partner: the best possible strategy compared with what it actually does, programs sorted.

Read together, these results suggest that the high prices are less defended than undisturbed: a well-planned deviation would pay, but the programs do not try it.

This may help reconcile our results with a later paper by the same group, Calvano et al. (2023), which concluded that the collusion in this setting is largely “genuine”. The difference seems to lie mainly in the deviation checked: theirs is the one-period cut, ours is the best possible deviation. To rule out differences in how the programs were trained, we retrained a set of pairs exactly under their stopping rule and reran every check. The results hold in that smaller set, and the profitable deviation is smaller but still present, at 10.4%.

A possible reason: less exploration

Our tentative explanation for both results, the missing threat and the drop with strangers, has to do with exploration. Recall that the programs experiment a lot at first and less over time. As experimentation fades, the programs spend almost all their time in the handful of situations around their usual prices. Those entries of the table get refined over and over. Everything else, which includes every situation a punishment would need, is visited rarely or never and stays close to its random starting values. This would explain why the rulebook looks like noise, and why the programs behave like random ones once they are pushed far enough.

The same process that settles the high price also makes it unlikely that the high price is ever put to the test. Game theory has a name for this kind of outcome, a self-confirming equilibrium (Fudenberg and Levine, 1993): each side’s beliefs are right about what it actually sees, and never checked against what it does not.

This account also makes a prediction that can be checked. Give the programs more possible prices, keeping the same training time, and the same amount of experimentation gets spread over many more situations. The groove should be shallower and collusion weaker. This is consistent with what we observe: the collusion index falls from 0.74 with 15 prices to 0.43 with 200. Prices stay above the competitive level even then, in line with Klein (2021), who found similar sensitivity to the number of available prices.

Figure 10. Average collusion index as the number of available prices grows from 5 to 200, with the same training time. The black diamond is the 15-price setting used by Calvano et al.

Where this fits

We are not the first to doubt that these programs really threaten each other. Asker, Fershtman and Pakes (2024) show that changing how the programs learn pushes prices back toward competition, which suggests that high prices come from limited experimentation rather than strategy. Hansen, Misra and Pai (2021) obtain high prices from simpler algorithms that have no way of punishing anyone, and Abada and Lambin (2023) reach a similar diagnosis. These papers show that high prices can arise without threats.

Our contribution is narrower. We look at the drop-and-return experiment itself, on the same programs that produce it, since it is the piece of evidence most often associated with the idea of punishment.

Our results are consistent with what Calvano et al. measured, which we reproduce. What we question is one reading of that behavior, not the measurements.

What it means for regulation

The harm to consumers does not go away. Prices with the original partner are robustly high, and nothing in this paper changes that. What changes is how one would respond to it.

Competition law is built around agreements, and its hardest question for algorithms is whether programs can reach something equivalent to one (Harrington, 2018). If there is no threat to point to, it is less clear that the pair resembles an agreement. The results also hint at a different lever: in our experiments, changing who plays against whom substantially reduces coordination among frozen programs, which suggests that entry, rotation or retraining could matter. Whether programs that keep learning after the shake-up would simply form a new habit, and how fast, is something our data cannot answer.

The warning for anyone reading market data is methodological. Real markets only show how one particular set of algorithms behaves with each other. A drop-and-return in prices looks like retaliation, and without controls of this kind it may say little about whether a threat exists. A pricing system that performs well against the systems it was trained with has shown a property of that group, not of itself. Every check in the paper works on frozen programs, so it can be applied to more complex systems too, including deep reinforcement learning.

What we don’t know yet

The results cover one kind of program (table-based Q-learning), two firms and the Calvano et al. market. The comparison with random programs relies on the small, finite number of situations, and does not carry over directly to neural-network agents. Whether those agents learn real threats is open, and the checks were designed to answer that rather than assume it. The other open question is the one regulation cares about most: if learning stays on after the partners are shuffled, does a new habit form, and how quickly? That is the next experiment.

References

• Abada, I., & Lambin, X. (2023). Artificial intelligence: Can seemingly collusive outcomes be avoided? Management Science, 69(9), 5042–5065.
• Abreu, D. (1988). On the theory of infinitely repeated games with discounting. Econometrica, 56(2), 383–396.
• Andreoni, J. (1988). Why free ride? Strategies and learning in public goods experiments. Journal of Public Economics, 37(3), 291–304.
• Asker, J., Fershtman, C., & Pakes, A. (2024). The impact of artificial intelligence design on pricing. Journal of Economics and Management Strategy, 33(2), 276–304.
• Assad, S., Clark, R., Ershov, D., & Xu, L. (2024). Algorithmic pricing and competition: Empirical evidence from the German retail gasoline market. Journal of Political Economy, 132(3), 723–771.
• Brown, Z. Y., & MacKay, A. (2023). Competition in pricing algorithms. American Economic Journal: Microeconomics, 15(2), 109–156.
• Calvano, E., Calzolari, G., Denicolò, V., & Pastorello, S. (2020). Artificial Intelligence, Algorithmic Pricing, and Collusion. American Economic Review, 110(10), 3267–3297.
• Calvano, E., Calzolari, G., Denicolò, V., & Pastorello, S. (2023). Algorithmic collusion: Genuine or spurious? International Journal of Industrial Organization, 90, 102973.
• Friedman, J. W. (1971). A non-cooperative equilibrium for supergames. Review of Economic Studies, 38(1), 1–12.
• Fudenberg, D., & Levine, D. K. (1993). Self-confirming equilibrium. Econometrica, 61(3), 523–545.
• Hansen, K. T., Misra, K., & Pai, M. M. (2021). Frontiers: Algorithmic collusion: Supra-competitive prices via independent algorithms. Marketing Science, 40(1), 1–12.
• Harrington, J. E. (2018). Developing competition law for collusion by autonomous artificial agents. Journal of Competition Law & Economics, 14(3), 331–363.
• Klein, T. (2021). Autonomous algorithmic collusion: Q-learning under sequential pricing. RAND Journal of Economics, 52(3), 538–558.
• Lanctot, M., Zambaldi, V., Gruslys, A., Lazaridou, A., Tuyls, K., Pérolat, J., Silver, D., & Graepel, T. (2017). A unified game-theoretic approach to multiagent reinforcement learning. Advances in Neural Information Processing Systems, 30.