I’m on the hook to do a SciML webinar next week:

• Bob Carpenter. Multiscale generalized Hamiltonian Monte Carlo with delayed rejection. SciML Webinar, 28 September, 2023. 11 am (Eastern US time).

These are organized by Keith Phuthi (who is at CMU) through University of Michigan’s Institute for Computational Discovery and Engineering.

Sam Livingstone is moderating (though they got his affiliation wrong on the announcement—he’s at UCL, not here at the Flatiron Institute). This is presenting joint work with Alex Barnett, Chirag Modi, Edward Roualdes, and Gilad

I’m very excited about this work as it combines a number of threads I’ve been working on with collaborators. When I did my job talk here, Leslie Greengard, our center director, asked me why we didn’t use variable stepwise integrators when doing Hamiltonian Monte Carlo. I told him we’d love to do it, but didn’t know how to do it in such a way as to preserve the stationary target distribution.

Delayed rejection HMC

Then we found Antonietta Mira’s work on delayed rejection. It lets you retry a second Metropolis proposal if the first one is rejected. The key here is that we can use a smaller step size for the second proposal, thus recovering from proposals that are rejected because the Hamiltonian diverged (i.e., the first-order gradient based algorithm can’t handle regions of high curvature in the target density). There’s a bit of bookkeeping (which is frustratingly hard to write down) for the Hastings condition to ensure detailed balance. Chirag Modi, Alex Barnett and I worked out the details, and Chirag figured out a novel twist on delayed rejection that only retries if the original acceptance probability was low. You can read about it in our paper:

• Modi, Barnett, Carpenter. 2023. Delayed rejection Hamiltonian Monte Carlo for sampling multiscale distributions. Bayesian Analysis.

This works really well and is enough that we can get proper draws from Neal’s funnel (vanilla HMC fails on this example in either the tails in either the mouth or neck of the funnel, depending on the step size). But it’s inefficient in that it retries an entire Hamiltonian trajectory. Which means if we cut the step size in half, we double the number of steps to keep the integration time constant.

Radford Neal to the rescue

As we were doing this, the irrepressible Radford Neal published a breakthrough algorithm:

• Radford Neal. 2020. Non-reversibly updating a uniform [0,1] value for Metropolis accept/reject decisions.. arXiv.

What he managed to do was use generalized Hamiltonian Monte Carlo (G-HMC) to build an algorithm that takes one step of HMC (like Metropolis-adjusted Langevin, but over the coupled position/momentum variables) and manages to maintain directed progress. Instead of fully resampling momentum each iteration, G-HMC resamples a new momentum value then performs a weighted average with the existing momentum with most of the weight on the existing momentum. Neal shows that with a series of accepted one-step HMC iterations, we can make directed progress just like HMC with longer trajectories. The trick is getting sequences of acceptances together. Usually this doesn’t work because we have to flip momentum each iteration. We can re-flip it when regenerating, to keep going in the same direction on acceptances, but with rejections we reverse momentum (this isn’t an issue with HMC because it fully regenerates each time). So to get directed movement, we need steps that are too small. What Radford figured out is that we can solve this problem by replacing the way we generate uniform(0, 1)-distributed probabilities for the Metropolis accept/reject step (we compare the variate generated to the ratio of the density at the proposal to the density at the previous point and accept if it’s lower). Radford realized that if we instead generate them in a sawtooth pattern (with micro-jitter for ergodicity), then when we’re at the bottom of the sawtooth generating a sequence of values near zero, the acceptances will cluster together.

Replacing Neal’s trick with delayed rejection

Enter Chirag’s and my intern, Gilad Turok (who came to us as an undergrad in applied math at Columbia). Over the summer, working with me and Chirag and Edward Roualdes (who was here as a visitor), he built and evaluated a system that replaces Neal’s trick (sawtooth pattern of acceptance probability) with the Mira’s trick (delayed rejection). It indeed solves the multi scale problem. It exceeded our expectations in terms of efficiency—it’s about twice as fast as our delayed rejection HMC. Going one HMC step at a time, it is able to adjust its stepsize within what would be a single Hamiltonian trajectory. That is, we finally have something that works roughly like a typical ODE integrator in applied math.

Matt Hoffman to the rescue

But wait, that’s not all. There’s room for another one of the great MCMC researchers to weigh in. Matt Hoffman, along with Pavel Sountsov, figured out how to take Radford’s algorithm and provide automatic adaptation for it.

• Hoffman and Sountsov. 2022. Tuning-Free Generalized Hamiltonian Monte Carlo. AIStats.

What Hoffman and Sountsov manage to do is run a whole lot of parallel chains, then use information in the other chains to set tuning parameters for a given chain. In that way it’s like the Goodman and Weare affine-invariant sampler that’s used in the Python package emcee. This involves estimating the metric (posterior covariance or just variance in the diagonal case) and also estimating steps size, which they do through a heuristic largest-eigenvalue estimate. Among the pleasant properties of their approach is that the entire setup produces a Markov chain from the very first iteration. That means we only have to do what people call “burn in” (sorry Andrew, but notice how I say other people call it that, not that they should), not set aside some number of iterations for adaptation.

Edward Roualdes has coded up Hoffman and Sountsov’s adaptation and it appears to work with delayed rejection replacing Neal’s trick.

Next for Stan?

I’m pretty optimistic that this will wind up being more efficient than NUTS and also make things like parallel adaptation and automatic stopping a whole lot easier. It should be more efficient because it doesn’t waste work—NUTS goes forward and backward in time and then subsamples along the final doubling (usually—it’s stochastic with a strong bias toward doing that). This means we “waste” the work going the wrong way in time and beyond where we finally sample. But we still have a lot of eval to do before we can replace Stan’s longstanding sampler or even provide an alternative.

My talk

The plan’s basically to expand this blog post with details and show you some results. Hope to see you there!

A researcher who wishes to remain anonymous writes:

I am writing with a question about your article with Imbens, Why High-Order Polynomials Should Not Be Used in Regression Discontinuity Designs. In it, you discourage the use of high-order polynomials of the forcing variable when fitting models. I have a few questions about this:

(1) What are your thoughts about the use of restricted cubic splines (RCS) that are linear in both tails?

(2) What are your thoughts on the use of a generalized additive model with local regression (rather than with splines)?

(3) What are your thoughts on the use of loess to fit the regression models?

I wonder if the use of restricted cubic splines would be less susceptible to the difficulties that you describe given that it is linear in the tails.

My quick reply is that I wouldn’t really trust any estimate that jumps around a lot. I’ve seen too many regression discontinuity analyses that give implausible answers because the jump at the discontinuity cancels a sharp jump in the other direction in the fitted curve. When you look at the regression discontinuity analyses that work (in the sense of giving answers that make sense), the fitted curve is smooth.

The first question above is addressing the tail-wagging-the-dog issue, and that’s a concern as well. I guess I’d like to see models where the underlying curve is smooth, and if that doesn’t fit the data, then I think the solution is to restrict the range of the data where the model is fit, not to try to solve the problem by fitting a curve that gets all jiggy.

My other general advice, really more important than what I just wrote above, is to think of regression discontinuity as a special case of an observational study. You have a treatment or exposure z, an outcome y, and pre-treatment variables x. In a discontinuity design, one of the x’s is a “forcing variable,” for which z_i = 1 for cases where x_i exceeds some threshold, and z_i = 0 for cases where x_i is lower than the threshold. This is a design with known treatment assignment and zero overlap, and, yeah, you’ll definitely want to adjust for imbalance in that x-variable. My inclination would be to fit a linear model for this adjustment, but sometimes a nonlinear model will make sense, as long as you keep it smooth.

But . . . the forcing variable is, in general, just one of your pre-treatment variables. What you have is an observational study! And you can have imbalance on other pre-treatment variable also. So my main recommendation is to adjust for other important pre-treatment variables as well.

For an example, see here, where I discuss a regression discontinuity analysis where the outcome variable was length of life remaining, and the published analysis did not include age as a predictor. You gotta adjust for age! The message is: a discontinuity analysis is an observational study. The forcing variable is important, but it’s not the only thing in town. The big mistakes seem to come from: (a) unregularized regression on the forcing variable which randomly give you wild jumpy curves that pollute the estimate of the discontinuity, (b) not adjusting for other important pre-treatment predictors, and (c) taking statistically significant estimates and treating them as meaningful, without looking at the model that’s been fit.

We discuss some of this in Section 21.3 of Regression and Other Stories.

After reading our recent post, “How to quit smoking, and a challenge to currently-standard individualistic theories in social science,” Gur Huberman writes:

You may be aware that the incidence of Parkinson (PD) is lower in the smoking population than in the general population, and that negative relation is stronger for the heavier & longer duration smokers.

The reason for that is unknown. Some neurologists conjecture that there’s something in smoked tobacco which causes some immunity from PD. Other conjecture that whatever causes PD also helps people quit or avoid smoking. For instance, a neurologist told me that Dopamine (the material whose deficit causes PD) is associated with addiction not only to smoking but also to coffee drinking.

Your blog post made me think of a study that will try to distinguish between the two explanations for the negative relation between smoking and PD. Such a study will exploit variations (e.g., in geography & time) between the incidence of smoking and that of PD.

It will take a good deal of leg work to get the relevant data, and a good deal of brain work to set up a convincing statistical design. It will also be very satisfying to see convincing results one way or the other. More than satisfying, such a study could help develop medications to treat or prevent PD.

If this project makes sense perhaps you can bring it to the attention of relevant scholars.

OK, here it is. We’ll see if anyone wants to pick this one up.

I have some skepticism about Gur’s second hypothesis, that “whatever causes PD also helps people quit or avoid smoking.” I say this only because, from my perspective, and as discussed in the above-linked post, the decision to smoke seems like much more of a social attribute than an individual decision. But, sure, I could see how there could be correlations.

In any case, it’s an interesting statistical question as well as an important issue in medicine and public health, so worth thinking about.

Paul Campos writes:

Probably the biggest public health success in America over the past half century has been the remarkably effective long-term campaign to reduce cigarette smoking. The percentage of adults who smoke tobacco has declined from 42% in 1965 (the first year the CDC measured this), to 12.5% in 2020.

It’s difficult to disentangle the effect of various factors that have led to this stunning decline of what was once a ubiquitous habit — note that if we exclude people who report having no more than one or two drinks per year, the current percentage of alcohol drinkers in the USA is about the same as the percentage of smokers 60 years ago — but the most commonly cited include:

Anti-smoking educational campaigns

Making it difficult to smoke in public and many private spaces

Increasing prices

Improved smoking cessation treatments, and laws requiring the cost of these to be covered by medical insurance

I would add another factor, which is more broadly cultural than narrowly legal or economic: smoking has become declasse.

This is evident if you look at the relationship between smoking rates and education and income: While 32% of people with a GED smoke, the percentages for holders of four year college degrees and graduate degrees are 5.6% and 3.5% respectively. And while 20.2% of people with household incomes under the $35,000 smoke, 6.2% of people with household incomes over $100,000 do.

All worth noting. Anti-smoking efforts are a big success story, almost such a bit story that it’s easy to forget.

The sharp decline in smoking is a big “stylized fact,” as we say in social science, comparable to other biggies such as the change in acceptance of gay people in the past few decades, and the also-surprising lack of change in attitudes toward abortion.

When we have a big stylized fact like this, we should milk it for as much understanding as we can.

With that in mind, I have a few things to add on the topic:

1. Speaking of stunning, check out these Gallup poll results on rates of drinking alcohol:

At least in the U.S., rich people are much more likely than poor people to drink. That’s the opposite of the pattern with smoking.

2. Speaking of “at least in the U.S.”, it’s my impression that smoking rates have rapidly declined in many other countries too, so in that sense it’s more of a global public health success.

3. Back to the point that we should recognize how stunning this all is: 20 years ago, they banned smoking in bars and restaurants in New York. All at once, everything changed, and you could go to a club and not come home with your clothes smelling like smoke, pregnant women could go places without worrying about breathing it all in, etc. When this policy was proposed and then when it was clear it was really gonna happen, lots of lobbyists and professional contrarians and Debby Downers and free-market fanatics popped up and shouted that the smoking ban would never work, it would be an economic disaster, the worst of the nanny state, bla bla bla. Actually it worked just fine.

4. It’s said that quitting smoking is really hard. Smoking-cessation programs have notoriously low success rates. But some of that is selection bias, no? Some people can quit smoking without much problem, and those people don’t need to try smoking-cessation programs. So the people who do try those programs are a subset that overrepresents people who can’t so easily break the habit.

5. We’re used to hearing the argument that, yeah, everybody knows cigarette smoking causes cancer, but people might want to do it anyway. There’s gotta be some truth to that: smoking relaxes people, or something like that. But also recall what the cigarette executives said, as recounted by historian Robert Proctor:

Philip Morris Vice President George Weissman in March 1954 announced that his company would “stop business tomorrow” if “we had any thought or knowledge that in any way we were selling a product harmful to consumers.” James C. Bowling . . . . Philip Morris VP, in a 1972 interview asserted, “If our product is harmful . . . we’ll stop making it.” Then again in 1997 the same company’s CEO and chairman, Geoffrey Bible, was asked (under oath) what he would do with his company if cigarettes were ever established as a cause of cancer. Bible gave this answer: “I’d probably . . . shut it down instantly to get a better hold on things.” . . . Lorillard’s president, Curtis Judge, is quoted in company documents: “if it were proven that cigarette smoking caused cancer, cigarettes shoudl not be marketed” . . . R. J. Reynolds president, Gerald H. Long, in a 1986 interview asserted that if he ever “saw or thought there were any evidence whatsoever that conclusively proved that, in some way, tobacco was harmful to people, and I believed it in my heart and my soul, then I would get out of the business.”

6. A few years ago we discussed a study of the effects of smoking bans. My thought at the time was: Yes, at the individual level it’s hard to quit smoking, which might give skepticism about the effects of measures designed to reduce smoking—but, at the same time, smoking rates vary a lot by country and by state, This was similar to our argument about the hot hand: given that basketball shooting success rates vary a lot over time and across game conditions, it should not be surprising that previous shots might have an effect. As I wrote awhile ago, “if ‘p’ varies among players, and ‘p’ varies over the time scale of years or months for individual players, why shouldn’t ‘p’ vary over shorter time scales too? In what sense is “constant probability” a sensible null model at all?” Similarly, given how much smoking rates vary, maybe we shouldn’t be surprised that something could be done about it.

7. To me, though, the most interesting thing about the stylized facts on smoking is how there is this behavior that is so hard to change at the individual level but can be changed so much at the national level. This runs counter to currently-standard individualistic theories in social science in which everything is about isolated decisions. It’s more of a synthesis: change came from policy and from culture (whatever that means), but this still had to work its way though individual decisions. This idea of behavior being changed by policy almost sounds like “embodied cognition” or “nudge,” but it feels different to me in being more brute force. Embodied cognition is things like giving people subliminal signals; nudge is things like subtly changing the framing of a message. Here we’re talking about direct education, taxes, bans, big fat warning labels: nothing subtle or clever that the nudgelords would refer to as a “masterpiece.”

Anyway, this idea of changes that can happen more easily at the group or population level than at the individual level, that’s interesting to me. I guess things like this happen all over—“social trends”—and I don’t feel our usual social-science models handle them well. I don’t mean that no models work here, and I’m sure that lots of social scientists done serious work in this area; it just doesn’t seem to quite line up with the usual way we talk about decision making.

P.S. Separate from all the above, I just wanted to remind you that there’s lots of really bad work on smoking and its effects; see here, for example. I’m not saying that all the work is bad, just that I’ve seen some really bad stuff, maybe no surprise what with all the shills on one side and all the activists on the other.

Speaking of teaching . . . I wanted to share this story of something that happened today.

I was all fired up with energy, having just taught my Communicating Data and Statistics class, taking notes off the blackboard to remember what we’d been talking about so I could write about it later, and students were walking in for the next class. I asked them what it was, and they said Shakespeare. How wonderful to take a class on Shakespeare at Columbia University, I said. The students agreed. They love their teacher—he’s great.

This gave me an idea . . . maybe this instructor and I could switch classes some day, a sort of academic Freaky Friday. He could show up at 8:30 and teach my statistics students about Shakespeare’s modes of communication (with his contemporaries and with later generations including us, and also how Shakespeare made use of earlier materials), and I could come at 10am to teach his students how we communicate using numbers and graphs. Lots of fun all around, no? I’d love to hear the Shakespeare dude talk to a new audience, and I think my interactions with his group would be interesting too.

I waited in the classroom for awhile so I could ask the instructor when he came into the room, during the shuffling-around period before class officially starts at 10:10. Then 10:10 came and I stood outside to wait as the students continued to trickle in. A couple minutes later I saw a guy approaching, about my age, I ask if he teaches the Shakespeare class. Yes, he is. I introduce myself: I teach the class right before, on communicating data and statistics, maybe we could do a switch one day, could be fun? He says no, I don’t think so, and goes into the classroom.

That’s all fine, he has no obligation to do such a thing, also I came at him unexpectedly at a time when he was already in a hurry, coming to class late (I came to class late this morning too. Mondays!). His No response was completely reasonable.

Still . . . it was a lost opportunity! I’ll have to brainstorm with people about other ways to get this sort of interdisciplinary opportunity on campus. We could just have an interdiscplinary lecture series (Communication of Shakespeare, Communication in Statistics, Communication in Computer Science, Communication in Medicine, Communication in Visual Art, etc.), but it would be a bit of work to set up such a thing, also I’m guessing it wouldn’t reach so many people. I like the idea of doing it using existing classes, because (a) then the audience is already there, and (b) it would take close to zero additional effort: you’re teaching your class somewhere else, but then someone else is teaching your class so you get a break that day. And all the students are exposed to something new. Win-win.

The closest thing I can think of here is an interdisciplinary course I organized many years ago on quantitative social science, for our QMSS graduate program. The course had 3 weeks each of history, political science, economics, sociology, and psychology. It was not a statistics course or a methods course; rather, each segment discussed some set of quantitative ideas in the field. The course was wonderful, and Jeronimo Cortina and I turned it into a book, A Quantitative Tour of the Social Sciences, which I really like. I think the course went well, but I don’t think QMSS offers it anymore; I’m guessing it was just too difficult to organize a course with instructors from five different departments.

David Kane sends along this article he just wrote with the above title and the following abstract:

A community is a collection of people who know and care about each other. The vast majority of college courses are not communities. This is especially true of statistics and data science courses, both because our classes are larger and because we are more likely to lecture. However, it is possible to create a community in your classroom. This article offers an idiosyncratic set of practices for creating community. I have used these techniques successfully in first and second semester statistics courses with enrollments ranging from 40 to 120. The key steps are knowing names, cold calling, classroom seating, a shallow learning curve, Study Halls, Recitations and rotating-one-on-one final project presentations.

There’s some overlap with the ideas in chapters 1 and 2 of my forthcoming book with Aki, “Active Statistics: Stories, Games, Problems, and Hands-on Demonstrations for Applied Regression and Causal Inference,” but Kane has a bunch of ideas that are different from (but complementary) to ours. I recommend our book (along with Rohan Alexander’s Telling Stories with Data, Elena Llaudet and Kosuke Imai’s Data Analysis for Social Science, our own, Regression and Other Stories); I also recommend Kane’s article. Kane, like Aki and me in our new book, focus on ways to get students actively involved in class. I hadn’t thought about this in terms of “community,” but that seems like a good way to frame it.

I’m sure there’s lots more out there; the above list of resource is focused on modern introductions to applied statistics and active ways to learn and teach this material.

Geoffrey Yip writes:

You’ve written before on how numerical stats can mislead people. There are great visuals for this idea through Causal Quartets or Ascombe’s Quartets. Are there clear examples of the opposite idea, where four visually similar visualizations can have vastly different numerical stats?

My reply: Sure, graphs can be horrible and convey no information or even actively mislead. Tables can be hard to read, but I guess that, without actually faking the numbers, it would be hard to make a table that’s as misleading as some very bad graphs. But I think that a good graph should always be better than the corresponding table; see this article from 2002 for exploration of this point.

1. The action

Recently there has been some controversy about Supreme Court judge Clarance Thomas, who appears to have broken the law by not reporting a series of gifts given to him and his family by a rich donor. A judge breaking the law—that’s not cool!

2. The defense

Oddly enough—or, at least oddly at first glance—one of Thomas’s defenders in this affair is law professor and free-software campaigner Larry Lessig.

Lessig’s involvement seems odd at first because in 2011 he published a book entitled, “Republic, Lost: How Money Corrupts Congress — and a Plan to Stop It.” You’d think that if Lessig believes that money corrupts Congress, he’d think that it would corrupt the court system.

Actually, though, no! Check out this recent post from Lessig. On one hand, I think it’s kinda cool that a bigshot like Lessig is spending his time blogging—he’s just like me! No regular NYT or Fox News gig or whatever so he puts his thoughts out there in the ether, for all to read!—; on the other hand, I’m not so happy with what Lessig has to say:

Yet most of the attacks on Justice Thomas go beyond the failure to report some of his gifts. Most are attacking (1) that he took these or any gifts at all, or (2) that he took them from someone on the Right, or (3) that he took them from someone who is wealthy.

Point (3) seems odd . . . Point (2) is odd as well . . . Which leaves point (1), the point about this reporting that is most odd to me: One might well believe that Justices should not take gifts like this at all — that they should never visit others, or stay with others, that their vacations should be on their own dime. . . .

Whaaaa? So the two alternatives for a federal judge are: (a) take zillions of dollars in undeclared gifts from a political donor, or (b) “never visit others, or stay with others”? That’s just weird. I don’t think anything in the law prohibits judges from visiting people!

I guess that’s something they teach in law school, to introduce ridiculous slippery-slope arguments?

Lessig continues:

But if that’s your view, then fair reporting would ask whether Thomas’ behavior is unique or exclusive to him.

This is absolutely nuts! First off, when he says, “if that’s your view,” “that” refers to the position that Supreme Court judges “should never visit others, or stay with others, that their vacations should be on their own dime. . . .” There’s nobody who has the view that judges should never visit or stay with others, so he’s making a conditional argument (if A then B) for which A is the empty set.

The second weird part of his argument is, if you break the law, it’s not much of a defense to say that you’re not the only person to have broken it. Typically they write laws for offenses that happen more than once!

The third weird part is that he speculates that other judges have done the same thing, but offers no evidence of that happening.

Lessig continues:

Washington is filled with techniques for the modestly paid to live life as if they were rich. . . . Justices on the Supreme Court are paid at the level of mid-tier lawyers at big New York law firms. And when you’re paid less than you think you’re worth, you find ways to justify what most would think corrupt.

I don’t really get why a judge on the Supreme Court should think he’s worth more, financially, than a mid-tier lawyer at a big New York law firms. Mid-tier lawyers have a lot of responsibility, no? They can make or lose a lot of money for their firms, and they get paid a lot. If Thomas or any another federal judge thought he could get more money in this way, he’s free to quit his job and join a law firm, no?

He concludes with a call for change:

If we’re to address these corruptions for real . . . we need to stop pretending that the problem is individual. It is not. It is institutional. The corruptions that are destroying our government are woven into the systems of our government. It is these systems that must change if we’re to have institutions we can trust.

I kinda see where he’s coming from, but the first step toward reform would be to enforce existing laws, no? If your goal is reform, it seems like a step backward to minimize existing violations.

The other odd thing here is that Lessig leads his post with, “Justice Thomas’s interpretation of the reporting requirements applicable to him is wrong. The reading offered by Dahlia Lithwick and Mark Joseph Stern is correct.” The title of Lithwick and Stern’s article is, “Clarence Thomas Broke the Law and It Isn’t Even Close.” But Lessig didn’t say that “Thomas broke the law,” he just said “Thomas’s interpretation of the reporting requirements applicable to him is wrong.” By framing it this way, Lessig took a clear statement of lawbreaking and turned it into a fuzzy-sounding legalism. I guess that’s what bigtime law professors do.

3. The reaction

Paul Campos, a law professor at the University of Colorado, was not happy with Lessig’s post. Campos shared some statistics:

Clarence Thomas has a salary of $285K per year, which puts him in the 98th percentile of individual income in the USA. This seriously understates his true economic position, however, because

(a) He’s guaranteed this salary, COLA adjusted, for life, even if he were to quit tomorrow; and

(b) He. can make as much money as he wants publishing books — or “books” — via whatever wingnut welfare publishing outfit wants to help a brother out; and

(c) His wife gets oodles of money from those same sources for her work, or “work; and

(d) Larry Lessig makes about $600K per year as as senior HLS professor for teaching two classes a year.

I kinda wonder how Campos knows Lessig’s salary and teaching load. I guess the world of law school professors is small, and everyone knows these things, more or less?

Anyway, googling *lawrence lesssig salary* led me to this 2018 post, “Poker the Bear: The Sad Unraveling of Lawrence Lessig.” I have no idea whassup with all that but who knows.

Also this interview with Lessig on academic corruption:

Lessig: They receive money from interested parties to participate in the policymaking process. . . . in fields like economics and law, basically the soft sciences, the temptation to bend or to shade is always there. And if the return from bending or shaving is high, then obviously we have to worry that there’s that kind of a distortion going on.

Interviewer: Is there evidence that a payment’s changed the testimony or research focus or of people who are being paid?

Lessig: Well we know first that it changes the perception of the integrity . . . the perception alone is enough. . . . there’s this incentive to focus on the question in a way that keeps the supplier of the data happy, which again is the kind of dynamic that we would worry about if we’re worried about the actual research being bent in a way that’s not reproductive of the truth. . . .

Lessig seems so passionate about fighting corruption. It seems almost unbelievable that he’d go out and defend a federal judge who’s taken all these freebies.

4. The resolution of the paradox

Again, the paradox is that (a) Lessig hates corruption, he even hates the perception of corruption, especially in “fields like economics and law,” but (b) when a federal judge has been taking tons of money and free trips from a rich political donor, Lessig doesn’t think it’s such a big deal.

I was legitimately puzzled here. In his above-linked post, Campos suggests that Lessig is motivated by some sort of generalized honor-among-plutocrats principle, and that could be: Lessig lives in a rarefied world in which being “paid at the level of mid-tier lawyers at big New York law firms” is considered to be a mark of poverty rather than of comfortable wealth. But still.

Then I came across this news article from 2020, “Lawrence Lessig sues New York Times over MIT and Jeffrey Epstein interview.”

This story is relevant because Lessig was arguing that if you’re going to take money from crooks, you should do it in secret. Here are two quotes from Lessig, reported in that article:

Joi Ito was the academic entrepreneur and friend of Lessig who’s famous as the link between Jeffrey Epstein and MIT. (Incidentally, as an MIT grad, let me just say that the Institute has gone downmarket when it comes to scandals. Back when I was a student, the big debate was whether to accept military funding: at the time, we were all upset about Central America and southern Africa, but in retrospect maybe we should’ve been more bothered by what our military was doing in Afghanistan and Iraq. Now the controversy is about professors hanging out on a private island with sleazy millionaires.)

Anyway, Lessig’s statements leapt out at me: he doesn’t seem to have much of a problem with corruption at all! His problem is with open corruption. Just take your “blood money” (his term) but don’t let anyone know about it. He literally said, “Were I king, I would ban non-anonymous gifts of type 3 [“money from people convicted of a crime”] or type 4 [“blood money”].”

What the hell? If he were king, he presumably would have the ability to ban all blood money, right? But, no, he wouldn’t go that far! He’d only ban “non-anonymous” blood money. All right, then.

I don’t really understand Lessig’s position, but I guess he’s consistent. Payoffs are ok but only if they’re secret. This also is consistent with him being so bothered by perceptions. If the payoff is secret, there’s no perception. No perception, no problem.

The other thing is that I noticed a certain paradox-loving style of writing, the sort of thing that your English teacher in high school or political science teacher in college will really appreciate. I’m thinking of the bit where he says:

Everyone seems to treat it . . . I see it as exactly the opposite . . . rather than repeating unreflective paeans . . .

Geddit? He’s a subtle thinker, not like everybody else.

And now I see how he got this reputation for brilliance. “Paeans”! I don’t even now how to pronounce that word. Dude must have got a really high SAT score, also really good grades in law school. One of my pet peeves of this sort of ethical analysis is people making these clever good-is-bad-and-bad-is-good arguments. Any schlub can say that it’s unethical for a federal judge to take big gifts from political donors, or that a university should avoid taking money from criminals. It takes a very special legal thinker to say that the real problem here is that the donations are not going on in secret.

In all seriousness, sometimes it seems that the main qualification for becoming an elite law professor is to be the kind of person who could get perfect SAT scores and write snappy high school essays and then just stay in that position for the rest of your life. I can only assume that those “mid-tier lawyers at big New York law firms” have more on the ball than this.

5. Why does this have to do with statistical modeling, causal inference, and social science?

The connection is political science, power and influence, and the role of institutions in society in supporting or opposing various forms of political influence. Lessig’s just one guy and this post does not represent any sort of attempt at a systematic study. Indeed, it’s safe to say that Lessig is not representative of left-wing law professors at elite universities, most of whom I expect would not defend a federal judge for accepting large undeclared donations.

In our paper on stories as evidence, Basbøll and I talked about the way that good stories are anomalous and immutable, and the anomalousness would seem to contradict usual statistical principles. In short: a good story is “man bites dog,” but good statistical evidence comes from representative data of the “dog bites man” variety.

We concluded that stories are useful for learning because they represent model checks: a good story is something that really happened and whose facts can be checked (“immutable”) and that is surprising (“anomalous”); that is, it represents a stubborn fact that does not align with our usual view of the world.

The Lessig story is immutable (lots of evidence that he is much more bothered by political influence being public than by the donations themselves, to the extent that he defends the judge in part, it seems, because Thomas did not disclose all that he received from that donor) and it’s anomalous (a political liberal who is opposed to money corrupting politics, but then is not so upset about a prominent instance of money that was potentially corrupting political decisions).

Following our principles, the anomaly is interesting: the surprise is relative to our implicit model that a anti-corruption campaigner would be disturbed by a public case of undisclosed political donations. Indeed, my expectation, without having any clear background on Lessig, would’ve been that he’d jump all over Thomas and use this as another example of why money in politics is a problem. But Lessig didn’t do that; hence the puzzle, which makes it a good story and also makes us realize that more must be going on.

The question is, why did Lessig take this paradoxical position of defending the gift-taking politician. As Imbens and I explained, we can’t in general answer Why questions, but they can motivate further study.

Regarding Lessig, we have a few theories, which I don’t see as mutually exclusive:

1. Celebrity stick together. Perhaps Lessig has sympathy for Thomas because they’re both wealthy celebrity lawyers. That perhaps is more important than their political differences.

2. Law professors have a loyalty to the system. There’s this whole Supreme Court mystique, starting with the idea that the judges on that particular court get to be called “justices” (or even “Justices”). Lessig was once a clerk on that court. The institution just means too much to him.

3. General rich-guy cluelessness combined with Harvard elitism. Perhaps Lessig genuinely thinks of $285,000/year, guaranteed COLA-adjusted for life, as a low salary. And perhaps he thinks it’s beneath someone’s dignity to be paid as much as “mid-tier lawyers at big New York law firms.”

4. Fundraising. Lessig ran for president and he worked with some political foundations. It seems he got some big donations from some rich people on these projects. He was also friends with Joi Ito who got money from at least one famous rich person and I guess others too. The point is that, from Lessig’s perspective, money from rich people is really important, but he realizes that it bothers lots of people, which is why he’d prefer the money to anonymous. It would hard for Lessig to take a hard stance against taking money from rich people, given that he’s done so much of this himself. This leads him into tricky spots in his role as a campaigner against money in politics, and it doesn’t help that he can come up with various ridiculous too-clever-by-half ethical arguments.

Of all these reasons, the bit about the fundraising is the most interesting. I say this because I get a good salary (not in the Lessig range, apparently, but enough to live comfortably and focus my time on research, teaching, and service), and the sources are indirect. You could say that Columbia launders its money for me, in the sense that I don’t really have to worry about where the $ originally come from. Similarly, when I get federal grants and corporate support, I don’t think too hard about whatever bad things the government and the companies might be doing. So, yeah, it’s easy enough for me to get indignant about unreported political donations—I’m not raising money from individuals.

To put it another way: I’ve eaten a lot of meat in my life but I’ve never been to the abattoir. So who am I to criticize Lessig for being cool with anonymously-donated “blood money”? I don’t look at the ultimate sources of my own funds.

I can’t say I agree with Lessig—I still hold the commonsensical (to me) view that transparency is better, that Thomas should’ve followed the law and disclosed the donations, that secrecy is not a “saving grace” to taking money from evil people, etc.—but I can kind of almost see where he’s coming from.

Again, the social-science interest here is partly the direct importance of the influence of money in politics and partly understanding how an anti-corruption campaigner can, paradoxically, come out against transparency and in favor of secret donations.

P.S. One of the benefits of blogging is you get comments! Lessig’s post has 9 comments, almost all of which strongly disagree with him. Maybe this will motivate him to reconsider his views.

P.P.S. In the between when I wrote this post and when it is appearing online, I came across this post from Lessig defending embattled dishonesty researcher Francesca Gino.

Jeffrey Epstein, Clarence Thomas, Francesca Gino . . . quite the collection he’s got there! Paradox-boy is outta control.

In all seriousness, I wonder if one of the problems here is that, in elite law academia, you win by coming up with creative arguments that no one has come up with before. The bad news is that most of the reasonable arguments have already been taken, so to be truly original you need to color outside the lines. Any idiot of a journalist can make the obvious arguments that Clarence Thomas broke the law and that Harvard and MIT shouldn’t be taking anonymous money from the likes of Jeffrey Epstein. It takes a bigshot Harvard Law professor to take the opposite side of these cases. I fear that the system incentivizes the Lessigs of the world to make ridiculous claims. Of course, lots of Harvard law professors aren’t going around defending the recipients of “blood money”—but those professors aren’t getting publicity either. I’m not writing about them, right? If you want the fame along with that $600K salary, it helps to be ridiculous.

Joshua Brooks points us to this post by David Gorski, “The Cochrane mask fiasco: Does EBM predispose to COVID contrarianism?” EBM stands for “evidence-based medicine,” and here’s what Gorski writes:

A week and a half ago, the New York Times published on Opinion piece by Zeynep Tufekci entitled Here’s Why the Science Is Clear That Masks Work. Written in response to a recent Cochrane review, Physical interventions to interrupt or reduce the spread of respiratory viruses, that had over the last month been widely promoted by antimask and antivaccine sources, the article discusses the problems with the review and its lead author Tom Jefferson, as well as why it is not nearly as straightforward as one might assume to measure mask efficacy in the middle of a pandemic due to a novel respiratory virus. Over the month since the review’s publication, its many problems and deficiencies (as well as how it has been unrelentingly misinterpreted) have been discussed widely by a number of writers, academics, and bloggers . . .

My [Gorski’s] purpose in writing about this kerfuffle is not to rehash (much) why the Cochrane review was so problematic. Rather, it’s more to look at what this whole kerfuffle tells us about the Cochrane Collaborative and the evidence-based medicine (EBM) paradigm it champions. . . . I want to ask: What is it about Cochrane and EBM fundamentalists who promote the EBM paradigm as the be-all and end-all of medical evidence, even for questions for which it is ill-suited, that can produce misleading results? . . .

Back in the day, we used to call EBM’s failure to consider the low to nonexistent prior probability as assessed by basic science that magic like homeopathy could work its “blind spot.” Jefferson’s review, coupled with the behavior of EBM gurus like John Ioannidis during the pandemic, made me wonder if there’s another blind spot of EBM that we at SBM have neglected, one that leads to Cochrane reviews like Jefferson’s and leads EBM gurus like Ioannidis to make their heel turns so soon after the pandemic hit . . .

[Regarding the mask report,] perusing the triumphant gloating on social media from ideological sources opposed to COVID-19 interventions, including masks and vaccines, I was struck by how often they used the exact phrase “gold standard” to portray Cochrane as an indisputable source, all to bolster their misrepresentation. . . .

Gorski continues:

I’ve noticed over the last three years a tendency for scientists who were known primarily before the pandemic as strong advocates of evidence-based medicine (EBM), devolving into promoters of COVID-19 denial, antimask, anti-public health, and even antivaccine pseudoscience. Think Dr. John Ioannidis, whom I used to lionize before 2020. Think Dr. Vinay Prasad, of whose work on medical reversals and calls for more rigorous randomized clinical trials of chemotherapy and targeted therapy agents before FDA approval we generally wrote approvingly.

Basically, what Jefferson exhibited in his almost off-the-cuff claim that massive RCTs of masks should have been done while a deadly respiratory virus was flooding UK hospitals was something we like to call “methodolatry,” or the obscene worship of the RCT as the only method of clinical investigation. . . .

But it’s not so simple:

Human trials are messy. It is impossible to make them rigorous in ways that are comparable to laboratory experiments. Compared to laboratory investigations, clinical trials are necessarily less powered and more prone to numerous other sources of error: biases, whether conscious or not, causing or resulting from non-comparable experimental and control groups, cuing of subjects, post-hoc analyses, multiple testing artifacts, unrecognized confounding of data due to subjects’ own motivations, non-publication of results, inappropriate statistical analyses, conclusions that don’t follow from the data, inappropriate pooling of non-significant data from several, small studies to produce an aggregate that appears statistically significant, fraud, and more.

Evidence-based medicine eats itself

For some background on the controversies surrounding “evidence-based medicine,” see this news article from Aaron Carroll from 2017.

Here’s how I summarized things back in 2020, my post entitled “Evidence-based medicine eats itself”:

There are three commonly stated principles of evidence-based research:

1. Reliance when possible on statistically significant results from randomized trials;

2. Balancing of costs, benefits, and uncertainties in decision making;

3. Treatments targeted to individuals or subsets of the population.

Unfortunately and paradoxically, the use of statistics for hypothesis testing can get in the way of the movement toward an evidence-based framework for policy analysis. This claim may come as a surprise, given that one of the meanings of evidence-based analysis is hypothesis testing based on randomized trials. The problem is that principle (1) above is in some conflict with principles (2) and (3).

The conflict with (2) is that statistical significance or non-significance is typically used at all levels to replace uncertainty with certainty—indeed, researchers are encouraged to do this and it is standard practice.

The conflict with (3) is that estimating effects for individuals or population subsets is difficult. A quick calculation finds that it takes 16 times the sample size to estimate an interaction as a main effect, and given that we are lucky if our studies are powered well enough to estimate main effects of interest, it will typically be hopeless to try to obtain the near-certainty regarding interactions. That is fine if we remember principle (2), but not so fine if our experiences with classical statistics have trained us to demand statistical significance as a prerequisite for publication and decision making.

Bridges needed

The above-linked Gorski post was interesting to me because it presents a completely different criticism of the evidence-based-medicine paradigm.

It’s not that controlled trials are bad; rather, the deeper problems seem to be: (a) inferential summaries and decision strategies that don’t respect uncertainty (that was my concern) and (b) research agendas that don’t engage with scientific understanding (that was Gorski’s concern).

Regarding that latter point: a problem with standard “evidence-based medicine” or what I’ve called the “take a pill, push a button model of science” is not that it ignores scientific theories, but rather that it features a gap between theory and evidence. On one side there are theory-stories of varying levels of plausibility; on the other side there are statistical summaries from (necessarily) imperfect study.

What we need are bridges between theory and evidence. This includes sharper theories that make quantitative predictions that can be experimentally studied, and empirical studies measuring intermediate outcomes, and lab experiments to go along with the field studies.

Omid Malekan, who teaches crypto finance at Columbia business school, writes:

I came across your blog post and obviously disagree, I work professionally in crypto, have been following it for close to a decade, written a few books, advise various Columbia student groups, etc.

Instead of trying to refute your claims about crypto being a scam, I thought it would be more fun to demonstrate its value. So I took the text of your blog post and embedded it inside the Bitcoin blockchain. You can see it here. I included a hash of the text as well for easier verification.

What does this mean?

It means this text now sits inside an append-only ledger, copies of which are kept by thousands of distributed computers all over the world. Unlike your original blog post, this data cannot be changed or deleted, because unlike the server that hosts your blog, it now lives inside one of the most secure networks on earth. It would cost someone tens of millions of dollars to even try to change it, and they’d likely fail. This ability to secure and preserve the integrity of information–be it blog posts or transaction data in a payment system–is a unique value prop of Bitcoin. No other digital service provides anything close to it.

But wait, that’s not a financial application!

True, but this “proof of existence” feature would not be possible without Bitcoin also having a native currency. Before the publication of the original Bitcoin paper in 2008, it was believed that securing an open and permissionless distributed system (where participants are free to come and go) was impossible, despite decades of work in the sophisticated field of distributed systems. Nakamoto Consensus was a novel solution that introduced financial incentives to help secure such a system, incentives that need to be paid in the platforms own currency. I had to pay the miner that processed this inscription around $8 for them to do it.

The ability for a distributed group of participants all over the world to reach consensus on anything (like the existence of your blog post) in an open, transparent and permissionless way is a pretty big deal, one that has major implications for the academy. I happen to run my own Bitcoin node (using about $300 worth of hardware). That means I have my own copy of all the data inside the ledger and get the latest updates as they happen. We are in the process of setting up Bitcoin and Ethereum nodes up at the B school to aid professors and students interested in research. Unlike traditional finance, where all the data is hoarded by private entities that charge an arm and a leg for access (at best) or just hide the important details (at worst) in crypto everyone gets to see everything.

And of course as you know all too well, payment systems are nothing more than than a series of records (of debits and credits). And while people paying each other in various cryptocurrencies remains the primary use, the open, decentralized and censorship-resistant architecture has since been expanded to countless other activities, from the movement of dollars (recently embraced by PayPal) to the issuance of consumer rewards (recently embraced by Lufthansa) to digital art (recently embraced by both Christie’s and Sotheby’s).

I’m skeptical about the digital art thing; then again, I’m skeptical about bogus social science on beauty and sex ratios, ovulation and voting, himmicanes, ages ending in 9, etc.—and lots of people who do these things have stable jobs and I guess will continue doing so. It’s a big world out there.

I don’t have any other comments on the specifics of Malekan’s note; I just wanted to share it with you, as he’s offering a different perspective than mine. Paying $8 to save one of my blog posts doesn’t sound quite scalable to me, given how much is out there on the internet; on the other hand, the foolish claim has been made by a celebrity scientist that each scientific citation is worth $100,000. I don’t know what all this means, except that valuations of intangibles can be controversial. In any case, I appreciate Malekan giving us the above background.

Regarding the “scam” thing, I received the following email from Gur Huberman, also at Columbia business school. Gur asked:

1. Do you need a scammer to initiate or support a scam? Would you call Satoshi a scammer? Ditto for Vitalic Butterin (Ethereum)?
2. Is gold a scam?
3. What exactly assigns value to money as we know it?

I replied that a scam builds off some existing institution. Ponzi’s scam was based on certain postage products. The stamps were real. Theranos was based on a real institution of blood testing. Etc. “Crypto” can be a scam even while based on real computer programs. For that matter, gold is not a scam—I have some in my teeth!—but the people who try to sell you gold on TV are scammers. In my post on Dan Davies’s book, I characterized frauds as being linear or exponential.

The point of my earlier post

My post that started this particular discussion was called “Crypto scam social science thoughts: The role of the elite news media and academia.” What was and is most interesting to me, and the point of the post, was the role of media elites in keeping bubbles afloat. So if you disagree with me regarding some benefits of crypto, you can just think about the media fluffing given to FTX etc. in recent years.

Some weeks ago, I posted in Mastodon (you can follow me there) a thread about “How big problem it is that cross-validation is biased?”. I have also added that text to CV-FAQ. Today I extended that thread as we have a new paper out on estimating and correcting selection induced bias in cross-validation model selection.

I’m posting here the whole thread for the convenience of those who are not (yet?) following me in Mastodon:

Unbiasedness has a special role in statistics, and too often there are dichotomous comments that something is not valid or is inferior because it’s not unbiased. However, often the non-zero bias is negligible, and often by modifying the estimator we may even increase bias but reduce the variance a lot, providing an overall improved performance.

In CV the goal is to estimate the predictive performance for unobserved data given the observed data of size n. CV has pessimistic bias due to using less than n observation to fit the models. In case of LOO-CV this bias is usually small and negligible. In case of K -fold-CV with a small K, the bias can be non-negligible, but if the effective number of parameters of the model is much less than n, then with K>10 the bias is also usually negligible compared to the variance.

There is a bias correction approach by Burman (1989) (see also Fushiki (2011)) that reduces CV bias, but even in the cases with non-negligible bias reduction, the variance tends to increase so much that there is no real benefit (see, e.g. Vehtari and Lampinen (2002)).

For time series when the task is to predict future (there are other possibilities like missing data imputation) there are specific CV methods such as leave-future-out (LFO) that have lower bias than LOO-CV or K -fold-CV (Bürkner, Gabry and Vehtari, 2020). There are sometimes comments that LOO-CV and K -fold-CV would be invalid for time series. Although they tend to have a bigger bias than LFO, they are still valid and can be useful, especially in model comparison where bias can cancel out.

Cooper et al. (2023) demonstrate how in time series model comparison variance is likely to dominate, it is more important to reduce the variance than bias, and leave-few-observations and use of joint log score is better than use of LFO. The problem with LFO is that the data sets used for fitting models are smaller, increasing the variance.

Bengio and Grandvalet (2004) proved that there is no unbiased estimate for the variance of CV in general, which has been later used as an argument that there is no hope. Instead of dichotomizing to unbiased or biased, Sivula, Magnusson and Vehtari (2020) consider whether the variance estimates are useful and how to diagnose when the bias is likely to not be negligible (Sivula, Magnusson and Vehtari (2023) prove also a special case where there actually exists unbiased variance estimate).

CV tends to have high variance, as the sample reuse is not making any modeling assumptions (this holds also for information criteria such as WAIC). Not making modeling assumptions is good when we don’t trust our models, but if we trust we can get reduced variance in model comparison, for example, examining directly the posterior or using reference models to filter out noise in the data (see, e.g., Piironen, Paasiniemi and Vehtari (2018) and Pavone et al. (2020)).

When using CV (or information criteria such as WAIC) for model selection, the performance estimate for the selected model has additional selection induced bias. In case of small number of models this bias is usually negligible, that is, smaller than the standard deviation of the estimate or smaller than what is practically relevant. In case of negligible bias, we may choose suboptimal model, but the difference to the performance of oracle model is small.

In case of a large number of models the selection induced bias can be non-negligible, but this bias can be estimated using, for example, nested-CV or bootstrap. The concept of the selection induced bias and related potentially harmful overfitting are not new concepts, but there hasn’t been enough discussion when they are negligible or non-negligible.

In our new paper with Yann McLatchie Efficient estimation and correction of selection-induced bias with order statistics we review the concepts of selection-induced bias and overfitting, propose a fast to compute estimate for the bias, and demonstrate how this can be used to avoid selection induced overfitting even when selecting among 10^30 models.

The figure here shows simulation results with p=100 covariates, with different data sizes n, and varying block correlation among the covariates. The red lines show the LOO-CV estimate for the best model chosen so far in forward-search. The grey lines show the independent, much bigger test data performance, which usually don’t have available. The black line shows our corrected estimate taking into account the selection induced bias. Stopping the searches at the peak of black curves avoids overfitting.

Although we can estimate and correct the selection induced bias, we primarily recommend to use more sensible priors and not to do model selection. See more in Efficient estimation and correction of selection-induced bias with order statistics and Bayesian Workflow.

Social Work Bldg room 903, at 4pm on Mon 18 Sep 2023:

Evaluating Visualizations for Inference and Decision-Making

Research and development in computer science and statistics have produced increasingly sophisticated software interfaces for interactive visual data analysis. Data visualizations have also become ubiquitous for communication in the news and scientific publishing. Despite these successes, our understanding of how to design effective visualizations for data-driven decision-making remains limited. Design philosophies that emphasize data exploration and hypothesis generation can encourage pattern-finding at the expense of quantifying uncertainty. Designing visualizations to maximize perceptual accuracy and self-reported satisfaction can lead people to adopt visualizations that promote overconfident interpretations. I will motivate a few alternative objectives for measuring the effectiveness of visualization, and show how a rational agent framework based in statistical decision theory can help us understand the value of a visualization in the abstract and in light of empirical study results.

This is a super-important topic, also interesting because in many cases people think evaluation is a big deal without thinking too hard about what are the goals of the graph. It’s hard to design a good evaluation without having some goals in mind. For example, see this discussion from a few years ago of a study that was described as finding that “chartjunk is more useful than plain graphs” and this paper with Antony Unwin on different goals of infoviz and statistical and graphics.

Another thing I like about the above abstract from Jessica is how she’s talking about two different goals of statistical graphics: (1) clarifying a point that you want to convey, and (2) providing opportunity for discovery. Both are important!

Tues 19 Sep 2023, 8:30am Pacific time:

Better Than Difference in Differences

Andrew Gelman, Department of Statistics and Department of Political Science, Columbia University

It is not always clear how to adjust for control data in causal inference, balancing the goals of reducing bias and variance. We show how, in a setting with repeated experiments, Bayesian hierarchical modeling yields an adaptive procedure that uses the data to determine how much adjustment to perform. The result is a novel analysis with increased statistical efficiency compared with the default analysis based on difference estimates. The increased efficiency can have real-world consequences in terms of the conclusions that can be drawn from the experiments. An open question is how to apply these ideas in the context of a single experiment or observational study, in which case the optimal adjustment cannot be estimated from the data; still, the principle holds that difference-in-differences can be extremely wasteful of data.

The talk follows up on Andrew Gelman and Matthijs Vákár (2021), Slamming the sham: A Bayesian model for adaptive adjustment with noisy control data, Statistics in Medicine 40, 3403-3424, http://www.stat.columbia.edu/~gelman/research/published/chickens.pdf

Here’s the talk I gave in this seminar a few years ago:

100 Stories of Causal Inference

In social science we learn from stories. The best stories are anomalous and immutable (see http://www.stat.columbia.edu/~gelman/research/published/storytelling.pdf). We shall briefly discuss the theory of stories, the paradoxical nature of how we learn from them, and how this relates to forward and reverse causal inference. Then we will go through some stories of applied causal inference and see what lessons we can draw from them. We hope this talk will be useful as a model for how you can better learn from own experiences as participants and consumers of causal inference.

No overlap, I think.

Campos quotes from one of the many stories floating around regarding ridiculous of cryptocurrency scams.

I’m not saying it should’ve been obvious in retrospect that crypto was a scam, just that (a) it always seemed that it could be a scam, and (b) for awhile there have been many prominent people saying it was a scam. Again, prominent people can be in error; what I’m getting at is that the potential scamminess was out there.

The usual way we think about scams is in terms of the scammers and the suckers, and also about the regulatory framework that lets people get away with it.

Here, though, I want to talk about something different, which is the role of outsiders in the information flow. For crypto, we’re talking about trusted journalistic intermediaries such as Michael Lewis or Tyler Cowen who were promoting or covering for crypto.

There were lots of reasons for respected journalists or financial figures to promote crypto, including political ideology, historical analogies financial interest, FOMO, bandwagon-following, contrarianism, and plain old differences of opinion . . . pretty much the same set of reasons for respected journalists or financial figures to have been crypto-skeptical!

My point here is not that I knew better than the crypto promoters—yes, I was crypto-skeptical but not out of any special knowledge—; rather, it’s that the infrastructure of elite journalism was, I think, crucial to keeping the bubble afloat. Sure, crypto had lots of potential just from rich guys selling to each other and throwing venture capital at it, and suckers watching Alex Jones or whatever investing their life savings, but elite media promotion took it to the next level.

It’s not like I have any answers to this one. There were skeptical media all along, and I can’t really fault the media for spotting a trend that was popular among richies and covering it.

I’m just interested in these sorts of conceptual bubbles, whether they be financial scams or bad science (ovulation and voting, beauty and sex ratio, ESP, himmicanes, nudges, UFOs, etc etc etc), and how they can stay afloat in Wiley E. Coyote fashion long after they’ve been exposed.

Crypto is different from Theranos or embodied cognition, I guess, in that it has no inherent value and thus can retain value purely as part of a Keynesian beauty context, whereas frauds or errors that make actual scientific or technological claims can ultimately be refuted. Paradoxically, crypto’s lack of value—actually, its negative value, given its high energy costs—can make it a more plausible investment than businesses or ideas that could potentially do something useful if their claims were in fact true.

Jennifer Zhang, who took my applied statistics class a few years ago, writes:

I am now working at the Consumer Financial Protection Bureau in DC. I’m writing to share an exciting job opportunity that I hope some of your students would be interested in.

The Consumer Financial Protection Bureau (CFPB), a 21st century government agency that implements and enforces Federal consumer financial law and ensures that markets for consumer financial products are fair, transparent, and competitive, is recruiting this fall for the Director’s Financial Analyst (DFA) position to start in June 2024, and we want to encourage graduating seniors/recent graduates to apply.

ABOUT THE CFPB DFA PROGRAM

A highly selective, two-year rotational fellowship designed to engage young professionals in meaningful public service work. Each year, the CFPB hires a new cohort of Director’s Financial Analysts to gain a unique experience working alongside senior policymakers and researchers to help make financial markets work better for all.

WHAT WE’RE LOOKING FOR

Director’s Financial Analysts utilize their analytical capabilities to support initiatives including, but not limited to, research, benchmarking, comparative analysis, data analysis, problem solving, collaboration, and project management. DFAs will complete two, year-long rotations in various offices gaining an understanding of the CFPB’s mission and work through exposure to multiple policy areas of a mission-driven agency.

Visit the CFPB DFA Program webpage at https://www.consumerfinance.gov/about-us/careers/students-and-graduates/cfpb-directors-financial-analyst-program/ to learn more about program eligibility and application requirements, upcoming informational webinars, and how to apply during the live application window.

If you have questions or want to be added to our mailing list, please send an email to CFPB_DFA_Program@cfpb.gov. You can also reach out to me at jennifer.zhang@cfpb.gov to hear more about my experiences as a DFA.

UPCOMING INFORMATIONAL WEBINARS

The CFPB will host three virtual information sessions about the DFA Program and the application process, including a question-and-answer period conducted over Webex. You must register to attend. Register for a session using the links below.

Webinar Session 1: Wednesday, September 20, 2023; 5:00 – 6:00pm ET

Registration Link: https://cfpbgov.webex.com/weblink/register/rf83b18f475aba74f93747d6d6e2988c7

Webinar Session 2: Wednesday, October 11, 2023; 5:00 – 6:00pm ET

Registration Link: https://cfpbgov.webex.com/weblink/register/r817528d8932b85b5244efe2ef61f69aa

Webinar Session 3: Tuesday, November 7, 2023; 5:00 – 6:00pm ET

Registration Link: https://cfpbgov.webex.com/weblink/register/r4986349e41b145814ff509d5300e3717

Join us for our drop-in Open House sessions held on the days after each webinar. No new information will be presented at these sessions, but they serve as an opportunity to meet with DFA Program Management, Recruiting Leads, and current DFAs to ask any remaining questions about the DFA program and application process.

Open House Session 1: Thursday, September 21, 2023; 1:00pm – 3:00pm ET
https://www.microsoft.com/microsoft-teams/join-a-meeting
Meeting ID: 220 187 354 225
Passcode: eXDib8

Open House Session 2: Thursday, October 12, 2023; 2:00pm – 4:00pm ET
https://www.microsoft.com/microsoft-teams/join-a-meeting
Meeting ID: 228 863 610 690
Passcode: DGFYmb

Open House Session 3: Wednesday, November 8, 2023; 3:00pm – 5:00pm ET
https://www.microsoft.com/microsoft-teams/join-a-meeting
Meeting ID: 217 715 703 572
Passcode: bBP8hy

Someone who would like to remain anonymous writes:

I’m a student during the school year, but am working in industry this summer. I am currently attempting to overhaul my company’s model of retail demand. We advise suppliers to national retailers, our customers are suppliers. Right now, for each of our customers, our demand model outputs a point estimate of how much of their product will be consumed at one of roughly a hundred locations. This allows our customers to decide how much to send to each location.

However, because we are issuing point estimates of mean demand, we are *not* modeling risk directly, and I want to change that, as understanding risk is critical to making good decisions about inventory management – the entire point of excess inventory is to provide a buffer against surprises.

Additionally, the model currently operates on a per-day basis, so that predictions for a month from now are obtained by chaining together thirty predictions about what day N+1 will look like. I want to change that too, because it seems to be causing a lot of problems with errors in the model propagating across time, to the point that predictions over even moderate time intervals are not reliable.

I already know how to do both of these in an abstract way.

I’m willing to bite the bullet of assuming that the underlying distribution of the PDF should be multivariate Gaussian. From there, arriving at the parameters of that PDF just requires max likelihood estimation. For the other change, without going into a lot of tedious detail, Neural ODE models are flexible with respect to time such that you can use the same model to predict the net demand accumulated over t=10 days as you would to predict the net demand accumulated over t=90 days, just by changing the time parameter that you query the model with.

The problem is, although I know how to build a model that will do this, I want the estimated variance for each customer’s product to be individualized. Yet frustratingly, in a one-shot scenario, the maximum likelihood estimator of variance is zero. The only datapoint I’ll have to use to train the model to estimate the mean aggregate demand for, say, cowboy hats in Seattle at time t=T (hereafter (c,S,T)) is the actual demand for that instance, so the difference between the mean outcome and the actual outcome will be zero.

It’s clear to me that if I want to arrive at a good target for variance or covariance in order to conduct risk assessment, I need to do some kind of aggregation over the outcomes, but most of the obvious options don’t seem appealing.

– If I obtain an estimate of variance by thinking about the difference between (c,S,T) and (c,Country,T), aggregating over space, I’m assuming that each location shares the same mean demand, which I know is false.

– If I obtain one by thinking about the difference between (c,S,T) and (c,S,tbar), aggregating over time, I am assuming there’s a stationary covariance matrix for how demand accumulates at that location over time, which I know is false. This will fail especially badly if issuing predictions across major seasonal events, such as holidays or large temperature changes.

– If I aggregate across customers by thinking about the difference between (c,S,T) and (cbar,S,T), I’ll be assuming that the demand for cowboy hats at S,T should obey similar patterns as the demand for other products, such as ice cream or underwear sales, which seems obviously false.

I have thought of an alternative to these, but I don’t know if it’s even remotely sensible, because I’ve never seen anything like it done before. I would love your thoughts and criticisms on the possible approach. Alternatively, if I need to bite the bullet and go with one of the above aggregation strategies instead, it would benefit me a lot to have someone authoritative tell me so, so that I stop messing around with bad ideas.

My thought was that instead of asking the model to use the single input vector associated with t=0 to predict a single output vector at t=T, I could instead ask the model to make one prediction per input vector for many different input vectors from the neighborhood of time around t=0 in order to predict outcomes at a neighborhood of time around t=T. For example, I’d want one prediction for t=-5 to t=T, another prediction for t=-3 to t=T+4, and so on.

I would then judge the “true” target variance for the model relative to the difference between (c,S,T)’s predicted demand and the average of the model’s predicted demands for those nearby time slices. The hope is that this would reasonably correspond to the risks that customers should consider when optimizing their inventory management, by describing the sensitivity of the model to small changes in the input features and target dates it’s queried on. The model’s estimate of its own uncertainty wouldn’t do a good job of representing out-of-model error, of course, but the hope is that it’d at least give customers *something*.

Does this make any sense at all as a possible approach, or am I fooling myself?

My reply: I haven’t followed all the details, but my guess is that your general approach is sound. It should be possible to just fit a big Bayesian model in Stan, but maybe that would be too slow, I don’t really know how big the problem is. The sort of approach described above, where different models are fit and compared, can be thought of as a kind of computational approximation to a more structured hierarchical model, in the same way that cross-validation can be thought of as an approximation to an error model, or smoothing can be thought of as an approximation to a time-series model.

One of them was hyped in the media as a clean-cut fresh face that would restore fan confidence in a tired, scandal-plagued entertainment cartel—and the other is a retired baseball player.

(Reposted from last year because the topic remains important and because I think it’s a funny line.)

Michael Stutzer writes:

This study documents substantial variability in different researchers’ results when they use the same financial data set and are supposed to test the same hypotheses. More generally, I think the prospect for reproducibility in finance is worse than in some areas, because there is a publication bias in favor of a paper that uses a unique dataset provided by a firm. Because this is proprietary data, the firm often makes the researcher promise not to share the data with anybody, including the paper’s referees.

Read the leading journals’ statements carefully and you find that they don’t strictly require sharing.

Here is the statement for authors made by the Journal of Financial Econometrics: “Where ethically and legally feasible, JFEC strongly encourages authors to make all data and software code on which the conclusions of the paper rely available to readers. We suggest that data be presented in the main manuscript or additional supporting files, or deposited in a public repository whenever possible.”

In other words, an author wouldn’t have to share a so-called proprietary data set as defined above, even with the papers’ referees. What is worse, the leading journals not only accept these restrictions, but seem to favor such work over what is viewed as more garden-variety work that employs universally available datasets.

Intersting. I think it’s just as bad in medical or public health research, but there the concern is sharing confidential information. Even in settings where it’s hard to imagine that the confidentiality would matter.

As I’ve said in other such settings, the authors of research papers have no obligation to share their data and code, and I have no obligation to believe anything they write.

That is, my preferred solution is not to nag people for their data, it’s just to move on. That said, this strategy works fine for silly examples such as fat arms and voting, or the effects of unionization on stock prices, but you can’t really follow it for research that is directly relevant to policy.

Xinru Wang, Lauren Kennedy, and Qixuan Chen write:

The two-phase sampling design is a cost-effective sampling strategy that has been widely used in public health research. The conventional approach in this design is to create subsample specific weights that adjust for probability of selection and response in the second phase. However, these weights can be highly variable which in turn results in unstable weighted analyses. Alternatively, we can use the rich data collected in the first phase of the study to improve the survey inference of the second phase sample. In this paper, we use a Bayesian tree-based multiple imputation (MI) approach for estimating population means using a two-phase survey design. We demonstrate how to incorporate complex survey design features, such as strata, clusters, and weights, into the imputation procedure. We use a simulation study to evaluate the performance of the tree-based MI approach in comparison to the alternative weighted analyses using the subsample weights. We find the tree-based MI method outperforms weighting methods with smaller bias, reduced root mean squared error, and narrower 95% confidence intervals that have closer to the nominal level coverage rate. We illustrate the application of the proposed method by estimating the prevalence of diabetes among the United States non-institutionalized adult population using the fasting blood glucose data collected only on a subsample of participants in the 2017-2018 National Health and Nutrition Examination Survey.

Yes, weights can be variable! Poststratification is better, but we don’t always have the relevant information. Imputation is a way to bridge the gap. Imputations themselves are model-dependent and need to be checked. Still, the alternatives of ignoring design calculations or relying on weights have such problems of their own, that I think that modeling is the way to go. Further challenges will arise such as imputing cluster membership in the population.

Paul Alper writes, regarding my post criticizing an epidemiologist and a psychologist who were coming down from the ivory tower to lecture us on “concrete values like freedom and equality”:

In your P.P.S. you write,

Yes, I too am coming down from the ivory tower to lecture here. You’ll have to judge this post on its merits, not based on my qualifications. And if I go around using meaningless phrases such as “concrete values like freedom and equality,” please call me on it!

While this sounds reasonable, is it not sort of anti-Bayes? By that I mean your qualifications represent a prior and the merits the (new) evidence. I am not one to revere authority but deep down in my heart, I tend to pay more attention to a medical doctor at the Mayo Clinic than I do to Stella Immanuel. On the other hand, decades ago the Mayo Clinic misplaced (lost!) a half liter of my urine and double charged my insurance when getting a duplicate a few weeks later.

Alper continues:

Upon reflection—this was over 25 years ago—a half liter of urine does sound like an exaggeration, but not by much. The incident really did happen and Mayo tried to charge for it twice. I certainly have quoted it often enough so it must be true.

On the wider issue of qualifications and merit, surely people with authority (degrees from Harvard and Yale, employment at the Hoover Institution, Nobel Prizes) are given slack when outlandish statements are made. James Watson, however, is castigated exceptionally precisely because of his exceptional longevity.

I don’t have anything to say about the urine, but regarding the Bayesian point . . . be careful! I’m not saying to make inferences or make decisions solely based on local data, ignoring prior information coming from external data such as qualifications. What I’m saying is to judge this post on its merits. Then you can make inferences and decisions in some approximately Bayesian way, combining your judgment of this post with your priors based on your respect for my qualifications, my previous writings, etc.

This is related to the point that a Bayesian wants everybody else to be non-Bayesian. Judge my post on its merits, then combine with prior information. Don’t double-count the prior.