@tristanbuckmaster

2/3 OpenAI is drawing a distinction that its answer to me erased, despite the fact that my question explicitly made that distinction.
We are not required to reverse-engineer OpenAI’s internal training pipeline to establish what happened. Only OpenAI has the relevant data for that. If Sellke and Bubeck give a categorical denial, it must disclose the basis: product and privacy settings, relevant datasets and checkpoints, and what “de-identified data derived from usage” means.
I disabled model training on 29 June. That control is still only a promise whose implementation users cannot audit, and it is prospective: it does not answer what happened to earlier conversations or to derivatives already selected. OpenAI’s answer did not mention the setting or any account-specific check.
There is also a mathematical parallel, though a less explicit one. The approach of Kun and myself was not the main line of attack on non-soficity, in fact there were other more promising approaches along the line of quantum games etc. OpenAI’s detailed command of our techniques therefore made me wonder how the model found this route. If nonpublic research supplied by users improved a model and the provider then used that model to race those users to publication - without informed consent, disclosure, or credit - that would be ethically indefensible. De-identification may remove a name; it does not remove the intellectual content of a mathematical idea. Sellke and Bubeck seem to be blind to this simple moral aspect.

@tristanbuckmaster

1/3 I want to record a number of points that I realized after reading up on the controversy. In fact it recalls an exchange I had with OpenAI after its non-sofic-group announcement. It exposes a serious, unanswered question about whether researchers can trust OpenAI with unpublished mathematics.

I wrote in an email to Mark Sellke and Sebastien Bubeck shortly after the surprising finding of a non-sofic group using the methods of Kun and myself:
“Another point is that I and a colleague in Dresden were discussing the expander matching problem and various extensions of the work with Gabor Kun actively over the last months with ChatGPT, so that we are of course curious if that was part of the training data or accessible to the reasoning process. There is a certain (frankly unacceptable) lack of transparency here; and I fear it will damage the communal process of math more than the new AI-generated results will benefit the subject.”

Mark Sellke’s complete answer was: “Regarding your conversations with ChatGPT: that did not happen.”

I had explicitly asked about two different things: (1) whether our conversations entered training data, and (2) whether they were accessible to the solving process. The categorical answer now looks as though it addressed only direct access under (2). No such qualification, explanation, or evidence was given. I take this as dishonesty to say the least.
OpenAI now says in the Buckmaster-Alpöge case that no specific user data was accessed, but adds that it “cannot rule out that de-identified data derived from their usage of our products helped improve our models.”

@tristanbuckmaster on training, the answer to your question is that if you didn't "opt out," your data is part of a large corpus of chats that is used for RL. Unclear when their last RL run was, and how likely it would have been for models to memorize that very small fragment of RL data. Being scooped and then pressured to commit academic fraud is the bigger story to me though.

I want to again express my profound gratitude to the outpouring of support from the greater mathematical community. I also want to remark that there is a far bigger story here than the one in my statement: the sheer magnitude of what frontier models can now do, and what that means for us all. I hope the labs can see this and set the petty posturing aside.

@tristanbuckmaster

"I asked when the first prompt had been sent by them. This question was not answered directly by OpenAI for some time. Eventually it was agreed that it had been sent in the past few days, after information about our work had reached OpenAI.

I asked whether the model had been trained on, or had access to, our sessions in Codex, into which we had been putting all our drafts for the whole of this project. I was told the model did not look up user data. I asked again, about training, and I did not get an answer."

There would be little doubt in my mind that they would have access. These companies have zero scruples.

You may not wish to hear this, but it's the price you will pay for selling your soul to the devil. The devil will keep you to the bargain.