GPT as Math Research Assistant?
Yesterday I suddenly remembered that GPTs existed. Could I use it to create a “research assistant” for my work by adding a bunch of relevant papers to its knowledge base and using a GPT to query the information in those papers? I started the GPT builder and told the creator chatbot this request. (For all chatbots, I enabled the code interpreter.)
Surprisingly, that attempt fell flat. I had thought that since GPT is supposed to be good at summarizing documents I could ask it questions about the documents I uploaded. Instead, when I asked it what $\text{Ver}_4^+$ was, it told me that $\text{Ver}_p^+$ was only defined for $p$ prime. (I had in fact uploaded the Benson et al. paper defining the higher Verlinde categories, so...I also knew that parts of the paper give the concrete definition that I've been working with for, like, the past year??)
OK, maybe that was too hard. My second attempt involved giving it just Jantzen's book on Representations of Algebraic Groups. No more research assistant, the prompt given to the creator chatbot was just to make an expert on the book who could help someone reading it by giving definitions, examples, theorem proofs, and intuition. (I admit that the book's OCR is not 100% accurate, but it's mostly fine.) I proceeded to ask the chatbot how Jantzen defined the Lie algebra of an algebraic group and I got basically “it's related to the distribution algebra by taking $\text{Dist}^+(G)$“. When I pressed it for an actual definition, it told me Jantzen never defined the Lie algebra. It did manage to repeat verbatim the definition of an induced module, though. When I asked it what the interpretation was in terms of invariant functions it kinda crashed, so I gave up.
OK, maybe no more math. Third time's the charm – let's make a chatbot to help me with Postgres stuff! I asked the chatbot for a GPT that was an expert in the Postgres C API and could explain internals to someone who wanted to modify Postgres via hooks (me). I didn't need to upload any files here. I asked the generated GPT about how a query showed up in the various fields of the SeqScan node. Boom, it summarized everything I had been trying to figure out this weekend. Definitely using it for future work.
I guess even though the Postgres API feels just as obscure of the stuff I'm working on, it's actually way less obscure because only like 9 people think about the stuff I do, so the ONLY exposure GPT is getting to Verlinde categories is their work, whereas many many people work with Postgres.
tl;dr just make GPTs for coding to save time reading documentation, not for niche research only you and your advisor and his collaborators do.