ChatGPT: Useful At Last

Y'all, I finally found a use for ChatGPT (and other similar LLMs) in my work.

Have you ever read a patent, and felt like it wasn't written in English? Even if you know something about the field, it just doesn't seem to communicate concepts in a way most humans can understand.

I've spent a lot of time trying to read and understand patents, and I've spent a lot of that time not understanding what I am reading. Even some of the simpler inventions I've shared--like the cat tree in this post--may have sections difficult to comprehend. 

Well, for my third post on ChatGPT (and other LLM AIs), I can tell you it has finally helped me in a significant way: ChatGPT will explain a patent to you in simple, plain English.

Requesting an explanation

Most of the explanation provided

Of course, significant limitations should be applied. If you provide a patent number to the free, historical info-only version of ChatGPT, chances are it will tell you about the wrong patent.

Thanks for explaining something else

And if the text you copy from a patent into the text entry box is too long to get around that, it will cause an error and not be able to respond. 

I'm uncertain what amount of text is too much

But the live info-version of ChatGPT that powers Bing Chat Enterprise did provide a good explanation for Rice's most recent patent, US 11,767,291 "Preparation of Secondary Amines with Electrophilic N-Linchpin Reagents", without me needing to copy and paste the patent's text. 

This was a really good explanation that included molecule diagrams!

I have found that just asking for info about a patent, or providing just a number or a title, can lead to some incorrect information. So if you want accuracy, pasting sections of text or specifying patent number, title, and maybe even more might be best. 

Bing decided it didn't want to deal with my corrections

You might also get a "link" that is just plain text

Oh, and you have to use Microsoft Edge to access Bing Chat. Not my preferred internet browser.

But even with these somewhat significant drawbacks, at least I got an explanation for what all these embodiments and disclosures really mean.

ChatGPT: Making Librarians Look Good Since 2022

Well, my mind hasn't changed: AI won't be coming for librarian jobs any time soon. Until someone builds and AI that is focused on data and research--instead of a language model like ChatGPT--replacing us is a laughable concept.

Some of my fellow PTRC representatives have been experimenting with ChatGPT since I last posted about my initial adventures in February. For those who may not recall, I found that the language model AI was unable to perform any sort of patent or trademark search, or provide specific data on patents or trademarks.

That has not changed over the past few months' updates. I tested again by asking for a specific patent number applied to the patented plant featured in my last post (Hibiscus 'Panama Red'). Again, ChatGPT was unable to give me a patent number.

ChatGPT still can't provide patent numbers

I decided to try the approach made by my fellow rep mentioned above, Stella Mittelbach of the Los Angeles Public Library Science, Technology & Patents Department. She requested Cooperative Patent Classification (CPC) numbers that may be applied to an invention, and was provided with an extensive list of potential matches. I was curious about the quality of CPC numbers ChatGPT could provide.

In recent publications and casual conversations, I've learned that ChatGPT struggles to provide concrete data and references, sometimes producing "hallucinations", which is made-up information presented as factual. Furthermore, searching CPC numbers can be challenging, and narrowing down the candidates to a category that most closely matches an invention is comparable to the initial stages of a patent search. It surprised me that ChatGPT was capable of returning such specific information.

Thanks to teaching courses on searching for CPC numbers and using them to search for patents, I have become extremely familiar with a few groups and subgroups. I decided to ask ChatGPT to suggest CPC numbers for the hypothetical invention I typically use: a new contact lens material that makes it safe to sleep in them, that is an improved silicone hydrogel with higher oxygen permeability. I know two potential CPC numbers that could be applied to this invention, just based on that description.

These numbers are so wrong, don't bother looking too closely.

ChatGPT was off-base. It supplied numbers that were mostly unrelated, yet has some aspects that used similar terminology. I followed up with asking for the definition of the CPC number G02C 7/001, which I knew was for bifocal contact lenses. Again, ChatGPT failed to give accurate information.

It's also wrong

I corrected the AI.

My correction spurred it to give some better information

 

Second correction, with a suggestion

Then I asked where it was obtaining its information. It vaguely cited "general knowledge", which means nothing. CPC numbers are something I would classify as "specific knowledge", rarely known to any extent by the vast majority of the populace. I'd wager most people are unaware the CPC exists. 

ChatGPT seems unsure of its information sources

Since the AI is supposed to learn from interactions, I wanted to test if my corrections were somehow absorbed. I emailed PTRC colleagues a copy of the transcript, and requested someone else attempt my questions, and share the answers given.

Dave Bloom, Science and Engineering/PTRC Librarian of University of Wisconsin - Madison tried my initial prompts verbatim using multiple ChatGPT accounts and one for Google Bard, a few times each. He found each response was "noticeably different, not necessarily informed by the other answers, and, most importantly, not reliable."

A sample of Dave Bloom's interactions

 

Of course I tested this myself, on two subsequent days to see if that made a difference. 

Similar to Dave Bloom, I got multiple answers. When I submitted a correction, I was able to choose a "better" answer, but this is not an option that someone actually trying to find a CPC number using ChatGPT would have. Furthermore, the better answer is based on the specific feedback I provided. A different account might receive the wrong information, and not be able to correct it. OpenAI should program it to stick with an instructional response, which is never wrong.

The next day

Before I made a suggestion through the system

After my suggestion through the system

Over a couple of days, and especially when I provided feedback not in chat, but through the feedback form, there was some slight improvement.

On the second day...

the answers seemed slightly improved...

but are still inaccurate.

Dave Bloom later reminded me that when using a large language model (LLM) like ChatGPT, "it's tempting to evaluate each individual response on the basis of accuracy. [...] It’s not about whether any individual answer is right or wrong or whether ChatGPT or Bard is better at generating “correct” answers, but that, even with the exact same prompt, LLMs do not produce the same results. Replicability is a baseline expectation of credible scientific research, and we should apply it to search tools, too." They don't provide reliable source responses, because their function is to produce answers that appear to be written by humans. Dave explained that this means the fake citations and URLs are therefore often very convincing.

So, even when ChatGPT can provide an answer, they're at high risk of being very bad answers. Librarians have dodged the bullet again!

I wasn't worried about you, buddy; my concern is decision makers who like to save money

My AI Replacement: Coming Soon?

Given the massive amount of attention OpenAI’s new chatbot, ChatGPT, has received (especially as a harbinger of the imminent AI takeover), I thought I'd test its performance as a PTRC representative. So, I spent a recent rainy afternoon formulating questions and interrogating ChatGPT's abilities. I was relieved to find my immediate replacement is not likely, but I should still watch my back.

ChatGPT generated fairly accurate, if brief, instructions on how to search for patents on Espacenet, with or without a given topic. 

Basic steps for searching with Espacenet

However, it has not yet learned about the USPTO’s Patent Public Search. When I asked for Patent Public Search Advanced instructions, I was directed to the WIPO. Though the instructions seemed fine, PATENTSCOPE and Patent Public Search were both misidentified.

Instructions for WIPO, not USPTO

I rephrased the question to stipulate a USPTO search tool, and was extremely reassured to find ChatGPT still thought PatFT and AppFT were available. I have a year's advancement beyond the AI. Of course, I took a moment to provide feedback and inform OpenAI that the answer was incorrect.

Wrong tools, right agency

ChatGPT is completely incapable of performing a patent search. When I requested both general and more specific patent searches, it responded with an unequivocal “No!” (if in more polite terms). Even when asked to find basic information on granted patents or patent data, like how many patents were issued to Rice University in 2022 (17), it could not answer.

CharGPT cannot provide patent data

I followed up with an easier search, asking for the grocery store nearest to Rice University, which caused several errors. Later, it managed to locate a Randalls—with the exact same address as Rice.

This is gratifying. I’ve been telling people for years that the searching abilities of librarians are irreplaceable, and AI has a very long way to go before it can try.

Following that, I changed topics, and asked general and specific questions about trademarks. I was surprised when ChatGPT immediately provided instructions for obtaining a trademark in Togo, the smallest country that first came to mind. Other basic questions, such as “Does Rice University have trademarks?” were also easily answered.

How to obtain a trademark in Togo

Questions about specific entities, trademarks, searching, or data were not possible. ChatGPT could not tell me what Rice’s first trademark was, or how many trademarks Rice University’s athletics department has. I anticipated as much from roughly similar types of patent queries.

ChatGPT is also uncertain about trademark data

Lawyers may be in bigger trouble than librarians. ChatGPT generated two fairly plausible examples of a patent content, one using its own example and one using mine. They were incomplete, and included substantial warnings about relying on an AI and the advisability of seeking legal counsel, but still composed some sections. I imagine an inventor could input enough information about a real innovation and receive a fairly helpful set of claims or descriptions.

The first part of an example patent document

Warning provided for all patent document examples

Next I asked for trademark applications, first with no information beyond brand name, and then with additional details like product and business location. The more details provided, the better the application--to an extent. The content was very generic and left many blank spaces to be filled, but it managed to correctly classify the product (athletic shoes, 25).

Example trademark application with less information

Example trademark application with more information

ChatGPT noticeably didn’t make any distinctions in application content for any IP office, and definitely did not consider type of mark or mark specifications. Again, similar to patents, it produced something that could guide a person struggling to write a trademark application.

I’ll keep an eye on it, though, and let everyone know when I’m being replaced by an online, open-source AI chatbot.