ChatGPT: Making Librarians Look Good Since 2022

Well, my mind hasn't changed: AI won't be coming for librarian jobs any time soon. Until someone builds and AI that is focused on data and research--instead of a language model like ChatGPT--replacing us is a laughable concept.

Some of my fellow PTRC representatives have been experimenting with ChatGPT since I last posted about my initial adventures in February. For those who may not recall, I found that the language model AI was unable to perform any sort of patent or trademark search, or provide specific data on patents or trademarks.

That has not changed over the past few months' updates. I tested again by asking for a specific patent number applied to the patented plant featured in my last post (Hibiscus 'Panama Red'). Again, ChatGPT was unable to give me a patent number.

ChatGPT still can't provide patent numbers

I decided to try the approach made by my fellow rep mentioned above, Stella Mittelbach of the Los Angeles Public Library Science, Technology & Patents Department. She requested Cooperative Patent Classification (CPC) numbers that may be applied to an invention, and was provided with an extensive list of potential matches. I was curious about the quality of CPC numbers ChatGPT could provide.

In recent publications and casual conversations, I've learned that ChatGPT struggles to provide concrete data and references, sometimes producing "hallucinations", which is made-up information presented as factual. Furthermore, searching CPC numbers can be challenging, and narrowing down the candidates to a category that most closely matches an invention is comparable to the initial stages of a patent search. It surprised me that ChatGPT was capable of returning such specific information.

Thanks to teaching courses on searching for CPC numbers and using them to search for patents, I have become extremely familiar with a few groups and subgroups. I decided to ask ChatGPT to suggest CPC numbers for the hypothetical invention I typically use: a new contact lens material that makes it safe to sleep in them, that is an improved silicone hydrogel with higher oxygen permeability. I know two potential CPC numbers that could be applied to this invention, just based on that description.

These numbers are so wrong, don't bother looking too closely.

ChatGPT was off-base. It supplied numbers that were mostly unrelated, yet has some aspects that used similar terminology. I followed up with asking for the definition of the CPC number G02C 7/001, which I knew was for bifocal contact lenses. Again, ChatGPT failed to give accurate information.

It's also wrong

I corrected the AI.

My correction spurred it to give some better information

 

Second correction, with a suggestion

Then I asked where it was obtaining its information. It vaguely cited "general knowledge", which means nothing. CPC numbers are something I would classify as "specific knowledge", rarely known to any extent by the vast majority of the populace. I'd wager most people are unaware the CPC exists. 

ChatGPT seems unsure of its information sources

Since the AI is supposed to learn from interactions, I wanted to test if my corrections were somehow absorbed. I emailed PTRC colleagues a copy of the transcript, and requested someone else attempt my questions, and share the answers given.

Dave Bloom, Science and Engineering/PTRC Librarian of University of Wisconsin - Madison tried my initial prompts verbatim using multiple ChatGPT accounts and one for Google Bard, a few times each. He found each response was "noticeably different, not necessarily informed by the other answers, and, most importantly, not reliable."

A sample of Dave Bloom's interactions

 

Of course I tested this myself, on two subsequent days to see if that made a difference. 

Similar to Dave Bloom, I got multiple answers. When I submitted a correction, I was able to choose a "better" answer, but this is not an option that someone actually trying to find a CPC number using ChatGPT would have. Furthermore, the better answer is based on the specific feedback I provided. A different account might receive the wrong information, and not be able to correct it. OpenAI should program it to stick with an instructional response, which is never wrong.

The next day

Before I made a suggestion through the system

After my suggestion through the system

Over a couple of days, and especially when I provided feedback not in chat, but through the feedback form, there was some slight improvement.

On the second day...

the answers seemed slightly improved...

but are still inaccurate.

Dave Bloom later reminded me that when using a large language model (LLM) like ChatGPT, "it's tempting to evaluate each individual response on the basis of accuracy. [...] It’s not about whether any individual answer is right or wrong or whether ChatGPT or Bard is better at generating “correct” answers, but that, even with the exact same prompt, LLMs do not produce the same results. Replicability is a baseline expectation of credible scientific research, and we should apply it to search tools, too." They don't provide reliable source responses, because their function is to produce answers that appear to be written by humans. Dave explained that this means the fake citations and URLs are therefore often very convincing.

So, even when ChatGPT can provide an answer, they're at high risk of being very bad answers. Librarians have dodged the bullet again!

I wasn't worried about you, buddy; my concern is decision makers who like to save money

USPTO Cooperative Patent Classification (CPC) Meeting with Industry Users, Sept. 19, 2016

The Federal Register Notice ,Vol. 81, No. 167, Mon., Aug. 29, 2016, p. 59200, provides the following information about the Third Annual USPTO Cooperative Patent Classication Meeting With Industry Users:

SSUMMARY: The United States Patent and Trademark Office (USPTO) is hosting its third annual Cooperative Patent Classification (CPC)meeting with industry users at its Alexandria Campus. CPC is a bilateral classification system jointly developed by the USPTO and the European Patent Office (EPO). CPC is jointly managed and maintained by both offices, and is available for public search for classification.

The upcoming meeting is open to the public and will inform attendees on the latest and upcoming developments concerning the CPC. The meeting

also will be an opportunity for attendees to share their views about the CPC.

DATES: The meeting will be held on Monday, September 19, 2016,beginning at 9:30 a.m. Eastern Daylight Time (EDT), and ending at 5:00 p.m. EDT.

ADDRESSES: The meeting will be held at the USPTO in the Madison Auditorium on the concourse level of the Madison Building located at 600 Dulany Street, Alexandria, Virginia 22314.

FOR MEETING REGISTRATION: There is no fee to register for the meeting and registration will be on a first-come, first-serve basis. Early registration is recommended because seating is limited. Registration on the day of the meeting (September 19, 2016) will be permitted on a space-available basis beginning at 8:30 a.m. EDT.

To register, please provide your name and phone number to CPCannualmeeting@uspto.gov. Registrants also may choose to identify their company or organization, and their position thereat, so that a workshop to be held during the meeting may be best tailored to meet the attendees' needs.

FOR FURTHER INFORMATION CONTACT: Christopher Kim, Director of Classification Quality and International Coordination Division (CQIC), Office of International Patent Classification (OIPC), by telephone at 571-272-7980, or by electronic mail message at CPCannualmeeting@uspto.gov.

SUPPLEMENTARY INFORMATION: The CPC is a detailed classification system in effect at the USPTO and based on the International Patent Classification scheme. The CPC incorporates best classification practices from both the U.S. and European systems.

The USPTO and the EPO designed the CPC to enable patent examiners and patent system users worldwide to conduct more efficient prior art searches. It provides an increased number of breakdowns compared to the U.S. Patent Classification System, allowing for targeted searches with more focused results. The CPC also was designed to encourage work sharing initiatives focused on enhancing efficiency by reducing duplicative work. Additionally, revisions to the CPC system can be made by both Offices on a regular basis, allowing for a rapid response to filing trends and emerging technologies.

At the September 19, 2016 meeting, key USPTO executive staff and project managers will brief attendees on the progress of the CPC, including recent changes and updates to the CPC. Attendees will receive information concerning external user interaction, accessibility, and outreach related to the CPC. In addition, a workshop will be held to provide training on the use of the CPC. Similar informational meetings have been held in Europe.

For further information about the CPC and the September 19, 2016 meeting, including the agenda for the meeting, please visit www.cpcinfo.org.

Dated: August 23, 2016.

Russell Slifer,

Deputy Under Secretary of Commerce for Intellectual Property and Deputy Director of the United States Patent and Trademark Office.

See the complete Federal Register Notice at https://www.gpo.gov/fdsys/pkg/FR-2016-08-29/pdf/2016-20700.pdf

Cooperative Patent Classification (CPC) External User Day Event

On Friday, Dec. 14, 2012, from 8:30 a.m. to noon (EST) the United States Trademark Office (USPTO) will host the next Cooperative Patent Classification (CPC) External User Day event at its campus in Alexandria, Virginia. Information about the event is available from a USPTO press release or on the CPC Web site. The event "is designed to provide external stakeholders with in-depth information about the October 2012 CPC launch package, as well as insight into upcoming CPC training platforms, in advance of the official launch of the CPC system in January 2013."

There is no fee to register for the event. Registration will be on a first-come, first-serve basis. To register, please email your name and phone number to CPC_Users_Day@uspto.gov.

If you are not able to attend in person, check the USPTO website at www.uspto.gov the day before the event for instructions about accessing the webcast.

New Patent Classification System User Information

[Federal Register Volume 77, Number 122 (Monday, June 25, 2012)]
[Notices][Page 37879]
Federal Register Online via the Government Printing Office [www.gpo.gov]
[FR Doc No: 2012-15447]
-------------------------------------------------------------------------------------
DEPARTMENT OF COMMERCE
Patent and Trademark Office
[Docket No. PTO-P-2012-0026]

Cooperative Patent Classification External User Day
AGENCY: United States Patent and Trademark Office, Commerce.
ACTION: Notice.
To read the complete text of the notice, access
 http://www.gpo.gov/fdsys/pkg/FR-2012-06-25/pdf/2012-15447.pdf  
------------------------------------------------------------------------------------
SUMMARY: The United States Patent and Trademark Office (USPTO) is 
hosting a Cooperative Patent Classification (CPC) External User Day 
event at its Alexandria Campus. CPC is a partnership between the USPTO 
and the European Patent Office (EPO) in which the Offices have agreed 
to develop a joint patent classification system that will incorporate 
the best classification practices of the two Offices. This CPC event is 
the next step, in a series of steps, to be undertaken by the USPTO in 
educating and informing external stakeholders about the current 
development and future implementation plans of the CPC.

DATES: The event will be held on Tuesday, July 10, 2012, beginning at 
8:30 a.m. Eastern Standard Time (EST), and ending at 12:00 p.m. EST.

ADDRESSES: The event will be held at the USPTO in the Madison 
Auditorium on the concourse level of the Madison Building located at 
600 Dulany Street, Alexandria, Virginia 22314.

For Event Registration: There is no fee to register for the event 
and registration will be on a first-come, first-serve basis. Early 
registration is recommended because seating is limited. Registration on 
the day of the event (July 10, 2012) will be permitted on a space-
available basis beginning at 8:30 a.m., Eastern Standard Time.
To register, please provide your name and phone number to
 CPC_Users_Day@uspto.gov.
FOR FURTHER INFORMATION CONTACT: Derris Banks, Supervisory Patent 
Examiner, TC 3700, by telephone at (571) 272-4419, or by electronic 
mail message at derris.banks@uspto.gov, or Linda Dvorak, Supervisory 
Patent Examiner, TC 3700, by telephone at (571) 272-4764, or by 
electronic mail message at linda.dvorak@uspto.gov.