Like most people with an internet connection I've been thrilled (and a little concerned) with the recent developments in AI, specifically Large Language Models (LLM), like Chat GPT.

The potential is obviously massive and I wanted to have a play with Chat GPT while working on a small app. I have always thought a nice idea for a browser extension would be one that 'fixes' website tables, by automatically adding sort handles to the table columns. I expected it wouldn't be too hard and so decided to use it as a test case to experiment with the AI interface.

** 1. First **

I made a few soft rules for myself before I began. First, I wanted to use ChatGPT as my only resource, as much as practical. I also didn't want to create a table sorting library, so that would not be part of the test. For the record, I have made some simple browser extensions, years ago, but I'm not an expert. My method would be to simply ask questions to the "Default (GPT-3.5)" version of ChatGPT and follow the response.

The first question I asked was, appropriately, "How to make a browser extension?"

It returned a list of steps which were pretty straight forward web development with the writing of a manifest.json file the only thing I needed help with.

I realised my question was too broad, so I specified "How to make a chrome browser extension?"

This answer was much more detailed, and included coding examples for the manifest.json and advice on the project layout, suggesting I should make popout.html and popout.css files. It even provided instructions on how to load my code into Chrome for testing and how to publish it on the Chrome Web Store, with links to the store and the official extension documentation.

This is quite impressive. It was able to go beyond simply repeating the steps it had previously answered with and I could load the extension to my browser using the Chat's instructions.

However, with the next group of answers, things started getting shaky.

I asked how to add JavaScript to the extension, and followed up with a question on the difference between background and content scripts in the manifest file example.

The answer looked pretty good and explained it convincingly, but the examples it provided didn't work. The console.log() I added would only log inside my extension developer console, not in the browsers. After triple checking my code, and a lot of back and forth with the Chat, I begrudgingly looked up the official Chrome Extension documentation where it was obvious that it had combined the v2 and v3 manifests in the code it generated.

Once I fixed this it was simple to find an open source library for the table sorting and load it into the extension.

In a real life situation this little mistake would be quickly caught. It is interesting, however, because it illustrates an issue when using Chat GPT or other LLM. Due to the novelty of the technology and the speed and plausibility of its answers, we assume it is accurate. It will even argue its case in some instances, with examples and references that may or may not be factual.

One improvement for this is the source of the data. If the model was reading directly from the documentation it would be a lot more accurate. I could have also specified my questions to be for one version of the manifest. But I was basing my questions on what it told me, which was a feedback loop to nowhere.

A LLM doesn't really 'think'. It predicts plausible looking text from the corpus of data it has been fed, but that doesn't necessarily mean the sources are correct or that the output from combinations of the sources are correct, even if they seem to make sense. From discussions I've had with people, perhaps the problem is more with us humans anthropomorphising the machine than the machine itself. This point requires its own post which I may explore another time.

I'm happy with how this experiment went. Obviously, it’s contrived but it has given me an insight to the potential and limitations of the current tech, and a basis for using it in the future. In reality, developers will use Chat GPT and similar, in tandem with traditional google-ing and bothering senior colleagues for answers to their problems. I have already started using Chat GPT in my work as a developer and it (or perhaps my use of it) is only going to improve.