Fun with LLMs, Part 1

Modern AI has demonstrated such amazing capabilities, leaving many people fearful that humans will be pushed out of their jobs and replaced with ChatGPT and friends.  Indeed, big tech has been laying off a large percentage of their workforce, claiming that the use of AI will be able to replace, and maybe even surpass, the results generated by the intellects of their former employees.  Is this the end?  Should we acknowledge our inferiority to the machines?

No, I don't think so.  AI still has so many failure modes (and will have them for the foreseeable future) that it is supremely unwise for a company to replace a large chunk of its employees with calls to ChatGPT.  I will catalog some of the failures that I have encountered recently.  After reading them, I hope you will agree with me that humanity isn't done just yet.

Today, let's consider a problem that is very straightforward but involves data-fetching and some tedious analysis.  Each state in the U.S. elects to Congress two senators and a number of House representatives that depends on the state's population.  The most populous state, California, has 52 representatives.  Several states with low populations have one representative each.  Suppose we wish to know how many states have four representatives or fewer.  This seems like a good task for an LLM, right?  Let's ask Google.

Sounds good.  But let's ask again to make sure.

Huh?  Uh, let's try a few more times...

Hmmmmmm.  Thanks, Google, that was very helpful.  Google performed the addition correct in each case, but it is getting bad information from somewhere.  I was surprised that Google's AI had so much trouble with this, since it is pretty easy to find a list of the states and their corresponding numbers of representatives.

For the record, the correct answer (if I did my math right) is 21.  If you calculate it yourself, let me know if I actually did it right.  😁  Surprisingly, Google didn't even come up with this answer by accident.

So take pride in the fact that your brain isn't obsolete, for the time being.

preventing viewers from seeing stale static pages

Until recently, the front page of amecy.com was generated by a CGI script.  There was a program, in a file named index.cgi, that ran every time you visited www.amecy.com.  The output of this program was sent to your browser, and this was the page that you saw.  While this setup worked fine, it was a little slow, so I started thinking about ways to speed it up.

Soon, I realized that the front page didn't need any of the dynamic behavior that a CGI script allows.  I decided to just run the script and save the output in a file named index.html.  In this way, I was able to speed up the site without doing a lot of work.  To make changes in the future, I can edit the CGI script, test it in the browser to make sure it works, re-run it on the server, and save the output to index.html.

Now, if you visit the site, the front page loads lightning-fast.  If you reload the page by clicking the logo in the top-left, the reload will be even faster: the server returns 304 Not Modified, and your browser can use the version of the page that is sitting in your browser's cache.

This is all straightforward, right?  Well, not exactly.  Some browsers -- Firefox and Chrome, at least -- will serve a static page directly from the browser cache, without checking with the server first.  If you update the page in between a viewer's visits to the site, the viewer may end up seeing the old version of the page, not the new one.  To circumvent this behavior, the server must explicitly tell the browser to check with the server before showing the cached version of the page.  This can be accomplished by sending the following header with the response:

        Cache-Control: must-revalidate, max-age=0

must-revalidate tells the browser to check before displaying the page.  However, the "check" here is not exactly checking the server; the browsers that I tested will often perform this "check" by checking against the browser cache.  To force the browser to check against the server, I had to set an expiry on the cached page.  That is what max-age=0 is doing there.

There is also a middle ground between "always check with the server" and "never check with the server."  If you are fine with letting someone see a page that is slightly out of date -- let's say an hour -- you can send the header

        Cache-Control: must-revalidate, max-age=3600

This way, someone viewing your site will only see a stale page for an hour.  The upside is that this will be faster, both for the browser (it can often show the page without having to use the network) and for your server (you won't have browsers connecting to your site as often).  The benefit is especially significant if the page in question is a "hub" page that you expect people to visit often while navigating your site.

If you have been on the Internet for a while, you have probably encountered this problem once or twice, when visiting a static site repeatedly.  You will see the latest version of the site the first time you visit, but on later visits, you may have to refresh to see updates. Many web pages tell you to refresh them; this is another manifestation of the problem.  The problem can be avoided, as I just explained, but it is not obvious to everyone that there is a problem (unless you go looking for it!).

That's about it.  If everything in this post was already obvious to you, then good... I am happy for you!  But when I started seeing unexpected behavior around static pages, I had to piece the solution (and the explanation) together from a few different places around the web.  I have written this post in the hopes that it will help someone who encountered the same problem as I did.

here we go

The year is 2019, and I have finally decided to start a blog!  Better late than never, as they say.

I intend to write about the projects that are listed on amecy.com (Checkmate Champ and the new amecy games), as well as related topics like chess and programming in general.  Along the way, some novelties or personal stories may slip into the mix.  It will be fascinating.

Stay tuned.