ChatGPT seems to be an interface moment for language-generation AI. Everyone is talking about it. Sure, there are a few companies worth over $1B combined that do AI for marketing copy, but ChatGPT seems to be captivating even the most non-technical people.
I experimented with ChatGPT to write sports articles. I gave it the following prompt:
On December 13, Argentina played against Croatia in a World Cup match at Lusail Stadium in Doha. Julian Alvarez scored 2 goals, the first in the 39th minute and the second in the 69th minute. Messi scored one goal at the 34th minute. Argentina won the match 3-0. Write a 300-word brief about the match.
It outputted the following brief:
On December 13, Argentina faced off against Croatia in a World Cup match at Lusail Stadium in Doha. Argentina came into the game with a strong lineup, led by superstar Lionel Messi. The team was looking to bounce back from a tough loss in their previous match, and they started the game with a high level of intensity.
In the 34th minute, Messi scored the first goal of the match, putting Argentina up 1-0. The goal was a thing of beauty, as Messi used his incredible dribbling skills to dance past several Croatian defenders before firing a powerful shot into the back of the net.
Just five minutes later, Julian Alvarez scored the second goal for Argentina, putting the team up 2-0. Alvarez made a great run down the right wing, beating his defender with a series of sharp cuts and quick moves. He then blasted the ball past the Croatian goalkeeper, who had no chance to save it.
The second half saw Croatia try to mount a comeback, but they were unable to break through Argentina’s defense. In the 69th minute, Alvarez scored his second goal of the match, putting Argentina up 3-0. The Croatian players fought hard until the final whistle, but they were unable to find the back of the net.
In the end, Argentina emerged victorious with a 3-0 win over Croatia. The win was a much-needed boost for the Argentine team, and they will now look to carry this momentum into their next match.
This brief is amazing! The only problem is that it’s wrong. Messi didn’t score by dribbling past the defenders, he scored on a penalty kick. Also, Argentina won their last game.
I updated the prompt I gave ChatGPT to reflect that Messi scored on a penalty kick. It outputted the following brief:
On a chilly December evening in Doha, Argentina faced off against Croatia in a World Cup match at Lusail Stadium. The Argentine squad, led by captain Lionel Messi, was determined to secure a victory and boost their chances of advancing to the next round.
The match started off evenly, with both teams displaying strong offensive and defensive strategies. However, in the 34th minute, Messi managed to capitalize on a penalty kick, putting Argentina in the lead. Five minutes later, Julian Alvarez added to the score with a powerful shot from outside the box.
The Croatian team fought hard to even the score, but Argentina’s defense held strong. In the second half, Alvarez struck again, scoring his second goal of the match in the 69th minute. The Croatian side continued to press, but were unable to break through Argentina’s defense.
As the final whistle blew, Argentina emerged victorious with a 3-0 win. The Argentine squad celebrated their victory, with Alvarez receiving the Man of the Match award for his impressive performance.
The win was a crucial one for Argentina, putting them in a strong position to advance to the next round of the World Cup. The team will now turn their attention to their next opponent, with hopes of continuing their winning streak.
Overall, the match was a thrilling display of soccer talent, with Argentina showcasing their strong offensive and defensive skills. The victory was a well-deserved one, and the Argentine squad will look to carry this momentum into their future matches.
Again, the brief is spectacular, but it’s factually incorrect. It wasn’t a chilly night in Doha, and Alvarez didn’t win man of the match. Messi did. Along with his goal, he had a brilliant assist to Alvarez which wasn’t provided in the prompt.
There’s a limitation that explicitly affects anyone using this technology without supervision: ChatGPT knows how to write, but it doesn’t know what it’s writing.
AIs, like humans, are trained to do a task by practicing. Practice requires example data, and that example data comes from past experience. To learn to write, ChatGPT read all of Wikipedia, among other things. It derived style, nuance, grammar, and colloquialisms. Then, it modeled how it writes off what it read.
The AI clearly read plenty of articles commenting on Messi’s amazing dribbling ability. It also surmised that the top scorer in a match is probably the player of the match.
A human writing an article for accuracy could guess that Messi scored by dribbling past the defenders, but they’d look up a highlight to determine. They could also think that Argentina lost their last match, but they’d look it up. Of course, in the World Cup knockout rounds, you lose, and you’re done.
The AI doesn’t check for that information. The AI doesn’t know any of those errors from anything it got correct, such as the Croatian team playing hard until the final whistle.
The AI knows how to write. It knows how to be interesting in its style, using a phrase like “dance past several Croatian defenders” and “a thing of beauty,” but it doesn’t know if it truly was a thing of beauty. It doesn’t know that Alvarez actually ran down the middle of the field. It made that up.
Sometime, probably sooner rather than later, AI will do that sort of fact-checking. Then, it’ll become a search engine, reporter, blog writer, screen writer, and more. For now, it’s best as a first draft. You never know when it’ll be confidently wrong.
Thanks to Miranda Wagner, TJ Ryan, Bill Bloom, and Shannon Waller for reading drafts of this essay.
Evan Ryan 