Unraveling the WGA’s MBA with ChatGPT: Expert Analysis or Algorithmic Bias Towards Legalese?

On Oct 9, the Writers Guild of America (WGA) overwhelmingly voted in favour of ratifying the 2023 Minimum Basic Agreement (MBA), which will apply until May 1, 2026. While most news outlets have reported on this as a major win for the writers guild, I grew curious on whether there were any shortcomings in the agreement since the the entertainment industry’s capitulation seemed to come quite abruptly compared to their former hardline opposition to any concessions. Since I am not a legal expert on contract law, I turned to ChatGPT instead to see if it could provide a fresh perspective.

Given that the WGA provided both the complete 2023 MBA as well as a summary, I chose to try inputting both into ChatGPT, focusing particularly on the Artificial Intelligence clauses. However, the results made me ponder whether ChatGPT’s analysis might be influenced by bias or randomness.

ChatGPT’s analysis of the summary and the complete 2023 MBA:

Upon analyzing the summary, ChatGPT initially gave off a skeptical tone, stating a “notable absence of concrete safeguards specifically addressing the use of writers’ past material for training AI models”. It specifically criticized that “While the WGA reserving the right to assert prohibitions is a step towards safeguarding writers’ past materials, it lacks specificity and does not proactively establish explicit protections or limitations on the use of such material for AI training.”.

However, ChatGPT’s tone changed after I provided it with the full detailed legal document. It stated that “these clauses safeguard writers from being replaced by GAI for creating original content. They ensure that even if AI-generated content is used as a foundation, writers’ rights, credits, and compensations are preserved.”, as well as stating that “Writers or the WGA can challenge any exploitation of their literary content for AI purposes.”.

My thoughts on this small experiment.

The intriguing part of this test was ChatGPT’s differing opinion on whether the MBA adequately protected writers from AI developments based on the material it was given. Both the original summary and the legal document explicitly stated that “The WGA reserves the right to assert that exploitation of writers’ material to train AI is prohibited by MBA or other law”, and ChatGPT acknowledged this in its analysis. However, it seemed to believe this was ‘insufficient protection’ when analyzing the summary, while conversely saying it was “comprehensive” and a good ‘safeguard’ for writers when analyzing the full legal document.

This brings up the question why ChatGPT believed this clause was sufficient protection when analyzing the legal document, while stating it was insufficient when given the summary. Perhaps it was the more detailed and legalese nature of the full legal document that made ChatGPT perceive it as more ‘official’ or ‘stringent’, which led to its interpretation that such legalese served as sufficient protection. Compared to that, the summary’s more simplified nature which ommitted more stringent and binding wording may have been viewed as insufficient. It could also be that this was a mere quirk of the randomness of generating ChatGPT responses, and that if I re-generated the answer enough times, I’d find both approving and critical ChatGPT responses for the same summary.

If the reason is the former (that it was due to the legalese and stringent wording), this could imply a bias by ChatGPT in favour of ‘legal/technical wording’. If the reason is the latter, then that would greatly reduce ChatGPT’s reliability when asking it to analyze and judge legal agreements since its judgment could be positive or negative based purely on chance.

Thank you for reading, and I hope you found it informative and interesting.

I would love to hear any thoughts from you on my experiment with ChatGPT. If you are interested in discussing more about how ChatGPT can be used in such political contexts, or have any questions, contact me at hello@simplawfy.ca

Disclaimer: This story is only intended to be used for educational or recreational purposes. Responses by ChatGPT, if mentioned in this story, should NOT be relied upon as factual.