The Benefits and Challenges to Writing Guides using ChatGPT

        Looking back on how I created the tips for writing ChatGPT prompts in August using ChatGPT(published on my Medium account), I noticed both several noteworthy benefits in using ChatGPT as well as a few significant downsides. The greatest benefit was the ease of creating content, yet turning the mass generated content into something readable that matched my intent precisely was the real struggle. Here are some highlights and lessons I learned while making those tips.

        To recap, in August I published a series of ‘useful tips’ for writing prompts in ChatGPT to get the precise results a user sought. I created those tips by asking ChatGPT itself extensively on ways users could get more accurate results from ChatGPT, and then kept on asking for more unique or different ways (to improve the prompts) until ChatGPT’s responses became slightly repetitive. I further asked ChatGPT to organize those tips based on broad categories, though I did not specify the categories and allowed ChatGPT to create them itself. Once adequate (to my evaluation) categories were formed, I further asked ChatGPT to describe those tips in more detail, with clear examples, in the style of a blog or guide, which is what I later published. 

ChatGPT pumps out content, but how to pick and use the best one?

        One of the major benefits I noticed was that ChatGPT was far more efficient in generating ‘content’. For instance, for the tip “repeating prompts with slight variations”, ChatGPT instantly generated (upon my prompt requesting it) the example of “Tell me a dad joke → Tell me a dad joke about animals”. Moreover, it could continue generating new, unique examples until I received a satisfactory one. Similarly, I could have ChatGPT create infinite variations of the same blog post until I found a specific tone and writing style that I desired.

        However, although this saved me time I would have otherwise spent personally creating examples, going through the mass-generated examples until I found a satisfactory response also took time. It was all too easy to get stuck in the trap of repeating prompts with slight variation to get the ‘perfect’ response from ChatGPT rather than settling for an adequate response in a fraction of the time.

        A significant downside to using this method was that I struggled to get ChatGPT to perfectly ‘copy-and-paste’ the good examples into longer ChatGPT-generated posts. For instance, if ChatGPT managed to produce a perfect example, and I wanted ChatGPT to give me that exact same example as part of its response in a later prompt, ChatGPT often gave me a slight variation rather than the exact earlier response.

If You’re Struggling, Ask ChatGPT Itself!

        One lesson I did not learn until rather late was that it is often much faster to ask ChatGPT on why something isn’t working out (relating to prompts) rather than trying to search up the answer myself or trying to create a workaround. For instance, the issue I had regarding ‘copy and paste’ was actually due to the token limitations on ChatGPT. I discovered this by asking ChatGPT (after numerous frustrated attempts) why it was unable to copy and paste responses exactly.

Me: You told me earlier that you cannot copy and paste things, because you do not store that information, not even for a short duration. But you seem to be able to recall perfectly my prompt to you in the past. Isn’t this contradicting what you said?

ChatGPT: When you send a single request to me, that request usually includes the entire “context” of our conversation up to that point, up to a certain length limit… If that context includes prior parts of our conversation, I can reference them.

Me: what is your length constraint here?

ChatGPT: … For models like ChatGPT based on OpenAI’s GPT-3 and GPT-4, the length constraint is typically measured in “tokens.” A token can be as short as one character or as long as one word (e.g., “a” or “apple”). For GPT-3, the token limit is 2048 tokens.

Me: How exactly does token limit translate into words, sentences, or prompts?

ChatGPT: …Estimate in Words: On average…If we assume an average of 4–5 tokens per sentence and an average sentence length of 20–25 words, then 2048 tokens might represent approximately 400–500 sentences

        While I did know about token limitations on ChatGPT, I did not understand how this applied to my practical usage until ChatGPT itself had explained to me. Asking ChatGPT directly helped me realize part of the problem may be due to the length of my conversations and prompts/responses exceeding the ‘token limit’.

Confirm and Test ChatGPT’s responses to Ensure their Accuracy and Completeness.

        Although asking ChatGPT can be the fastest solution to any prompt related problems you encounter, you must confirm and test whether ChatGPT’s explanation is accuracy and completeness. In my example, although the token limit explanation that ChatGPT gave is certainly a part of the reason, it is not the whole truth.

        I confirmed that the token limit is likely true by asking it to tell me the first prompt I had given in a long conversation, and the response it gave was a prompt somewhere in the middle of the conversation. This indicates that the ‘first prompt’ was likely too far back in the conversation to be included in the ‘context’. However, when I asked it to recall the prompt and response given immediately prior, ChatGPT gave me that prompt and response, but in a ‘regenerated’ manner, which indicated ChatGPT was in fact able to view the prompt at least and provide a very similar response. This result does not seem to be explained by the token-limitation.

        Asking ChatGPT on this discrepancy led to a more detailed answer: it revealed that ChatGPT’s response generation model and factors like prompt complexity may lead to slight inaccuracies or variance in copy and paste accuracy. This still does not entirely explain to me why ChatGPT did not copy and paste even simple prompts & responses perfectly, and the process of getting sufficiently detailed explanations from ChatGPT required me to both test and question ChatGPT beyond just accepting its initial answers. Nonetheless, if I had asked ChatGPT earlier on why copy and paste did not work properly, I would have saved myself a lot of time struggling to cobble together multiple ‘perfect’ responses into one ‘perfect’ lengthy result.

Concluding Thoughts

        I found ChatGPT to be an incredibly powerful tool for producing ‘raw content’. That is, in terms of sheer quantitative output, ChatGPT saved me a lot of time thinking ‘creatively’ and allowed me to focus more on ‘editing’. However, if I already had a specific result or framework in mind, getting ChatGPT to that result took some effort. Trying to edit and nudge multiple responses into a ‘final product’ style result, such as trying to use ChatGPT to produce a book chapter or lengthy guide, was very challenging — especially due to the unreliability of copy and paste.

        Perhaps the two most useful tips I learned in this process were; 1) to not get stuck trying to make something work, and instead ask ChatGPT itself on why I wasn’t getting my desired result, and 2) to always check and confirm myself on whether ChatGPT’s responses are accurate, while also questioning ChatGPT extensively if there were discrepancies to ‘draw out’ the most detailed response.