Do AI Generators Know What They Do Not Know?

Marketing Viewpoint by Ruth Winett

Experimenting with Two AI Generators

I have been experimenting with AI content generators, but I will continue to write my own newsletters. The following observations are based on using Constant Contact’s AI and OpenAI’s ChatGPT 4’s content generators. Many of the observations will apply to other content generators as well. Constant Contact’s AI is designed to help subscribers who post newsletters, product promotions, and announcements. ChatGPT is older and is designed for a more general audience seeking answers to questions. It generally provides more in-depth responses than Constant Contact’s AI.

I posed a dozen different questions with varying results. Here is what I learned about the content generated and the format or style of the output:

How Is the Content? Good, but Not Yet Great

Constant Contact’s AI generator offers several choices: I selected newsletter as the end-product, text as the format. Then I selected an informative tone. Importantly, you can request additional versions (options) if you do not like the first two versions generated. The quality of the content generated is a function of the quality of the questions posed. Specific questions produce better results: “Which states do not celebrate Juneteenth?” “[Hawaii, North Dakota, South Dakota, and Montana.] “Requesting examples, or tips, or statistics often yields better results.

“Examples of government websites with free data” yielded data from three obvious agencies: Census Bureau data, climate data from NOAA, and crime statistics from the FBI, without including URLs or other less well-known agencies.

“How much food is wasted annually in the US?” Some statistics were provided without much context.

“What are some ways in which the US could reduce food waste?” One option enumerated several steps that people could take, but most were self-evident, such as creating a shopping list when heading to the grocery store.

“What are other countries doing to reduce food waste?” Requesting still more options and requesting sources from ChatGPT, yielded information on the initiatives of five countries, plus two or three government sources for each of the new options.

Often, responses included examples, but no explanations or definitions. A question about new market research approaches yielded two undefined techniques: social listening (tracking social media comments) and eye-tracking (tracking what people look at when viewing online content). Follow up questions are often necessary.

Constant Contact AI and Chat GPT 4 do not generally include sources used or the year when the source material was published or the data collected. This makes it harder to distinguish fact from fiction. It also makes it difficult to ascertain if the information is out-of-date. You can add “provide sources and dates” to your query.

Do AI Generators Know What They Do Not Know? Not Necessarily

AI generators cannot accommodate all queries. When AI software is out of its league it may provide misinformation. Here are two surprising results from so-called book reviews from Constant Contact’s AI:

Lessons in Chemistry by Bonnie Garmus is a novel about the struggles of a gifted female chemist in a male dominated profession. Constant Contact AI describes the book as a chemistry textbook with sections on organic and inorganic chemistry!

Horse by Geraldine Brooks is about a racehorse and his master in the Civil War era, not the Crimean War where Constant Contact’s AI places it.

Open AI’s ChatGPT 4 is fast and robust, but states that it cannot address queries concerning matters that occurred after September, 2021, when the AI generator stopped collecting data. This is a major limitation. ChatGPT 5 is due within a few months and should have more recent data, but will it also cut off data collection?

How Is the Format? The Style?

Constant Contact AI, and ChatGPT 4 AI content generators provide well-written responses in seconds in a format and style that an English teacher would approve. The responses include an introduction with a hook and a beginning, middle, and a few examples followed by a summary. Sentences are grammatically correct, and the vocabulary while average is used correctly.

The Bottom Line

Often AI generators provide responses that are very general, for instance naming market research approaches, such as sensitivity analysis, without explaining them or evaluating them. However, by asking more specific questions you can generate more useful results. Also, Constant Contact’s AI invites you to request more options, which sometimes provide more in-depth information. Note that AI generated text is so well-written it could convince you that the content is accurate and meaningful. Thus, it is important to evaluate responses critically. Like any research conducted online or with traditional sources, you can better evaluate the results if you are familiar with the topic.

AI is evolving. Constant Contact’s AI is a beta version with a disclaimer: “Please check the generated content for errors, accuracy, and suitability.” In addition, ChatGPT 5 will be released in a few months, with more up-to-date content. But, it may not add more content in real time.

Resources:

Thanks to Jacob Richman who helped me install and use ChatGPT 4.

To test ChatGPT click on https://chat.openai.com/auth/login

To test Constant Contact’s AI generator https://app.constantcontact.com/pages/dashboard/home/#/ (for subscribers only)

Actionable Business Insights

For Email Marketing you can trust

articles index | home | back