Written by 
Unlimited Data |
BY Jim Kulich |
5 MIN READ
Artificial intelligence (AI)’s role in the workplace continues to evolve, and as it does, so too must our understanding of the balance between what humans and machines have to offer.
Understanding the Role of AI Assistants in Modern Workflows
I recently overheard a fascinating conversation on my daily commuter train ride. A group of young professionals, who I surmised had backgrounds in business and IT, were talking about ChatGPT. One member of the group, who appeared to be well-versed on the topic, discussed how he relies heavily on AI for producing code and how his colleagues do the same, supplanting much of their manual coding efforts.
One of his comments was especially provocative; that he trusts the feedback on his code that he receives from AI more than any feedback he receives from his human colleagues!
As striking as that thought is, it doesn’t generalize well. As Ryan Elmore, Senior Vice President for Data and AI in North America at Equal Experts and my fellow instructor in Elmhurst’s online master’s of data science and analytics program, achieving impactful enterprise-level results with AI requires alignment with business goals, quality data and modern engineering.
The Potential and Pitfalls of AI Assistants
I recently taught an introductory course on generative AI for students in my graduate program as well as interested MBA students. Their goal was to produce a basic AI assistant that would address a meaningful business problem. They learned a few things.
First was the initial excitement about what even a simple AI assistant can do. Even with basic prompts, amazingly good results were attained in some cases.
Next came the challenge of generating consistently useful outcomes. One student focused on improving the descriptions of products arriving in the receiving department. They created a reasonable persona for the AI assistant and prompted it to create appropriate product descriptions. The tool responded, but only in a superficial way. Like most students in the class, this individual quickly came to understand the need to narrow the scope from a business perspective, sharpen the prompts and provide the tool with solid, relevant data.
As class members experimented with ways to add useful knowledge bases to their AI assistants, they discovered another interesting twist. The first sources added to a knowledge base improved the tool’s performance, sometimes considerably. But, if too many resources were included without good attention to their organization, performance could degrade. The AI assistant began to get confused when it encountered conflicting information. More was not necessarily better.
Enhancing AI Assistants with RAG Technology
This is a familiar phenomenon in data science, namely, the idea of overfitting. At the start, all information added to a predictive model helps, as the model needs something to work with. After a while, though, a predictive model starts to mistake the inevitable random patterns in large collections of data for signals that are useful. This is an unavoidable reality in this work that must be managed.
Researchers and practitioners are actively investigating ways to address these issues. Retrieval augmented generation (RAG) is now a well-established approach for adding knowledge bases to generative AI assistants. Advanced forms of RAG combined with technical advances in the AI tools themselves offer promising improvements to these systems that aim to address the kinds of problems my students encountered.
The Limits of AI Assistants: Accepting Performance Tradeoffs
Yet, this will never be perfect. As is the case, in more traditional machine-learning as well as cutting edge AI, performance tradeoffs will always exist. 90% accuracy may be phenomenal in one business setting and entirely unacceptable in another. These kinds of judgements make or break AI projects.
An approach one of my students followed provides an illustration for how you might proceed, especially at the early stages of this work. Their initial project goal was to create an AI assistant, trained on internal documents, for employee training purposes. As expected, this was too much to ask at first. The next step was to narrow the scope to required safety training – better but still too much.
However, there was one pain point: an ongoing need for fresh test questions regarding safety protocols, grounded in the company’s internal documentation, that could be used on the required periodic safety exams for all employees. Company leaders were running out of ideas for good questions. This looks to be a perfect use case for a simple AI assistant.
Finding the Right Balance Using AI Assistants
So, unlike the group on my train ride who seemed eager to simply turn over the keys to AI, perhaps it is better to realize that AI, in its current state, can readily provide a tailwind for many things we do every day. As my students learned and as I continue to learn, finding ways to bring the advantages AI offers into the flow of work we control is almost always a winning proposition.
Understanding the ins and outs of ever-changing AI is one of many ways a master of data science and analytics degree from Elmhurst University can help you level-up your career. To explore the program further, visit elmhurst.edu/DataScience.