Episode 67: Trisha Causley and Maria Hofstetter - Content design and AI

Kristina Halvorson

So when you are doing that, is that what you mean by training the LLM?

Maria Hofstetter:
Sorry.

Trisha Causley:

What was that?

Maria Hofstetter:

Don't know.

Kristina Halvorson:

It's the LLM going, "No, don't talk about that."

Maria Hofstetter:

I think so, yeah.

Kristina Halvorson:

Stop it.

Trisha Causley:

So funny.

Kristina Halvorson

Hello, my friends and neighbors, and welcome back to the Content Strategy Podcast. Once again, I'm your host Kristina Halvorson. I know that with the introduction I introduced myself twice, but that's just in case any of you are not listening the first time around because you're distracted by the outstanding theme music that was written by Sean Tubridy. Okay. I have been looking forward to this conversation for a very long time and I'm so grateful that these two amazing humans have made time to have it, and I'm grateful that you were able to tune in and listen to it. So I'm going to tell you who they are. These are two folks who work at Shopify. Trisha Causley is a senior staff content designer at Shopify in Toronto, Canada, where she works on AI-powered product features. A one-time AI skeptic, she's learned to appreciate its potential for removing toil from content crafting and editing. I haven't said toil out loud maybe in my whole life. Love it.

Maria Hofstetter is a staff content designer at Shopify crafting AI features. She's honed her expertise in UX content strategy and design solving complex user problems for software companies, including OpenText, Manulife Financial and McAfee. Now, delving into AI, Maria sees large language models as a fresh avenue for harnessing the power of words and digital experiences. Welcome to the show, Trisha and Maria.

Maria Hofstetter:

Thank you, Kristina. So happy to be here.

Kristina Halvorson:

I'm so happy you're here. Before we dive into the wonderful world of AI, I would like to ask each of you, as I always do at the top of my show, to share with me your journey through content strategy and content design and how you have landed in your current roles. Let's start with Trisha.

Trisha Causley:

Yeah, so I actually started in linguistics, that's my educational background, and originally I worked on a lot of dictionaries for indigenous languages. So that's where I started my word career, but eventually I to get a job that actually paid money, so I started working at IBM as an information developer and an information architect, and I was at IBM for a long time, like 16 years I think, but I moved around a lot because there's a lot of different industries. IBM has its hands in lots of different places, and probably the most fascinating place that I got to work was in healthcare because we'd bought a population health software company, and that was just fascinating. I got to work with nurses and developing things, like training materials for them and job aids and that sort of thing. It really gave me a perspective on what an important job nursing is and how committed they are and how if I'm going to complain about not finding the right word, they have a lot bigger problems to complain about than I have.

And then I moved to Shopify. So I started building... I started on our app store. The first big project I did was with a taxonomy for all of our apps. So we have thousands of apps in our app store, and so I did a lot of taxonomy work and a bunch of other projects. And then I got tapped about a year and a half ago to work on the first AI projects that we were tackling at Shopify, and it was brand new, like GPT was just out, GPT-3 was just out, and so nobody really knew what they were doing, and so it was really trial by fire and figuring out what are these things capable of, what do we actually need to do? How do you even do this thing called prompting and what are people going to expect from us? And it was really great. It was very... I think probably I learned more in the last year than I've learned in my entire career and I get to work with Maria, which is really great. So that's where I'm at right now.

Kristina Halvorson:

I'm going to suggest that what you just described is a little bit more than a word career, but thank you very much. Incredible. I do want to say before we move to Maria, that I think it's really important for listeners to understand that when ChatGPT-3 came out, everybody was making it up. I think that as soon as that happened everybody was like, "Wait, I don't know what's going on. I don't know how to do this. What should I do?" And then very quickly folks realized, "Oh, wait. Nobody knows how to do this." So really looking forward to diving into how that period of experimentation went. All right, Maria, you're up.

Maria Hofstetter:

Awesome. Well, I was a lit major to begin with, so all I knew way back then was that I wanted to work with words. In literature, I was super drawn to how words can build entire worlds, and I think in our digital realm the same is true, but we think about it through words building digital experiences. So it's this interest in creating something out of language that's been the thread through my career and the exact path that I took is not unlike many other content designers. I actually started out doing freelance social media marketing for small businesses. This was at the time when Instagram was just kicking off and small business owners didn't have time to run their own social on top of running their whole business, so I did it for them.

This took me down the marketing path initially where I wore a few different marketing hats and copywriting hats, but this is also when I began to work on product content and user flows before actually knowing this was an emerging field. So totally what we now call UX content design, but it was just being formalized as its own thing back then. And when I learned about UX and content strategy and realized this was what I was already doing, I made it official and moved into UX, and have worked at a number of great companies throughout that journey. Most recently at Shopify, and at Shopify have been on a few different teams from our largest enterprise products to our mobile team, working... So two ends of the spectrum there, big screens, little screens, and then about just over a year ago got tapped on the shoulder to join Trisha working on AI features.

Kristina Halvorson:

Fantastic. I do want to let you know that the number of career paths that folks have described since I started this podcast are way weirder than what you just described. It's not that unusual.

Maria Hofstetter:

I'm in good company.

Kristina Halvorson:

That's right. I just love that you both were coming at this with such a love of words from the very beginning. So the first thing that I would really like to do, because I never want to assume that my listeners are savvy with the language and with the acronyms that we use, I'd like to have you describe some of the basic acronyms that are thrown around daily in the world of AI. So the first thing that I'd like to have you describe, what is an LLM?

Trisha Causley:

Okay, an LLM is a large language model. That's what LLM stands for. It's a type of artificial intelligence. When people talk about AI today, they often conflate the terms AI and LLM, but an LLM is just a subtype of AI. So an LLM is trained specifically to understand and generate human language. So other AIs might know about vision, so something like Midjourney. Those are models that understand visual data. An LLM is purely language-based. So it knows about language. And this is part of one of the things you'll find with LLMs, it's not great at a lot of other tasks. It's not great at math because that's not the way it's trained, it's been trained with language. So that's in a nutshell what it is.

Kristina Halvorson:

Terrific. And I do want to come back around how one trains an LLM because I am still figuring that out. Alright, so moving on to a completely different word. Can you please describe what a prompt is?

Maria Hofstetter:

Yeah, so a prompt is basically a set of instructions given to guide the creation of something. So in the context of text generation, which we'll likely be talking the most about here, a prompt is a specific set of directions or guidelines that we give the large language model to generate text that meets certain criteria.

Kristina Halvorson:

So can you give an example of a prompt?

Maria Hofstetter:

Yeah. Generate five marketing taglines for a website selling soup, would be one example. It's not a great example, but it's a set of instructions.

Kristina Halvorson:

Listen, I asked you for an example, I'll take it. It's great. So if anybody has played around with ChatGPT or any other AI tool that is like, "Enter what you want and it'll spit back the words," that's a prompt.

‍Maria Hofstetter:

Exactly.

Kristina Halvorson:

Terrific. So let's move on then to, you talked about training large language models. Can you describe what that means?

Trisha Causley:

So most people are not actually doing training. That's sort of done for you when you start using an LLM. The LLM is pre-trained and it's trained on a ton of text data. So it'll be trained on books and websites and movie scripts and chat logs, and so through all that data, it's recognizing patterns and absorbing the structure of language. It learns about the grammar of the language in those texts, it learns about common phrases, it learns conversational behaviors. What does a question look like, and usually a question is followed by an answer. So it's learning that from all of the examples that it's sort of absorbed, it's ingested. So what you can do, there's those base things. So GPT has GPT-3 or 3.5 or whatever version has been trained on all of this data.

If you wanted to do your own training on a model, you can give it more information that's specific to your industry or to your company so that you would train it on basically your own data using... Often they'll use stuff like a question-answer pairs or input-output pairs to sort of show the model, "If somebody says this, then you say this." So you're training the LLM on what we expect you to do with certain content, if that makes sense. So most people are not doing training. When you talk about training, that's actually a pretty specialized task. You probably are taking a pre-trained LLM and then you are prompting, so you're basically saying, "You have all these skills and now my job is to give you instructions on how to use those skills."

Kristina Halvorson:

So really then what you're describing is that there are two ways to build and let's say influence what an LLM puts out or cranks out when it's given a prompt. One of which is you feed it data from wherever, from the entire internet or from books or from user manuals, et cetera, or customer service scripts, that's one, and it recognizes the patterns and the phrasing and the language and so on, and the second way is to specifically have human input that says... or otherwise it says, "When somebody says this, you say this." Am I getting that right?

Trisha Causley:

Right. And I mean you can do... Those instructions were giving a very basic... When we say instructions, that sounds really simple, but the instructions could be, "Here is a book. I want you to use this book as input and then answer any questions that I have." So for the context of the work you're doing, you can feed in... It's actually called context. You're feeding it in sort of resources that you want it to use in particular. So that's not the same as training because you can basically say, "I'm writing a research paper. I'm going to feed you a bunch of content that is going to help you write this paper." So it's not part of its base knowledge set, it's part of the context you're giving to it in that task.

Kristina Halvorson:

So would you say that the chatbots that have been around for a long time, is that what is happening with those chatbots?

Trisha Causley:

I think it depends on the context. So I think some companies would be building a chatbot and they would be potentially training an LLM so that it's part of the LLMs base knowledge. Everything to do with that company, let's say all of your return policies or your legal stuff, all of that might be built into the LLM or it might be that the chatbot is prompted to go and look stuff up, "Every time someone asks a question, go look this up in our library of information." That's probably the most common case because that's a little bit simpler to implement and easier to update. So you basically have a help center that you can send the chatbot to go and check for content.

Kristina Halvorson:

Yep, I suspect that that's what most of us have encountered over the last several years, and it always sucks because oftentimes what you get is, "No, none of these answer my questions," and you hit other and then it just kicks you back. Anyway. They all suck. So I'm hoping that actual AI might help them, but I'm suspecting that with many of those companies it may not because I'm jaded. Okay, let's come back around then to content design and how you're using AI in your day-to-day content design practice. Can you talk a little bit about the relationship between the two to start off?

Maria Hofstetter:

Yeah, so I use AI largely as a large language model daily now, and I was kind of like Trisha in the beginning, a total skeptic, but here we are and I've kind of dug into it with curiosity and it's now a daily part of my workflow. So I'm using it as a pairing partner, idea generator, using it to get a scrappy first draft down. So basically beating that blank page syndrome. So I'm using LLMs to get me started. I actually have different prompts set up and saved for different tasks in my daily workflow. So I totally have a robot content designer that I pair with. They're not perfect. I'm happy, I'm there to help guide them along and rev on ideas, but it's a helpful tool for ideation and iteration. So those scrappy starts combined with my own brain power and my own critical thinking can help me move ideas forward faster than if it's just me staring at a blank screen.

And I think the total opposite is true for me too. So I do use LLMs to edit. When I have super complex ideas in my head that I need to make sense of and then communicate, I'll just brain dump it into an LLM and work through revisions to get it to simplify and clearly communicate what I need to articulate. And I think it's important to call out, either way, this isn't a one and done for me. I very much use it as a tool and I play a huge role in using it as a tool to iteratively make those improvements. So kind of think about it like any other tool, like a hammer. I still have to do the hammering, it's not doing it for me yet. It might someday, but it's easier than just getting the nail in there without the hammer. So I'm kind of using LLMs that way.

Kristina Halvorson:

Can you provide a specific example? So let's say you have to write X. Can you give an example of what that would look like and how you might use an LLM in that process?

Maria Hofstetter:

Yeah. I guess recently it came up when I had to write... Well, I'll stick with content design actually. If I had to write an error message, I enjoy writing error message, they're not the favorite part of my job, but they can be fun to jam on. I will use an LLM to get me started now. So I'll kind of put in some loose prompt around, "You're a content..." I actually tell the robot, I use my content design prompt and position it as you are a content designer working at a tech company, you're creating user-friendly software for... I get specifics which we'll talk a bit about if we get into best practices for prompting, but I'll kind of set the context and then explain the message I'm working on. So write a short error message for a user that has run out of... I'm trying to think of an actual error. The feature's down, they can't use the feature, and then giving it some instructions around the tone I want it to be, what it actually needs to communicate, just feeding in some of those specifics.

It'll generate a first draft and the first draft is going to be... Depending on how good my prompt was. In that example, my prompt wasn't that great. So the first draft is going to be a first draft and then I'll kind of look at it as a content designer would in a content crit and think about what is this error message doing well, what is it not doing well? I mean, I'm not breaking it down, it's coming natural to me because I'm a content designer, but then I can kind of like... Maybe it's given me enough that I can take and then just rewrite. It gave me an idea to follow, and done. It doesn't need to be a big process, especially for an error message. But using that in different context. If I'm working on an entire webpage that I need to communicate, especially marketing content, just using it to get started, looking at the result, thinking critically, giving it more instructions, which then is another prompt. You're prompting it each time you're iterating through what it's generating to get to something that feels good to you.

Trisha Causley:

It's funny, I like to think of the LLM as a really new intern. So you've got an intern on your team and so what tasks would you give that intern and how much coaching are you going to have to do? So giving the LLM a really tedious task is great. So if you've got something... I'm sure people... You've got this information, you've got a list and it's in the wrong format because there's something like a piece of information that's at the end of every phrase, but you actually want it at the beginning. You can actually use the LLM to do that rote thing that you would've given an intern. If you said to an intern, "This is all written so it's..." I don't know. The date format is wrong or they've written it in third person, but we need it to be directed at the user. You can give an LLM that and it will just do it, which it's kind of nice because it does not mind tedium.

So you can make it do it and you could say, "That's not right, do it this way," and you could do that 10 times and it doesn't get angry because its job is there to just do those tasks for you. If you can think of it in terms of what kind of thing would you give your lowest skilled learning employee on your team and then give it instructions that you can't assume that it's going to know which date format or what the capitalization is that you want or what the length requirements are. So you tell it that and if it doesn't do it right, just tell it didn't do it right, or if you forgot to give the instructions, you say, "Well, sorry. I forgot to mention you actually need to do this in a single line."

And so I think with that mindset it feels less like this thing is here to take our jobs and more like it's here to actually assist you with those tasks that would take you a lot of time and all you need to do is give the right instructions and then it takes you no time, basically. Your time is just supervising and instructing as opposed to actually doing those tedious tasks.

Kristina Halvorson:

You know what cracks me right up, is I was just thinking when I do ask it to do something 10 times, I do want to say, "Oh, no. I'm sorry, that's not what I meant. I'm sorry to ask you to do this one more time, but..." And then 99% of the time I will thank the robot like, "Oh, thank you so much."

Trisha Causley:

I say, "Great. That's great."

Kristina Halvorson:

That's right, that's right. And then I'm just like, "Wait. It doesn't care," but it is, I do walk away sometimes with guilt like, "Oh, I really put it through the ringer this time." But I think that those are great examples. So when you are doing that, is that what you mean by training the LLM?

Trisha Causley:

You could train an LLM on your punctuation rules for your style guide. If you trained it, you gave the actual model enough examples, you could build that in so that's what it knows. It knows, "I always use Oxford comma," so that you would never have to tell it to use the Oxford comma in your instructions because it's part of its base knowledge. So that's where the training becomes sort of a permanent part of the LLM, and that takes, like I said, special skills and special tools and now you have to host the LLM and all that sort of stuff.

Us retail users are doing is giving instructions every time, and so you basically have to say every time, "I want you to use the Oxford comma," if it's not doing it by default, or, "I want you to use sentence style capitalization." Maria and I have a joke that we can always tell when people are using GPT to do their, whatever, work because it's like it loves title case. It uses title case everywhere, every chance it gets. It loves capitalizing, which we never want it to do, but you basically always have to say, "Use sentence style capitalization." You have to teach it that because it's not part of its base knowledge that it's been trained on.

Kristina Halvorson:

It cracks me up. We just hired a project manager, and I mean, we received hundreds of applications within 24 hours, but a real give for me was when the cover letters ended, "In summary."

Trisha Causley:

It's true. So this is what people don't realize is that the LLM is not... It's only drawing on the most obvious cliched examples. You have to untrain it from those things because using what it sees the most frequently is what you're going to see reflected back at you. So if you're looking for uniqueness or if you're looking to stand out, do not use your LLM to get all your content. You need to give instructions for it to be unique or you need to do more than the default because the default will be as generic and repetitive and predictable as possible. It's all based on prediction. So it's like I can predict the way this ends because this is what everyone does, and so what you're getting is the most common way to end a cover letter.

Kristina Halvorson:

So this really leads well into my next question, which is you kind of described how an organization or a content designer would use style guide rules specifically like, "Don't use this word, use this word," or, "Don't use title caps, use sentence caps," et cetera. How do you drag things like tone? Because I know that when I ask ChatGPT for something, it'll spit out something relatively formal and when I say, "No, no. Do it more casual," then it comes out like the worst game show host you've ever heard, and I'm just like, "No, not like that." So when an organization has very specifically defined voice and tone, how are you able to guide the LLM to get there?

Maria Hofstetter:

Yeah, I would say in general, right now anyways, LLMs aren't great at one or two word labels as instructions. So if I said, "Use a casual tone," it's going to give me something and it thinks it's a casual tone, but it might not be specifically what I'm looking for. So it's better to almost provide a list of specific attributes that equal casual tone to you. So I think this is also why content people have a unique superpower here because we can get in there, unpack and articulate what is giving a casual tone or a playful tone. So when I say playful, I might mean I want you to use puns or I might mean I want you to use emojis, but defining what good looks like for what you're aiming for with specific traits or attributes.

Trisha Causley:

Right. So use contractions or use metaphor. So you actually need to tell it what it is that makes you think it's casual or makes you think it's literary or makes you think it's, like Maria said, playful, because I think when we hear that we think, "Well, it's playful," or it's, "Oh, no. This is a formal tone," but the LLM is actually kind of guessing, and if you ask it to analyze a passage of text, the same passage of text, and you say, "What's the tone here?" If you ask it multiple times on the same piece of text, you'll get a different answer every time. So it's actually not understanding what that tone is anyway, so the label isn't what's important, it's those attributes that you actually need to nail.

Kristina Halvorson:

So for a very long time, Shopify's content design team and practice has really been held up in our field as the model of what a content design team can be, and we see content designers taking on different and new responsibilities specifically in leadership, and now here you are working hard in the practice of AI, which of course is continually evolving everywhere. Talk about what you like about working at Shopify.

Maria Hofstetter:

Yeah, I can kick it off, Trisha. I think it's a really cool time to be answering that question in our focus on AI because I think... I don't know if this is happening everywhere, but it's certainly happening at Shopify that content design is uniquely positioned to get involved at the offset of AI projects and features, and I think it really comes down to they are language models that we're working with at the end of the day. So having an understanding of language goes a long way in this work. We were talking a bit about prompting and iterating through examples, and I think an LLM will always give you an output. It's amazing still as we're in these early days of AI that some people don't actually read the output or read it critically, and I think content designers are the exact people who will look at an output, and, like we were talking about chatbots earlier, we know there's still crap in a lot of places.

So we actually look at it, we think critically about it, and then we nerd out on the language. We're like, "Okay, this isn't good because of these reasons. It could be great if we did this." So I really love that Shopify content design right now is getting involved in that work and just getting in there and figuring things out. Everything that I think Trisha mentioned at the beginning of the call, we've learned more in this past year than maybe our entire careers before then, and part of that is just getting in, getting curious. We don't know what we're doing, we still feel like we don't know what we're doing most days and the technology is changing at an incredible pace. No one can keep up. We're all learning fresh as things change all the time, but it's really cool and exciting that content design at Shopify is right in there in the mix getting involved and contributing and building features.

Trisha Causley:

I do think that there's a recognition too from engineering, from product, from our product designers that we are very necessary to the whole AI enterprise. So I feel like it is... I don't want to say... We've always been important, but I think all of a sudden it's coming to the forefront that we are necessary on a lot of these things, but also we hold a lot of these things. So Maria and I were on projects with engineers where they're building tools for evaluating output, but they can't do the... What is the evaluation? What are they looking for? So they can build the tools, but we're the experts on what to look for and it's clear and it's really obvious I think to all of those people. Our data scientists, same thing. It's like what should we be counting, Maria and Trisha? And I feel like that's starting to spread out throughout the company. Every time a new AI project gets kicked off, they look around the room and they say, "How are we going to do this?" And there's a really natural fit for content design.

Kristina Halvorson:

Do you feel like it's necessary for content designers applying for jobs in AI to have specific experience in AI?

Maria Hofstetter:

I think curiosity is the most important thing there. I think if you're curious and you've played around with the technology and you just spend time with it, I honestly think that's the most important thing right now because that's how you learn. I mean, of course there are courses and resources out there available. If that's interesting to you, by all means, but the curiosity and the understanding that things are shifting all the time and will continue to shift, that curiosity is going to set you up for success as things do shift and continue to shift.

Kristina Halvorson:

One thing that I have really seen over the last couple of years is that where content designers are getting more and more involved in AI projects are in organizations where it's okay to be wrong. I see that there are so many enterprise cultures in particular, well, I mean at any size company, where people are scared of being wrong because they're worried that it's going to affect their job, it's going to affect their profile within the organization, it's going to affect whether or not people want to work with them and bring them onto projects. And so that I think is one thing that I have always admired about Shopify's culture is that it's evident that they are open to people being wrong and experimenting and trying again until they stick the landing. So I did just want to call that out, that I think it's unique and I think it's valuable and I think it shows in your product.

Trisha Causley:

That's so interesting that you say that because I feel like the other cool thing I guess about the projects that we've got to work on is that being wrong is actually... If you can say this didn't work, that's actually a learning that's valuable to everyone. So we're actually encouraged to... It sounds funny to say, we're encouraged to fail. We're encouraged to try things and figure out what doesn't work as much as what does work, because what doesn't work is actually... If you didn't try that, you actually don't know that that's not a path that other teams should take. So I mean, maybe it's because we're early days, so we're still shaping what the strategies are that other teams should use, but it really is a learning to say, "Hey, do you know what didn't work? This." And everyone goes, "Oh, that's fascinating." So I think that's how you position it. You position not that you were wrong, but your experiment showed this is not a viable path.

Kristina Halvorson:

Well, that indicates that folks that are leading teams and leading the organization get that, and so that's an incredible important value I think that other organizations can really learn from because there are a lot of people who are working at places where it's like they do the same wrong thing over and over and over again and it's because people are afraid of being wrong. So good work, Shopify. I'm proud of you. So let me ask the question that is the elephant in the room for all content people everywhere.

What I see over and over is, "Oh, AI is going to require humans because we have human perspectives and we don't want the robots to take over, and it's more important than ever to bring humanity into the product," and I keep looking at that and thinking, "Oh, no. AI will bring humanity into the product. Pretty soon that's what's going to happen," and then our robot overlords of course will take over. But in terms of remaining relevant, do you feel like you've got a good line of sight in how content designers can think about it, move forward? Should they have confidence? Should they be looking to expand their skillset in other areas? What is your take on that?

Trisha Causley:

It's funny, because there's... Because Maria and I, obviously we use LLMs for stuff all the time, I think we have the clearest perspective on how necessary we're going to be for a long time because there are tasks you cannot get it to do. It is actually really tough to get an LLM to do good UX writing, for example. You can get it to give you a sentence, like some placeholder copy or that error message that Maria was talking about. Even if you give it really good instructions to get an error message out of the box and have that be an automated thing that it's just going to give error messages, we are really far away from that, and so there's I think a really vast gulf of, I don't know if it's taste or instinct or something that the LLM is really not good at, and so we, I think, are going to be more shapers and curators, but I don't see... I am not worried for my job.

I mean, I think there are probably jobs that are going to be tougher to... I don't know enough about it, but those videos that OpenAI released, the video generators where there's a woman walking down the street in Tokyo and you can see all these people around them and it's amazing that they've generated all this content, and I thought, "I wonder if that means that there won't be anything like you won't be an extra, you won't need extras anymore in your movies because you could just fill them in." So maybe those jobs go away, but I think what we do, there's so much about that curation and the sense of taste and reason that I feel like there's a long time to go before we'll be irrelevant. Yeah.

Maria Hofstetter:

Yeah, I think it's... We've already seen in our own roles, like you and I, Trisha, how our roles have changed where we are still entirely focused on content design, but it just looks different than it did before we started working on AI projects, and I could see that continuing to evolve where we're more thinking about a strategy for text generation or evaluation. We're kind of responsible for the text that's generated, but we're not the ones actually writing the text in the same way anymore. So I could see things like that happening more and more. But yeah, I agree. I feel good about the future.

Trisha Causley:

And I also think that there's... What we need to get better at, what I think the skill that I feel like you really need to flex is that ability to make explicit the things you know so that you can make the tool give you better stuff. So that making explicit your knowledge is something, I think, content people are pretty good at it because we're able to say, "Here's a guideline, here's an example." We're good at that in a way that, I think... We can do that with words in a way that, I think, it's tougher if you're a visual artist, for example, to say, "Why is this a good painting versus this other thing?" I think it might be tougher for them to put that into words, whereas we're much better at making those things explicit, which is that's actually what we need to continue to grow in is how do you take what you know and make that an external instruction that you could give to a model?

Kristina Halvorson:

And you're talking about prompts in particular?

Trisha Causley:

Prompts, but also evaluation because the part of the thing, if you're sort of running something, I mean, it might be a trained model or you just want to know are the prompts working at scale? You need to be able to look at output and say, "What's wrong here? What do we need to go back and fix, or what do we need to either fix in some other way or fail as an output?" You can't just say this is a bad answer, you have to say it's a bad answer for these four reasons and be really clear, because... I mean, unless you just want to hoard all that information and be like... My job will always be to review output, but probably you don't want to do that job. So how do you take your knowledge and say, "Why would you call this a better paragraph than that?" And so making that explicit I think is the muscle that we have to flex.

Kristina Halvorson:

So if somebody is a fantastic content designer, if they know how to get ChatGPT to do what they needed to do, or if they are really skilled on their own without anybody having asked them to use ChatGPT as, for example, really working through first drafts and honing in on final drafts if they need to, do you have any suggestions as how people can learn to flex that muscle? I mean, it's almost like going to the AI gym. What weights do we need to lift? What exercises could we do that then we could either even present in a portfolio, "Here's how I had a conversation. Here are the prompts I used, and then here's how I got to my final output, which isn't it fantastic." Do you have any advice around there?

Trisha Causley:

That is actually a great point. I never thought of it, but you're right that a portfolio nowadays should... I mean, if you're applying for that kind of a role, the portfolio should probably show off your ability to get good output based on your prompt structure or your examples, or whatever it is that you're doing. That's really great. Yeah, I think that that's like... If you're asking how do you get that, I think what you do is you use the tools to do what you would do by hand and flex the muscles of being the director as opposed to the typist.

Maria Hofstetter:

Yeah. I'll add too, which I think we've talked a lot about already, but I think our motto this year has been define what good looks like, and that act is, I think, practicing defining what good looks like in different contexts. So what is a good heading? What is a good description? But those specific attributes we've been talking about, like is it concise? What do we mean by concise? What's the character count? What's the tone? What do we mean by that tone? What are the specific characteristics of that tone? That definition work is, I think, the proof that you understand this was the output I originally got, this is where I landed, this is how we got it from A to B because this was our rubric, and showing that work I think will become important in portfolios for this type of work.

Trisha Causley:

Yeah, and I think that it's funny, for someone that's more of a marketing writer, I feel like I am not the person to evaluate marketing writing. I'm not a marketing writer. So I actually think that you are still needed for your taste and for your abilities to spot good because... You just can't do it if that's not in your skillset. I am never going to be able to tell if what the LLM spitting out is a good marketing headline. It might sound okay, but I haven't seen enough to know if it's trite or if it's boring or is it likely to do something. So it's still using your specific domain knowledge. It's not like, oh, anybody can come in with the right prompting skills and then they don't need a marketing person anymore. It's still necessary that you have the skills or the taste, all of those things have to be part of the team, right? So the LLM is doing some of the work, but you are the person, the expert. You need the expertise still. I think that's, I guess, the bullet.

Kristina Halvorson:

Well, we are at time, although I know there's so much more to this conversation. Thank you so much for joining me and for being so articulate and transparent about how you all are using AI and how others. If folks want to find you on the internet, where can they go? If people want to stalk you on the internet, where should they start?

Trisha Causley:

I would say LinkedIn, at my LinkedIn, but I don't post much.

Maria Hofstetter:

But for people that work in technology and AI we're kind of like... In our spare time, we read and knit.

Kristina Halvorson:

Well, in my spare time I watch K-drama, so although I know I should be posting on LinkedIn. Really, I think the last thing I posted was in fact about K-drama. I have just decided to write about whatever I want on LinkedIn. I'm really tired of being like, "I have to be smart." No more. No more.

Trisha Causley:

No more.

Kristina Halvorson:

That's right. Anyway. All right, thank you so much. I cannot wait to share this conversation with our listeners and I hope to see you around LinkedIn or elsewhere. Thank you.

Trisha Causley:

Thanks.

Maria Hofstetter:

Thanks, Kristina.

Kristina Halvorson:

Thanks so much for joining me for this week’s episode of the Content Strategy Podcast. Our podcast is brought to you by Brain Traffic, a content strategy services and events company. It’s produced by Robert Mills with editing from Bare Value. Our transcripts are from REV.com. You can find all kinds of episodes at contentstrategy.com and you can learn more about Brain Traffic at braintraffic.com. See you soon.