Making P.E.E.C.E. With AI
A framework I created to organize my understanding all things AI/LLM
Gah! Not a day goes by that there isn’t some new AI something in my feeds. I’ll try not to recount many of those here, else this post becomes old news a few days after it’s published!
Instead, I want to share the simple framework — something a bit more lasting — I’ve been using to make sense of it all. It’s a framework I created for myself to organize and monitor various topics — issues, really — associated with AI, and how they’re being addressed (or not).
NOTE 1:
While I’m using the term “AI” here, I’m mostly referring to the current era of massive pre-trained models and reinforcement learning— ChatGPT, Claude, Gemini, and so on. AI is a much broader field that also includes earlier technologies such as rule-based systems and traditional machine learning approaches.
NOTE 2:
My thinking here is framed in terms of concerns. Despite this seemingly negative framing, within each area of concern, I track a mix of things that are ‘Exciting’, ‘Alarming’, or ‘Promising’. This framing is my way of holding criticism alongside celebration. Why concerns? There’s plenty of hype and exciting new demos. This seems be all we see. We need more pragmatism to balance out the euphoria. As someone who prides himself on thinking about second order effects, systemic issues, ethical considerations, and so on… I’m frustrated by the unchecked hype with AI (and apparently, you can’t think critically about this technology without risk of being branded ignorant or afraid of change). With so much focus on the tools themselves, and surface level use cases, there’s far less discussion of failures (brushed off as “hallucinations”), and even fewer conversations — in popular media at least — around the risks, hidden costs, or “behind the curtain” issues. Why not be objective about all the things, so we can make rational decisions? This framework holds space for all of this — the good, the bad, and the ugly. From the environmental costs (and potential fixes) to the more straightforward question of “is this piece of tech something we can depend on with our customers?”
Those disclaimers out of the way, here’s the framework…
The Framework
I’ve organized my thinking into 5 areas of concern, easily recalled with the acronym P.E.E.C.E. (slap me, it wasn’t intentional — it just happened!), which stands for:
- Practical
- Economic
- Ethical
- Cognitive
- Environmental
While the labels should be self-explanatory, let’s go through each briefly:
Practical
Is it good at this task? Is it reliable?
As a product manager, this has been a top concern. You want the stuff you ship to be reliable, and yet… generative content can be wildly unpredictable. Behind the scenes, this leads to all kinds of tuning, blending of models, and so on — all to improve performance. But, seeing the raw results come across as weighted numbers, and knowing where and how things might fail — knowing the the inner workings — can be more alarming than comforting. While each new day brings some exciting improvement, or curious failure, I feel like knowing a bit about how it all works (it is ‘just’ math, at the end of the day!), makes it all the easier to see the cracks.
Whether the product risk is acceptable or not varies greatly with the use case. While there is no limit to how to use this AI technology, I’m broadly chunking things into one of four use cases:
- Computational Tasks — Extract, Tally, Sort, Filter, Produce (AIs tend to be pretty good at this; as a non-programmer, I love being able to do computational things without needing to write queries)
- Transformative & Summative Actions — Summarize, Rewrite in style of, Detect Sentiment, Cluster…
- Generative Actions — Explain, Write code, Compose…
- Coaching & Calibration — Is this a good KR? How do I write a good HMW statement? How do I format this query to…
Regarding this last bullet… I’ve written previously about Coaching-as-a-Service, building knowledge and best practices into the (digital) tools we use. While there are simple rules-based things we can reliably enforce without using AI (“does this task start with a verb?”), I’ve been monitoring how the models are and can be improved to offer more reliable coaching for more abstract or heuristics-based topics.
Nothing hard and fast about these buckets, they’re just something that seems to be working for me; your mileage may vary. For a more comprehensive classification of AI use cases, check out this post from Matt Webb on “Mapping the landscape of gen-AI product user experience.”
And for a slightly different take on Matt Webb’s framework, check out Josh Clark’s version in “The Shape of Sentient Design.”
Economic
What are (and will be) the operational costs? Is this economically viable?
There are at least two ways to think about economic costs:
- As a customer of these services
- As a creator of these services
Customers applies to both to individuals playing with the latest and greatest as well as every company building AI into their products (or even building their company around AI!). While many things are being given away for free or below cost at the moment you have to consider the costs, both now but also in the future, which could go up or down depending on any number of variables. And costs aren’t just to the AI creators. I first encountered last year, when a team I was leading had to upgrade our servers to host a specific clustering model — the raw computing power we needed meant switching to the highest end servers, which meant an increase in monthly spending substantial enough that we needed executive approval. And this was for a single feature among 100s of features within our product! Later, we found ourselves in a more common situation: Assessing the price (and performance) differences between the latest model compared to one from just a few months ago — while these per transaction costs are fractions of a penny, the newer model was nearly 10 times the cost of the other, and if usage took off, there was a scenario where we could end up upside down, paying more to AI creators than we were charging our customers. As a product manager, I felt it was responsible to do these back of the napkin explorations, to assess IF the financial investment was worth it, for features with questionable usefulness. For those whose products depend on AI, this is what I mean by economic concerns.
Creators are of course the OpenAIs and Anthropics of the world. I’ve seen an increasing number of articles looking at the financial sustainability of these companies, given the outsized costs needed to maintain their services. This recent article argues we’re in a generative AI bubble:
Should the bubble burst, startups and venture funds alike face possible extinction, and a big enough drop from the Magnificent Seven could spark skittish markets to panic, leading to wider economic contagion.
— Bryan McMahon, “Bubble Trouble: An AI bubble threatens Silicon Valley, and all of us.”
Depending on your perspective, something like DeepSeek’s R1 AI model can either be hopeful (open source code, significantly cheaper training costs, fewer hardware requirements) or an existential threat to the business model of most AI companies.
Like everything in this framework, we can add the alarming examples, as well as counterpoints, rebuttals, and possible fixes. It’s really just a space to hold and organize our thinking.
Ethical
Who or what is harmed by this?
Honestly, there are so many ethical issues, this box should be triple the size! And, to be honest, I should probably organize these into sub-topics, as I did with the Practical use cases (above). That said, here’s an unsorted dump of the kids of topics that fit into the ethical box:
- Sourcing data without attribution or credit
- Data theft
- Lack of transparency on training data
- Outputs lack interpretability
- Output may include toxic or biased content
- Exacerbates social inequities
- Concentration of power
- Framing as a ‘human‘ entity
- Absence of ethical regulations
- Hidden worker exploitation
- Dangers to our information ecosystem
I’ve split this box in half, to indicate those ethical issues we do see discussed from those that tend to get buried. For example, while there’s quite a bit of talk about stealing content, there’s false less talk of hidden worker exploitation.
A few other comments:
Lack of transparency on energy consumption is an ethical issue that correlates with the “environmental” area (below). We know there is a lot of energy being used, we just don’t know exactly how much, as this is something that’s not divulged. That’s one type of openness that I hope companies are pressured to be more transparent about, as energy consumption and use of natural resources kind of directly affects us all!
As far as openness of the technology goes, I like (and filed away for future reference) this ‘checklist’ of sorts from Timnit Gebru:
Speaking of wisdom from Timnit Gebru, she also makes an interesting case for replacing ‘AI Ethics’ with ‘AI Safety’:
I have never taken an ethics course in my life. I am an electrical engineer and a computer scientist however. But the moment I started talking about racism, sexism, colonialism and other things that are threats to the safety of my communities, I became labeled “ethicist.” I have never applied that label to myself.
“Ethics” has a “dilemma” feel to it for me. Do you choose this or that? Well it all depends.
Safety however is more definitive. This thing is safe or not. And the people using frameworks directly descended from eugenics decided to call themselves “AI Safety” and us “AI Ethics” when actually what I’ve been warning about ARE the actual safety issues, not your imaginary “superintelligent” machines.
— Source
[Hmm… I suppose I could label this the ‘Safety’ box, making the acronym for this framework PESCE… as in fish. Or fishy! 😜]
There’s more I could do with this Ethical area — for now it’s just a catch all box; I’m more concerned with the next one…
Cognitive
How does this affect our thinking?
No surprise, this is one of my top concerns. Those who know me, know I deeply value learning, and the active construction of knowledge. I fear the way many folks are using these tools is harmful, as it short circuits critical thinking. I’ve written elsewhere about this:
Not surprising, there’s a growing number of studies demonstrating the harmful effects of LLMs on critical thinking skills. Examples:
- AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking
- The Impact of Generative AI on Critical Thinking: Self-Reported
Reductions in Cognitive Effort and Confidence Effects From a
Survey of Knowledge Workers
To be fair, this is less about the tools, and more about the humans using the tools. And inflated expectations. But, these tools were designed to perform in a way that exploits our human biases. Comparisons to con-men and horoscopes are warranted. When we don’t understand how these tools work, we attribute cognition where there is none, and we confuse computation with probability. In many cases, this can be harmless (see use cases, above). But, spotting errors takes work, and without this work we start to believe in these mathematically generated phrases, when all it takes is a questioning prompt for the confident tool to do a sudden U-turn and apologize for the errors. That is, IF we know enough to — or take to the time to — challenge the AI word salad.
On a promising note, I’m delighted to see people teaching how these LLM tools actually work, such as this recent workshop from Alexandre Eisenchteter:
…. Is to become one!”
And… I’d love to see more thoughtful constraints, like this specialized version of Claude “designed to develop students’ critical thinking skills rather than simply provide answers to their questions.”
A few additional thoughts related to LLMs and cognition:
1/ Writing is thinking!
Writing is a form of learning. When we write, we engage in a kind of dialogue with ourselves that helps us to clarify our own thoughts. Skipping this process… Yikes! Just in writing this article — describing a framework I already use — helped me to clarify my own thinking, and articulate some things I hadn’t fully considered.
2/ Are the current wave of LLMs tools anti-social?
Something else I’ve been thinking about lately is how solitary the use of these tools is — this current wave of AI tools isolate us. Yet, when it comes to learning, we know that learning is a social activity, that we figure things out when ideas are challenged and explored. Within product management and design, I see artifacts (customer journeys, PRDs, roadmaps, etc.) that used to take days or weeks being cranked out in seconds. But for many of these artifacts, it was never the output that’s valuable. I’ll say that again: It was never the output that’s valuable; the value of these tools was always in the dialogue, disagreement, discovery, and discussion that happened along the way. Theoretically, that discussion can still happen, in response to the artifact, but… Do you see that happening? 😐
Environmental
How does this affect the environment?
Here is where I track things articles commenting on the (massive!) energy usage these technologies require, things like ‘every query waste half a liter of water’ (a questionable heuristic), or this alarming prediction (from former Google CEO Eric Schmidt) that “99 Percent of All Electricity Will Be Used to Power Superintelligent AI.”
As appalling as the statistics can be, I can also use this space to dig deeper, and explore what is being done — or, as is more often the case, what could be be done — to address these issues. Collecting these articles has helped me separate out issues with training the models versus ongoing usage costs. I can also use this space to dig deeper, and ask hard questions like ‘while this seems like a lot of energy consumption, how does this compare to media streaming or a traditional search query?’ This line of thinking can make you pause and reconsider the environmental costs behind all kinds of technologies we enjoy. In the end, all of this curation and holding up different perspectives side-by-side helps me to think more critically about these topics.
On things that could be done to reduce energy usage (ahem, responsible AI), I save things like this:
- A fascinating study demonstrating that a simple change to how the math is calculated could reduce energy consumption by as much as a 95%!
- Microsoft researchers who recently shared they’ve created a hyper-efficient AI model that can run on CPUs.
🤔 Existential?
Is this the end of us?
Note, I also have a space for Existential concerns — the Terminator and The Matrix style predictions that ask ‘Is this the end of us?’ ‘Will the AI take over?!’ Until something changes, my conclusion is that these fears are largely non-existent and are a distraction from the actual — as in happening right now — Environmental and Ethical issues. There’s enough of these kinds of existential apocalyptic news items that I needed a place to store them, but I don’t consider them a core part of this framework.
A quick note on spatial arrangement: These are sorted from the most visible, more frequently discussed topics to the less visible, less discussed, and harder to spot issues, with a horizontal diving line cutting across the ethics box.
How I’ve been using this framework:
Primarily, I use this to organize articles I save. And, I also save open questions or developments I’d like to monitor. I have a giant poster sized version of this in my digital whiteboard of choice (think Miro, Mural, Figjam, etc.), where I have started to actively organize interesting or newsworthy articles I come across.
Organizing things in this way has had at least two benefits:
Benefit 1: Chunk for comprehension.
With so many articles comments, announcements, and the like happening almost daily, being able to organize things into this framework has helped me to make sense of it all. Or, I can at least break a complex topic down into smaller, related sub-topics.
Benefit 2: Track improvements and rebuttals (not just concerns).
Being able to trace how things are evolving, or potential “fixes” to yesterday’s concerns, helps me stay current and objective. Despite the dominant framing of all this as concerns, within each box I look at things though a kind of “Rose, Thorn, Bud” lens, or (as mentioned earlier) I’ve shifted the labels a bit: ‘Exciting’, ‘Alarming’, or ‘Promising’.
Example of improvements: Early on, I observed that the results for some niche coaching applications were frequently wrong or less than useful (‘Alarming’). But then, RAG (Retrieval-Augmented Generation) entered the picture (Promising’) followed later by TAG and CAG, and other improvements. And on it goes.
And… that’s it! 🎉
This is a visual I’m using to make sense of a complex and ever-changing topic. I created this primarily for myself, though I’m sharing it here in case others find it useful; I’ve shared this with a few people in coffee chats, and there seemed to be interest, so… 🤷♂
Feel free to use this framework (here’s a PDF version that scales to very large sizes). If it doesn’t quite suit your needs, you’re free to make your own version.
Enjoy!
NOTE: This is the 2nd in a 3-part series of showing how to organize information for understanding:
- The first was my post on “WHAT Does A Product Manager Actually Do?”
- This one was about how I make sense of AI topics.
- The next post, will be a bit more… personal!
