The episode opens with Mike sharing two stories that set up a theme: context beats dogma. First, a bike rack bolt snaps seven miles from home with his toddler on board; he “hacks” a fix using a strap to limp back safely—imperfect but right for the moment. Second, he yells to stop that same child from leaning over a railing—normally a “don’t,” but justified to prevent harm. Bridging to software, Mike argues that sometimes you should break best practices: a hard-coded, partner-specific access control once shipped as a pragmatic stopgap, worked for years, and only now is being replaced with a proper, general solution.

From there, the group explores when and why “best” practices stop being best. Dave frames it as “there’s always a best move”—for this context. Will and Kyle note performance work routinely trades readability and safety for speed; measurement is essential, or all you’ve done is make code harder to read. They contrast language and ecosystem philosophies (Python’s “one right way,” Ruby’s malleability, Java’s explicit structure), agree that humans are the expensive resource (optimize for mental load and boring, readable code), and acknowledge domains (firmware, game engines) where constraints force “ugly” but necessary code. The team also debates two coexisting feature-control systems—slow but self-contained env-based flags vs. instant, granular runtime flags—concluding both are needed because different roles value different trade-offs.

They close on practical guardrails: prototype fast, even “sloppy,” to learn and validate; refactor after you’re green. Use YAGNI—don’t solve tomorrow’s problems today—and be kind to “future you.” Keep a backlog of intentional hacks, prioritize cleanup time, and recognize that some code paths matter far more than others (optimize the hot ones; duplicate templates when sharing adds needless complexity). Break rules deliberately in sandboxes to learn (e.g., Juice Shop, OWASP exercises), but in production favor simplicity: make it easy and explicit unless you’re forced not to—then measure, mitigate, and circle back to clean it up.

Transcript:

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I'm Mike, and I will be hosting again today. With me, I've got Jordan, Will Archer. We've got Dave.

DAVE: Howdy, howdy.

MIKE: And Justin, and Kyle.

And, as usual, I'm going to tell a story [chuckles]. Actually, I've got a handful of them here. I'm not sure if I can share all of them, but I think I want to introduce the story by first telling a story about cycling.

The great Sandi Metz shares a lot of good stuff. She talks about cycling all the time. I can do lots of cycling stuff, right? So, I'm going to tell a cycling story. So, I was riding with my kids, and I had my youngest in a bike seat sitting on a rack that was over my back tire. And he was getting a little big probably for it [chuckles], but it worked fine. I could do it.

What I didn't know is that maybe getting that top-heavy on the rack I had was putting a lot of stress on the bolts that were holding it up. And I was doing a loop, and it was, like, seven miles from home, luckily, not that far, but seven miles from home. And the bolt sheared off that's holding the rack up [chuckles]. Not a great thing, you know [chuckles]?

DAVE: With him in the seat?

MIKE: With him in the seat. That’s right.

DAVE: Because of the weight on the bolts. Yeah. Okay. Yeah.

MIKE: Weight on the bolts. And so, the rack drops on the tire, suddenly stopped. I’m going up a hill when this happened, you know, rocking back and forth because I'm going up the hill, of course, putting the stress on the bolts. I suddenly stop. You're not moving anymore, right? Now what [chuckles]? Seven miles from home. I can't really push the bike because now the rack's sitting in it. I can't take him off the bike and make him walk seven miles. He's, like, three years old [laughs], right? That's not going to happen. I also had a tether to pull the other two kids.

So, it was a bad situation. What do I do now? No one was at home to come pick us up. I could have called, like, an Uber or something, but what do you do? "Uber driver, can you come put three bicycles and a car seat [chuckles]?" There was no good way out of the situation.

DAVE: You got two kids on a tether. Dog sled.

MIKE: Dog sled [laughs]? Not going there [laughs].

DAVE: Fair.

MIKE: After some careful analysis, I got an idea, and I had a strap that I use for strapping stuff to the frame, like a water bottle. And I connected the strap to my bike seat, to the rails on the bike seat, to hold it up, and wrapped it around one side of the rack and pulled it really tight. So, it was just hanging from the strap by my seat.

I found that if I sat down on my seat…and I was standing up to peddle, right? I went as gently as I could. It didn't hit my spokes very often [laughs]. I got back on, and I rode seven miles home that way. And I made it. And I took off the tethers. The other kids, they rode independently [laughs]. They can make it seven miles. It was fine. So, we did it. We made it home. And that was not the purpose that those straps were made for [laughs]. There was nothing in there that was serving the correct purpose, but we got home.

Okay. So, that was story number one. I think it's a good one [laughs]. So, a tiny little story. You shouldn't yell at your kids, right? I think most people would say, “No, don't yell at children.” That accomplishes nothing.

Yesterday, my youngest, again, he is a common thread in this set of stories, was upstairs, and he leans over the railing, just jumps up, like, jumps up on the railing and starts leaning way over. And I yelled because [laughs] I was horrified. “You need to get down. You're going to fall off and something terrible is going to happen.” And I don't regret that.

He didn't like it. I didn't like it. There was some hugging and talking and making sure he was okay afterward. But, you know, I broke the rules, and I did something that normally I would not be proud of. And I made him sad, and I don't feel happy about that. But I was scared, and I wanted him to live. I didn't want anything, you know, any of the horrible consequences that could have come from this to happen. So, I said something, you know, I did not do what I would say that any parent should do. In this situation, those rules didn't apply [chuckles]. You know, you do what you can to save somebody.

So, we talk about software, right? I'm talking about outside of software. Let's talk about software. Sometimes you should not follow best practices. Sometimes you need to strap something on, and that's the right thing to do. Go ahead.

JUSTIN: I mean, if you're going to apply that back to your bicycle, I think you should still be riding with your kid hanging off that strap today [laughter]. That's really what happens [laughter].

MIKE: Well, okay, so, I've got a software story. Years ago, literally years ago…I don't think I'm saying anything that's exposing anything horrible [laughs].

There was a partner who had a feature request that was just for them, where they just needed certain employees to have access to a certain feature, and we didn't have a framework to make that happen. In this context, we just didn't need it, so there was no need to build it out. Nobody else was requesting it. And to build the framework to make this happen so that it would be generally applicable across all the users would take months. And the customer needed it, like, already, right? And there was no way that's going to happen.

So, we put in a hack. We just hard-coded the users that got access to the feature. If you're one of these users, you get access. We didn't make it too terrible because it was configurable, and you could set up in your environment. It wasn't, like, in-line in the code, but, yeah, it was a hack. And the customer was happy. Didn't build the feature, and years later, it still works [laughs], you know, the straps holding it on. And, in this case, it wasn't that important, right? It was a pragmatic fix to a nasty problem, and it was kind of a nasty fix to the problem, but it worked. Not too many people…there are actually a few other people who have requested the same feature, so it's grown.

There is a plan to mitigate this. It's actually probably going to be happening in the next quarter. But after years of doing good work, this hack is finally going to be retired. Man, it did its job, and I don't regret it [laughs]. I'm actually kind of proud that we solved this problem, and it worked, and nothing was harmed. Sometimes it's the right thing to do, and a brilliant hack, I think, deserves some respect. Oh, I've got some thoughts about some rules about when to break the rules. Well, I'm curious what you all's thoughts are. I threw some stories out there to seed some discussion.

DAVE: There's a phrase that I've trotted out in front of most of the people on the call at some point, if we've been working together, which is there's a fantastic book from years and years ago called “Bobby Fischer Teaches Chess,” Chess Grand Champion from the ‘70s. One of the lines that I remember from Bobby's book is he says, “There's always a best move. You might not have any good moves, but there's always a best move.”

And this is kind of what I see as that being, right? Is it OSHA-certified [chuckles] to strap a rack and then hang a child from it that you're actually related to and theoretically care about [laughter]? Of course not. But you're seven miles from home, right? That's probably the best answer, or at least one of the ones that qualify for that. If you'd had somebody with a pickup truck nearby, that would have been the better option, and it's fine.

The thing that I get thinking of is when we say best practices, I hear Best with a capital B, or Practices with a capital P. And I see it engraved on a leather-bound book that looks suspiciously like scripture.

MIKE: [laughs]

DAVE: And all you have to do is see somebody misapply scripture in the real world to realize, okay, yeah, it's called best practices because that's the name on the book. The gotcha to a best practice is the context. Is it the best practice for this specific moment? Usually, what a best practice is, is absent other context, this is a really good way to do it, and we probably should. And we've given thought to these things that you may not have considered, like security or, you know, integrity, or something like this.

And just jamming in a quick user ID check, of course, that's going to be rife with problems, so it's not a best practice. But I argue that having cash flow come in is absolutely the best practice, 9 times out of 10. And getting it working, getting it shipped, minimum viable product. And then if nobody ever used that thing, you throw it away at some point, and if somebody does and it gets really popular, you refactor it. Or, like Justin said, you ship straps with every bicycle and no bolts, because that's just how we do it now [laughter].

WILL: I mean, I'd say, like, you know, from a software engineering context, I don't know, I can't think of a single performance optimization that isn't deliberately contrary to all software engineering best practices. It's going to make it more fragile, more complicated, more state, harder to read, harder to update, everything. Everything you do to make things fast, right, is going to make it worse, right? So, is it worse or is it better? It's faster, and it's harder to deal with, right, and that's --

KYLE: Actually, what I was going to bring up, too, is that's where I've seen a lot of pitfalls in, quote, unquote, like, "best practices for coding," is when you want to get performance out of your system. They don't tend to go hand in hand.

WILL: Nope, cross purposes, every time. I would say, like, 100% of the time, they go cross purposes. I can't think of a violation of that. Sometimes you get performance stuff via the platform or via the language or via anything like that. But if you've ever sort of followed the internal turmoil of an open-source project that's really trying to get their performance game in order, let me assure you that the tears are there. Just because they're not yours, they [inaudible 11:34]. Going fast hurts, you know. Most racecar engines, you know, you couldn't even get started if you weren't a serious mechanic. That's how it goes.

DAVE: I used to own a Jeep Wrangler, and we called it, it feels like driving a basketball because the suspension on it is built for off-road, not for the highway. The thing you mentioned about optimization, this isn't that quote, but it's absolutely based…treats your statement as bedrock. The thing that I will tell people is you have to measure. When you do performance enhancement, you have to measure. And here's the quote that shows the bedrock of what you just said. Because if you don't measure, the only thing you know for sure is you made the code harder to read, and that is straight it, yeah.

MIKE: So, there are multiple different dimensions of good, and they're sometimes at odds with each other. And it's very easy for us to get fixated on one of them and to the detriment of things that we should be caring more about.

WILL: Unless forced otherwise. Unless I'm forced otherwise, I will always, always optimize for what's the least amount of thinking that I can do to get this project done? That's my North Star. What's the least amount of thinking? I just do it the easy way, like, mentally, mentally load. I optimize for my mental load, unless I'm forced to do some other thing. Going back and fixing bugs, high mental load. What did past Will do? That asshole? Oh my God. I don't want to think about him. I want it done. You know what I mean [chuckles]?

And so, that's it. I don't want to think about it. And if I do have to think about it, I don't want a migraine pill [laughter] reading through whatever the hell I did. And so, that's my North Star. Make it easy. Make it easy. Just finish it one time. Make it boring. Make it clean. Make it stable. And then if I'm making so much money that people will, like, back a truck up to my house so that I have to make it fast, fine.

MIKE: [laughs]

DAVE: Deciding in advance what the best criteria is going to be, when those criteria are kind of...how do you tell them apart, right? I rushed into Python from Perl because...I said in the pre-call Perl is a write-only language. I love Perl. I've spent a lot of time down in it. I've blessed a hash or two. If you know what Perl is, that tells you about where I was. I've never met Larry Wall, but that was next. I love Perl, but it's very, very cryptic. And Python comes along, and it's all about make it clean; make it readable. And I'm like, yeah, let's do that.

And Python has this rule, which is there's one right way to do it. And the supporting quote that they will use is…because you come back and you say, "Well, but if I want to do a map reduce, that should be a slick inline because I'm just doing it." And they're like, "No, no, no, no. You need to have the for loop. You need to have the..." I mean, they have comprehensions now. I'm old. This was a long time ago. But Python, you had to have the colon, and the indent, and da-da-da. And if you want to do it twice, you had to have another one. And their argument that they would say is the special case is not special enough to warrant a special case.

What they're saying is if you can look at the structure and you know what the structure is, you can infer meaning from the structure, then that's going to increase your understanding. It's going to reduce the mental load. And you then put your...the difficult portion of your algorithm is in the parts of that structure where you would expect to go look for.

And after I'd been in Python for five or six years, I started realizing there's times when the special case is really, really more convenient. Like, doing it the one right way is the worst way possible to do it. And all these promised features of, "Well, it's going to be more readable," no, it isn't. None of that actually paid off. It turned into scripture for me. And as soon as it became dogmatic, it started to really leave a bad taste in my mouth.

This is why you've heard me talk about coding, where I'll be like, I like to program in a language where a big idea should take up a big space on the page. But a small idea should be compressible down to something very, very small. And I have written a 300-character line in C with a comment in front of it that said, "Don't touch this unless you know what it means. This is a big, hairy, scary…" And, literally, it was vector math, so I didn't want people touching it.

It was literally, if you touch this, the motorcycles will fall over, because I'm literally applying a torque on the Z-axis of this motorcycle to keep…it was a racing game for when I worked at Acclaim. And I wanted a very dense, very packed, very...I wanted the line to scare you, not because you needed to be afraid of it, but because I needed you to respect it.

And when we did actually make a change to that line, it was two of us at a whiteboard, and we spent three hours unpacking the math, unwinding it to deoptimize it, to make it easier to read and understand. We made the change we needed. And we packed it back up because all it was, at the level of the method we were looking at, it was just one thing that was, make sure the bikes don't fall over. That's all it was. It was just give me a little bit of a torque impulse in the rotational axis. And I guess that's what torque means.

But we wanted that compressed down to be something really, really hard. Would I write production code like that all the time? No, not unless that was the...I wouldn't write the whole method that way. This was a 2,000-line physics method, and that was one teeny, tiny piece of it. And I wanted it to be one teeny, tiny piece. And if we had unrolled it, it would have been half the function for one-tenth of the functionality.

And unfortunately, optimization...this was on a PlayStation. I couldn't make a function call because I did not have the budget to push things onto the stack, make it jump, and do it over there. So, I literally couldn't extract it out of the source code [laughter]. It had to be packed into that spot.

WILL: Oh, I remember that, yap. I can't allocate memory in this loop. This loop, I cannot allocate memory and maintain the performance guarantees that I need to keep my frame rate up or my bit rate up, or whatever.

DAVE: Mm-hmm.

WILL: Mm-hmm. Mm-hmm. Yeah, you're going to see nasty work in situations like that, real, real ugly work. And you don't do it unless you have to.

DAVE: I toured in...my grandfather was an oil man, and I remember riding in the truck with him as a little kid and looking at the pump jacks. And I remember at one point looking at this machine, and it's hydraulic cylinders 12 inches in diameter, and it smelled of sulfur and just awful. It was tar everywhere. It was filthy. It was awful. It was a miserable experience. And me as a six-year-old kid going, "This is gross. Can we just not be here, please?" And my grandfather says, "Look at how beautiful that is." And I'm like, "What?" And he was basically saying, this is a big machine that does a big job. It's important, and it's a messy job. It's shoveling muck, almost literal muck, right?

WILL: It’s literally shoveling muck.

DAVE: It's pumping sticky, gross crap out of the ground so that we can have civilization. And that big, gross, tar-covered, sulfur-smelling machine was doing exactly what...it was supposed to be covered in tar and smelling like sulfur. That's what it was. That was its point.

WILL: Yeah, man.

DAVE: These analogies are getting weird. I like it.

WILL: Yeah. Well, no, I mean, I think that's, I mean, here is specifically, like, this is one of the grievances that I've always had with Ruby in particular. And [inaudible 19:16], I mean, I love the flexibility and the dynamism of the Ruby language, but I feel like there's a foundational trade-off. It's a foundational strategic error about human nature that Ruby programmers got kind of wrong.

And as much as I hate to say it, I feel like despite being in my soul a Ruby programmer and I really love it, but the language itself is completely malleable. It can do whatever you want. You can write whatever domain [inaudible 19:46] language that comes into your mind, and you can make anything happen, and people frequently do, right? Which is a wonderful thing because you can, like, express these beautiful thoughts very succinctly, and things can just flow. And you can run the mother of all demos where you just, like, bam. You just kind of log up in real time right in front of you because all these things magically unfold, which is a beautiful thing.

But versus, I mean, the counter-example of that is, like, just nasty, crusty, fusty old Java, right, where everything, like, where's this class? And I'm like, well, I'd tell you. I'll give you a hint. It is laid out right in the class. There it is, boom, boom, boom, boom, boom, directly structured. I could follow that chain right to the file, and I know exactly where the hell it is, and I know exactly what it does, and I know what it will do, and what it won't do, what it expects, and, like, et cetera.

But it's, like, big, and it's crusty, and fusty, and, like, ugh. But it's all laid out explicitly. And, like, the problem that I think where, like, I was like, okay, well, why is Ruby such a niche language? I’m like, no, well, it's kind of because of this. Because if it's just Voodoo and it happens just magically, right, well, how do you know the incantations, the magic spell? You know, where's the grimoire, you know, that I'm going to use to summon it up this web page? What's in the documentation? And what do programmers not really get right? Yeah.

DAVE: The fact that they don't ever document, let alone document well.

WILL: Nothing. Yeah, and so, like, you get the sort of, like, graveyard effect, right, where, like, the grand wizards, you know, they know everything, you know [laughter]. But, like, but if you're not the elite Dumbledore-level sorcerer, you can find yourself stalling in the dark for a very long time trying to find, like, where did this stuff go?

And, you know, Rails starts to mitigate it because, like, they're like, hey, just put it in the right place, okay? But that's, like, you hope you put it in the right place [laughter]. You hope the other guy put it in the right place, you know? But maybe they were busy. Maybe it was a Friday afternoon, you know? Maybe, like, somebody put it in, and they're just like, ah, just put this one next to the other one; it'll be fine, you know? Anyway, it's a little bit of a tangent. But, yeah, and it's just, like, I'm just trying to be as lazy as I can get away with and still keep my job.

MIKE: So, you're saying, building on what you said before, making things explicit, boring, easy to read should be the goal in almost all cases.

WILL: Just keep it easy, man. Keep it boring.

WILL: Like, I’d love a boring day at work. Do you know how bad I love a boring day at work?

MIKE: [laughs]

WILL: Oh my God. Well, I could, like, put on a podcast and, like, I'm not, like, toweling my brow off, where I'm trying to, like, figure out what's happening. Or I'm just, like, I could just pretend I'm [inaudible 22:48], you know what I mean? It's, like, this is just easy, ta-ta-ta. Put some music on, you know what I mean? Just vibe out. Like, I'm not crunching, like, numbers, like, splitting the atoms in my brain. Oh, I get three of those days a month, maybe, and I mark them on my calendar [laughter].

DAVE: That's my grandpa's good day, right? Is go out into the oil field and pump jacks, and all he's got to do is just check on them and make sure they stink. Yep, they stink. We can go home.

MIKE: Well, I was thinking about this as you were bringing it up before. Hardware, so modern hardware is, in general, fast, and storage is cheap. I'm going to say never, but it's pretty close. Yeah, you're never going to use all that CPU, right? You're never going to use all that memory. There are cases when this is not [crosstalk 23:40]... There are violations of what I'm saying. But, in general, the hardware is cheap.

You know what's not cheap? Us. We are not cheap. And if you make us more expensive, you've done something wrong. Technically, we could hand-roll loops in Assembly and be optimizing all the time. But nobody does that unless there's some very special need because it is incredibly time-consuming and error-prone. And that costs time that relatively highly paid people need to take. And whereas make the hardware do it; make the compiler do it, means that we get to think in something much closer to English, or whatever our native tongue is, right? Something similar to natural language.

WILL: [laughs]

MIKE: What’s that?

WILL: I said, please no English.

MIKE: Yeah [laughs]. Something similar to whatever our native tongue is. Something that reads largely like natural language but is unambiguous. Forget the fact that LLMs exist nowadays, and they might do what you want, and they might not. Unambiguous language.

We're writing code for us. I've heard you say this, Dave, a lot. We write code for humans. Most of what we're doing is writing this documentation to ourselves and our peers, and the computer goes through a whole lot of work to make sense of it because we've set up a bunch of rules. But we design these languages around our understanding because, in the end, it makes financial sense. And, in general, we're doing this because we want to get paid. We're trying to support our families, and we're doing whatever it is that our aspiration is. We want to eat, therefore, we write code. And we don't want to take more cost than it should. So, we write things that make sense.

KYLE: One thing I'm thinking about is you said hardware. There's another environment where I feel like best practices get ignored even more and more, meaning more, like, reliability and modularity, and that's when you're working with microcontrollers. Anything actually in, like, firmware-based, you're very limited on the resources that you have. And the code, if you start looking at firmware code, because your memory modules are so small in some of these things, it's not readable. You need that master's degree or that doctorate to be able to even go through some of that code to figure it out, and be, you know, that these controllers are in Java these days or whatnot, you know.

WILL: No, they're not [laughs].

KYLE: No, they're not [laughter], but, you know.

WILL: It's really not because if you write it in nasty, old C, I can say [inaudible 26:39].

KYLE: It will run better. Yeah, but point being is, like, it's still, like, looking at a different language at some point, right, even if it's one that you're familiar with because it is so condensed. And you're not going to see the same best practices that you would if you were, you know, going for a full-fledged app for software-based.

WILL: I mean, I’d say, like, embedded device drivers, right, like, they have their own best practices, and it's, you know, I mean, like, it really is. I mean, like, they're not...I think everybody's trying to do the least amount of work they can before quitting time. I mean, and a lot of that stuff, a lot of it is sort of, like, you know, like, you're really not doing it in a way that’s, like, hard. It's hard work to think about, but it's also a lot narrower, right?

Because, like, you know, like, device driver, what are you doing? You're pulling bits off a bus, right? You're turning that into data, right? You're putting that data out to an address, right, you know, your memory mapped I/O, whether it's like, I'm going to save it out to a buffer so that, like, somebody can pick it up downstream. Or I'm going to, like, send these pins for this controller to move this, you know, motor or whatever the heck you're doing. I mean, like, it's actually…so, I mean, on some level, I don't want to, like, undersell, like, how hateful a process it is because it's not fun, and you don't want to do it. But it's also, like, once you break it down, you know, it's not so bad, you know. If you want some really, truly awful, hateful work, performance compilers.

MIKE: Gotcha [laughter].

WILL: You're getting all the smoke, all the smoke. You need to know, like, all the nasty work that's being done on the platform level, on the hardware level, at the language level, and, like, you've got to see anything that comes. Anything that comes in the door, you know what I mean? If your lexer, you know what I mean, will acknowledge it as a valid program, you've got to figure out some way to make it happen. Oh my God. I'm making web apps. No more of that for me [laughter]. [inaudible 28:58]. I'm done. No more.

JUSTIN: So, to kind of bring this back a little bit to, you know, you write some code. It does the thing that you need it to do in the moment. What's the next steps? Like, I've often created tickets that I put on my backlog, and I specified, here, I did a thing because of X, Y, and Z. We should fix it sometime. And then, of course, it sits there on the backlog until we, you know, we just ignore it for a long time. But do you guys do that? Do you guys, like, [inaudible 29:35] code, or do you create a ticket?

WILL: I love it, man. I'll put it in the sprint. I'll just be like, you know, I'll put it in the December sprint and just be like, "Hey, I did this," and, like, it seemed like a good idea at the time. And, I mean, honestly, it's like exponential backup, right? Where it's like, okay, listen, I have to hit my ship date, okay? I got to hit my ship date or else, you know, we're all fired, you know? So, okay, you know what I mean?

And then, like, maybe, like, two weeks out of the ship date, we're going to see where we're at. And then maybe a month later, and then, like, six months later, and then, like, a year later, and then, like, you know, like, every year. We'll be like, Hey, it's, like, a week before Christmas, you know? So, let's all look at our stockings. Let's all check our stockings for lumps of coal. Just sometimes when I'm like, listen, I know I'm not doing anything. It's, like, the day before Thanksgiving. We sure we're still good with this? Okay. You know, whatever. And, you know, it could be like [inaudible 30:43], where it's like, yeah, I did a bad thing but, like, the blast radius is small. The value is high. And I'm not, you know, I don't know [inaudible 30:55], you know. That's the next guy's problem [laughter].

MIKE: It's something that I was thinking about before, you know, before the call, I was writing down some notes. Some things are more important than others, you know, that sounds almost [inaudible 31:13] logical [laughter]. If there are two things of different importance, one is more important than the other. But it actually matters. There are things that matter a lot, and there are things that matter a little.

For example, if you have a piece of code that is getting hit over and over and over and over again…I had this happen once. There was some permission checking system in an app I was in that was, you know, of course, it was used dozens of times for every request you made because, you know, it's just checking all these permissions all over the page. And it was used all the time, and it was slow. It was terrible slow.

We optimized that, and, like, just that one thing. We optimized permissions, and the app went, like, twice as fast or several times faster. That code was important, right? It really mattered. We needed to make that run well, and so we did, like, some caching on it. Again, it makes it worse to read, but it mattered. It was enough. It mattered enough that it was worth getting the attention.

On the other hand, if you've got, like, a template, and you're on one page, and you like the idea of having it shared but, no, this other page needs to be a little bit different, then copy that thing. There's almost no cost to having another file and making it look a little bit different, yeah? There's a little bit of duplication. But you know how easy it is to copy a template? And you know how hard it is to try to share something between templates and make sure that that's right? And it's off a little bit, and then now it doesn't match anymore, and you're fighting that for years. That's a nightmare, right? Copy the file. Nobody cares. It'll work. And these are supposed to be [inaudible 32:51]. If it was going to be the same, you wouldn't have been asked to copy it in the first place. There are things that matter a lot, and there are things that don't.

JUSTIN: I mean, that goes back to the importance of grooming your backlog and everything. It’s like, you look at your backlog, and you look at all the tickets you've created in the last six months or month and a half or whatever. And you're sitting there, and it's like, what's the biggest bang for the buck? And you have a limited amount of time. The company has a limited amount of budget. That thing that you did to fix prod, that's gone. That's still working. And you have the choice between fixing that versus shipping new products that could be new revenue. That's an easy choice.

DAVE: I worked with a guy years ago who wanted to put some backlinks on our website for SEO. So, basically, he wanted us to give him the ability to just drop links into our website that would point to his other sites. Okay, that's fine. He's basically farming. What do the kids say? Farming aura but with Google. He's just trying to increase the amount of Google index rank that he can get. And I'm like, really? You're just jamming extra links? You're just stuffing links into all of these sites so that it radiates back? And he paused, and he goes, "It's a cheap trick, but it's a time-honored cheap trick." And I'm like, that's another great way to say best practice, in my opinion [laughs].

WILL: If it makes money, it makes sense, you know.

DAVE: Pragmatism. If it works, it's true. Absolutely. Pragmatism.

WILL: That's the truth, man. I can't fault it, really.

MIKE: In that backlog, we should probably not dive too much into this because we could end up spending a full session on this. We probably have before. But you've got to have some time on your schedule to be working on that backlog because if you leave a bunch of garbage around, eventually, you've got a big pile of garbage. And you do need to be cleaning up. That's, I think, a different discussion. You do put it on the backlog. You prioritize it. And you spend a significant amount of every sprint, or whatever your time interval is, working on it. Maybe that number is 20%, maybe 30%. Maybe you had a dedicated team that’s going through and just cleaning stuff. I've seen that work as well, and they're a meaningful part of it.

WILL: I’ll do it in my free time.

[laughter]

MIKE: Sure [laughter]. You've got to dedicate some time to it. I did take that rack off my bike, and I came up with a different solution. And I've got a trailer that's mounted to my seat now. It works great. The alternate solution has to be built. You're going to have to deal with that at some point. It doesn’t mean you made the wrong choice at the beginning, but you do have to get to it. So, I'm not advocating, yeah, make the mess. It'll take care of itself. But you should make the mess when you have to and prioritize what you fix, and then be working on the top priority item.

DAVE: You said something a few minutes ago, Mike, about some things are important and some things are not. And, for me, one of the toughest things to start, and this goes back to don't treat best practices like it's a Bible, the thing that's important might change. And two different things might be important to two different people at the same time.

In our architecture meeting today, we talked, and I think everybody here...Jordan might not be aware of this. We have two different ways to control whether or not a feature appears on the website. We have one way. We have a library that we wrote called system settings. And it will look at environment variables coming into the system, and you can set an environment. And we have these local files that we keep in the project that you can override or supply an environment variable if it's missing, and they're cascaded by environments. It's a nice fine-grained way to do it.

And It's super convenient as an engineer because I can say, here's this feature. I'm going to put the feature flag in the code, and I'm going to turn it off. And in my local code, I'm going to turn it on. I can test it. We can play with it. I can give it to QA. They can turn it on, play with it, test it, and da da da, and down the road we go. And when it's ready to go out, we just ship it to production. We don't have to involve DevOps. We don't have to touch anybody else. We don't have to do a dual deploy. We are good to go.

However, our deploy process is kind of lengthy. It's fairly long here, and the review process is even longer. This is not a fast way to turn a thing around. And sometimes, I think I've said this before, when the egg hits the fan, you need to be able to turn off the fan or the egg, one of the two. And for that, you need a kill switch.

We have this other system called Kipper, and Kipper is an open-source project. You can get it right now and play with it. It's fantastic. Kipper, you basically go to the Kipper library and say, "Hey, is this feature enabled?" And we have the advantage of saying, for this merchant, or for this customer, or for this class of users, or whatever, you can give it actors. And Kipper can say, "Well, it's enabled for that person, but not this," or "It's enabled for this logged in user, but not that." So, again, you can hide people back and forth. And you can go to Kipper, and you can turn it off, and it's off instantly. So, if something blows up, you can make it stop blowing up, or you can turn it off.

But this is the reverse problem. Now, if you want to ship something into prod, you have to go to our Kipper site, in whichever environment you're in: local, preflight, prod, whatever, dev. Whichever environment you're in, you have to go find that in Kipper. You have to type that variable in, which you're going to mess up once, right? We all did it once. And you're going to figure out why it's not enabled on the site. Am I the wrong user? There's all that mess.

There's a batch deploy. There's a tandem deploy for how to do this. And different users find these things indispensable. With Kipper, you can say, "I want to slowly cut traffic. I want just 10% of my traffic going here." Or "I want 50-50. I want to do A/B testing between the two." You can do that all with Kipper. System settings, now the flag is either there, or it's not. You could go in and say, "Here's 10 merchants that I want to be included in the system setting," and keep that in environment. But if you want to change those 10 merchants, you've got to do another deploy, right? Again, it's slow.

And we kind of went around the table a little bit in architecture because there are people on one side who didn't see the relevance of the other side, and vice versa. People are like, "Well, I need this. Who cares about that thing?" And the other people were like, "Well, I care about that." So, where we ended up with is, right now, we're going to continue using both systems. But we have an increasing desire.

And this won't surprise Mike, this has been coming up in architecture for three years, that maybe we'd like to have something that can do both. And it's just one place to do it in the code, and you can deploy it if you need to. You can turn it off instantly if you need to. You can filter it, A/B, whatever. But it's taken us three years of going around trying to determine what is the best practice, which one...and we haven't had a good understanding yet of like, "Oh, we need a solution where the best practice is to solve all of your best cases and all of your best cases." Those are constraints. It's not a best practice. There's two groups, and they both need the thing that they need, and because they're both true at the same time, we have two systems. If we replace them, we're going to have to replace them with a system that can do both things.

MIKE: But you're able to live with both of them for a while.

DAVE: Yeah, and I can't get my manager to give me six weeks to go write a new library to build a new thing that would do the thing. I'll make it a Rails engine. We can plug it in the app. It'll be so slick. Yeah, yeah. It'd be nice. It'd be nice. Making money for the company is more important.

MIKE: Let me throw one other time when I think it's a good idea to violate best practices. Play. There are times when you should break it on purpose when it's not in production.

DAVE: Yes. Yes.

MIKE: If you want to learn something, one of the best ways to learn it is do the wrong thing. There's this great thing, Juice Shop. It's [laughs] this system for learning web security, where it's a website that has everything wrong with it. It violates all the best practices for security. And you can go in, and you can do horrible things to it [chuckles] that should not be done because it allows you to do it. It's like a scavenger hunt. What are all the things that you can go in there and you can break? You wouldn't normally put up a web app like that, but you learn so much by having done it wrong.

I did something similar once years ago where we built our own web application. Over a series of weeks, we had a session a week where we would learn one of the OWASP top 10, and somebody would be assigned to violate the rule. We went through each of them and added some code somewhere in the app that violated one of the OWASP top 10 rules. And then the rest of the team, once a week, would have a chance to try to break it. And the team that I was with learned so much about security.

I don't think they ever went and put any of those vulnerabilities in their code ever afterward because they got it. I think that you learn a ton from violating best practices on purpose in a sandbox and should be actively encouraged to do so. There's something about the mental exercise of it. But we should deliberately set up situations when we're learning something, to break the rules, to figure out what those boundaries are because sometimes maybe you should. I mean, there's probably some cases where you should. Security? Maybe not. There's probably an exception here. Well, there's there, the Juice Shop. You violate security on purpose so people can learn.

In general, no, you never, ever, ever want to do that, and it makes it really hard. I remember trying to allow some SQL injection in Rails, and it was hard. I had to [chuckles] break so many things to let that through. And you learned a lot of what you had to do [chuckles].

JUSTIN: I, unfortunately, have seen that problem before in Rails [laughter]. They tried really hard, and they succeeded [laughter].

MIKE: It is possible, but you have to --

WILL: It's Turing complete. You can always break it.

MIKE: I saw a whole reporting system built once that was reliant, at its core, on SQL injection.

WILL: Oh.

DAVE: Fantastic.

WILL: I would also say, like, you know what I mean? Maybe, like, adding on to, like, play or a corollary for play, I think the value of a sloppy, like, prototype is just way, way, way undervalued. I find myself, like, because I like to write good code, and you know what I mean? And I don't like people dogging me out, you know, pull requests, you know? Because, like, I did something sloppy, and I'm just like, all right, I like, you know, I have a tendency towards golfing, you know?

But there's a lot to be said for just going out there and, like, just rip and run. And just make it fast, and ugly, and nasty, and just make it work by any means necessary. By any means necessary. Do things you know are wrong. Just like, yeah, this is all one function. Yeah, that's right.

MIKE: [laughs]

WILL: Psychometric complexity who? Get it. Like, we're doing it all. And then, like, you know what I mean? And then when it's done, like, I find, like, because again, like, I don't like thinking really hard. And it's a whole lot easier to look at, you know, an ugly, finished, working piece of code and look like, who wrote this? What idiot excreted this thing? You did this wrong, and this wrong, and this wrong, and this wrong, and this wrong. And this should go here, and this should be a class. This should be in a module. What are you doing? You know, like, that's easy work. Like, I find it effortless to be a hater [laughs]. And so, you know, just slap something together and fix it in post. I know that's, like, I know that feels counterintuitive to most people.

DAVE: Does it?

WILL: But, for me --

DAVE: Because what I just heard you say was get to green fast and then refactor.

WILL: Exactly.

MIKE: What I heard, like, I've been working with one of my kids through an English class, and he was assigned to write a first draft and make it sloppy [laughs]. He said, "Don't worry about grammar. Don't worry about making it pretty. Don't worry about what order things are in. Just get it out there."

DAVE: Yeah, I'm trying to teach you the process. Stop trying to produce a perfect document because you've got a way of generating that document implicitly without thinking about the process. And that means you are end-running the process to generate the document. I need you to stop generating the document and focus on the process. I may have had that conversation with another programmer recently, so I'm a little head up about that. Now, do it to hear…Okay, yeah, yeah. No, come back. Stop jumping ahead.

MIKE: You know, that first draft lets you start putting in that criticism. It's a whole lot better than having nothing to criticize. You may have your beautiful opening sentence. That doesn't get you very far. You can get a whole lot farther with a working product that you can then tear apart and put back together pretty.

DAVE: There's a million perfect sentences. And if you haven't figured out what the story is about, you've got a one in one million chance of getting it right. It could be a perfect sentence. It could just be the wrong one.

This goes back to that idea of the waterfall. You run the risk of working your way all the way up the ladder to discover that you built it against the wrong wall. And you don't know until you're at that top rung because it doesn’t work until...you can't even turn it on until you've got that last piece plugged in.

MIKE: Wasn't it a long-running joke in the Peanuts comic that Snoopy was writing a novel?

DAVE: Probably.

MIKE: And he had the opening sentence, "It was a dark and stormy night," and never got beyond that [laughter]. I think that strip was out for decades, and I don't think he ever got beyond the first sentence.

DAVE: I don't think so.

MIKE: And it was a perfect first sentence but didn't have the story. So…Go ahead.

JORDAN: I think this sounds similar to my personal experience of working on projects. Like, I know that I should be following...They tell us in school that you should be following best practices and doing things the right way. But when you have an idea, and you have motivation, and you're doing it slowly by following best practices, sometimes you just lose that motivation. But you have a perfect start, but you don't have anything as a result.

It goes back to your goals and pros and cons. If I have this motivation, and I make it quick, but it doesn't look the greatest, or it doesn't follow the best practices on readability and stuff, at least I'll have something, and it's there. And I think it's a lot easier to go back and fix it rather than continue trying to build something perfectly, so…

KYLE: You just described an MVP that they decided to ship to production [laughter].
​
WILL: [inaudible 48:48]

DAVE: I would sometimes call that the triumph of YAGNI, where you'll come back to some code, YAGNI is Y-A-G-N-I, You Ain't Gonna Need It. And, basically, it's just pull yourself back a little bit and say, don't write a solution to tomorrow's problem. Trust tomorrow you to do that.

And so, sometimes I get into code, and I'm like, this whole thing is missing. I'm supposed to be adapting this, and the code to support it is gone. Who wrote this crap? Oh, the person who wrote it didn't need this piece. And thank God they didn't write it because it was factored differently than the way I would have needed it. This is not a stupid oversight or omission. It feels like a stupid oversight or an omission, but it's not. It's a triumph of the principle of YAGNI because you've left me clean field to build what I need to build. -

JUSTIN: Then you look at the blame history, and it was you. It was you.

DAVE: Oh, it's always me [laughter]. The call is coming from inside my own head, absolutely [laughter]. I have started...I've said this, but this is genuinely a sentiment I've experienced. I have started trying to be kind to future me when I write code. That's an actual thing I try to think of, be kind to future me. And you know what I've discovered? I've discovered that past me has started being a little bit less of a butthole when I'm working on my code. It's the weirdest thing. I have no idea what future me thinks about it, but past me is getting nicer and nicer. I'm actually enjoying it.

JORDAN: I feel like best practices, though, like the idea of best practice is to be nice to future you. So, like, I guess it depends on, like, what kind of practices you're following. But, in my mind, best practices is to make it easier for future you, even though, I guess, like, the whole point of this discussion was, what is the best thing to do in the present? So…

But it's the recurring theme that we come back to. If it's not simple, then you're probably doing it wrong, like, that should be your guiding principle. Make things easy on future you. And you only make an exception when you have to. There are times when that's right. You may have a performance reason you need to do something. Building a sophisticated caching system is a nightmare, but it solves a lot of problems, and there are times and places. But avoid it until you have to. I think that's a good place to leave today.

DAVE: This was awesome.

MIKE: Make it easy, unless you have to, and, you know, clean up if you can, if it's important.

DAVE: I would add, consider...make it easy on future you. But from time to time, think about how much it hurts right now and ask yourself if it's because of the thing that you did to make it easy. Sandi Metz gave me some advice about do this thing and then stop. And I got to that point, and I knew it needed to be refactored, and I spent two hours refactoring it.

Then she said, "Great. Can you understand your code?" And I went back and realized that after that stopping point, everything I did just added ceremony and unreadability to the code. And it was all what I thought was best practices, and I realized this was dumb. I should have been asking myself if I can understand this.

MIKE: Great. I think it's a perfect place to end. Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, Mike kicks things off with a cycling story that serves as an analogy for problem-solving in software engineering. Planning a long ride to Illinois’s highest natural point, he had to carefully map his route with handwritten directions before realizing he could quickly write a small program to calculate distances. The story highlights how coding, even outside of professional contexts, provides practical tools for organizing complexity and solving problems efficiently. This segues into the episode’s theme: the value of side projects as creative outlets that both challenge and refresh developers beyond their day jobs.

Dave picks up the discussion by sharing his own side projects, ranging from building automated tools for games like Minecraft to recursive Sudoku solvers. He describes his habit of scripting repetitive tasks at work and how tinkering with small, often quirky coding challenges keeps his skills sharp. Will chimes in with his perspective on solving Sudokus using deduction instead of brute force, sparking a lively debate about problem-solving strategies and approaches to recursion. They also discuss playful experiments like writing adventure games in SQL or porting Doom into Postgres—projects that might not have practical business value but showcase curiosity, resilience, and creative problem-solving, traits they argue are vital in startup or complex development environments.

Later, the conversation broadens beyond coding to explore balance, curiosity, and learning from outside experiences. Will reflects on being new in a role and choosing not to code outside of work, instead focusing on absorbing context, leadership, and finding inspiration in non-technical pursuits—whether it’s parenting, reading fiction, or woodworking. Dave shares his experience building cigar box guitars, while Mike recalls colleagues whose hobbies, from rose gardening to counseling, enriched their professional lives. They conclude that having creative or restorative pursuits outside of work—whether technical side projects or entirely different hobbies—ultimately strengthens problem-solving skills, resilience, and perspective in software engineering.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. Dave and I are going to do kind of a joint hosting thing today.

DAVE: Hello.

MIKE: We've got Dave Brady here, and we've got Kyle, and we've got Will Archer. Actually [chuckles], we've got the Archers [laughs].

WILL: No relation.

MIKE: [inaudible 00:00:37] relation [chuckles], Kyle and Will. But the four of us are here today, and I'm going to start with a story.

As Dave and I were talking, we were kind of debating on the pre-call who was going to host today. And we're both going to do this, but I've got the story. I've got the story. So, that's [laughs] why I [inaudible 00:00:57]. I have mentioned before that I've done a lot of cycling. I won't go too deep into it. I've done a lot with my kids, which has actually helped strengthen me. And, occasionally, I've gone out solo and had these great, long rides.

I went on a long ride a couple of weekends ago. And the ride itself is not where I'm going here. But ahead of the ride, I needed to find out the route. I was planning a new way I'd never been before. I went to the highest point in Illinois [laughter].

DAVE: 12 feet above sea level?

WILL: 13, baby. 13.

DAVE: 13?

MIKE: Okay. So, I misspoke. I went to the highest natural point in Illinois. The highest point in Illinois is not actually a natural point. It's the top of a high-rise in Chicago.

WILL: Is it a tower?

DAVE: Like the Sears or whatever, yeah.

MIKE: Yeah. It’s now called the Willis Tower.

DAVE: Is it still --

MIKE: It's called the Willis Tower.

DAVE: Willis Tower. Is it still the tallest?

MIKE: It is. But there is a...the highest natural point in Illinois is called Charles Mound. It's 1,235 feet, which my house is about 800 feet, so it's not that much higher. Except it's a ways away, so it's more about the distance.

Okay, I'm going to detour slightly. It actually is relevant here. It's in a part of the Midwest known as the Driftless Area. That's sometimes just called driftless. They could put different words after driftless, but driftless is the key word. It was unglaciated during the last ice age. So, unlike what you'd normally think about as the Great Plains, which are made by glaciers scouring everything flat, it is not scoured flat, and it is actually extremely hilly, not mountains, but exceptionally hilly [laughs].

And this high point is actually just, like, a half mile from the Wisconsin state line. Wisconsin has this as well, this very hilly area. And there's rivers that cut deep canyons in between high hills on both sides, hundreds of feet high.

And to get there [chuckles], I had to wind away through back roads through this hilly area, which means that there were a lot of turns. And if you ever tried using Google Maps, it only lets you get to, like, 10 steps along the way, and then it just bails. Like, nope, that's all you can do. And so [chuckles], that wasn't going to work. I actually did it in sections, and I carefully wrote down the turns and the distances between each one of them in a document. And I built this all out in just a text document because I'm a software engineer. That's what I do; I write it out in a text document.

And [chuckles] I got through it. And there was one point there was an alternate. I wasn't sure if I was going to be able to go [inaudible 03:39] because it was just, like, a farm road, literally through a farm. But there were no signs posted, and Google said to go that way, and it was actually in Google Maps with Street View. I'm like, okay, well, I'm going to do it, and I did. It was beautiful, by the way [laughs], and it was fine. Nobody yelled at me. I wasn't sure [inaudible 03:55] way.

So, you know, it was a text file, but there was some weird stuff in it. I got through it and said, okay, so how many miles is this? And [chuckles] what am I going to do? Hey, I will just write some code, so I did. I pulled up a console, and I had my answer in, like, three or four minutes. I had to do a little bit of cleaning to sanitize some of it, remove some of the stuff, and put it all together.

And it reminded me just how useful code can be. It allows you to do amazing things to solve problems. This was something far afield of normal software engineering work, and it still ended up [inaudible 04:33]. I'm going to use code as a tool here, and it allowed me to solve the problem quickly. If I had tried to add that up by hand on a calculator, I don't want to think about that. Maybe I could use a spreadsheet, right [chuckles]? But then I would have to deal with all the edge cases in there because there's the alternate routes that would have been a pain. Pulled it off quickly just by writing some code. It's an amazing tool that helps us in many things.

And today, we're going to be talking about our side projects. Normally, we talk about our day job, but we thought today we'd dig a little bit into the things we've been working on outside of work and how that's going. And it gives kind of some context to our day-to-day because these side projects tend to be weird [chuckles]. They’re the miles that you get to a high point. They just are way different than your normal work, which makes them fun. It breaks up the monotony. There's my story. And I've got a couple of other things I've worked on recently that I may bring up, but I'm going to start with that. Dave, over to you.

DAVE: Well, I don't want to follow up story time with Uncle Mike with story time with Uncle Dave, but here we are.

MIKE: [laughs]

DAVE: So, I'm constantly writing stuff. My career hack for people is, if you have the time, which most people you got to decide. We all get 24 hours a day. But I no-lifed programming for most of my 20s and 30s. I would go to work. I would work all day. I would go home. I would keep writing code. And insert story about mental health here. Work-life balance was something for other people to do.

And what I found is that there's not a problem...there's not an interesting question that I can come up with that I usually can't immediately think, I wonder if I could solve that with a computer? Off the top of my head, without going into full-on stories, although I might tell the clicker story because we can talk about AI for research as part of writing custom stuff.

But in the last 30 days, I have worked on a planner sheet thing that literally people listening at home won't be able to see this. But if you've ever looked like a Franklin-type, you know, day planners, that kind of thing. So, I figured out how to make a program, draw it out, fill in all the dates and times, and send it off to the printer. I built an auto clicker for Minecraft because I was AFKing and needed resources.

The other weird one, and this is harder than it sounds, or no, this is exactly as interesting as it sounds, is trying to calculate the cost of a recipe in Minecraft. If you want to build sticks, right, you need two boards to build four sticks. But to get two boards, you need one log, and that makes four, and there's, like, surplus numbers. And if you want to build, like, some of these late game, really complicated things, you can have these build palettes that are, you know, with thousands of blocks that you have to know what you need.

And normally, we just stand next to the crafting table, next to your inventory, and just pull what you need and build the next piece. And you just keep, do I have what I need? And on the other times, sometimes I'm like, no, I want to know how many, you know, how many chest full of, you know, cracked bricks do I need to have ready to go before I start this?

And being able to say, “I want three of these, but it's made up of this many of that,” you can see very quickly, like, this nested hash of, like, the dependencies to make this item requires these things. And they require this, and they require this. And it's just algebra, but it's tedious, and that's what computers are great at. Computers love tedium.

And there's half a dozen other things that I've written. Some of them are work-related. Like, I will...if I do a thing in git more than three times in a day, I'll start thinking about, do I want to just wrap a script around this, or do I want to keep this memorized? So, I'm constantly tinkering with things. Games as well.

I think you were saying...you said Sudoku puzzle. Did you mean programming is like a Sudoku, or do you mean writing programs to solve Sudokus? Because that was a very fun thing that I did about 15 years ago. James Edward Gray used to do, oh, it was to sit down and write some code in an hour. I'm blanking on it, Ruby [inaudible 08:46]? I'm blanking on the name.

But he would do this thing every month, where it's like, here's a programming challenge. It should take you about an hour. And son of a gun, he said, “Write a program to solve a Sudoku. It should take you about an hour.” And I'm like, what? That's nutso. And now I might be able to, but back then, it took me, like, two days to write the thing. And--

MIKE: So --

DAVE: Yeah. Sorry, go ahead.

MIKE: Sudoku comes in multiple flavors. The simple case, I did this probably more years ago than that. Simple case is easy, but there are some of them that require backtracking, where it's not fully specified. So, you have to start branching, and it gets a lot more complex.

DAVE: I have written a recursive solving solution that I now have a repo on GitHub just called Game Players, not Game Playing, but Game Players. And it's programmed to play other people's programs. And the pattern emerged very, very quickly of, here is the board. Here's the thing in it. What are the legal moves? Can I make one? What state does the board go into? Now recurse, and keep going until we...if you can exhaust all the moves out of the board, then great.

WILL: I see, like, I don't know whether I buy that sort of you need to backtrack problem. How I always solved Sudokus, in my general case, was always a little bit of a dynamic programming approach in that I would say, all right, here is the...so, I would create a board, create the blank board.

And then I would say, okay, so, for every cell, I've got a set from 0 to 10 that could be here, and then I would just sort of start adding things in. And then that would populate the board, and then I would eliminate the set. I'd be like, okay, nobody else can be a 1 on this row. Nobody else can be a thing. And then I would just finish that, populate that whole thing. And then I would go through the things until I found a...what do you call it? Something that there would only be one thing, and I’m going to put that in. I wouldn't even need to backtrack because like --

DAVE: What do you do if you put a 1 in the top-left corner, and then build out the rest of the board and find out that the 1 isn't correct there, and you have to go back and start over with a 2?

WILL: But that's not possible. I would never put...so, it's always a matter of...it's always a matter of you never make an illegal move.

DAVE: Oh, okay. You're writing a thing that actually does the deduction. Yeah.

WILL: Yeah.

DAVE: Mine was just a brute-force solver, just throwing numbers at it until something drops out.

WILL: That's probably faster to write, you know.

DAVE: The deduction would probably be a lot more fun to write though because then you'd get to understand what are the rules to it. What are the second-order implications?

WILL: So, you just muscled it through, huh?

DAVE: Yeah. And once you start seeing every problem looks like recursion, then everything is a smashed thumb. That's a terribly mangled metaphor, but yeah. When the only tool you have is a hammer, you're going to smash your thumb a lot. And so, yeah, I will approach things in terms of can I make a board out of it? Can I determine what the legal moves are? And does the legal moves reduce the solution set? If those three things are true, that is the definition for recursion will work here.

And in late game, I started reaching for recursion first because I had the board set up. I knew all I have to do is fill in this method, and give this a loader and a saver, and I'm good to go. And down the road you go, and it's crazy, so...

WILL: Interesting.

DAVE: But yeah, deducing is also a good one to do as well.

WILL: It seems more efficient. I don't know. I'm not an animal, just, like, hit it with a rock until it stops moving. All right. That's...okay. I went to school for this. I'm not going to like -- [laughs].

DAVE: I dropped out of school for this.

WILL: Yeah, well [laughs]. Interesting. Interesting. Sorry, I guess it's, like, you know what I mean? With, like, grad school, I spent my time working on sort of deductive decision trees for other things. So, that's just sort of, like, a...oh, well, that’s obviously what you would do [laughter], you know? Anyway, so yeah, regardless.

MIKE: Well, there's some sort of tree in that Sudoku solving at work. We're just talking about interesting stuff here, right? But there's...where it's not fully specified, right? Where there is..it's I am ambiguous which branch you have to take. Well, there's going to be some sort of tree structure, whether you're doing dynamic programming. There's different ways of representing that in terms of data structure. But conceptually, there has to be branching of some sort.

WILL: I don't think I’ve ran into that problem. I don't think I’ve ran into a situation where, if I had examined all possible scenarios, there was one where there were two legal moves.

MIKE: Yeah. The worst Sudoku puzzles have that. And that's where it gets...and, suddenly, it's way harder. It's not that hard to write a solver for a puzzle that's a beginner puzzle. But then they start to have those where there's multiple legal moves, and one of them is going to be a dead end.

WILL: Interesting. Interesting. I have to think about that a little bit. I have to think about that a little bit more than I have.

DAVE: Arguably, it's the same solver if you combine the two steps, where the first step is, is there a move that I know I can do? Because why would I guess if I can deduce what the correct move is? And then if you run out of...if you don't have a deducible move, then it's time to start guessing and putting...yeah.

WILL: Oh, that’s fair.

MIKE: Then you get your tree, right? You're going to have to branch down some direction, and sometimes it's going to be a dead end. And dynamic programming can address that, right, if you structure it right because you end up just taking the path that works, but you're kind of going down every one. I'd have to think about exactly how to set that up.

WILL: Well, I mean, basically, you represent the Sudoku as, like, a three-dimensional array, I guess I'd say, right? Like a two by two, you know, by three in that, like, okay, so I'm going to say, oh yeah, I'm going to...Each cell has a set, right, of possible legal moves, right? And so, you iterate down until that set comes to a fixed point, right?

And then if you have a situation where it's like, okay, well, the best I can do is this one has a set of three moves that I could do, right? Then you iterate, and you say, okay, I'm going to say it's one. I'm going to put a one in there, and then I'm going to go through. And then you continue to iterate, iterate, iterate, until you get to a situation where one of those cells has a null set, right, where there's no legal moves.

Like, that's where you know, like, oh, I screwed up. This is not solvable, right? And then you backtrack, you know, until you get to that iterative point. It's like, okay, well, I'll try and put a two in, and then you'll go through. And then like, you know, oh, null set. I failed. And you'll go back, and you'll say, okay, I'm going to try and put a three in. And it will either work, or you'll get a null set, and then the Sudoku puzzle in itself is --

MIKE: Is invalid, right [chuckles].

WILL: Is invalid, right?

MIKE: And interestingly, you're talking about two broad approaches. So, Dave, you’re like, oh, I'm going to do recursion, and you're going to come up with an iterative approach. Two pathways that diverge but end up in the same solution.

DAVE: Awesome. Years and years ago, I was goofing around with somebody, and I'm like, “I need to drop this table.” And I wrote drop table, and I hit Enter. And somebody responded with, “You dropped the table. There is a table here,” like the old Zork commands, like the old text adventures where you pick up the sword and take the lantern and go explore the cave.

And we giggled about this, and I joked, and I said, “You know, PSQL or PL/SQL is Turing-complete. You should be able to write an adventure game in pure SQL.” And everyone was like, “No, it can't be done.” I'm like...So, I wrote one to win a bet, and it was just fun, and stupid, and silly. And I lost the source code. It was like 2006. I lost the source code to it, and I've tried to recreate it a couple of times, and I just haven't gone off it.

And I was thinking about it a week ago, and somebody said, “You know, there was a guy that ported Doom to Postgres, right?” And I'm like, “No.” And it's for real. Somebody figured out how to get it...It's text. The graphics are terrible. It's ASCII art. But, apparently, it's even got a decent frame rate, and it's a first-person shooter in Postgres in PSQL.

MIKE: Wow [laughs]. I'm not quite sure what to say about that. We're talking about solving problems that don't necessarily need to be solved.

DAVE: And solving them in interesting ways, which then makes it so when I'm hiring somebody, I will ask them, “What are your hobbies and your interests?” And if I get somebody who's weird like this, if I'm at a...I've never hired from an enterprise position where I need somebody who's stable and just a metronome; you’re going to clock in at 8, clock out at 5, get 1,000 lines of code, da-da-da. That's not a good person for that.

But if you've got dragons and you're in startup mentality and you need weird solutions because you have weird problems, I love finding that person. Not because they know how to write Doom in Postgres, but because they know how to look at a tool and go, I wonder if I could turn that around backwards and use it this way and find a completely new way to use it?

MIKE: It speaks to curiosity and play, which are characteristics of somebody who can solve problems.

DAVE: Also, resilience, interestingly.

MIKE: Interesting.

WILL: I like that, right? I like that. And I like to play, and I like projects which reflect sort of, like, tinkerers. But I'm curious if you guys have a perspective on, like, most of my job is just reading other people's stuff and figuring out what the hell you people do. What do you people do?

DAVE: Code forensics.

WILL: There's very little Greenfield stuff. And so, what are good projects that demonstrate...like, honestly, the pivotal requirements around enterprise software development, where you're just sort of, like, groveling through the context, legacy codebases decisions that go years from people who've gone. And they've left their footprints in the fossilized clay, and you have to backtrack through what happened. Which is just a lot of reading code and a lot of patience and grinding through things, right?

I mean, because, on one hand, like, yeah, that's the job. But on the other hand, if you're doing this in your free time, you're a psycho, you know what I mean? Like, what? Was it cold outside so you couldn't catch any small animals to torture and kill them, like, you know what I mean?

[laughter]

DAVE: I can neither confirm nor deny, but if you want to look in my freezer, you're going to need a search warrant.

[laughter]

MIKE: There's a thing online where people will post a picture, and people will try to figure out where it was taken from.

DAVE: Like GeoGuessr?

WILL: Those guys are also psychos or, like, undiagnosed autists.

MIKE: Well, it's relevant here. I heard about...I don’t remember the exact story. This is a few years ago. There was somebody who was lost in the mountains. And his phone was dying, and he managed to post a photo from his location. And his phone died. And whoever got the photo reached out to people online and said, “You know, my loved one's lost in the mountains. What do I do?” And this community of people who locate where these pictures are taken from, found it, and figured out exactly where this person had gotten lost, and that's where they sent the search party.

DAVE: That’s fantastic.

WILL: Wow. Did they find him?

MIKE: They did. They found him. And I believe it was a him, and he lived because of these people.

WILL: It’s always a him [laughter].

MIKE: Off trail, he'd, like, gone down a cliff or something, so he was not where he should have been. But they found him anyway because this photo got posted. But that, you know, talking about that skill, I feel like that specific skill is the one that ends up becoming the most useful for senior engineers who are in the code. What is going on here? Where am I? What is the context, and how do I get here? What's here around me? And, you know, it's in the code. It's not out in the world. But it's that kind of thinking.

DAVE: 100%. 100%. I call it code forensics, where you start reading, and it's text on a page or on a screen in a fixed font, but you can still read other people's handwriting. You can absolutely tell, oh, this was written by that person who likes this particular pattern and likes to name their variables this way. And that means that when they go to solve this thing, they're not going to use this pattern. They're going to use this one. So, I'm not even going to look in this other sort of...or I'll look in the other place, but not first.

I play a lot of Chesterton's Fence with myself mentally. It's like Chesterton Fence, you find a fence in the woods, and it's in your way. Should you get rid of it without knowing who put it there and why? That's the thing. Some people very quickly are like, no, I'm efficient. We need to get this out of the way and get moving. And other people are like, somebody had a reason to put this here, and they went to great effort to do it. Let's find out why.

So, when I see something in code that doesn't make sense, I don't immediately assume that it didn't make sense to the person who wrote it. I start stepping back and going, okay, if you wrote this on purpose, what situation would you be in? And you end up spending a lot of time, like, playing with, like, git blame, looking up old PRs, and trying to see, okay, what problem were you trying to solve?

And I like it when I find some new thing in the code that's like, oh, do I have to follow this pattern? And then I'll git-blame it, and I'll find out that it was, you know, written by a developer in Poland in 2017 and hasn't been touched since. And I'm like, okay, either this code is rock solid or abandoned. And I've never seen this module come up in our Rollbar log. So, I'm pretty sure it's abandoned, but it might not have been. It might have been written two weeks ago by one of our contractors. And so, that greatly affects whether I will make the code match my style or whether I'll make my style match the code.

MIKE: So, we’ve dug a little bit on that playfulness and the skills required to deal with the kind of software, which are not, like you say, Will, they're not that greenfield kind of development very often, you know, it's new on day one and never new again [laughs]. And even after that, you're building within some framework, some context that you have to be aware of.

WILL: I mean, I enjoy... So, I was thinking about this a little bit, right, as you guys were talking about it, like things you’re working on right now. Like, so, I'll be honest, right, like, right now, I am, like, three or four months into, like, a new role, you know, with a new codebase, new tech stack, like, new everything, right? And I'm not coding nothing outside of work. Nothing. Nothing.

And I shouldn't be coding nothing, and, like, you know what I mean? And you shouldn't feel bad about coding nothing because, like, I'm drinking from a firehose right now. I'm, like, running. I'm still, you know what I mean? I can get things done, right? I'm clearing tickets. I'm getting things done. Like, there's all kinds of technology. Like, I'm in meetings, like, all the time.

And I encourage everybody to do this; I'm just, like, writing, like, little notes to myself, like, what is this? You know, like, and it's documented. You can ask people and read about it and just get things going, right? Not that hard, but I'm constantly just, like...And when I get done, you know what I mean, the [inaudible 25:09] free time that I have where I have, like, the ability to, you know, work on stuff, like, I'm not working on code.

And so, these projects you need to be working on don't have to be, you know, [inaudible 25:20] related because, like, I get a lot of inspiration anymore these days about, like, sort of what I'm doing professionally from things outside of work, you know? Because I’m pretty technically solid, you know?

I know my work, and I need to, you know, how does this API work? How does this library work? How does this, you know, third-party service, like, interface with our third-party service? Like, I have things to do. But, like, you could derive a lot of engineering work, and especially when you start getting into, like, interpersonal work and leadership work, and dealing with people. You can learn a lot about management from having kids.

MIKE: Amen.

WILL: You can learn a lot of stuff from reading fiction. Learn another language. Make something with your hands. Make something with your hands that does something, right? Like, just carve it out of wood, chop up wood. Build a deck, you know. There's so many things in life.

And get a good night's sleep, if you find yourself at work, where it’s just like, man, I’m just frazzled all the time, and I feel stupid. If you would like another 3, 20 IQ key points, just 3, get 8 hours of sleep, dum-dum. Yeah, it's just like, oh, today would be a lot cooler if I was 10% smarter, 8 hours of sleep, dummy, you know? Like, don't feel like, I suppose, like, you know, like, we can't all be Dave Bradys who are just, you know, programming freaks all the time. Like, you've got a life, and, like, you should live it, and coming in with a mindset, you know, because there's seasons, right?

Because you'll also find a season where you know everything, and you know everybody, like, every burial ground, every, like, you know, poorly covered, shallow grave, and/or mass grave, you know, in the company, and you know where things are. And you find yourself in a rut and rot, and everything's just boring. I’m just doing the same thing over and over and over again. Well, then it's time to start tinkering. Then it's time to start investing in things.

But like, you know, when you're drinking from the firehose, which we will spend at least half of our time just completely drowning...and do something else. Read a book where there is not a robot to be found anywhere.

MIKE: [laughs]

WILL: Not a [inaudible 27:47] robot. Not a one. You know? Like, write poetry or go geocaching. Do something else. It feels like you're not progressing, but you absolutely are. Doom scrolling on social media probably I'm not going to count that one.

DAVE: Not as good, not as good.

WILL: But you could watch a Ted Talk, you know. Anyway.

DAVE: Learn music would be the other thing. I messed up my elbow, and I could no longer fret. I'd been taking guitar lessons, and I could no longer fret a sixth string. And somebody said, “Why don't you play a cigar box guitar?” And I'm like, that's weird. And to shorten a very, very long story, somebody said, “They're even more fun to build than they are to play.” And I'm like, that's not [chuckles] even remotely possible. And then I ended up building...this is number five. I've built one and a half more since then. So, I'm working on seven right now. And they're ridiculously fun to play.

MIKE: My first boss I had as a full-time software engineer, had a rose garden at home. And I still think that was relevant. And I don't always talk to everybody and know all they do. But a recent person I've worked with does counseling with people who are getting married on the side, right? And does software during the day and this very different thing at night.

I'm building on what you said there, Will and Dave, that it really does matter. Having that other thing improves your ability to do your primary thing because it gives you new ideas.

DAVE: Drastically. Absolutely.

MIKE: I led by talking about that bike ride. I was thinking about this directly relates to software because...very indirectly relates to software. But figuring out how to make that work if you're going to go a long distance is actually challenging. Getting enough sugar that you don't run out of sugar and then have a, not a fall-off-your-bike crash, but a physiological crash where you just can't move anymore [chuckles] is a real problem. And if you don't deal with that, you're not going to be able to do it.

Figuring out your logistics, you know, you need water. And there's challenges in those things that you have to work out. And you have to be very detailed and meticulous about it. And developing those skills pays dividends in the daily job, for sure.

Honestly, my life is, I have not done a lot of coding projects outside of work recently for some of the same reasons. Probably one of the larger ones I did is I redid the...I made an alternate intro music for the podcast using the project we talked about, like, two years ago. It's called Sonic Pi, where you use Ruby to write music. And that's the intro and outro music that we're using sometimes. Based on the original one we had, just mixing it up a little bit. And we said we'd do that, and we did.

It really didn't have anything to do with the job at all [laughs]. It let me try something different, look at things from a different perspective.

DAVE: This is very cool. I'm looking at the Sonic Pi website right now. So, you guys finish the episode. I'm going to go look at some music code.

MIKE: Until next time on the Acima Development Podcast.

The panel digs into the perennial question: how much SQL should developers know? Kicking off with a war story, Mike recounts a hyper-growth phase where ~20 performance issues were fixed—almost all by database changes, especially adding (or rethinking) indexes—yielding order-of-magnitude speedups. The moral: ORMs are great for safety and productivity, but when things get slow, it’s “usually the database,” and knowing how indexes, JOINs, and query patterns work is what unblocks teams. Will adds a blunt rule of thumb: apps are slow because of “bytes on the wire” or “the database,” and you can’t rely on ORMs alone to prevent N+1s or inefficient access patterns.

From Ops, Kyle reinforces that troubleshooting still lands on SQL: monitoring tools can point to hot spots, but root-causing and backups often require direct database savvy. Eddy shares a counterexample: moving search from Elasticsearch to Postgres full-text revealed that a GIN index on a high-churn table actually slowed writes—illustrating the trade-off that heavier indexing speeds reads but taxes inserts/updates. The group also debates concurrency: for most web apps, you can push work “down the stack” to the database and avoid complex threading; true low-latency, hard real-time concurrency is rarer than many think.

Stepping back, the crew frames SQL as a declarative, optimization-friendly paradigm—closer to functional transforms than procedural loops—which is precisely why database engines can do so much heavy lifting. Resources like SICP and Lisp/Scheme are recommended for learning to “think in streams” and transformations. The consensus: programming languages and frameworks come and go, but SQL endures—and senior engineers still write ad-hoc queries daily to answer business questions and debug production. They close with a teaser for next time: a lively debate about testing private methods and how to design for testability without committing “crimes” against your runtime.

Transcript:

DAVID: Hello, and welcome to the Acima Development Podcast. I'm David Brady, your host today. And we've got a fun panel today. We've got Kyle; we've got Eddy; we've got Mike; we've got Will, and we've got Matt Hardy. Welcome, Matt. Nice to see some new people, well, not new, recurring people. We don't have anybody new, new, new, but nice to see the regulars.

Today we're going to talk a little bit about SQL. Like, do developers need to know it, and if so, how much? And this came as a suggestion from Kyle, who works more in ops rather than frontline web dev. And so, I'm very interested in hearing where he wants to go with this. But first, as our tradition, let's start with some story time with Uncle Mike. Mike, what do you know about SQL?

MIKE: [laughs] So, I do have a story, and I was prepared for this. A few years ago, I say a few, at least five years ago, maybe five or six years ago, I was in a big application that was growing up [chuckles]. We were actually getting a lot of traffic where we hadn't been. It's the startup journey, right? And this actually was at Acima, you know, as we were in our rapid growth era, not that we're not still growing, but, you know, back in the startup days.

And it's natural for every application, once you start going through that growth period, like, oh, wait, there's some things that don't work very well, let me say, that don't perform very well, that don't perform very well. There's things that work just fine when you had a data set of a thousand that don't work very well when you've got a data set of a million [laughs]. Things just don't work quite the same anymore.

And so, I went through with the team, and we worked on that for maybe a couple of months. And over that couple of months, we made the application run about 10 times faster, which is a big deal, right [laughs]? That's a major performance impact. I think it was more than 10 times. I mean, it was a big deal. You know, going to 10 times faster has a big effect on your infrastructure, and I think it even went even beyond that.

But I kind of quit keeping track after a while [chuckles] because it's, you know, you start resetting your baseline. Okay, well, we got to go faster than that, got to go faster than that. It’s not the only time I've done this, right? It's just the most recent time that I have fresh in mind.

And one thing we found is...let me say a couple of things. Every single one of the performance improvements we made, except for one, were database improvements, and most of them were adding missing indexes. So, there's a database table missing an index, and, by the way, it's always a missing index. If you have a performance problem, it's a missing index [laughs]; I'm just going to say that, except there was a couple of them that weren't. A couple of them were some kind of weird cases.

There was some really gnarly JOINs. It was running, like, JOIN against 10 things using a lot of LIKE queries. We fixed that one by using an external index, using a reverse, one of those external reverse indexes. It was Elasticsearch at the time. And that made a tremendous difference as well in our database load.

And there was one that was not a database issue, you know, so we fixed, like, 20 issues, one of them wasn't. And it was because there was something that should have been in the database [laughs] but wasn't. There were some permissions that were being loaded really strangely from flat files and weren't being cached, and every request was reprocessing them. And by putting a little bit of caching and changing the processing, we took that from taking, like, half of every request time down to unmeasurable, right? It just went essentially to zero. It was great, you know, app went way faster. Everybody's happy. Everything's wonderful [laughs]. And we work on other problems.

The lesson I learned from this, though, is that, yeah, it's always the database. And here's the critical thing: adding a database index is something that's really, really easy to miss if you don't know something about how databases work. And most of the time, you don't have to think about it at all, except for that time that you do, and it's the thing that matters most. I found that throughout my career, that yeah, it's always a database index [laughs] or something to do with the database because you don't have to think about it except for that time that you do. And if you don't know that time, then you're burned. And somebody who can come in and can fix that can make a tremendous difference in your application.

Oh, there's the story that I was thinking about as we talked about this topic.

DAVID: Awesome.

MIKE: Do you have to know SQL? Well, I come from an older generation, right, where I cut my teeth writing raw SQL. And I thought it was kind of amazing when we started getting ORMs that would do some of that automatically. There's a whole lot of advantages to that. So, you avoid a lot of SQL injection flaws. You probably get better queries most of the time. So, overall, the switch that most applications have made to using an Object-Relational Mapper (ORM) is a great one. But that experience that we got when we had to do it manually is still incredibly useful. And I worry a little bit that the new devs coming in now are not getting that experience that you usually don't need, except for that time that you do.

WILL: There's only two reasons that your app is slow: bytes on the wire, or your database sucks. That's it.

MIKE: Right [chuckles].

WILL: And if your API is bad, it's just because their database sucks. That's how it works, you know. And, I don't know, I love an ORM. ORMs are great. ORMs are great, and I use them all the time. And, like, 99% of the time, it does it every time. But that's why we make our money. That's why AI is not going to come and get me this year. It's because sometimes...if the junior devs could do it, they would have done it, and it wouldn't be on my desk. And, I don't know, like, you've got to run some gnarly SQL.

I'd also say two things: one, it's not like regular programming. It doesn't follow the same rules. It doesn't work the same way. It doesn't work the same way. SQL, if you're writing a for loop in SQL, you’re f***** up. My one F bomb for the podcast. Let me take it early.

It doesn't work like that, right? And it's not that it's inefficient, but it's just it takes a foundationally different logic to it, and it's everywhere. My PM he was having terrible troubles in Jira this week because our team and all of our Jira stories were getting communicated up to management that was saying, like, “Thumbs up, thumbs down,” Roman emperor style.

And this Jira query was screwed up, and I'm like, I don't know JQL, but I know JQL, and let's face it. And I just got in there, and I fixed it. And it's like, it's everywhere. It's everywhere. It's a foundational programming paradigm that is in everything. You want to do some GraphQL? You should know SQL. It goes in everything. There’s all these sort of, like, abstract layer data management things, and they're all heavily derived from SQL because they're solving the same kinds of problems.

MATT: You should know what's under the hood if you're working on the car. Simple as that. And it's really easy, with an ORM, to get in trouble and write a bunch of N+1s if you don't understand what's going on and what's happening under the hood.

WILL: Absolutely. Absolutely. Like, if somebody asked me and they came in and they were like, “Hey, listen, I have time to learn one thing this year. I could do concurrency, or I could do SQL.” I'd be like, “Just save yourself some time and learn SQL.”

I believe, sincerely, that we're pretty close to the level in programming where most developers will never need to know anything about a thread, not really, you know? You've got an async/await. You can run some coroutines, just kind of spawn a job queue, which you can do, not knowing anything about real concurrency. And you can live and die a productive life without having to wrestle that pig. SQL is not that way, though.

MIKE: Agreed. I may have told it before, but the gnarliest problem I ever worked on as a professional developer was a concurrency problem, and it probably shouldn't have been [laughs]. You get yourselves in all kinds of trouble there. And suffice it to say that there was multiple bugs in the library I was using [laughs], and I had to fix them. And solving a problem that is both concurrent and distributed is hard [laughs], especially with problems in the library.

MATT: Yeah, and few applications require true concurrency.

MIKE: Exactly. If that concurrency had never been in there, it would have worked better [laughs]. And it was the wrong solution for the problem. But I use SQL, well, all the time. This morning, somebody said, “How do I get this data?” and I wrote a query for them.

MATT: I mean, here at the company, we have hundreds of services, and I think one or two actually require real concurrency.

WILL: Yeah, I mean, I'd be surprised, like, what services do you have that require real concurrency, like, a web app? Like --

MATT: Underwriting-related things.

WILL: What's that?

MATT: Underwriting-related things.

MIKE: Lots of parallel requests going out to third parties.

WILL: Still skeptical, but I don't know enough about it to have, like, a strong opinion on it. I mean, because, I mean, I'll tell you, like, honestly, like, really how, like, how things work for really real is, like, all that concurrency, like, modern web apps are concurrent as hell, super concurrent. But you know who does it all? SQL [laughs]. Like, oh man, I got all these parallel requests that I need for, like, high-performance web apps. I’m like, oh man, you don’t need all that. Just kick it down to the database. Let them sort it out.

MATT: Yeah, I mean, it's necessary when you need responses from 12 different third parties at the same time, right?

WILL: Nah, [inaudible 11:19] can wait. You just wait them all. Wait on all of them. You don’t need a semaphore for that. That’s like...when I say, like --

MATT: I would argue, Will, that SLAs say differently.

WILL: Naah. No, no way. Not a chance. Not a chance. Like, your performance isn't that good. Like, if I’m going to network stack, my performance...I don't want to say it like thaaat. But, like, so I mean, for background, right, when I'm talking about, like, real serious concurrency, like, what I refer to specifically in my work, right, is that you work on the radio stack, not TCP, not IP, like, bits and bytes on the wire but wired less, right? Like, working the radio.

I also do a lot of, like, real-time audio programming, and that stuff is serious concurrency. And there's other stuff. It’s like async/await. It's okay. I'm going to fire it off on async, and I'm going to wait for it to come back. We'll, you know, join all my promises together, so I have this big chunk. But, I mean, like, milliseconds, like, a millisecond is an eternity, you know, and, like, you know, on the network, a millisecond is almost not even measurable, you know. Anyway, maybe there's something exotic, and if you're not doing, like, high-frequency trading, like, I don't know, probably the technical detail’s not something that's appropriate to get in here anyway.

DAVID: There's a CS professor at the University of Utah, can't remember his name, but the best quote...he was an audio CS guy in the ‘90s back when, you know, a 486 was all you had and, you know, at 33 megahertz grinding away.

WILL: [inaudible 13:06]

DAVID: And a friend of mine was in his audio processing class, and efficiency was the buzzword. And he finally says, “So, I have to do this before the next stroke of, like, sending bits to the sound card.” And the professor goes...I'm sorry, the professor wasn't in audio. He did some audio, and he didn't realize he was talking with my friend about audio. And he looks at my friend, and he goes, “Wait, you're doing audio? You'll have milliseconds,” and walked away. Literally, that is a lot of how concurrency is today, right? It's like, I got a four gigahertz processor. I can chew this elephant from front to tail serially and just take a blocking operation, and it's two, you know, two milliseconds or whatever, and you get through it.

I've had one app where concurrency was critical, and it's anywhere you need to be handling multiple threads of attention live or giving the appearance of live. I wrote an IRC client, so you have to catch keystrokes coming in from the user before they hit enter, so you can't do an input read statement. You have to stroke, stroke, stroke, stroke, and update that. Meanwhile, the server is throwing network bytes on demand at you over the fence, and you can't re-request them, or it's like, you have to handle those network packets when they come in. And that was the time when we had to interleave.

MATT: You just gave me flashbacks.

WILL: Yeah, the battle days.

MATT: I've written a few IRC clients back in the day.

DAVID: Getting off topic, but IRC is what broke me of Python. I was in love with Python. I'd come from Perl, which is super cryptic, and messy, and dense, came into Python, which was clean and beautiful. And you cannot condense it down to a single line. And when you are scripting an IRC client, you have one line to do everything, and that's what pushed me...It pushed me back to Perl and then forward to Ruby.

WILL: Interesting. I don't know, here there be dragons. Don't mess with concurrency. Don’t do that.

MIKE: Exactly. That's the moral of the story.

DAVID: Touching back, though, a little bit, there are times when you have a high-level framework, and it starts acting weird, or you change a feature, and something completely unrelated changes. And if you are very familiar with, like, SQL and how database engines work, that's when you go, oh, we're reading the leases table on a new column. I wonder if it's not indexed because that would slow everything down in this chain, right?

And it's, yes, we do...I'm old enough that I was in the generation of people that had the math teacher that said, “You won't always have a calculator with you. You need to learn your times tables.” Well, I have an iPhone. Well, I have an Android. I've got a computer in my pocket now. You were wrong about that, Mr. Nelson.

WILL: No, he wasn’t.

DAVID: SQL might be headed that way, but right now, you don't always want to pull out your phone to be a calculator. And when you can just look at a website and go, that's misbehaving in this specific way, and I can see through the ORM to the underlying technology, it’s super powerful.

MATT: Well, it provides value.

WILL: Well, Matt brought this up, and I think that nailed it. I mean, in that, like, it's really easy if you're in an ORM to write N+1 queries, right? To write a double-nested loop instead of, like, just getting the whole enchilada.

And I think I would liken it to memory management in that, like, nobody, I mean, much like, you know, much like concurrency, like, nobody's malloc-ing anything if they have any sense in the year of our Lord 2025. But you still need to know how it works because you can definitely leak.

MIKE: It's not that hard to do. Just make a global hash in memory, and keep it persistent.

WILL: Yep. Yeah, man, I have a ticket in my backlog right now, which is, like, go leak hunting. You’ve got to dump that heap. Kyle and I we've dumped a heap or two, you and I [laughs].

DAVID: Actually, that reminds me of something else. We said at the top of the call, Kyle, you live over in Ops land. What are you doing talking to databases?

KYLE: What am I doing? A lot of it has to do with we manage our RDS instances, and it's a lot of the time troubleshooting or backups. Those are the bigger issues. That's kind of where I thought about this topic is because in prior organizations, I mean, older codebases, older companies, they didn't use ORMs. A lot of it was just straight SQL. And so, everybody that I spoke to they knew and understood SQL.

And one drawback that I have seen with ORMs, while they are great, is when you need to get in and troubleshoot and dig around, you need to find the engineers on your team that actually know SQL, because not all of them actually know SQL, in order to help troubleshoot with you, right? There are tools out there that kind of bridge some of the gaps and will identify, oh, you don't have an index or something on your table, which we utilize. We've got New Relics and Data Dogs of the world that kind of help us identify some of those things. It gives us benefits in some areas.

In other areas...I think somebody said, 99% of the time, it's great. And then you run into the 1%, and then you run into an engineer that is not very versed in SQL. Not to say that I'm great with it either, but it does slow down troubleshooting. It does slow down figuring out the root cause of the database problem. Because, like Will pointed out, if your app is slow, the first thing I'm going to is the database. Is something happening with the database, right? Because that's where most of the problems come from. If it’s not the database, it's something like Redis, another pseudo database, or --

MIKE: Another database.

KYLE: Another cache location, right? It's always in these locations. And ORMs make it really easy to develop. They make it a little bit more difficult when you need to troubleshoot. And I won't even say just ORMs, ODMs, too, anything that's doing it for you or helping you without actually teaching you the underlying layer. But, yeah, that's kind of where it is for me.

And even some of the toolings for doing backups, when it comes down to it, we've chosen to do backups with straight SQL. Because this is speaking outside of the ORM, but some of the tooling we have that we've tried it's not as fast, and we've had to optimize with straight SQL commands.

WILL: Well, I mean, I say it's a clutch move that I've run into many, many, many times. It's like, when you push the logic down a layer, if it's things that the database thinks about very efficiently, right, you can push things down to the database because the database has all of these internal optimizations and tools, and it doesn't have to go back and forth. You could skip steps.

And if you can have, I mean, obviously, there's caveats there, but if you can have the thinking happen on the database, you're looking at 10x improvements because databases are optimized to an incredible degree, internally, internally for the things that they think about, right? You would never put business logic in there, but in terms of, like, munging data, it's so much better than, God help you, Ruby [laughs].

MIKE: Well, you say 10x. 10x, that is super pessimistic. Because just making that network call, the moment you make that network call, and we’ve talked about this before, it dominates every other thing by far in that request. Because making that network call, you're going from milliseconds or microseconds to, like, seconds sometimes, right? You can't define that.

Now, it's probably milliseconds, right? But it might be 30 milliseconds or 50 milliseconds, whereas if you're running that thing in memory, we're talking less than a millisecond significantly. And it may be your database. Maybe it's a really hairy, gnarly query, and maybe it'll take two milliseconds in your database. That is still wildly faster than bringing it back into memory on your side, doing everything, and then making a request for every one of those pieces, creating a new network request back to the database. If you are only getting a 10x improvement, you might have done something wrong.

KYLE: I worked with an engineer a while back on card processing. And they needed to do the card processing in a specific time window. And it was one of those things where they really wanted to blame the database, and they really wanted to narrow it down to that. And I got in there, and I looked, and we're sending, I can't remember at the time, but it was hundreds of thousands of requests to this database.

And I got looking, and the highest turnaround time for any one of those requests was about two milliseconds. And it's just like, you're already offloading a ton of work to this database. Scaling this database isn't going to do you any good. We're already executing this stuff faster than it's transferring across the wire. Like, we need to look elsewhere.

But, like, I bring that up in this context just to say, like, yes, databases do handle things really well. And also, like, in this case, an ORM was used, so, I mean, it was efficient. Like, there's definitely efficiencies there and good use cases for sure.

WILL: I mean, sometimes you can flog the database a little bit too much; believe me, I’ve done that play, you know? Can I get a bigger instance? No. Oh [laughs].

MATT: You don’t need to tell us, Will. Kyle and I just dealt with that, what, two days ago [laughter]?

KYLE: Maybe. Maybe.

WILL: Yeah. That's when things start getting really gnarly, when it's like, okay, can I just put a bigger engine in the car? Actually, no [laughs].

KYLE: Well, and it's interesting, too, because it's not even...usually, at least in the cases that I've seen, it's not that the database can't handle it, so the engine is big enough. It just doesn't have enough seats to handle all the connections. In most cases I've ever run into, it's, we're tipping over databases because we're just talking to it too much. We're sending way too many connections to it.

MATT: That was exactly what was happening.

KYLE: I didn't want to throw you under the bus [laughter], but since you spoke up [laughs]...

MATT: It’s okay. I mean, to be fair, I did not write that, but...

KYLE: That's fair.

MATT: But we did get it resolved.

WILL: The last guy out always takes the arrows. It's like, all right, who quit last? [inaudible 24:26]

MATT: Well, I am responsible for the service that was happening, too, so I will take that.

WILL: You will take responsibility for finding the guilty party and seeing that they're punished.

KYLE: Matt identified the dip. It's fine [laughs]. We know who did it [laughs].

DAVID: Fantastic.

MIKE: That's why it's called git blame.

DAVID: Yes.

KYLE: Blame, right?

DAVID: Subversion, the predecessor to Git, had SVN blame, and the author hated that it was such a loaded term, so he added an alias, he or she; I don't know who actually maintains that. They added a praise, and it's just an alias for blank, an SVN praise to see who did this. I don't know if they still have it. I think it got retired because nobody used it. Absolutely nobody is going to Subversion anymore.

MATT: [crosstalk 25:10] uses Subversion anymore.

DAVID: Mm-hmm. While Subversion was still extant, they were like, yeah, nobody is using this. We only ever go to the code in anger.

MATT: It's true. I don't think I ever used SVN praise. I did use SVN blame.

WILL: I don't know. I agree. I agree. We shouldn't have called it blame. But that was probably Linus' fault, and, like, he probably did it for a reason.

MATT: It’s snarky. It's a little funny.

DAVID: Good times. Eddy, I messaged you during the call here. The story that Mike gave us at the top had an interesting phrase in it, which is, it's always a missing index, except when it isn't. And we actually ran into that. I don't want to steal any of your thunder. Do you want to tell the story of the stuff that we worked on with Bill and why we had to do it?

EDDY: Sure. I'm not very inclined with exactly, like, I don't have all the context in what we were running into. I'm not a database guru. But the bare bones essentials is we have a project that's currently leveraging something called Elasticsearch, right? And, historically, you know, anytime we needed to make any changes to it, right, it's brittle in our context. You know, it's always been a little bit shaky, you know, and inconsistent, always caused a little bit of turmoil with experience.

So, we had some time to really research, you know, an alternative, you know, to replace something like that and just sort of bring it in-house. And so, we had some time to look into something because we use Postgres at Acima. And so, it turns out that Postgres actually handles text searching and matching natively, called Full Text Search.

And sort of the way that works under the hood, right, is that it uses a GIN index that gets added into a table, which then passes in a cascading TSVector into any joins table that it wants to match any strings on it. The problem with adding an index to that degree, especially for a table that we were adding it on, was that it had way too much activity on it. I think it was up to, like, it was for every 20,000 inserts, you had, like, 10 times updates [laughs]. So, like, the amount was basically tenfold with updates versus inserts.

And so, what we ended up doing is we ended up bringing a copy of production. I didn't see any of that data, I should say. It was piloted by another database.

DAVID: For DBA. For a DBA.

EDDY: It’s for a DBA. So, I had no access to that whatsoever. The bastion for that individual was only to that one person, no one else. I didn't see anything, so no PII. Anyways, the point is we were able to mimic, quote, unquote, “activity” to what we were to expect to happen in a production environment, but in a lower environment setting. And it turns out that by adding a GIN index with the nature of how that works, right, we actually ended up slowing down the number of how fast that table was able to process connections.

So, like, long story short, I mean, I guess, like, 9 times out of 10, you can say, oh yeah, index is always the way to go, except in our scenario. When we're adding a GIN index, it actually made it slower. So, really something to keep in mind where, like, you do have that really one edge case, right, but it’s not one tool at all, right? Like, you do have to be cognizant of what index you're leveraging, and it can have, like, cascading effects like what we saw, so..

DAVID: Absolutely. The general case of this is that the more heavily you index data, the slower your inserts and updates are going to be because every time you insert or update a record, you've got to go update the indexes, right? The whole point of an index is it's pre-calculated. So, when you modify the data, you've got to go redo the pre-calculation, and that totally makes sense.

I cut my teeth in databases, oh, man, [inaudible 29:25] to SQL base. But the one you might have heard of would be MySQL was where I started. And MySQL, like, 3, 10, 15 years ago, generally, had two types of tables that it liked. One was...and I can't remember what they were called, but one of them was basically fully indexed, and inserts were kind of slow, I mean, normal speed, I guess, because this was the normal type of table.

But it also had kind of like a logging table, and inserts into that table are very, very fast, but you cannot put indexes on the table. You can only read it sequentially. You can read it any way you want. It's just slow because it's not indexed. And so, if you've got, like, an event stream coming off of something, one of these tables is great because you're not ever going to query it in production. You're only going to query it, like, in the data warehouse. And when that happens, then, yeah, strip all the indexes off because that's going to speed up your inserts and your updates.

MIKE: One thing that Will mentioned a while ago is that SQL is very different from other programming. And I wanted to latch on to that a little bit because I think that there's some...I think it has a lot to do with what we're talking about. We don't use it because it's weird, right? And the way that it's weird, I think, is relevant.

So, SQL is declarative; it's not procedural. That is, it's not allowed to have side effects. You just declare the transformations on the data and what you get out, and then your compiler for your SQL, right, does everything. And that may sound familiar to anybody from functional programming, which tries to make the language like that as much as possible because situations where you can just be declarative and not procedural can be optimized like crazy.

And they actually represent what we're doing most of the time as engineers. We're taking data from one place. We're changing it around a little bit, and we're sending it to another place. I mean, that's our jobs, right?

DAVID: Mm-hmm.

MIKE: We grab the data from somewhere; we tweak it a little bit, and we send it somewhere else.

And if you have a language that will only do that, you can have your interpreter for your SQL that will create some really optimized code that will do that, and you don't write the procedure at all. You have no idea how it's working, and you don't have to because you're thinking about relational algebra, and [laughs] how does this work from a high level?

Much like you would do in a functional language, where you're just thinking about the output and the transformations, and you write your code as a series of transformations. You worry about types and about output, and you don't worry about how to do a for loop. And that's usually considered best practices in modern coding is to write your code in a functional style that writes those transformations because it avoids a whole class of bugs.

So yeah, it's weird, but it's good weird, right? Now, it's an old language. It's got some warts, right? It feels kind of old, maybe, but the principles it's built on are solid. You learn the syntax; it works great. And thinking that way is, I think, one of the most important things that an engineer, and I tell this to junior engineers all the time, it's one of the most important things that will help you in writing good code.

If you can think about your transformations as being a series of transformations, well, what's my start data; what's my end data? Don't worry about how it gets there, and then let the compiler or your interpreter do the job, right? And instead of you doing it yourself, you're less likely to have bugs, and it'll probably be faster. Yes, it's weird, but [chuckles] if you can start thinking that way, you're probably going to write your code better anyway.

MATT: Old weird, but good weird. One word, Erlang.

DAVID: Erlang, mm-hmm, getting on the beam. I'm scouring my bookcase from here, and I'm in the middle of moving my office, so I can't find the books that I'm looking for. I cannot recommend highly enough SICP, “Structure and Interpretation of Computer Programs.” This is a free class taught...you can download these online. There's a book for it. This is a CS class. The video quality is terrible because it was recorded on a VHS in, like, 1986. It is old, old, old as dirt. And if you are a web programmer, there is stuff in there that is rocket science world --

I went through the SICP lectures in 2009, and I was furious halfway through that none of the senior developers around me had a clue what was going on in this class. One of the biggest things that they will teach you in that is how to implement streaming, how to think in terms of streams. Instead of in loops or in long blocking things, it's just, what's the next thing, what's the next thing, what's the next thing?

And you have to understand streaming if you want to do work on a collection that could potentially be infinitely long or human input, where it's, like, every keystroke is just a new keystroke, and you can't iterate over every keystroke that's going to be entered into the program before they've been entered into the program. So, you literally just have to sit there and wait for each keystroke to come off.

Handling those streams, because it's from the ‘80s, the same math problems on the same whiteboard drop out when you go talk to the database theory people, as the SICP guys. When they start talking about streaming data out of the system, they're like, “You guys are writing a database engine. 100%, this is a database engine.” Coming back from SICP, the drawback to SICP is it will teach you how to do Lisp. The advantage to SICP is it will teach you how to do Lisp, and learning Lisp will make you a better programmer, I promise you that.

MATT: 100%.

DAVID: I don't know a single programmer who has learned Lisp who has not 100% agreed with that statement. It will make you a better programmer.

So yeah, Structure and Interpretation of Computer Programming. And if you want the light version, “The Little Schemer.” Scheme is a dialect of Lisp. You can get it on your computer right now. It's free, and you can learn to program in Scheme, which is, again, a smaller dialect of Lisp. And the first time you learn how to write your own multiplier, when the only tools you have in your pocket are add, subtract, and is this number equal to zero? You have no if tests other than is this zero, which is fantastic.

Write the equals equals for a pair of integers when all you can test for is zero. You're going to need a subtractor, and you're going to need a lot of time if the integers are large.

This has been a good bit of chatter. I think we're at a pretty good closing up point.

MIKE: You know, one thing I wanted to latch on to, again, and I think Will brought it up also, as to AI not taking my job tomorrow.

WILL: Not this year.

MIKE: And I alluded to it before. I have, you know, as my career has evolved over the last few years, I don't write very much code anymore, mostly directing things, right [laughs]? I'm moving things from here to there, you know, communicating. But I still write SQL regularly, close to daily. I write in queries because I need to get information out. And they're ad hoc, right? And they're going and pulling out data that other people don't know how to pull out because, you know, they're not sure where the data is, which says something to me. When everything else goes away, what's left is SQL [chuckles] because it endures forever. You know, your career starts with that, and it seems to end with that as well. It is just always needed.

And yeah, you might not need it for every piece that you're writing because the ORM probably should be used to do most things. But knowing that is probably, you know, the most valuable thing I've learned in my career in terms of programming. Other languages come and go; SQL endures.

MATT: I would agree with that. It's been the only thing that's always been there in my career. Tons of different languages, always SQL.

WILL: Like, when I'm cracking up a codebase, right, and I'm like, okay, what are we doing, right? Like, the two things I look for, when I could find them, especially, like, I go to the schema, like, that schema.rb. What's in there? Who was your daddy, and what does he do? And then I look for the API requests. If I've got the schema and the API requests, I know what's going on. And it's usually not that hard to read. You know, I mean, like, you'll have, you know, maybe 50 tables, you know, and, like, you know, 50 tables, and, I don't know, half as many API requests. And if I know what's going on there, like, I got you, you know, maybe for web apps at least, you know.

DAVID: 100%. I had the great privilege of hanging out quite a bit with Sandi Metz, and she's just an amazing programmer and coach. And she's really, really good at teaching people how to think, which is so much more rare than being able to think. She can teach thinking, which is incredible. She and I were chatting at a conference, and I was talking about, like, things that change over time. And we got to talking about databases and ORMs, and she said something really interesting. She said, “I've never seen a company evolve its database stack,” like, moving forward.

And I thought about it, and I'm like, have I seen...and what she said was, “Your tech stack will change. Your database will live forever.” One day you're going to wake up, and somebody's going to say, “Let's get rid of Ruby on Rails, and let's switch everything over to Groovy,” and it'll be a three-year project, and your tech stack goes away.

I have migrated databases under the same tech stack, but they have all been the same migration, which is migrating to Postgres away from a very expensive vendor lock-in, like SQL Server, or, you know, one of the big, big, big, big expensive ones. That's the only time I've ever changed database flavors mid-roll on a serious production system.

WILL: Yeah, I was going to say that is why I think many developers of our vintage have an almost pathological aversion to that one database vendor.

MATT: We all know what you're thinking [laughter].

WILL: And if you know the one, you know the one. And if you don’t know the one, it's because whoever raised you [laughs]...like, we don't say that word out loud, like, Voldemort [laughs] for developers that were like...

MATT: I have a friend who's very high up at that company.

WILL: And I think that [inaudible 40:37] a whole lot of money.

MIKE: And if only there was, like, a really wise person you could...I was going to say, if only there was, like, a really wise person who [inaudible 40:43] you could go talk to and ask this question to, that would be really helpful.

DAVID: Sandi is incredible. She is the first person that...Mike, you said something earlier. You said, always, always, always accept when you don't. And she and I were at loggerheads over, do you test private methods? And we went around for, like, 20 minutes. And then, by the end of it, she went, okay, yeah, never test private methods unless you want to, and then test them as much as you want. Because she had found the edge case, right, which, well, what if I really, really want to? And she’s like, if you really want to, go for it. Anyway, that's a side ramble. That's a P.S. for the podcast. I think we’re good.

WILL: Wait, so are you on team test private methods or team don't test private methods?

DAVID: I'm on team never write private methods, ever. So, testing...

WILL: What?

DAVID: If you've got a private method, you have logic that could break and should be tested. But you've put it deliberately at an impedance mismatch to the stuff that you're testing. And you probably have a better...you have a public class hiding inside that class that if you just name that thing, that becomes a private class. You pull it out. You make it a public class. Those methods will become public. And you guard access to that privacy by making that class private in the class that wants to use it. You're making your poop face. I love it.

MATT: Sounds like we have the topic for our next podcast.

Mike: [laughs]

DAVID: Oh, we could do a battle royale on testing private methods or not, absolutely. Because there's a... I've been very dissatisfied with all three positions. So, I don't actually have a, what do you call it, I don't have a sacred cow in that hamburger factory.

MIKE: I've got a strong stance on testing in that it doesn't matter whether you have private methods or not. Because if you write your tests right, you're just following every branch of logic, of input logic. What are all of the different types of input that you have that could be branched on in the code?

WILL: I've existed in ecosystems where testing on a public-only policy requires some crimes against God and man in terms of manipulation of mocks, and run time, and stuff like that. The big, ugly one is React, where there isn't a direct run loop input-output. If you're dealing with sort of like, hey, if I do this, what's the UI going to look like? Am I going to see all these elements the way I want them to? Which is obviously a test that you would have an interest in writing. But you have to commit crimes against God and man to sort of force that run time and then loop to arrive at a sync or a fixed point or whatever. So --

DAVID: Cool. Will, let me put a pin in that for our next episode, because we could go another hour real easy because I'm ready for this. This is awesome, so yeah.

MIKE: So, we've got the teaser for the next episode.

DAVID: We’ve got the teaser for the next episode. Fantastic.

MATT: And it won't get heated [laughter].

WILL: Horrors of testing.

[laughter]

DAVID: It won't get what?.

MATT: I said it will not get heated.

DAVID: Get heated. No. No, we're all friends here, so...

MATT: It’s true.

DAVID: Fantastic. Yep. Thank you for listening to the Acima Development Podcast. We have been Acima Development. And, Will, you’re honorary. You're an alumnus, so...

WILL: And also Will [laughs].

DAVID: And also Will. There we go. People we love and also Will and...so, thank you for listening.

WILL: [laughs]. Acima [inaudible 44:16]

DAVID: Yes, I like it.

The episode frames engineering as an investment rather than a cost. Mike opens with a parable: leaders who put capital (or engineers) to work create value; those who “protect” resources without progress destroy it. The panel builds on that: impact comes from choosing work that improves tomorrow—planning enough to ship, avoiding ivory-tower perfection, and recognizing that “cheap” talent can be expensive in management bandwidth. Good contractors and senior ICs pay for themselves by shipping and by freeing organizational attention.

They dive into technical debt with a pragmatic stance: prefer YAGNI, refactor when the change is hard so the right change becomes easy, and pay down debt “as you go” when you trip over it. Treat debt like a family credit card—budget a consistent tithe each sprint for the top, highest-leverage items and let the bottom 10% freeze or die. Too much debt maxes the card; you end up servicing interest (slow teams, broken features) instead of building. For growth and product strategy, pursue small, adjacent bets that serve existing customers and reduce risk rather than sweeping reinventions.

A major throughline is investing in people. Effectiveness scales when seniors make others effective: clear plans for distributed teams, social onboarding, and apprenticeship-style pipelines that move interns, QA, and juniors into productive engineers. Pair programming and XP practices are championed as systematic knowledge transfer—training juniors up, spreading domain context, and keeping work flowing despite constant change. The most rewarding ROI, they argue, is the compounding payoff of people who grow, stay, and ship.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'll be hosting again today. With me, I've got Dave.

DAVE: Hello.

MIKE: Eddy.

EDDY: Hello.

MIKE: We've got Will. Yeah, we’ve got Will Archer. We've got Jordan.

JORDAN: Hello.

MIKE: We've got Justin, and we've got Ramses.

RAMSES: Howdy.

MIKE: I'd like us to start off with a story. This time I'm going to go biblical.

WILL: Oh God.

MIKE: [chuckles] So, [inaudible 00:00:51] I’m going to tell an old story in a new context. No religious context, but I think it applies here. So, a boss is going to go on a year-long sabbatical, maybe not that realistic, but, you know, humor me here [chuckles]. And they're going to leave, you know, put some employees in charge. And they're going to put them in, you know, these are, you know, leaders, leaders of the company, and they're going to give one person responsible for 10 million worth of funds, another one 5 million, the third one 1 million. So, they're going to be responsible for some funds here while the boss is on sabbatical.

A year later, the boss comes back, and she asked each of the employees how things went. And the first employee, well, they've hired a new team, and they’ve built a new product that's going to bring in $10 million a year. So, "Well done," she says. So, the second employee has invested their millions in acquiring a small dev shop, brought them in, and they've increased the size of the dev team. And they expect that to contribute to $5 million in projected annual growth. So, "Well done," she says.

So, the third employee let go of some of their developers because they wanted to save their money. They wanted to make sure...I've only got a million dollars. I got to make sure that it stretches for this year. Customers are complaining because now there's not the support on the product. The product's been losing users because everybody's complaining. Reviews are going down online. And the boss says, "You know, why didn't you invest the money? You could at least put an investment account. You just sat on it." And the employee says, "You know, I was just scared to lose the money. So, I made sure I kept it all." The boss says, "You're fired [laughs]." And she gives the money to the first employee.

So, we hire engineers as an investment, right? Engineers are builders. And we build stuff because it makes companies money. You know, we're doing this on our own. We do it because we're trying to make money. I mean, we enjoy the job, but the reason that we get paid is so we can make the company money, and I think it's critical to remember that. Every minute we spend is an investment. And is it a good investment [chuckles]? Was it worth investing in us? When we get to next year, is the boss going to look at us and say, "Was it worth having them as an employee? Did we actually do better because of that?"

And it also affects the way that the company sees the engineers. There's sometimes a mindset where we're seen as a cost, just net cost. Like, "Oh, they're just the cost we have of doing business." And I think that's a dangerous way to think because it doesn't appreciate that you've got to spend some money or, you know, I got to put in some risk to make money. Money doesn't just come walking in the door. It happens because you've built something and maintained it.

So, in the broader picture, I'd like to talk, for today's discussion, I'd like to talk about what it means to treat ourselves as an investment and what that means for engineering. Go ahead.

DAVE: Can I jump on a tail on that?

MIKE: Please.

DAVE: I love when ideas from very far apart in my brain snap together. It's a psychological thing called synthesis. And you just synthesized this thing for me, which is, are we investments, or are we a cost?

And I took a financial literacy class 25 years ago. And the guy stood up and he said, "Do you have any assets? And what do you have are liabilities?" And we all started going through all the things. Well, I've got assets. I've got a car. I got, you know, and we just started listing all the things we had. And he's like, "Every single thing you just listed is a liability, not an asset." I'm like, "What are you talking about? My car is an asset." He’s like, "Is it going up or down in value?" "Oh, it's going down. And you had to pay for it." "Yep." And it doesn’t make you...well, I mean, it gets me to work. Okay, that's an investment. That's something you have to do. But you can't sell that car for more money than you had it. It is not an asset in and of itself. You can use it. And it's good, but it is not an asset.

When you acquire a company or you're in a company that's getting acquired, the first thing management does is say, "Nobody leave," right? "Here's the retention bonuses. If you stay for the next two years, here's this." Why? Because you are an asset. You increase the value, and your continued presence here continues to increase the value of the system you are in.

So, anybody who thinks their employees are a liability...you can point at a specific liability or a specific liability and call them an employee. That's an interesting wording, but you get what I mean. But that's like a onesie-twosie that the job, when an employee is a liability, is because they are failing to be the asset that you need them to be.

WILL: That wasn't my experience, but that's a tangent.

MIKE: So, well, what's worth investing in? If we are assets, if we're worth investing in, what is? What should we actually be investing in? Because there's a lot of stuff you can do in a day. There’s choices. That's a broad question, intended to be open-ended.

JUSTIN: [laughs]

DAVE: I think on our last podcast, I mentioned the generic advice of prioritize later. And anything you can do now to improve tomorrow or improve later, I think, is a great way of an asset. And so, that's a very generic answer to what should we do as employees, but that's the thing that I try to do. I try to look at, like, where does it hurt the most, and what can we do about it?

MIKE: So, where does it hurt the most?

WILL: The question I've got, right, is, like, sort of, like, in what context, right? I mean, sort of, you know, as an individual contributor, like, I manage a department of one, right? You know, so I have things that I need to be doing, right? It's all good, you know? And so, you know, my discretionary spending, right, is limited, not zero, right? Because there is lots of interstitial time that I'm waiting for something, right? You know, like the factory is idle for whatever reason. It’s just sort of, like, move up the ladder, right, as your scope gets larger, like, that sort of time, like, you know, the latitude you have to make decisions gets bigger. And so, I guess the question is, like, who, right? Who would be investing and, you know, what?

MIKE: Well, I was thinking about this before the call as well. It also depends on the scale of the organization you're in. If you are alone, if you're the startup, you know, investing in yourself, building a new product, what matters probably is very different than if you're on a team of 50 or 100 or 1,000.

WILL: What if you're a startup and, like, you're not, you know, like, if you're not generating a profit, everything's an investment, isn't it? You know? It's a thing. I mean, like, you know what I mean? Like, you're building something, right? Like, the investment is fairly direct and, like, unambiguous and kind of cut and dry.

MIKE: But you can go for some perfect ivory tower implementation that's going to take you 10 years, and you don't make it to market before you run out of funds. So [laughs], I think it does matter. I think when you're that startup, you need to have a goal of getting in money fast. It's very sink or swim. Like, you need to prioritize, and you may make some cuts that you wouldn't do otherwise.

JUSTIN: So, I think it goes back to a little bit what you mentioned right there, Mike, is, like, you've got to plan, or you've got to have a plan. And if you don't have a plan, you have no plan. And so, whatever you do may not fit within that overall plan of making sure that I get paid tomorrow, next week, and, you know, next year. So, time spent planning, I think, is, you know, you don't want to do too much. But it usually pays off in droves.

DAVE: When I was a contractor, I always had to consider, when this contract ends, I will go back to the sales cycle, and you don't get paid for sales. That's a pure investment in speculative opportunity. And I get where people are like, "Well, your hourly rate is really high." And I'm like, "Yeah, because you're hiring me for a very short contract." And they're like, "Well, we only need you for this time." And I'm like, "Yeah, and then I'm going to go be unemployed. You understand that if I've got a two-week sales cycle and people hire me for two weeks, I'm unemployed 50% of the time. You're going to pay me 200% of the market value. That's just how it's going to work."

MIKE: Yeah. Fairly typical for contract work for that reason.

JUSTIN: So, talking a little bit more about investment, too, and I'm lucky enough to, like, be hiring people. And, unfortunately, the people I'm hiring are in India. But having said that, the investment that I make is getting up early and spending time with them to make sure that they understand what the vision is and what the plan is. Because I don't see them for, you know, I only see them for, like, two hours out of their eight-hour day.

And if they don't have a clear plan on what they need to be doing, the money that I'm saving by, you know, having people in India is getting thrown out the window because, you know, they're only productive for two hours. So, the time invested in helping other people be successful and helping make sure that they understand what the plan is really worth it as well.

And it's interesting because, like, a lot of this is not, like, coding. You know, this is not necessarily individual contributor work. It's like, "Hey, I need to plan. I need to make sure other people understand. I need to make sure I understand." All of that is very important before you sit down and code.

MIKE: So, you're saying if you help other people be effective by helping organize the work, you've made more efficient use of your time than if you just went in there and tried to build it yourself?

JUSTIN: Yeah, which is rough.

MIKE: Sure.

JUSTIN: Because if I go in and build it myself, I know it's done right. And, you know, I can be the hero because I did the thing.

MIKE: [laughs] Absolutely.

JUSTIN: [laughs] But it’s like one person is doing the thing.

WILL: So, would you say that you're investing in your sort of, like, contract team? Like, you've got an offshore contracting team that you're investing, like, not only in your team, your enterprise, but you're investing in them, right? So, you've got, like, contractors, right? But, like, you're training them.

JUSTIN: Yeah, and, like, any sort of success that I have is dependent on the success that they have.

MIKE: That's interesting, that idea you're exploring there, Will. You're investing in the people. And this is an idea that has come up a lot, I think, on the podcast, you know, people are human. You can't just drop somebody in, "Okay, you're productive, and you'll do this unit of work, and then you're done."

JUSTIN: [laughs]

MIKE: And that's just not how work works. There's that human element that requires the coordination that requires the training. And we say training and some of that is...it's not necessarily, you know, learn this language because that's kind of the baseline. That's the expectation a lot of times going in. It's rather, you know, "Here's the systems we work on. Here's the way we do things. And here's how you can be effective." It's around things that, like you say, aren't necessarily coding although there may be coding style and things in there. It's a lot of bringing somebody into the group.

DAVE: Mm-hmm, social onboarding.

MIKE: Yeah, well put: social onboarding. And that's an investment, and it's...you actually are lowering your return on investment if you think, okay, I'm going to hire these people for one month or two months, or anything less than, I mean, even up to a year or more. Sometimes, you know, I think that productivity is still climbing. We've got contractors I've been working with for almost 10 years, and I would hate to lose that, right? That is a valuable thing to have. We've got relationships, you know, we're friends. And we've got, you know, trust. These people know exactly what to get done, and they do it well. And that matters. You lose that if you just treat that relationship as disposable.

WILL: So, like, what's the con? What’s the flip side of that? Okay, you've got a contractor, right? So, I mean, the contractor and the expectation, I think, that I certainly work under, and I think a lot of people work under, is sort of like, "You're going to get it done. Like, what have you done for me lately," right? Like, there's a level of investment that you don't expect and you shouldn't expect. You shouldn’t expect to get hired if you're not making things happen now, right?

MIKE: Sure.

WILL: We discussed this as sort of like the mismatch, right, between contracting in that, like, a lot of shops can sort of, like, compete on a cost basis as contractors, right, for a job which, you know, if you think about onboarding, right, like, a contractor you're always onboarding. And that’s hard. That's really hard. And so, like, the expectation is that maybe that investment and, like, your sort of technical skill set is priced in already. And it's something you pay...you pay for something if you've already done it, right? Where's that line appropriately drawn?

Because this is also, like, sort of the converse where, like, you have, like, long-time contractors that are employees in all but name. And that's sort of, like, a weird, like, bureaucratic cul-de-sac that commonly people will find themselves in.

MIKE: It's an interesting question, I mean, because you're hiring somebody expecting them to know something [laughs]. Otherwise, you wouldn't be hiring them. So, how much do you invest? You know, how much is it worth investing there? If somebody doesn't really know what they're doing, how much do you put in? And I guess it does depend somewhat on the cost, right? How much you’re paying for it.

I'm thinking about if I was asking somebody to build a house, if I was hiring a contractor to build a house. Actually, earlier today, I was talking to a roofing contractor. My roof is getting old and needs to be replaced. And I did talk to multiple contractors, right? And I compared the price. But I didn't go and say, I'm going to find the absolutely cheapest contractor I can possibly find. And that was a deliberate choice. Because I knew that that would not save me time, or energy, or grief in the long run.

I made sure that the prices weren't way out of line, and it was somebody who was reputable and, you know, local, lots of reviews [laughs], somebody I can go back to six months from now and say, you know, "You messed something up,” and expect them to actually do something about it. Because it mattered, right, to get somebody who knew what they were doing and that I could have some trust in.

On the flip side, I wasn't actively looking for the most I could spend, you know, there's no, like, gold leaf on my roof [laughs]. You know, it's a roof. I needed it to get the job done. And it seems like there's some applicability there. I’d think that going for the cheapest you can possibly get probably is not going to save you money because there's going to be so much investment afterward in making it work. But you probably want to go a little bit higher up that ladder, right, and pay a little more and get somebody you can trust.

And then, you know, maybe thinking about people like engineers, right? Somebody is like, okay, I'm going to have these people with me for a few months. They have some basic competence enough that I can trust them to get the job done if I spend some time. I would advocate for something like that so many times. Go ahead.

WILL: I'll toot my own horn here as a fairly expensive contractor who still manages to get hired on a fairly consistent basis. When I say fairly consistent, I mean I've been overbooked forever, since the dawn of time. But the cost of your contractor is not strictly defined by the dollars that they're charging you because, in all honesty, there is organizational bandwidth that is a zero-sum game. There are so many managers in the company. They can get so many projects done. They have so much attention. They have so much planning. They have so much specking things out. And they can only deliver so many things per year. There's a hard limit on the number of things that they can push out per year, period.

And so, you can say, like, "Wow, you know, this guy's only, like, 10 bucks an hour." 10 bucks an hour at Bangladesh he’s so expensive. Yeah, but, like, he's taking up 20% of your management bandwidth for the entire year. You're not getting anything else out. It's not going to happen. You know, versus somebody who's much more than that, but they'll get it done, and you don't have to worry about them. Like, I'm saving your bandwidth. And that's, I think, a value proposition that most people in the business who have been there for a while understand. And they're like, "Yeah, but he'll do it, and he won't bother me. It'll just happen."

MIKE: We recently hired, I won't give names here, recently hired a contractor to work with a team, local. And he stepped in and immediately started getting stuff done, went and found some organizational efficiencies, called them out, and improved things just kind of from day one. He doesn't know everything, right? You know, nobody's...of course they don't. But they’ll make a huge difference. And I'm like, "Wow, give me 10 more people like this."

Again, not perfect, nobody is expected to be, but knew what they were doing. And I can trust that they'll get the job done. And that was worth its weight in gold, right [chuckles]? Like, [inaudible 19:24] making us money. Absolutely. That ability to get stuff done just is absolutely critical. You can have people who are less experienced, but only up to a certain critical mass. They need to have basic competence, right? Somebody you can tell them what to do, and they'll get it done. But you need to have people who can help direct that, who can work really independently.

Because if you have these people who are less experienced and can't just get stuff done, they reach a critical mass, and then nothing gets done, right? You reach a level of work, and it doesn't matter how many people you hire. You're not going to get anything else done because there's just a lot of people running around and nobody knowing what to do. The only way I've seen things be effective is you have people who really know what they're doing, who can help those other people.

So, Justin talked about working with a group and spending a lot of time working with them, helping them know what to do and keep them going. That gets work done. But there's a limited number of Justins [chuckles], and you have to have more. You have to have enough of those hubs of the wheel to keep those pods moving, or else it just falls apart.

So, we’ve talked some about this balance of getting what you pay for and being willing to invest in people. I'd like to hit a little different topic related to this: technical debt. So, we talked a lot about investment, and then there's the flip side, where you're taking on debt. And you may be taking on debt in order to invest, right? So, you're building something. So, going back to the startup idea, if you don't get the product out the door, you run out of money, and your business fails. Maybe you incur some technical debt along the way. And that might be very much the right decision. You're taking on some debt to invest.

So, if we think about this idea of investing in ourselves, when should technical debt be paid? Where do you balance? Like, I've got this money. Do I pay down the debt or build something new?

DAVE: Mike, all I heard you say was, "It's too quiet in here. Let's start a fight."

MIKE: [laughs]

DAVE: I will say that as I've gotten older as a programmer, I have learned more and more, (YAGNI) You Ain't Going to Need It, right? So, a lot of technical debt came from preemptively trying to prevent the wrong technical debt, trying to build something that it turns out we never ever needed. And we see this all the time. Eddy, you and I have ripped some stuff out this week, I think, that somebody built, and it never happened, right?

There's a thing on our team right now where we had a build-out so that the individual developers could be running Docker to standardize our configuration. And then we realized that we have a service-oriented architecture. You can't run 12 Dockers on a laptop. It's just not going to work. It makes a great lap warmer, but it doesn't make a good development environment. But there was all this stuff to support the Docker initiative that just died. And then that code got maintained for another three years because it's separate. So, YAGNI, YAGNI, YAGNI.

So, as I've gotten older, I've started thinking, when I'm done with something, I will do the refactor to try to clean and make things readable and understandable. But I try very, very hard to never let myself say, "We need to do this for this future thing." I've gotten much, much better at saying, "Tomorrow, Dave can handle that." We all joke, that sounds like a future me problem. No, it really is. This sounds like a future me solution. This sounds like something future me absolutely can handle. I'm going to do stuff today to be kind to future me, but I'm not going to try and tie future me's hands or try to paint the wall that future me hasn't built yet because it's just a nightmare. You end up with a bunch of dried paint in the shape of a house. It's like, how do you put that on?

But if you've got Cowboy Coders on your team, they will weaponize that against you because they will go out and crap out all the technical debt they can to get their stuff done and polish all the rivets they want because they know you are going to come back and clean up their latrine. And I've started fights over this. Legitimately, I'm just like, "Yeah." I have so many metaphors that are inappropriate for this. Cleaning up the latrine, I'll just leave it there. I've yelled at people in this company recently, so...not recently, right? Months ago. I'm much calmer after the sabbatical, so...

MIKE: [laughs] So, you're saying build enough and not more.

DAVE: Build enough...oh, sorry, there's one more instinctive habit that I just realized I skipped over. All of this is enabled by the specific habit of when you get an assignment. I'm a very high-latency developer. You give me an assignment, I'm going to take a while to get started on it because the first thing I do is refactor. I will look at the code, and I'll go, the change you want me to make is very, very hard to make.

Kent Beck, famous quote, if the change you need to make is hard, refactor the code until the change you want to make is easy to make, then make the easy change. So, like, Kent was like, I'm really lazy. I only make easy changes. And then he's like, yeah, I spend most of my week doing the hard work to make the easy change possible. You have to do that. So, it's like, anytime I get a new ticket, I'm like, yep, the technical debt that I wasn't paying off before, I have to pay it off now because now I know where the house is. We have to paint this wall before we cut a hole in it.

MIKE: Are you suggesting that you should pay off technical debt as you bump into it?

DAVE: As you go. As you go, yeah, basically. Because when you bump into it, now you know, I really do need this. You can't say, "You Ain't Gonna Need It" when I'm literally stubbing my toe on it.

WILL: I don't know, I like...like, having done this for a long time, like, I do consider technical debt. Like, the YAGNI principle there I think it has a lot of merit. I personally think you should just...I think you should [inaudible 25:18]. Like, you know what I like? I like to stack rank technical debt. The bottom 10% of the technical tickets, relegate them. Relegate them to the freezer.

So, I think the family credit card is a good analogy for technical debt. I mean, you should have a bunch of, like, you should have a wish list, right? And, like, every once in a while, get your leads together and rank them, you know? Bottom 10%? Yeah, yeah, probably. Probably that's just never going to get fixed, you know? And you should take a certain number of points, you know, maybe every sprint, maybe a quarter, maybe something, you know?

Like, yeah, yeah, you know, the bottom 10% if things have been, like, nah, we don't really care that much, you know? Versus, like, you know, honestly, like, there's stuff that, like, you know, like, people do care about. People will, like, get out and fight for it. Like, that stuff actually matters, like, you know?

Like, one form of technical debt that I've found particularly pervasive is the sort of, like, performance creep on, like, developer workstations as a result of, like, IT overhead, let's say. You know, there's administrative IT, things running on these workstations, which I don't care. You [inaudible 26:42] work my workstation all day, all day, you know? Like, that's no problem. I have a smartphone that I can do all my personal stuff on. It's not a problem.

But performance tends to degrade as more and more things are being checked and exclusions aren't being maintained. And that's the kind of thing where it's just, like, you know, people will rumble with you, but it'll never show up on a sprint. It's just...it's a silent tax across an organization, and that's a little bit of a personal anecdote.

DAVE: I remember talking with Mike a couple of years ago, and I'm like, "I keep getting bumped off the VPN." And Mike said, "Yeah, that happens," I’m like, okay, okay. And I talk...I keep getting...and it was starting to sound like Dave's kind of a complainer. He's constantly whining about the VPN. We all get bumped off the VPN. I got so annoyed one night that I cracked open the laptop. It's like Tuesday night, and I wrote a thing to just check, is the VPN up, and then log it.

And then I came back, and I'm like, okay, I've got this graph. Every 6 minutes, my VPN goes down, and that's a 30-second timeout, and whatever I'm doing just halts. And I go to Mike, and I'm like, "So, should I be losing my VPN 70 times a day?" And Mike was like, "What? No, wait, wait, what?" But the patterning, the social onboarding was my VPN is going down. No quantity information supplied. Everyone else assumed quantity being far, far less. And I'm like, no, I'm not oversensitive. I'm getting my face punched in by this stupid VPN [laughs].

WILL: Yeah, right.

DAVE: We fixed that, and my life got so much better. So, yeah, communicate, I guess, is the answer to that.

WILL: Oh yeah, well, exactly. I mean, it's just, like, you know, just, like, tithe, you know? Just pay your tithe to the tech debt gods and, like, rank them and relegate them, you know?

MIKE: Well, I agree with it. I agree with what you're saying strongly. Take a chunk to every sprint, you know, schedule it so whatever percentage your company's okay with, you know, 10% of your time, 20%, 30%, you know, that you're working on each sprint, be working on paying those things. And you said, you know, don't worry about the bottom 10%. Honestly, it's rare to get past the top 30%, right? Because new things come up.

DAVE: Top 10, yeah. Pareto's rule rules here, like, the power law. 90% of benefit and 10% of effort.

WILL: Just delete them or, like, you know, [crosstalk 29:03] just get rid of them. But yeah, you're right. I don't know, I mean, like, in the limit case, right, a long-running enterprise, right, like, a long-running technical enterprise, you're paying your technical debt tithe. You pay. You either pay because you're actively working on it, or you pay because it's, you know what I mean, you're taxing your organization. But in the limit case, as, you know, T approaches infinity, right, you're paying your card down more than --

DAVE: You pay for it in customers that go to Amazon instead of you because their site comes back in under eight seconds every time.

WILL: Yeah. You don't get out of it. There's no off-ramp. Where you can find yourself in a really nasty situation is when you're in a situation where credit card is maxed, right, and the minimum payment is substantial, and you don't have the bandwidth to fix it. You just have to work around it, right? So, like, product development is as slow as can be sustained. And the technical debt is taxing you with every ticket. And now you're stuck.

MIKE: And it can get so bad that you can't move forward anymore. You can't build new stuff because you break other stuff. You reach a stasis point, right? A balance point where you cannot actually introduce new features because everything you do fights so hard that it breaks something.

We all live in reality. There's a quote that I've been bandying about that I just realized is a... I love that Will and I, like, we end up in violent agreement so often because we come at things, like, orthogonally, but we're saying the same thing. I've been telling people, "You cannot build software faster than you can build stable software." It's the same thing. You will eventually end up paying interest. All your money is going to the interest. And the only way to get out of it is to make your minimum interest payment, and if that's your entire paycheck, that's great. But you've got to go run a lemonade stand and make 20 cents extra to pay a little bit. You've got to get the principal down.

MIKE: So, if you want to be effective in making money, you've got to keep paying that payment, bringing it down consistently.

So, we've tackled kind of broadly how to invest. We've talked about paying down technical debt. Let's think about it a little bit bigger in terms of how business works.

In a large corporation, in an established business, you've got some sort of product line, and you're maintaining that. You're adding features, and things are great. And you can treat it like that's your only gig forever, and that is risky because the world changes. You ask Standard Oil [chuckles], or you name your big business of the past that has since been forgotten. You look at the S&P 500, and it's not the same as it was 10 years ago, and very different than it was 20, and completely different than it was, you know, you go back a few decades, and you see names you’ve never even heard of anymore because the industry changed, the world changed, and those businesses did not adapt as that change happened.

So, how do you build new products? How do you go about investing in something new, even though it might even have a negative impact, might even cannibalize some of your current business? How do you do that effectively? So, this is kind of a bigger question than engineering [inaudible 32:50], but kind of a business question. What are y'all's thoughts?

WILL: Small deltas. I love a tiny, tiny delta. I mean, like, you're doing business right now, right? Something is working, right? Hopefully, at least, something is working, right? So, like, prospecting is so expensive. So, what more can I do, right, for my customers, right? I mean, this is, like, way afield from, like, software engineering and stuff.

MIKE: Sure.

WILL: How can I expand to adjacent customers, and how can I sort of, like, add some sort of value onto, like, my existing customers? Now I'm putting on my, like, I ran a software product company for 10 years hat, which is not, like [laughs], what we do here but, like, you know. It’s just the most --

MIKE: But it matters.

WILL: Well, for most places, like, you want to mitigate risk, right? Like, we did do...like, we’re Acima, right? We went over to Upbound. We got a very natural move, the natural, like, I'm taking one block over, right? I'm taking one step over. You know, Upbound was like, hey, we're doing this thing, and now this is, like, this is where things are going, and so it was a natural outgrowth. And so, like, and it's acquisition of work, like, that played. It was successful. And I think it was successful for those reasons, right? So, you want to mitigate your delta as much as you possibly can.

You don't want to get into a...even, like, there's a...I think it was...I forget the one. I forget whether it was Bentley or Rolls-Royce, right? Luxury car company, right? Well, how did they get started? Well, they were carriage companies, right? They were carriage companies and sort of, like, they built the top of a carriage, like, a horse-drawn carriage, right? Like, the top that people rode in, right? It was, like, real nice and, like, you know, the furniture was nice because people hung out, you know, for a while. And they were, like, okay, well, I'm building a carriage for a horse. When you got rid of the horse, you still had to sit somewhere, right? And so, the buggy-wheel people, those guys are gone, but the carriage company is doing great.

MIKE: Because they were willing to pivot. But you said small delta. They didn't switch over to making trains or boats.

WILL: They didn't even build engines for a long time, right? Not even engines because engines were hard.

MIKE: They just built the top of the vehicle. That's interesting. Yeah.

DAVE: There's a fun story. I wish it was true that the wheelbase on your car, the distance between your wheels, is based on the fact that the rail gauges were of a similar size. Well, the reason the rail gauge is the way that it is is because of the width of a carriage. The reason a carriage is that way is because if you drive in Europe on the roads that were built by the Romans, they have grits in the stones, and it will break the wheelbase if you're not the right width. And those grits come from horses two abreast, pounding down.

WILL: [inaudible 36:03]

DAVE: I wish it was true. I'm sure that it was an influence. But a tape measure will tell you it's not true. You can just go out and measure any of those things. They don't actually match. But oh, I want that story to be true.

MIKE: You're right, influence.

DAVE: Influence, absolutely.

MIKE: I'm sure it's not completely true, right? There’s differences between horses.

DAVE: Not a governing principle.

MIKE: Yeah. But the general idea...you don't have lanes on your road that are 30 feet wide or 3 feet.

DAVE: Or 70 feet wide.

MIKE: Yeah. They're within an expected range that came from history.

WILL: Well, I mean, I think, honestly, it probably is very, well, no, no, that's not. I mean, the reason roads are now is dictated almost entirely by technology and sort of, like, width, right, and, like, the speed that people travel and stuff like that. So, I mean, like, right now, in the year 2025, it's all technically specified, road [inaudible 37:08], road, you know, the camber, like, you know, like, stuff like that. That stuff is all, like, you know, we tried it every which way. And I'm like, if you're going to go 80 miles an hour, you need this, and this, and this. We need to have a crown on it so water comes off and, like, all this stuff, right? And I need to have a semi go down it, too, and semis are built, you know, a certain way. Anyway.

DAVE: I think roads are now in the should phase, right? The RFC 2119 is the one that gives you the definitions of must, shall, should, should not, must not. And should means you must build it this way unless you have a good reason. So, if you're in a small town or you're building a driveway straight up a cliff face, you can have a four-foot road. You can do it that way. But yeah, if you're building a federal interstate, it's going to be standard width.

WILL: Yeah, well, I mean, like, it's almost always, like, you have, like, a very exotic environmental, you know, situation. And you have some sort of, like, grandfathered in. My dad actually was, like, putting...he was rebuilding his cabin, right? He lives, like, way up north of Michigan, and he’s got a fishing cabin. He wanted to, like, go and, like, build it, right? And so, it's, like, way out in the woods and, like, he had to widen the road. He had to, like, build out the road because the road is, like, a hundred years old and, like, they needed to move the equipment down to, like, dig a septic tank. And you had to build out the road because you just couldn't do it.

But yeah, these days, civil engineering is a pretty solved problem, you know? We might not maintain the roads, but we know how to build a road that will last for a thousand years. That's figured out. We just might not do it.

MIKE: [laughs] We've detoured into civil engineering.

WILL: Yeah. Save us. Save us, Mike. Can we bring it back? We can bring it back. We're supposed to be about woodworking [laughter].

MIKE: Going back to the idea, I'd like to go back to the idea that we talked some about early on, and that's about investment in people. This is something that's kind of dear to my heart, right? It's something I care about.

So, I'll do another analogy. Hundreds of years ago, generally, every town had a blacksmith shop. And the blacksmiths, like every other human being, pass on, right? They're not there forever. So, you have people who want to go into that trade, and they maybe couldn't stay in their own town. They find a town somewhere in the region where there was somebody who was taking on an apprentice, and they'd bring them on. They would learn the trade over a number of years, and, eventually, they'd take over the job of the master that was helping them out.

And that worked out pretty...there's flaws in the system, right? But it was a persistent system for a reason. It stayed that way for a long time because it worked because the kind of...and it works for many trades. We still have it in many trades. You have apprentices, right? You have an apprenticeship program. It's pretty standard practice. People work up, you know, apprentice, journeyman, and they eventually work up to the master of their trade because learning alongside people works.

And sometimes in engineering, we forget the lessons of the past and think, oh, we'll just bring people in, and they'll know exactly what they're doing, and it doesn't work that way. There's always a need to do some of this apprenticeship. And people can sometimes come from non-traditional backgrounds in that way as well if they can take the time to apprentice. I think it's very important to always have somebody growing and a group of people.

So, you don't have everybody be the experts then you lose an expert. It's a huge loss. Instead, you have some experts and you have some people who are less experts, and then you have some people just getting started. And you keep this pipeline going because it's a system that works. And if you forget that and you shut down the pipeline, it works for a little while until you've got no water flowing, right? Everything dries out, and you're in a really bad situation.

So, what do you all think about the importance of career pipelines? I’m going to take this a little further. I’m just going to...storytelling. There's a number of people that I've worked with who worked in QA or application support who switched over into engineering and have become very successful. Some of those people might even be on this call. And I am very happy that they were able to make that move and become successful by taking the time to learn the trade, partner with other people, develop the skills, and go on to become successful.

And by providing that opportunity, we ended up with people with huge domain knowledge and know what they're doing, know the business, and were able to kind of step into the role and be successful very quickly and be strong contributors. Whereas if we just pulled somebody off the street, they would have taken as much time to get up to speed and would not have the same kind of domain knowledge and the social cohesion. We're already friends here.

So, I think there's huge value in having that kind of pipeline. We may also have somebody who brought in as an intern who joined us on the call [chuckles]. There is value in people coming through that kind of pipeline.

So, I'm just going to sing the praises of that idea, and I'm saying, yes, you need to be investing in that. If you don't invest in that...I've described it before as eating your seed corn. You're fine this year, and it's a lot worse next [laughs] because you've got nothing to grow with. I'm laying down my stance here. I'm taking a strong stance in favor of the criticality of having a healthy department, of having this sort of pipeline in place.

DAVE: I could not agree more.

WILL: It sounds like the point of the podcast where I start to proselytize for extreme programming.

DAVE: Kind of.

MIKE: Please do, yeah.

DAVE: Please do, yeah. That is very adjacent to the story I'm about to lay, so please.

WILL: Because, you know, that's how you do it in extreme programming, because everybody is pairing all the time, which is how you do knowledge transfer. And you train juniors to become seniors, and seniors to become staff, and staff to become, like, whatever, super staff, or whatever the heck.

And, like, that sort of flow of training is endemic and intrinsic to sort of a healthy, functioning engineering organization because, like, nothing is ever static. Everything is in flux. Everything is changing. People are coming. People are going. People are moving up. People are, like, I don't know, probably not moving down, but, like, moving out, right? And so, I don't know. Like, there is this sort of you have to accept, like, that intrinsic, undeniable, like, flux, change, right? It's happening all the time.

And, like, I think, you know, the reason I harp on XP is because I have not encountered a systematic way of addressing this as a first-class citizen that it is. I would love to just be able to be, like, hey, what if I just pay somebody a lot of money, and then, like, I don't have to think about it? And, I mean, like, you can do that if you could find the people, but, like, it's, you know what I mean? Like, Google can't do it, so can you, you know?

Like, Google can do it, and, like, they write a check, you know? And, like, if you talk to anybody from Google, they're, like, yeah, it's better, but, you know? So, anyway, so, you know, not to harp on it, but, like, you've got to get it done somehow. And I haven't heard a lot of answers. I've heard a lot of people sort of, like, taunting or advocating or throwing their hands up in despair. But, like, if you're actually trying to solve a problem, once again, extreme programming, shooting [inaudible 45:16] away.

DAVE: Pair programming, yeah, absolutely.

Related to both of those things, we have kind of a career pipeline here, and I love that. I can't remember if I was super interested to be hired here because we had one or if you guys were interested in me because you wanted one. But, like, I was coming out of CoverMyMeds, where we had established a career pipeline, absolutely.

And, like, when I was there, we had this rule that everyone codes. Like, we had code from the CEO. It was terrible, but it was there. We had people from QA that were contributing code into our corporate…Everybody codes, but the QA people would just sit quietly and get crapped on by the dev teams. I was the one that started going in and saying, "No, we're going to pair program."

I would grab a QA person and say, "Come pair with me." And they would look at me like, I'm going to waste your time. And I'm like, are you kidding me? You're QA. You know where all the bodies are hidden. Sit with me, please. And very quickly, they’d find out that even the juniors have an incredible amount of contribution. So, I cannot sing the praises of that high enough.

But Dan Pink wrote a book called Drive, and he talks about what motivates us. And mastery, autonomy, and purpose are the three biggest things that will get you out of bed every single day. And when you take somebody and say, "You can grow your skills. You can increase your mastery. It's hard to do, and you can get better at it, and that feels great." Autonomy means we're going to open a hatch at the top of your career pipeline, so that when you are at the top of your pay scale, you can slide into the middle of the pay scale of the next harder level. If you want to go up in a belt rank in the dojo of engineering, we've got another team for you where it's harder. And you're a much greater investment because you're solving even bigger problems, and it's fantastic.

The reason I get so excited about this, and this is kind of heavy, so I realize you guys are...I'm usually the poop joke guy. But when I was at CMM, I had two people come...well, I had about seven or eight people come to me and say...this was when Ruby Rogues was really, really famous. I wasn't the most popular person on the panel, but I was there. And I would tell people, come work with me. Come work with me. Come work with me. We're fixing the healthcare system.

And now I'm telling people, "Let's..." At Acima, we are servicing a part of the economy that is wildly underserviced, underrepresented, and it is heavily predated. There are a lot of predators down here. And being able to operate in that environment and help people is incredibly emotionally fulfilling.

But the thing that knocked my head right off my shoulders was it happened back-to-back over a two-week period. I sat down with a guy, and we're just coding, and just out of nowhere, he says, "I work here because of you." I'm like, "Wait, what? I didn't interview." He said, "No, I was listening to your podcast. And I'm like, that does sound cool. I want to go do that. So, I'm like, okay, hey, that's really, really cool. I thought that was my shining tiara. I brought somebody in. That was great.

And then, two weeks later, I was programming with a woman. And that's significant because women are wildly underrepresented in tech. And we're typing along. And I don't want to niche or brag or politicize. She was also trans, which is even more underrepresented, and that's the only reason I bring it up. It's even more underrepresented. It's an even more hostile environment.

So, I'm sitting next to this woman. We're typing along. And she said the same thing, except I knew that Cover My Meds was her first job. And she didn't say, "I came here because of you." She said, "I got into programming because of you. You convinced me that I could do this. You convinced me that I could survive in this environment, and here I am." And I'm like, "Can I give you a hug?" I will carry that to my grave as one of the top ten core moments of my career.

Yeah, invest in people. It is so worth it. It is so worth it. That's my speech.

EDDY: I've said it before, and I’ll say it again. Dave was the first person here at Acima that taught me how to create a branch and push it up to that repository. No one before then had taught me how to do that.

DAVE: Awesome.

EDDY: And so, the fact that you did has gone and paid in credit, I think, so...

DAVE: And I forgot that we did that. And you were subtweeting me in the room. You were like, "Well, I got to be one of...there's an engineer. He sat down git da-da-da-da-da." And I'm like, "That guy sounds really cool. You need to do that for people." Eddy’s like, "Dude, it was you." And I'm like, "Wait, what?

So, yeah. Don't do it for the credit. Don't do it because you want. I'm just saying do it, and when it pays off, it feels amazing. Plus there's all these other programmers that are now going and having careers that are...and that's the real purpose.

MIKE: Absolutely. I've said it for years. It's the most gratifying part of my career is when I can help somebody be successful and see them move on. And when you see that happen, it's just incredibly rewarding. And it goes back to this idea of investment. You watch that investment pay off [chuckles]. You invested in that crypto coin 10 years ago, and now suddenly you're rich [laughs].

DAVE: Survivor bias fallacy. Yep.

MIKE: There are some investments worth making and investing in people because you care, I think, is worth it.

Now, this is not to say that we can function exclusively as an educational institution because that doesn't work either. As we talked about, if somebody is not ready and the business is there to make money [chuckles], that doesn't work. But if somebody is competent and willing and able to do that work, give them all the support that you can.

As Justin was saying, his time is best spent helping other people be more successful. And as you move up and move forward in your career, whatever it is, as you advance in your career, that becomes more and more true. The more that you can help other people be successful, you're going to be accomplishing more than if you're just walling yourself off into a corner and banging out some code.

DAVE: Stephen Covey, if there's no margin, there's no mission. There's times when the people have been bright and ready to learn. And we had two weeks to ship eight weeks of code, or we were all going to lose our jobs. We didn't do a lot of skill-building that week. We did a lot of overtime and a lot of typing because that's all the margin we had for.

MIKE: But when you have some, you pay down your debt, and you invest in the future.

DAVE: And there's a valuable investment to trauma bonding with your co-workers [laughter]. That's funny, but I'm not joking. That is absolutely a true statement.

MIKE: It is true. I laugh because I identify it, not because it was wrong. So, I think I've covered everything I wanted to talk about today. Anybody else have any final words they'd like to throw in here before we break?

WILL: Everybody should be doing XP all the time. Do it. If you have the power to do it, do it. If you don't have the power to do it, do it anyway. Just don't tell your boss. Invest in people. If you can’t invest in people, invest in yourself. There’s always this, like, because you're going to find yourself...As an individual contributor, you're going to bump up against just these things where it's like, what is that? What is that? What is that?

Just keep, like, a slush pile of things you're run into, either, like, a piece of the codebase, a library, just some kind of thing that it's just like, what the hell is that? And just keep a list. It's just a list of stuff, like, Google's I need to do, you know, in the future. And just, like, just check it out.

Just keep stuff in your down cycles because you're going to get them, right? They're going to happen. And it's always about, like, you know, getting better and getting smarter and building extra understanding. Because, you know, civil practical fact, right, like, none of the work that we do is really that complicated. None of it's that hard. It's really pretty simple.

Like, most of the stuff you did in school is more complicated. Like, I built an operating system, like, an entire operating system with a memory manager in school, and I don't have to do anything like that at work, you know. But I do need to know how a bunch of, like, dumb business logic operates. And, like, how does the DI that gets our cookies to authenticate a user, like, how does that get initialized, and when? Because if it's not ready, then you don't log in. That's a problem for some people.

And there's nothing about it that's special. You just have to know. And, like, what's the difference between, like, the best guy in your team and, like, the worst guy in your team is, like, the best guy in your team, like, he paid that toll already, and he knows. And, like, it's just a matter of showing up every day and, like, just getting a little bit smarter.

I mean, because, like, to be perfectly honest with you, like, completely average developers can become exceptionally productive, and valuable, and talented ,and have wonderful careers, just by doing, like, a little bit extra. It's not that much work, but people won't do it. And it's a matter of investing in yourself, you know. And that kind of stuff is entirely within your control.

You know, XP now, XP forever.

MIKE: I’m not going to argue about the pairing [laughs]. So, I'm going to conclude by supporting that. So, we've got Jordan here, who was an intern recently. And one of the best things that happened to me this summer is, at the end of the summer, the interns were doing their presentation. And they brought up pairing on their slide and said, you know, we've been doing this all summer, working almost the entire time pairing. And I was like, yes [laughs], yes, learn how to work together. That's, like, one of the most...maybe the most important thing you could have learned this summer, and you did. It worked.

JORDAN: I want to say that this summer, compared to last summer, just felt so much more productive. And I kind of...I low-key became kind of dependent on pairing because we would have the hour in the morning before stand-up where we just worked on our own stuff, like, I don't know, catching up on our tickets or, like, the Exercism.

And I would sit there, and I'd be like, well, I can't get anything done. Like, I don't have Chloe sitting here, like [laughter], to discuss on what we should be doing next or, like, what we're currently working on. And so, we’d pair, even when it felt like there was nothing to do because we could talk and, like, find something to do. And that just felt so much more productive. So, I'm a big advocate for that [laughs].

WILL: Yeah. And keep a slush file. Every day, and usually a couple of times a day, I'll come up against something that's just like, how's that? What did you do? And there's always a story, right? It's always a story. It doesn't matter whether it's, like, it could be, like, a Git, Jira spelunking story, or it could be, like, some kind of trip thing, or it could be, like, some technical debt that maybe needs to be on the list. Just keep a note file of just like, what the f*** was that? You know? Like, I can't encourage it enough, you know, because there's always something.

EDDY: I will say, Jordan, you don't necessarily need a Chloe to talk to, right? Like, you need something just arbitrary just laying next to your desk, you know, something where you can express yourself to. I mean, I can't count the number of times that, like, I started typing out a message to someone, and I'm trying to articulate all the things that I've done, right? And I'm like, dude, I'm stuck here. I can't figure this out. I've done this, this, this, this, this, this. And then I'm about to send them like, ah, wait, you know? And I just laid everything out, right? And --

WILL: ChatGPT is my work wifey [laughter]. Not like that. Not like that [laughter]. But, like, I can [crosstalk 57:14] you know what I mean? Like --

MIKE: Probably a good slot to close. The social aspect matters. Invest in your people.

Until next time on the Acima Development Podcast.

DAVE: Cheers.

In this episode of the Acima Development Podcast, Mike hosts a roundtable with Eddy, Dave, Chloe, Jordan, Ramses, and later Will, to explore the differences between small and large projects. Mike kicks things off with a personal story about industrial dishwashing as a teenager and how that shifted his perspective on “small” problems like doing dishes at home. He ties this to software by noting that scale fundamentally changes how you approach problems, whether that’s navigating your neighborhood versus traveling cross-country, or building a small app versus managing a complex, multi-team system. The theme: what works at one scale may feel trivial or even inadequate at another.

The conversation then moves into real-world engineering trade-offs. Dave contrasts working in application development versus database engineering, where different constraints (compute vs. storage) flip what’s considered efficient. Eddy stresses the importance of incremental, low-risk changes in large systems, while Chloe shares how her internship taught her that upfront design decisions carry far more weight in big projects than in small, flexible ones. Will and others riff on the “temptation of perfection”—the idea that if you just planned enough, you’d get everything right upfront. Instead, they argue for agile approaches: accept mistakes as inevitable, design for change, and value rework as part of progress. Several panelists tie this to the importance of embracing ignorance, asking “stupid” questions, and adopting a growth mindset to avoid paralysis when decisions inevitably prove imperfect.

The group also digs into the human dimension of scale: coordination. Jordan notes that large projects require consensus and approvals, whereas small projects let you move unilaterally. Mike reframes bureaucracy as a natural outcome of needing structured communication and shared understanding, not just red tape. Dave and Will debate decision-making heuristics—how long to defer choices, what counts as “enough information,” and how to balance planning versus execution. They conclude that small projects build the fundamental skills and intuition needed for larger systems, while big projects demand humility, adaptability, and collaboration. Ultimately, the episode emphasizes that success isn’t about perfect planning but about making good-enough decisions, coordinating effectively with others, and being willing to course-correct as conditions change.

Transcript:

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. With me, I've got Eddy, Jordan, Dave, Chloe, and Ramses. Now, Jordan and Chloe are our interns. You all were on before, right [chuckles]?

CHLOE: Yeah.

MIKE: So, it's not the first time. It's not the first time here. But today is the last day of the internship that we had for this summer. And as a result, we're choosing a topic that I thought would be really relevant, something that we've talked about some with the interns, if they can give us some insight as to what they've learned. And having somebody new on a problem often reveals things that other people don't see. That's why sometimes grad students make good teachers [chuckles]. They're fresh to it, you know, they haven't been so distanced. So, I'm really interested in hearing about this.

Notice I haven't mentioned the topic yet. I'm going to give a little intro to introduce here. And I'm going to talk about a summer in my teen years where I worked at a Mexican restaurant [laughs] washing dishes, not just washing dishes. They also made me a prep cook and I heated up, like, beans [laughs]. But I washed a lot of dishes, and this was, you know, industrial dish washing. You can imagine what the pans looked like after they've been cooking whatever Mexican food they were cooking [laughs]. Just imagine, you're probably right.

EDDY: Did you use a pressure hose? It's probably the best way to clean dishes, especially large dishes.

MIKE: So, we had two things that happened. So, there was the customer dishes that came in, and those we had a big industrial dishwasher that we kind of did a conveyor belt style. You’d load them in, and they’d go in there, and they’d steam, and spray a bunch of water, and that worked pretty well. Except if you have, like, cheeseburgers, you know, cheese would melt onto it and stick to the plates [laughs]. I see Chloe smiling. She may have done this before [laughs].

And on the other side of the area where we did the washing, there was a series of three sinks, and that's where all of the stuff from the cooks, right, that's where that stuff would get cleaned. And that was the gnarly stuff. And a pressure hose...there was nothing that would take gunk off a pan when it first came over. And this wasn't, like, very authentic Mexican food [laughs]. They had, like, I don't know if it was blackened, you know, Cajun, but along those lines kind of fish that they would do. And they’d burn it all on the pan, all the seasoning that they would have around, and then we'd have to clean that off. So, imagine burned on fishy oil with spices and gunk. So, first sink was a soak with lots of soap. Second sink was for rinsing. And third sink was bleach. Soap, rinse, bleach.

And I’d spend hours every night. Well, before I ended up, like, hosing down the floor and stuff. Weird smells come out from behind kitchen equipment when you hose them down at the end of the day [laughs] [vocalization]. It's unrelated, just something I think of sometimes, that smell [laughs].

This is this industrial operation, right? And I would spend hours doing this, you know, the soaking and then you go to scrub the thing and if it wasn't coming off yet, then you’d go back in the soap soak. And, eventually, it was like magic. You’d soak it long enough, even the worst caked on stuff would eventually come off. And you'd rinse it and get it over to the bleach and out, and rotate back around to the cooks. And I got used to this. I'd been doing this for weeks.

And then came my rotation, remember, this is my teens, came my rotation to clean the dishes at my parents' house. And I went to do the dishes, and I just started laughing. It's like, what are these? Toys [laughs]? It's a little scrub brush [laughs] and, like, a sponge. Like, I used to think this was a chore. This is going to take me five minutes. What kind of problem is this [laughs]?

And it totally changed my perspective around doing the dishes. This is an entirely different job. I have different tools. I don't have real tools anymore, like, and I had to, you know, find the scrub brush. Like, okay, the scrub brushes work better at this than the sponge [chuckles], but still, it just all felt like toys. I laughed. Every time I did dishes that summer, I laughed because it was just so much of a different problem than what I was doing at work.

Today, we're going to talk about big projects versus small projects. And we're not washing dishes [chuckles], but maybe somebody's doing software for, like, a microcontroller for a dishwasher or something. So, maybe somebody out there is washing dishes. But a lot of these principles apply. The scale really matters. And this comes up often, actually, in our podcast. And you’re like, oh yeah, scale matters. It came up recently when we were talking about vendors. If you are very differently sized from a contract shop, then you're probably not going to get the right level of service because they don't cater to your needs. And when you're building stuff, scale matters, too.

And I've got a lot of thoughts on this. You know what? I'm hosting, so I'm going to throw one other thought out here: Navigating inside of a neighborhood. In fact, this is something I've seen in my neighborhood. Where I live the most prominent landmark is the village water tower. We've got a big water tower. It's one of the larger ones in the region, and you can see it from almost everywhere in the neighborhood.

We have a daughter of a friend who once got lost because she got into a part of the neighborhood where she couldn't see the water tower. And, you know, there's a little bit of hilliness, and so she got to a low spot. It's like, I don't know where I am [laughs]. And she was lost for a while because she had nowhere to go.

DAVE: GPS is unreachable.

MIKE: Yep [laughs]. And, you know, she may have been, you know, she was probably, like, she may not have had a phone. This was probably 10 years ago, maybe a little bit before everybody had a phone [chuckles]. She didn't have a GPS. But yeah, the water tower is the GPS that that's where you're going --

DAVE: That's what I mean, yeah. That’s what I meant, yeah.

MIKE: The water tower is the GPS. And so, yeah, she didn't know where to go because navigation is different when you've got a small scope versus a large scope. Once you get outside of that, well, you actually have to know which way you're going. You have to know where your cardinal directions are maybe, right? You have to recognize more landmarks than one. And, you know, traveling across the country requires a different set of skills than walking around the block. They both require skills, but one of them is fundamentally different.

And a lot of times it is hierarchical in that some of the skills that you would apply for that navigation of the, you know, of the small scale map may apply to large scale map, but you're going to have to say, “Okay, well, I need to get from Chicago to Des Moines.” And you figure out at a high level where that's going to take. But then when you get down to the actual implementation, you know, you have a detour. And then you switch down to a different level, and then you have a smaller map, a smaller map in your mind that you're dealing with. So, your hierarchy changes. So, you have to break down the bigger problem into smaller problems.

I wanted to throw out a couple more ideas to kind of get the thoughts. Also, Will has just joined us. Will Archer, welcome.

DAVE: Welcome.

MIKE: [inaudible 08:22] So, glad to have him with us. So, I've talked about big problems versus small problems. What are y’all’s thoughts? You know, open discussion.

DAVE: I will throw a weird curveball into this, which is that I just came here from a meeting with our DBA, Bill. And I think I've mentioned this before that, like, when I first went to the data team, it shocked me that on AppDev, we spend all our time...everything has to be small. Don't do big joins because you've got to get to the database, get your thing done, get back out. It's synchronous. It's, like, single threaded. We don't like doing big updates. We like doing small batches, everything.

And then behind the data center wall, compute everything because storage is ridiculously expensive. And when I went to the data team, they were, like, yelling at me, like, “Stop that. Stop that. Stop that,” because storage is free, and compute is vitally expensive because we're spinning up cloud servers [inaudible 09:19] down. And it was mind blowing to me. And we approached everything...I was so excited when they did, like, this 30-table join, and it came back in, like, one and a half seconds. And then, on the flip side, we tried to do select count from this table, and it took one and a half seconds. It's like you could do anything in one and a half seconds, but you couldn't do anything in not that amount of time, and I'm like, that's the difference. The get in, get done, get out is low latency. So, it's very, very orthogonal thinking is how you come to it.

And it was kind of exciting to talk with Bill about, like, “Well, how would you refactor?” And he's coming from a world where they would take the whole system down, like, to rename a column, take the whole system down, rename the table in the database, rename the table in the code, bring the whole system back up. And this would be after, like, a week of testing it in a replica to make sure that the name change was going to work.

And I'm like, yeah, we don't do that. We're coming from, like, finance and healthcare, where it's like, your system cannot ever go down. So, if you want to rename a table, you're looking at five separate deploys, you know, add the column; backfill the copy; add code to shoot to both places and to read from, you know, and to read from the new place. And once you're syncing always, then you start reading from the new location only. And each one of those is a separate deploy. And yeah, he was shaking his head. And he's like, yeah, that's fair. So, instead of big versus small, it's like breadth first versus depth first approach to covering the entire graph.

MIKE: But it requires different tools.

EDDY: Going the slow but sure route, I think, is the one that I've always appreciated the most, meaning don't go destructive; get it done in a day, but rather do it in a five-step process to make sure that everything you're doing is correct so that you don't lose any data, right?

DAVE: Mm-hmm.

MIKE: That's interesting. You're absolutely right, Eddy. You have to do that on a big project. But what if it’s your own personal project that hasn't even gone out yet? Do those same rules apply?

EDDY: If something's not actively being written to production, is that what you're saying?

MIKE: Yeah, that's right.

EDDY: Then go as crazy as you want, I think. You don't have any sort of sensitive data, like, actual factual data from customers or clients, right? So, I'd say you're free to play around.

MIKE: Now, Chloe and Jordan, you have been working on a project most of the summer, and that scale is different than what you do at school. What have you learned about working in a larger project versus a smaller one?

CHLOE: I think something that I've noticed is that the way that you design things and those decisions at the beginning can have a large impact on the project. When something's really small, if you're, like, I'm going to structure it, you know, this certain way, it's halfway through, and you're, like, hmm, maybe that's not the best, it's really easy to change. Versus a large project, those decisions are usually pretty concrete by the time you realize you might want to [chuckles] change it.

And so, I think it puts a lot more emphasis on the front end of thinking through the process of what's the overall game plan of this project? What do we want it to look like? Do we expect it to grow? Do we need to look at tools that can grow with that? Or I think there's just a lot more questions you have to ask yourself before starting versus when it's smaller, you can kind of just hit the ground running and then deal with problems as they arise. But you're not always afforded that luxury with a large project.

WILL: Ooh, so waterfall.

CHLOE: Yes.

WILL: The siren song of the waterfall.

MIKE: [laughs]

WILL: [inaudible 12:52] sensation to just be, like, okay, listen. So, listen, what if we just didn't make any mistakes in the beginning and we knew [laughter] everything that we were going to do? Despite being literal students, right, and this being a learning project, we're going to get it. We're going to nail it. We're going to tie everything [laughter] together. And it’s just going to be like, boom, bullseye right in the very beginning.

I think that is a very natural impulse that will be with you for your entire career. Like, you're always going to be, like, oh, what if we just didn't screw it up, you know, essentially [laughter]. It's a temptation. It's a very fundamental, natural human temptation. And I think, you know, you could get better and better and better on it. Because you go on and you have better instincts. You make better, like, just sort of, like, rules of thumb, you know, kind of stuff.

We talk a lot about, like, carpentry and building houses and, like, physically constructing things. If you’ve ever watched somebody, you can...you can watch people who are really, like, just good and, like, good framing carpenters, and they can just throw up a shed. And they just kind of know where everything's supposed to go. And it all just fits together, and it's right. And it makes me really mad because I'm okay with effort. But you’ll get better at it, but you're never going to be good at it [laughs].

And it isn't, like...the solution isn't, like, don't try because absolutely try. But there's also a feeling of, like, you know, let's just go and learn. If you accept refactoring and rework and reevaluating of your prior preconceived notions as a necessary process, when you hit that point, and you’re dead end, and you need to turn around, turn the car around and drive back, you'll identify it, and you'll do it, and when it's time, as opposed to, like, just, like, okay, I'm just going to muscle through. You know what I mean?

CHLOE: Yeah, I think that makes sense. I think it's inevitable that you're going to make decisions that eventually you might want to change. And so, instead of, you know, having this idea of no, we're going to make this decision now and it's going to be perfect, embrace that things are going to change and kind of learn to work through those things. I agree. I think that's a good point.

MIKE: The agile software paradigm in a nutshell.

DAVE: That actually came up in my chat with Bill that his database thinking is old school. The reason I had the chat with him was because we're migrating phone numbers in our database. And we're literally moving from one table to another. And I had a chat with him of, like, we've been working on this for, like, six months, and, you know, what should we have done?

And I realized when I invited him to the meeting yesterday, I said, “I'm going to ask you what we should do. And I realized that the first thing we need to do is jump in a time machine and go back and not forget to have asked you what we should do before we do it. Because it's so much easier when you get the design right first.”

EDDY: It's so much easier to let someone else do the design that they're really good at to do it for you than [inaudible 15:55] [laughs]. Agreed.

MIKE: But it's inevitable. And we're talking about large projects. It's inevitable that on a large software project, the business domain is going to shift. So, even if you made all of the perfect decisions on day one, which you didn't, the ground is going to shift under you when what you were building --

DAVE: And if it doesn’t, the company you're at is going to go under. Sorry, if the software you're building doesn't shift, the company you're on is going to go under because the world is changing every day.

MIKE: Well put. You build really good horse carts, right [laughs]? You may have it perfect. You've still got a problem.

DAVE: Yep. I make the best abacus in town.

MIKE: [laughs] That necessity of change is going to come up on you. Tech debt can hit you even if you didn't know that you were going into debt in the first place, just because of that change in the world.

So, you know, we've talked a lot about this, Chloe. You talked about, yeah, I want to get things right in front. You do, and then you can't, right [chuckles]? You do all that work, and then you find out that you were given an impossible task. As Will said, you still try, and you do the best you can. You still measure twice, cut once [inaudible 17:22] carpentry [laughs]. It still might be wrong a year from now.

WILL: This is maybe advice that, like, maybe is specific to me, but I think it generalizes, like, at least decently. I've always been smart, and I've always been good at my job. And I've always been known for being smart and for being good at my job. And I think a lot of people in the business self-identify as smart and confident. And it leads me down a dangerous and destructive path when I fail to embrace my own ignorance and stupidity [laughs]. Embrace stupidity. Lean into it. Lean into it.

Like, you get used to being...you can get too comfortable being smart, especially if you're really smart. You know, like, we don't work with...nobody in the office...if you walk down the office, right? There's nobody...nobody is dumb. And there are very, very few people, maybe in sales, I don't know, who are average, you know? There's nobody. And so, you know, yeah, embrace stupidity.

MIKE: I'm going to build on that because I think that is a profound concept. Have you read any of the work of Carol Dweck, a researcher in learning?

WILL: The name is familiar, but I'm ignorant of her work.

MIKE: She’s done a lot of growth...I think I’ve got this right [chuckles]. She's done a lot of research on what she calls a growth mindset. And what she's found is that people who think they're dumb and people who think they're smart end up running into the same dead ends because the kids who think they're dumb, well, they don't try. The kids who think they're smart have their self-image invalidated when they make a mistake. And so, they don't try because they think, oh, I'm actually not smart. I'm not good at this. And they don't like that feeling, and so they stop.

My recollection is that this actually tends to affect girls more than boys because a lot of parents are like, “Oh, you're so pretty. You're so smart.” And they say a lot of “You are this,” rather than “You did this,” and that can harm the girls, right? Like, you think, well, no, there's nothing wrong with being told you're good. But it can actually be counterproductive when you say that you are frozen in this good state rather than saying, “Wow, you are a hard worker.” Now that completely changes the way you see the problem, right? Like, “Wow, you made a lot of mistakes today. You are working hard. Look at those scrapes you've got. You are tough.”

That is a fundamentally different way to approach the problem and is much more healthy. And the kids who think that way, that growth mindset, are the ones who tend to succeed. And that matters a lot when you're in school where you have to grow a lot. But then to Will's point, we're all going to get stuck. And if we don't recognize that, yeah, we're going to have to be dumb sometimes; we're going to have to be uncomfortable and grow, then we've run into exactly that problem.

EDDY: Yeah, the best advice I was given when I first started is, don't be afraid to sound dumb, right, kind of to add to that, right? Everyone has come from that point, right? And the sooner you accept the fact that someone who's been an engineer for 10 years is also dealing with the same thing as you are, things become much easier.

DAVE: There's an easy thing that I...it was probably me who told you that because I'm --

EDDY: Probably.

DAVE: I’m still happy, Eddy. About six months ago, we were standing around talking, and I asked a pretty obvious question. And you looked at me and said, “You are fearless about asking stupid questions.” And then three people in the group said, “I don't know the answer to that either.” And I'm like, not so stupid after all. And Eddy wasn't asserting that. I'm, like, no, that's actually a pretty smart, stupid question.

And so, the hard work versus...sorry, getting it right versus dealing with it, in my opinion, that's the same shape as being told you're perfect and not wanting to risk damaging that, which makes you nonfunctional and less productive and less successful versus being told if you're here, you need to get here. Just take one step up, and then if we come back to you in five years, you're going to be way ahead of the other people because every day you take one step up where the people who were smart to begin with haven't been.

And this is why I wanted to say this is getting the design right the first time is getting the design perfect the first time. When I put the phone number thing in, it's currently running in the most awful way possible. And I'm proud of it, so I don't feel like this is me shaming the company. We currently have data living in two places, and it's ugly, and it's messy. And both Rails models have triggers in them that when I get updated, I'm going to go check the other data location and if it doesn't match me, I'm going to fix it.

Well, first thing I'm going to tell it, “Don't update me back because you have the same code to update me when you save,” right? So, this data is constantly fixing. And basically, anytime you touch one of these things in the database, it settles with the other one. So, literally to backfill the data, you literally can just say, “Load an object, touch it, save it. Load an object, touch it, save it, load an...” And that's literally the backfill because the data can repair itself in motion. And all you have to do to backfill the data is just touch all the data, and that's great because, now going forward, any new data going in the system comes out correctly.

I went on leave for four months. I came back. That code is still there. We are running on that little donut spare tire for this data. And there's a part of me that's going, this design is terrible, and it's inefficient. There were many extra cycles...da da da. And then I'm going, we are one step closer to having the database right, and the website works. We're selling leases. We are giving people the things that they need. We are taking care of business. We have cash flow. And I love that. And, longitudinally, we are headed towards the right data schema, database schema, and I love that.

I would much rather be good at correcting an error than getting it right the first time because getting it right the first time doesn't scale.

EDDY: Well, I'd be surprised if any of you have been part of a large project where you got it right the first time. I mean, I'm happy to be proven wrong.

CHLOE: I certainly have not [laughter].

EDDY: Yeah, no. I'd be happy to be proven wrong. I think that's a very fair assumption.

DAVE: There's a rule that we found in the ‘90s, which is that a waterfall software process works very successfully if you have built it three times already. So, if you're making version 4 of the thing and it's the same thing, you're not reinventing it for a whole new world, right? It's like nobody has built 3 AI embedded products. That's not what I'm talking about.

I worked on a company that was making graphics cards, and we were making the 3K version, which was after the 2K, after the 1K. And we've reused all the same processes because the way you manufacture silicon hadn't changed. And so, waterfall worked really, really good.

Do you know where we ran into trouble? That 10% of extra stuff that was all new. And we ran into all kinds of trouble, some of which broke the silicon. And I stayed up late one night swapping red and green in the software because somebody had made a change upstream that was now using the wrong channels for the colors in the silicon. Couldn't be fixed in the silicon without turning the chip in the fab, which was going to be, like, a $10 million impact. So, they're like, “No, fix it in software. It will be all right.” We got it wrong.

MIKE: To pull this back to where Chloe originally started, well, you have to think about the decisions beforehand. Well, you do. But you need to make a good decision, not a perfect one. And those are different [chuckles]. You do, in a large project, need to make a good decision. But it's infeasible to make a perfect one, so you shouldn't try because you can't. And it is a fool's errand, and you will end up spending all of the resources allocated to your project trying to make it happen in planning up front, and then you'll still be wrong.

EDDY: So, Mike, are you saying that a good decision implies that you can be agile about your design?

MIKE: I am. Because if you head out in a good direction knowing that some parts of it are going to be wrong, then you step in knowing, hey, I'm going to have to change. I'm going to have to take feedback and course correct over and over again. But that means that at any point in your project, from inception to finish, you're continuously pivoting. And you never reach that perfect ivory tower; instead, you make money. You actually bring in value [laughs] to your employer.

DAVE: Have I told you my mantra on...you get told, if you don't have time to do it right, when will you have time to do it over? That is waterfall thinking. Agile thinking is, if you don't secure the cash flow right now, when are you going to have a paycheck to do it right?

WILL: So, the question I've got, right, like, I love the conversation, right? And I feel like, okay, you know, smart, pat on my head. I made a good point. Everybody likes it. So, you know, I'm being applauded in my mind. My therapist would be very proud of me. But, like, I think, okay, let's take it a step further, right? We said, like, don't do this, right? Like, don't plan too much, right?

And balance in all things is, like, the essence of engineering, right? So, we say, don't plan too much, right? And I think everybody [inaudible 27:29] on that, right? But, like, don't not plan, right? And so, like, how do we balance the scales between, like, okay, you know what I mean? Like, I'm not just, like, shooting off half-cocked, but I'm also, like, not analyzing this thing and creating a 100-page PowerPoint before I even break ground, right? How do we balance that?

DAVE: I have a quote from Sandi Metz. Agile means deferring a decision to the point of maximum information and no further. That's the key, right? It's deferred as late as possible because information always goes up over time. But she does say --

WILL: I don't like that at all [laughs]. I hate that. Maximum information?

DAVE: Everyone does --

WILL: Maximum information. Like, that's, like, I don't...I am deeply unsatisfied by that, but I bet you have more to say.

DAVE: So, information increases over time, but at a certain point, you've lost the opportunity to make the decision because things have just been decided for you over time. And so, that's the ideal perfect time because, like, the schema is turning, turning, turning, turning. No, we have to shift, but now you have as much information on what the correct schema to go with is.

You are absolutely right. She's not saying don't make a decision. You do have to make decisions. And a lot of decisions if they're not impactful, just make them, right? It's pushing not all decisions are the same.

MIKE: Procrastinate as long as you can get away with.

DAVE: Procrastinate when it's appropriate, which is probably more than you think.

WILL: I find this deeply unsatisfying, and I'll tell you why. I think the fundamental thesis is right, but I don't like it as a heuristic because I think what I believe will eventually...if you're a perfect person and you're diligently following, like, that thing, what you'll run into, I believe, is the uncertainty around the information that you can get where it's just like, I don't know. How's it going to work? I don't know. What's going to happen here? I don't know. I don’t know.

DAVE: Not all information is equivalent, yeah.

WILL: You know what I mean? Because you don't have enough data. The uncertainty, I guess, the signal-to-noise ratio gets to the point where, like, you're making decisions, you know what I mean? And there's just no way of...there's no way of possibly knowing, not because, like, the information is not out there, but because you don't have it, right? You run afoul of...you just have to have a lot of wisdom and professional judgment.

And, like, you just need to have been around the block a few times. And you need to have, like, very much that beginner's mindset to be able to have, like, both the wisdom to say, like, “I don't know. How would I possibly know that?” And the judgment and the courage and the humility to say, like, “I don't know.”

And, like, to stand up to say, like, “I have no idea what the answer to that question is, Mr. Vice President. I would like several million dollars of your budget so that we could just, like, go off, you know what I mean, and see if this project lands.” And there might be some dumbass in the seat next to you without the wisdom or humility who's just like, “No, I know, and we're going to do it my way.” I think it's blind human nature and the sort of, like, emotional, psychological, social forces that these decisions are going to have to be made in. Does that make sense?

DAVe: I think so.

WILL: You know what I mean? Like, yeah, you're right.

DAVE: I think so.

WILL: We [inaudible 31:16] work.

DAVE: Yeah, I think not all decisions are the same shape. Like, I worked at a company where the CEO wanted us to look really good to the investors, so he splashed for a SQL server. And it was a nightmare. It wasn't easy to match. There was a lot of impedance. And because we went with SQL server, that forced a lot of decisions that were free decisions until he made that one. And five years later, we moved everything to Postgres. And we moved things to Postgres here at Acima. I'm not talking about Acima, just to be clear.

But that's the kind of thing. Like, if you make a decision I'm going to do it this way, and it's going to lock me into 73 other side effects that I haven't even considered, that maybe was a decision that could have been deferred and studied a little bit better. I'm not saying push everything out. But I would say, by default, I tend to push decisions back unless there is, like, no, this has to be made, or I can see that the decision has very little cascading downstream effects, if that makes sense.

You could probably use some sort of heuristics or some sort of rule of thumb around, like, percentages. Well, this project is, well, this project's got a $500 million budget. So, you have something outrageous, you know, some big government contract. You probably want to spend more than five minutes making that decision. If something’s got a $2000 budget, you probably don’t want to spend more than a few minutes making that decision.

DAVE: Absolutely. I wrote a script to keep track of what branch I'm on in Git. And the very first version of it, I used YAML store, which is literally just a YAML file as my database, and that worked great for about six months. And I eventually had to upgrade to SQLite. I didn't want to go all the way to put a database server on my thing. So, I was like, yeah, let’s just install SQLite. I can do that. And that's enough SQL to do the things that I'm keeping track of.

But what I needed out of that decision was for the decision to be made quickly and to not slow the little project down. And that could have, like, if I was running a production business and I had made that decision, we might run into all kinds of scaling problems. Because pretty soon it's, like, well, we don't want to change databases, so we're going to write all this stuff to deal with the fact that we got this wrong. And welcome to the rest of your career. It's going to be solving your solutions is so much more painful than solving your problems.

MIKE: Going back to what I was saying about the different scales, you can say it in percent. What is it, 5%, 10% of your time should be spent up front making some decisions. And I don't know exactly what that number is. It shouldn't be 50% in general [crosstalk 33:56].

DAVE: I would be tempted to think the number changes, but yeah.

WILL: I mean, I would say, like, rather than, like, sort of, like, a percentage, right? Like, I would say I want a story for, like, a thing. Like, this is a thing, and this is going to happen. And somebody is going to open thing up, and they're going to go on a journey, right? They're going to go on a journey. And, like, I'm going to, like, I want to make a web app. Like, okay, well, what's the first thing I'm going to do? I'm going to log in, right? And then I'm going to see, like, a to-do list, right? Something dumb. I'm going to make a to-do list. I'm going to make, like, a photo, whatever you're going to do.

Like, make a story for, like, this is the person, and they're going to do the thing. And then what are they going to see, and what are they going to do, right? Like, that's...like, it's a school project, and I'm going to write, like, a database. And I'm going to enter data, and I'm going to have a row. And it's going to be a thing. Just have a story of, like, just what this is in plain, simple language.

And then, the story gets more or less detailed when you're in a big organization where it's, like, okay, well, I'm going to go in here on the website. And it’s going to be this team, and it's going to be this page. And this is what we're going to do. And then I'm going to talk to this service. I'm going to talk to this thing. And this isn't...it sounds like, oh, this is a waterfall, but it's a simple algorithm, like, the planning that you're going to need to do in a big organization versus a little organization.

Write out the steps. Don't have a lick of code. Like, I'm talking about plain English and napkin drawings. Like, you can do all this stuff on a whiteboard. Write it out. No code. Nothing. You’re just like, I'm going to talk to the Prometheus service to get a user log in because you got to do that. And you know where you have to go, or you should know before you're building it, right?

DAVE: We might be in violent agreement, Will. When you say push the design out and do it on a napkin, that's what I mean. You have just avoided choosing a tech stack and a server host. And that's the decision I'm talking about pushing out.

WILL: Yeah. I mean, but just a story, like a story, you know what I mean, like, yeah, nothing. No keyboards. Write it out with your fingers, those little [inaudible 36:16] popsicles, and, like, write it out. Go ahead.

MIKE: Are you suggesting, Will, that if you can tell the story, then you're at the point where you can make a suitable decision?

WILL: I believe, like, you'll write out the story, and then, like, somebody in your team will be, like, well, what about that, right? And you'll flesh out the story. Well, what about that? And you'll flesh out the story. And eventually, and this is, like, woo-woo and vague, and, like, okay, it is a craft that you learn, you know what I mean? You’ll get to a fixed point. Like, you’ll get to a fixed point where, like, you’re going to know what you're doing, you know what I mean?

And then, I think, some of it, unfortunately, I hate to like [inaudible 36:58] vibes out there, vibes. But you're going to start feeling some vibes, you know. And you will also blow it because it's just a story. It's a napkin drawing. And you'll be, like, oh, well, what do we think of that? Like, yeah, it's a story, you know.

MIKE: Emotions are sometimes looked down on, but they're there for a reason. You have those vibes because it's telling you something. So, I don't think we should be ignoring them. We should be using them to their maximum potential. And if you reach the point, like, hey, this feels good; this story resonates, that's probably telling you something.

DAVE: What you're saying, Will, matches very strongly with my view of, like, top down versus bottom up. When you build bottom up, you start with, like, the write me an adding machine. Now write the multiplier. And at every step, you can prove that what you have built works. When you start top down, you might get to the bottom and find out you can't actually do it. You made an assumption about the latency of some system or other. But you do know from the very beginning that you're building the right thing.

And we've all had that thing where you build top down and you find out that, oh no, the database literally doesn't have enough latency to support this entire approach. We've also built bottom up and then found out that we climbed all the way up the ladder and found out the ladder was on the wrong wall.

MIKE: We’ve really run with Chloe's idea [laughs] about planning ahead of time and how that really is dramatically affected by scale. I'm curious, Jordan, what thoughts you have about the difference between large and small projects?

JORDAN: Yeah. So, I would say that the biggest thing I face, well, not face, but experience, and maybe this is just, like, the difference between working, like, by myself versus working in a business, was the need to get approval on many ideas and, like, changes to move in a certain direction, I think. So, like, similar to, like, the architecture and design, like, if I'm working on my own small project, I'm allowed to do, like, any change I want, like, willy-nilly compared to, like, here. Like, if I want to do something, I need to, like, propose it, like, architect out, and then, like, talk to people and get approvals there.

And so, I guess that's, like, I can have a radical idea, and then I present it to people, and then it's more, like, normalized into something that might work better for everyone, I guess. So, I guess that is the biggest thing I've experienced.

MIKE: You know, as I was preparing to host today [chuckles], I wrote down some notes. And one thing that occurred to me is that on a large project, you deal with coordination, which is a challenging human endeavor. And, you know, solo project, you're not coordinating with anybody but yourself. But when you get to a certain scale, you can't do it yourself. And now you have to add in another layer of complexity, which is how do I make this work with other people [chuckles]? Maybe I need version control so I can share this with somebody else, so we don't step on each other's toes. Maybe we need an API contract that we agree on so that [chuckles] when I send a message, they can actually receive it.

And working together is hard because you have to get shared understanding, and that takes a lot of work. It's the only way we can build big things. And the more big and complex, the more work that requires because the things have to be precise, right? And you have to have an interface that actually aligns.

I think that sometimes we think, well, there's all this bureaucracy. And, I mean, bureaucracy can be heavy handed. It can also be a representation of the natural work involved in coordinating across groups. You're going to have to have hierarchies of information that travel upward and then are coordinated between each other, which sounds a whole lot like bureaucracy. And it exists for a reason. Now, it can be misused like any tool, but it's going to have to exist because the alternative is anarchy, and then you can't coordinate anything at all. And that's not our goal.

And Will made the point about balance being so fundamental. I agree on that in many ways. I think finding a balance is tremendously important. But my key idea here is that you're going to have to do some coordination. And the bigger the project, the more coordination you're going to have to do.

EDDY: Well, you have more hands in the pot, right? So, you've got to satisfy more people.

MIKE: And that's not free. So, yeah, big project, you're going to have to get signed off. You're going to have to get people to agree with you. You might have to find a consensus or at least get somebody to make a decision [laughs].

DAVE: Jean-Paul Sartre, "Hell is other people," or in our case, hell is other people's code.

MIKE: On the flip side, I was reading somewhere sometime in the last month. I would have to go find the code. I don't have the source on it. But it was a researcher on human happiness. And if I remember the quote correctly, it said, "Happiness is love: Full Stop." Which is to say that hell is other people and so is heaven.

DAVE: Yeah. Yep. I heard a good one on this today, or last week, which is happiness is your circumstances minus expectations. I thought it was kind of interesting. Being present where you are instead of looking over the shoulder of the present moment to see the next problem that's coming.

MIKE: So, Jordan, you talked about the coordination that is involved in working on a larger project. What does a big project look like structurally versus a small one?

EDDY: I'm not exactly sure I understand your question. I mean, if you were designing a new service or whatever, I'm sure you have stipulations on things that you have to meet. In order for that service to work, you have certain metrics. You have certain clients that you work with. You have certain design that needs to fit. I'm not entirely sure I understand how broader than that you're asking.

MIKE: Well, let me give you an example. Let me give you my example. So, at Acima, we've done a lot with Ruby on Rails. It's been a popular web framework for a couple of decades now. And it gives you its particular take on model-view-controller architecture. Here's where your database interface layer goes. Here's this models directory. Here's where your interface goes to the outside world, you know, your controllers. Here's your views. Here's your templates that you're going to use to display things to your end user.

That works really good for a small project. You build a blog. You can throw everything in that structure, and it just works. If you have a large multi-year project, it doesn't answer hardly any of the questions because most of your business logic doesn't quite live there. And it might be needed to call from background jobs, and I could go on and on.

And so, you're going to build a massive set of business logic, some sort of structure to represent your business logic that is behind the scenes and not in that model-view-controller. Now, you might think of it as a logical extension of one of those. But if you try to cram that all into one of those out-of-the-box layers [chuckles]...I'm seeing looks of horror [chuckles] from people on the call. It ain't going to work. The scale deeply matters to the architecture, to what your code looks like.

MIKE: I heard this talked about by Uncle Bob. What's his last name?

DAVE: Bob Martin.

MIKE: Martin. He said that if you look at blueprints, you can generally tell what kind of building it is.

DAVE: Oh, I like that.

MIKE: But sometimes when we start building with a framework, we say, “Oh yeah, it should look like the framework.” No [chuckles], he says, “That's wrong. In a large project, it should look like what your business domain is.” So, you can look and say, “Oh, okay. I see what you all do. You know, here's the kind of the center.” It's centered around...you probably have a directory representing your primary business domain.

If you're doing something in finance, you're probably going to have something financial, right, at the center of your business logic. And if it's not structured like that, if instead it says, “Oh, here's your multi-controller,” then you're going to have an awkward time because it's built around your tool rather than built around your business.

DAVE: I was on a team once where my assessment after six months on the project was I stepped back, and I said, “I don't know about anything we're doing in the business to make money, but we have written an amazing program to support a database.” We were so bottom up. We were just, like, get the database, get the database. Everything had a table; everything had a trigger everything...da da da.

And a year and a half later, very little stuff to support the actual business. It was a Greenfield project, like, an investor's baby, where it's like, go do this, and do this for me. And so, we had a lot of funding. And after a year and a half, we started having some difficult conversations about what was coming out of this money. And I'm like, you got a nice program about your database.

MIKE: So, you talk about the structure needs to be different. How can we use some of the things we learn from small projects? Yeah, go out and build a small prototype in school. So, Jordan, Chloe, in school, you build a lot of small projects. You can almost think of those as prototypes, right? They are very purpose-built, specialized things to prove a point that you'll never actually use in the real world [chuckles]. But you do it to learn about something, and we do that in the business world as well, right? We want to learn how something works. Would this idea work? If we try this out, is it possible? So, we build these prototypes.

That prototype is probably going to have all kinds of bad decisions. You're going to have to rethink to build the real thing. But how can we apply what we learn from the small projects to the big ones? Is there any carryover? Do you have to learn an entirely new set of skills? Going back to what I was talking about in the beginning about navigating in your neighborhood, have you learned anything there that you can actually apply to the larger problem?

EDDY: I've learned to play nice with other people. I think it's really key. And be very receptive to feedback that you otherwise wouldn't.

MIKE: So, you're saying a large project, the larger the project is, the more the human element matters?

EDDY: 100%. You can't just make changes willy-nilly, assuming that it's a benign change. You've got to have adoption across the board. So, it takes a lot more coordination between everyone.

DAVE: That poked my brain. We were having a chat in the engineering...actually, Mike, you started it. We were talking about how does AI help you? And somebody pasted a tweet from DHH that said, "When I use AI as my pair programming partner, it's really skilled, really good. When I let it drive, I stop thinking. I retain nothing, and it's useless."

And, in my mind, that's very close to what you just said about do we build a prototype and then leverage it? Or did the AI build it for us, and we've got nothing? We used to be raised on "build one to throw away," right? Because you're going to learn so many lessons from building the first one. And the important thing is to throw it away when you're done because you've made so many mistakes. Don't fix them. Just build the new one without the mistakes, right? And, like, in AI, if you let it just do it for you, that's not a prototype. You can't leverage that much anyway.

MIKE: That's interesting. You didn't learn from it because you didn't build it. Cheating on your homework.

DAVE: Yeah. And there's times when that's valuable, though. I needed a thing that would log in with Google, right? Just OAuth. Simple. I’ve built three of these in five years. I've been on teams that have built 50 of them over the years. And I’m like, can't remember how to do it. AI, go build this login for me. I don't need you to learn me how to make an OAuth login. It's a small problem.

EDDY: For the listeners, I don't know if we have anyone out of state, but here we're in Utah, right? Salt Lake City area, whatever. If I have to drive to, I don't know, like 30 minutes away to another city, typically, I will drive there. I know exactly how to get there. I know all the directions. I know what the subroutes are in the event that the road is closed or whatever. At that point, it's just becomes routine, right? Like, I’m on autopilot, and I have to just do it for the sake of just doing it because it's expected of me to do it.

It gets to a point where I'm just like, dude, if I had an autopilot on my car, like a Tesla or whatever, I already know how to do it. I don't have to focus to do this because I already know. So, I'm just going to tell it to do it for me, and eventually, we'll get to the same spot, and I know that I got there.

So, like, for mundane tasks, you know, where you know 100% you can catch issues or bugs early or whatever, I would say that's fine. But when you’re having to go to a different place that you've never been to before, you're a lot more attentive, and you’re running the risk, you know, of autopilot kind of telling you where to go without paying attention. So...

DAVE: You just synthesized the thing that we've been talking about this entire time. Prioritizing the future value or assessing the future value of the knowledge and the skill. Driving somewhere, you can't manage a 70 mile an hour night knife fight in rush hour traffic when you are still learning clutch, brake, shift, right? You've got to get the basic skills in.

I've got a note on my monitor that says, “Insight is not cure. Get it under your fingers.” Now, this is a guitar teacher that told this to me. He's like, you can't be thinking about chord, voicing, mood changes if you don't know where your fingers are, and if you don't know where the notes that you need to target are.

And peeling that back, when I told the AI, “Go build me an OAuth login,” I put no future value in me learning that, but I have in the past. 20 years ago, I sat and sharpened the whetstone in software of, like, I'm just going to build this thing. I'm just going to do just this pattern. I'm going to do just this thing. You have to get those skills under your fingers then you can start mentally composing the larger things built out of that fluently, fast enough that you can actually start to create (I'm muddling my metaphors horribly.) but so that you can create music in your software in how you develop it.

And what you just said about driving, to me, it’s the same thing. You start composing those things. And it's all based on, the decision I'm making right now, what is the future impact? What is the cost of it? What is the value of learning this skill? Prioritize later, I think, is what I've heard that said. Not put prioritization off until later, but right now, prioritize the things that will create value later. And when enough later has gone by, you're going to be up here instead of right where you started.

MIKE: So, we talked about building fundamentals, because they will still be useful in the larger structure, but they'll be composed. It'll be made up of many of them. I’m hearing that from various voices here. Suggests that learning your skills on small projects and doing that schoolwork without cheating actually gives you something that will provide value as you work on the larger things because it will give you those components you'll know how to put together. And you may not know how to put them together yet, but if you don't have the pieces, you can't even start.

Well, I think that brings us to a pretty good place. We talked a lot [laughs], a lot, a lot about making decisions and about not getting mired in too much decision-making, but trying to find that right spot where you make a good decision and continue to make decisions, continue to pivot and get feedback rather than getting stuck.

We’ve talked about the human aspect, about the absolute critical human aspect on large projects because that's where maybe most of the work is. Well, not maybe, that's where most of the work is, is in coordinating across the other people so that you can build something bigger together. And we've also talked about building up from small to large. That's pretty good for an hour, if you ask me.

Thank you, and until next time on the Acima Development Podcast.

This episode of the Acima Development Podcast focuses on the complexities of evaluating and selecting vendors, whether for software, contract labor, or specialized services. Host Mike opens with a cautionary tale about a vendor that initially impressed with a knowledgeable representative but turned out to have major security flaws, poor integration practices, and overall incompetence once the contract began. The group discusses how this mirrors a broader challenge—companies often get only a brief, curated glimpse of a vendor before committing, with little opportunity to see the full team’s capabilities. This leads to a conversation about the importance of scale in vendor choice: a large enterprise might thrive with a feature-rich, expensive solution, while a startup could be better served by a smaller, more agile provider. Needs change over time, and the vendor that fits now may not be the one you need later.

The conversation then shifts to the vendor-client relationship from multiple perspectives—Mike, Dave, Will, Eddy, and Kyle all share stories from both sides of the table. Will highlights the unstable economics of contract shops and the difficulty in sustaining top talent, while Eddy points out the importance of assessing a vendor’s responsiveness to feature requests. Kyle and Mike discuss how larger organizations often limit vendor options through partnerships, which can lead to compromises in quality or fit. The group also addresses the consolidation push in big companies, where leadership may favor a single vendor for cost control, even if that means losing specialized capabilities. They emphasize balancing department needs, cost implications, and redundancy to avoid single points of failure.

The episode wraps with a practical checklist of red and green flags. Red flags include security issues, legal troubles, impossible promises, conflicts of interest, overreliance on one “all-star” representative, and immature products. Green flags include a vendor that’s “boring” (reliable and consistent), mature, used successfully in your industry, and willing to connect you with your actual account rep. The hosts stress building vendor-agnostic integrations to allow easier pivots in the future, negotiating service levels, and reevaluating relationships regularly. Final advice: know your needs, verify vendor claims, plan for change, and avoid long-term dependencies on critical functions that should remain in-house.

Transcript:

MIKE: Hello, and welcome to the Acima Development Podcast. I'm Mike, and I'm hosting again today. With me today, I have Kyle. I've got Dave.

DAVE: Hello.

MIKE: We've got Will Archer, and we've got Eddy. And we are going to be talking about something that comes up over and over again in the industry. And I'll just jump right into it [chuckles]. We're going to be talking about how to evaluate a vendor.

We talked about build versus buy, I don’t know, a few weeks ago, maybe a couple of months ago. Today, we're going to go specifically into strategies when you do decide to buy. And this also applies even if you're going to go with, like, an open-source project. How do you evaluate the vendor? You know, how do you choose when there's a variety of options?

And story time. Years ago...I think I can be a little open about this because the company isn't even in business anymore [laughs]. I worked for a company that did content management for publishers, mostly newspapers-newspapers and magazines, small newspapers. There's a lot of them out there actually. And we did a lot of their websites. And we were evaluating a partner, like, a business website builder. That's a common thing out there. There's lots of ways you can get your content out on the internet [chuckles]. And there are some niche products that allow you to do it in a specific way. We were looking for a partner to do it in this niche way.

And I'll get a little bit into the business model because it actually is interesting. With the newspaper industry, sometimes people think it was killed by other media, and that's a little bit true, but not really. Newspaper industry was killed by Craigslist because they made their money off of classified ads. If you're old enough, you remember that when you used to want anything, you'd go get a newspaper.

When you needed something like an apartment to go rent, for example, you'd go get the newspaper because everybody in the community knew it was there. All the business in the community knew it was there, and so they published there. And we were trying to come up with a way to kind of preserve that model where businesses could have a permanent presence within the newspaper website. And we were working with a partner to help with that.

There was a friend of somebody in the executive suite who had a company to do something that was loosely associated with what we wanted to do, who said, “Hey, we'll just use my buddy's company.” And they sent a tech rep over to get some evaluation. They did most of the stuff without evaluating with the engineering team [chuckles]. They sent one person over to the engineering team.

And I talked to this guy. He was their...well, I'll get back to that. But he was great. He talked very clearly. He was able to talk about their software. I talked to him about what you do with security. He answered it very well. He was able to talk about their stack. He knew what he was talking about. I didn't see any real red flags. And really, I was just supposed to be putting a rubber stamp at the end because the business deal was basically already made.

Well, when we went to integrate with these folks, we never saw that person again [chuckles]. Turns out he was their best person. And he was trying to keep the company running because nobody else really knew what they were doing. Very quickly, we saw glaring security issues. We had to send a password to them, which they could provide to us in plain text. So, they kept all of their customer passwords in plain text. If you don't know, that is, like, the quintessential security faux pas. You never save a password in plain text [chuckles]. It's just the wrong thing to do.

DAVE: That's not in OWASP anymore because we all know not to do it. That's the only reason...it would be top of OWASP if we were all still doing it.

MIKE: Exactly. The partner was a disaster. The security issues were just the beginning of it. They were hard to integrate with. They were a difficult partner. And they're not the only company I've worked with that was like that. We brought in a vendor, and then it was a disaster. And none of us want to have that.

I just read, earlier this week, there's a new website that allows you to rate your date. It's popular in urban areas where there's guys who will go around and be bad dates, mistreat their dates a lot. It's open only to women, apparently. It allows them to rank the guys and say, “No, this guy's a creep.” And they're happy about it, right? You can identify the creep.

There's lots of problems with that business model, by the way. There's some challenges. There's several companies that started it and then failed because it can be abused. But it'd be great if we could have something like that, right? You get some crowd-sourced, semi-objective information to say, “Yeah, this is a bad vendor.” But we don't. You never get that. What you get is more like what I got, like, here's the best person from this company. Try to evaluate whether they're a good vendor or not. You've got 10 minutes. Go. And it's a challenging problem. It's a challenging problem.

So, what do we do? How do we evaluate a vendor and determine they're the ones you want? How do you stack them against each other and choose which one to go with? And I've got some thoughts in here. I've got lengthy notes that I can start talking, but I don't want to do all the talking. I shared one horror story. You all probably have some as well. I'll kind of open the floor here. Do you all have any horror stories? And maybe you don't have horror stories. Maybe you've got positive stories. What are your horror stories, and what did you learn from it?

DAVE: I have a quick scoping question. When we say buying from a vendor, this can be anything from buying a $100,000 site license for some software, to purchasing, to, like we were talking in the pre-call, to contract houses where we literally...where this is the opposite of buying, that we're literally renting workers. But it's the same idea, right? We’re going to go [crosstalk 06:34] for money instead of for a high-risk [inaudible 06:37]

MIKE: Exactly. It's the same principle because our expertise only goes so far. There's only so much that you can build. And [chuckles] you're going to have to buy something, almost certainly, that's a rare company and probably doesn't even exist that can manage everything themselves. So, at some point, you're going to have to work with a vendor.

DAVE: So, I wasn't going to tell this story, but you just jiggled it in the back of my brain. And Eddy’s heard this story this week. I built a project 20 years ago that I called Data Monkey. It was all written in Perl, and you gave it a DDL, basically a schema, a DSL of a database schema, and it would go build your database for you. And somebody told me, “Have you seen Rails?” And I'm like, “No. What's that?” And they’re like, “Oh, you need to see this.”

And, Data Monkey, I had spent over a year and a half building this. And I was running a content management system for a bunch of online webcomics with it. It made us very happy. But it was clunky, and I was the only maintainer, and developing and adding features was hard. And I couldn't do migrations. I could not understand how to do deltas on a migration. So, you had to throw away the entire schema and then run Data Monkey again and build it all over again. It was a 1.0, right? It was a very early thing. And it only ran on MySQL because that was the only database that I used as a single person.

Somebody showed me the DHH's How to Build a Blog in 15 Minutes Using Ruby on Rails, the viral video that kicked Rails off. And I killed the Data Monkey repository, like, 30 minutes after watching that video. I'm just like, there's an entire team building out all the other database adapters. So, you get Postgres; you get SQLite; you get SQL Server. You get all that stuff. And they figured out, like, delta, like the harder problems that were still...like, I scratched the worst of my itches, but it just got harder and harder to reach, and deltas were in there.

Being able to pick it up and buy it versus building it yourself there is a huge, huge ego component. And this story has stuck with me because only after the fact did I realize, I really should have had some ego in this. This was a personal passion project that I was making money off of. I had every reason to want to defend Data Monkey, and I didn't because what I wanted was all of the features. And I wanted a full team supporting it and building it up. It helped that the price was free. I mean, total cost of ownership my time involved, like, the next 20 years of my career has been sucked up by Rails, so I can't exactly say it was free.

But I would say that, if you are in pain, and you are scoped down, and you're trying to fan out into something, yeah, don't build, buy.

MIKE: Well, did anybody have any, you know, I threw out my story of the awful vendor [chuckles]. Any of you worked with particularly awful vendors that you'd like to share that it's not, you know, maybe go back a few years [laughs]?

WILL: I myself have been a vendor, right? I myself have been a vendor, but I was, like, a solo freelancer running my own shop vendor. And consider the reliability of the narrator. But I had been successful at it, and people had been really happy about it. And people had been like, you know, generally pleased and happy to refer me, and hire me again, you know, when they had the budget, and they had work that needs to be done.

And I've hired vendors, and I've had a very difficult time with it. All the people who have hired vendors and then had a difficult time with it I have been, like, Mr. fix-it, you know, have pinch-hitting for vendors that didn't work out.

I was trying to zoom out to, like, and here I'm talking about, like, hiring development shops. And one of the issues that I think I always think about, like, sort of like the macro, right, like, what are the incentives that drive this thing? Like, get it in a messed-up state.

And what I'd say, economics of it are weird, or they may be unstable, right? Because, like, you know, I'll be straight up, right? Like, if you are a contractor, right, you're parachuting into some other big enterprise, some big project, some big codebase, some crazy thing. And you're going to drop in like SEAL Team Six, and you're going to get stuff done. That's hard, okay? It's just not everybody on your team that's going to want to do that, going to be able to do that. That is tough.

And, you know, to be totally honest with you, most people don't want to do it. Most people are just like, yo, I'm going to get in this codebase. I'm going to get familiar with everything. Like, I'm going to know who the people are, where the people are, what levers to pull, how to get things deployed, how to get things debugged, where the skeletons are buried, so that I can be more effective and be more productive as a developer. And that's a lot less strenuous a way to work.

And so, if you want to get some people like that, right, you want those people in, why are they going to do it, right? And so, it could be like, you could pay them a bunch of money, right? Hey, cash talks, flexibility of lifestyle. You know, like, I myself am a freelance gun for hire right now because, like, I'm at the phase of life where, you know, working remotely is pretty attractive to me. And even in our RTO world, there's still a lane for you if you can deliver the goods, and you're willing to work contract, you know? Like, there's a little bit of a loophole there, right?

So, where these big sort of, like, software developers as a service come in is there's a lot of incentive, I think, for the business to keep as much of that money as they can and then work the people as hard as they can. And it's very difficult to find a place that can sustainably, you know, hire and retain the kind of developers that they would need to be able to deliver on their promises and achieve them. Because, like, the business wants to make money. So, they make money when the cut is bad, and they don't make money when the cut is good.

And so, you know, you have a lot of people coming in doing a very hard job for, like, not a lot of money, and, like, that's not sustainable. Like, people who are good at their job will have other opportunities. So, the business just sort of, like, it just kind of degrades. And I find, like, there's an inevitable life cycle to these places is then, as they get bigger and older, they just all kind of fall apart in the same way.

MIKE: That's an insightful take about the challenges of a contract shop.

EDDY: I was going to say, I mean, I got to imagine, I mean, I don't have much experience with integrating with vendors or whatever. But I got to imagine that part of the decision factor when you're deciding to integrate or buy a product is how easy is it to request a change or a feature change, right?

Because maybe the product that you're buying doesn't fully encompass what your business actually needs, right? So, if you're trying to evaluate the pros and cons, like, you know, one can only assume, you know, is that something that you...is it a request that you, as a vendor, can...or as someone who has purchased it or licensed it, like, can they actively make that change themselves, or can they request a change? You know, what's the turnover on that, right?

MIKE: Well, you actually...you're touching on something, Eddy, that actually ties into what Will was saying indirectly, what I was thinking about before we got together. And you talked about them. Will they make the change? Do they have the features you need? There is a scale issue, which means the same vendor may not be the right choice for everybody, and, in fact, almost certainly aren't.

If you are a large, big, mature company, your needs are different than a startup. You might need all the features. You might want the enterprise plan and have the deep pockets to pay for it. If you're a startup and you're going with the enterprise plan, you're probably doing it wrong because [laughs] you don't have that money, and they don't care about you. You know [laughs], that big company, they're just trying to get money from other big companies. That's their business model.

And you might want to go with a small company that doesn't have very many features because, you know what, you don't need them. If they can give you what you need and it's cheap, that probably meets your needs. And your job is to stay afloat. Your job is not to spend all your money on a product that might work for you 20 years from now. And those needs are very different.

And, you know, Will talked about the contract shop, you know, growing and kind of degrading. Well, that can happen at the full enterprise level. But it also happens even within a single organization. You might get a team. They've got a new team, and they're great. Maybe even within your own company, you've got a new team. They're great. They grow to the point where they're not very manageable anymore. Maybe as you grow the pay is becoming more normalized, and some of the best people start leaving. You might have that exact same kind of problem happening within your company, and part of it has to do with that scale. And there's some tricky timing there.

The vendor you need today may not be the vendor you need 10 years from now, and you have to make those kinds of evaluations, both when you're hiring a team and when you're buying some software.

Think about a team, big companies, the real big companies, Microsoft, Google, they hire contract shops. They'll hire engineers by the thousand, and they have a longstanding relationship with them where they can manage these long-term things. They've figured out the requirements where they can make that work. But if you're mid-size, you're not going to get those precise requirements because you don't have the resources to make that happen, and if you're tiny, even less so. You're going to get what you get.

And finding a single really good engineer is probably going to serve you a lot better than going and trying to contract with some huge shop that is nameless to you. And they're just going to rotate through a series of people you don't know and probably don't care. But if you're really big, you can set up a long-term relationship that’s not so much different from your internal employees. There's scale effects that really matter here. And you've got to do some deep soul search here about where am I, and where am I going to be a couple of years from now? What is it that I actually need today?

WILL: And one of the things, I mean, you say there's not that much difference in terms of, like, you know, the difference between a long-term contractor and a long-term employee. Well, I'd say one of the big things...so, first off, why would you have a long-term contractor that didn't have a personal relationship with the company or a specialized skill set?

A long-term general contractor is like, oh, he's been on contract with us for five years. That's weird, and probably not...difficult to justify. Because one of the important things is if you have an employee, you have an understanding and a say in their working conditions. You know what I mean?

One of the problems with a contractor is somebody who...say they’re here with you for five years. So, they go from a junior, maybe not a junior, but, like, a mid-level engineer, all the way up to a senior, maybe even a staff. You can make staff in five years, if you’re a hitter, right?

MIKE: Sure.

WILL: So, you keep on bumping their salary because they're doing more for you. But are they getting it? Are they getting it? Is that money making it to them? Maybe. Maybe not. Maybe. You know what I mean? But possibly. But it might not be that way. You know, what's their working environment, right? What are the hours worked? You know, there's all kinds of things that are outside of your control. Like, a vendor might just say, “It's not your business. That's my business,” right?

I don't think, well, this is...we're going a little bit afield of the topic, but I don't think companies should have long-term core competencies outsourced. Like, you're always going to need QA, right? QA is your business. QA is your job. You have to have QA all the time. It’s like, oh, I'm going to get a QA shop to do my QA. Like, but why? Like, why? You can outsource stuff that isn't part of your core competency. But if you're running a software shop, QA, if it isn't a core competency, you need to make it one. You need to make it one today.

It's like, oh, well, we don't have a mobile team, right? I don't have a mobile team, so I'm going to outsource my mobile team. But you have a core part of your business is running this mobile app. You need to have mobile people running a mobile app because this is a direct line of business.

Anyway, I mean, there's always a line. But it’s like, I'm going to have these people work for me for five years. Like, just hire somebody. It's a right-to-work state, baby. You could fire people. Trust me. In the year of our Lord, 2025, you can fire a developer.

MIKE: Look, I'm not in disagreement because there's a lot of stuff I'm with you on, on what you had there. There are some circumstances where having a close relationship with a long-term contractor might make sense. For example, we've got a partnership with a fairly small contract shop in Eastern Europe. And we've had some engineers from there where it would be challenging for them to have a full-time employee because we don't really have a presence in Europe. But they're able to work with us on a contract basis, and they are just part of the team.

The fact that it's a contract is a necessity of the international labor exchange, but they are effectively part of the team. They are treated like part of the team. They have the respect of the team mutually. We respect them. They respect us. They meet with us. They act as an extension of our team that happens to be outside the office. I feel like that is a success story.

On the other hand, if you just have some generic person that you don't really know who they are, you give them a task, and then forget about it and come back a few days later, “Hey, did you get it done?” Well, that's a very different sort of relationship, one that you probably should never have gotten to in the first place. But if you just maybe have the gun-for-hire person who really can get it done, but that becomes a much more challenging relationship to maintain, especially at scale.

KYLE: So, I'm sitting here thinking about this and vendors. Right here we're talking about developers and stuff. I'm thinking vendors and software vendors as well. You're talking about the size of a company. You brought that up a couple of times. That, I wouldn't say matures, that transitions, I feel like, as you go up the line.

If you're a smaller company, you're picking and choosing. You're going with anybody. You're really kind of nitpicking what you want within your budget. But I feel like that almost narrows in because when you're a small, mid-sized company, you're like, oh, I want this contractor. And you kind of get personal with them. You know them.

Where I've seen in larger businesses, it's, hey, vendor that we're using, we need somebody with this expertise. We don't vet them. We don't do anything. They hand us somebody, and then we try and work with them. It's kind of that same thing, but with software vendors that I've seen, too. But instead of the smaller, mid-sized companies, it's, you know, I can go with anybody. It's, okay, you can go with these guys because we have a partnership agreement with them. They scratch our backs. We scratch theirs. And you kind of get that partnership going on. Then your vendors are almost limited, and then you may not even be able to get the best resource for your organization because you have a limited vendor pool, I guess is what I'm saying. And so, like, I’ve ran into that in enterprise locations as well.

MIKE: That's interesting.

WILL: Yeah. I'd love to hear more from you, Kyle, about sort of like that...We’ve talked about this sort of like software consultant contractor thing. I would like to hear more about, like...the thing that I think is most interesting and critical is, like, okay, I'm going to get a vendor, and I'm going to get a big site license. I'm going to write a big check for a big license for a core piece of my business. It makes sense for me to buy versus build, right?

But I don't know these people, and I have two, three, four vendors that I could go with. I don't know them, right? And I'm going to be stuck with them for a year or two at least, even if they dog out. How do you vet people? How do you vet quality of service? How do you vet quality of support? Like, what are the questions that you ask to figure out, okay, this is what I'm getting into, right? Because, you know, they'll promise you the moon. The salespeople will promise you the moon. And, in the meeting, like, they're going to bring out the all-stars. But when rubber hits the road, how do you figure that out? You know, how do you vet?

KYLE: That's currently a problem that we're actually dealing with. Like I say, in kind of a [inaudible 24:05], or I'm just saying, in a company's atmosphere, we're more worried about money most of the time, right? And so, we're asking one vendor, “How much will this cost?” We're asking another vendor, “How much will this cost?” They're giving us a price. Well, that one's a little bit higher. These ones are giving us a price. They're not telling us who. We're not really interacting with who might be doing the work. We're not vetting those individuals out. They're just like, “Here, work with this team,” right? And so, there's that atmosphere.

And then when it comes to the software world, you know, SaaS products, it's a similar situation. And then it kind of gets even larger because you have engineers that will use the SaaS products, for example, that will have favoritism as well. One team will appreciate a product for the application monitoring. Another team will appreciate a product for their tracing capabilities, right?

Well, this SaaS product is really good at that one, and the other SaaS product is really good at this other one. And then, you know, they're not good at the other one. And it's like, we're running into that as well, and, like, how do we mesh that? That's actually one reason why I was excited about this conversation. I’m like, I need to know, too, because those are active problems that we run into day in and day out.

Now, I will say, when I've worked in smaller companies, it was much easier to meet face-to-face with the individual that was, like, going to be performing the work if they were a contractor, you know, and discussing because then it was easier to go back to your manager or whomever, the hiring person, whoever's writing the paycheck, and say, “Hey, I do or do not recommend this individual to perform this work.” But that's not been my experience lately. If that answers your questions, Will [laughs].

MIKE: Well, you know, you've raised a lot of challenges. And two vendors, they're both decent. They offer somewhat overlapping services, but they're not fully overlapping, right? And how do you go with which one? Which team do you choose [laughs]? And that's a tricky one because you're going to have to evaluate from a business perspective which department's more important to get their needs met. And that's hard.

KYLE: Then there's a lot of time where it's like, consolidate to one vendor, right? There's always a push from upper management, “Consolidate to one vendor,” when it's like, well, they both facilitate our needs in different ways. And it's just like, go lesser evil, I guess. I don't know.

MIKE: Well, one thing you can do is, sometimes you're going to lose something, right? You can't win every battle. There are some cases where you can legitimately make the case, but you've got to speak with money. You can say, “Well, if we consolidate on a single vendor, this is what we're going to lose, and this is how much it's going to cost us.” And if you can compare costs and say, “This will actually cost us more year over year,” that's a pretty easy argument to make to people who are trying to save money. And if you can't make that argument, then it's uncomfortable.

But you might need to look a little bit inward because they have accountants for a reason. They've got a finance team for a reason, or a spend management because things do tend to get out of control. And you're combining departments, you know, buying, doing acquisitions. Sometimes you're going to get redundancies. And going with a good enough solution, you're going to have to give something up. And that's always painful, but it might still be the right answer.

WILL: Is there a counterargument to having somebody else bidding against them? I mean, you don't want one vendor for everything, do you? You know what I mean? You know, going at...payment processing, it's nice to have a few payment processors, where they just say, “This is your [inaudible 27:50] I'm like, I don't think...for all the traffic? Maybe not, you know.

MIKE: So, single points of failure are very much worth the cost to avoid at scale. Again, startup may be different from enterprise. But as you grow, the more those single points of failure cost you, you know, if something goes down.

We certainly had vendors go down before. And recently, we’d mitigate a single point of failure. If a vendor went down, like, hey, well, we've got a backup [chuckles]. And it was fantastic because we had done the work. And we were in a position of much more strength.

WILL: Well, I mean, not for nothing, but, like, you know, business organizations can fail, too, and they fail in different ways. But it’s like oh, well, you know, founder decided he wanted to cash out, get that private equity money, buy a boat. God bless you, you know, I'd love that for you, but I might not love it for me so much.

MIKE: And then changes in ownership can have a big impact on what that company is doing for you. A number of times, we've gone with a vendor because we preferred their product, and then they got acquired by their competitor [laughs]. And there you go. You didn’t really [crosstalk 29:10]

WILL: That's a for real thing. It's a big thing to think about in the software space just because, okay, so let's say if you're going out and your vendor is a venture capital funded company, right? You know a couple of things, you know, with a high degree of confidence, never 100% confidence, but a high degree of confidence. They're on a burn rate, right? They're funding. They're funded, and they are spending more money than they take in for growth because that's what venture is like. Otherwise, you're just a private company that has some investors, right?

And the venture model, 90% of those companies fail, right? You know that, right? Those are just the rules. Those are the cards you're betting on. So, fold or acquire. Or, you know, maybe they get a lot bigger, you know, and they get a lot bigger. And they go from, like, a series A kind of a thing to a publicly traded kind of a thing.

That's a brand new company. That's an entirely different organization, and it's going to happen fast. Two years, they're either a completely different company because they won or a very different company because they lost. But, like, they will not be the same company two years down the road. It won't happen structurally.

KYLE: That goes into, you know, acquiring companies, which is, I think, what you’re saying there. Where we’re at, we have Acima. We have RaC, and we have Brigit under our Upbound umbrella. And all of these have different ecosystems that were already in place. And then, how do you consolidate those to the specific vendors, right?

Like, so, when you're acquiring these companies, big or small, how do you make that transition and decide, you know, well, you've got a large group of people over here that they prefer this vendor? However, the core organization uses another vendor. How do you consolidate? And that's a hurdle, too.

MIKE: And somebody's going to not get what they want. Somebody's not going to get what they want. And, sometimes, those decisions do come down to money. Sometimes they come down to size. Well, more people want this vendor than want this vendor. Or the costs to transition are going to be huge.

A very specific one that I think I can share...You talked about RaC and Acima. RaC used Microsoft, and Acima used Google Docs, and we had to choose, right? And, in the end, the tech company who was able to, you know, navigate tech are the ones who had to change because they could [laughs]. They didn't get what they wanted because it was a core competency. They were good at dealing with tech changes. And so, they didn't get what they want because they were good at it [laughs].

And it's a little painful, but it's probably still the right decision because the cost to go the other direction would have been really high and really painful for the company. And that's a difficult choice and not one that made us, on the tech side, very happy because we had to change. But it's still probably the right decision.

KYLE: I had an older engineer that I worked with when I was at NCR, and he made a comment to me one day. I grew up in a mentality...I was very fanboyish. I liked what I liked, and I didn't want to go outside of it. And we were a C# shop at the time, and we were transitioning to being a Java shop. And so, I was kind of teasing him, and I was just like, “Hey, you know, are you going to hate this? Are you going to stick around?” And he's just like, “I'm absolutely going to stick around. I'm a software engineer.”

And I was just like, “I don't quite understand what you're saying there.” He goes, “I'm not a developer. I am not a software developer. I am not tied to a language. I am an engineer. I will use whatever tool the company provides me with, and I will get my job done.” And I feel like that's kind of stuck with me, in the sense of, like, that should be everywhere. It's just, you may not get what you want. You may not get what you've always wanted to work with. But they're all tools. Figure out how to use them for your job.

MIKE: Nailed it, right [laughter]? One thing I wanted to bring up here that we haven't talked very much about are red flags. Because we talked about sometimes you're not going to get what you want. And we've talked a lot about going with contract vendors, you know, with the contract shop you go with, and how scale matters, how there's some challenges with that relationship altogether that you need to be really watchful of. But just in general, and this probably applies more to buying software that you're going to use, I think that there are a lot of red flags. I compiled a list coming [chuckles] into this call because there's a bunch of things that I've seen.

And I mentioned the first one in the intro, security issues. If you do a security, in fact, we recently...unnamed vendor. We evaluated a new vendor. They looked like they had the product we needed. The security team looked at it, and they got an almost failing score. Like, oh, actually, never mind. We're not going to go with them. I feel like any security issues are an indicator of something deeper. If a company doesn't care enough to take care of something as fundamental as security, then what else are they not taking care of? Every time. So, I started listing companies that I've seen security issues on, and it was just a debacle [chuckles].

Way back to right at the beginning of my career, company security issues, total disaster, maybe even sunk our company I was working with at the time. I mentioned the one earlier. I've seen, more recently in my career, partners where they were real fast and loose. They were really willing to share sensitive credentials over insecure channels. As soon as you see that, like, wow, that's not a partner I want to do business with because it's a symptom of an organization that doesn't care.

And it's led me to think, well, there's a strategy. If you can somehow, like, ask them, like, “Could you send us some credentials?” You know, ask them in a sideways kind of manner. Give them an opportunity to do something wrong and see what they do with it. You could probably learn a lot.

So, there's my first one. Security issues are a huge red flag. And if you can somehow find a way to allow them to reveal their nature there...and you can't always. But if you've already got a contract with them and you learn that they have those, it's time to start looking for somebody else because, in my experience, it never works out well. Like, they're not going to fix it. Thoughts?

WILL: I love it, man, just the idea. Like, maybe more broadly is, like, are they minding their Ps and Qs, right? So, not, like, the salespeople, right? But, like, is the contract, like, is the contract right? Are there, like, just goofy things? Like, you know, like, they messed up the contract. They messed up...there are typos in their stuff, like, not in what they say, but, like, in that.

You're looking for, like, I think the quality, the personality quality, is called conscientiousness, right? Conscientiousness. Are the Is dotted? Are the Ts crossed? Because you can see...if you're doing enterprise stuff, right, like, big enterprise things, there's work product, the kind of work product that isn't a good sales guy doing a good job, right? Like, the guy just, like, boring, gritty, pick-and-shovel, get-it-done stuff that is, unfortunately, enterprise software development. You can see that. That will leak out if you're looking for it.

MIKE: And, you know, taking that a little further, next red flag I was thinking about, is there legal action against the company? Now, if they get big enough, pretty much any company has probably had some sort of legal action, which may have more or less credibility. But some stuff has very obvious credibility [laughs], right? And, you know, if there are multiple attorneys general signing on multiple times, it's a red flag, at least, that maybe there is something not going right. And it goes back to the care. If somebody is careless about their customers in that way, they may not be treating you very well either.

WILL: I mean, I'll say, as a small shop, like, I'll say two things, right, like, here I am generally showing my ass a little bit. I'm not a voyeur, and I'm small, and I don't have, like, you know, all that stuff. But I have definitely...when you are a little fish in the pond, there are a disturbingly large set of people that will bully you through the legal system. I have been involved in more than one or two legal actions, all of which, every single last one of which was somebody trying to rob me and then saying, “Do something about it.” So, you know, beware, but I would say that, like, a bigger company would have learned their lesson the first time, right?

MIKE: [laughs] Well, that's why I try and preface that. Well, pretty much every organization is going to have spurious legal action, or at least in a gray area, where, well, yeah, maybe, you know. There's somebody who doesn't want to pay you, and then there's Enron [laughs].

WILL: I get patent-trolled. Like, do you get patent-trolled? I got patent-trolled, you know, and it's just, like, bro, what?

MIKE: [laughs]

WILL: Taking a credit card, like, that's the patent you're going to come and get me on? Like, if you could really afford that patent, you wouldn't need my money [laughter].

MIKE: Yeah. So, at the end, there's a tricky one. It's not that there will be no legal action, but you should at least look. You should at least look. And it's not like companies I've worked with have not been involved in legal questions before. But I've left a company before because they were doing some legally questionable stuff, and I'm glad I did. You know, there's a line, and a great deal of it from reliable sources often means something.

WILL: Absolutely. Yeah. Yeah. And I'll say this, especially legally, you know what I mean, highly regulated industries, you know, minding your Ps and Qs. Acima has always been scrupulous in following the law to the letter. Other big companies that I have worked with they've all been on their game. But Acima is definitely the leader of the pack in that regard. The current gig is a close second, but, like, yeah. This is the financial stuff, right? You've got to have it together.

And, like, living in Utah, I will say that Utah...there is some business in Utah. We do some stuff in Utah [laughs]. I'm not familiar enough to pursue that one. I’ll let that go. That sounds interesting.

DAVE: I am, and I'm also not going to pursue that one, at least not on the call [laughter]. We’ll talk afterwards.

WILL: You can just do a quick Google. And this has nothing to do with me. But how many Utah attorneys general have gotten indicted in the last 20 years? A lot more than you think. I think we're batting over 500. I sincerely, hand to God, I think we're batting over 500. This isn't Mississippi, man [laughter]. What are we doing?

MIKE: Illinois has had a streak with governors [laughs], so I can relate a bit, and other political entities. Moving on.

WILL: You also have a reputation. Yeah, moving on [laughter].

MIKE: Impossible promises. If they're promising you the moon, walk away. Just walk away. It gets to some of our biases, to wanting what they have to offer.

Relevant story here. So, this isn't a vendor, but I'd say it was an employee I hired once who was doing some sales, or business sales, like, relationships. And he made some big promises, and they all turned out to be hollow. And we listened because I'd actually done work with the guy before, and he'd brought me business before. He seemed credible enough. But I got that bias, like, oh, this sounds like the ideal work for me and my shop. And so, I overlooked some of the red flags there. It was too good. It was too good. And got to watch out for it.

So, on the flip side, I've got a list of green flags as well. A vendor should be boring. If they're boring, that is the biggest green flag ever. If they're boring, that means they do their job and it's not all marketing. There's nothing better than a vendor who is just, like, watching paint dry, right [chuckles]? You know, there is nothing interesting about them at all. That is your first choice. Because --

EDDY: So, I'm halfway there. Is that what you're saying?

MIKE: [laughs] So, boring, great sign. That means that they know what they're doing, and they're probably doing it really well.

So, going back to the...well, I'll give an example of impossible promises. And I'm going to make a positive shout-out to Google here. They were pitching us on some tools, some AI tools, and they told us what we'd get from it. And it was honest [laughs], and it was not a huge, dramatic change, but it was something that could make a meaningful business difference.

And it's hard to argue, you know, that was such a good sign. It was such a positive sign to see somebody who would give you honest assessment that was not going to make everybody jump up and cheer, but was something that would be enough to make it worth it. You know, it's always possible that the numbers weren't perfect, and they probably weren't. They weren't for many vendors, you know, from any...it's hard to get a perfect number. There's maybe no such thing, but it was believable.

A couple of other red flags I thought of: possible conflict of interest with internal leaders. I mentioned that with the vendor at the beginning of the call. You know, if somebody might be getting a cut, or somebody has a long-term relationship there, you know, you want to tend to trust that, like, oh, so they know them, so it’s good. Well, maybe, but give it some scrutiny.

If there's a conflict of interest there, you should be really careful. Now, especially if you're going with a smaller vendor where you don't know much about them [chuckles], but, yay, this is my friend; he knows exactly what he's doing, give it a little scrutiny; often a good way to get a contractor. Maybe not such the best way when it's not...again, we're talking tech. The non-tech executive who brings in somebody and says, “Oh, yeah, we're going to go with them because they're my friend,” that's a red flag.

Another red flag: they have one person who talks to you, and they're great. It's easy to have one person who can talk well, and they may even be really good. They might be their only real good person [chuckles]. If you can get a team of people who can all speak confidently, that says something about their culture. Wow, I’ve got a group of people who know what they’re doing. In a really large organization, well, maybe again that’s easier to fake, sending their best, and they’ve got a lot of people who aren’t the best. But at least for a smaller vendor, [crosstalk 45:02].

WILL: Is it possible to say, like, “No, I want to talk to the person who's my account guy?” Is that a reasonable ask? Because, I mean, on one hand, like, I...how do I put it? Like, if I'm busy doing support, right? Like, I don't have time to do sales. Sales isn't my job. I'm busy supporting the customers I have, and I understand that. I understand that. But, like, if they don't have time to talk to me before I write the check, what are they doing after?

MIKE: [laughs]

WILL: Like, I understand that, like, you know, you don't want to burn up their entire time. But, like, I want to talk to the guy, not, like, your best guy. Don't give me the all-star. I just want to...I want to talk to the person that I'm going to be dealing with now.

And I also, like, maybe as an aside, you know, you guys have...you guys have bigger, like, all the software that I've bought, like, I buy, you know, off the shelf, you know, these are just SaaS, like, you know, because I'm not...I'm not big and bad and all that. Like, you can negotiate sort of quality of service in these big enterprise contracts, can't you? Where it's just, like, if I send you an email, I want an email back, you know, in this time frame [inaudible 46:10].

MIKE: Absolutely. You can set up all those quality of service items in the contract. And I think they're fairly typical. You know, I wish...you said, “Hey, can you go talk to that person?” I wish I'd done that several times. And maybe something that I'm going to ask next time, “Can I talk to my actual account representative? Not your best person you sent to me, but, you know, who's the person who's actually going to be working with us? Can I talk to them?” I think that is a fantastic question. And if they'll make it happen, it says something about the vendor as well.

I threw out a big list there. And I haven't gotten a lot more than, like, nods and mm-hmm.

DAVE: You're covering it well. Yep.

MIKE: [laughs]. Yeah, I thought it through. Anything you'd add to the list of red flags?

DAVE: I would add a thing on there. You mentioned, like, SLAs and, like, quality of service. The thing where it has hurt me recently is people saying, “Oh, I want this. I want this, I want this.” And then, they come in, and then here is the third party we're going to use. And then, you hand this out to all your employees. And then you never hear back on...you know, the only SLA report you get is from the vendor, not from your people. And so, yeah, you can run into some fun things there.

Joel Spolsky, the guy who originally...he ran FogBugz. I think Stack Overflow, I think, was his baby way back in the day. He talked about selling software in the 2000s, and he called it steak and stripper-based software sales, which is where you take somebody from the C-suite out to dinner. You buy them steak. You show them a good time, and you sell them the software. And then they go home, and they tell everyone at work, “This is what you get to use because it's great.” No, they had a great steak, and they had a fun after-dinner entertainment. They never even looked at the recs, right?

And you used to be able to make really good money selling software that way, and then come to find out that once it actually hits the ground, the service you're getting, yeah. Make sure you've got transparency into, like, how the QoS is working, how the SLA is being honored.

MIKE: That's an interesting one. And it says something about being willing to push back if you can. If you've got some capital to spend there, social capital use it to do real evaluation because there's a lot of things that slip through when you don't scrutinize. Go ahead.

KYLE: Another one that I've got is maturity. At least to me, it's a red flag. If it's a B1, if it's a beta, I'm sorry. I'll probably pass until you're a little bit more mature. I don't want to take that gamble.

MIKE: That goes back to the scale. If you're a startup and they have exactly what you need, nothing more, that might be the right choice if they're in beta. If you have gotten any scale and you actually need that thing to stay running, you should probably walk away.

WILL: I mean, a big piece of it, too, is what kind of software are you buying, right? If it's ops-type software, I don't know, man. Maybe not. Maybe not. You know, there's a lot of people where it's just, like, just write Oracle their check. It's fine, you know? [laughter] Maybe not that kind of a check. But, like, you know what I mean? Like, somebody with some gray in the beard, you know, like, there are things you can mess around with, and there's things you can't. Some software, it's just like, yeah, I want it. I want it battle-tested.

KYLE: Even there, you know, and I'm saying maturity within their product line, too, because there are features specifically, you know, I'll throw out AWS. I don't think that's bad to throw out. We have done evaluations from time to time, and it is generally better to wait for V2 [laughter]. We have run into that recently and have had to backtrack our implementations, and we're waiting for V2 of a product.

MIKE: That's interesting. You could think about AWS as a bunch of little companies [chuckles].

KYLE: Yeah, you really can.

MIKE: Or a bunch of little vendors, yeah, because they operate so independently.

KYLE: Yeah, if you want to see that, just go to the different services page for each of their products. It's designed by a different team, every single one of them [laughter].

WILL: I can't keep up, man. AWS is coming out with a new, like, something every week. I remember when AWS came out. I was an early adopter, right, because, like, it was brand new, and, like, we were just getting going. We had just moved off of the data center model. And, like, our team, our stack was, like, just brand fresh out of the oven, and we didn't have a lot of dough.

And so, we just got right on AWS, like, right off the jump, I mean, nearly at launch, and whoa, it's stunning. They’ve come out with so many things where I'm just like, I sort of start to question, like, who needs all this? This is crazy.

MIKE: The interesting thing is a lot of their products came from internal, at least originally, came from stuff they were doing internally.

WILL: Yeah, absolutely.

MIKE: If there's a niche [laughs], there's, you know, there will be product to fill it.

WILL: Yeah, absolutely. Well, anyway, a little bit of a tangent, you know. I liked AWS over the other maybe things because they did, like, less, you know, and now they're very much doing more.

KYLE: They are doing more, but I will give them the benefit of the doubt in this scenario. AWS is the more. If you want less, you do, what is it, AWS Lightsail? That's kind of your small individual one, you know.

MIKE: So, we’ve talked about some red flags, some positive things, right? They should be boring. They should be mature, have some longevity, used by other companies in your industry. They have multiple competent people who come and talk to you, you know, they'll let you talk to your rep, and they leave a good impression. There are some positive things. We’ve talked about a lot of red flags.

WILL: I have a red flag for you.

MIKE: Okay.

WILL: So, I won’t be too specific, although, you know, if you know the space at all, you'll probably know who I'm talking about. So, there was, early in my career, I worked on the corporate side of a multi-level marketing company, which don't get me started. I was young, and poor, and hungry, and I needed to eat.

DAVE: Needed the money.

WILL: Hey, I needed the money, you know, whatever. So, there was a vendor for their sort of, like, downline payment processing payout for their tree. Tree was upside down, but it was a tree. Not a different geometrical shape, but they were nightmares to work with. Absolutely horrible. They were expensive. They were slow. The SLA was more of a middle finger. It was pretty intense. It was pretty bad.

And the red flag...but it was also, right, it was also textbook case of, like, buy it; don't build it, right? Because it was this very specific, very niche financial product. You wanted it to be right. Like, I'm here to be the pharaoh on top of the pyramid, not, you know what I mean, this sort of, like, esoteric, arcane financial billing thing. Like, that's not my job. I'm here to do whatever it was that company did. You know, it didn't have anything to do with software.

But the red flag is, like, they were the sole player in this space. They did not have competitors, like, meaningful competitors, right? Because it was the 800-pound gorilla in this very niche space, and, like, there wasn't a negotiation, right, where we're, like, we're getting our enterprise contract. They were, like, this is it. That's what it is, right? Nobody was coming behind them. They ran things, and they set terms.

And so, like, you know, like, the old, like, Ma Bell monopoly. When you deal with people like that, you're going to get a...they're established. They're very big. You're going to get what you get, and you don't get upset. And I don't know if there was a number two, but I feel like the number two would have cared. They would have pretended to care at least a little, I think.

MIKE: [chuckles] Well, there's times when you may not actually have choice. But the small choice might be the right one, the disruption.

WILL: Maybe if we were a bigger account things would have been different. You know, I think fly-by-night, multi-level marketing, like, startups, are not [laughs] afforded the respect that you feel they might be deserved to. I know what I thought about the place, and I worked there. They were my only client. And I was just like [laughter], make the check out to cash.

MIKE: [laughs] One thing I wanted to touch on before we close is you've got the relationship. You've got the vendor. What do you do in your first year, I'll say? It may not be your first year. You've got a contract for a year, and you’ve got a contract for two years. How do you establish that in a way that makes sense going forward?

And I've got one thing that I wanted to touch on. Try to build your code. Try to build your integration to be vendor-agnostic. Build an integration service that you go through before it goes to the vendor. And this isn't always possible if they're running your infrastructure. And, Kyle, you know [laughs], sometimes the infrastructure can't be agnostic.

Well, that's a lot more...infrastructure is a lot easier to move now, right, to be multi-cloud than it was in times past. If you can make it vendor-independent, then you save yourself a lot of pain going through a year from now and changing the name of that vendor everywhere in your code where you're stripping it out and completely changing to call somebody else's API.

If you build a generic integration service that just abstracts that idea, you've got to change it in one place, or more or less. You know, you can minimize the damage. And maybe they're a great vendor, and they were a venture-funded startup that goes out of business. There's all kinds of reasons where maybe they were even a great partner at the time, but it's not going to work out in the long term. And having the ability to pivot is good. I think that building that in is worth it almost every time. What do you all think?

KYLE: Yeah, whenever we're evaluating new infrastructure components, that is something we take into consideration. I'm thinking specifically AWS. But, you know, we try not to run anything that only AWS provides, for instance, for that reason. You know, we're using Postgres. We're using OpenSearch, you know, stuff that the other cloud providers will also offer. That way, if we do need to pivot, those type of things will historically bite you, and those are large migrations.

Even monitoring tools, you try and make it a drop-in. So, kind of like you're saying there, Mike, you write a layer or something to where you can drop those in, and they're not hard integrated into your environments. That way, you can pivot.

MIKE: We do this audio only, but we record with video [chuckles]. I see nods all around the room [laughs]. All around. Well received. Any final words?

DAVE: I would say be careful to pay attention to what's important. If you listen to a recruiter's ad, they will tell you that we solve the problem of how hard it is to find the right people. If you're hiring people through a contracting agency, that shouldn't be their primary sell. I mean, it should be. They should absolutely be finding the people for you. But if the only thing they're doing for you is finding the people, are they managing the people? What work are you outsourcing? And especially if the only reason you're doing contract-to-hire through an agency is because you're bad at hiring or bad at recruiting, you're probably paying through the nose for not having somebody good at recruiting. Focus on what's important.

EDDY: I think at one point or another, all of us who have used a service or software have said to ourselves, man, I can build this product so much better [laughs]. And if you can, maybe that says more about the software you're using [laughs].

KYLE: It's also a chance to reflect, though: are you using the software correctly? Is your practice correct? Is something I would warn against there.

MIKE: Well, and it goes back to being willing to evaluate. If you don't do some evaluation and say, “Is this still the right vendor?” Then you may find yourself in a long-term bad relationship, and it's worth it to do some introspection. There's no shame in reevaluating that choice because times change, and maybe it was a bad choice to begin with, or maybe it's an even better choice, right? There's cost to moving. And if it's okay, that's good enough, right? But be willing to reevaluate. It's not quite presence of mind here, right? Have the business willingness to revisit your choices.

Great discussion. One of our --

DAVE: This was good.

MIKE: There was a lot of stuff to cover here, and we covered a lot of it. Hope that you get something from this and can make some better and better decisions when you're picking your next vendor. Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, host Mike opens with a gardening metaphor to frame the episode’s central theme: cultivating healthy company culture. Drawing from his personal experience transforming barren soil into fertile ground, Mike contrasts short-term fixes like chemical fertilizers with long-term strategies like feeding the soil itself—likening these to how companies often rely on perks or high salaries to attract talent rather than investing in sustainable, growth-focused environments. This sets the stage for a broader conversation about what truly nourishes both plants and people—ecosystems, relationships, and meaningful investment.

The discussion evolves into a rich conversation among remote team members, including Justin, Javier, Jorge, and Will, who share their experiences working in international and distributed environments. They highlight the importance of communication, inclusive documentation, and feeling culturally and professionally supported. Javier and Jorge reflect on challenges faced by Latin American contractors, including language barriers and differing communication styles. They emphasize that well-documented processes and clear expectations can bridge time zones and cultural gaps, especially when employees are empowered to grow and contribute meaningfully.

The group critiques superficial attempts at building culture, such as gift cards and perks, and stresses the importance of leadership that fosters personal growth, purpose, and strong communication. Leaders who spend time mentoring, understanding their teams, and clearing barriers can create “fertile soil” where people thrive. Whether it’s through offering growth opportunities, creating psychological safety, or enabling people to do impactful work, the team agrees: you can’t shortcut your way to a strong company culture. You have to invest in it intentionally, consistently, and with heart.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting again today. With me today I have Justin and Javier. And this should provide some good perspectives. Justin previously worked for Acima [laughter] [inaudible 00:39] and is now elsewhere; I'll leave unnamed [laughs]. Javier is a contractor that works outside the office, outside the country, which will provide some good perspective because we'll get good perspective to the topic today. I'm going to announce the topic in a minute [laughter]. Although you could probably see it from the title of the podcast recording but, you know, at least I'll pretend to leave some suspense there [laughs].

JUSTIN: I'm looking for a good story, Mike [laughter].

MIKE: I'm going to talk about my garden.

JUSTIN: Oh.

MIKE: I've enjoyed gardening since I was a kid. I like growing things. I like being outside. I do have a garden. To be honest, it's maybe a little neglected as of late, but [chuckles] I do have a garden. And I say neglected, I've let some things grow and spread. So, I've got a big patch of raspberries, for example, because I love raspberries. What's wrong with letting the raspberries spread [chuckles]? They're delicious. I've got quite a bit of garlic, and I've got some other berries. I’ve got some goji berries, some garlic chives. So things that have kind of spread some, but I like them, so it works great. And I have a lot of kale because I love kale. I eat kale every day.

I've read a lot about gardening. I, at one point, considered changing careers. I was going to work for a gardening company on their tech side for a while. I didn't end up doing that, but strongly considered it. And my original major was botany, so definitely very interested in this topic. I've got, like, a hundred cactus plants in my office that I love. Most of them I grew from seed [laughs]. I like plants. Pandemic, you know [laughter].

So, I'm going to talk about a thing that people regularly get wrong, and even to the point of agriculture sometimes gets wrong, and they're turning from this. And there's been a turn over the last few decades to fundamentally shift the way people think about not just gardening but farming. Where I live, I have quite a bit of farms nearby. In the state of Illinois, there's a lot of corn and soybeans. If you've ever been there, you know what I'm talking about [laughs]. There's a lot of corn and a lot of soybeans.

They used to just plow everything under every year or just treat the waste from the previous year's corn and soybeans as waste, and just kind of ignore it, let it blow away. And when they’d till it under in the fall, the soil would blow away all winter. And there's places you can go and they'll show you how deep the topsoil used to be and how deep it is now [laughs]. On the rest stops on the side of the road, they'll show you how much topsoil has been lost because it's a big deal, like, half or more of the topsoil has been lost in areas. And it's a really serious problem because now you can't grow crops as well because you've lost that deep reservoir of fertility that allows you to grow.

And so, they've got techniques they do in farms. One of them is that they’re moving away from tilling. They allow the stubble to stay on the ground so the soil doesn't blow away. But it also means that it decomposes into the soil. And there's a reason they have corn and soybeans also. Because soybeans are legumes, they've got nodules on their roots that feed microbes, that take nitrogen out of the air and make it into a form that the plants can use. So not only are soybeans high in protein and nutritious, they also leave the soil better than before you planted them. So, they improve the soil.

So, it's part of a rotation where they grow the soybeans to improve the soil. Then they grow the corn, which is really hungry and uses up all that nitrogen. And then, they'll grow soybeans again to improve the soil again. So, there is this cycle of rotation, which is something that was done years in the past. People have done crop rotation for probably a millennia, realizing that certain plants will take certain nutrients from the soil. This is getting to engineering, by the way. This is getting to engineering [laughter].

JUSTIN: I'm getting a minor in botany right now.

[laughter]

MIKE: In my garden, so my garden, when I moved into where I'm at now, I took the worst soil in the yard, along the back of the property. And they’d cut away the topsoil [chuckles] when they built the house, and it left just subsoil. And it's just...it's clay, and sand, and gravel, and really nothing else [laughs]. And I thought, this is a project. I want to see if I can make this grow stuff. And I guess the previous owner of my place was a grass seed salesman. He couldn't get grass to grow [laughter]. It was just bare dirt on the area where my garden is now, and there was ravines all over through it. Cutting down, you know, it was bad.

And so, to the key point, what a lot of people do is they'll go buy some chemical fertilizers, and they'll throw that on the soil, and the plants can actually grow in that. You know, you can get some things to grow. In the extreme case, you can do, like, hydroponics, and you can actually kind of make it work with just chemicals [laughs]. It's a fussy business. And the minute that you stop augmenting with those chemicals, the whole thing falls apart, right? And everything dies [laughs]. You need to continuously add those chemicals in order to make it work.

And they actually degrade the soil over time because they usually come with some salts that will build up in the soil. So, you'll get these minerals that build up and actually make it harder and harder to grow things. It's a problem. It's a real problem. And it reflects a misconception as to what you're doing there in the garden. Because I have a garden...it's a small scale. I can really focus on that. And the change in mindset that makes all the difference is you don't feed the plants. You feed the soil. You don't feed the plants. You feed the soil. If you feed the soil...so, think about the difference that would make. So, I'm going to focus on improving the soil.

Going back to those farmers who have lost all the topsoil, like, oh, wait, we need to stop this, right? I need to preserve the soil I have and improve it. In some places, they actually are improving the soil. They're growing it again. If you feed the soil, then the healthy plants will follow because the plants don't live in a desert, in a vacuum. We're not on a space station. Even if we were on a space station, plants have not evolved to live in a sterile environment, and don't live well in a sterile environment.

You may remember the book and movie The Martian that came out a few years ago [chuckles]. A guy had to poop in his garden in order to get microbes to grow because the plants wouldn't grow without it. It’s the same idea. You need to have this ecosystem, or the plants just don't do right. And if you build healthy soil by feeding all of the stuff that's living in the soil...because it's not just the plants. There is a huge network of fungi, bacteria, protozoans, insects [laughter], rodents. There are all kinds of things that live in that soil. And your plants are just one of them.

If you have incredibly healthy soil, by adding things to feed the soil, feed that whole ecosystem, then the healthy plants will follow because you're focusing on the right thing. You're building the healthy ecosystem, the healthy environment, and then the plants will grow well as a natural consequence because they'll be in a place they want to live. All of the things that would be good for them are there. And so, you don't have to go and find all of those precise chemicals. Oh, I need this much nitrogen. I need this much phosphorus. I need trace amounts of copper, whatever the case may be. You don't have to worry about that because everything else is taken care of. That will just naturally be taken care of because everything else is so healthy.

And you might have to worry about weeds [laughter] because they'll grow abundantly in your garden as well, but that's a problem you like to have. You want to have a garden where you can actually have things grow. You can actually solve weeds the same way. We actually want to diverge. You feed the soil on top of the ground where you don't have the plants you want, and then the weeds cannot grow there. If you put a lot of mulch there, then it will gradually break down, become good soil, also smothering weeds.

Which was a long introduction [laughter], but I think it's hugely relevant here because we're going to talk about company culture and how you build it. And talking about this garden shows some of the ways you can do it wrong [laughs], as well as the ways that you can do it right. There's a lot of ways that you can kind of fake it. You can add those chemical additives, and you can pay really high salaries, for example. And yeah, you'll get a certain sort of mercenary person who wants to go for the high salary. But it doesn't really build a healthy culture a long time.

Not that it's bad to pay people a decent amount, don't get me wrong [laughs]. But you can't solve some problems with money. If people hate being there, really, there's not enough money you can pay. There's an example of this that's been in the news somewhat of late. I guess Facebook has publicly somewhat denied it, but there's been news reports that they are offering seven-figure salaries to AI researchers if they'll switch over to Facebook. You start adding up those zeros, and that's annually, right? That is huge amounts of money. And I think they've got, like, one [laughs] new researcher that's --

JUSTIN: I think there were a couple that came over, but yeah, it just goes to show that, I don't know, they may be focusing on the wrong thing.

JAVIER: I just wanted to share an experience because, previously to Acima, actually, I was working in a company earning probably 30% or 40% more than now. I really hated my job.

JUSTIN: Oooh.

JAVIER: Yeah. So, definitely, you know, money is good, and I do have money. It's good for me and my family. But the culture sometimes, you know, the way you feel doing your job can be definitely different, even with good money. So, I’m completely agreeing with you.

JUSTIN: Yeah. And if you somehow get that combination of good money and good culture, I mean, you get people who are willing to go the extra mile for the company, and that is fertile ground for explosive growth. Going back to that garden thing, you know, you look at what you reap from a really, really fertile garden. It may take a season, but you're going to get two, three, four times what other people are getting because you spent the time to prepare that ground. And not only that, but it's like, you'll be spending less time weeding. You'll be spending, you know, all your focus will be on growth or productivity. So, that is, like, the ideal.

And so, I've been in a couple of places that have been good, and I've been in a couple of places that have, basically, it seemed like they were salting the earth [laughter], and people were leaving left and right. So, it's like, wow, what a great analogy you have here, Mike. But you look at it and you're like, it's not something that just happens. If you keep on going back to the analogy, the gardener or the farmer, the person in charge there of the soil, they are spending time preparing that soil. And they are just, like, you know, doing everything that they can to prepare it. Because it's like, they aren't the ones that are necessarily producing the thing. They're preparing the soil such that the thing could be produced.

MIKE: You hit on a key point there. They're not producing it [chuckles]. They are building that environment so that the production will happen on its own. And I think that distinction is so important [chuckles] and really easy to miss with short-term thinking.

JUSTIN: Yeah. And this is funny because I'm always torn about the huge amounts of salary that CEOs get paid. And you look at that and you're like, how could that guy, usually guy, how could that person be worth that much money? And you realize that sometimes they aren't, obviously. Sometimes those are massive failures. But sometimes they are amazing at farming.

MIKE: Yes [chuckles].

JUSTIN: They're growing people. They're growing a company. And they have the right ideas to prepare that soil such that that company can be very successful. And I can think of a couple of places that I've worked that have gone through a couple of different CEOs. I think it's been long enough now, but I specifically worked at SoFi. And they started out with a decent, I mean, they recognized that, but they had kind of bad leadership at the top.

And then, the person that came in as current CEO, whose name is Noto, who I really respect still, he came in, and he has created a culture that even though I don't work there anymore, I recognize that, hey, that was an amazing culture that fixed a lot of the problems that the previous one had, the previous CEO had, and he did it from the top. And he came out and figured out all the intermediate leadership. And to the point where it's now, I still have friends that work there, and they are very happy with the way things are going. And so, it's just like, I look at that and I'm like, that is an example of a great leader who recognizes how to grow a company and the people.

MIKE: Well, let's talk a little bit about what might be some of these chemical additives [laughter] they might use to cope for a little bit, but don't really have long-term success, and then what actually matters.

A lot of years ago, I used to have a job where I'd take the same bus to work every day, and I got to be kind of friends with the bus driver, who was also a university professor. That's the problem with adjunct professors [laughter], is that they have to go drive a bus to actually make the money. That's a topic for a different day, but there's some challenges in academia.

So, he was a university professor who also drove a bus, and he talked a lot about the incentives that companies will give to try to get their employees to do something, like, they'll give out, like, here's a $50 gift card, to give an example of that. And a lot of companies will turn to something like that, to, like, oh, we need to improve our culture, so we'll give people lots of these little gifts. He was not a fan, I'll say [laughter], because he said, “You know, people can look...” So, he was thinking of somebody who wasn't getting paid enough at his job, so he had to go get a second job. He says, “You know, people can look and see, you know, I'm getting this $50 gift card. It's maybe nice today, but it's not feeding my family [laughter]. It's pretty superficial.”

Not that it couldn't be a little nice to do now and again, but if your entire plan for helping your team be happy is to give them a gift card once a year, you're probably not going to be very successful, especially, I'll say, if you're working with a distributed team where you've got contractors all over the world, you know, where a lot of times contractors aren't getting that, right, you're not going to build a company culture off of handing out those little gifts. It's an utterly inadequate strategy.

JUSTIN: I do have to say, I knew of a company that took that to the extreme. They handed out those all the time, and that was the culture [laughter].

MIKE: Oh.

JUSTIN: And so, every time he saw somebody, he was like, “Here, here you go.”

You remember Overstock? That went through a lot of different phases, right, growth and shrink. Well, actually, you probably aren't as aware with, you know, where you're living right now. But here in Utah, overstock.com had a very interesting CEO who had phases where he would grow the company a lot, and then it seemed like there was a great culture. And then, something would break culturally, usually the CEO, and then they’d go through a bunch of shrinkage.

And it got to the point where it was kind of a joke within software engineers in Utah because you all did your time at Overstock [laughter], you know, you'd go in...I interviewed there, and luckily, I declined their offer. But it was interesting because they went through phases where the culture was actually really good for software engineers. And they were well-compensated, and they were well-regarded within the company. And they were producing a lot of successful websites, and the company was growing, and things like that. And then, they went through periods of time where the budget got shrunk, and all the little perks that they had gotten used to got cut, and the pay got cut, and then they went through several rounds of layoffs.

And so, it's interesting because you have stuff that's external to the culture that gets cut, I mean, if the company goes bad and their budgets get cut, all of a sudden, those little perks that were part of the culture they go away. And you're left with, oh, why am I here if I'm not getting that $50 gift card, or if I'm not getting that great quarterly vacation to Las Vegas, or something like that? And people were looking at it like, you wonder if they ever got to the point where they were an actually good engineering organization, or was it that they just had a lot of neat perks [laughs]?

And so, the perks inspired a certain amount of loyalty for a certain amount of time, but it was, like, those specific chemicals that got added that...and when those chemicals got taken away, people didn't want to stay. And it went just beyond just, like, the cuts to the, you know, the layoffs. They had people actively leaving, you know, looking for other jobs because the company culture got toxic. And it was interesting. It was an interesting story. And yeah, I don't think I have any friends that work there anymore, and the company's gone through several reinventions. They're now known as Bed, Bath, & Beyond, actually.

MIKE: Heard that.

JUSTIN: But it's funny because the culture got so bad, and they got such a bad reputation that they actually had to rename the company so that people wouldn't associate it with them anymore. But yeah, a lot of my experience, obviously, has been with local culture, and, you know, we've been over some of the bad things that happen locally. How about you guys...can you guys talk a little bit about the bad things that may happen, you know, badly for remote? And then we'll get to the good stuff, right? So, that's the intent.

JAVIER: Well, you know, as remote developers, sometimes communication is a big deal. Well, for example, here in our team, something that is very important for us, I think, is that we are able to have a community of Latin people. Because, you know, Latin people is a bit more fun people. I don't know how to say it. Maybe Jorge has a better idea. But yeah, that is part of our challenge, is to have communication with, let's say, the other side. And when I say the other side, I mean no Latino people. Sometimes it's for culture, or whatever, it's difficult to have this communication, and that is difficult to handle. Yeah, that's my thoughts right now. I don't know, Jorge, if you have some opinions about this, or...

MIKE: By the way, Jorge has joined us. He joined us a little bit late [laughter].

JUSTIN: Jorge, where are you located at?

JORGE: I'm in Lima, Peru.

JUSTIN: Lima, Peru? Oh wow. I've never been. That sounds exotic and awesome.

JAVIER: You have to go, best food in the world, I think.

MIKE: Oh.

JUSTIN: Oh wow. Hey, I'm going to have to go now [laughter].

JORGE: Yeah, but if you come here, you're going to return to your country a little bit fat because here is, like [laughter], a lot of food, every day [laughter], every hour.

JUSTIN: Gotcha.

MIKE: Do you have any thoughts, Jorge, on what is critical for company culture? So, Javier said something interesting there. He said that not being, like, on an island, but having a group of people that you can identify with. And that can mean a lot of things, but maybe most importantly, it means that you're not alone, that you've got somebody else that you can pair with, that you can ask questions to, that you can feel like is a friend, seems like is pretty important.

JORGE: Yeah. In my experience, as Javier told us, communication is really important, and even not only communication but defined processes are too important. Why? Because, for example, sometimes we don't know about the best practices inside a company to fix a problem or an issue, for example, and that's a really good point. That's one of the reasons I love a lot of the process here in Acima because we have, like, a really big Confluence page [laughter]. We can find whatever we need inside of it, right? But that's a good practice that a lot of other companies don't do, right?

So, that's key in the company and in a good technical culture because if you need to do something, at least you can check your recommendations and try to fix something in a good way, in an expected way for maybe the teams, right? So, I think it's a really good, important part of our jobs to get that documentation as updated as we can, right? As we can. So, I think both of them, the communication and the processes, are the key in a remote culture and work.

MIKE: It’s interesting. You mentioned the documentation. Have something written down. Go ahead.

JAVIER: I have something to say about this because, again, as a Mexican, as a Latino...and I think this is many of us. I think many of us we think the same, because sometimes...of course, this is not offensive, not at all, but, you know, the American communication could be different if you compare it with Latino communication. And sometimes documents and documentation itself is prepared thinking in that culture, and it's perfectly okay.

Just for me at least, it sometimes could be difficult to understand the sense. Maybe I can understand the words in English, you know, but the sense, for example, you guys use a lot of acronyms [laughter], and Latino people make me crazy sometimes with acronyms [laughter]. And sometimes the descriptions are very small, and maybe Latino people need a bit more, you know, something a bit more [SP] más carne.

JUSTIN: Más carne. [laughter] Meaty --

JAVIER: Yeah [laughs], and sometimes that makes you feel a bit like I'm part of another subculture inside of the culture of the company, in this case, the companies from United States. At the same time, this subculture could mean you're in a different group, and that sometimes feels like you can, you know, be completely apart. Maybe I'm not expressing totally because my English is not so good, but yeah, hopefully, appreciate my thoughts.

JUSTIN: So, I have some thoughts on this. I've worked with contractors from Colombia, Medellín, and Argentina, and a couple of other places, and as well as with contractors from Eastern Europe and from India, and, hopefully, I'm getting a couple of contractors from India in the next month or so.

But you brought up some really good points there of, like, you know, making sure that you're included on the culture and that the documentation is really good. Because you can go read documentation, and if it's good, you don't have to, like, necessarily ask questions, and it conveys what the culture is. If the documentation is not good, oftentimes, you're kind of, like, left in the dark, and you're trying to communicate with people, and nobody will respond, and so on.

But if the documentation is good, it helps you understand, and it enables you to understand and fix the problems yourself. And also, if there's, like, a culture of fixing documentation, or adding to it, that is really key as well to, like, helping people not be frustrated. And this is something that I've noticed, is if you can enable people to be successful on their own through documentation, through clear processes—I like how you talked about that processes part—if you can, like, show how the processes work, and what the theory is behind them, and why they work the way that they do, as well as, like, have good documentation, you're enabling those other people to be successful. And you don't have to be there to, like, answer every question.

And it's especially important as you get, like, further time zones away, and you're not, like, overlapping. Because, you know, all of a sudden, if somebody offshore has a team, and they, like, have an eight-hour difference to you, they'll write their message. You'll get it first thing in the morning, but they'll already be asleep. And you'll reply, and all of a sudden, this task that, you know, in reality should have taken them a couple of hours at most, really ends up taking, like, a week or two or more.

And nobody likes to be unproductive. If you are being unproductive, generally, it's, like, you are unhappy, at least myself. And, all of a sudden, you start looking around, like, oh, this company is not utilizing my potential. I'm not growing here, and I'm just banging my head against the wall. And that'll enable you to go look for something better, and even that something better could be a pay cut. So, it's, you know, all those things. And I've rambled a little bit here, but yeah, those are some of the items I know.

MIKE: You touched on a couple of things that I wanted to bring up. You said that everybody wants a chance to grow, and everybody wants a chance to do something meaningful. And both of these go down to deep human need, right? We have [chuckles] our human nature in that we're not robots. We have things that we need, and one of those is to grow. It's just deeply wired in us. And, again, we want to do something meaningful.

If we're not growing and if we have no opportunity to do anything that feels like it really matters...you work on a project that seems trivial, and then it gets dropped. And then, you move on to something else [laughs], and that project gets dropped. And you never actually get anything up that actually accomplishes anything. You're not going to like that job. You're not going to like anything about it. You're going to feel like I just wasted however many years of my life doing something that didn't matter at all, and you leave.

Whereas if you feel like you participated in building something that mattered and you get to see that actually going and affecting your customers, making your customers' lives better, then you're invested. You care. And there's probably other things. But coming into our session today, the things that I thought were most important for culture are those opportunities for personal enrichment and fulfillment in our job.

I think there's also an aspect that Javier touched on, which is having people that you like working with, having those conversations, being able to interact with people and have positive interaction, feeling safe. We've talked a lot on this podcast before about psychological safety, feeling like you can be yourself, like you can be open and be respected; there's another key attribute as well. Those things trump just about anything. If I can go to a job where I am able to grow in my skills consistently, where the things that I do matter, why would I want to leave? It's giving me fulfillment in my life.

JUSTIN: And that pays you something so you can live, but yes.

MIKE: Yes. It is important to have enough money [laughter]. If you decide you're going to undercut all the competition on pay and pay less than everybody, it's probably not going to go well. You have to meet basic standards. But there's research that shows that once people reach the middle class, more money doesn't make that much difference, and this is outside of work, necessarily. But, you know, just in general, being super rich, there's a little bit of a bump as you get more money, but there's a lot of very unhappy rich people [chuckles].

But if you have enough, if you look around and you're more or less on par with other people, approximately equal, that's usually okay. It doesn't have to be the best pay in the industry, as long as it's not the worst. As long as you're making about what's normal for the industry, it's fine.

JORGE: I love to work with international teams. I started about six years ago trying to work with American companies and in international teams. And I love it because all of this stuff that currently we are talking about it's really awesome to know people from other countries. I only have the opportunity to go to Mexico City. I was living, like, four or five months in 2019, nothing else. But I need to visit a lot of other countries.

But I work with other teammates from a lot of other countries like here, for example, Colombian people, American people a lot, people from Europe, even from India. It's a very rich experience, at least for me, because I try to learn a lot from other developers and how they handle or how they solve a problem because they have other way to think and way to do the things, right? So, that's awesome. And, obviously, I'm taking advantage of improving my English skills every day [laughs]. So, it's really good, at least for my career, and I hope to still be getting close with all of these kinds of international teams. It's really good.

JUSTIN: So, I see Will joined. Hey, man. So, we got, like, three remote people. Will, are you also remote 100%?

WILL: Yep. It's been...

JUSTIN: Wow.

WILL: It's challenging, man. It's really, really tough. It's been frustrating. Yeah, I'm not exactly sure where the thread of the conversation has gone. I have personally been impacted a lot with the RTO mandates, and it's been challenging.

JUSTIN: So, I guess my question for you guys is, it's interesting because the company I work for went through a big, huge return to office and five days a week. If they don't have an RTO plan that is done well, you have high attrition. And it is interesting that that done well part...because I've worked in this industry for 20 years, right?

Before COVID, I was almost 100% in the office, and there were parts of the office that were annoying and parts that were fine. But the office, you know, when you're in the office, if you have good leadership and good recognition and you're working on good things and everything, and you have recognition and opportunities for growth, and your commute is not bad and things like that, there's a variety of things that make the office doable.

Like, a good example is today, I, you know, I camped out up in the mountains above here with my wife and one of my kids, and then I came down into the office at 8 o'clock. I had a shower, worked for four hours, and then I took a lunch break and did a workout for my lunch break, and then came back. And I'm, you know, working for another couple of hours. And it's not bad, but at the same time, I'm still working from home one or two days a week, and, for me, that's actually a good balance.

But every single person is different, and every single person has to make that decision for themselves, you know, what they're willing to do to return to office. But it has helped so much if the leadership makes that return to office palatable and, you know, enjoyable even. And they have the right incentives to go back to the office, and not just incentives, but, like, they have the right culture. So, otherwise, I forget what the attrition numbers are sometimes, but they are pretty high.

WILL: Yeah, like the big, long bus ride, like, heavy-duty commutes. A lot of people in big cities are doing serious, serious commutes, and that's a big...it's a big deal. I mean, like, two-hour commutes out on the West Coast is not unheard of, not unheard of, not crazy.

MIKE: Well, I think it's interesting. You talked about it as a perk, and we talked a lot about perks early on and how perks can keep you there for a while [chuckles]. But in the end, the things that matter the most are an opportunity to grow, an opportunity to do something that matters, and enough money. You have to have enough. You don't have to have the best in the industry, but you have to have enough.

So, as a leader, so I’m talking to any leaders who are listening [laughs], if you give the people you're working with an opportunity to grow their skills consistently, make time, schedule time. Say, “Take some time to grow your skills.” Give them projects specifically to give them a chance to grow their skills. Find out what they want to work on. Try to give them those opportunities.

Actively work with the people that you're working [chuckles] with and lead by giving them opportunities, clearing out the way, making their lives easier, and trying to give them an opportunity to grow. Also, give them opportunities to do something that matters. Don't stop somebody in a corner to work on a really long backlog of issues that nobody has reported in five years because it didn't really matter [laughter] that much. There may be some value. Most people probably aren't doing that.

But there's ways that we do that are less blatant than that, where you don't give somebody the support that they need. And you were talking about this before, I think it was you, Justin, talking about when they've got time differences, time zone differences. So, you don't give...the communication is so bad that nobody can actually get anything done because they don't know what it is they're supposed to be working on.

So, really address head-on some of the communication challenges that Will was talking about. You have to. And we've got people all over the world. We've got people here in multiple countries here in this call. If you don't do the work to try to make it easy to work with people who are doing the work...and most of the big companies, I think, are working with people not local.

Now, locally, you might have people in office, but you're going to work with people remote. And if you don't make that experience positive, then you're not going to have a healthy culture because people are going to be frustrated. People are not going to get things done. As a company, it'll be inefficient, and on a personal basis, people will be frustrated. And you won't keep people who want to do good work because they'll get frustrated, and they'll leave.

But on the flip side, there's levers you can pull, right? You can feed the soil. And for those who arrived a little late, we started the call by saying that, in a garden, you don't feed the plants. You can just throw a bunch of chemical fertilizer, and the plants will grow for a while. But as soon as you take away the fertilizer, they die. They go into dramatic decline. Instead, you feed the soil, and then you build a fertile ground where everything just kind of grows without you thinking about it.

Put the work where it really needs to go. Do the work involved to give people opportunities. Know who they are. Know what kind of opportunities would be good for them, and give it to them. And, secondly, build the kind of communication structure and processes that lead to people getting the information that they need so that they know who to talk to. They can talk to them. They have the information they need. You've got things documented as you build them. You've got your processes documented so people can be effective. And that will go so much farther than a bunch of gift cards [laughter].

JUSTIN: I'm trying to think of some specific things that leaders have done for myself or my team that have resulted in growth. And some of the things include, you know, when I've had technical leaders, they've actually been in the code and doing code, you know, writing code and doing code reviews and things like that, spending the time to get to know technically how things go, how things work. And so, that's always been great.

Another one is, you know, it's hard to do if you have multiple, multiple layers, but, you know, the engineering director or the engineering VP them going down to individual engineer level and, like, chatting with them for 15 minutes about themselves and their careers and things like that, where they spend the time getting to know the rank and file. That is something that I've seen be really helpful in terms of, like, encouraging culture growth at a company, and recognizing, you know, asking them, “Oh, you know, what are you working on?” You know, “What's the next step in your career?” You know, “What do you want to do with yourself five years from now?” All those sorts of things.

And it doesn't have to take long. But, you know, spending those 15, 20 minutes to get to know the people who are actually doing the grunt work has resulted in a lot more...a better view of leadership than it not.

MIKE: We've covered quite a bit of ground. We've talked about the importance of actually building, you know, what matters, and allowing people to grow on their own, and kind of used that as the framework we've focused on, about the downsides to thinking that you can just throw perks at the problem and call that culture because it doesn't work in the long term. And we've talked about things specifically that you can do. You know, we've talked a little bit about growth.

You know, I will say, schedule time. One thing I've seen be incredibly effective is just saying, you know, 30 minutes a day or whatever the number is, do something to study and grow your skills. And then, you have to lead by example. Not everybody can do this, but a lot of times your tech lead, your media leads can do that and show they're doing that. And maybe get other people doing it, you know, pair. Pair with people so that they can learn from you.

And there's probably a great number of other things you can do. But it has to be on top of your mind. You have to actually know the people you're working with. Recognize what their needs are and do something to meet those needs. And then, finally, set up the conditions where they can work effectively, where they're not always stuck, where they're not blocked by something. And so, they can, you know, work effectively and get stuff done like we all want to do.

WILL: I'd like to close in saying that extreme programming solves all these problems [laughter], and we shall rise again [laughter].

JUSTIN: I've never regretted time invested in making sure people understand, you know, how to be a better engineer, like, whether that's very specific on a task or a, you know, how things work. It's like an investment in time that will pay off later on. But that's the thing, you've got to spend the time.

MIKE: You've got to spend the time. You can't cheat [laughs]. Great. Well, thank you. Interesting discussion today. Hopefully, you've gotten something from it. And until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, Mike kicks things off with a humorous comparison between the DMV and employee onboarding, using the DMV’s predictability as a metaphor for what onboarding should feel like: smooth and well-organized. The team, including interns Chloe and Jordan, along with veteran team members like Will, Matt, and Tim, dives into the realities of onboarding experiences. Chloe and Jordan reflect on the value of strong documentation, access to multiple mentors, and feeling emotionally supported. They emphasize how overwhelming onboarding can be when information overload hits or support is unclear, and how even small gestures, like developer lunches or Slack channels, can make a big difference in building comfort and connection.

The discussion expands to include insights from more senior voices like Will and Matt, who underline the psychological and emotional complexity of onboarding, particularly for seasoned professionals used to excelling. Will points out that new hires, especially mid-career professionals, often face an identity crisis when they’re suddenly inexperienced again, and that trust between leadership and new employees must be earned, not assumed. He stresses the importance of proactive communication, asking questions, and building relationships over time. Meanwhile, Matt emphasizes that leaders must take responsibility for initiating that trust and creating a culture of safety and availability, even if time and organizational bandwidth are constraints.

Finally, the group turns to strategies for remote and global onboarding, with Tim detailing Microsoft’s best-in-class processes that pair automation with strong support systems. The consensus is that technical logistics like IAM provisioning and documentation matter, but they’re not enough on their own. What truly shapes a successful onboarding experience is human connection: pairing new hires with mentors, creating safe spaces for questions, recognizing cultural differences, and setting realistic expectations. The episode closes with a collective realization that while automation can streamline processes, it’s trust, empathy, and communication that ultimately empower new team members to thrive.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. We've got a good crew here today. We've got Tim, Kyle, and we've got a couple of interns who are joining with us today¬¬—Chloe and Jordan, great having you. I’m excited to hear your input. It's very topical for today. We have Justin and Dave, and, finally, Will Archer, the crew here. I think I got everybody. Did I get to you, Kyle? I think I mentioned you. If not...[laughs]

Let's start with an ordeal I'm going to have to go through in the next couple of weeks. So, it's been five years or so since I last renewed my driver's license, so I have to go to the DMV next week [laughs].

JUSTIN: I want to see how you're going to tie this together with new hires. This is going to be really entertaining.

[laughter]

MIKE: People do not look forward to going to the DMV. There's an old song that comes to my mind that says, "I've been to hell. I spell it DMV [laughs]." I think of that every time. I went and looked up the lyrics. There's some other lyrics in that song I'm not going to share [laughs]. I'm going to point out the reference. But it makes a point that I think most of us tend to agree with.

However, when I go to the DMV, I know exactly what to expect because they have done this a million times, right? Maybe literally a million times, maybe more than a million, you know, many millions in a large state. They just do this over and over again. So, they have a routine. I know I walk in there. I'm going to get the number, and then they're going to send me down to a seat and wait for who knows how long [laughs]. They might have one of those little counters to let me know when the number is coming.

At the one DMV I would go to, there's, like, three different desks, so there's actually three different lines [laughs]. You have to know which one you're getting to. But they have signs. They've got tape on the floor that sends you to the right direction. And then when they call you up, they've done this so many times that the people there they don't even, like, see your face anymore [laughs]. They just walk you through the routine.

And it's so standardized because they need to make sure that it always works every single time. And it generally does [chuckles], as long as you didn't forget to bring whatever document you needed to bring, and then they send you to the back of the line or send you home [chuckles], and you have to come back. If you get everything right, it just works because they've totally standardized that process.

Now, it's totally impersonal, and [chuckles] you wait forever sometimes, and nobody likes that. I have heard that they've got, like, some...There are offices in Utah, and I've heard that they've got, like, some express DMV out there that works really well. I've heard some people say some good things about that. I'm seeing some nods out there. Because they’ve cut the process...streamlined so you can go in there and just take care of that part that's fast, and they run you through. If you set up an appointment, you go right through.

So, it actually can be pretty good when you have everything lined up and planned ahead of time. I think that we've all been involved in starting something before that did not go so well [chuckles], when things were not planned, and it can be an absolute disaster. We're going to talk today about onboarding new employees.

And I bring up the DMV because if you have everything lined up, you might not enjoy it, but it'll be relatively fast. You're probably not going to be there for a week [chuckles], and they will take care of your needs, and you will leave with your business taken care of. And sometimes when we onboard new employees, it does not go like that at all. I've seen some horror stories where people did not get their computer for a month [laughs], and...I’ll work someday, you know [laughs], and nothing goes through. It can be a total disaster. And somehow, we've been doing this for however many decades we've been doing it in software, and it still takes...it's still hard. It's still hard.

That's what we're going to talk about today. We talked about onboarding, actually, about a year ago. You can go back, and I think it was July of 2024. We talked some about this. And that time, we focused a lot on the mentoring aspects of it. Today I thought we'd talk a little bit more...to have a little more opportunity for horror stories [laughs] we’ve seen. And we've got some people who've recently onboarded. We can talk about the good and the bad. And we’re going to talk about a little bit more of the hands-on, nuts and bolts. How do we make this work?

So, I would like to start by asking our recent hires here [chuckles], so Chloe and Jordan, what went well, and what did not go well about your onboarding process?

CHLOE: I can start. I feel like one thing that went really well is there was a lot of documentation, so a lot of really good resources to get help when it came to onboarding, as well as I was paired with somebody who has recently onboarded just a few months prior. And so, they had a lot of really fresh experience as well as advice when it came to onboarding because they had just done it, so they had already kind of figured out a lot of the things that could go wrong or any problems that I might have encountered.

Something that was a little bit more challenging was, I think, when you are onboarding, it just feels like you are in a pool of water, and there's these big waves. And it's really hard to feel like you can catch a breath and feel like you're understanding everything, and that's common to a lot of places. But it feels like there's so much to learn, and you always feel behind, which is a tough feeling to feel when you're coming into a new company.

MIKE: Great. Thank you. Jordan, any thoughts from you?

JORDAN: Yeah, so this is, like, something that is kind of obvious, but I think good documentation is a must. And this isn't any fault of the company, but maybe of my own from last year. Since we started the project, there's very minimal documentation for it [laughs], and it's simple enough where you can get it figured out. But I think that was a little bit of a struggle trying to remember what I did and asking some people, like, “Did I do this right?” So, there's that.

But something I thought was helpful was having some early low-hanging fruit tickets that got ready for us, basically. He had chosen them and said, “These are good tickets for you guys to kind of remember what you're doing or learn how to do things again,” so that was very nice.

MIKE: Nice. Well, thank you. Now, we've talked to you all who just recently started at Acima. But we also have some others here on the call. Will has recently started a new position, so I think he's got some fresh onboarding stories in mind as well, and I think some others as well. So, go ahead, Will.

WILL: Well, I've onboarded a lot, and I think, like, oh gosh, like, I'm getting through it. I mean, part of it, like, the biggest thing, I think, to keep in mind is you're just going to have to embrace the suck. You know, like, it's going to be bad, and it's always going to be bad. There's no comfortable way, I think, for, like, ordinarily high-achieving people who are used to not looking like an idiot to be real dumb and real helpless and completely ignorant for a period of weeks, if not months. There's just no...you know what I mean? Like, there's not a lot of, like, C minus students [laughs] getting onboarded, but, like, you're just going to have to take...you're going to have to take a bunch of Ls.

And I’ve sort of, like, as I look...so, I mean, like, having done it, I always find the process really, really rewarding when I'm through it. But I have to remind myself, like, “This isn't supposed to feel good [laughs]. This isn't supposed to feel good,” at every point there. And so, if I'm reminding myself, like, as I'm through it...because, like, it's me and a buddy. We both came in here. We got recruited at the same time, and sort of, like, we're both sort of our support group, right? Because, like, I'm extremely old, and I get hired on because people expect me to come in, you know, plug and play. Like, I'm expected to produce.

I'm not an intern, and there is, I mean, like, I think everybody is understanding, but, like, there is a certain level of expectation that I have to, like, get things done. And so, like, me and my buddy are just sort of...we're constantly reminding each other: you need to be vocal. You need to be out there, like, asking questions. You need to be bringing people in. You need to be sort of getting on pairs. You need to be proactively searching out documentation and following it and not being surprised when it's wrong because it's frequently wrong.

And you need to be aware of asking questions in the right way, where, like, if you're asking a question, you need to be doing the work. You need to be visibly working harder than whoever you're asking a question of. You need to be sort of, like, cognizant of chain of command, whereby you go up to your team lead, and then you go up to your manager, and maybe you go up to, like, a staff engineer, a principal or whoever, and then, like, if you've got to go...if you have to go to a director, you can go to a director, and I will.

But I'm not going to go there without the checklist filled out where it's, like, I would do this and this and this and this and this, and this led me to you. This is what I'm trying to do. This is the problem, and this is what I need you to sign off on, right? And then, as you do that, eventually, the sun will come out, you know, come out through the clouds, and you can start getting things done, and you're not going to piss too many people off in the process. I don't know. Sorry. It was a vague prompt, and I'm trying to distill, like, you know, what to do, right [chuckles]?

MIKE: Will, you and Chloe both pointed out something that I think is really interesting. You both talked about from the perspective of somebody who's being onboarded and said it's hard, and Chloe swimming against those waves [chuckles]. And you talked about, you know, you're just going to feel dumb, and that is hard. I’ve worked with --

WILL: You're going to be dumb [laughs]. It's not a feeling. You suck [laughs].

MIKE: Well, I've worked with...I've seen a number of people who switch careers later on, and some of them really, really struggle because they were successful before and now they're not, right [chuckles]?

WILL: Yeah.

MIKE: They're the noob, you know, they don't know what they're doing. And they are making mistakes, and they can't figure things out. And they feel like everybody knows this better than them. And that is not a good feeling. But you have to just embrace that for way longer than is comfortable if you want to be successful. And you have to say, okay, yeah, I'm back in high school. Maybe I'm back in junior high [laughs], and I just have to live that role again.

WILL: Well, man, I think...I don't know. I mean, this is maybe, like...this is not an intern problem. Like, you guys have been dumb recently enough that, like [laughter], you can still remember the feeling and cope with it. But, like, I'm 45 years old, and I've been good at my job for a very long time. And, like, imagine, like, going back to school, going back to high school and, like, not doing great [laughter]. But I feel like that's a trap. I feel like it's a trap that later on in your career you can get trapped in, later on in your life, you know.

Like, you could get into your mid-40s and be like, I always wanted to learn, I don't know, basket weaving or, like...you know what I mean? But, like, I'm so good at everything else in my life. If I go and do this thing, you know, I wanted to learn ballroom dancing and, like, I'm bad at it. And there's no skipping over that first...that first rung of the ladder. You're just going to be bad at it. So, you're going to either figure out a way to, like, embrace stupidity later on in life, or you're going to stagnate, and that will just be...you'll just be stuck forever. And I think that's --

MATT: We've all gone through it, right? And we still do, regardless of our level. You talked about going up through leads, managers, staff, principals, directors. I'm currently a director with a company.

WILL: I'm sorry.

MATT: And I still go through it. We don't always know everything, and we're always going to need some help. Part of it is just accepting that and understanding that everybody goes through it. Other people go through it, and more likely than not, they're going to be willing to help you through it. So, go in with that mindset, and I think you'll do all right.

I don't require someone to go through all of the chains of command to come talk to me, at all. If it's something really trivial that they could have leaned over to the guy next to them or the girl next to them and got an answer really quick, then maybe I'll say, “Hey, did you ask these guys at all?” But door's always open. You need help --

WILL: Yeah, but they don't know that, Matt. They don't know that. They don't know you. They don't know who you are.

MATT: Yeah, it's my responsibility to make that clear, right, and, hopefully, I have with Jordan and Chloe, you know, as I’ve talked to them over the time they’ve been here but --

WILL: I’m going to lean back on that one, Matt, because it cannot be done. Like, what you’re talking about, you could say that, and you could say it clearly and effectively, and you could say it with clarity and confidence. You can say that in that sentence, right? But, like, you cannot establish that level of trust in a 15-minute getting-to-know-you meeting. Like, that has to be grown over time.

You can't just be like, “Listen, my door is always open.” And I'm more open, and I'd say, like...you know what I mean, like, most critically, I think virtually everybody, you know, at a director level or VP level, like, if you reach out to them and you need help and they can help you, they'll help you, but you got to get on their calendar. And the reason you go through the chain of command is, like, A, you know what I mean, like, yeah, don't waste your director's time; don't waste your VP's time. But also, B, you know, that pyramid gets pretty narrow towards the top, like, you know, you can't be on your calendar just at a whim. But your team lead, yeah, man, if I need five minutes from my team lead, yeah, they better give it to me, you know [laughs], like --

MATT: Yeah, and that's fair. That's fair. Calendar can certainly be tough, you know, it's generally booked all the time. But the way you earn that trust is, try me. You know, you have something you need to talk to me about; you want career advice; you want software advice, try me. And if I don't give you that time, then that trust isn't earned, right, but if I do, then we can establish that rapport. And, yes, it does take a little time to build it, but that's how you do it.

WILL: I don't know, man. I think you have people for that reason because that trust is something that is built over time. You don't have...you can only...what is it, the Dunbar's number? You know what I mean? Like, how many relationships can you maintain? Well, you got to have...that's why you've got people to build that trust among, you know, people that you can't meet with every day or every month, really, you know? It just can't be done.

MATT: And that's correct. You know, it's much easier as you're down the pyramid because the bandwidth is there, right? But ultimately, trust comes from the top. If I can’t trust my leaders, then I don't want to be wherever it is I am.

MIKE: There's something there to be said about the leaders making themselves genuinely, like, doing something to make themselves a little bit vulnerable, like, oh, hey, I can trust you. You know, the thing that, you know, the dog rolls over on its back, shows its belly. And, like, oh, I could hurt you, but I'm not going to. It's giving the opportunity to be hurt so that you know that, hey, I'm here, and I'm not here to hurt you. And the person in that role, in that leadership role has to do something like that, reveal a little bit of vulnerability, and that's, you know, that's just part of relationships.

They still have limited time. That calendar is still going to be limited, no matter how much they do that, and that's a challenge. And you're a new person. You're that new person. You say, “Okay, here's somebody I can go talk to three weeks from now on Tuesday at 8:00 a.m. [laughs],” and that's a problem. So, you're going to have to be assertive with the people closer to you.

MATT: Somebody --

MIKE: Go ahead.

MATT: Yeah, somebody once said to me something that really struck a chord, and that is, we all have time; it's, do I have time for you?

DAVE: Don't say, “I don't have enough time.” Say, “I have too much to do.” Because you can't give yourself more time, but you can reduce the things you've committed to.

MATT: Yes, you can rearrange your schedule because we have time. We have 24 hours in a day, every day. So, it's priority, right? So, am I going to make you a priority? Is my superior going to make me a priority? And that's something that I think we all need to try to do. Sometimes it's not possible because there's other commitments and other people are relying on us, but we can make time.

WILL: I mean, the reason I'm pushing back on this is because you can't. It's a zero-sum game. You can burn yourself out. You can spread yourself too thin, you know? But I think the reason I'm like, no, you can't command trust; you can't command a relationship; you can grow the relationship, but your bandwidth is limited, and you're only going to be able to maintain so many relationships. If you don't respect that process, then you'll find yourself in a trap.

We’ve diverged really badly from onboarding, but I do think there's a point that needs to be made, where you'll go to somebody and you'll say, "Listen, my door's always open. I need feedback. If something's going bad with this project, I need you to let me know. I need reality. I need this relationship to be strong." But then you say that, and you mean it. It's not that you don't mean it, but because you haven't grown that trust, that relationship isn't actually there. And you rely on it, but it isn't actually there.

And then, you can get these really weird communication mismatches. And you can get these really weird things because you said, "Do this thing," and they said, "Sure, boss." But it isn't there, but you think it's there, and you rely on it, but it isn't. It's ephemeral. You have to respect the need for cultivating those things and the limits around what you can and can't do.

DAVE: I think you guys are both right. I'm hearing something interesting from both of you guys. What I'm hearing from Will, and I think you're right, that there's a boundary. You can teach me something, but you can't understand it for me. You can offer psychological safeties, but you can't make me trust you.

But I think Matt is right that you have to get up and go offer it. If you're the training or the onboarder, it behooves you to actually make an explicit, “I have to go do this. I have to go talk to you and let you know that my door...I have to tell you what the culture is, unless I'm assuming that you're weapons grade good at reading rooms,” which most of us aren't. So, I think you guys have both got the right...you can only go up to your own boundary. And so, what Matt is saying, you've got to charge your boundary if you're a leader. You've got to get right up to their boundary. And I think Will's right that you can't go over it. You can't make them take it.

MATT: I totally agree with that.

DAVE: I have an illustration for that, but I figured I would yield the floor.

MATT: There's responsibility and accountability on both sides, right, for --?

DAVE: 100%.

WILL: Kind of, sort of, but not equally divided. Like, if you're in charge, then you're in charge, you know?

DAVE: Scope of power.

WILL: If that reporting relationship is not good and you're a people manager and managing relationships is your primary function or role, then, like, yeah, that's on you.

DAVE: You’re bad at your job.

WILL: It's not 100% you, but it's 80-20, and besides, you know, if they can't do it, like, you hired them, so what? But I'll say this, right? In the context of onboarding, right, because, like, this is something that I don't want to talk out of school. But there are some people who are having a hard time integrating organizationally. And we're bumping up against this sort of, like, lack of trust, which must be cultivated in the chain of command because they need, let's say, resources. I'm trying to be really vague, right, because it's not my story to tell. But they need resources out of the organization. They need, you know, engineering allocations, and they're not getting that, right?

Like, they're like, “Hey, I need access to this. I need explanations of this. I need a...” you know what I mean? “Endpoint opened over here. I need, like, a security review here because, like, I got a hot deadline,” and they're not getting that stuff. And they need to go up that chain of command, and they're running into that lack of trust. Somebody that I know who's also onboarding, and they've been talking to me, right, because, like, we're homies, and we started together. And we're sort of, like, blind men in the dark trying to figure out, like, how this org chart actually operates.

And, you know, I'm just like, listen, I'm just going to email the director because I am a door kicker by nature. And I know that if I go in with good intentions, I will be forgiven even if I do the wrong thing [chuckles], you know? Like, yeah, I've been running my mouth for a long time, and, like, occasionally, I screw up, but I'm always forgiven because I'm not...you know what I mean? Like, I'm not out here, you know, trying to, like, I don't know, take from the organization. I'm just like, you told me to get this thing done, and I'm running into this wall. I’m stuck.

But, like, this is exactly...like, this relationship of trust has yet to grow, and sort of this guy's like, I don't know, “What do I do? What do I do? Nobody's getting back to me. Nobody's returning my emails. I'm stuck.” And not everybody is a door kicker. It's for the best that most people don't think and act like I do. [chuckles] Society kind of depends on it.

[laughter]

DAVE: We can't all be fuel rods. Society would go nuclear. We need control rods between us.

WILL: All gas, no brakes, is not the way to run a functional engineering organization.

DAVE: Yep. All kite, no string. Yep.

MIKE: But you talked about that person getting stuck. You know, I gave examples before of some people who... switching careers later. And I’ve seen people be successful, too. So, like, I’ve seen it go both ways. And the people who are successful are the people who find themselves stuck, and then they go and they ask somebody. And if they don’t get an answer there, they go and they ask somebody else. It’s the people who are willing to do that that are successful because it’s going to happen. You are going to get stuck. It’s inevitable.

DAVE: There’s a really powerful bit of self-help going around the internet these days. I mean, it’s been going around forever, but it’s popped up on my feed, which is that you are responsible for you, right? Nobody’s coming to save you. You are here to save yourself. And yes, it’s wonderful when you’ve got co-workers and managers who will come help you, but this is just your job. You are in charge of your career, and nobody else is in charge of your career.

There are times when getting onboarded or getting the knowledge you need is going to be tough. It’s going to suck. It’s going to be hard. And what you have to do is sit down and go, well, when I started this career, did I think this career would be easier? Did I think it would be hard? No, I thought it would be hard. Okay, well, this is what hard feels like. And that can kind of help you light a fire underneath your breeches without burning your pants off. That’s a weird metaphor [laughter]. But you get what I mean. You’ve got to get up.

And so, it’s like, if you are in that 20% of the 80-20 and you’re not getting that 80, it behooves you to master your 20, to own it, try and get it to 21. And that’s the kind of thing. It might not save your job, but it will absolutely save your career. It’s a tiny, little percentage of investment that pays off over time is take the initiative and go get it. Because if you’ve got a manager like Matt who’s like, "Door’s always open," and Matt has an employee that’s like Will that will come kick the door in and say, "Hey, you closed your door, and I need it open because we need to talk," that’s going to be a good relationship. And it only takes one and a half of you to make that relationship work.

JUSTIN: So, having said, like, just to pivot a little bit, you know, you are responsible for yourself, but as leaders, you know, we’re trying to onboard these folks so that they’re most successful. And I’m bringing this up because, starting next week, I’m going to be onboarding some folks that are very remote. So, it’s, you know [laughter], I want to do what I can to make sure that they’re successful. And we’ve had a couple, you know, I’ve onboarded other folks onto my team, and, you know, so I have a process and everything. But, you know, the very remote is always difficult. What have you guys run into, like, when you’re onboarding folks that, like, you’re not sitting next to, you know, or you’re not even in the same time zone or even in the same hemisphere?

TIM: So, I will steal from what I learned from Microsoft. So, Microsoft, they scale instantly overnight, over a week. They’ll hire offshore engineers for a sprint, and then they’re done right after that sprint for two weeks.

WILL: Woow.

TIM: Or they will hire someone for an epic, which is three months, and then they’re done after that point. So, they have to get their shit figured out. By day two, everything is done and running. So, how do they do that? So, first is, the first step recipe for them is IAM automation. It’s all Entra ID, all automatically driven through Entra ID. And you plug in your own tool, whether it’s Okta, Ping, whatever you’re doing there. That’s the first step: IAM provisioning and automation.

Second is engineering environment setup. Microsoft, since 2014, has had this really cool tool called 1ES, or One Engineering System. And it does exactly what Will was saying in the kind of pre-podcast is, how do I standardize an engineer’s developer platform? That’s what Microsoft does. The One Engineering System automatically gives engineers what they need, and that image, so to say, is built and maintained and automated through their tech leads.

Second is the corporate and security, the stuff that we always overlook that we do within, like, the first four hours, but it’s, you know, it’s really important, kind of compliance policies and conditional access, and signing all the junk.

And then, they have what they call an onboarding buddy. So, they have, like, a 30-60-90-day plan. Someone is always there to answer questions, to help escalate issues. They’re setting goals for those 30-60-90s, and they have what they call buddy KPIs. So, what are you going to be trying to achieve within that period of time? Because engineers thrive on getting stuff from the to-do to the done column. And if it feels like you’re getting stuff done, you still will want to work there.

After that, it’s just like what Chloe and Jordan were saying: documentation and internal wikis. Holy cow, dude. We need it so bad. When I started, my manager said, "Okay, here’s everything that we’ve got." And he just showed me this torrential flood of mismanaged SharePoint folders and Confluence pages that were just all over the place. I’m like, how in the world am I supposed to divine what you want me to learn out of all of this? No idea. So, a well-crafted wiki system is so important.

And then, finally, the analytics and feedback loop, you know, the managers, the HR team, the recruiters, everyone is going to ask you after that 30-60-90 window, “Are we hitting the mark? What are you missing?” Stuff like that. I have never seen an organization do onboarding better, faster, and more efficiently than Microsoft. Those guys are so good at it. But again, that’s just what I’ve been exposed to. I’m sure someone else is probably better at it than them.

MATT: There is one key piece of this missing with Microsoft and how they do it, and I happen to know who their partner is. They also have dedicated teams.

TIM: Yes.

MATT: So, if they need to spin up a team, that team has already worked with them and familiar with their domain and ecosystem, which is why they can move so quickly, and that’s key. However, all of those other things are good, and, you know, everyone should be doing it. But the instant onboarding, that’s a key piece, is those teams are dedicated to Microsoft and only working for Microsoft when they get spun up.

TIM: Because the money’s there, right? They can afford that.

[laughter]

MIKE: You just pointed...That’s a fascinating thing you pointed out there, that the team was prepared to join. You know, interesting, some years ago, when I say some years ago, like, 30 years ago, maybe more than 30 years ago [chuckles], there was a movie that came out. What was it called? Stand and Deliver, maybe, about a teacher who took his inner-city school math class and helped them all pass the calculus AP test. And it was this inspirational movie. Everybody watched it.

And I’ve read some of the back story about that, and they left out...and it’s true; there was this inspirational teacher based on a real person who got his class up there. But they left out something really important. This math teacher had been working with all the teachers in the junior high and in the previous levels in the high school for years, prepping them to teach those students. So, by the time they got to his class, they were already ready, and if you leave out that part, it doesn’t work.

There’s some baseline preparation that you have to have. And it is an inspirational story, but they left out the best part is that he did the work. If you’re going to make this work, you have to do the prep work and lay the foundation. Microsoft, it sounds like they can make this work because they have partners that already know, you know, they’ve got people who already kind of know what’s going on that --

WILL: It sounds a lot. I mean, I love the system, right? Like, they did all the stuff that I just dreamed of, right? I think that’s so cool [laughter]. But at the same time, it sounds a lot closer to, like, an internal transfer than onboarding a new hire, right? It sounds like these are, like, "We’re going to put these assets that have worked with us a bunch and know our systems and stuff like that," and like...Like, oh man, I was twisting arms for a week, literally, to get access to all the Git repos. I was chasing people down.

I mean, in terms of, like, I mean, we can call this directly, right? You’re talking about onboarding India hires, right, or people from Asia, where you’re on the other side of the world, right? I mean, the biggest thing that I have seen people really trip up with, like, these cross-time-zone hires, right? Because it doesn’t really matter. It doesn’t matter so much, like, cross-country, but, like, cross-time-zone, you put people on an island where they don’t have real-time support.

It’s like, I can’t work like that, or, more to the point, I can’t work effectively and efficiently like that, and it’s profoundly demoralizing. Like, if your first two weeks are just sort of, like, sitting there like, [vocalization], I work 15 minutes, stop, wait eight hours for people to wake up, right, like, that’s brutal. I don’t know. I mean, like, I’m pretty good at my job, but I’m not that good. And it’s profoundly demoralizing. And it’s sort of, like, I don’t know. I think it establishes a relationship where, like, you’re kind of, like, you know, you know what I mean? And then, if you treat people like that, then they produce like that.

So, I mean, the thing is, if I could do anything, I mean, and this is not, you know, necessarily good news, but, like, people are going to lose some sleep, you know? Them and you, you know, because, like, you just got to have a block of time where, like, you’re just getting them up and running, you know?

MIKE:  Exactly.

JUSTIN: I really like that. It's like, you can prep all you want, but you got to invest in them for them to be successful. And all the time that you invest in them it's definitely going to cost you something. And you may have to adjust your hours for a while, but it pays off long term. And, you know, whether that's on the front end where you are prepping all the documents and prepping all the training videos and things like that, or it's on your own time where you are, you know, walking them through the application and answering all their questions, and debugging an issue, and walking with them, pair programming through their first PR, you know, all of that is well worth the time to me because, you know, that pair programming time is, you know, it takes up your time, but it enables them to be much more successful.

WILL: Well, I mean, like, a broader thing, but, I mean, like, in all honesty, you know, like, you never really get out of it, right? I mean, like, you could never, you know, I mean, like, if you're going to have somebody on the other side of the world, like, okay, we all know why we do that. But, like, you know, that relationship still has to be cultivated.

Honestly, I've been in a lot of places where they didn't really...I don't know, they're a human being, and they're a human being just like you. And they have the same feelings that you have, and they have the same everything. They're just as smart as you, and if you treat them like an equal, they'll produce for you. And I've seen so many places that just completely...they completely bungle that relationship, and then, you know, everybody knows the horror stories, and, like, I've seen it over and over and over and over and over again.

Because, like, when you, yeah, you're saving a bunch of money, and that's fine, but, like, you're also taking on managing this relationship across this very challenging set of circumstances. And, like, if you're just sort of, like, just trying to save a buck, and you don't want to treat people fundamentally with the exact same amount of respect that you would expect somebody to treat you with, then, like, you know, you're going to get what you get.

MIKE: Well, and you don't save money either.

WILL: No, no, no. No, you do not.

MIKE: Do you know how expensive it is to hire a huge team of people that you then completely ignore and can't accomplish anything for months at a time [laughs]?

WILL: Oh my God.

MIKE: You develop a culture where nobody cares whatsoever because they're not paid attention to. It doesn't matter what they do.

[crosstalk 39:09]

DAVE: They care. They care. They're just caring about the incentives.

WILL: I don’t think [inaudible 39:12]. It ain't your father's India offshore. Like, the developer market in India is competitive these days. If you treat people like a dog, they're going to bail, and if they're good, Microsoft has a shop in India, and so does Google, and so does Facebook. And there are bigger fish than you swimming in this pond, and, like, good people are not any easier to find there than here. Your good people, like, they'll split.

Like, I was surprised at how competitive some place that I was working with before. They were spinning up in an India office, and, like, we had some guys that were like, yeah, hell yeah, like, yeah, this is...okay. All right, I'm [inaudible 40:00]. It sharpened my game up a little bit. This guy is giving me a run for my money. Like, oh, he got a job with Netflix. Bye.

MIKE: You know, I love what you said about treating people as the human beings they are. You can't have an effective relationship without a relationship. It’s not going to work. And I've seen it be very successful working with teams from all over the world. And it's harder the farther apart the time zones are. It absolutely gets harder. Every hour you add to there is another level of difficulty. But if you put in that overlap, are willing to make it work, take that time, it can become a well-oiled machine. And you can have great success with very talented people from wherever.

KYLE: I think, too, one thing that I've had to learn is cultural differences with these distributed teams, right? Yeah, it's easier. Time frame does make a difference. But also, the culture just really affects, like, how you're talking to them and how you might be understanding why they might be stuck or something, right? Because I've worked with teams in, you know, India or in Poland, and they're closer in time zones. But it's definitely different hurdles.

WILL: Oh, man, I love an Asian-European team. If you ever wanted it straight, man, they'll give it to you.

[laughter]

MATT: It's true. We work with some good ones.

WILL: Yeah, wear your helmet, but, like, you'll definitely get that feedback.

[laughter]

MIKE: The cultural differences are a real thing. In some cultures, I have noticed this...we're talking some about India. It's probably not universal, but there is often a hesitance, I found, from some people to speak up, to ask questions because they feel like, oh, I shouldn't be asking people. I should just know this. And it takes some extra work, and good people will recognize that, read the room, like, okay, they're wanting to talk to me, extra work to reach out and say, “Hey...” make it clear that you want them to ask. Make yourself available. Show that vulnerability. Let them know that you're willing to have that relationship. I think that's really important.

TIM: What I learned...and this is coming from someone from Bangladesh who told me, you know, every time you work with a team offshore in these areas, always explain what you want people to do. And it's customary or culturally acceptable for the person who is listening to you to constantly say, “Yes, yes, yes,” to be an active listener. But over at Stateside, when someone says yes like that, it's generally an acknowledgment that I understand what you want me to do, and I will do that. But over there, it's just kind of that unconscious that's how we proceed the conversation forward, not necessarily that --

So, his feedback was, at the end of describing what the task is, ask that individual to repeat it back to you in their own words so that you know the translation, the communication process has occurred successfully. Otherwise, it's almost a surefire recipe for miscommunication. To be honest, that doesn't even have to be, like, an Americanism to another culture. That's probably just good practice for any kind of communication.

MATT: Yeah, that's actually a really good observation, and I haven't thought much about that, but I think it's probably correct. You know, we've been doing a lot of talking, but I'd be really interested to hear some perspective from Chloe and Jordan. A lot of us are leaders that have been talking, and as you guys are coming into this [laughter] career and going through the process, I'd love to hear what your thoughts are. What are some things maybe we're missing that we could help you with?

CHLOE: I can share. One of the most helpful things that's happened since I've been here was, like, very early on, there was a developer lunch. And so, we all kind of were able to go out, have lunch together, and get to know one another. And I was able to get to know one of the team leads (Later that day, we were going to shadow), and just had a very normal conversation. He likes to play video games. My husband plays a ton of video games. We talked about that.

And so, when I went to go shadow him earlier, I felt so much more comfortable asking questions because it wasn't like, this is a team lead that I don't know. It was, oh, this is a person. Because I think, as you guys are talking about how people that we're hiring are people, we should treat them like that, but so are our managers, the leaders in the company. They're also people. And so, I think having that experience made me feel so much more comfortable asking questions, and it didn't feel like the questions were unwanted.

And I think as well with that is, as someone new, I have so many questions, and it's not fair to put that all on one person. Even if they're so well-meaning and they want to answer the questions, they probably don't have the time. So, creating an opportunity where there is a wide range of people to ask questions to helps each of those people not feel overwhelmed. That also helps me not feel like I'm the annoying one asking this one person questions all the time, and also, I get to create more relationships. And so, those are some things that have been helpful, but I think just continuing to have a pool of people for me to go to or for new hires to go to is extremely helpful.

MATT: Do you think, and this is something that we haven't historically done, but what you just said made me think it might be a good idea, do you think something like a roundtable with current employees and full-time staffers would be helpful?

CHLOE: It depends on the nature of the roundtable, but I think, yeah, it could be very helpful of just kind of giving some of that feedback, but also just creating an environment where questions are open, and you can get to know a lot of people at one time.

MIKE: Well, so one thing we did with the interns this year is we identified a group of people, and we're deliberately having them rotate through a pool of people rather than just dropping them on one person like, hey, “Here's your job [chuckles]. Help these people,” because then they have split loyalties, and they don't have time. But having a pool of people they can work with, we very deliberately set that up so they could have a group of people to ask questions to.

I know I'm going to be talking to this lead tomorrow, and this lead the next day, and so on, and then that lead can assign somebody else on their team. So, there's an intent to give structure, to give them lots of opportunities to meet with that group of people and have many people to ask and not just one. I don't know if that's been useful or not. So, I'm curious for feedback on that.

Before I ask, though, one other thing that Chloe and Jordan have done is they've compiled lists of questions, and that was awesome. I've already mentioned that to them. Multiple times...I've been working closely with them this year. I've received a list of questions in a document. Can you answer this, this, this, this, this, and this? We've already explored the options, and now I've got these questions. And that is so helpful because it's a little bit asynchronous. I can go through it when I get a chance. These are well thought out, and it's documented. I can put the answers there. Being willing to get those lists of questions together was actually a really helpful thing.

MATT: Also, something we could share with future internships, so...

MIKE: Absolutely.

MATT: Really valuable.

MIKE: So, back to the question. I'm curious if having the pool of people was helpful. Maybe it wasn't. Maybe it was a failed idea. And I'm also interested in your thoughts, Jordan, because Chloe has shared her thoughts, but you haven't this time. You shared at the beginning, but coming back --

JORDAN: I kind of cheat a little bit because this isn't my first time here, so I know some people.

MIKE: That’s true.

JORDAN: But I think just having a pool of people that you can go to for questions is phenomenally helpful. For example, just onboarding in general, I had multiple people I could ask for different services and all that, but I think being able to know or meet someone is great. And so, I think on the subject of having a list of people that we can go to every day, I think it's very valuable. Having one mentor is nice, and they'll have lots of context on what you're doing and be able to help you very quickly, but that is to the detriment of them not having any time for themselves. So, I think it is valuable to have multiple or even just having someone to help you, so...

TIM: To Chloe and Jordan, I'm curious, would it be more valuable to have a buddy that you can ask for help when you need it? Like, Jordan, you just described, it's valuable, but it could be to their detriment. Or would it be more valuable to be plugged into a Slack channel or a Teams channel where you have onboarding specialists, people who know the questions you're going to ask, and it's a safe space for you to ask stupid questions like, “Hey, I can't get to this. How do I get to this?” Or “I need to submit a ServiceNow ticket. Who do I assign this kind of ticket to?” Which would be more valuable as a newly employed person?

JORDAN: Ideally, both [laughs]. And I say this because having one person that you can ask quick questions to, so you don't have to wait for a random small thing for, I don't know, a couple of hours, is very nice. But, at the same time, being able to document the questions, like the intern chat, was so helpful this year. Like, I came across some personal blockers that, coincidentally, someone else had experienced last year, and I could just go back and read the thread. And if I'm missing some context, I could just ask very quickly to get up to speed. But I think having both a safe space and someone, or people, to ask very quick, simple questions to is great.

TIM: Like the Stack Overflow of onboarding. There's a place you can look for commonly asked questions. Yeah, that's good. I'll be honest, I've been here with Acima for, what, nine months or something like that, and I still will run into things like, where do I send this stupid ticket to? And I debate. I sit there for, like, three minutes. Like, do I put this in general engineering and ask and look like that one guy that can't seem to find that one Confluence page? Or do I just bug someone who's already being bugged [laughs]?

WILL: I love how we keep on looping back to that psychological safety, where it's like, well, I feel bad because I have one person that's onboarding me, even though you have one person that has been specifically selected and tasked to onboard you. And it's like, oh, but I'm asking them so many questions, right? And it feels uncomfortable.

It's totally...like, my point there is not, like...my point there is just, like, this psychological safety and these relationships and bonds and trust and integrating into a community, like, that psychological safety, like, it's so critical. And that is just something that has to be grown. And, as leaders, as people who are already embedded in the community, if you leave anybody with anything, it's to keep that at the top of your mind, and, like, growing that, and, like, being cognizant and aware of its lack, of its absence, right? Because, like, people just...everybody, right?

I mean, you have, like, people who are really senior, who've been doing this for decades, and we're still talking about this stuff, right? Like, me, like, I'm going through it the same way as you. And I suppose, like, the biggest difference is not in how I feel about it, but, like, that I just know that if I do the right things, I'll get the right results. So, I'm okay with, like, my trust fall, you know, into the organization.

MATT: Someone who's really, really good at providing that for people happens to be on this call, and that's Mike. People just feel safe working with Mike. And, you know, I used to report to Mike, and immediately, I just felt that way with him. Because he's there; he's reaching out, and he's making you feel that comfort. And I think, you know, as leaders, we need to be cognizant of that and take an example, right?

And as new people coming on...and it's one of the hardest things to overcome. It really is. That's why a lot of people struggle with paired programming, right? It's the ego, and I don't want to feel dumb. And what are they going to think if I mess up? We need to just accept that we're all human beings. We all mess up, and it's okay. And it's okay to ask those questions, even if it may feel dumb, you know, you're asking it for a reason.

CHLOE: I think I'd really just like to echo that sentiment. I think, coming into this internship, I felt, you know, very overwhelmed because it was in a different language and never had this kind of experience. And one of the first things that Mike said was, “You're not supposed to know everything.” And so, I think that took off the weight.

I think when you're interviewing, you have this pressure of, I need to prove myself that I'm worthy for this position. And then, when you get into the position, I think sometimes we still have that mindset of, I'm still proving myself. But you've got the position, and I think it's an opportunity for you to be, like, it's okay if I don't know everything. I got the position. I'm here. Now it's time for me to learn. And so, I think Mike did a really great job of setting that as the expectation was, I'm not going to know, but I'm here to learn. And I think if we can do that for a lot of people, I think that would help.

KYLE: I would wonder, Mike, you're saying that we've put the interns with a bucket of leaders, or people that they can cross-train with. Have auxiliary teams been included in this? Because do they feel comfortable going to IT, DevOps, security?

MIKE: We have put them with data, but we have not put them with some of the other teams yet. Although I'd love, actually, to have them rotate through some of those other teams. I think that'd be really helpful.

KYLE: Because, yeah, I wouldn't want them to be, you know, I'm on DevOps, and Tim over here is on security. I wouldn't want them to feel timid to reach out to either of us for anything, too.

JORDAN: You know, I think something that kind of happened as a side effect was that we were seated near data and IT, and IT, they're nice, and so they would talk to us. And I think as a result of being able to sit close to them, I'm more willing to go up and talk to them, so I just wanted to point that out.

WILL: [inaudible 55:20] is not all bad [laughs]. Mike wouldn't know [laughter].

MIKE: So, we've talked about a lot of things, and we've been here for a little while. We've talked some from the psychological safety aspect from the leaders. We've also talked a lot about your role as a person being onboarded, that you're going to have to assert yourself a bit. You're going to have to find that way, and you're going to feel stupid [laughs]. That's the reality of doing something new. And that also speaks to leaders. Well, recognize that you've got a bunch of people who are feeling that way, and try to give them the support they need in that very uncomfortable situation, and try to help them get through that quickly.

TIM: I think what's interesting the most out of this conversation, for me, and it speaks to kind of being a little myopic to my role. But every day I fight challenges with automating a lot of the IAM mechanisms of onboarding people, or fighting, how do we get the engineering environment set up, up and running?

But I swear, I think 80% of this conversation has been the soft skill side, the aspect of onboarding, not so much, okay, we get a new employee, and I have to populate a million attributes about this person in our directory, and try to build an automation so that if they have this job title, they automatically get these roles, and stuff like that. That's where my head is at all the time because, in my mind, that is a successful onboarding, that on day one, they have all the roles they need. But when you talk to, I think, employees at large, they're kind of more like, I actually care more about a buddy that's there for me. And I think that's absolutely true.

We hired a person for our team, and it was just to his virtue that I had gone through the same wall that he had months before, and I just had a list of everything that needed to get done because I had just done that. And I think that experience of him being able to lean on me and say, “Okay, what did you do to get things up and running?” I had that list, and that was more valuable to that person as an employee than a shiny system that everything figured out. Which doesn't mean I don't want the shiny system, but I think it's interesting that the human aspect is more in focus than I thought it would be.

MATT: Yeah, I think, and probably true for about every career, but those soft skills, communication, trust, are every bit as important as technical skills.

WILL: I mean, I keep on, like, I don't know, the further I go, the more I think it's more that than anything else. On a technical level, I've had really, really hard technical jobs, really hard technical jobs. But this isn't that. This isn't, you know what I mean, this isn't cutting-edge engineering research.

This is business logic, line-of-business stuff, making sure, you know, I mean, it can be tricky to take a block out of the Jenga tower without knocking the whole thing over, and that can be hard. But it's all about relationships and networks and people and, like, how you get people together to do the job. Like, that is the challenge of this role, not, like, you know, some fancy algorithm, you know, whatever LeetCode may have told you, but you're not going to be doing any of that.

MIKE: No. Maybe if you're a Ph.D. and you're getting hired to do some research somewhere, and those relationships are still going to matter [chuckles], even if you're working on that tricky technical problem, which you almost certainly aren't.

Thank you for joining us, Chloe and Jordan. It's been great having you, and thanks to everybody. It's been nice having a big pool here of people, with everybody contributing and offering different perspectives. It, I think, gave us some really good food for thought, how to do this better, do something that's really hard better.

Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, host Mike is joined by Kyle, Matt, and Ryan to discuss how technical interviews are evolving in the age of AI tools like ChatGPT and Copilot. The conversation begins with a story about a candidate who plagiarized code during an interview—years before modern AI assistance—which sets the stage for a deeper discussion on integrity and how easy it now is to cheat during remote interviews. The group reflects on how traditional coding challenges and rote questions are becoming less effective, given the accessibility of fast, accurate answers online.

Instead of focusing on textbook knowledge or code syntax, the team advocates for interview methods that prioritize problem-solving, communication, and curiosity. They emphasize the value of questions like “Tell me about a project you’re proud of,” or “When did you break production?”—questions that reveal depth of experience, passion, and the candidate’s ability to troubleshoot in real-world scenarios. They also suggest leaning into pseudocode or open-ended challenges that require candidates to explain their thinking, not just deliver a solution. These formats are harder to fake and better assess a person’s reasoning and learning capabilities.

The conversation concludes with a shared sentiment that the role of engineers is shifting away from just knowing code toward applying it creatively and collaboratively to solve meaningful problems. Tools like ChatGPT can assist, but they can’t replace real understanding, adaptability, and team skills. For both candidates and interviewers, the core takeaway is clear: be honest, be curious, and focus on showing—not telling—how you think.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike, and I'm hosting again today. With me, I have Kyle from our platform engineering team, who often joins us, and Matt from...well, you do a bunch of things, Matt [laughs]. He's a leader, I'll say that, and he's a director here at Acima, and is involved in doing a lot of things.

I'd like to defer introducing the topic and tell a story. I'm going to do that for...oh, and we also got with us, oh, there we go. We've got Ryan, who's a delivery manager here at Acima and should have some great input.

I'm going to start by telling a story. Oh boy, how long ago was this? Over, actually, I don't remember how long ago it was. I think [laughs] I don't even remember which company it was at [laughs], whether it was at Acima or not. I think that this was, like, 12 years ago, so I think this was prior to Acima. I was interviewing somebody, and I thought that I'd give him a coding test. It was a little hard to tell whether he knew what he was talking about or not. And I think I asked him to write some Ruby code that would output a Fibonacci sequence, output the first 10 digits of a Fibonacci sequence. And he said, "Okay,” and he just got quiet for a bit. This was a remote interview, by the way. I think he was overseas.

I waited a couple of minutes. He came, and he dropped some code, and there it was, and it worked. It was weird. It was, like, weird code [laughs]. There's stuff that's idiomatically normal for the language that you're writing in. And this was Ruby code, but it barely looked like Ruby code. It looked like somebody had pulled it out of some other piece of code that was doing something else and then tweaked it. It barely fit. It was just really strange. And you don't usually write really strange code when you're on the spot. You're going to write something really simple. And you might have a couple of quirks, but you're not going to have something that's just really weird.

So, I finished our interview, and I went and I Googled some of the code because it was weird enough that it was unique. I Googled for it, and it came up immediately on Google. He just copied and pasted from something he’d found online. And not only did he not get hired, but the contract shop he was coming with took a real ding because of that. There were some bad vibes thereafter [chuckles] between us and the shop because you sent us somebody, and he just cheated on the interview.

That was before all of the AI tools that we've got now [chuckles], before ChatGPT. Think about all the changes that have happened over the past 15 years or so. This was prior to that, and still managed to cheat. Well, he didn't get away with it [laughs], but he still managed to cheat the interview.

And that introduces our topic today. We're going to be talking about how to deal with technology, how to do interviewing in a world where we have technology now, where it makes it very easy for people to pull out answers quickly and hard to verify.

This is more broadly applicable even outside of engineering, you think about in schools trying to do testing. It's a general problem. We're going to talk specifically about the software engineering world, because that's what we do, and how to do interviews. How to do your interviews broadly, but specifically, how do you do interviews when it's really easy to cheat? So, I've told a story about [chuckles] somebody doing that sort of thing. Any of you have some stories or similar experiences that you've had?

MATT: Well, I don't know if I have a story. However, prior to this call, I was just conducting an interview. And something I think is really important, especially these days where a lot of people are working remote and interviews happen virtually, make sure their camera's on. And ask questions that, A, might not be as easy to Google, but require some conversation. And a demonstration of understanding of concepts, I think, is important. I think these days you have to dig a little bit deeper and ask the whys, just not the hows, on a lot of these technical questions.

MIKE: I've had other interviews where I've asked somebody a question, a technical question that was a little broad about architecture, and they answered it really fast. He talked about extracting services, for example, to clean up code and decouple things. And they answered them really well and quickly on all of them, but their answers were brief. This was, like, they gave the right answer really quickly and then never elaborated.

And I'm thinking about one example where this candidate did that repeatedly, all the right answers, one after the other, like, "Okay, they seem like a pretty good candidate." Then, when we actually employed them, they didn't seem to do anything [laughs]. They apparently were really good at either Googling the question or had researched interview questions before, so they knew what to say, but they didn't actually have the deep knowledge. They were able to give quick answers.

So, one thing I've learned is a red flag, and you've touched on it, Matt, you've got to ask questions that allow them to elaborate. Talk about that “Why?” That conversation is hard to duplicate. You can't fake it in the same way. If you can come up with a tool that will do that, well, then you've got a tool that can maybe do the job, but we’re not there yet.

MATT: Or if I have my phone with ChatGPT on listening in conversational chat, it's going to respond as fast as I could, right? And that makes it a little tricky. By the way, I think I was present in that same interview with you.

MIKE: [laughs]. You may have been.

RYAN: So, I guess the question here is, are you guys against using such tools like Google and LLMs, agents out there that an interviewee can use to answer your question?

MATT: Well, I want people to have an understanding of what they're being asked. I use tools all the time to help me with my work, and that's just the way of the world these days, right? If tools are available, let's use them. But I need to understand what I'm doing, or the next problem that comes along, I'm not going to solve properly. And these tools are not a replacement for a human being.

All of us have probably used Codepilot at some point, or Copilot at some point, or Cody, or something similar out there, right? That code needs review, and it doesn't always solve your problem the way that problem should be solved. So, yes, I support using tools, but if I'm hiring someone, I want them to understand the problems they're trying to solve.

RYAN: And I think that is the crucial point here that we want to discuss is, like, because ChatGPT or any of the LLM model what do they do? They go and surf the internet, right? So, when you get your answer back, it doesn't know whether that's the right answer or not. It just knows that this is an example it found somewhere on the internet, right? And the person who put that together had to understand what they put together. So, when Mike said, "Well, this guy found that answer on the internet, and it worked.” So, why does it matter if it's strange? If he understands and he can explain why, so why does it matter at that point if it's strange?

MIKE: Well, you hit on something important. If he had told me, "Well, I'm going to Google something and then talk about it," then that would have changed the whole conversation, and we might have even hired the guy. That's not what happened. He passed off the work as his own.

MATT: Integrity is important.

MIKE: Yeah. So, there's deception there. I expect people to Google for their job [laughs]. I think we've talked about it here. That's a key aspect of engineering. It's hard to do engineering without Google. I remember the first time I used it. My boss told me, “Go use Google.”

RYAN: It's going to change the nature of how we do interviews because there's no way we can...even with camera on, and even if we ask the questions that need interaction, there's no way to prevent them from looking up information on the side screen. We can't see it. I mean, you can probably have a big enough screen with multiple windows open. You can do that right there and get away with it pretty easily. But I think the nature of the interview has to change. Instead of asking, “Hey, go write me a function that generates Fibonacci or do a binary sort,” or something like that...because that kind of stuff is not useful, right?

But you can say, “Hey, I have this problem here. How do you solve it for me? And feel free to use the tool that's available out there. You can use ChatGPT or whatever out there. But you need to be able to explain why you put that solution together for me and explain it to us,” right? I think that the nature of the interview has to be like that, instead of asking what or how to do some basic stuff. We have to move beyond that type of interview.

MATT: I agree with that. You know, I'm not one who's a real big fan of code challenges anyway, but, you know, maybe it's as easy as, “Explain to me in pseudocode how you would solve this problem,” right? And I'm not going to give you some academic computer science problem because most of the time they aren't real applicable in the real world unless you're doing low-level stuff, right? But a real-world problem.

RYAN: Yeah, yeah, yeah.

MIKE: I had the same thought about pseudocode as I was preparing for this session. I thought, you know, pseudocode goes a long way because you can't usually Google it [laughs]. But even --

RYAN: But you can put your thought process in there, and you can see it, you know, forming. And even though it doesn't compile, compiling might give people, the interviewee, that anxiety, like, oh, code doesn't compile [chuckles], you know, and then they just drift away from the main problem they’re trying to solve.

MIKE: Which is what you're after, right? You're trying to see how people think about the problem.

MATT: Yes.

MIKE: The solution itself is almost irrelevant [laughs].

MATT: It's the approach. I wholeheartedly agree. And I think there's a soft skill involved here as well. And that is just the more you interview people, the more you can read people, right? The more human interaction you have, the more you can read them. If I see someone on camera and they're taking a second to answer a question, I can see if the gears are turning in their head, versus if they're trying to type something on a keyboard quickly, or vice versa, right? If someone's thinking something through, I think for the most part, you can pick up on that. But we're getting into a whole new world of the way we have to do things these days.

KYLE: I have to kind of tack on with what Matt was saying there. That's kind of...when I've been interviewing, I haven't necessarily cared even what level a person is because regardless of the level, it's how much can they learn? And I think that's more of what I focus on in interviews is like, does this person seem like someone that I can teach, someone that can pick up on what I'm trying to tell them? And, respectively, a junior I would give more leeway to.

But it's one of those things where, like we've spoken, do we really care that it's solving these generic coding challenges, or handing them a marker and telling them to go write on the whiteboard, even on camera? Just, I want to see what your thought process is. Can you solve this? And I almost feel like we can still use those coding examples or whatever it is that we're wanting to use. But even historically before AI, I would give someone homework in the sense of like, “Hey, go spin up an EC2 instance, and give me a script to do that.” And then, I would judge based upon the quality of that script.

What script did they do it in? Did they use Terraform? Did they use Pulumi? How did they do it? How repeatable is it, or did they just give me a set of instructions to do it? What was the thought process behind this, and how far along are they? I was judging on those criteria because you can, even in today's world, you can ask AI, “Well, how do I spin up an EC2 instance?” Well, depending on, you know, it's going to give you answers, right? But are those the answers that you want to give during an interview?

So, I think we kind of need to take those into consideration, too, because, yeah, these generic answers aren't what we're all going to be wanting. And even taking issues that we're currently running into as a team, I guess, and saying, “Hey, this is what I'm actually facing. If you were on my team, how would you solve this?” and kind of seeing how they would think through that that would be helpful, too.

MATT: One of the things, and Mike's probably sick of hearing me say this because I sound like a broken record in all of our meetings, more important to me than ability to write code is ability to communicate and ability to solve problems. If you have those two things, the code can be taught. I don't see that as even being a big hurdle. But those are the things I'm really looking for, and AI just can't fake that.

KYLE: Well, and in today's world, right? I mean, code can be taught. At some point, will code even be needed? You know, that troubleshooting and ability to learn and communicate is going to be the top skills that we're looking for.

MIKE: I think it already was, and it just increases it, right?

KYLE: Yeah, more prevalent, right?

MIKE: Yeah, because you're removing some of the esoteric knowledge that was, to some degree, a waste of time, you know [chuckles]. If you have to spend a long time learning the language, that may not be a good indication that you're using the right tools for the job. Instead, you want somebody who can use the best human skills, their ingenuity, their creativity, to come up with an effective solution, and then make tools available that can make it easy to put that solution into practice.

KYLE: An engineer I used to work with at a previous company I always thought he had an interesting mentality for when he was doing interviews for his team, and this was way before AI. But I would ask him...because we were a Java shop. But I was in one of his interviews, and I was like, “Why did you not have any Java-related questions? Why aren't we worried about whether or not these people know Java?” And he said, “If I'm looking for a Java developer, it's just that. They are a developer. I'm looking for an engineer. I'm looking for somebody that can use any language, so I don't care how much they know Java.”

And that's always kind of stuck with me. It's just kind of like, okay, there are situations...I understand there are situations where you want a Ruby pro or a Java pro, but there are situations where you're wanting an engineer, someone that can use any tool, and you want to vet out that situation.

MATT: I think I want those more so than a specialist in any language.

KYLE: Yeah, exactly.

MIKE: Yeah. In the end, our job [chuckles] isn't to write code; our job is to solve problems. And the difference matters.

MATT: It does. I don't need to be able to write code to make software do something, right? I need to know how to solve the problem that we need to solve, and I can use code assistance to help me write the code if it's in a new language. I can, you know, yes, I need an understanding of software engineering and design patterns and, you know, all of those important things. But anyone who wants to or is a software engineer should really focus on being language agnostic because you should be using the right tools for the job at the right time.

MIKE: I don't want to give the wrong impression that you shouldn't get good at using your tools.

MATT: Oh, absolutely not.

MIKE: I agree with what you're saying. But one thing I've heard said is you should get really good at least one language, and I agree with that. It's a mark of professionalism [laughs]. And, you know, if you care about the tools you use, you'll learn at least one of them really, really well. And they do differ. It does matter some. Ryan is really good at functional language, who’s here with us, and they tend to approach a problem in a different way than more procedural languages.

Likewise, if I was interviewing somebody who's completely unfamiliar with object-oriented aspects of a language and I was doing Java, I'd [chuckles] know that there was going to be a lot more time required for them to learn the tool. So, I do think that there's something to be said for knowing something, but I think that that kind of expertise will probably decline somewhat in importance over the coming years as the ability to automate a lot of that work increases.

MATT: Someone can be good at a language, you know, and you see this really often in the Ruby community because all the bootcamps for years were Ruby, right? And people know how to do things in Ruby, but they don't understand the language and what's happening behind the scenes and under the hood. So, I think, to your point, Mike, I think knowing a language really well and being an expert at a language, you have to understand what's going on and why you're doing the things you're doing. And I agree with that 100%.

KYLE: This kind of goes back to a previous topic which we had, which was learning languages. I can't remember what the topic was specifically, but it's entirely that, where you get good at one language, and it makes learning new languages that much easier. And that will show in an interview.

RYAN: I have a story about, like, doing interviewing. So, I'm a big fan of Elon Musk [laughs]. People don't like him. It doesn't matter. The way the guy approaches interviews or his work ethic it's just something I'm a big fan. So, I follow him closely and listen to a lot of the things that he posts.

So, one of the things he said about how you weed out the people who just know how to talk really well during an interview versus the people who actually do the work on a specific project is to ask them to tell him about the detail of that project. So, he was like, you know, “Tell me something that you...some code you worked on in the past,” and then drill them on it because the people who actually worked on it they’d get excited about that kind of stuff.

They would go into detail that you didn't even care to know, but that would tell him whether this person actually worked on something, or they just read about it, or just heard about it. Because when you get to a certain degree, you can talk your way through a lot of these things without having the detail of it, right? But when you have to actually explain the detail of a certain implementation to solve that problem, then it shows whether you actually worked on that problem or not.

So, one of the guys that I interviewed with in the past he kept going on and on about how performance tuning is one of his biggest hobbies that he enjoys. It's not even his main job. His main job is a developer, but he would spend time on his own and try to get into codes and try to fine-tune it, right? And I thought that's an interesting subject. Then I started drilling him on it and said, “Okay, so what...” It was in .NET, and it was an ASP.NET web application, but they run on Windows server. It was not even on .NET Core.

But anyway, so when I started drilling him on what he'd done on it, he'd actually start explaining stuff that people not only wouldn't know. He would go into how he would go to the web server, change the thread pool so that you have, by default, 100 thread pool. He'd maximize it up to 1,000 thread pool on the web server because he'd only get one dev machine to test his code. So, he wanted to be able to run concurrent, high-concurrency code, [inaudible 22:19] high throughput. He was like, “I figured out how to get into the configure at the root and change the thread pool to allow 1,000 threads by default.” And he would increase the performance for the process power of the CPU up to 100%. He'd max out that CPU running on that server. And he'd just go and do that kind of stuff.

When you hear that, it's the enthusiasm behind the story. The excitement behind the story is kind of a telltale sign that he actually worked on it, and he knows what he’s doing there. I mean, of course, I hired that guy [laughter]. But nobody goes into a web server and changes the thread pools. And when you work on Windows, it's like the worker thread pool versus the IO thread pool are different. And people who don't just write web server code they don't understand what that means. One is run the receiving of the request come in. One is run all of the IOs processing behind the scene. But they have two different thread pools. And he actually understood how to explain all of that. I was like, “Wow, okay, you impressed me [laughs].”

But, you know, I mean, that's the kind of thing I've been using as my tactic on the interview. I ask that kind of question, drill them on it. And if they don't give me enough information, then I'll just mark that as, you know, he didn't really actually work on it.

MIKE: That's actually become my favorite interview question as well. “Tell me about a project you're proud of and go into detail about what was involved.” It's interesting hearing what people are proud of, and then interesting hearing about the details. If they’re like, “Oh, yeah, I don't remember it very well...”

RYAN: You didn’t work on it [laughter]. I would make that assumption. If you don't remember, you didn't work on it. Because if you actually spend days and nights trying to make it work, you remember why it didn't work or why it worked [laughs].

MIKE: Absolutely.

MATT: Days and nights.

RYAN: They might not remember the code level, but they know exactly what the problem is, how they solved it, how they figured out what the problem is. Most of the time, you just had a problem. You don't know what the root cause is. And just to spend time on it and figure out what the root cause is, it kind of differentiates the type of people who have no problem putting the time to either solve a problem that they're interested in. Whereas if somebody just showed up and, you know, meet their hours and go home, it's --

MIKE: I can think of...Yeah, you just talked about that. I can think of a clever solution I was proud of that I implemented over 20 years ago [chuckles]. And I couldn't tell you at all what the code did. I remember this because a few years later, the company reached out, and I hadn't worked there in years. The company reached out to me. Like, “Do you remember how this worked [chuckles]?” Because it was in Java, and I don't think they had any Java engineers on staff. They didn't know what was going on.

So, they just reached out. Like, “Hey, we found in the commit log who wrote this,” tracked me down somehow and reached out to me and said, “Hey, can you help us with this?” And I talked them through it because I could still, right? Because I remember that, that I could go back and talk about it. And I remembered because when I looked at the code, I didn't recognize the code at all. I hadn't looked at that code in years, and I had no familiarity with it. But I remember exactly how it worked. And if somebody can't say that, I think you're right. They weren't engaged or, more likely, they really weren't making those decisions.

MATT: They weren't interested in making a difference. And those are the kind of people we want, right? People who come in, want to make a difference, want to be innovative, and think about those tough problems. The level of gratification you get out of solving really tough problems for a company, even if you don't get compensated for them, it's still huge. It's a big deal. And it's something to be proud of.

MIKE: Yeah. We could probably all think about solutions we're really proud of, even from a long time ago.

RYAN: That's why we get into tech in the first place, right?

MIKE: [laughs] Yeah.

RYAN: It’s those kinds of little wins that keep us going [laughs].

MIKE: We all probably have some significant other in our lives who's seen us reach that moment where we solve the problem, whether it's debugging [laughs], or you finally get there. And they know [laughs]. They've seen it a few times [laughs].

MATT: Absolutely possessed or obsessive in the process. You haven't been to sleep for two days [laughs].

RYAN: There was a...Oh man, this was a fun problem that I had to deal with. So, I was a .NET programmer for a long time [laughs], so a lot of my fun stories come from the .NET days. I think by the time I got to Haskell, things just got easier because of Haskell [laughs]. It’s crazy because when you get to Haskell, it gets boring because things just work [laughs]. I think that’s --

MATT: Said no one ever, Ryan.

[laughter]

RYAN: I was like --

MIKE: It's true. It just takes longer to get to the working.

RYAN: Yeah. So, when it works, it just runs, right? You don't have to do anything. So, on that day, we released these brand-new features. I used to work for this analytics company where we bring in customer or user survey data and then do all kinds of analysis on it. So, the different kinds of math equations that we have to run on the data one guy decided that instead of using the data that had been passed through the pipeline, he would clone it so that he can make the calculation on it without affecting the original data going.

It sounds like a good idea because there's a lot of calculations that he had to run on that piece of data. So, every single piece of data going in will get replicated two or three times based on the equation that we had to run on it. And then, at the end, he would just return the result back and then just change the initial object, right? Just by copying it three or four times, it tripled or quadrupled the amount of memory we used on the web server [laughs].

And we have to roll back, like, the release night it come in the next [inaudible 28:57] because it merged, and we didn't have that kind of like we roll back plan that we have here where we just deploy the previous thing. It merged with the code when it build. So, now we have to undo what he did and then release a new version on the release night. And just all of a sudden, the server would just markdown. Everything slowed down to a halt when we started running automation tests.

Automation tests, what they do is they run hundreds of thousands of requests through the application [laughs]. The server just exploded in the middle of the night, on release night. Oh, that was a fun problem, got it figured out by the time everyone realized that it was the memory consumption. And I was like, what jinx? And then, by the time we looked at the codebase, it was a copy. It was a copy, and Windows and .NET didn't really do a very good job of memory management. When you make a copy of an instance, it doesn't reuse memory. It just copies a whole brand-new copy of it. So, by the time we got through it, it was three days’ worth of death march to roll back this code and retest it [laughs].

But then, when I go from .NET into Haskell, and I was like, how does that work with the immutability, right? Because when you change something, you practically have to make a copy of it. You can't just change the object it passed in. It doesn't allow you to change the object it passed in. You have to make a copy of it. And I have a nightmare of that problem that I had to deal with. I was like, so, how does that work in here? But it turns out Haskell has something called light thread.

So, basically, it reused memory behind the scenes without even compromising the ability of immutability of the data. So, with the data coming, you can't change it. But if you changed it on a copy, it still somehow referenced the old data. So, it didn't make a full copy of it [laughs].

MIKE: Nice.

MATT: There's that passion we were talking about.

[laughter]

MIKE: Yes.

RYAN: I just take a look [inaudible 30:56], product right? The way it sucked in this entire XML, and it just parsed out a piece of it lazily. And that’s kind of, like, the power of the Haskell right there. It just got me excited. But the thing about Haskell when it work like that, we hardly see any problem anymore, right? We keep rolling out [inaudible 31:14] product and the XML...look at the Grand [inaudible 31:18] XML. It was huge. But it didn't bring down the server at all. Memory consumption on it is so small on it. But, anyway, that was a fun problem to solve [laughs].

MIKE: As Matt was saying, there's the passion. You care about that. You love the tools that you're using. Love the aspects...but there's probably things you don't like about the tool, too. But there's things that you just absolutely love about it, and you just can't help but want to talk about. I heard, I don't know why it took me so long to hear, but I heard some time in the last year the joke, “So, how do you know somebody's going to run a marathon? They'll tell you.”

MATT: Oh, they'll tell you.

RYAN: Yep.

[laughter]

MATT: Burning Man, they'll tell you.

MIKE: [laughs] It's the same deal, right? If you care about it that much, you just can't help yourself. You're just going to be talking about it. And that's the kind of questions that I think are evergreen, right? They're going to keep working. They're going to work 10 years from now because what you're really exploring is somebody's humanity, and that's a powerful thing.

RYAN: It’s a bragging right. When you solve a problem, it feels good, and then the company would benefit from it. It feels good. The younger years of my life [laughs], I didn't care. I’d sleep in the office if I had to [laughs].

KYLE: I was analyzing Ryan's stories as he was going along and thinking how many questions I didn't have to ask him if I was interviewing him, right? Because now I know that Ryan can troubleshoot. I know that he has a really good in-depth understanding of C#. He's translated that over to another language and made that applicable to another language. So many of these different aspects of an interview that we are looking for were solved with one question saying, “Tell us about when you solved a really big problem and how you felt about it.”

MATT: Memory management. [crosstalk 33:26] You know, the interview I just did, I had a list of about nine questions that I was going to ask. I asked my first question. By asking my first question, he answered all nine of my questions that I was going to ask throughout the interview. Because he was very passionate about what he did, he demonstrated his knowledge of what he did. And I didn't have to ask him any technical questions because it was really clear he had a great understanding of technology and the technology he needed to use.

He talked about memory leaks. He talked about resource management. He talked about interfaces with APIs, like all of these things I was going to ask him about. My first question, which was one of the questions Mike likes to ask, and it's, you know, “Tell me about what you've been doing and what about in your career have you done that you're really excited about?” And he just went off. And about a half an hour later, I said, “I don't have any more questions.”

[laughter]

MIKE: I found another question that kind of goes the same direction but flips it. “Tell me about a time something you did failed.”

[laughter]

RYAN: It's kind of like [inaudible 34:56] break production [laughs]. You need to break production to earn that [inaudible 35:02] [laughter].

MIKE: And I asked that one. “Tell me about a time you broke production.” If they say, “I haven't,” you know you don't hire them because either they haven't been in the industry very long, or they're lying.

[laughter]

MATT: Yep. Anyone with any amount of experience has brought down production. And we all wear that badge [laughter]. It's when you do it often when it becomes a problem.

RYAN: Right. With the same scar over and over again.

[laughter]

MIKE: That question's great because, again, is somebody going to Google, get on ChatGPT, find, you know [laughs], examples of how bad somebody did? It's not something that's...you can't make a shallow replica, right? You kind of have to go into your human experience. And you can hear about the problem-solving skills. A lot of times if they're not talking about who they collaborated with, then that's suspicious, right? So, you broke it alone. You solved it alone. Why was nobody else involved? There's a lot of things that come up.

MATT: Yeah, and if you ask them, “Tell me about a time you brought down production,” and part of that response isn't how you fixed production, also a red flag.

MIKE: Yes [laughs].

RYAN: Troubleshooting is a skill that can be taught. You're just going to have to learn over time with the work that you've done. There are people who have that instinct. You can hear them go through the problem one by one. There are people who’s like, “I don't know where to start.” You've got to start somewhere. I mean, they can fix the problem if you tell them what it is, but they just don't know how to start.

MATT: Yep. And starting somewhere is better than not starting at all.

RYAN: Right. So, back to the AI topic with interview, the junior definition is going to change a lot, right? The junior dev definition. We no longer need to ask people what is the definition of, you know, and something anymore, right [laughs]? Use ChatGPT and [inaudible 37:23] [laughs].

MIKE: Yeah, it expands your brain.

RYAN: Right [laughs]? Crazy.

MIKE: And there's technologies that have been doing that for millennia. Writing expanded our brains because it allowed us to expand our memories. Now your memory can last, not just short-term, but for years, and even across generations. And across geographic differences, you can send the book somewhere. So, writing was able to expand our brains and has had a dramatic influence on human culture.

And think about the printing press that allowed that to be expanded dramatically and computing. The rise of computing has allowed people to expand their mental capacity, same process, I think. Now we get to expand our functional intelligence. Some of the things that before we had to rely on amassing this deep well of knowledge are less important. And we can use our problem-solving skills more quickly. That's a big deal, right? That’s a big deal.

MATT: Yeah, what we do isn't going away. It's just evolving.

MIKE: Right. It's interesting you say the definition of a junior engineer changes. That's true. They don't have to have as much of the, like you said, the specifics. They don't even have to know it that well because they can use some tools to help...to bring about, how do I do this? Well, it’ll show you right away. But the problem-solving skill, that's something that you have to practice.

RYAN: Yeah. I don't know if you guys have done this before, but like, I would print out an exception stack and have somebody read it and tell me what they think. I've done that before in the interview because, like, if you can't follow this mumbo-jumbo blob of text to find out where things break, you're not quite at the mid or the senior level yet. You're not even there. You're mid-level. Expect to at least know how to read an exception stack and figure out where things break [laughs].

MATT: I love looking at Grafana logs every day, Ryan.

[laughter]

RYAN: Well, that's the thing, though. When you see a problem, what do you do first, right? You hear that, I would go look at log, or I would go look at the exception return. Where do they start? And that's important.

MIKE: I've done something similar by just showing a file. You’re doing a remote interview, pull up a file in your code, right? And say, “What does this do?”

RYAN: Yeah. You can --

MIKE: And it goes a long way. Go ahead.

RYAN: Yeah, you can read between the lines, and most of the time you can read between the lines and just guess 90% of the function there, unless it's a trick question.

MATT: Well, if you understand code, right?

MIKE: Exactly.

MATT: And it weeds people out really quickly. And I've seen Mike do it. I mean, we've been in so many interviews together. I probably can’t even count. But some of the answers you see with that question are really surprising. And it's a great question because it shows if someone can follow code, understand where things are going, what the dependencies are, what's coming in and out, some key things that you just have to understand.

MIKE: And the questions they ask are sometimes more important than what they tell you.

MATT: Almost always.

KYLE: It makes me think back to...I had an interview several years back, now at this point, but somebody did that. They threw out code in front of me and they said, “Tell me what's wrong here.” And I looked at it, and I was just kind of like, “I have no idea.” And I looked at it a little bit closer, and finally I was just like, “I don't know that there's anything wrong with the code, but this comment doesn't make sense.” You know, they’d commented the code.

And he's like, “Oh, that's what I was looking for.” He'd thrown code in front of me, and he's just like, “Yeah, we wanted to make sure, basically, that you were paying attention to all aspects of the coding window.” And I was just like, okay [laughs], you know. But he was literally looking for that I would read the comment and then read the code and determine if they actually fit together. And stuff like that can even be helpful.

MIKE: Nice. We've covered a lot of ground here on what matters for interviews. We've talked about not just doing a coding challenge because it doesn't really get, many times, to the right kind of information that you're looking for. But if you do...and sometimes it makes sense to talk about bigger problems and then do it in pseudocode. Do it at a higher level where people are talking through their thinking. Ask why rather than what.

MATT: Some advice for those of you out there listening that may not have a lot of experience with this. Be honest. Be transparent. Ask questions. Those are the things that are going to get you hired.

MIKE: And that's the main thing we've ended up focusing on here, right, is asking people to talk about what engages them, and then, let them do the interview [laughs]. Let the person you're interviewing speak for themselves and reveal what they care about, what they're good at. And those kinds of probing questions work and will continue to work for a long time.

I think it's been a great session. Hopefully, you can take this with you in your own interviews.

Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, host Mike welcomes returning contributors Kyle, Will, and Dave for a lively and insightful discussion on communication and its failure modes—especially in engineering and development teams. Mike kicks things off with a humorous but illustrative story involving his children, hoverboards, cactus spines, and a major context mismatch with his wife. This leads into the first failure mode: people interpreting the same situation differently due to differing contexts. The group agrees that a lack of shared understanding often derails conversations and projects, especially when assumptions go unspoken or expectations aren’t clarified. Will and Kyle emphasize the importance of providing full context when asking for help or collaborating remotely, noting that even minor omissions can significantly delay progress.

The conversation then shifts to another failure mode: assuming communication is complete after initial planning. They highlight how this mindset leads to integration issues near the end of projects—when it becomes clear that vital tasks or dependencies were overlooked. Dave tells a memorable story about a sound engineer who foresaw this issue and left a self-contained module called “OYWNS” (“Oh Yeah We Need Sound”) that could be plugged in later, exemplifying proactive thinking. However, the team debates whether these are truly communication issues or just planning failures, ultimately agreeing that both planning and communication must be iterative and responsive, especially in complex, cross-functional environments. Mike brings in the Agile principle of “customer collaboration over contract negotiation” as a more effective framework to reduce these last-minute failures.

Toward the end, the group introduces a third major communication failure: speaking without tailoring the message to the audience. Whether it’s explaining unit testing to a non-technical relative using car analogies or trying to influence C-suite executives without drowning them in technical jargon, they agree that effective communication requires strategic translation, not just transmission. They also discuss how hierarchy and communication chains create distortion, using examples like long managerial handoffs or segregated teams that never speak directly. The episode closes with a call to action: be deliberate about context-sharing, break down silos, speak in your audience’s language, and ensure someone owns the responsibility of facilitating true collaboration.

Transcript

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting again today.

With us, we have long-standing contributor Kyle and Will Archer. Kyle Archer, Will Archer-no relationship to each other [laughs]. We have Dave Brady--

DAVE: Howdy, howdy.

MIKE: Who hasn't been here for a little while, but he's here today. He's been out on leave because of some medical challenges, so we're really happy to have him here with us today.

DAVE: I'm delighted to be vertical and to be here.

MIKE: [laughs]

DAVE: Yes, that was optional. That was an elective for me, was being vertical. So...

MIKE: So, great having you with us, Dave. We are looking forward to having a good conversation. And speaking of conversation [chuckles], that's what we're talking about today. We're going to talk about communication and communication failure modes.

I'm going to start, as usual, with a story, but I’ll introduce the topic first. To tell you a story...A few weeks ago, I got to do a little bit of setup here. So, you probably all know about hoverboards. They're not hoverboards, right? But it's like a Segway without a --

DAVE: Segway with no sticks?

MIKE: With no stick, exactly. The self-balancing scooter that you stand on and move around. They're ubiquitous, especially among young people and kids, and they're all over the place. We’d had one in my house for several years, and the kids played with it, and they enjoyed it.

My wife discovered something. You can put a seat on it, with a bar that comes out from it, and you put your feet on it. And it's got handlebars that you can lift up and down, two bars that you can lift up and down. So, you can adjust, you know, you can move it forward or back like you normally would by using your hands. And it makes it into, essentially, a little electric go-kart [laughs] because you've got a really low center of mass, you can just go full speed and have all kinds of fun.

So, my kids at home, at Christmas, that’s what they all got, and they love it [chuckles]. There's a ring around my house that’s...[inaudible 02:32] dead. [laughs] They have flattened it so much, not every day, but often. And [laughs] they have a great time. So, that’s the first line. So, I’ve got to tell a couple of things that are going on here.

In my office, I actually have a lot of cactus plants. Back during the depths of the pandemic, I had a little extra time and started doing some grafting experiments with cactus. It was fun. I've got quite a few cacti that I had some fun with that are now growing [chuckles] and taking up a lot of space in my office. But I like to take them outside in the summer.

So, a few weeks ago, late spring, yeah, the weather was going to be good, so I started bringing out these cacti. And I've got the gloves on, but sometimes you get the spines and especially the little fine...they're called glochids. I don't know if you're familiar with them, but they're fine, little spines, and they fall off really easily. And they get in your gloves, and they get in everywhere. And they're really hard to pull out, and they're itchy and awful [laughs]. So, they just get in everything. They are very hard to get rid of. It's like sand in the car after you've been to the beach, you know, it’s everywhere. Same kind of thing, it gets in your gloves.

So, I was bringing out these cactus plants, and I was getting more and more in my gloves, and my hands were getting itchier and itchier. And I got most of them upstairs, and my kids were out [inaudible 03:49] around the house on their hoverboards. And [chuckles] my four-year-old he comes up a lot in these stories because he’s a great source of stories. He ran his into a raspberry bush [laughs].

DAVE: Those are spiny. Oh.

MIKE: They are very spiny. I think he even did it deliberately, like, "Oh, I'm going to run into the bush, see what happens." Ooh, and then he realized that was not good. And then, he blamed the scooter because he thought...and he blamed the bush, “That shouldn’t be there [laughs].” So, he was angry at his scooter and refused to get on it because he thought it would hurt him again. And so, he started chasing my daughter, wanting her scooter. And --

DAVE: Because her scooter has never driven him into the bushes. His logic checks out, yeah.

MIKE: That’s exactly right. So, he's chasing her around the house, screaming. And I’m like, I should do something about this. But my hands are full of spines. And the thing is, I’ve been going up and down. My belt was kind of loose, and my pants had been falling down. So, I’ve got this series of problems: If I start running around the house, my pants are going to fall off. But if I try to take my gloves off and redo my belt or pull my pants up, I’m going to get spines all over my belly, which I don’t want. So, urgent situation. I can do nothing about it.

So, I take off my gloves [laughs], set them down, lift up my pants, adjust the belt, and I’m about ready to go out. And my wife comes out, like, “Um... what are you doing?” I’m standing there with my belly hanging out [laughs] with my belt while my son is screaming. He’s really yelling out there. Somebody’s going to, like, call the authorities.

There was a lot of context that was missed in this situation. And the way she was perceiving this situation was very different than the way I was perceiving the situation. We both knew there was a problem and me standing there I was trying to solve the problem. And [chuckles] it certainly did not look that way to her.

DAVE: Right. “What are you doing?” “I'm spanking our boy. What does it look like?”

[laughter]

MIKE: Not the topic—not spanking—but that kind of situation is incredibly common, where one person has context, the other person doesn’t. And you may be thinking you’re talking about the same thing, but you’re not, because both of you see the situation so differently. And neither of us was wrong. She looked “There is a real problem going on here. Why aren’t you doing something about it?” I was thinking, “I am doing something about this real serious problem, and you’re not seeing it.” It was very cordial, by the way—no yelling at each other. She just was wondering, “What are you doing? [chuckles]” We worked it out.

But this kind of idea where people see things differently it’s just everywhere. And I think it's one of these failures of communication, not just this kind, but there are a variety of these failures of communication failures that happen in engineering. And they are the problem, I would say [chuckles], with almost anything. It almost always comes down to some sort of communication failure. And I could tell stories, a variety of stories, and I probably will tell a few here. We're going to talk about, what are these communication failure modes, and what do we do about them?

My introductory story gives an example, you know, different context. You may be talking about the same thing, but your context means you're seeing things very differently, and unless you resolve that, you’re not going to see eye to eye. There is one example. Feel free to run with that. I would like to hear from the panel here: What are some communication failure modes you’ve experienced, and what’s the underlying reason behind it, and what can you do about it?

WILL: The biggest thing that jumps out at me when I think about sort of failure modes of communication is a lack of, I don't want to say respect, like, regard, care given to making sure that people have the same context for a problem that you do, right? Like, when you'll be working on something, it'll be all day, every day. You've been chewing on this bone for a couple of days now, right?

You narrow it down to some external team, some other partner, some other person. And you ask them a question, but you're sort of like...it isn't that your question doesn't make sense; it's the question assumes a certain level of context they don't have, right? And then, you run into all kinds of problems, which is sort of like a bunch of follow-up questions, like, sometimes people can get defensive, right, where it's like, “Hey, I've got a problem with your API.” “Well, if you've got a problem with my API, you've got a problem with me,” right?

And I think you can mitigate a lot of that defensiveness with, I mean, that's a separate problem, right, like, it's a separate issue. But if you can really lay out your case and your context and what's been going on, why you're doing it, why that path has led them to my door or your door or whoever, right, like, what brought you here and the reason that you think this person can help, right, like, there's just a lot there.

And really breaking that down is pretty laborious, but it improves outcomes dramatically, especially in, like, a distributed workforce. Like, you’ll find yourself in a situation where, like, you'll drop some Slack message on somebody. And to a very large degree, whether you get a reply in five minutes, or an hour, or maybe today, depends directly on how much context you provide, how much “What the hell did they even mean by that?” that you're asking them to do in this public help Slack channel.

MIKE: I've seen that to be so true. I'm thinking about a junior developer saying...it doesn't even have to be a junior, but it could be a senior developer. “I'm getting an error when I try to sign in,” and be like, well --

WILL: Story of my life, buddy [laughter].

KYLE: You just described every message that comes into the DevOps channel [laughter].

MIKE: What you've just done is you've asked other people to go and find all that context for you. And it's a real problem. It means that...I could go as far as to mention this happens in the DevOps channel. It's not very polite, but even if you're doing your best, I mean, if you're ignorant of it, you're not going to get a good answer. But if you come with somebody who says, “Well, I can't log in, and this is what I tried, and this is the error message that I got,” well, then that's catnip to an engineer who's like, oh, there's a problem. What am I going to do about that? And they'll go in and they'll solve it. It's just a totally different experience when you provide that context.

DAVE: If you can't give me context, at least give me steps to reproduce the problem, right?

KYLE: Yeah.

DAVE: Because then you need context.

KYLE: Links. Links are huge.

DAVE: Links, logs.

KYLE: Let me know what you're looking at.

DAVE: There's a reverse to that, though, right? Because it's not just that if you don't provide all the context, it's bad. It's more like, you have to do an impedance mismatch because we've all had the problem where you're fighting with the compiler, and what the heck is going on? You just moved into a new machine, and everything was fine, and now everything is just thrown up on this one service. And you sit back and you're like, why won't this stupid thing work? And somebody two desks over will go, “You got a new machine?” “Yeah.” “Disable pthreading.” And they go back to typing. And you do it and it works.

Because just the sound of your error, the tone of your voice, the time of day you were having this problem, right? You just got out of your sync meeting, and just the mention of “Why doesn't this work?” was enough for this person to grasp all the other context, assemble it, and hand you the correct answer.

MIKE: So, context is huge and one of the key failure modes, right, I've seen in communication. I think we'll hit on some other critical ones here, but I think that that is absolutely one of the important ones. That’s what I led out with, right? And the way to solve it, as we've been talking about, is give people those clues, right? Give people the breadcrumbs they need to be able to do something. If you don't build that shared understanding, then one of you is going to be standing there fixing their belt, and everyone's going to...”What are you doing out there [laughs]?” This is just going to happen.

I think one thing that I've taken to doing is in almost any group conversation where we're talking about a project, I will start by laying out the reason that we're working on that project and a summary of what we're doing and why. A lot of people in the meeting, I'm assuming, already know that, but almost I always get expressions of appreciation. Because even if you have been working on it, maybe it's been a week, and developing that foundation for the conversation is useful. Even if it feels a little silly, even if it feels a little redundant, even if everybody knows it, it gives you something to run with. Without that shared understanding, you don't really have language.

KYLE: That's an interesting one, because we actually had a leader that posted a topic at our work here a while back, and it really kind of stuck with me. He talked about how his father would start conversations with people, and he would provide that context, like, hey, I'm such and such from this interaction, from when we did yard work together, or just to kind of give context so that actually clues people in on who you are and where you're coming from, why you're talking to them.

You gave a very specific answer to engineering, which is great for this topic but I'm just saying, like, in general, for everything, just providing context, even in introductory conversations with people, it's very helpful and can save embarrassment, even.

MIKE: Yeah. I loved that also. Like, “Hi, I'm Mike. This is where we met.” Because they're thinking the same thing, like, you may not remember their name, and maybe you don't. And that way, you're giving them permission to do the same, right? You're saying, it's okay if you don't remember me. That's normal. Let's build some shared understanding.

DAVE: How do we make sure...so, I've got this idea bouncing around in my head. It might be a dangerous one or a dumb one. We won't know until we release it into the world.

I've been fascinated for decades on unit testing my own brain, like, coming up with things that are external to my meatware because a telescope can look at anything but itself, right? A microscope can examine anything but itself, right? Same idea. And the brain, it's hard to diagnose what's going on, especially because the order of operation is always back...like, we always think that we reason our way through something, and then we feel good about the decision. No, you integrated it; you synthesized the emotion, and then you back-justified it in your brain. We've got the receipts on the fMRI to show that that's actually what you did. But your brain tells yourself, no, I did it the other way around. You literally give yourself this reverse receipt that it's backwards because that would be logical.

Context: How to debug context is starting to feel, to me, like an integration test for my brain, where there are things I can do with my unit test brain. Like, I can pull up a video game that I've played tons and tons and tons that's challenging for me. And if it kicks my trash, I know I'm done. I know it's late at night, and I've lost my dopamine from the day, and I've lost my focus. I should put the computer down. If I open it, and I play it, and I just get a high score, I'm like, okay, I'm as locked in as I think I am. Let's go be productive.

I'm wondering about integration tests, where if we are following these rules, we should see this change. We should see this thing happen. You know, given a situation where I know this, and you don't...and this is the beauty of it: going into the problem, you never know what the other person doesn't know, right? So, it is a challenging unit test. So yeah, I'm just thinking through because it does feel like that, to me, like an integration test where you say, you log into the website, go here, and in the back end, you're bouncing back through seven different services, da, da, da, and at the end, you end up with a lease or with the end product of our system.

And it makes me wonder, like, if you are in a leadership role, and you want to affect a certain change in the organization, and that has got to be passed down by telephone through five layers, good luck, right? I'm not even sure how to approach that. I'm at the phase where I just thought of this thought, and now I have some interesting questions, but I have no interesting answers. It's just kind of a spicy thought.

MIKE: Well, it's an interesting idea. A way to get that shared understanding is that you test it. It’s like, do you know...and that's something I've seen work as well. Like, have you worked with this API before? It's a non-judgmental question. If you haven't worked with it, that doesn't judge you. It doesn't say you're a bad person. It just means you haven't gotten to it yet. So, asking those kinds of questions, “Well, have you worked with this API before? Are you familiar with this one?”

And if you ask a few of those questions and some of the answers are blank stares or “No,” “Well, let me talk about it a little bit, and then explain how it works.” If you're not running those tests, right, then you're almost certainly not going to have the kind of shared understanding that you want, because you haven't taken the time to check whether it's there. I like that. Testing in production, not so great. Not so great.

DAVE: There was a thing that Marty Seligman did. Marty Seligman wrote Authentic Happiness and Raising the Optimistic Child, which I think should be required reading for every parent ever because there's things that you can put into a child's wetware before the age of seven that will bake in and will stay with them, and that you can dramatically affect their health care, their mental health outcomes over their life with that.

WILL: Dang. My kid’s seven. Oh, well, I could save one of them maybe.

DAVE: What’s that?

WILL: I have a seven-year-old, so maybe I can save one of them.

DAVE: Right. It's on the bubble. Yeah, speed-read it. Speed-read it.

DAVE: But he did...I wish I could remember what it was called. I want to say it was, like, a case, like, it was a case study, but, like, CASE was an acronym. It was, like, Context-Aware Semantic Evaluation or something like that, where they were taking dead people and analyzing their speeches, like presidential stump speeches and that sort of thing. Because it’s science, you have to come up with a hypothesis, make a prediction, and then test your prediction. And so, what you do is you gather statistically improbable phrases.

And so, where am I going with this [laughter]? What's the context that you have on this? When I sit down to work on a ticket with a coworker and I ask them, “How does this tie into the rest of the system?” if the answer is, “I don't know,” and they're the senior and they're the person who knows that system, then that's significant. We should maybe put a tick mark on that, right? It's like, I'm supposed to get this from you. I would expect you to have this. Who would you expect to get this from? And that's kind of an interesting idea.

I certainly, like, hanging out with Ramses, like, Ramses knows where...I don't know if Ramses knows where all of the bodies are buried, but he certainly knows where all the memorial services are held. And he and I worked on something a couple of months ago, where it was going to send an email. The system was going to send an email. And that piece got written eight years ago and never got tested. I mean, we went on production; we verified manually that it works, and then we walked away. And so, I'm like, “So how do we test this?” And he's like, “Um...” And when Ramses says, “Um,” I get real nervous. My tail goes way bushy because that means we're in the dark times in the code. And you do still find those. You do still find those.

And so, anyway, the point being, coming up with, like, identifying specific phrases, think back on this...machine learning, you can examine your past life for data. Come up with the phrases that you think were maybe telling a certain thing, and then watch for them. I don't know if this is a genius idea or not. I'm just telling you the experiment that I'm going to run next, and I'll tell you guys later how it went, so...

MIKE: Well, the principle you're suggesting here is to test it out, right?

DAVE: Yeah, test it, yeah.

MIKE: Verify it. It makes perfect sense. You verify it, and then you fill in the gaps.

DAVE: Yeah. We measure this as a cadence run rate, right? Like, somewhere in the org there's somebody whose job it is to just, you know, obsequiously, fastidiously, you know, slick up Jira like a fat daughter for a beauty contest. Sorry, that's a weird southwestern phrase. But just somebody's job who it is to just fuss with Jira all day. And this is the person who knows how to make a burnedout chart and a burnup chart and an Eigen chart and a Gantt chart and all this stuff. And these are the people that, you know, in every organization, there's somebody that can tell you these teams have this much velocity, and if we ask them for this feature, it will take them this long to get it, right? And we aggregate this over time.

And I'm actually suggesting so that you think that's, like, the wave function, right? We aggregate and sum them all together. I'm actually talking about, like, taking one interaction and putting a point on it and saying, “Okay, start the stopwatch. We're going to measure this one, and we're going to watch and see how this input affected the output. Did this one run long, or did it run short?” You'll have to do it multiple times because, you know, there's going to be 73 external circumstances to everything.

MIKE: So, I'd like to maybe, if you all are good, because we've talked a lot about how to deal with this specific problem about context, to talk about another communication failure mode. I think that we've all experienced this. So, you've got, like, five teams working on something, and the project is down to the wire. You've got about a week left, and then somebody realizes that the API between two of these systems isn't working. And everybody knew it had to be built. It was in the requirements six months ago, right?

But you reach this moment like, oh, we didn't do that. Because you all got in your silo, and you didn't talk to each other about it because you thought, oh yeah, we’ve figured this all out. We've done the communication. We're done with that. Let's go work on our stuff. And it's somewhat related to the missing context, but it's a little different, right? This is assuming that the communication is done and not iterating on it.

Big projects, as you're coming near to the end, it seems like they always have something like this, right? Like, oh, wait, we missed that. We forgot to either...well, there's always the one thing, right, the thing that doesn't work and takes way longer than everything else. And there's that, but a lot of times there's also that, oh, wait, how did we miss that? And we didn't miss it a lot of times. A lot of times you knew about it, and just somebody didn't write it down in the stories that can get done. So, have you seen this? And what do we do about it?

DAVE: I have a fun, short story about this. Years and years ago, I worked at Acclaim Entertainment. I worked on video games for them, Jeremy McGrath Supercross. And when you work on a video game 25 years ago, all you care about is 3D pixels, texels, voxels. How much graphics can we stream? And you can literally have a AAA title get nine months down the road on a 12-month dev cycle, and there's no sound effects anywhere in the game. There's no music; there's no sound, nothing.

And the friend I had at Acclaim that I most closely worked with was the sound guy, and he was a sound programmer. That's what he did all the time. And he put a module in there called Oywns, O-Y-W-N-S.cpp, right? And we would compile it, these libraries, and he would just check it into the repo, and he's like, “When you need it, let me know.” And we walked off.

And he actually left the company. And time went on, and it's like, do we need? Oh yeah, we need sound. We should go do this. And we pulled it up, and we realized, O-Y-W-N-S. Oh yeah, we need sound. He called it from the get-go that you were going to...like, I'm going to tell you in advance the thing you're going to forget. There you go. And bless his heart, he wrote the sound engine to have no dependencies so that you could just pick it up and plug it in. You only had to fight with the upstream code. You didn't have to fight with the sound. That was a genius-level, like, master development. Like, to see it, to call it, to walk away and leave working code in the future for somebody, it's amazing to me.

MIKE: And he'd learned that because he'd seen it happen [laughs].

DAVE: The other one is, like, you get to the last week, and it's not that you forgot to write the API but that you never tested the API at scale. And if it's a third-party company...I won't name names, but this has happened recently where we connected up to the remote API, and not only did it not work at scale, but it didn't work at all. Like, they had written it, and they had signed off their acceptance test plan for their unit test.

They had never talked to us. We were the customer. They had never spoken to us. And we connected to it, and it just...you know, to this day, we still see messages come through from the QA team saying, “Hey, just so you know, this guy's API is down, so you're going to be seeing a problem, you know, coming back from this.” So, I'm not going to name names, but I'm seeing nodding around the table, like, we've seen this. We've seen this.

And I like this because...like is the wrong word. There's not an obvious scapegoat that you can point to and just shriek at and say, “This is all your fault,” right? It's all our fault. This was too complicated, and we left one of our PFAs, our Potentially Fatal Assumptions, we left it unexplored until way, way, way, way late. And then, that's when you have late-night coding sessions to code around the problem.

MIKE: So, I gave, like, the overall idea. Dave, you gave a couple of specific examples. I'm guessing, Kyle and Will, you've seen similar problems in your careers, and if so, what do we do about it? Whether or not you've seen it, how do you address that problem? Because it's everywhere. I have some thoughts here, but I'm going to be quiet for a minute and let you all --

WILL: Is that a communication failure, really, or is it a planning failure? No offense. That's one of the hard parts of, like, just planning, and there’s things that are...I don't see it as, like, we could express this, or we didn't express this, or we said it, and it didn't land. Everybody was like, “Oh yeah, it’s sound.” Everybody was like, “Sound,” but everybody's focused on, oh, voxels, character design, game player, whatever, right? And everybody's just like “Sound,” right? Yeah, yeah, it’s sound.

But then, I mean, it's an organizational strategic sort of, like, planning kind of an issue. And that's just, you know, in my experience, it's just the nature of very complex plans and integration. So, it doesn't matter. Something is always last on the list, and that is, like, the further down the list you go, like, the more strategic risk, let's say, that it's not going to get looked at. And, you know, the reason it always comes at the end of the project when everything is integrating is that's when everything on the list is getting checked off. To call it a communication failure, I think, is not how I experience it, at least.

MIKE: Well, let me --

DAVE: I think I agree with you. It's both, right? It was a planning failure, but it's exacerbated by the communication failure around it. It's kind of how I see that.

MIKE: Well, let me give a take on this because there's a lot of truth in what you're saying, right, is planning. So, should we all do waterfall-style planning, where we just really lean heavily into planning, make sure we get the planning right?

DAVE: If you've built this thing five times before, yes, you should. It's a proven method.

MIKE: Good point. There is an approach that gets away from some of the upfront contract negotiation and makes it an iterative communication process, where does this work? Does this work? Does this work? I have just pulled up the Agile manifesto, and one of the principles it says, “Customer collaboration over contract negotiation.” One of the key items in there is that if you lean heavily into the plan, well, that is a really expensive way to solve the problem. And some of those things are going to be left out because it's just so hard to get right.

And it is a planning exercise, but to a degree, planning is all about getting that shared context, right? It's how are we going to be working together? And if you change that to having a repeated, iterative conversation, then you're still going to miss stuff, but the costs are lower.

WILL: Sure. I mean, everything's a waterfall at a small enough scale.

MIKE: Sure.

WILL: I don't know. I mean, still, yeah, I still see it as planning. I still see it as a planning problem more than a communication problem. I don't know. I think six of one half [inaudible 30:29] the other, right? Because what's planning other than communicating, right? Like, planning is communication of an idea.

DAVE: I think I can split this correctly. You're seeing this as...I'm not seeing...in the communication that happened, I'm not seeing a defect in that communication, and I think that's valid. I'm saying that the communication should have happened here, and it failed to happen at all, and that's the slice that I'm kind of approaching this as. Like, it just didn't happen, and you can make an argument that it wasn't anybody's job to do it. It was all of our job, but nobody...I don't know if that makes sense.

I agree with you that these, like, much of this communication that I described absolutely fell down on planning. And the oywns, my friend, Sean, he wrote oywn, and he had brought up sound at the team meetings, like, every single week for, like, six months, and everyone would just, like, you know, whatever, the sound guy is talking while they're doing it, and, to me, that's a communication failure.

But it led to, yeah, failure to plan of, like, when are we going to...there was nothing on the Gantt chart of, hey, let's make sound effects work, and you can better believe that when we released the game, the sound was not a tight fit. It wasn't Crazy Taxi where you could rock out to the game because the sound was tight and had been up since the beginning. It was just like, yeah, it's a soundtrack. We bolted it on, and you could feel it. You could definitely feel it.

KYLE: But in that scenario, the communication was attempted, right? Is that a problem in communication, or is that a problem in that we didn't have a project manager or somebody else that wasn't prioritizing it correctly?

DAVE: Good question. In my opinion, the answer is yes. I have kind of a...I don't want to get too many southwestern colloquialisms in there, but I grew up with a phrase called tough titty [chuckles]. And what that meant was suck it up. Yeah, this sucks. I can literally go back to an interaction and say, “You were the victim in this interaction, but you were also the only agent that you had control over what you would do. So, this is not your fault, but it's your responsibility. If you go back to this team meeting again in a week, you are the one who gets to present this.”

And, at some point, what do you do? It's like, I presented it to the team. I said, “This is getting urgent.” I communicated that as well as I could, but nobody wanted to pick it up. And, at some point, you just have to drop it at other people's feet and say, “It was sitting on your porch. All you had to do was open the door and pick it up.”

WILL: Yeah, I'm assuming that communication failure mode, right, if I explain, like, I need a context or a task, something, clearly, concisely, to the point in a constructive fashion and you just don't care, we are definitely in a communication failure mode, but I can only go so far [inaudible 33:23] don't care [laughs].

DAVE: Is that communication, or is that culture, or is it psychological safety? This is a beast with many symptoms, right?

MIKE: Well, I think it was Kyle who touched on this a little bit. If you don't have somebody who owns it, who is going to listen, then you have everybody caring about their own silo of stuff, and they're not going to care because they have no reason to care because they are worrying about their own responsibility. And if you say, “Well, it's everybody's responsibility,” then everybody has a tiny piece of the responsibility. If you get a crime that happens in a large city and lots of people are watching, well, everybody thinks somebody else is going to do something about it, right, so nobody does anything.

And if you don't have that designated person who's like, “Oh, I'm going to go in and do something,” you can hope people will be proactive and fill your world with heroes. Well, that's great, but it goes against human nature a lot of times. You need somebody who says, “Oh, yeah, this is my job. I am going to listen to this person, and I'm going to figure out if that fits in the plan.” And I think if you don't have the designated person, and I've seen this over and over again, if you don't have somebody who knows it is their responsibility to care about what happens, then nobody cares.

KYLE: This reminds me. I had an old manager. He was always in meetings, and he was always very...what I called silver-tongued, and I always teased him. I said, “You're quite a salesman. You're quite a salesman.” Finally, he stopped me one day, and he says, “You realize you have to be this way, right?” I was like, “Why do I have to be a salesman and an engineer? What are you even talking about?” He kind of explained it to me. And I have wondered if this is what the sound guy, you know, maybe it was above, they weren't willing to listen, or maybe it was the sound guy not selling it correctly.

Because what you have to explain to get an engineer to understand is going to be massively different than, say, a C-level or somebody in sales. You have to sell your idea completely different. You have to sell the priority of your task. And I guess maybe that's what I'm recommending is take a salesman's approach in your communication just to get your idea sold, to get your priority sold in these kinds of situations.

MIKE: Okay, so you didn't say it directly, but you just identified another failure mode of communication that's even bigger than that. If you don't speak to your audience, you've lost them. Oh, this is simplistic. If my four-year-old asked me, “Why does the sun come up in the morning?” And there's multiple ways I can answer that, right? Many of which would be ineffective. And if a professor asked me [chuckles], “Why does the sun come up in the morning?” They're probably looking for a very different answer.

And if I'm not changing my answer...and sometimes people get uncomfortable about this. They're like, no, I should just tell the truth. Well, that's a lot squishier concept than that, right? Your goal is to communicate something to somebody, and language is not very precise. So, communicating that message is not going to be the same, depending on your audience, because you have to meet them where they are. So, that's not a matter of breaking the message to filter it in some way, right? It's a matter of actually attempting to share it better, to carefully tailor the message to the audience. And if you don't do that, the communication does not happen, or at least does not happen well.

KYLE: Yeah, I feel like I do it all the time, because, for example, everybody you're speaking to, when you're explaining a problem, right? My father is a mechanic, so when I'm talking to him about anything computer-related, I relate it back to a car, you know? How does this work with a car? How does this work with the systems he's aware of? And yeah, just breaking that silo, I guess, we brought the word silo up earlier, breaking down that silo so that you are able to communicate on a commonality. Without that, I mean, right then, he's finally interested, before then, he doesn't care at all. It's just computers to him, right? But turn it into a car, and he's engaged.

DAVE: My father-in-law was a plant engineer for years and years and years. He had a full machine shop in his garage. He would make engines, like, actual engines that would burn gas or steam. He liked making steam boilers that would run. He was working with that level of precision and tolerance. And he didn't understand computers, bless his heart. He was born in the 1930s. He was a Depression child, and he worked at one company his entire life. He was the last of the boomer career, one-job career people.

And I remember trying to explain to him testing and finally saying, “You know when you went to make this cut, and you stepped back, and you made a jig so that you could calibrate the cut exactly where you needed it?” He's like, “Yeah.” “The unit test is that jig.” “Oh, okay,” and then he was on board. He's like, “Okay, it’s going to tell you this is going to be exactly where you want it, when you want it, at the time you want it.” I’m like, “You got it exactly.”

MIKE: Absolutely. And, you know, you mentioned the C-suite, Kyle. I've seen this. I've even done it. I've helped somebody write a slide, right, you know, make a PowerPoint, and put in some language that I thought was right. And I go, “Yeah, this is exactly what happened. And, you know, here's the technical reasons.” And they're like, “No, that doesn't convey it at all. If you have any technical details in your description, you're doing it wrong.” “Oh, okay [chuckles], but that's my job is to provide the technical answer, right? I'm the technical person.” “No, no, your job is to translate it to somebody who's not looking for a technical answer.”

And thinking about that differently is a critical realization if you want to fix that communication. Otherwise, you know, eyes glaze over, and communication did not happen. Like you said, you don't engage somebody. But then if you engage with something they actually do care about, it makes all of the difference.

DAVE: There was a thing --

MIKE: Go ahead --

DAVE: Something that I got taught years and years and years ago was when you're writing, like, an email to somebody where you want them to do something or you want to answer their question and it's detailed, you write out the answer. Then you read through your answer and drill it down to, like, one sentence. And back then, we called it the executive summary, right? And so, it was literally like, “Hey, can we use Hibernate to, you know, backdate the JVM with the database this way?” Executive summary, “No.” [laughs]. And then you give them, you know, three pages explaining why.

And we still do this to this day. You'll see people write this post and then at the bottom they go, TL;DR, too long, didn't read. That's perfect, except the TL;DR should have been at the beginning. Write your message, write your TL;DR, and then move it to the top so that somebody can read the TL;DR and then decide, “Oh, yeah, that is too long. I don't want to read that.”

MIKE: Some of the best blog posts I see do exactly that. They'll say TL;DR, a semicolon, and then give you the two-sentence answer. And, honestly, if they do that, I'm more likely to read it sometimes.

DAVE: Absolutely.

MIKE: Even though I've already got the answer. Because here's somebody who respects their communication.

DAVE: Yeah. I’ve wanted to --

MIKE: Go ahead.

DAVE: I've wanted to do, like, a YouTube channel where literally every video starts off with, “Hey, YouTube. I'm Dave Brady, and today I'm going to teach you how to...” you know, and in the first three seconds, I've told you what the video is going to be about. And you won't see this in the wild because the YouTube algorithm is explicitly designed to punish that. It incentivizes watch time. And what’s the easiest way to drag out the watch time? Is not give you the answer you need for as long as possible. And it's evil. It's awful.

MIKE: Which is why every recipe on the internet is a nightmare. Have you noticed this?

DAVE: Mm-hmm.

MIKE: How many pages do you have to scroll down to? The nice people at least put a link, “Take me to the recipe.”

DAVE: Yeah. I tweeted this, like, a few years back that if you have a problem with your computer, like, “I can't get Windows to resize, you know, when I'm in this mode. And when I'm in full-screen mode, I can't get this window to resize.” And you search it, and if the top-level search begins with an explanation of why you might be wanting that answer, in other words, like, “Full-screen apps are a great way to use your computer to its maximum potential, but some people...” you're not reading the answer; you're reading a content farm. And somebody is pumping up their SEO to get out in front of that, and they're doing the same thing. They're just dragging out the watch time.

MIKE: Let's not do that. The Internet is becoming a sea of AI-generated slob and declining in usefulness. We can do better in our individual communication, and we can be concise and tailor our message to our audiences.

So, we've covered three failure modes of communication. We've talked about lacking context. We've talked about everybody owns it, and so nobody does problem. And so, you know, how do you get somebody who will make sure that things don't get dropped and is keeping that iteration going? “Well, did we miss this? Did we miss something? Did we miss something?” And we've talked about not talking to your audience. Are there any other key failure modes in communication you'd like to talk through today?

KYLE: I've got one that I've been thinking about bringing up and trying to figure out when to [inaudible 43:28] in. But that is the chain, the chain of the communication. And what I mean by that, the best examples that I can think of right now is when you've got people lined up and you draw on their back. And then, the person there, the next person will draw what they think was drawn on their back, and it goes down and down and down. By the time it gets to that last person, the context is completely different. And the longer that chain is, the more context that is either changed or is missing from the original context.

And so, point being, in, like, the engineering world, that's how many chains of leadership something will go through before it actually gets to you or how many team members it will go through before it gets to you, meaning, like, you go too far, and by the time it gets to you, it's not actually what was wanted. And that's where I'm just thinking of situations where I've completed tickets only to find out, “Oh, that's not at all what they wanted,” and I had to go back and talk to the person directly and could have saved a day's worth of effort going directly to the individual rather than having it go down the chain to me.

And I've seen this problem be a concern even just one level where team members aren't allowed or are discouraged from communicating with one another. And they have to go up through their manager over to the other person's manager and back down to the other person to get the communication done, and it just sits there and runs through. You don't have that direct communication, and things just start to break down.

DAVE: Microsoft did an amazing study in the late ‘90s, early noughties, where they ran all the project data longitudinally for, like, Windows 3.1 all the way up through Windows 2000, which was excellent at the time. And they were basically, “What causes bugs? What causes bugs to take forever to fix?” And they found that it wasn't team size or project complexity. It was the number of layers of management you have to go up and back down to get into the right leg of the trousers to get the solution that you need, and it's exponential. So, if you have to go up one level, it's twice as hard. But if you have to go up two levels, it's four times as hard. If you go up three, it's nine times as hard, and it gets insane.

And going back to contract negotiation versus collaborating with customers, when you're stuck in that hierarchy, what you have to do is say, “Draw it again,” and then somebody comes and draws on your back again. You're going to argue over how much resolution of data I need on my back in order to fit, right? You're negotiating a contract. Just turn around and talk to the person, right, which is against the rules of the game.

But the point of the game is to show you that the game is broken, right? It's like, this is literally an exercise in doing it badly. So, when that happens, attack the hierarchy. Don't go after better contracts. Don't try to make it so, “Well, if we could just get it so that the DBA team and the engineering team and the ops team, if they could just pass everything back together as tickets, this would all be solved.” Nope. Nope.

MIKE: [laughs] Oh man. I've seen that one as well. “Oh, yeah, just make a ticket system then we won't have to talk to each other, right? It's all just organized.” And, of course, the opposite happens. It's a nightmare. But you said the solution is embedded in the description of the problem. And I've seen this, and I've seen it a lot from good, well-run organizations is the first thing you do is when you identify the teams that are dealing with this, you get people on the ground, and you connect them together, and then you step away.

If somebody is going to claim ownership of that communication and say, “No, everything's got to go through me,” then they are destroying their organization because you are preventing communication. And people who will instead say, “My job is to facilitate the right conversations. So, my job is to make conversation happen. That doesn't mean it has to come through me. In fact, I'm probably the wrong way for that to happen. I'm going to find the people who need to talk and get them talking.” And that kind of leader will make great changes in their organization, make things move forward. Again, I've seen this happen.

So, I've seen that kind of leader fairly often, and I deeply appreciate when I do because everybody's lives are so much better. And the best negotiations I've seen are exactly that. You might have some leaders who get in like, “Oh, yeah, we need to do this. Let's get the engineers and put them together and have them figure it out.” And when you don't see that happen, you're like, “Okay, this is going to take a few months.”

DAVE: We've talked about this on the podcast in the past that one of the most amazing things you can do if you're on a siloed team is to violate the borders of the silo and just go talk to somebody. We would do team swaps, and now you know somebody over there that you can just poke.

I'm just thinking about this, that we have call processors that talk to the customers, and they have to use our website on a side of the website that I hadn't seen anybody use because I was always facing another set of customers, and all of a sudden, I'm facing these people as they are my customer.

So, it's all remote. I jumped in on a remote meeting with one of the call processors to just shadow them on some phone calls. And if I had been in office, I would have picked up my laptop and gone and sat next to them, like, physically next to them because I want to actually create rapport. I want them to know what I look like and what my name is and that I'm a nice guy so they know they can come talk to me.

And just sitting with them, I immediately realized, “Oh my gosh, we are making you type with thumbtacks on your keyboard. It does not have to hurt this much.” I added, like, a dot sort. There was a thing, four or five merchants in it, right, and it wasn't sorted. And that's fine if you need to read through five things, but these poor processors were seeing 500 stores unsorted, and they had to find them by name. And they were just...and when you're a call processor, you're making minimum wage or barely above, and when you're in that job, you don't complain, right? You just put your head down, and you just do what it is.

I'm watching this, and I'm like, “Oh, my gosh, you're working in a Dickens novel. Who did this to you?” And I'm like, “I did this to you. I'll be right back,” right? And so, like, the next day, I, like, poked him, and I said, “Hey, just go to Jira,” and he went and he looked, and there was a ticket to fix the thing, to just sort the merchants. And that's a one-line fix, right? It's literally dot sort. It's a seven-character fix.

And I have friends on the processing team now because I took the one thing that hurt them the most in their eyeballs, and my point is it was collaboration. It was crossing through the DMZ and saying, “Show me what you're working on, and show me how you use this product because, in theory, we built it for you. Let's collaborate.”

KYLE: See, and without that direct communication, how slow would that have been? Because that's processing up to some project manager. Then it would be prioritized down to you, and you wouldn't care. You'd be like, “Sure, yeah, this ticket, throw it back over,” you know? And you would have never heard any update. But now you get that personal interaction, and you've got friends over there that are just like, “Thank you, Dave. Like, this is great.”

DAVE: The truest proof of that is the fact that this bug had been in their system, and with this particular...you guys know we have one merchant that's, like, an aggregate. They've got thousands of merchants right under them. It was that merchant, that super merchant, right? And we never checked it on the engineering side, and they had been dealing with this for years.

KYLE: Wow.

DAVE: And no ticket had been created. Because if you're processing, you don't get to complain about the software. You don't get to have input on what would make it easier. And just having an engineer sit down and go, “Oh my gosh, we need to fix this for you,” it was world-changing, right? Literally, there was somebody going, “Let me make this not hurt so much.”

And we talked about this a few minutes ago. A gift card is so much better than just more salary, and a nice bonus is so much better than, you know...I mean, to be fair, I like getting cash as a bonus. I love that. It's fantastic. But those aren't the things that I remember, right?

You'll find anybody on WordPerfect the second year that they were in business, and when the company took all of them to Hawaii for two weeks as the Christmas bonus, right? They all remember that because they were wildly profitable and spending like stupid because it was the middle of the ‘90s. You get the idea.

MIKE: Well, I think that is a great place to tie things up today. We could probably go and find a variety of --

DAVE: Oh, I've got eight more topics we could go into on this. There's different ways that communication can falter down [inaudible 52:11]. This is an evergreen topic. We could easily circle back to this [crosstalk 52:16].

MIKE: Absolutely. But we hit some big ones. And importantly, these all have relatively straightforward solutions. You just have to do them. You just have to use the solution, right? Test the context. Make sure that you have it. Cut through the hierarchy and go meet with somebody directly. Speak to your audience, right? Think about who you're talking to. Or designate somebody. Designate the person when you have a large group that needs to communicate who is going to facilitate that communication.

Let's work on that and make our communication better because, in the end, it's most of what we do is trying to figure out how to do things together as humans. And with that, until next time on the Acima Development Podcast.

The episode opens with Mike sharing two formative stories that highlight the importance of speaking up against inappropriate behavior. The first story recounts a moment in the late 1990s when a manager made a blatantly racist comment during a hiring discussion, leaving Mike stunned and regretful for not speaking out. The second story contrasts that moment with a more positive example—Jonathan Schwartz, then a Sun Microsystems executive, swiftly and calmly shutting down a sexist remark during a public tech event. These stories set the stage for a broader discussion on moral courage, preparation, and the power of speaking up in professional settings.

The panel—Mike, Will, Justin, Kyle, and Tim—then share personal anecdotes ranging from experiences with crass coworkers to moral gray areas in technical and corporate decision-making. Justin recounts a story of an inappropriate joke at a company meeting and the swift exit of the offending employee, highlighting the importance of understanding professional boundaries. Will reflects on his past confrontations, including a fiery exchange with Alan Kay and a more recent situation where he lost a technical argument at work but chose to “disagree and commit” to preserve team cohesion. Tim and others emphasize how reading the room, respecting legal and ethical boundaries, and choosing the right hills to die on are essential strategies for navigating workplace culture.

The conversation closes with strategies to foster healthy debate and encourage junior team members to voice their thoughts. The group stresses the importance of frameworks like DACI for decision-making, teaching engineers to argue both sides of an issue, and maintaining psychological safety in teams. Ultimately, the takeaway is that speaking up—whether against clear misconduct or in nuanced technical debates—requires preparation, emotional intelligence, and humility. And sometimes, the bravest choice is knowing when to push, when to let go, and when to walk away.

Transcript:

 MIKE: Hello and welcome to another episode of the Acima Development Podcast. I’m Mike, and I'm hosting today. Today, I've got with me Tim, joining us from security. We've got Will Archer, as is usually joining us. Thank you, Will. We've got Kyle, and we've got Justin. We've got a great panel here, and we've got a good topic.

I've been bouncing this one around for a while. I'm excited to talk about it. And I'm going to start with a couple of stories about...well, I was present for both of them. One of them is more about me, and one is more about somebody else. The context for the first one...and let's start by making this a little awkward. So, I'm a white guy [laughs].

WILL: Oh no.

JUSTIN: No, no.

[laughter]

MIKE: So, my dad's family's mostly British. My mom is an immigrant from Norway. So, white guy, right? This...value-neutral on that one, but it affects what comes next. So, I lived in Louisiana back in the late ‘90s, and I was working at a construction company, upholstery [inaudible 01:24], an interior construction company. So, we’d work in office buildings, so we’d tear out the previous setup. When one tenant moved out, another tenant moved in. It's not all we did, but it's a lot of what we did, and it worked.

I was working in the warehouse as a warehouse manager at the time. So, I was downstairs. People would walk in and ask for a job sometimes, and I couldn't make the choice. I'd refer them to the manager who worked upstairs in the offices. So, I was sitting at my desk one day and somebody walks in, asked for a job. So, I called upstairs [chuckles] to my manager upstairs who could make this decision. And his response was, “Are they white or black?”

And I was so flummoxed [laughs]. I couldn't believe that he'd said that. I was just so floored. I just stared at the phone [laughs]. I just stared at the phone for 20 seconds. And then, he goes on. He says, “Okay, just say W or B”. He thought the reason that I wasn't saying anything is because I was afraid of incriminating myself. So, I just stared at the phone, like, I didn't say anything else. I just stared at the phone, and, honestly, to my regret [chuckles] because I should have said something, but I had no idea what to say. I was so, so completely gobsmacked. I couldn't react. He came down.

And the race of the person who was applying doesn't really apply here, right? What matters is, I wish I would have said something. And this was going on 30 years ago, and I still think about that because I wish I had had something better to say. I wish that I had prepared ahead of time, you know, had gone through this scenario in my mind. I just had not gone through this scenario because I didn't expect somebody to say something like this, and so no good reaction. It certainly changed my perception [chuckles] of the manager upstairs, but I didn't know what to say. So, there is my first story.

Second story. I was at a Java Users Group in the early 2000s. And Jonathan, I made sure I looked this up, and Jonathan Schwartz, who later became CEO of Sun Microsystems, was coming to speak at this Java Users Group. So, there was a lot of people there, hundreds of people there probably from the Java community. And Sun Microsystems was pushing their own distribution of Linux at the time. It didn't go anywhere, by the way [laughs], and neither did Sun for that...well, they were eaten by Oracle a few years later [laughs]. But --

WILL: Solaris was all right. Solaris wasn't so bad.

MIKE: They were a meaningful company at the time. I was actually doing a little research on this before the call thinking about it. I guess they had a lot of their business in finance, and the crisis in 2008 did not serve them well [laughs].

WILL: They were pretty well cooked by then, though.

MIKE: I think they were. They were stretching a bit. Which is not really the topic of the story. I was at a Java Users Group...but, you know, it gives you a little context. I don't think he was CEO. I think he was, like, a VP at the time. But he came there, and he was representing the company certainly. And he was going to give out free copies of their new distro of Linux. So, I was like, oh, cool, the Sun distribution of Linux. And he used it...I got one, and it was, like, they hadn't even removed all the branding. I think it was SUSE Linux. They changed a little bit. But they were going to repackage and sell it.

And the person introducing Mr. Schwartz, talking about getting things started, was talking along and made just this badly sexist remark, shockingly sexist joke in front of hundreds of people, right, and thought that this would somehow be okay. And, interestingly, Jonathan Schwartz immediately shut it down. “No.” He didn't escalate it. He just, “No,” and then moved on.

And I've always been impressed with that. He clearly had prepared for that moment, and he knew if somebody said something that was out of line, he would put them in place. And this wasn't somebody who worked for him. He just had practiced this muscle that he stopped what the presenter was saying, you know, that's not okay, and then he went on. And here I am again going on 30 years later, and I remember that. I remember that because I was so impressed with how well he handled that situation. He said something, and he handled it with grace. He didn't make a big deal about it. He just made sure that's not acceptable behavior. We're going to move on.

Both of these are for very specific kind of speaking up. There's an instant, clearly something out of line. Right in the moment, say something. There are lots of other kinds of moments where maybe it's not quite so clear, right? Maybe there's a decision that is complicated, that's not quite clear. And a lot of times, there's technical decisions. Do I get involved in this? And they're hard. But I'm going to reference what I didn't do, which was prepare for that kind of situation ahead of time, and what Jonathan Schwartz clearly did do, which was think about it ahead of time, prepare, and practice so that when he was in a moment where the decision came, he was able to say the right thing at the right time.

Oh, I'm going to start by asking my panel here. What are some times that you wish you had spoken up and you didn't, and maybe sometimes, you know, maybe the opposite? You spoke up when you shouldn't have.

JUSTIN: So, I've got two instances, one that I was present at and one I was not. One I was present at. A company I worked for this guy was very, like, I worked directly with this guy, and he was very, very crass. Like, if there was a dirty joke, he knew it, and he would say it. And most of the time, like, I had known this guy for, like, you know, I was working with this guy for a couple of months. And most of the time, it was like, oh, okay, he's a decent engineer, but, man, is he crass. He's not going to last long at the company. Somebody's going to make a complaint.

Well, the next month rolls around, and they have a company meeting. There was the CTO who was a woman, and it was a meeting of the engineering department. So, there was probably, like, 30, 40 people in this room. And this guy says a joke really loudly, and it was very, very crass, something about humping somebody else, I don't know. It was really, really bad. And the room deafens, and everybody looks around. And the CTO she just starts the meeting right there. She didn't say anything to him right then, but she probably should have.

But, in any case, he was gone within two weeks. So, the next time that she flew out, it was to talk to him specifically and escort him out. So, that was a case of reading the room and understanding who you're talking to. I think that's a skillset that people need to have. It's like, understand the audience of who you're talking to.

And then, the next story I have I wasn't at, but I definitely heard about it. I work for a large bank now. They recently had a large initiative to return to office, and the CEO was having a couple of town halls to talk about it, and somebody very senior...well, not very senior, somebody who was probably two levels above me, but they were still an IC. It's basically architect-level position, super architect-level position. He asked directly to the CEO something to the effect of, you know, you do realize this is going to really ruin morale. And I don't see how this can't ruin morale.

And so, what happened next the town hall went on because the CEO didn't have a good answer for that. And what happened next was that guy's boss's boss somehow heard about that question and fired him in the elevator later that day. Then the story doesn't end there.

WILL: Ask me almost anything.

JUSTIN: Ask me almost anything [laughter]. And the guy later on got his job back.

WILL: Whoa.

JUSTIN: Because it was specifically, you know, it was an ask-anything type of situation. And the boss's boss had overstepped his authority and, well, I guess he fired him for the wrong reason, so...And that sort of --

WILL: Yeah, you better get that constructive dismissal. There are other steps. You got to make him [inaudible 11:17] and do the hokey-pokey [chuckles].

JUSTIN: So, I guess my two stories were examples of be aware of what you're speaking, and sometimes it's worth it to speak truth to power, and sometimes it's not worth it to be very crass.

WILL: Yeah. Damn, I'm taken aback at just the stories of people just being just woefully out of line. I have not had a lot of experiences with people talking like that, I mean, I don't know. I believe you guys. I know these guys are out here. I just haven't...I haven't had that experience.

For me, most of the stuff that I've been exposed to that's been really difficult has been within the bounds of professional communication, right? Like engineering teams, they're all hard problems, and very often, the solution to those problems is not immediately apparent. And you're choosing without prior foreknowledge between two equally good strategies, and that's where you see a lot of people butting heads. Although I will say I talk a lot of shit, and I've gotten better at it. I've gotten a lot better at it. But I have a ridiculous story if you guys want to hear it.

MIKE: It's story time.

WILL: When I was younger, like, when I was young, right, I was, like, 25 and I was at the University of Utah. I was at the Graduate Student Advisory Committee. So, we were a little selected group of grad students that got to sit in faculty meetings, which was a profound waste of time but very interesting. And we got to talk to, like, faculty candidates would come in and be like, “Okay, well what are the grad students like?” And we had no power. It was like a nice little pat on the head committee, but I appreciated it.

And we got to meet people. And one of the people that we got to meet when he buzzed by was very famous in certain circles, University of Utah alumnus Alan Kay, who is just a wretched, dried-up, useless, blowhard. Like, you know Alan Kay's name because, Alan Kay, for the entirety of my lifetime, has completely abandoned any form of useless work to tell the story of Alan Kay ad nauseam.

We had a lunch meeting, me and a bunch of grad students. Like, you had the privilege of interacting with diet Will, who most of his fangs have been pulled...I am so much...I'm sweetness and light. 25-year-old punk rock Will was not as retiring and diplomatic as I am now, and it did not go good. He spent...I don't even remember what we were arguing about. We were arguing about, like, operating systems, like, in the weeds of the programming languages. And I was a pretty serious, like, compiler operating systems nerd at that time in my life. I didn't have a lot of time for Smalltalk, Smalltalk the language, rather than small talk, like, chitchat. Like, we were deep in it, and Alan Kay he was ridiculous, really bad.

And it wasn't only me that he made a really poor impression on, but I'm not going to out anybody else because they had the good sense to keep their mouth shut. And he spent the rest of his lecture in front of the entire university...I got, out of a 45-minute lecture, I got about 15 minutes directed at how bad I suck. He didn't mention me by name, but everybody who knew...and there was a blast radius among my faculty advisor. And other people were just, like, Will, what did you do?

And the right thing to do is to just let the old man flap his gums at you because it doesn't matter. Oh, he sucked so bad. Sorry, I can't read Alan Kay articles anymore having interacted with the guy and seen the absolute wasteman of his professional contributions for the last 50 years. Anyway, moving on.

MIKE: So, you and Alan Kay don't get along [laughs].

WILL: There is no we [laughter]. I doubt that he has thought about me twice in the intervening 20 years. He’s probably dead now.

MIKE: That's the worst moral to take from this story [laughs]. The real moral...total sideline, Calvin and Hobbes, the greatest comic ever written, there's an arc about Snow Goons, where he builds monsters, and they go out and wreak havoc. And [chuckles] the final line of the panel...[inaudible 17:07] if it's a panel. The final strip of the arc is Calvin learning his lesson. And Hobbes asks him, “So, what did you learn from this?” He says, “Snow goons are bad news [laughs].” And Hobbes observes that he's actually learning nothing from this [laughs], and that's how he likes it.

WILL: Well, I actually think it segues into the general strategery for these sorts of disagreements, which is, at that time, I hadn't developed this muscle yet, but, like, now I've got it, and I still talk a lot of shit. But I get along pretty well with engineering organizations, even when we don't agree and I don't get my way...is when you can tell that things aren't going your way.

This conversation that I had with this rich, old fossil, who's completely high on his own supply, and he was in an orbit somewhere completely detached from the actual state of the art in 2006 or whatever, there was no reaching that guy. There was no communicating with that guy. There's no conversation we were going to have. So, just vibe out, eat my expensive, free lunch, let him ramble, and move on.

More recently, I have learned my lesson. And there was a big debate that happened a few years ago. And I was on the losing end of it, where they were getting rid of...we had this Java server, like, I work for a big retailer, like, a big retailer. We shop there. And they were getting rid of their...they had this big initiative to get rid of this Spring Boot application server, which was sort of, like, our gateway behind the firewall downstream services. They wanted to get rid of all of this old, crusty Spring Boot infrastructure and replace it with team-specific Node JS microservices, I guess you'd call it, like, gateway microservices. And every team, every functional group would have their own.

And it was a dumb idea. It was dumb. Like, why do this? Like, why...it would make it slow. Because it was based on the false premise because that was in JavaScript and many of our other systems were in JavaScript, it would make the engineers fungible across these radically different domains, which has never happened. It's two years on. It never worked. I was like, why? This is going to be slow. It's not going to buy us anything. It's going to slow down application development. You guys don't have the appetite to pay down the tech debt of moving this stuff over. So, you're going to build a new service, but you're not going to get rid of the old one. And they haven't.

MIKE: Oh [laughs].

WILL: You know? All this stuff happened, and I was right about everything. And I told them, and they said, “We don't care.” And I said, “Cool.” [laughter] Disagree and commit. Because, in the end, if I had continued paddling the boat the wrong way, we would just spin in circles. And it is an absolute guarantee that we will fail. If we can't disagree and debate and get at everything back and forth, and then we're going to say, we're going to go, and, like, we're going to go. And, as such, we did not get an optimal solution, and there have been consequences for that solution, but we are moving forward slowly.

And I still have a good reputation in the organization. I got my ‘I told you so’ points. I got to dig in a time or two privately. But I'm fine, and they're fine, and the project is fine. And was I right? Yes, I was. Were they wrong? Yes, they were. Does it matter in the end? I don't know. Maybe, maybe not. But that's how you have to deal with it. If you are on the losing team, take your L, and shut up [laughs]. And that's how it goes.

I have a friend, a good friend of mine. He got into a similar slapfight on some GraphQL implementation architecture, and he couldn't take his L. And he was a contractor, really senior guy, made a lot of money, but he was still a contractor. And contractors can be fired very cheaply, and he's gone. And when he was going to get hired back...he subsequently got hired back. He's an excellent engineer, and he has been doing great work, both before and since. But he got into this fight with a staff engineer and a director, and they got his ass. He was done. He was out of work for three months.

And when he was trying to get back, they were on the phone with a manager who was trying to bring him back on. He's like no, man, get rid of this guy. He's a jerk. Get him gone. We don't want him. You don't want this dude. And it's a fine line to walk. If you want to have a career and still talk your shit [laughs], you got to figure out how to balance it.

MIKE: Interesting. You talk about reading the room, and you say that's the key. And the room may be you and one other person. Being attentive to the person or people you're talking with is a huge amount of the communication, and that's not easy for everybody. I mean, there's people who are neurodivergent who have a really hard time with that. And so, that's going to be something you're going to have to work on and work on and work on because it matters. That's where a lot of the communication is. I think there was a ton of insight from your stories, Will.

WILL: [chuckles] Well, I've been talking shit for a long time.

[laughter]

TIM: I do think these kind of moral boundaries are a little bit endemic with the tech industry, and these companies are in a public position where it gets...like, okay, so to go back to your original story there, Mike, how you've got a kind of construction related business.

MIKE: Yes.

TIM: And I hate to say it, but it is kind of like a stereotype sometimes, you know, the street construction workers, there's a certain level of crass that just comes with that industry. In some places, it's just more publicly accepted, and that's unfortunate. And that's not at all desirable or a place where you want to work. But within the tech industry, you just go back 10 years, and there's patterns of it getting...okay, so, Uber, the bro culture in Uber with...I can't remember...there was a couple of executives that ended up getting canned because of sexual harassment and discrimination and just kind of dehumanizing pattern of events.

The same thing happened in Activision, if I recall. They had that bro culture in their executive leadership that eventually got outed, and people were let go. There was Google. I think Google got caught about 10 years ago just kind of...it was an internal memo that got leaked. And one of the executive leaders were disparaging the gender wage basis, kind of things like that. And they made some really insensitive comments in ink that got leaked in a memo that...it's just something that happens in the tech industry. And when it gets out, there's a zero-tolerance policy about how it is accepted in the industry. I think that's one good thing [laughs].

Like, if someone in our executive structure were to do something stupid like that and it got leaked out, they'd probably be on their ass in a couple of weeks. And it's just when to speak out. Sometimes in the tech industry, the industry will speak out for you [laughs]. You will not be accepted otherwise. There's a pattern within the tech industry of this kind of stuff of getting, how would you call it, a Darwinian approach to how you're going to be received [laughs].

WILL: I would say it's corporate size, like, entrepreneurs are a different breed of cat, they are. And they're not going to act right, but they started the company. So, you could kiss their grits, and it's good and bad. Madness and genius go hand in hand. And people with an entrepreneurial gene, or personality disorder, or whatever, they're out there, they just are. But you have to leave a substantial portion of who you are at work.

Like, you guys have never heard me tell you a dirty joke. And let me tell you right now, I love dirty jokes [laughter]. Like, I love them. I love them. You've never heard one out of my mouth. Even now, like, I'm not on the clock. I'm doing this one for free, but sorry, nope. Not going to do it. Sorry. Like, I might have gotten this podcast a PG-13 rating with my trash talk. But I'm actually not going to swear over the rest of the podcast because that was enough. That was enough. That PG-13, woo. Mm-mm. Not a word.

TIM: I don't know. I think you get the Silicon Valley or slopes around here. You get that attitude with a lot of these kind of startups. But I've met some really impressive people in the leadership realm who are...I think my favorite one that I've ever met was Scott Farquhar from Atlassian. At the time, he was the co-CEO of Atlassian. We had an opportunity, me and a couple of my peers, to have lunch with him. And I've never met a more straight-to-the-point, like, honest, no-BS kind of person.

He wasn't propped up. He wasn't inflated, wasn't anything like that. I was like, “I really like your stuff, like, what you guys are doing and how well you guys have developed. But you guys just don't support HIPAA certifications,” at the time. And I said, “What do you guys want to do about that?” And without even skipping a beat, his answer was, “Nope. We're not going to do that, at least not right now. I can tell you right now that's not on my roadmap, and I have no interest in doing it.” Complete respect.

I've never had a CEO executive reply in a straightforward manner. They're always diplomatic. They're always politicians with how they give you their answer, the majority of them, what I've got. But that guy, he and, I think, Mike Cannon, those guys, straight shooters, just some of the best guys I've ever seen. Really a good pleasure to be with those guys.

WILL: But that's a unicorn of a unicorn because I think that guy was the founder, right? Like, he was the dude, right? And so, entrepreneur. He got his business to, like, the ramen profitability, paying the bills. This is a thing. And then, he took it to a multi-billion-dollar level. And so, I mean, that's a unicorn of a unicorn. But even that guy there's a difference between the CEO that comes up through the ranks, and I was a director here, and now I'm the VP here. Now I'm a CTO here. Now I'm a CEO here. And you just sort of jump up, jump up, jump up. Like, this guy, it was his shop. And it is such a refreshing difference.

And it's why Founders-CEOs are still coveted to the degree they are. Because a professional manager who was appointed by the board and beholden to the board is just going to be like, “Well, we have a lot of quarterly initiatives, and we can get that on our roadmap. All our partners are valued,” and just some more word vomit.

So, your eyes glaze over and you wander off, whereas the founder is just like, this is my shop. Yeah, I can't do it strategically. It doesn't make sense for our company to take on this amount of infrastructure to support HIPAA, which I know a little bit about, and it's a tremendous amount. It's a tremendous lift, and he's just like, I can't afford it. And he'd just be like, no. But yeah, I don't know, man. There are a few entrepreneurs, and there are ducks [laughter].

MIKE: Let me observe something that's come from a number of these stories. You talk about complicity. Tim, you talked bro culture taking over sometimes. When that happens, there are people feeling uncomfortable. Obviously, there are people being mistreated, which is why it eventually surfaces and becomes a scandal is because there are bad things going on. But one person starts seeing it happen, and they don't say anything, and then they decide to join in. They follow the pattern, and then it grows. You end up with a bunch of people...you come in, and humans we absorb the norms around us, and we tend to follow them.

And some of us have an inherent punk rock attitude [laughs] and maybe naturally resist those norms. And some people are more prone to say, “Hey, I'm going to follow what's going on here.” In any case, there's a human tendency, and we can see this outside of...we can take this to extreme examples. Obviously, people cite fascism in Europe, you know, a hundred years ago, where a lot of people just kind of went along with it, even if they were uncomfortable.

There's this tendency where when a group's doing it, you don't say something. And it's interesting because even if you're reading the room [chuckles], maybe the room says, “We're all okay with this, and you're not.” And that's an uncomfortable situation to be in. So, what are the boundaries? It's not just about protecting your job, right? Because maybe it's worth leaving your job. When is the time when you say...well, so I'll take this further. If you were working for Enron and you didn't say anything, that probably is a black mark on your resume. There's times when there's clearly a line that's been crossed, and you should have said something. It was worth losing your job over. So, where do you find those boundaries?

WILL: Yeah, that's tough. I mean, I think, for most commerce, like, a good bright line is legality, right? I will not go to jail for any job. There's no paycheck. Like, I won't do it. There's not a chance. I have worked myself...personally, I've worked for defense contractors, many weapons of war, you know, in the hands of the United States military. We're killing people, and I have some sliver of accountability for that. So, is it morally gray? I have to say yes. I have to say that's a moral gray area because it's just the act of war, the act of violence, of taking human lives. But is it wrong? Ooh, no, I can't say that. Can I say it's right? Ooh, not purely. And everything exists somewhere in that continuum.

So, I'd say legality, like, that's a bright line. If you find yourself being asked to do something that is not legal for a job, for a paycheck, you should stop. And so, then it becomes a situation where if you disagree with something, I think you should argue, like, you should raise up your hand and say, “Hey, I don't think this is cool. I don't think this is okay. We shouldn't say anything.” And then, to the degree that you're empowered to act, I think you should act to do the right thing.

And then, when it's out of your hands, when it goes to your boss's boss, and then you're like, nope, this is this, then it becomes a situation where you have to ask yourself, do I stay, or do I go? And that's, I don't know, man. I have left jobs, and it was very, very difficult because...well, I'll tell this story.

So, I ran my own software company for 10 years. We made, a bunch of mobile app fitness products. I started it with a good friend of mine. We built it up. It was a seven-figure company. It is probably a seven-figure company. I had a significant equity stake in it. And we had some business relationships and some business arrangements such that I thought that the company was basically limited.

We had an artificial ceiling, where it was never going to be anything more than a job. And I had to ask myself, is this a job that I want? And the answer, to me, was, no, we should sell the thing, get a nest egg, get, like, a modest middle-class retirement out of it. That's what we would wind up with. We would wind up with a robust 401k. It was on the order of, like, [inaudible 34:55], think about it like that.

And me and my co-founder we had diverging of opinions. My co-founder saw it as this is the job I want. I'm the CEO. I can do anything I want. I'm the king of my little kingdom, right, and I love it. And I'm like, ah, but I'm not the king, not that it wasn't cool. I did a lot of cool things that you can't do in a big company. But we had this fundamental diverging of opinions. I had to leap. I had to go, and it did not go good. But that's sort of one of those forks in the road where it's like, okay, this is dumb.

There were other things that I mentioned where it's like, yo, we're making these technical strategic decisions that I think are really bad. I don't think they're going to work out, and they wound up not working out. But still, I'm just like, okay, well you're writing the checks, and that gives you the right to say, Yes, No, or Maybe. Like, I'm delegating that to you for the paycheck. So, you get to make the call, and I let you make the call. And then, this other thing where it's a strategic line of the road where it's like, I get this one life, and I'm not living it the way I want to. If we can't come to an agreement, I got to go, and that's that.

And so, how do you differentiate between, like, I don't think this is a good idea, but okay, which happens I think to most of us a lot. You can think about something you did today, where I'm just like, ah, I wouldn't have gone that way. How do you make that determination?

TIM: I see it happen all the time, but it never escalates to the Uber level of the bro culture. But I think every software development shop we always know there's a group of brogrammers that they act a certain way, and most of the time it’s just harmless. They're their own kind of way. But what happens is it just turns into a click, just like every other workplace. Whereas this time it happens to be software engineers, and they feel that they have the momentum and the numbers to act a certain way without consequence, and that turns into the bro culture.

And those always self-destruct over time because, eventually, one of those guys is going to do something stupid that HR is going to come down or the law and order of public...your PR is going to kill you when you do something that bad. But I think when you're in the engineering workplace, you just have to be self-aware enough to be able identify who the brogrammers are that may have a negative click approach or impact on their peers, fellow employees. And sometimes you just kind of have to steer clear of those people. Otherwise, you just add more fuel to the click, and then you're just, okay, great. We've got that going on, and yeah.

WILL: I don't know. I think the click-ish culture is, I think, I mean, I’ve run into some brogrammers here and there [laughter]. They’re really few though, to be perfectly honest with you. I’ve run into the bro boys. I’ve run into the bro boys in the sales team. I’ve run into the bro boys [inaudible 38:22]in marketing.

TIM: The sales engineers.

WILL: Less so MBA dudes, those guys can bro down. But devs, like, bro devs, pretty rare. It's just, like, the psychological skillset that you need to be an effective developer it doesn't sit well with the bro dogs. There's a lot of sitting still that you need to master if you want to do that job and brogrammers do not, as a general rule, excel at sitting quietly for many hours in a row.

MIKE: Well, I think that there may be some simple rules here. You said legality, easy. Don't break the law. There's another one, and this goes back to something we've talked about a lot: psychological safety, treating people with respect. If somebody's doing something that is dehumanizing another person, I think that can be a bright line. Like, no, I'm not okay with that. I'm going to say something, and that's my cue to leave.

I did leave a company a number of years ago because their marketing tactics were maybe legal, but certainly not ethical, taking advantage of people to the point where I just couldn't stomach it, so I left. When you join in that, by proxy, once you're dehumanizing, you've lost some of your humanity [chuckles]. And you've said, well, this kind of person is better than this kind of person. That kind of thinking doesn't have a good end. And I think that you can recognize that, look for that, and see that as a line.

I think there's a lot of things though that we haven't talked much about. It's just a technical decision, and we're going to have a disagreement. And, in most of those cases, you don't have to win [laughs]. You’ve been saying, Will, I will raise my opinion. I will share it. I will emphatically argue for it, and then I will agree with the group and [laughs] go with the decision that's made. A lot of times it's more important to have a decision than the right one, than the optimum one. Better to be going...you talked about rowing in the same direction.

If you're generally going in the direction that will get you where you need to go, you'll eventually get there, and it's a whole lot better than going in circles. So, I think there's a lot to be said for put in your paddle and just rowing. After you have clearly and effectively shared your guidance as to which way you think we should go, you get down and you just follow the leader.

WILL: Yeah. I mean, that's it. The only way it works is if you trust me to do my job and I trust you to do yours, and that's it. Like, as long as you understand where I'm coming from and why I wanted to do it, and, hopefully, I understand where you're coming from and why you're choosing this other thing, then that's what we got to do. And in terms of like, yeah, you can't work at an abusive or dehumanizing place. You can't do it for long. If you are in one of those places, you've got to get out.

I was not in this meeting, but a meeting was reported to me very recently at the big corporation that I’m consulting for that one of the VPs went in an AMA, and he said, “There's not a single individual contributor in this org that is operating at a staff level.” So, this is, like, hundreds of engineers, including many staff engineers on a level. And he's like, there's not a single one of you that's at that level right now. And I'm just like, woo, that's spicy, man. That's bold. Sure that's not you? [laughter] It was tough. It was a tough meeting. I was just like, wow.

MIKE: It's interesting you've said maybe it's you [chuckles]. If you're leading an organization and everybody is underperforming, it's usually something systemic. And you can blame people, or you can do something about it [laughs], and that's on you, I think.

WILL: If you gave me a hundred chances to either develop, recruit, or retain, like, a staff-level engineer, I could get one. I could give one of them, man, come on [chuckles]. That's just me. It is the little, old me [chuckles].

MIKE: That goes to the creating a culture of respect, that it’s just absolutely vital to have healthy dialogue.

TIM: There was one thing you brought up, Mike, the process of every time...so I had the experience. We had a new mobile app up and running. It was doing well. It was a primary mechanism during COVID for patients to get their COVID-19 testing everything. It blew up bigger than we thought. After we kind of hit a peak run time, the principal engineer brought up the idea that they wanted to move all the microservices into one monorepo. And the logic behind it, which was baffling to me...because I was on the DevOps and the CloudOps side of it at the time. The logic behind it was because Google that's how they do it. And this individual pointed out multiple big people out there who do everything they do in a monorepo.

But from a DevOps perspective, we had built custom pipelines in this whole ship-it mechanism that was fine-tuned, and we were easily bootstrapping a new microservice, getting it up and running, delivering a feature, all within a sprint, which is mind-blowing. The system is working. And then, they're like, oh, let's flip it to a monorepo because we don't want to have to keep doing patches on each repo on...And I was like, okay, just use Dependabot. Do something to help automate this thing. And it got to the point where engineers were getting very colored and very open about their feelings one way or the other.

And I don't think there's any argument in the world that will swing one way or the other that a monorepo or a microservice model is the right way to go when they're so religious about, you know, they have these stakes in the ground. If you've done this for a while, you just need to learn to step back and know your role as either a contributor to the conversation...this is my opinion, and this is why I think we should do X, Y, and Z. But, at the end of the day, it's the decision maker who's going to make the call, and you've given them the best information that you can. The decision maker makes the call, and you run with it.

And some people got so bent out of shape that one guy decided to quit because it didn't go his way, and it wasn't because...I mean technical reasons one way or the other, but it was more of an emotional decision for him to get out of there. And it's like, I don't think that was the right hill to die on because every piece of software expires in 10 years. We always replace something with a new something after a certain amount of time. So, what's the point? Just do the job and then move forward and let the decision maker who's hired to do that do their job.

WILL: Well, I'd say...I agree with you and, yeah, totally agree with you. But I would say this, if there is a series of technical decisions, strategic decisions at the management level where they're making these broad strategic decisions that just make your job suck, it just sucks to go into work over and over and over because it's just, like, this sucks, then you should find a new job. You should find someplace where you will thrive, and you will feel comfortable. And sometimes it's just a reset, where you just need different tech debt [laughter].

TIM: Yeah, that’s true.

WILL: [inaudible 47:27] Because we've all got our stink. But at the same time, you shouldn't do it in a huff. You should do it because, like, you made this decision, and I said, “Don't do it,” but you did it anyway. And now I'm stuck in this living nightmare every day. And that's a great reason to quit a job.

JUSTIN: So, I know we're towards the end of the podcast. But I did want to kind of get you guys’ feel for, you know, all of us are kind of more senior, right?

WILL: Allegedly.

JUSTIN: Allegedly. What are some good strategies to encourage discussions like that? Because, oftentimes, we could get to the point where we think we know what the idea is and everything and where we want to go. And we're trying to steer the direction of the discussion and go for the decision-making process that we think is correct, but oftentimes, we need other people's input. We need to encourage the next generation of engineers to speak up. So, what can we do? What are the strategies there?

WILL: I was in the debate team when I was in high school, and when we would do our little practices, everybody was expected to be able to take either side of the argument. And I think in an engineering discussion, it's almost never black and white. There are going to be pretty clearly good and pretty clearly bad decisions, but it's almost always a trade-off between two differing sets of ideals.

And I think in every good-faith engineering discussion, you should be able to articulate both sides. This is why I think your idea makes sense. This is why I think my idea makes sense. These are the fundamental values that we are fusing between for this and this. If we go this way, we get this. If we go that way, we get that. What is it that we value? And if we value set A or set B, then that dictates our strategic direction. But I think everybody should be able to do that in a good-faith discussion.

TIM: I had an awesome SVP once, and the way he did it when it came to these big decisions that had implications about framework changes, or big wide impact, he said, “Okay, I'm going to set the discussion up with the DACI framework.” So, Driver, Approver, Contributor, and Informed. “Whoever wants to do this thing, that's the driver. I'm the SVP, so I'm going to approve one direction or the other. And then, I want subject matter experts to weigh in as a contributor.”

And his advice to everyone who was a contributor was, if you tell me all the pros about your idea and you can't tell me one con, then you're way too religious about your perspective, and I don't want your information. You need to be able to tell me the pros and the cons of your position and remain objective. Otherwise, you're not furthering the conversation and giving me valuable information. And just setting that framework right up front really led to some productive decisions that maybe they were wrong, that's fine. But we were able to get through that process so fast and realize we were wrong quicker than debating it for four months and then finally getting there.

JUSTIN: I like that. I’ve kind of heard about that sort of thing before, but you put it together really well. Like, you have certain roles, and you expect people to play those roles, and encouraging people to play those roles makes sure that you get that variety of opinion, and it encourages people to fulfill that role. If you have a junior engineer, it would be like, “Hey, guess what? You are going to be doing this role here. I want you to prep for this meeting, and this is what we're going to be talking about.”

TIM: Yeah, 100%.

MIKE: I really like that, too. Come prepared to take the stance against your position.

JUSTIN: [laughs]

MIKE: And I'll often do that for myself in a meeting because I don't feel good about a decision that's not had any negative feedback. It may still be the right decision, but a lot of times you know it's the right decision because you hear the arguments against it, and they're all bad [laughter]. You have to hear both sides and sometimes that actually...in fact, oftentimes, that makes it really clear what the right decision is because you start arguing against it and being like, yeah, you're not making a very good argument, you know, it helps. And sometimes you dig and you find something that genuinely is a good argument and you realize, hey, well, maybe I wasn't coming in with the right idea.

WILL: I just default to constructive laziness. Like, what's the least amount of effort that I could do to get my job done over the long term, over broad, like, whatever my planning horizon is, which is not sprint level but quarter, if not year. And whatever's going to win me the quarter with the least amount of heartbreak and stomach acid, I'm doing that. Past a year planning becomes real hopes and dreams and moonbeams [laughter]. But you can plan a year. Like, a big organization can plan a year. [inaudible 53:23] plan two years. Don't [inaudible 53:25] your fairy tales [inaudible 53:26] [laughter]. Tell that to the investors. Save it for the board meeting, dude.

MIKE: I think we've covered some good material here. We've talked some about how to deal with just straight-up inappropriate things that shouldn't be said. It's okay. Speak up. Shut it down [laughs], and it's worth it. We've talked about a lot of these areas where it's not so clear and that we should say something politely [laughs] but emphatically.

There's a recent idea that maybe take a counterpoint so that you're clearly looking at both sides of the issue. And just the importance of not dying on every hill, of getting with the program, that a large organization or an organization of any size is only going to go somewhere if you're all going in the same direction, and push it forward even if it's not quite the way you want to go. And as long as it's not making you hate your life and your job, it's probably okay. You can live with it.

WILL: Yeah. If you hate your job, you should go. Nobody should spend all day, every day at a job they hate. You should go. If you hate your job because you didn't get your way, that is a deeper concern. I think you should probably introspect.

TIM: And you're still going to get paid. So...cool [laughter].

WILL: Yeah. Yeah. I mean, many times I have returned to my fundamental axiom, which is that I get paid either way.

MIKE: This is some good stuff because this is hard. Going back to the beginning of the call, I was in a situation. I didn't know what to do. I was young [laughs]. I was young and foolish. And I wish that I had today's experience then but you don't.

Hopefully, you've learned some things. When you're in your situation where you don't want to be regretting it 30 years later, thinking about what you could have said. Thank you everybody for sharing. I think we're in a good spot.

Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, Mike, Will, and Kyle explore the intersection of language learning and programming. Mike kicks off the discussion with a personal anecdote about studying German in school, noting how little he’s used it since, and how learning Spanish as an adult through Duolingo was more rewarding. This transitions into a broader discussion about how the process of learning any language—human or programming—reshapes cognitive patterns and makes it easier to acquire future languages. They draw parallels to AI, explaining how training models on multiple languages makes them better at understanding new ones, suggesting that the mental “training effect” of learning is more important than the specific content being learned.

The conversation then shifts toward practical advice for aspiring developers. Will emphasizes the concept of “the lore”—the unwritten, experiential knowledge in software development that can’t be easily taught in schools or bootcamps but must be passed down through mentorship and hands-on practice. He shares his own journey from fixing computers to compiling Linux source code at 15, which gave him a massive edge later. They all agree that building projects, no matter how scrappy or simple, and putting them out into the world is the most effective path for beginners. This “build and share” approach helps attract feedback, builds credibility, and accelerates learning in a way that no tutorial can match. The group also acknowledges the tough job market, especially for juniors, and suggests finding mentorship even in “sketchy startups” as a legitimate way to gain real experience.

Finally, the hosts reflect on how the landscape of software development is evolving in the era of generative AI, emphasizing that while tools are getting smarter, the core need remains unchanged: people who can think clearly, break down problems, and express logical solutions. They argue that learning how things work under the hood—whether it’s operating systems, threading models, or build pipelines—is still essential for becoming a great developer. Will humorously champions “vibe coding” and encourages makers to lean into their curiosity and creativity, while Mike reminds listeners that even though the languages may change, the deep thinking and hands-on experience involved in building and debugging remain the bedrock of the profession.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting today, as I have usually been doing. With me today, I have Kyle, and I have Will Archer. Actually, I have Kyle Archer and Will Archer [laughs].

WILL: No relation.

KYLE: No relation here.

MIKE: No relation, but they sound alike [laughs]. And, sound like, speaking of sounding like, I'm going to start today by talking about something I did long ago in high school. I took a German class. I took four years of German. I started in junior high, and then got to four years of German. And you know how many opportunities I've had to use German since I got out of high school? I think the answer is zero [laughs]. I think the answer is zero.

Actually, no. I ran into...maybe four years ago, I was visiting my parents, and I was going on a walk with my parents. And my dad ran into somebody who he knew who was from Germany, and he said, “Hey, Mike, so you speak German.” And I froze. It had been, like, 30 years [laughs]. I’m like, “Probably not right now [laughs].” I didn't have much software. If I'd taken a little bit of time, I could have done something, but no.

If I see, like, on a packaging, like, shampoo bottle or something, you know, and they've got instructions in German, you see that, and you don't even think about it. You just start trying to read it. You’re like, oh, I recognize that, and you start reading and try to make sense of it. That's about my entire experience using German is reading the back packaging that happens to have a translation in German, you know, since I got out of high school.

However, a few years ago, I said, you know what? I want to learn a language that I might actually use [chuckles]. I started using Duolingo, the language learning app. I even have a paid account now, because after a while, I started using it more. I've been doing that pretty consistently. I think, daily for the last four years, I've been learning Spanish. And I actually completed the course a while back, but they give you just review, and I've been doing that. And I got everybody in my family doing it, too. So, we're doing it together. And I went through it the fastest.

And I've read this: it turns out that after you've learned one language, it's easier to learn another. And, in fact, they actually recommend that if you want to learn a language very different from your native language, for example, for an English speaker, if you want to learn an East Asian language, if want to learn Chinese or Japanese, you learn it faster by learning a European language first.

WILL: I believe that.

MIKE: Because it trains your brain to think about things in a different way, to think about language differently. And once you've done that, you're primed, and it's easier to switch to --

WILL: So, you’re saying if I wanted to learn Chinese, right, I want to learn Chinese, it has very few loanwords or commonalities, then I would go faster if I learned Spanish, and then went to learn Chinese, than I would if I had only learned Chinese.

MIKE: That’s right. Which sounds crazy, right?

WILL: It does sound pretty crazy.

MIKE: It sounds crazy. Like, how could you possibly learn Chinese faster by learning another language first? But I've read this. I don't have the source. I don't have it right in front of me, but I have read this before that it actually is faster to learn a different language that’s more similar to your native language. So, you know, again, if you were a native Chinese speaker, it might be faster to learn maybe Korean first and then go learn a European language because it changes the way your brain thinks. I think that's fascinating.

Maybe the source that I read is totally wrong and leading me astray, but there's some intuitive sense to it that it trains the way you think. And regardless of whether it's universally true, there's probably some meaningful truth there that there's some training about how to think that is different from learning the specific syntax.

To take this further, AI models have been doing...they've run some of them in lots of different languages. So, they just train a text on lots of languages, and specifically, they train on text that's been translated from one language to another. And the fascinating thing that they find is the models that have been trained on one language or on many languages actually are better at learning a new language than they were at learning the first language.

So, if you have a language that can translate between English and German, and English and Spanish, and Spanish and Turkish, and just name your language pairs, and then you you add a new language to the mix, like Igbo or something, then if you've already had those other languages, your model will be faster and even do better than it was if it was just trained on that language. And there are a couple of reasons for that.

First of all, you have more data from commonly represented language pairs. There are probably really common ones, like English-Chinese, for example, you know, where there's a lot of stuff translated out there. And so, you have a higher body of language between really commonly used languages out on the internet, but it's more than that. There seems to also be just some universality of learning to translate that makes the models better. When you've learned between a lot of languages, you actually improve them all together.

WILL: That makes sense. That makes a lot of sense, to me, actually, I think. Because you're dealing with, how do I put it, like, you're dealing with...I don't want to use the word means but ideas and a more robust mapping of linguistic phrase and the poor, I would say, human idea. We're all people, and to a degree, I think language shapes the way we think, but only to a limited degree. And so, the more mappings, the more cross references you have between a turn of phrase, or a word, or something and the fundamental idea of the human condition that you're trying to express, then I think the more, you know, evocative and robust your translation between the two things would be.

MIKE: And it's interesting you say that. You can say there's this human experience that it gets closer to you. And they actually talk about, also in AI research, the embodiment problem, which is that some concepts have been really hard for AI to learn, partly because they don't have the human experience. I believe the...I can’t remember the philosopher who did this...was it Plato? Plato and the cave? I'd have to go look it up.

WILL: That sounds about right. That sounds right.

MIKE: So, imagine you're sitting in a cave, and people are broadcasting pictures of something up in the wall in front of you. And so, your whole experience comes from that broadcast, you know, looking at this shadow show. It’d be difficult to learn some aspects of the world. You could learn a lot, but it'd still always be second-hand, and that's how these AI models are learning. They don't have the body, typically. I mean, there are some robots now with, you know, physical bodies that can go out and live or experience the world. I'll use the word live very loosely [laughter]...to go out and experience. And so, there's some ability to learn there.

And, in fact, simulation is hard for robotics. You know, it's a lot better if you use a real robot, but then you break your robot. It’s expensive [laughter]. That embodiment problem is tricky. It's hard to learn if you don't have those experiences. You're suggesting there's this human experience that is independent from the modeling we use in language, and that if you model it several different ways, you actually get closer to the human experience, which is a fascinating idea, which we won't dig too deep into, but it's still a fascinating idea you said there [laughs]. I like that.

I mean, we haven't even hit the core topic that we're talking about today because this initial idea is so interesting that leads to it. I wanted to ask, so present it, if you were going back, if you could go talk to your old self that was first learning to code, what would you tell them? How would you have them do different things differently? And I think this idea of language and this idea of experience and language being different is a fascinating analog because it's another language, kind of language, that you're using to accomplish a similar purpose, but in a totally different context.

WILL: So, it's been a long time since I've learned how to do this, and I'm always, I don't know, I'm aware of, like, you know, how much unconscious knowledge that I've just sort of absorbed and accumulated and I deploy without thinking in just the daily process because I'm learning all the time, right? The machine never stops. The treadmill never stops [chuckles]. I'm always learning new, different more horrible ways to break machines.

But I have brought on board a fair number of junior developers. And the one thing that I look back on the most is what I refer to, colloquially, as the lore of just these are things that you have to know because they're not your job. But you're never going to be able to effectively work and to make an artifact that functions without the lore, which is just, like, anecdotally, I remember...

So, my first job I was working in a computer shop. I was a computer repairman. And I got exposed to a lot of the lore, you know, back when it was really like a dark art. It was really like a guild. When I started programming computers, there was no internet. I mean, it existed, obviously, but, like, there was no internet. There was literally no Google, nothing.

And so, I got in this computer shop, and I got, like, some graybeards who taught me some stuff. And then, I got exposed to Linux, like, as it was rolling out. Like, as it was coming out, I was getting on Linux, and I had to, like, I ordered...I purchased a slug of six or seven CD-ROMs with the source code to Linux that I had to compile myself [laughter] as a 15-year-old kid to have Linux. And it was such a hateful, abusive process.

But then I went and I got into college, and I got onto the programming team. We had a little competitive programming team. And I was wildly more effective than any of my peers because I had this lore around Linux and making the shell do things, reading files from the shell like basic commands. How does a directory work? How do you mount a device? Like, all this crazy stuff that is both absolutely foundational to the process of creating a running, functioning artifact, at least for me, and is not the kind of thing that anybody would bother to write down.

And so, that, to me, is the biggest and craziest thing, and, like, how do you get? How do you get the lore? And that is the thing that almost...I don't know of another way other than being in the company of professionals. It has to be passed, almost literally, from hand to hand. You need to be, not necessarily in the room, but in a distributed workforce, we're finding you need to be either in the room or in a very, very close virtual analog. I mean, it has to be live, one-on-one transfer of information between you and me. I think a one-on-one Zoom call would be sufficient, but I think a webinar or some analog that wouldn't do it.

And so, then it sort of leads into this problem of, like, I think, in 2025, we're being extremely hard on junior developers in this year, at least. At this moment in our industry, we are being very, very, very hard on junior developers because productivity is going up. And people, you know, productivity wants to go up. Salaries want to go up, which means the demands to get a salary job are also going up.

So, if I'm starting from zero, all that in mind, this is sort of, like, the background of like, okay, this is the problem that I need to solve. I know nothing. I need to get up enough so that I can demonstrate enough value so that I can sort of start the flywheel turning where I work, and I get paid to work, and then I get to work more, and I get paid more, and then that virtuous cycle where I can sort of put energy in, and I get enough money to, like, go full-time, or part-time, or something like that.

So, what I would say to do is...I suppose that’s the goal. It does assume the goal is to go pro. The goal is to go pro, to have a job, have a career, you know, like, really do it. If you just want to go out and have fun, I think there's never been a better time to go out and have fun doing programming. I think there are a lot of places, a lot of frameworks and stuff, or a nominal fee people will be happy to just get your stuff out there in the world. I know C was a Rails shop, and, like, what are the guys who, like, you just sort of put your artifacts up, and they just manage all the service stuff for you.

MIKE: Oh, Heroku?

WILL: Yeah, Heroku. There are places like that, I mean, analogous ones, I'm sure, for, like, you know, JavaScript, or Node, or React, or whatever you want to do. And so, those kind of places are really good. You've got your generative AI sort of coding assistance that could go out and do it. So, honestly, I would just make things and get them out into the world in the dumbest way possible. And this would be really hard for me, especially as, like, a broke kid. Like, I would spend some money to, like...I'm the guy who compiled Linux from source at, like, 15 years old, so obviously, I'm a little bit of a psycho, but that's the biggest thing that I could see.

And I’ll say that I think I'm monologuing a little bit, so I should take a step back. But, like, one of the patterns that I see in terms of people who are trying to bootstrap a software engineering career successfully is people who are trying stuff, putting it out on the internet, writing a blog, making a thing, like, I'm trying to make this thing. And then, you just put it up, put it up, put it up, put it up, put it up. Here I'm building this thing; here you go. Here I'm building this thing; here you go. Here I'm building this thing; here you go. And you put it out into the world. You're proactively doing the work. You're asking good questions.

Because most developers, most engineers, are very, very, very, very, very forthcoming with helping anybody out there who’s actually doing the work. If you come to somebody like, “This doesn't work. I don’t know.” Then, you know, you may get a cool response. But if you come out like, “Hey, I was trying to do this thing, and this is what I tried, and then this thing happened. And so, then I tried this, and this, and this, but then that happened. And I'm worried about, and I'm thinking, like, this is maybe a thing that I could try. But I have this question and, like, how do I implement it?” People will do backflips for you. [crosstalk 16:54]

MIKE: These are professional problem solvers. They can't help [laughs] but want to just solve the problem.

WILL: Exactly. And so, the cycle that I'll see in terms of people doing it right is they'll go out. They'll start working. They'll put their work out into the world, at whatever level of quality that they're doing. And they'll start, and then they'll get some feedback. And they'll put some more out, and they'll get some more feedback. And they'll pull some more out. They'll get some more feedback. They'll put some more out. They'll get some more feedback. And then, the blog is done because they have a job, and they're too busy to write the blog anymore. And I've seen that happen over and over and over. And so, that's kind of what I would...

Generative AI, I think, is a little bit of a tangent, because, I mean, really, it's just, like, start making things and put it out into the world publicly, and share it with everybody. There has never been, nor do I think there will ever be, a surplus of people creating things and pushing them out in the world. People who are willing and able to do that stuff are rare gems. We've never had enough of them in the world. We never will. And if you demonstrate yourself as that kind of person, there's a spot for you. You just have to, like, raise your hand and say, “Here, I'm out here. I'm doing it. Here I am. Look at all the stuff I'm doing.” And you’ll get scooped up.

MIKE: And I've seen that repeatedly. I’ll give a couple of examples, you know, go and build stuff. So, a woman...I think we may all know her, but I don't want to, you know, like, speak for somebody without, you know, tell their story without talking to them. Home school --

WILL: Is it bad [laughs]?

MIKE: No, it's not bad at all.

WILL: I’m kidding. I’m kidding.

MIKE: It's actually really good. So, she was a homeschooler. She had been teaching her kids for years and found that she needed to organize her stuff. And she’d wanted to code for a while, so she learned to code to build an app to organize her stuff. She was not out working professionally. She just built something because she knew she needed it. And she learned so much about how to get something done by solving this need that she joined an apprenticeship, quickly got to speed, and became an effective, professional developer.

Another woman I knew was working at the front desk at a company, started working with the engineers and writing some stuff. Like, 10 years later, she's running an engineering shop as a professional, you know, she became an engineer, ended up running her own shop, like, very much so. That process of building something, and, you know, from whatever background you're coming from, learning that lore, building, building, building, absolutely, I’ve seen to be the case. And the people who are really effective tend to be people who've done it for a while.

And you're talking about doing it as a kid. I started doing some coding in...well, I had done a little bit, like, as a really little kid, but then I didn't have a computer [laughs], so I didn't do any for a while. But then, when I was in, like, junior high, I got a computer, pulled out the big manual, GW-BASIC, and started writing some stuff, starting from those things, and building it, and then I didn't do it for a while.

But then I came back to coding in college. It went much more quickly. It went much more quickly because, man, I'd done this. I had practiced. I knew what I was dealing with, and it just pushes it forward so much. I also did some...talking about fixing computers, I did tech support for Windows 98 and Me [laughs]. And you learn some things about how a computer works and how to solve things that serve you well in the industry.

WILL: Absolutely, yeah. And I remember selling a lot of Windows 95s. I remember like, yeah, I did a lot of weird stuff, a lot of weird stuff, because I predate Windows 95. I was doing, like, Windows for Workgroups, or before or earlier, a lot of like, MS-DOS type stuff.

MIKE: I remember Windows came out thinking, oh, do you need that [laughs]? I've been doing all this on the command line [laughs].

WILL: Yeah, yeah, yeah. I don't know. And there's a big piece of me, I mean, this is a little bit of a tangent. But, like, I do wonder, to a degree, whether...I wonder how, you know, generations coming up right now, where, like, all the really big bad, like, gross stuff has kind of been watered down. It's been, you know, it's been tamed, more or less. I think there are a lot of computers that are effectively appliances, and there's really nothing to worry about.

If you’ve got a Chromebook, if you’ve got an iPad, you don't have to worry about all that. You don't have to worry about it, unless you want to be a professional, in which case, you know, here there'd be dragons. It's like when everybody opens up their car these days. You pop the hood on your car, and it's nothing but plastic shrouds. You can't see any of the weird hoses, and belts, and cables, and stuff like that. It's all in there, but you don't look at it anymore. You have to take the cover off, and it's like, oh yeah, it's the same nuts and bolts. It always [inaudible 22:13]. So, I mean, I just don't know. I don't know --

MIKE: You're touching something. You’re implying that you do know. You're implying that if you really want to learn this, that learning a language isn't necessarily, like, I mean, it's a part of the process. But you're talking about this lore that you had to learn, even more important than language, because the language of today is probably not going to be the language of 10 years from now. It might be. Java is still around. It’s been around for a while.

WILL: Yeah, Java is going strong. It ain’t going nowhere [laughter]. Well, and Kotlin, right?

MIKE: But it's not the number one anymore. There's new kids on the block. You know, there's Python, JavaScript, and I'm sure Python and JavaScript will be here 10 years from now. But there'll also be another new kid on the block, probably, and particularly talking about GenAI, you know, there's new tools, and that's always been the case. There's always been new tools, and those tools are going to change. And that doesn't change the fact that somebody who is trying to build stuff, and has grappled with hard problems and thought about how to address those problems in a systematic way, has developed skills that are universal.

WILL: Absolutely. There’s sort of, like, call it, like, cognitive, simple machines, like, where there's sort of processes that, like, I could explain to you in five minutes, and I will, and it won't even take me five minutes, but, like, foundational ideas. Like, always debug from the known to the unknown. Always debug from the known to the unknown. And if your stuff isn't working, dumb it down until it works.

I have had a professional day, where I contracted for my full rate, which is substantial, and they hate it, and they were happy to do it. And the butter would have melted in my mouth when I cashed the check, where all I did was make the damn thing turn on. That's it. It didn't do nothing. And that was [laughs] a hard day.

And everybody's had days like that, where you're just getting going. You talk about the lore. That's the lore right there, where it's just like, oh no, no no, just make it work, right? That is a dead, simple idea. Go from the known to the unknown, and if it's not working, dumb it down until it's working. Make it stupider. Like, how much stupider? As stupid as it takes, and then you move forward from there.

I could get a tattoo of that [laughter]. I could write it on my body in indelible ink. And you learn, sort of, like, simple cognitive machines like that conceptually, intellectually. It's a lever, a pulley, things like that. How do you break down these problems? I don't know. I don't know. I mean, nobody ever taught me any of this stuff. A lot of it was just, like, I just had a knack for it from day one.

MIKE: Well, so you say you had a knack. I'm sure there's some truth to that. Well, you say you sat around with a bunch of gray beards [laughs]. I think that software engineering has historically been an apprenticeship industry, and I think that's underappreciated. I think I've said it before on this podcast. I think it's true. You know, a few hundred years ago...there's a reason the most common surname in the United States is Smith is because there's a whole lot of people who are Smiths. There was a whole lot of need for metal workers who could, you know, a kind of cottage industry, who could take the metal and make it into important implement.

And you learn that by going and spending time with somebody who did it and working with them for a few years, maybe 10 years, for some extended period of time, looking over their shoulder and learning all the things that you learn by just being there. And I feel like software engineering is very much the same way. We try to short circuit it, and you think, oh yeah, we can teach this in school, hmm, or, you know, bootcamp. And there's some truth to that, but I think a lot of what you're learning in those places is coming from trial and error and watching the people around you.

WILL: I mean, the principal problem of teaching programming in schools is, I can teach you programming if I know it, but if I know it, do I really want to be teaching a bunch of high school kids, ehhh, you know [laughter], I mean, they pay me pretty good to just do it. And that's problematic if you want to teach people.

I think I've said this before. I think I've said this before on this podcast, and I don't really think of programming as a four-year bachelor's degree type of subject. I think if you graduate from school with a bachelor's in computer science, you have proved yourself to be smart enough and dedicated enough to be trainable [laughs]. The industry, historically, has been willing to absorb the sort of finishing costs to take this raw [inaudible 27:40] and turn it into an actual developer. And I feel like that appetite might be waning.

But if you compare analogous professions in terms of like, skill, you know, which I will make up out of whole cloth in my head, I think what you're expected to know is a lot more like somebody with a master's degree than a bachelor's or a high school diploma, or something like that, you know, all this stuff. Because it's an apprenticeship thing, you could do with a high school diploma or not even. You just have to sit with somebody to fill your apprenticeship. And companies, large companies, especially, they can afford to post that sort of a gate on you.

I'll say this, too, if we're worried about apprenticeships, you know, it's like, I'm a junior developer. I need an apprenticeship. You guys just took the graybeard’s advice on faith. And it's like, you got to have an apprenticeship, you know, to really learn the trade. Well, how do I get an apprenticeship then? The three little magic words: sketchy startups [laughter]. Sketchy startups. Sketchy startups. They're going to pay you in Amazon gift cards. Like, it's going to be in somebody's house that smells funny. Like, you're going to get a paycheck that you can't necessarily cash on the same day [laughter], like, all of that stuff, all of that stuff. But you will get your apprenticeship in.

And this is, like, one thing. There's a lot of stuff that you can and probably will have to let go, like a W-2 form, or health insurance, or anything like that. But the one thing you can't afford to let go is you do have to have at least a journeyman to be learning from. Like, there are a lot of sketchy startups who are like, uh, we don't have anything. You, did you get a web page up one time? You're the CTO/co-founder.

And you got to have somebody better than you, like, a couple of steps ahead of you so you can learn from them, or else it's just going to be a disaster, like, a bad experience for everybody. But, in the end, yeah, sketchy startups, man, so many people, so many people have gotten their start on some, like, really, rinky dink fly by night what do we even sell startup.

MIKE: Another wonderful thing about said startups is you wear a lot of hats. You do a lot of things, and you get to work in a lot of aspects of the business that you're not going to get so much in a large enterprise [chuckles].

WILL: Absolutely.

MIKE: You're setting up the infrastructure [laughs]. You don't have a DevOps team. You are the DevOps team and the engineer. And you've got to do both, and you learn it, or else it doesn't get done [laughs]. Having been at multiple places where they laid off the rest of the team [laughter] and I was doing everything, you learn a whole lot that way, when it's you or nothing.

WILL: That's the truth. But, I mean, it is pretty sketchy. It's pretty intense to go in without any sort of support, especially when you're just a baby dev. I think it's difficult, and when you find yourself in situations like that, the fundamental trade off you make is a lot of people...it's a really high risk, high reward situation, where there are a lot of people who would have made it if they just had anything, anything backing them up, anything at all. They would have made it, but they had nothing. And so, the ones that make it through those guys are going to be serious.

MIKE: Sure.

WILL: But you're going to find a lot of people who, ordinarily, they could have made it, you know. They could have done it. They just didn't have anything, you know. I believe in taking risks and stuff like that, but I don't believe it takes stupid risks. And I think you can get yourself in a bad situation like that.

KYLE: That's a good point, where you can’t get discouraged too fast, like, I don't think I took a necessarily traditional approach. I wanted to go straight into dev, and that backing that you're talking about, or that support, I don't think I ever got it. It just was not available to me. And I ended up finding myself in QA. I found myself in QA for seven years before I really found my niche in DevOps. And saying that, I'm saying, you kind of have to keep at it like. Maybe you don't get the opportunity, you know, right away. But if you stick around and it's something that you really want to go for, maybe the opportunity will present itself. Like, not everybody gets it at first try.

WILL: Oh, absolutely, yeah, completely.

MIKE: Looking for work during the dot-com bust was not fun [laughs]. That was not a great time.

WILL: Oh, it was so rough. Yeah, I worked at a gas station. I worked at, like, straight up, I worked at a gas station, like, I was the ninth shift manager at a gas station.

MIKE: [laughs] I ran up my credit card bill pretty high [laughs], and I very nearly went back to doing construction, because I made money doing construction.

WILL: [laughs] It’s the truth, man, yep. Yep.

MIKE: But on the other side, it got better, which, you know, kind of going back to the theme, you notice how we're talking very little about the language. We said, what did I wish somebody had taught me? I think that you should learn a language. If you want to learn to code, yes, go learn a language. You might not code in it in your job; in fact, good chance you won't, but you're going to learn some things. You are going to learn some things about yourself. You're going to learn some things about the computer. You're going to learn some things about the computer you wish you didn't know [laughs]. You're going to learn those simple tools.

You're going to learn, maybe if I cut this problem in half, then I can very quickly do a binary search and get to the root of the problem. You're going to learn how to write a configuration file, which you will do an endless number of times in your career. You're going to learn how to do all of these lore sorts of things that change your perception of the world. You're watching the master blacksmith banging on that metal, and you're learning what the colorless sparks mean. And it's going to change you more than the language.

Although the language is going to be important, you got to learn something. You need to go out and start building something. And I will express that strongly. I think you said it really well in your monologue [laughter], Will, that by building those things, you're going to learn a different set of skills. And, you know, I came into this, this podcast thing because I wanted to express something about you got to learn the language, but that's not the most important thing you're going to learn.

WILL: Like, I almost think...one thing, and this is something that I’ve thought for a while, maybe I'm just out of touch, but I really feel like...how do I put it? In the age of generative AI, ChatGPT, and stuff like that, like, I've never thought all that much about programming languages, and I plan to think so much less in the future, because, like, what’s a language? A language is a way to express a series of procedures, a series of thoughts, very precisely, which human language does not do, will never do, but, you know, sort of even a high-level programming language. It is a language to express logic, procedures, stuff like that, you know, really specific way.

And ChatGPT, generative AI is amazing at translation, the translation of ideas across, you know, from English to Spanish, from Spanish to Japanese, to Korean, to Smalltalk, you know, whatever you want to call it. And so, you know, language Kotlin, Node, Python, Ruby, Swift, whatever, the ideas are important. The language is nearly meaningless at this point. Maybe that's just me, because I understand the concepts really, really well, and I know exactly what I want. I know exactly what I want. And so, how do I say that in Python? I almost don't even care, you know.

MIKE: Will, I'm suggesting that you've got to pay your dues, that it is helpful. It's helpful to take that stack of CD-ROMs and of source code and compile your operating system. Because you know what? CD-ROMs don't matter. And that particular build of Linux you probably used for a little while and forgot about [chuckles], you know, the specific problems you were solving were irrelevant. But you had to do some work grappling with it, and you have to learn something, I think.

You know, we all, as a child, we don't know how to speak, so we watch people around us speak. And I don't think it really matters what natural language we learn. You got to learn one, and then you can communicate. And then, if you need to go learn another language, that's what you do, another natural language. Likewise, here, you've got to learn how to speak. And I do think that you have to go through some work, and, like you say, work for sketchy startups. Go write a bunch of stuff. Go do a bunch of vibe coding, you know, go use a bunch of generative AI and see what you can build, and see how it's broken, fix it, and [laughs] push it out there. Work on something.

And it's that working on it where you learn those skills. So, I do think you need to learn something specific. You've got to drill down enough to be able to understand. So, if you're writing apps using generative AI, they're going to write it in the language. Underneath, there's an interpreter, compiler. There's pieces that you're still going to have to understand because it's going to do it wrong, and you're going to have to go and learn...say it's Python. You're going to have to go and learn Python enough to fix its bugs to know, hey, this is wrong. Where did it go wrong? And go and fix it, or, you know, ask it to rewrite it again, figure out which one looks best.

You're going to have to learn the underlying structure as you go. I think learning one matters. I would recommend that. Learn one thing well before you generalize. Maybe not perfectly, but I do think the learning one thing really well does help you generalize. Going back to what we talked about at the beginning, [inaudible 39:22] one thing further.

WILL: For sure, for sure. Make something, build something, and put it out in the world. Make something and put it out in the world. Vibe coding is great. I am pro vibe coding, you know, like, just making things. I love making things. Like, that's the foundational thing. There's a big piece of this where it's like, if the idea of making things just for fun, to scratch your own itch, because I want to make stuff, because I like making things, and I'm just going to do it and put it out there, put it out there because there are people who desperately want to find you, but you have to give them an opportunity, some avenue to reach you.

If you want to work, you have to find some way so that somebody who's searching for you desperately but they don't know your name could find you in some way. I think that's absolutely foundational. But if the idea of just making stuff just to make it doesn't really sit well with you, where it's just like, well, but who's going to pay me? There is a big question, I think, you need to ask yourself, which is this for you? Because that is the fundamental thing.

If you don't just love making stuff, then is this really a good idea for you to waste your one precious one-time-only life doing? I don’t know, there's lots of wasted, like, just make a bug. And that's fundamentally why, like, you know, ultimately, like, that foundational loop of making things and putting things out in the world it always works if you're just really into making stuff because not everybody is and, like, we could smell our own.

MIKE: [laughs] It is true.

KYLE: I would tack on, too, to go along with making stuff, understand how stuff is made or what is already out there. Because I feel like that's one thing that I missed going through my career is I didn't take the time to stop and understand what it was that I was coding against. Like, what was the underlying operating system like? How is this running on Linux versus Windows versus, you know, the other OSs? You know, just having that fundamental understanding underneath the hood just helps you understand what you're coding as well.

WILL: That is such an important concept. And this is one thing that I think we're getting very, very, very, very, very wrong in terms of the vibe coding, like, generative AI thing. Like, hey, man, I understand that we all want to make things. We want to push these things out in the world. We want to make things, and that's, like, the goal. But one of the other cognitive levers, the cognitive simple machines, is like, when you do something and you're like, I don't know how that worked, read the manual, man. Just read the manual. You did something, and it's like, oh, I'm just sort of trying, like, I’m just trying something. Oh, that worked. Whoa, whoa, whoa, what did you do and why?

Like, taking that step back, that is another one of those intellectual, simple machines that you really need to master. If the other one is a lever, then this one's a pulley, where you integrate these things, and you get a larger understanding of how stuff works. That's why I don't care about languages anymore, because I've done this so many times that I understand the foundational things. Like, oh, we're doing a [inaudible 42:53]? It's like, oh yeah, that's a thread, but it's got rules around it. I'm like, okay, that's cool, you know, because I understand how that works.

And, like, you know, oh, promise. It's like, oh, this is how a promise works. This is how an async function rewrites my JavaScript with all these freaky returns and stuff that it puts in there, where I'm like, okay, these are the rules for taking this async promise away thing and turning it into what appears to be an imperative function, just all that kind of stuff and, like, that's another one of those fundamental things. And it's one of the things we get so, so wrong about generative AI, because generative AI goes both ways. Like, not only can you ask it, “How do I do this?” You can ask it, “Whoa, what the hell did I just do?”

KYLE: Yeah, we need to do that more often.

WILL: Yeah, we need to do that way more often, way more often. You're talking about the lore. That's it, the generative AI it could do that for you, just basic stuff, where it's just like, you know, like, oh, I Googled this thing on Stack Overflow, which is still really, really relevant, although some of those Stack Overflow answers are getting pretty creaky, where it's like, the accepted answer is, like, 10 years old. And I'm like, I don't know whether that's best practices anymore, homie. I don't even think that'll compile. That's from 2009. We're in a brave, new world.

Like, that Stack Overflow answer can drive a car [laughter] because we’ve moved on. But you can read it. You could feed it into generative AI and say, like, “What is this?” And it'll tell you, and, like, it works actually a lot better. Like, what the hell is this? So, it'll be like, it's this, and this, and this, and there will be references. And you should look at the references because sometimes it'll hallucinate [laughter], but the references are usually pretty good. There's all kinds of stuff out there.

I think it’s never been a better time to learn how to make software, learn to code, learn how to make things using software. I don't think there’s ever been a better time of your life. And I think for the people that have the knack and really want to do this, I think the doors are as open as they've ever been, really. I don't think it's that much harder than when I graduated, you know, college at the middle of the dark days of the dot bomb era, you know, and I had to work graveyard shift at a race track, sell my plasma in the hopes of getting into the industry.

MIKE: I did that, too [laughs].

WILL: I don’t get...Pro tip: drink a lot of water. I'll water that down. You're not stealing my vital fluids for nothing. They gave me the side eye. They're just like, “This is looking pretty thin, dude,” and I'm just like, whatever. I don’t care [laughter].

MIKE: Another pro tip: tums.

WILL: Tums.

MIKE: Calcium. They use an anticoagulant that they'll make you feel funny, but if you have lots of calcium, it counteracts it.

WILL: Interesting, interesting, okay.

MIKE: Totally off topic [laughs].

WILL: I mean, you know [laughter] --

MIKE: Make sure you’re giving blood, too.

KYLE: You need to know how to pay for it. You need to know how to pay for your education.

[laughter]

WILL: That’s right. That’s right. It wasn't for free. Yeah, it wasn’t for free. But, anyway, I don't know, I remain optimistic. Nobody is throwing their phone in the trash, you know, nobody's spending less time on the internet. I think I'm probably going to stay fed, and you can, too. There's room for you. If you're listening to this, get in here.

MIKE: Yeah. So, during the pandemic era, interest rates were really low, perhaps artificially low, and so big tech companies invested a whole bunch of money in themselves because they couldn't invest anywhere else and make money, as much money as they could by investing themselves. Interest rates go up, and they look and say, hey, I can take this cash I've got, and I can go make whatever percent off of it really consistently. I don't have to risk hiring a bunch of people and hope they build something that'll make me money, so they let a whole bunch of people go.

WILL: It’s true.

MIKE: So, there is a flood of engineers out on the market right now, and it makes it really hard to find a job. I actually read a data analysis just this week about this topic. And there is --

WILL: Send it to me, would you? Will you email it to me?

MIKE: Yeah. I think it was in The Washington Post, if I remember right. Basically, they found no evidence thus far that generative AI has had a meaningful impact on hiring in software engineering.

WILL: I believe it. I think there's a lot of weird stuff in terms of big tech. This is a little bit of a...no, it's a substantial diversion from what's going on. So, I work with a lot of people out on the West Coast. And what I have found is that big tech let a lot of people go, but it's having a very local impact, because big tech it's all on the coasts, you know, a lot on the West Coast, to a small degree, on the East Coast. Well, all these guys required people in the office, so you have all these people who are unemployed on the West Coast. But they live on the West Coast, and they need West Coast money. And they can't make Midwestern money, Utah money, mm-mm, that won't do because they live in San Francisco, and they have to pay their San Francisco mortgage.

And so, I feel like a lot of the unemployment effects from big tech losing a lot of jobs are very localized, where, you know, like, I’m pretty senior, and I get paid pretty good, but I don't get paid pretty good San Francisco money. Like, nobody's cracking half a mil. For me, that's not the dough I'm making. You can make some, you know, you can get on levels.fyi, and they'll tell you the Bay Area is a very lucrative place to be an engineer. But you don't get that money, you know, telecommuting from Idaho anymore. They kind of shut that down because they've been doing layoffs.

And so, demand is very high, and they can say, like, no, if you want Facebook money, you're coming in Facebook. You're going to have to get on the half-pipe and skate around every day, or whatever they do [laughter], you know. You're going to have to come and eat our sushi if you want that money. So, I don't know, it's one of those things, that very top tier, very expensive market, very, very, very high pressure stuff. Like, yeah, those guys are hurting pretty good. But I actually don't know whether it's so bad if you fly out of the country.

MIKE: For new hires, it's real bad. Yeah, it's real bad. I was interviewing recently for an internship position, and just the number of applicants we got alone.

WILL: Interesting.

MIKE: I think we posted it, and we had...I don't remember the number. I think it was over 300 applicants in one or two days for an internship position, not even, you know, like, a full-time, long-term job, just to get an internship.

WILL: For how many positions? For one?

MIKE: Two, yeah.

WILL: Two. Mercy, that's brutal.

MIKE: Yeah, it's tough out there for people starting.

WILL: So, I should start, you know, exploiting these [laughter] beautiful young minds for my sketchy startup. You’re saying, Will, start a sketchy startup, like, there are people to be trafficked.

[laughter]

KYLE: Oh geez.

MIKE: And on a more positive note here, it’s always --

WILL: That was great. I’m very optimistic [laughter]. I have so many sketchy ideas. I'm going to be a head in the box. You'll never see me. I'll just Venmo you sporadically [laughs].

MIKE: I think Will said it really well, not very long ago: the world needs people who can think clearly about problems, who can break them down and express unambiguously and in a well-structured way about what the problem is that needs to be solved. This is the problem I need to solve and solve it. That need is great today, just like it was every other time. People have been hiring software engineers for the last, what, 70 years [chuckles]?

WILL: Yeah. All the way back to Grace Hopper, man. Like, who's going to fix all this crap [laughter]? Yeah, fascinating. All right, I'll have to cook up some sketchy startup ideas.

MIKE: Oh, any final words?

WILL: No, no, that was it. That’s the final thought. Like, I’m going to start cooking up some sketchy startup ideas [laughter]. There are exploitable minors out there.

MIKE: Expect some job listings in the near future [laughter]. Until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, host Mike kicks things off with a personal story about his wife’s jewelry-making hobby to introduce the episode’s main theme: the classic “build vs. buy vs. open source” dilemma in software development. Using analogies like 3D printing and sourcing beads, Mike illustrates how creators often rely on third-party resources rather than building from scratch. The conversation, joined by guests Justin, Kyle, and Will, highlights how this decision-making process mirrors software strategy—balancing time, cost, scale, and business priorities when choosing whether to build proprietary solutions, buy from vendors, or leverage open source tools.

As the discussion unfolds, the group explores real-world applications of these choices, especially at scale. Will draws from his experience in academia and enterprise to explain how even cutting-edge innovations are often built atop existing open-source frameworks. Justin stresses the importance of aligning technical investments with what drives a company’s revenue, while Kyle adds that factors like security and infrastructure complexity often make buying solutions the safer choice. The team discusses examples like login systems and reporting platforms, agreeing that while DIY solutions might work early on, scale and security typically justify outsourcing or adopting proven services like Auth0 or Snowflake.

Toward the end, the conversation veers into the implications of AI and large language models (LLMs) on the future of development. They debate how junior engineers might adapt, how architectural decoupling can future-proof integrations, and how the rise of AI could alter productivity expectations. With humor and insight, they emphasize that every decision—whether to build, buy, or adapt—requires thoughtful consideration of timing, scale, and evolving business needs. The episode wraps by reinforcing the idea that good architecture, adaptability, and a keen sense of timing are essential for sustainable tech development.

Transcript:

MIKE:  Hello, and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting again today. I apologize: my voice is a little bit raspy. I've had a little bit of probably a virus [chuckles] going through the family. I feel much better today than yesterday, which is good [laughs]. But I may be a little bit raspy.

I've got a small group with me today. We're actually moving offices at Acima, moving to nicer, I guess, you know, a little bit updated offices just a couple of buildings down. So, same complex, it's not much of a move, but [laughs] upscaling a little bit today. So, a lot of people are tied up in that. But I've got with me here Justin and Kyle. And we should have a great, more close personal conversation today than we sometimes do. That'd be great.

I'm going to start by talking a little bit about my wife [chuckles].

JUSTIN: I'm a little curious how you're going to spin this because this is the build or [laughs] buy --

MIKE: That's right [laughs]. So, my wife is an artist. She doesn't paint very much. She makes stuff. She's a maker, and she's made a lot of things. But the thing that she does most, and she's done this for years, she makes jewelry. I cleared this with her beforehand [laughter], you know.

JUSTIN: You’re going to be internet famous, hun.

MIKE: Yeah, exactly [laughs]. I've mentioned her before. She makes stuff. She makes jewelry, not, like, diamonds necessarily. She uses natural stones...diamonds are natural stones. But she generally does more, like, beadwork. She'll do complex subbead things, you know, of patterns of natural stones. There's one she uses a lot called labradorite that is just this beautiful stone. Kyle's nodding, like, I know the one [Chuckles]. It's translucent, but if you look at it in the light, it has this blue shine in it. It's really pretty. But she uses other stones as well. And she'll make these patterns and put, you know, she uses a lot of, like, so it's like a larger stone as the centerpiece. It’s kind of chunkier sort of stuff, really pretty.

And she spends a lot of time, you know, like, hand tying it between the beads [chuckles], and she'll say, “Oh no, I messed something up. Now I got to undo, like, two hours of work [laughter].” And so, she'll have to undo two hours of work to go and retie it and move on. And she spends a lot of time building this stuff. Really cool. And so, you know, each is one of a kind; that's how she does it, which is really cool. You get these one-of-a-kind things that she makes.

But there's some interesting aspects about this. She doesn't make the beads. She doesn't make the stone. I mean, the earth made the stone [laughter], and then you have somebody in the supply chain, you know, somebody whose job it is to go collect the stone from a quarry somewhere and distributes that out to suppliers. Some of the suppliers will buy the stones. They'll take those stones and make them into...and they'll make lots of different things, but some of them will make them into beads of different sizes. So, she'll buy it several steps down from the original source.

She buys the stones. She doesn't make those herself because we don't have the manufacturing equipment, nor would we. I mean [laughs], it would require a lot of upfront costs. Now, if that was your thing, then that would be a very reasonable thing to do. She was actually talking to me the other day like, “You know, I see these people who have their own stone shops, and I'm tempted to [chuckles] start small and go into that.” I don’t know if we'll do that or not [laughs]. But she's thinking about expanding and doing some of that, you know, to actually start getting some of that stuff wholesale. We'll see.

This ties into our topic, by the way. This isn't just an aside. It's very relevant [laughter]. So, we could potentially buy some of that equipment, but it has not been worth it thus far based on the business that she does. She doesn't do this, but I know people who have 3D printers, so who do buy the supplies and start making it themselves. Justin's, like, pointing [laughs] to 3D printing equipment over [laughter] in the corner. We only publish this in audio, but we record it with video. Maybe someday we'll do the recording, you know, post the video, but [laughs] we only post the audio.

So, he's got the 3D printer, and 3D printing is super cool. You can print out all kinds of cool stuff, but, like, the stuff for making the beads out of the stone, you have to buy the equipment. And you can download the designs. They're great. I mean, you can download all kinds of cool open-source stuff on the internet, where people have published the 3D models, and you can print them out yourself. But you've got to buy the stuff.

You've got to buy the resin. You know, you're taking responsibility for everything in the process. Whereas if you're not doing that, you just order from somebody who does it online, and they ship it to you unless you get to a certain scale. Justin, have you reached the scale where it's actually been economical for you?

KYLE: [laughs]

MIKE: Hear the laughter? Hear the laughter, listeners [laughs]? Of course not, but it was cool. So, you did; you got to do it [laughs]. I've got Matt who's been on the podcast quite a bit. He has a 3D printer. I've got a nephew who does it. I don't think it's ever been economical for any of these people, but you do get to do...I know Matt just does a lot of his own 3D designs. I don’t know if you do that, Justin, or not.

JUSTIN: Not as much as I'd like but, I mean, that's the intent, right?

MIKE: Right. The intent. So, it's one of those, someday [laughs]. It's like my big, beefy computer I've got on my desk here for machine learning that I'm not using right now [laughter]. When I get a moment, we'll use. So, we talked about...oh, and Will is joining us. He's been a bit delayed, but Will Archer is joining us, which brings us to our topic for the day. And you're joining us right on time. We're going to talk about, in software, when do you build; when do you buy; when do you use open source?

And we've just talked about very practical, tangible items, but software follows a lot of the same rules, right? And there are times when any one of those three is appropriate. And you think about things in the tangible world, and a lot of the same rules still apply. And I'm going to stop talking for a little bit because I've set the stage, and I'm curious what your thoughts are based on kind of the stage that I have set.

WILL: I feel like I've had a little bit of a unique career in that I've spent a decent amount of time kind of on the edge of technology, like, actually doing brand-new stuff that has never been done. Because I came from academia, where we're, like, hardcore research, like, stuff that isn't commercial, and it's not going to be commercial for a while, and then brand-new technologies where there weren't libraries. It wasn't, you know, necessarily sort of your standard web CRUD systems kind of integration stuff.

And what I have found, personally, is you will almost never be going soup to nuts, brand-new stuff. The most out there, woo-woo stuff that I ever developed was a static analyzer for embedded device drivers. And what we were trying to do is we were trying to prove memory safety, where it was this low-level super tiny code set, and we were trying to use...and what we were doing there was, like, we were using a publicly available open source, another woo-woo research project theorem prover. And so, we were sort of trying to analyze the code, you know, get a formal specification for this embedded network, you know, firmware driving...I'm, like, throwing bits on your, like, you know, like, real, real nitty gritty thing.

But I was only doing this within the ecosystem of this previously available open-source stuff. I did a bunch of, you know, kind of pushing audio processing forward on the iPhone when it came out just, like, real cutting-edge stuff. But even then, I was taking a bunch of existing open-source stuff for Linux like C++ drivers, and files, and processing stuff, porting it over to this thing, making it run there, then adding my tiny, little secret sauce, which was cool. It was cool. I did stuff that pushed hardware forward a little bit.

But even then, I was doing just a little bit, like, just this extra thing. There was this decoder thing. There was the analyzer thing, and none of them worked for this, and I made them work for that. And then, I added on some sauce. As far as I've gone, it's just incremental improvement. And that's in a case that I think, you know, outside of those two gigs, I don't do stuff like that anymore. I'm doing big enterprise line of business stuff, where I'm, you know...I hate to call it an assembler, systems integrator, you know, because I write plenty of custom code, nothing like that. Even as far out onto the ice as I've gotten, that's all it's been, you know what I mean, so...

MIKE: Absolutely. So, standing on the shoulders of giants.

WILL: Well, exactly. Absolutely.

JUSTIN: Personally, I think, you know, the question is, you know, build, buy, or open source? It comes down to what makes your business special? Because you need to determine what is making you money, and you got to maximize that and minimize everything else. And if what is making you money is something unique that you have to write custom software for, yeah, go do that. But if it's something that's not core to your business, buy it or open source it. Push everything else away. Just focus on what makes you money. At the end of the day, if you aren't making money, you know, what are you doing as a business [laughs]? I mean, maybe you got a lot of money pouring in from investors, but long-term, you got to maximize your, you know, the thing that's making you money.

WILL: Okay. So, I'll play devil's advocate here because one of the things, you know, when I was working with Acima, and one of the things that I do, like, right now for my large retailer, is at big enterprise scale, small efficiencies add up. One of the things that like, you know, like, I don't feel like I'm talking too much out of school here, but I spent a lot of time dealing with various, you know, financial infrastructure providers trying to get a couple of basis points on a billion dollars’ worth of credit cards.

JUSTIN: That's making you money.

MIKE: That’s making --

WILL: That's not Acima’s business. It's never been Acima’s business. It is not core business. That's not core business. That's just like, yo, at this scale, if you can get me a couple of basis points, you just paid for your whole team forever. But that isn't why Acima exists. It's nothing to do with Acima’s core intrinsic value proposition, whether I can run cards for, you know, 3.5% or 3.45% has nothing to do with whether the business lives or dies. But if I can get it, you know, 500s of a percent more efficient, that justifies the entire team's salary, [chuckles] forever because it is kind of cross-purpose, you know. It's not going to double the business. That's not going to --

JUSTIN: But like you said, it's going to pay for stuff.

WILL: Sure.

JUSTIN: So, if you can identify something that you can write that is custom that is worth a lot to you or to the company in some way, you focus on that, and that's what makes you unique, probably. And, at the end of the day, though, it's like, do you have to write your own code to optimize for that, or can you buy something that optimizes for that? And so, you're always going to have to do this juggling, this evaluation of is this ultimately going to be worth it and make more money for the company? You know, buying versus building.

And a really good example for this, where I'm currently at, or every place that has a login, to have a good, secure login, that takes a lot of skill. And there are companies that have figured out how to do that at scale and sell it as a third-party provider, you know, Okta, OAuth, you know, Okta, sorry, OAuth.

WILL: Auth0.

JUSTIN: Auth0. And, you know, there's a number of other providers and Omni, for example, or, you know, where I'm currently working, we're not in the business of login. So, obviously, we're going to be, you know, hey, this is very important that we get something that did it right. And so, we will shift the risk and the responsibility to this third party, and we will pay them gobs of money to own that. Because, at the end of the day, that's still going to be more secure and cheaper for us than writing it ourselves.

KYLE: I think you brought up another variable in what you were saying there, too, depending on what it is that you're looking at, different variables play a larger or a lesser role, and in login, security is massive. You can roll out your own login stuff, but it's like, do I want to risk it, or do I just want to go with the golden standard? I'll pay that premium for that golden standard, the tried and true that we know works.

And I think we run into that with a lot of SaaS products, where there might be a really good, even open-source product out there, but it may not have the name. It may not have the GitHub stars or whatever you're using to qualify it. You know, you're not willing to risk your business. Even though it's a massive cost difference, you're not willing to risk your business in going for that open-source solution versus the paid-for solution.

MIKE: Well, you're not just paying for the software. Going back to what we talked about with the 3D printer [chuckles], software is free with open source, but that is not the only cost. You now have to own everything about this, make sure that every configuration is right, because if you get it wrong, then your business gets compromised [laughs]. And now you're paying for the security experts. Now you're paying for, you know, the liability that one of them has done something wrong. And you start adding up those costs, and it's not actually free [chuckles]. It's not actually anywhere near free, and the cost is likely higher than just paying for it.

KYLE: Well, and, to your point, how many of these larger companies have we seen spawn where none of their software really is proprietary? They're just using open-source software, and they're selling it to you as a service. There are so many companies out there right now that, you know, they'll use the ELK stack. And they're using, assumably, some of the more open-source type portions of that. Maybe it's licensed to some extent. But I'm saying you are just paying for the support. You are paying for them to manage your infrastructure for you and to have that reliability.

WILL: Got it [inaudible 17:19] not like us. We would never just put together a bunch of open-source software and sell it to people [laughter].

KYLE: No, never.

MIKE: Or [laughs], as you say, shoulders of giants. Does anybody build an operating system from scratch if you're not in the operating system business? No. And even the people who are in the operating business, a system business, if you're Red Hat, you didn't build it. I mean, you built some of it [laughs].

KYLE: You've contributed to the community.

MIKE: You contributed to the community. And even Windows is moving a little bit in that direction. It’s not a great model [laughs] to have to build the infrastructure. You know, everybody lives on top of that. And the infrastructure works really well when it's shared. And you look at all the cloud platforms. Yes, AWS, you could replicate a lot of what they do pretty easily, but you don't [laughs].

KYLE: No.

MIKE: Or, you know, I mentioned one cloud provider. There are others, of course, that are also very competitive in there, you know, Google Cloud Platform or Azure. They provide that.

WILL: Well, but AWS and Oracle, let's say, you know, some other ones, they do provide a counter narrative here in that, I mean, Oracle is maybe the furthest along the curve where their opacity has...like, when I run into some ops team who's just like, “Yeah, we're deep in Oracle,” I'm like, “Oh man, I'm so sorry.” Because Oracle runs...everybody I've ever talked to says that Oracle runs a real type database. They run good database.

MIKE: 100%.

WILL: Their stuff is as good as you'll find. But I don't know a lot of new people, a lot of new builds that are getting on them because they really charge you for it. And so, what you are running into...oh, maybe Kyle could speak to this better than I can. But I know a lot of people who have been pushing back on AWS pricing because you get real locked in real deep to AWS. I don't know, I have personally, in my own personal little micro projects, looked real hard at some of the other providers, you know, other than AWS for some of the stuff. There is a lot of people who are taking it back to the colo, the independently run colo just because AWS has been tightening in the screws bit by bit by bit. And that is the counter-narrative.

You get locked into a system, and there's some very smart MBAs. And if they don't have very smart MBAs, some hedge fund will buy them and install a team of very smart MBAs to [laughter] slowly bleed you dry, you know.

JUSTIN: I think we've talked a couple of kind of extreme examples, but I do want to look at one that may be a little harder to evaluate, and we all know what Snowflake does.

WILL: I don't know what Snowflake does.

KYLE: Well. [laughter] Yeah, we should probably touch on it for our listeners.

JUSTIN: I think Mike could say what Snowflake does.

WILL: [laughs]

MIKE: They provide data warehouse services, so backend, not real-time reporting data, real-time transactional databases, but business intelligence reporting, you know, all of the post-transactional database stuff and the whole ecosystem for making that tie in together.

JUSTIN: Yeah. And so, that data, Snowflake itself didn't generate that data. That data lives in our data stores, you know, in our databases. And, theoretically, business could come to the engineers and say, “Hey, I need this report.” And the engineers could go in and generate the report for them, right? And so, there's nothing that Snowflake is doing that is absolutely necessary for the business to run necessarily, you know. But it does [chuckles]...it does make the engineer's lives a little better since they don't have to interact as much with the business folks. It gives them tools to dissect that data. So, the question is, do we build a reporting system? Do we buy Snowflake, or do we use some sort of open-source thing? Like, what are the pros and cons there?

MIKE: The answer is don't ever build your own reporting.

[laughter]

KYLE: Please don't.

JUSTIN: Why not? It's our data.

MIKE: I've done it. I've done it before. Have you ever seen what happens when said business person runs a report against your production database that has grown a little bit bigger than it was a year ago [laughter]?

JUSTIN: Why isn't the website responsive [laughter]?

MIKE: That's what happens. Yes. And then, you end up using a data warehouse service like AWS Redshift. And [chuckles] next thing you know, you're rebuilding something like Snowflake. Now, you don't have to go to Snowflake. You don't have to go that far. But building reporting yourself, I mean, if you've been living under a rock for the last 20 years, data has become important [laughter], and it drives a lot of business and is actually, you know, what's driven all of this AI revolution that we've had over the last few years. It's the reason we have smart devices. You can talk to them, assistance, because there's a lot of data around. Data has become this huge, huge thing, and well-run businesses use that to run. They need that data.

And so, reporting has been done before. I don't know how clever you think you are. Just like you're not going to build your operating system from scratch because you're not actually that clever, it takes a team of very smart people years to build that. You're not going to build a better reporting stack for cheaper than what's out there. So, there is my strongly held opinion. If you are building the reporting, except perhaps if you're building some special visual chart that nobody else is doing, your own little special sauce on top of the framework that others have built, then there might be a business there.

But, generally, if you're not in the business of specifically reporting, I think the answer is if somebody asks you to build a report, you say, “Okay, what tool are you going to use for that?” That is the answer [laughs]. The answer is not, “I will build that.” Now, push back if you think I'm wrong, because I could be wrong.

WILL: I don’t know.

KYLE: I’ll tuck onto what Mike said there and say that, when you do that, what I've seen is you have a database that is then dual purpose like that. And as a platform, DevOps, whatever you want to call me, engineer, trying to support that infrastructure for my engineers, it's one of those things where scaling for those different types of workloads is vastly different. And so, when you have that dual-purpose database, how do you scale it? So, if you're not using another solution like this, it's almost impossible.

WILL: Well, no, I suppose I could say, you know, push back because I think all this stuff there are arguments for both. And the biggest argument for doing it yourself...and there’s maybe two. One is there's this super special thing that only I can do, and no one is doing it. It's a niche thing that's just unique to me. And then, what do you do? You take an open-source ecosystem and solution, and you build on your little thing on top of it. That's usually how it gets done. And then, two is you just need a little thing. You just need this one thing. You just need to run this one report. It’s just one. I just need to run, you know, sales for the end of the week, just tally it up, right? And everybody's laughing. It’s like, oh, doing it yourself is a dumb idea.

No, doing Snowflake for some tiny ass thing is the bad idea because it’s only a bad idea when the thing snowballs. And you have a goer, and this thing is working, and, like, you're doing more and more. Instead of every day, it's every night. Instead of every night, it's two times a day, and then things get bad, right?

But no one's counting the times where it was just a thing that you built, and then it didn't get any bigger, or you stopped using it in the first place. It was just something where it's just like, oh, it's a thing. It's like, ah, yeah, it was dumb. Whatever. Like, it is definitely worth investing in something that's a goer, and you're using it a lot, and it's starting to impact the organization. But, like, most of my ideas are bad, you know, I just try it out. Just try it out. You’re like, oh, that was a dumb idea. It's like, yeah, put it with the others [chuckles].

KYLE: Just, like, so maybe as a proof of concept or as a start, it's not a terrible idea, but if it grows at all, you probably want another solution.

WILL: Exactly. And then, the real devil becomes, oh, it's growing. But like, were you watching it? How big did it grow? How long has it been? How many people are running that report and how often? [inaudible 26:46] how that’s going to go [laughs].

JUSTIN: I think this goes back to if you are in a startup, like, a small startup, and you're trying to figure out how to make money, and you're, like, my budget is zero, and, you know, whatever needs to happen, you know, depends on me, and the money that squeezes out of my wallet that, you know, I made doing something else for 10 years. So, if you're in that startup mode, I think it goes back to what you said, you know, you are doing everything by yourself to see, you know, does it make any sense, you know, to run it at least once? Because a report, like you said, you know, I could run an SQL statement and done, and, you know, I can automate that and done then, you know, it grows and so on and so forth.

But it's like, a lot of this time, in startup mode, you are exploring everything because there's some sort of business need and, usually, if you can identify that business need, and it's some small X unit of time to execute it once, you're probably going to do it and try it out just so you understand what's going on. And then, once you’ve ran that proof of concept, then you can decide, oh, this took me, you know, half a day to do, but I could see it growing a lot. Now I can evaluate the open source, or I can evaluate the vendors, because I know what's required. I don't think you know enough to make those other decisions until you've tried it yourself. It's really hard to do that until you've tried it yourself.

MIKE: I should put an asterisk on what I said before, at scale [laughter], at scale, don't build reporting yourself. But I am completely in line with what's been said and figuring out what that line is. Where does this count as scale is the whole reason we're having this conversation; otherwise, it would be so obvious. We wouldn't have it in the first place. Because when you are small, you've got to build it yourself. You've got no other choice.

WILL: Yeah, everybody starts small. You know, everybody starts with nothing. Even in a big company, like, if you don't already have, you know, your Snowflake, your Red Shift, whatever, whatever you're doing...Even in a big company, if you've got a Snowflake contract, no problem. If you've got OAuth contract, no problem. All good.

But the first person to be like, yo, this is out of hand. Like, we've lost our minds. And then, the six-month grind to move all the old junky stuff onto the new, shiny infrastructure that's brutal. But we've all migrated, where it's just like, this can't go on, and it's just like, oh, we have to drain the swamp now. Okay, nuts. Well, that's going to take a while, and that's just the nature of the business.

It's like, was it Chesters + Spence? If you go back, you know, you don't assume perfect knowledge as you go back in time. All of the terrible engineering architectural infrastructure decisions that we've made that are actively ruining our lives on a day-to-day basis were not only reasonable, but the optimal decision at the time, you know?

JUSTIN: I was thinking about this earlier this week, and everything that we write will eventually be tech debt.

MIKE: I've heard it defined a little differently. All code is tech debt. It has a cost, right? It's infrastructure that you now have to maintain. It doesn't matter how awesome it is. It could be the most beautiful thing ever. It's still costing you money. You've got to maintain that thing. Unless it's out there bringing in revenue, it's pure cost [laughs], so it's only hurting you. And there's always...it's never in the middle. It's never purely making you money. There was a cost to building it. There's a cost to maintaining it.

WILL: Yeah. All my commits will die a hero or live long enough to become the villain.

[laughter]

JUSTIN: I'll put that on a t-shirt, Will, and send it to you.

WILL: Right [laughs]? Yeah. All that tickets, man [inaudible 31:07] nothing. Oh, dude, it is terrible when your tenure at a company has extended long enough that you start popping up in the git blames, where it's like, who is responsible for this?

MIKE: And you're the one doing the git blame [chuckles]?

WILL: Yeah. I’m like that fucker. [chuckles] God.

JUSTIN: So, I guess I got a question for this. It's like, given the fact that everything's going to change, I mean, you got your buy, your build, and your open source. I think there comes a time when you are going to be, you know, you're going to be switching in between those, especially, you know, you probably built a lot when the company was small, and now you're moving to buy and build. So, what can you do if you are small, or even if you're big, you know, what can you do to minimize the cost of that switching?

WILL: I remember reading this, and it sounded good, so I never checked their sources. But successful companies spend, I want to say, like, 10% of revenue on research and development. It could be 20, but I think it was 10. It was 10% is R&D money. If you're successful, over the long term, you're spending 10% in research and development. And I think, you know, you should be spending 20% on technical debt. If you're spending less, you’re screwing up. If you're spending more, then, hopefully, that's sort of you're paying off the old credit card bill, and it's going to get down to a slim amount.

But I think that's probably [laughs], like, probably, you know, 10% on research and development, new stuff, 20% on paying off the mortgage that, you know, your company makes its living on. And then, the rest is just sort of operations and line of business stuff. I think that's probably the right mix. Because if you're much under that, you know, I mean this is, like, long-term, broadly over the scope of a year, I think if you're much under that, you're running up your balance.

KYLE: What company gives you 20% for that? Because [laughs] I've never been in one.

MIKE: Acima, historically, said 30% for tech debt.

KYLE: Really? Okay.

MIKE: Yeah. There's been a bit of a tightening of the belt over the last year, but I think that that's going to be...we’re going to go out, like, another couple notches [chuckles] in the coming year, which is I think an important thing because I actually am aligned with Will. I think those numbers are about right.

KYLE: They sound good.

MIKE: You could argue is it 15? Is it 25%? But it's somewhere in that range. It's pretty close to the right amount. If you're not spending about that much time improving your stuff, it's not going to get better. But then, going back to the question, how do you avoid that problem? You know, sometimes you can't see the future, and you shouldn't try. But good software architecture never goes out of style [chuckles]. Decouple your things. Don't make a big, tangled mess. Build discrete components. I don't care if it's in a monolith or it's in separate services.

Make sure that you have separate components that can be separately maintained. There's a clear boundary between them, and [chuckles] they don't know about each other's business. Don't let things grow too big. And then, when you have to replace your component with the thing that you buy, it's not very hard.

JUSTIN: I totally agree, and I think that lends itself to not being a single-shop company. Like, oh, we're a Microsoft shop. Everything we own is Microsoft. Or we're, you know, a Salesforce shop. Everything we have is Salesforce, and it drives our business, and it drives our website, and things like that. So --

KYLE: Or an AWS shop.

JUSTIN: Yeah [laughs]. Or an AWS shop. The funny thing is, I was chatting...like, with all this AI stuff, and I hesitate to bring it in here, but the thing is Google and Microsoft both just released frameworks for having an Uber AI agent that understands all the systems that it's plugged into and --

MIKE: And so did Anthropic, by the way.

JUSTIN: And so did Anthropic, yeah. So, you have this mother chatbot that can do all these things that's owned by a single company. And, obviously, if it's like Microsoft, everything will be tied together in a single chatbot. And it gives my little security heart nightmares to think of that because if you crack that one, you got everything. But the thing is, can you just imagine, if you were a Microsoft shop and you had the Microsoft chatbot, whatever copilot they deem it be, it's going to be so hard to move off Microsoft because everything will be tied together.

KYLE: Yeah. You're at their mercy.

WILL: I think that's their hope. But I actually don't think it's going to go that way. I actually think a consistently well-designed AI-engineered sort of system is going to be predictable in a way that the coupling is going to be very portable from one chatbot to another. Because, yeah, Microsoft they're going to make their lock-in chatbot, but then Google is going to make their unlocking chatbot. And I’ve seen this thing is like the kind of thing that an LLM is going to be very good at porting over and translating from one to the other.

MIKE: I've got a case study at Acima. Not going to go into all the details, but we happen to have integration with an LLM because it's running part of our business. I’m not going to go too much into the details of the business [laughter]. But there's been some really important things we've built that use some of that AI tech, and we built a generic interface that none of our clients, internal clients have any idea which company is powering it behind the scenes.

Nobody is integrating directly with the provider, with, you know, Microsoft, Google, Anthropic. They integrate with our internal client. It makes you have to do a little more work, right? Now you have to build this in between, but because we have that, we could switch providers just like that. We'd have to probably rewrite some queries and do some validation, but none of the clients would have to change a thing. You absolutely can build your infrastructure. You can build with good architecture. If you always put that layer there to decouple, now you are not committed to that one provider, and you can swap it out.

WILL: I would say even if you're running on vibes, if you're running on pure vibes, say, okay, the LLM task is, okay, let's say it's my messaging queue. Okay, I'm an AWS guy, and I want to go out, you know, my...what is it? SNS queue. That's my event queue. And it's like, “Hey, LLM, write me a facade over this thing so that I have a uniform, not AWS-specific interface.” And that task is a task that an LLM, you know, coding buddy can take on and win.

MIKE: That's true.

WILL: Like, sophisticated refactoring? Maybe not for them. Because we're just moving data payloads around. This data goes in this pipe, you know, format the data for this pipe, format the data for that pipe. It's a translation. It's a translation problem scheme, you know, that you're coming through. The LLMs are really great at translating stuff. Really great.

I see the vibe, but I also see the win for, you know, empowering the vibes because you could define this ecosystem. It's not everything under the sun. It's our stuff that we fully understand, and we've exhaustively tested. I kind of like it. I'm kind of into this, you know, for just hammering out web apps? Oof.

MIKE: But that still led to good...I mean, you're doing vibe coding on an LLM, cool idea, but you're still architecting the system at a high level to be, you know...

WILL: Decoupled.

MIKE: What’s that? I thought I heard a word there.

WILL: Decoupled

MIKE: Decoupled. Exactly. Decoupled.

WILL: I mean, no, no, the vibe guy is on the beach with a pina colada. He's paying me to do that. Now you've transcended the vibes. The vibes are not enough. Like, now you need to elevate your game to somebody who actually does something. That's fine. You go prospecting, put your shovel down. If you strike gold, then you call me. Just be like, “I would like to keep this mine operating.” Why go to a [inaudible 40:32]? And I’ll be like, “Yes sir. Right away, sir. Make the check out to cash that.”

MIKE: So, you're saying the same thing we've been saying all along, though. You start small. You are willing to do things differently than when you hit that scale, and you should, and you should.

WILL: I love it, man. I love it. I see clearly the distance, my dream of the golden 30, where I could just get, like, 30 hours a week, just coming in, getting my stuff done, bang, bang, bang, being the overseer to a team of AIs. They're doing stuff, but somebody at some point in the chain needs to know what the hell is going on. That'll be me. God help a junior developer, you know, in the year of our Lord 2025. But I'm not that guy. I love it, man.

JUSTIN: My kids are all going through high school. Well, no. I got two kids in college and two kids in high school. And the high schooler, the youngest one, he is going through math class, and he has LLM helping him out. I mean, he basically really dislikes the teacher because he doesn't learn well from her for whatever reason. But he comes home, pops it in ChatGPT. ChatGPT solves it, and he's like, “Okay, really teach me how to do this.” ChatGPT goes through each step, teaches him half a dozen ways to figure it out. And then, he knows he learned better from ChatGPT than from that teacher. And my other kids have had similar experiences.

So, it's like, I'm kind of optimistic about the junior developer because they could go in, ask ChatGPT, “Explain this code to me, like I'm 5. Explain it like I'm 10. Explain it like I'm 5.” And they gain a deeper understanding of that code than a senior engineer perhaps would have the patience to teach them. So, I don't think it's all bad for the junior developer, other than the fact that the senior developer is suddenly going to be, like, 10 times more productive, and there may not be work for the junior developer. But for them learning, it’s --

WILL: Yeah. Well, I mean, that is effectively it. If I get more productive, then all the low-hanging fruit is gone. And the kinds of problems that I've worked with day to day, and the kinds of problems that I don't really see going away, LLMs are going to struggle because they're going to hurt. They’re going to hurt on the kinds of issues...because it's never just a repo or just a codebase or, like, how does this work together? It's just weird crap, you know? And that is --

JUSTIN: They may struggle now, but we're still early days.

MIKE: We are.

WILL: Yeah. You know, who knows?

MIKE: But, well, we talked about this a few weeks ago, that software engineers have become more productive consistently for at least 60 years, and the industry has only grown. The need for people who can think clearly about problems, and I'm going to say explain those, so be able to understand the problem that needs to get solved and understand the solution at a high level, is not going away. People who think clearly about problems are going to be needed, I think, for the foreseeable future.

Now, the tools that they use have been changing for all of 60-plus years, and I expect will change maybe dramatically with these AI tools. But businesses have never said...I've never heard of a business saying, “I'm making a bunch of money,” and then said, “Stop. Okay, that's enough. That’s enough. I'm making enough money. I don't need to do this anymore.” If we become more productive, so is every other company out there, and that competitive landscape just raises the stakes. And they're going to need smart people to drive the business further, or else they go out of business.

WILL: It's true.

MIKE: How is this any different than it was 10 years ago, or 20 years ago, or 50 years ago? You still have a competitive landscape, and now your competitor is using these tools, then they're way more effective. Well, you better be doing the same thing, and you better be hiring some experts who know how to do that.

WILL: That's true. I think it's true that we may be under...so, the technical sophistication that you need to quote, unquote, “vibe code,” it's like, oh yeah, I'm a senior software engineer, holy shit. Vibe coding is going to get rid of all of our jobs. And I'm like, yeah, but let's get a senior product manager and see whether the vibes are still flowing so freely, you know, not exactly the same level of vibes. I think Mike put it precisely. Our productivity has been going up and up and up and up and up decade by decade by decade by decade.. This is nothing new. And if I double my productivity, you are going to want three times as much output.

[laughter]

KYLE: The one thing that was pointed out to me on this was, you know, when I was coming up through school, it was, “Oh, you can't use your calculator. You have to do all this without a calculator.” And then, you start really thinking about that and how many places you'd never have a job where they say, “You can't use a calculator.” I mean, that's just ridiculous.

And to kind of go with what Will is pointing out there, it depends on your calculator as to how good you are. Some of the stuff that I've seen people do with those old TI-82s, or whatever, like, it was brilliant. I wasn't able to do, you know, some of the things and some of the equations and saving some of that stuff, you know, that they were doing. I was clueless. And I think we're going to run into that same thing with juniors, you know, they're going to be clueless what to really do, even vibe coding. You're going to need a senior. You're going to need somebody, an architect that actually understands what's going on in order to actually be able to efficiently use the tool.

WILL: I actually think...so, this is a little bit of a tangent. I'm going to go old man shakes fist at the sky. But, like, generationally, I think the late ex, early millennial, that we may have actually peaked in terms of technical sophistication because you still had to do it. If you're a teenager, do you know what a C drive is? You know, God help me, do you know what slash vol is? You do not.

You don't know that because you don't know...why would you invest in this arcana? But it matters. If you want to be in the business, that matters to you in ways that you don't understand. My generation we were both exposed to everything hot off the presses, like, every good technical innovation, like, up until this LLM, you know, it's a bunch of Gen Xers. We made you. We made LLM for you, Rugrats. You're welcome. It's our final gift to you, you know?

[laughter]

We got it all right as it was. People my age, like, 45, you know, thereabouts, early to mid-forties, we're not a bunch of COBOL programmers. All that hot stuff that the kids are getting, we made it for you. We made this, not you guys. It's not rock and roll or fashion or whatever. There's no new slang. This is just us.

MIKE: We're on a tangent. I'm going to roll with it and tie it back.

WILL: Yeah, I get it.

MIKE: So, I think there's something you said. Those of us who fit in that category, especially maybe the higher end of that category [laughs] in terms of age, you may also notice that there aren't so many car guys out there anymore. There used to be all these car guys. I'm saying guys. There are women. I'm using this generically. But there was a stereotype --

WILL: They exist. But you counted --

MIKE: They exist. But gender norms of the era [laughs]. There were a lot of these people who were car people, and it seemed like they were all over the place, and they’re becoming a rare breed. Because you don't need to have that expertise to keep your car running, because cars are better than they used to be, and they're becoming more technology-driven. So, you bring it to somebody who's an expert in that. And so, that generation is going. We hit peak car guy, right?

WILL: Oh yeah. Like way back ‘80s, maybe ‘70s, OBD2, baby. Just plug it in. Listen, tell me where it hurts, baby [laughs].

MIKE: So, you know, I think that there's something there. I mean, we've seen that happen at that other technology. We had this other technology where it's not like cars have gone away, but the person who can take care of everything end to end they don't exist much anymore. They don't even really exist. Those people work on classic cars. They don't work on the new cars anymore. And they might even shake their fist to the, you know, the sky around the new cars because the technology has moved on. It doesn’t mean the technology is worse. But yeah, I think that there's something to be said for that, that the end-to-end person is going to probably fall out of existence.

To bring it back to our topic, that build versus buy, you know, our expertise becomes specialized. And you've got to build the thing that actually provides value, and everything else you buy or you're working with another system. You build on that existing system and recognize that's how this works. You can't build everything yourself, and that's okay. In a complex civilization [chuckles], you wouldn't want to build everything yourself because there's a huge loss that you incur by saying you're going to do everything yourself.

WILL: I'm not smelting aluminum, man. I just want my Coke.

MIKE: Exactly [laughs]. Exactly. There are times you're literally part of the world that you should build. You should build your art and buy the rest, or use open source when you're at the appropriate level of scale.

WILL: Yeah. Get some samples. Get you a drum machine, some turntables.

KYLE: I think as we've hit some of these newer technologies, and kind of like you're saying with the car analogy, it's the same thing is happening in software where you can't have one...I guess things are more complex than the generalized do you buy, build, or open source? Because, in the platform world, we go between what's the appropriate option? And you got to look at all these things, too, where, oh, this is a great buy. But then we've got to employ the person that maintains that software for us. And then, you've also got to tack on, like, so that's been around, but the newer portion of this is infrastructure as code.

Well, this might be the top of the industry, but the next one down may look like a better option because, well, this one has infrastructure as code support. And maybe we don't want somebody that, you know, their entire job is to actually click ops this entire solution. We want somebody that's going to go through and automate all of this.

And so, now it's like options that maybe you wouldn't have taken advantage of prior you're taking advantage of because of other variables like infrastructure as code, or something like that. Because now you've got to weigh the costs of the offering, the costs of what it's going to cost to maintain, and then the cost of the code behind it. And those weren't things we worried about before. Like, you didn't worry about putting infrastructure as code there. That's, what, 10, 15 years new?

WILL: Yeah. They all want more. I mean, like, the past, you know, all the infrastructure, all of our technical infrastructure advancement, I think it's maybe thirds between better user experience, developer productivity, and increased performance, and stuff like that. But I'll be the first to tell you that, in my opinion, the past 20 years of Moore's law speed ups have all been consumed by shitty JavaScript.

[laughter]

KYLE: That's so bad, but it's true.

MIKE: You know, what's being consumed with now is kids asking ChatGPT to generate cat images.

WILL: God bless them. They're generating all kinds of images, all kinds.

MIKE: All kinds. That is true. And we're going to keep going, not on that topic [laughter].

WILL: Nope. Nope. You know, yeah, whatever comes out of the other end of your little creative writing essay about what you're imagining between you and God and OpenAI presumably.

MIKE: [laughs] I think we're coming to a good closing point here.

WILL: [laughs] I think we’ve shot way past a good closing point.

MIKE: Yeah [laughs]. We did. We went on this tangent where we started talking about the details of specific tech, but it comes down to these same trade-offs. It comes down to these same trade-offs. Hopefully, we've given you some things to think about [chuckles] as you're making these trade-offs. Where are you at? Where are you at in this cycle of your business? What trade-offs should you make, and what can you do? And Justin's question was well placed. What can you do to be prepared for the trade-offs you will inevitably need to be making in the future?

WILL: Absolutely. I think, in the end, I have to call it, like, an evolutionary process in that, whatever you're building, the initial step, I think if you're doing it right, if you're doing it the smart way, is whatever is the least resource impactful. If you are broke, if your budget is zero, you're going to slap something half-ass together, you know, that works and call it a day. If you've got a big enterprise, then get a hundred bucks a month subscription to something. Your boss will sign off on it. It's going to be fine.

And then, you take it as far as it'll go, and, at some point, the real art is going to be seeing that inflection point where you need to take a fork in the road. And either what you're doing is, in the gross majority of cases, something, you know, not essential to the business. Just buy something to take it off your plate. Or, you know, occasionally, you'll find something where it's like, this is a competitive advantage, and this is special sauce that only I could produce. Then you'll get there. And the art is identifying that fork in the road and not just getting pulled along, you know, whatever path you initially chose through inertia. That's really the art of it, in my view.

MIKE: That's perfect, I think where we'll leave things, and we'll see you next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, host Mike and the team—Will, Matt, Kyle, and first-time guest Vignesh—explore the topic of testing in production, using robotics and AI as a compelling entry point. Mike explains how reinforcement learning, while powerful, is limited in robotics due to the high cost and risk of physical failure. However, recent advancements in simulation environments have opened doors for safer learning before deploying robots into the real world. This analogy sets the stage for the broader discussion: the balance between risk and necessity in testing software systems directly in production environments.

The conversation shifts to real-world engineering practices, with Will sharing how his team manages mobile app rollouts using feature flags and staged deployments to control risk. Vignesh and Matt echo the sentiment, highlighting scenarios where pre-production environments can’t replicate real-world conditions, especially when working with third-party vendors, such as payment processors, where true testing isn’t possible until go-live. The team agrees that in these cases, observability, logging, and fast rollback mechanisms are critical. Production testing, when done responsibly, can create long-term resiliency in systems and processes, especially when it’s augmented with metrics, monitoring, and small, incremental changes.

By the end, the group reflects on the importance of ownership and accountability. They distinguish between testing and validation, noting that while testing should ideally be done before production, validation in production is inevitable and often essential. They also acknowledge the contextual nuance: startups may need to move fast, and students may operate in sandboxed environments, but in professional practice, taking testing seriously is a non-negotiable part of responsible development. The episode concludes with a reminder that thoughtful, well-instrumented production testing can be a powerful tool—if you approach it with caution, strategy, and care.

Transcript

 MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting again this week. With me, as usual, I have Will Archer. I've also got Matt, and Kyle, and Vignesh. Vignesh, is this your first time joining us?

VIGNESH: Yeah, this is the first time.

MIKE: Great. So, Vignesh comes over from the Rent-A-Center side, and we work with him a lot. He's a deep technology expert, been in tech for a long time, and he is a solutions architect. Should provide a lot of expertise and expect to be seeing him more in the future. So, great to have you with us, Vignesh.

VIGNESH: Thank you.

MIKE: Today, as usual, I'd like to introduce our topic by going kind of sideways and talking about something that will get us there. There's an interesting thing that we've seen in the world of AI. We've seen lots of text generation, right? If you haven't noticed, you've probably been living under a rock, but [laughs] there have been some really popular language models over the last couple of years that have made a huge impact. It's been a big thing.

And there's been other AI achievements, audio, you know, you can talk to your smart speaker, and it actually understands you. There's been progress in image recognition, image generation. But you know what there has been very limited progress in? Is improving robotics. There's been some, but it's been fairly limited, and there's a good reason for that. And it's really very simple.

If you want your robot to learn something, it's probably going to destroy itself while it does it. If you've got a robot that's strong enough like an industrial robot that can go and pick up a car, you probably don't want to say, “Hey, just industrial robot, go try and learn some stuff and see what happens,” because it's going to pick up things that are not cars that you don't want picked up, like, maybe itself [laughs]. Or maybe it's going to throw said car, or maybe it's going to use its internal hydraulics to destroy itself. In fact, this is a pretty common problem. And when you've got a million-dollar robot, that's a very, very expensive test.

There's areas of machine learning, and one of them is reinforcement learning. Reinforcement learning is when you try stuff out, see if it works, and then adjust your policy to be a little closer to what works. And, over time, you learn better what works. It's worked really well for games, but it's been very limited in robotics. And it really comes down to it's just crazy expensive. If you've got robots, you don't want to go break them. Robots are expensive and testing them in production, you know, is the only way to do it.

The only way you can test your robot is with a live robot. Now, that's actually changing a little bit in recent years, like, year, you know [laughs], they've built better simulation environments. If you can build a simulation environment that's really, really close to, you know, the real world and you can build a simulated robot, let it go out there; let it go ahead and destroy things; it’s fine. It's not going to break itself because it's just happening in software. And then, if you have it learn a policy that's pretty good, then you bring it to the real world and let it finish. Now that it knows not to do a lot of horrible things, it will do some reasonable things, and it's just getting a little bit better at them.

So, reinforcement learning is, you know, a big deal. It can be extremely effective. And they're just now getting in robots because if you couldn't test it ahead of time, it's just too expensive. It's just too expensive. They do have some lower-priced robots, you know, some people have managed to do with drones because they're fairly cheap. You can use a small one, and you can put it in a padded room. You can do some things to help keep it from being destroyed, but it's just really expensive.

There's a lot that we can learn about that, about testing. There are times when you have to test in production. You got your robot, and you want it to learn there. It's the only place you can. You cannot really do it in the real world. But, man, it is so expensive that almost nobody does. They don't release it because, without a test environment, you just can't learn.

Today, we're going to talk about testing in production, and [laughs] when we talked about this topic, you know, Kyle said, “Testing in production?” I said, “Never [laughs]?” [laughs] And we got a chuckle out of it. The truth is you wish it could be never, but sometimes you've got that robot you've got to teach, right? There's sometimes when it's your only option.

WILL: I do it all the time. All the time.

MIKE: Do tell, do, tell.

WILL: Well, I mean, so one of the main things that I'm doing right now for my major electronics retailers I work on their mobile app. And so, the thing about our mobile app is the release cycle is really long, right? So, I go out. I package the thing up. I send the app out to, you know, the approval board, right? Like, Apple's decently fast these days, but it's still maybe a day or possibly two to get approved, right? Google usually you can get it through the same day. And then, it rolls out to the customers, and it rolls out to the customers whenever their phones feel like downloading my update, right?

So, it goes out over the course of, I would say, like, a week or so, like, to everyone to 100%. And so, commonly, what we will do is we will release new features behind a feature flag, right? Which we can sort of manually turn on, you know, in test devices. And then, we can turn it on for a few people, and then a few more people, and then everybody when we feel confident. In fact, even our regular rollouts of a new version of the app we'll release it to, like, 1%, make sure everything's okay.

Because the other thing about it is we are releasing software on a platform that we do not own. Your phone, that's your phone, and you have a greater or a lesser level of control on what happens with that phone. You know, Apple's pretty locked down. Android, anything could happen. We don't have control over what's going on in your phone. You could be out of disc space. You could be out of memory. Anything can happen.

And so, there's a certain level of uncertainty around any release that we execute. That means we may need to claw it back. It’s easy to release to a production server when it's your server, but it's not my server. And I can only be so confident until it actually runs and everything is good, and it's like, oh, is it good on a Samsung? Is it good on an LG? Is it good on an Apple? You know, because they all rolled their own operating system, and they all have their own problems from day to day. Sometimes we need to claw it back. That's just how it is.

VIGNESH: I don't think we’re ever able to say, “No.” Everything done not in production, everything done [inaudible 07:21] environment go to production. That's impossible. Sometimes you have a lot of vendors. Like, if you're own grown solution, you may have all flexibility, but when you are integrating with others, you have to somewhere, you know, take a payment provider. You never have a real card testing, right?

You'll always have a dummy card, dummy things. They’re just a mock-up of data. There's not even running through the true system. They just do a lot of mock-ups on the side. This is the test card. You’re getting this number. I get test card. You return this response, right? So, then you only have option is before even rolling out to everybody as customers, you have to do your validation when you deploy. So, you need to have that production testing for certain cases.

MATT: Yeah. You hit on something really important, right? You need to control your risk, and that's extremely important. So, feature flagging, great way because you can react extremely quickly, and if something goes awry, you hurry and flip feature the flag off, and you mitigate your losses.

I just recently went through something, and, in fact, we were talking about robots and AI, AI-related, where we had to test in production because we didn't have the data we needed to properly test in a pre-flight or dev environment. So, you do as much as you can to mitigate your risk, and you have to keep a really close eye on so you can pivot quickly. And I think that's really the most important thing in production testing is you need to watch it extremely closely and be able to react quickly in case anything goes wrong.

MIKE: And you also touched on something there. In all the cases we mentioned, you don't have full control. And [chuckles] that's the reality sometimes, right? Sometimes you're working with a third-party platform you don't control, or you don't sort of control the data source. You have outside vendors. Credit card processing is a great example, Vignesh. It's one recently I’ve seen done where, yeah, you just can't. You want to tokenize a card to put in a mobile wallet? Good luck doing that outside of production because there is no sandbox credit card processing network [laughs].

MATT: And, Large Language Models, you could test a hundred times, and then it just takes that one request, right? And the model gets confused. It doesn't know how to respond, and you're going to get something totally unexpected. And that's when you have to make adjustments as you go. And I don't think there's really, you know, at this point in time, a way around it, unless you collect, you know, millions and millions of rows of data to throw against it and train on.

WILL: I’ll also say it's not all bad. Because one of the things about this sort of, like, a heavily feature-flagged environment, when you build that stuff out, you could ratchet stuff up and ratchet stuff down. Like, an emergent property of those kinds of things is you have fine-grained troubleshooting techniques where it's like, oh, with this payment provider, they're not having a good day today. Boom, they're off, right? No service interruptions, no problem. It's just like, okay, those guys aren't getting any of the traffic right now.

Similarly, if I'm having a problem, you know, there's a feature on my mobile app; it's, like, some backend, downstream services just having a day, and I'm just like, well, not today. You know, we're going to turn this thing off. You get everything else, but, you know, this aspect of the app is disabled until we can get it back up. This is one of those things where if you find yourself on shift where somebody else's, you know, experience or a backend subsystem is going wrong, there's only so much you could do. And having, you know, a button, you know, a panic button that you could press is a really valuable tool to have. And it's something you could get for free by testing in production [chuckles].

MIKE: Well, that's really interesting. So, the risk mitigation that you have to do to do something as risky [chuckles] as testing in production, when you're doing it right, when you're not cowboying it, but when you're actually saying, “Well, this is something I have to do, so I'm going to build as much structure as I can to make this as safe as I can,” is not single use. Having that system of feature flags actually builds resiliency into your system that can protect you operationally. So, when third-party service X goes down, well, you've already developed the ability to deal with that.

WILL: Exactly.

MATT: And we said, how often do we want to test in production? And everyone says, “Never.” You can also argue always.

WILL: Right.

VIGNESH: I don't think never is the right answer [laughs].

MATT: Well, right. And I would argue you always do. Yes, you want to go through a QA process. You want to do developer testing. You want to throw as many scenarios at something as you can prior to releasing, but you always want to go test and validate in production, post-release, always.

WILL: Yeah, that's the final...like, I think, informally, everybody does that, where you could have done every single possible peer review, code review, you know, linting, QA process, but I always check it. It’s like, okay, we rolled it out. I'm just like, okay, I'm just going to run one.

[laughter]

VIGNESH: Unless, you know, the point you mentioned, you know, I have a percentage of rollout, rollout for 1% or 10% of customers, monitoring a log. So, if something goes wrong, then I roll back. But you have to sit through that time, also, hey, that rollout process. But most of the system doesn't build that way. I know that those rolling out, like, percentage-based it’s very few systems right now and few companies does that one. Definitely, you need to validate yours in production, not just say, “Hey, it's going to work it out.” And a lot of times, you know, that's not the true statement.

MATT: Right. And if you don't, there's a really big cost because bugs go unnoticed and unreported because customers may be running into them and just bouncing, and that whole time you're losing money. Whereas if you would have went and validated in production, caught it early, you would have saved the company and yourselves a lot of time, and money, and effort.

MIKE: So, the monitoring instrumentation, visibility is how you make it work is what I'm hearing. Kyle, you might have some opinions on that [laughs].

KYLE: Yeah, I was just sitting here thinking, so it kind of seems like we can't, or we shouldn't be doing any of this testing in production if we don't have the observability behind it, if we don't have monitoring, if we don't have the logging because you want to do metrics. Because it sounds like, you know, feature flags and canaries is what we're talking about, sorry, canary deployments is kind of what we're talking about and doing a subset of data. And we need to know how each of these different systems are performing.

So, more than any other environment, it sounds like it's more important just to be able to have that kind of information for these testing scenarios, especially with the idea of load testing, too, because you don't get a more realistic scale of your services than load testing with real user data.

VIGNESH: Yeah. I mean, nowadays, due to microservice model, a lot of this new tech stack, which we will [inaudible 15:27], even though it's our environment, but we don't technically own a lot of things, you know, behind the scenes. Observability is the key. Nowadays, you know, even before starting working, the first thinking you should come in mind, say, how do I monitor this one? You know, what is my criteria failover scenarios? But I don't say 100% time as Matt said it. We don't need to test in production, but there is certain areas you definitely need to validate in production. Otherwise, you know, you may end up risking, especially, like, payments or some of the decision engine, something you have to run a dry run to make sure, hey, it's working [inaudible 16:07] connection.

Sometimes I saw, you know, deployment people think it works, and if one person deployed an hour before, especially when you are deploying cross-team application deployment, that's even critical your application connections is correct. If you think, I did it; my application is working; there's no logs, and you went off, then other one is doing, gone. One time I saw, like, six hours was an issue. So, that's where the five minutes taking just validating either through observability, I mean, yeah. One way of fixing is...I said two ways, right? One, you do real production testing, other is your observability is key point.

MATT: Yeah. I mean, it's critical. And when I say 100% of the time, I do mean it if we're talking about changes in business logic. If I say, go make that border a different shade of blue, probably okay to not go do production testing. But if you're changing business logic, I think it's really important, especially when there's the possibility of a lot of edge cases that just aren't going to get caught in quality assurance testing and dev environments.

VIGNESH: For example, this Monday, we are going to release, not this Monday, the following Monday [inaudible 17:30]. We said we have to test a program. We don't want to just say, “Hey, all the services deployed working. The customer is going to validate it.” No, we want to validate before even saying, you know, it goes to calendar, we have to: the new features, new integration, new business logic. It's worth to test it than just assume it will work.

MATT: Yeah, absolutely.

KYLE: There's a subset of testing, right? As much as we would like to test everything, there's RUM, right? And we can't get RUM without actually giving it to the customer, to the user. And that's, you know, as much as any developer would love that they use the product the way that we intended them to, they're always going to use them outside those bounds, be it a large percent or a small percent. It's not going to be used quite as we expected or hoped some of those times.

MATT: Yeah, that's really hard to predict, too. I mean, user behavior, you know, psychology, and you can't always predict everything somebody's going to try to do.

MIKE: So, it's interesting. We started, and we're still talking about testing in production, and it's almost 100% come down to validation and observability, mitigating risk by making sure you're watching it, doing slow rollouts. There's process around this risky thing you need to do that you can use to make it work. It's a fascinating idea, what do I say, and we all arrive to the same place. You make this work by being as careful as you can, and you only do it when you have to, but in some ways, you're always doing it.

When can you avoid? When can you avoid testing in production? Because there's a lot of times you think, well, I have to. So, give me an example. I'm going to be a little bit vague here because I’m not going to go deep into the business. But sometimes there's third-party providers that have sensitive data. You think about, like, credit reporting agencies, for example. They have a contractual obligation that you will not give them test data because they don't want to have their data polluted.

So, if you try to test with them in production, you're violating your contract, and that's not a good situation, right [chuckles]? So, there's, you know, there's a, well, you cannot, and in that kind of case, well, yeah, I'm not going to test in production because I can't. I'm not allowed to. My contract says, “No,” so all I can get is observability.

But there are other cases where like, well, yeah, I could probably come up with a test for this, but it'd be easier just to roll it out. And this happens especially at small companies or new projects, things that are pretty small. You're like, well, yeah, I don't have to set up my test environment. I can probably just roll it out there. What should that boundary be? Are there times when you really can get away with just putting it out there and seeing what happens?

MATT: Yeah, I think so. Something you have to keep in mind is how sensitive is the data that you're dealing with, right? Are you dealing with people's financials, health information, things like that, where it's critical that it remains secure? And then, there's other things that you can just throw out there and see how it goes.

I think, you know, we went through a phase here, I don't know, 10 years back. It started roughly with the whole gig economy thing. I think that's a perfect example. Everyone, not everyone, but a lot of people were out there trying new ideas and things, trying to be fastest to market with this gig economy and just throwing a bunch of stuff at the wall to see what sticks. And you had to move fast, or you get left behind, and then you find your big players in that market, right?, DoorDash, perfect example, Uber Eats. You know, there were a million food delivery services, and the survivors are the ones that move fast to market and spread quickly.

MIKE: Well, did they get there by having lax testing processes though?

MATT: No, I don't know that's the case, but I think they were a little more loose with the features they were rolling out and liberal.

MIKE: So, there's a difference there, right? Say, “Well, I'm going to test this out. I don't know how this is going to work from a business use case. And I don't know that we need to architect this really well because we just need to trial it.” But you probably want that feature to work and not damage your customer relationship by always just having broken code.

MATT: Sure.

MIKE: So, there's some nuances there I'd like to tug at a little bit. So, I agree, you're a startup. Getting out to market is your life, right? You do not survive as a business unless you're trying stuff out and doing, just as you described, getting out products that aren't complete that maybe have some weirdness to them, right? But it's another thing to say, well, I'm just going to throw out garbage. And I’m not even going to bother testing, just throw out whatever, because that might not actually be faster.

MATT: Oh yeah, I mean, you always pay the price, right? With your debt. You're creating your relationships. And if anyone's just pushing out garbage, they probably shouldn't be in the business honestly. You always want your stuff to work. And I don't know that in my entire career I've ever actually pushed anything that wasn't at least manually tested. You know, prior to the days of TDD, we did that all the time, right? But you test your happy paths, maybe an edge case or two, but you don't test everything.

I worked at a startup and was kind of the lone wolf for quite some time just due to resources and funding, where the CEO would just come in my office and say, “Hey. I want to do this thing. Can you have it ready by tomorrow?” And I'm thinking, well, that'll take three months, four months to build, but I guess [laughter]. And you prototype something, and sometimes that prototype is good enough to get out in the wild and just see how people behave to it, right? It's not perfect. You're going to run into issues, likely bugs, but it's good enough to test the market. And that's kind of where I was going with that.

MIKE: And you hit some important stuff there, that you care about validating, right, about making sure your system works, and any responsible engineer is going to care about that. On the flip side, there are going to be business needs, whether that be internal business needs or external ones, right? Internal ones say, “If we don't get this out tomorrow, we lose a client.” External ones say, “You cannot contractually test this,” or “We don't have a sandbox. Sorry. We have to test this in production,” you know, the credit card network, for example, that force you to test in production. And that's the reality of it [chuckles]. That's the reality of it. There's sometimes you have to. But you talked about debt. Every time you cut corners, you make things a little harder for next time.

MATT: Absolutely. Absolutely. And that's the price you pay to move fast.

KYLE: I wonder, too, how much has this faster movement affected testing in production? Like, when you've got a release cycle that is, you know, once a month or every two weeks or whatever, how is your testing methodology in production changing compared to if you can release to production 10 times a day? You know, are you making...I would hope at least that we're shipping out code that's easier tested and doesn't need as much testing type thing, and there are going to be much smaller and less effective changes in that case. But are we less cautious because we can iterate so fast?

MIKE: One thing...we've talked about this before. The larger your release is, the more complex it is. And it very quickly becomes impossible to truly test every path through the code, just mathematically [chuckles].

KYLE: Too big of a matrix.

MATT: Yeah. I mean, that's why, you know, moving into the world of CI/CD is so important. You are mitigating so much risk versus waterfall style where you're releasing, you know, 20 features in a single release. You find a problem. Those problems sometimes are extremely hard to track down, whereas you're releasing a small incremental change, really easy, less impact, and you can react way more quickly.

MIKE: You did kind of hint at something there, Kyle, about becoming complacent. And you go, wow, I've tested these are always working, you know, I'm really good at pushing out these small releases. I always test them. Maybe I forget to even think about validating it. And you knew it was important, but you just got so good at your process that you started missing parts. And you start believing that it's going to save you every time.

MATT: Yeah. And that's dangerous because you don't think about downstream effects and impact, right? You think, oh, well, this small thing I did is isolated here. But that change could impact things further downstream and cause real problems. And we've all seen that before.

MIKE: Yeah. So, the rules: while continuous CI/CD is huge for reducing risk, it's just reducing risk. It doesn't eliminate risk. And all the practices we've talked about, particularly the validation and having all of that instrumentation, whatever it is you do to make sure that it works, which may just be going up and testing it, is vital. That never goes away.

MATT: Yeah. I mean, we can get close, but we're never going to be 100% of anything. It's like solving for pi, right? The farther you go, the closer you get, but you're never actually going to solve all those problems. And you run into that a lot with machine learning and artificial intelligence. And sometimes the closer you try to get to that 100%, the further off it pushes you because of the impact downstream that you just aren't seeing at the time of change.

MIKE: So, we've talked about some good practices, about sometimes when you need to get there, about times to, you know, where you can't. So, we haven't talked that much about when you don't. And we've talked about, you know, yes, you're always testing in production. You've always got users who are using your stuff differently than you expected. So, I’m going to ask a simple question.

So, I mentioned it before, you know, when do you test in production? Well, you don't, yeah, never. Of course, it’s not true, as we've talked about thus far. But where does that sentiment come from? Why would we have the gut reaction, well, no, you don't test in production? Why is that the case? And what are we actually trying to say?

MATT: I think the reaction comes from data. You're muddling your data. When you test in production, you're not using real data. Your reporting can get thrown off, things like that and, you know, the risks you talked about before, contractual stuff, compliance issues, those types of things. So, maybe we think about things a little differently, and we talk about testing versus validation, you know, actual testing. Your tests should be written. You're not, you know, you're trying to get as much coverage as you can with written tests, with quality assurance, developer and peer testing before you send it to production. Once it's there, you want to validate. Sometimes you can't with reliance on third parties that you can't control. And that's when you're kind of forced to really test in production versus just do validation.

MIKE: I'm going to go back to the industrial robot thought, right? Let's say you've got a million-dollar robot. If you had a reasonable simulation environment, would you test in the real world without testing in the simulation environment first?

MATT: Absolutely not.

MIKE: It would be just burning money.

MATT: Yeah, and irresponsible.

MIKE: Irresponsible.

KYLE: But there's also a cost to getting that simulation environment.

MIKE: Exactly [chuckles]. That simulation environment is really expensive, and there's companies that have been spending a decade or more working on building that sort of thing. Companies like Nvidia who have a vested interest in having more companies using their product are trying to provide an open environment for people to test in because it benefits them. And so, they put millions, a billion dollars, you know, just vast sums of money into developing that because it is so expensive. Individual companies are going to have a really hard time doing that on their own.

MATT: Yeah, and it's like we talked about with the debt you're creating by moving quickly, right? The cost over time may be much greater than the cost you front load with in setting up that environment. It depends on what you're dealing with. You have to do the math and say, is it worth it?

MIKE: That's a very conscious decision. You say you do the math [chuckles]. You don't just kind of roll with your gut and say, “Ah, it'll be fine,” right?

MATT: Yeah.

MIKE: You look at the value of what you might damage, your relationship with your customers, financial information, you know, loss of information, and, you know, these extremely...well, I'll say particularly financial information, right? I mean, it's just not something you ever tinker with. The risk is too high. It's irresponsible to do anything with your customers that is not extremely well-vetted. Or even more, what about medical hardware?

MATT: Right. When you're dealing with people's lives.

MIKE: Yeah. You know, you test that in every possible way that you can before you even consider going into production. And where the risk is high, where the cost of the risk is high...it's not just financial cost but however you define that cost because, you know, people's lives that's high cost. The cost is extremely high then you're careful. You're careful.

MATT: Yeah. Use good judgment. I mean, everything isn't black and white, and not everything is objective.

MIKE: To flip it around, we said, well, there's times you don't, and we've also talked about low risk. And this is, I think, maybe where testing in production gets its reputation from. You've got students in school. They're writing the code for their project. And a lot of them aren't going to have a test environment. The risks are so low, right [chuckles]? Do I pass the class or not? You know, do I pass this assignment or not? And the cost of building a test environment is high enough if it's a small project, right?

Rationally, they look at the risk-reward and say, you know what? My test environment is the TA [chuckles]. I will be testing against the TA, and if they can run the code, then that is passed. And that's a valid test environment in that scenario. I could argue against it, and say, well, students should be learning to write tests. Well, they should, but if the system doesn't incentivize it, then they're going to make the rational choice [chuckles], which is don't do the extra work because I have a tester. I have a test environment built. In fact, they probably built one in the class that's going to test this for me.

MATT: Yeah. And, in that way of thinking, you're putting the reliance on somebody else, you know? Just a little bit of advice for any listeners: have some ownership; have some accountability and try and do as much of this stuff yourself, and don't always rely on somebody else to cover you.

MIKE: It's a sandbox, right? School, or, you know, bootcamp or, you know, your own test work, you know, your own learning. It's all sort of sandbox. But in the real world, testing matters [laughs] and not ever relying on somebody else is one of the things...now, let me state that a little differently. We rely on each other heavily, and it's the only way you get software made. But not putting the expectation on somebody else where you could be reasonably be expected to do it is the way that you end up building good software and moving ahead in your career because you care.

MATT: Yes, and good relationships with those around you. In this case, don't ever think it's not my job because it is.

MIKE: And that may be a good way to tie this one up [chuckles]. It is your job. All of us, you know, we've talked a lot about the exceptions, but in the end, in almost all cases, it's your job to validate before it goes into production and after it's gone into production. You've got to do both. And owning that, knowing that you, you know, this is something that you built, and it's your, what do I say, stewardship, you know, it's something that you take personally, is what's going to lead to the craftspersonship, where you truly become an expert who can build great things.

MATT: It's my name on my commits.

MIKE: Yeah. Well, thank you for listening. Thank you for listening. We've covered some good stuff and not one of our longest sessions, but really pointed the critical importance of visibility, monitoring what you're doing, testing it as early as possible in the process, which may be in production in some cases but doing everything you can to mitigate that risk so that you treat your customers and their data with respect.

And until next time on the Acima Development Podcast.

In this episode of the Acima Development Podcast, host Mike kicks off the discussion with a personal anecdote about pulling weeds as a teenager—an activity he once dreaded, but later learned to appreciate when reframed through a lens of choice and agency. He uses this story to introduce the episode’s central theme: burnout. The panel—composed of Eddy, Ramses, Justin, Dave, Kyle, and Will—dives deep into what burnout looks like, how it manifests, and how much of it is tied to control, meaning, and the perception of work itself. Mike cites symptoms and causes from the Mayo Clinic, underscoring that burnout isn’t just about being tired—it’s about disconnection, loss of purpose, and lack of support or agency.

Will shares a vivid analogy comparing burnout to hitting the wall during a marathon—you can’t just push through it, your body simply stops. He recounts how after pushing himself to the edge, he realized that managing burnout is more like being a professional athlete than a machine: you have to track your metrics, set boundaries, and plan rest deliberately. The group explores how burnout can creep up slowly, especially in tech where overworking is often celebrated. From Eddy’s rapid climb from QA to development and Matt’s startup experiences to Dave’s Tony Robbins-inspired mindset shift, the panel underscores that burnout is both cultural and deeply personal. They emphasize that having passion is not enough—without boundaries and recovery, even love for the work can spiral into obsession and depletion.

Throughout the episode, several key tactics emerge for preventing or recovering from burnout: identifying your personal signs of overload (like screen time or poor sleep), setting and protecting boundaries, using journaling and self-reflection, and finding meaning beyond work. The group also touches on the importance of environment—having supportive leadership, fair recognition, and a healthy workplace culture all matter. They wrap with a reminder that being human at work means embracing agency, even in small “petty rebellions” like working outside or choosing a task you know you’ll succeed at to rebuild momentum. Ultimately, burnout isn’t about weakness—it’s about energy management, self-awareness, and choosing sustainability over suffering.

Transcript:

MIKE: Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. It seems like I've done a long streak here [laughs]. I've got a great panel with me here today. I've got Eddy. I've got Ramses. I've got Justin, Dave, Kyle, and Will Archer, as is usually with us, is here with us today as well.

And I'm going to jump right into a story, and I'm going to talk a little bit deeper about that to introduce our topic. This is something we’ve talked about before. Well, I'm going to start with a story, but then circle back. I'm going to talk about when I was a teenager [laughs]. I think I was 18, 19, late teens. And when I was a kid, it was my parents, but mostly my mom would tell me to go out and weed the flower bed. And I hated it. I hated the fact that I had to go out there and do what seemed like a totally pointless task. It doesn't look that much different. The flower bed is going to look messy. Anyway, we got rid of the flower bed that had most of the weeds because it always looked bad, and we just put a lawn there [laughs].

It was going to look bad anyway. And I'm just going to go stand up there in the sun for, you know, an hour, pull these weeds. They'll be back next week. It just drove me crazy. I hated it, and it ate at me. So, it was this thing that anytime I had to do it, it was just, ugh, again. I’d complain about it, as kids are prone to do [laughs]. And I was 18 or 19, and I got a job, and I was pulling weeds [laughs]. And that old feeling came back, and I was starting to feel all those feelings.

And I thought about a friend of mine, a friend of mine named Dan. He's great. He worked at a plant nursery, so he’d been working at a job where he pulled weeds most of the day, every day, and had done that for years [laughs]. And I thought, what would Dan do about this? And I realized he seemed to enjoy it. I thought, well, you know what? I'm sitting outside, and [laughs] there was this sunrise. I thought about this as the sun was coming up. You start early when you do landscaping kind of jobs because it gets hot, so [chuckles], you know, early in the day.

So, I’m sitting there, and I’m seeing the sun come out. Like, I'm watching the sun come up. I'm outside. I like being outside [laughs]. I am doing something that I actually enjoy. I'm making money. I'm getting paid decently. And the only reason that I'm not liking this is because I've built a habit of not liking this. What's wrong with just following the weeds and thinking about something else? And it was actually kind of a formative moment in my life. And I said, “What would Dan do? [laughs]” It was more than that because I started thinking about it. I could be enjoying this. Why am I not enjoying this? And I got to make a choice.

Now, I think there's a lot deeper picture to that. There was a reason I didn't enjoy it [chuckles] because the history leading up to it, I didn't have a choice in the matter. And anytime you lose any agency, that personal ability to make choices, it takes something from you [chuckles], and it had been taken. And I’m like, hey, I’m going to take it back [laughs], and I was able to in that time. And I try to think about that when I'm in those situations, and sometimes you can't, right? You can't just say, “I'm going to be happy today.” One of the best ways to not be happy is to say, “I'm going to be happy today [laughs].” It doesn't work. You can't choose that.

Keeping along with talking about weeds, it's like a garden [chuckles]. You cultivate a garden and help things grow. The plants grow on their own, right? But you can plant the seeds. You can pull the weeds. You can cultivate around them. You can fertilize, and you can put mulch down. You can do all kind, you know, to retain moisture, you can water it. There are all kinds of things you can do to provide a great environment for those plants to grow, and happiness is the same way. You can't say, “I'm going to be happy today.” In fact, it's really offensive usually to say to somebody, “Why don't you just smile?” Don't say that [laughs] because it's patronizing and diminishes somebody's real experience, and that's not going to fix the problem.

And I think sometimes people are trying to say, “Well, you know, kind of fake it till you make it.” There's some aspect of attitude that can change that, but you got to cultivate the garden, right? And if somebody takes away your ability to do that, then those weeds are going to come up, and there's not much you can do about it. But there are things that you can do, which brings us to our topic today.

About a year ago, we did a session on burnout, and we're coming back to that today. If you haven't been living under a rock, living in a cave for the last several months [chuckles] cut off from outside civilization, or maybe you've been up on the space station and just landed, there's been a couple of people like that [laughs], you may have noticed that there are a lot of things going on in the economy [chuckles], a lot of things going on in the economy, a lot of lost jobs, and particularly in software. When the interest rates went up, a lot of the big companies said, “Oh wow, I can invest money in other things besides my people, and it'll make money.” And they did.

And they said, “Oh, let's go of all those people.” And so, there's a flood of thousands, tens of thousands of people from big tech companies that have gone on the market, and that, obviously, affects the market. A lot of companies are also asking questions. We talked about this a few sessions ago. What's AI going to do? What are all other changes going to do? And it's hard to find a job. So, a lot of people don't feel like they should move, and you maybe shouldn't. There's changes we have to make. And a lot of us are in situations where maybe some people have left, and we're doing extra work.

So, revisiting burnout seems like a great topic today. It seems like a really good place to go. And I'll maybe put a little disclaimer here. There have actually been some really positive changes at Acima in the last month or so that have kind of lightened the mood. But sometimes things are, you know, in the real world, sometimes things are up, sometimes things are down. And I think we should really grapple with this.

What do you do? What do you do to address burnout when maybe you can't change jobs? Now, when we talked about this before, we talked about some big things we could do. Maybe we can reduce your hours. You can work with your manager. Particularly if you're a manager, maybe you can reduce some hours for people so that they have more flexibility with their schedule, right? And there are a lot of things about the things that you can do there.

Well, we're going to talk about it today from a different perspective. If you're not the manager, if you’re not the one who can make these kinds of choices, how do you make life work when you're threatening burnout? And I'm going to start. I've been talking a little bit because there's a lot to lead here with. I think it'll help the discussion.

I actually went to the Mayo Clinic, to their website. And they have a list of symptoms of burnout and possible causes. So, I'm just going to mention some of these because I think it's very effective. It's like the definition from a leading clinic. They say, “Do you question the value of your work? Do you drag yourself to work and have trouble getting started? Do you feel removed from your work and the people you work with? Have you lost patience with coworkers, customers, or clients? Do you doubt your skills or abilities? Are you using food, drugs, or alcohol to feel better or to numb how you feel?” And they mentioned some other things that can come out of there.

And then, they said possible causes. And I think this one's really interesting as well. “Lack of control. Lack of clarity about what's expected of you.” I could go back to that lack of control. I didn't get to choose where I pulled the weeds [laughs]. “Conflicts with others. Too much or too little to do. Lack of support. Problems with work-life balance.” And they've got more material as well.

But I think that sets the stage, you know, I want to define what burnout is. It's like, you're not happy anymore, but it's more than that. You tend to get aggravated. You don't care about it. You don't want to do your job anymore. Just overall, this negative affect that's caused by work. So, lots of preface, and one shiny example of a time that I got better with it in a relatively easy situation to do so. What are your thoughts about burnout? Feel free to tell stories, you know, about what you're dealing with.

Remember, not everybody here works at Acima [laughs]. So, don't take this as an indictment of our workplace because we'll tell stories from lots of places of business. But what do you do, and what can you do when you maybe don't have as much control as you'd like to have?

WILL: I've got a burnout story from a different perspective. So, in the before times, in the olden times, I had my own software company. I ran my own software company for 10 years, right? 10 years is a looong time to be in the game. And so, it is good and bad, right? Because, on one hand, like, you're going to feel some burnout. You're going to get that burnout. And, on the other hand, like, you have effectively unlimited options to do it. You could do whatever you want. You're going to pay for it, but you can do it. And, like I said, 10 years. You can have unhealthy work habits for a year or two or three even, if you're a real hard charger.

But at 10 years, you are going to have to be familiar with your limits. It's going to be like you're a professional marathon runner or a bike racer, whatever, where managing your energy and your endurance is a fundamental job skill. And so, where I got really familiar with burnout was when it was maybe a year in, right? And I had a giant feature that I got out. I don't even remember what I was trying to get out, honestly. But I remember I worked haaard, like, no offense, but you don't work that hard for somebody else's shop. If you did, you'd have a mild form of brain damage, and God bless you for it. But it was like, sleeping in the office, like, extreme grind mode.

And I shipped my thing, and I was like, I know I'm loopy. I know I'm messed up. And I know I got to get this thing because the next thing is coming, right? I didn't do that for fun, and the treadmill never stops. But I knew I needed to get my legs back. And so, what I did was, I was like, okay, I'm going to go in. I'm going to wake up when I feel like waking up, and I'm going to go into the office, and I'm going to work until I don't feel like working anymore, and I'm going to get out. Like, okay, you know, for, like, a week at least or maybe two, I'm going to do that.

Because I manage my productivity like a professional athlete, like, LeBron James ain't got nothing on me. I had a program, which is an endorsement, called RescueTime. And there are other ones. RescueTime is the one I use. And all it does is it tracks every single thing that you do on your laptop, like, spyware to the maximum, right? And I track my productive hours, and my back off week, my rest week, and my ultra hardcore sleeping, eating, drinking in the office, everything at my desk, exactly the same, exactly the same amount of productive hours, dead even.

MIKE: Wow.

WILL: And I think that's the nature of burnout and the insight that you really need to take from that is it is...and I keep on going back to this marathon analogy because there's a thing called a wall, where you bonk in a marathon where you have exhausted your muscles capacity to generate energy. And you cannot run. A marathon is like this sort of, like, your muscles get sugar and blood and break down fats and whatever. They break down energy, break down energy, break down energy. And you're replenishing energy, you know, your anaerobic state, and you basically want to ride this way. So, if you're crossing the finish line and, like, the gas tank is empty, right? Because when the gas tank is empty, you're done [laughs].

And that was exactly where I found myself. I was done. I hit that wall, and I was finished. But failure was not an option. Stopping was not an option. And you'll see the people running marathons, where they soil themselves in various ways, and they're staggering zombie-like because they're not done. And the reason burnout is...you have to manage it. You cannot muscle your way through it. If I can get anything into anybody's head about burnout, it's that you cannot muscle your way through it. You can't do it. And when you hit that wall, you need to make some kind of a change. There’s nobody in your office, in the office of Acima that was as motivated as I was at that time to get things done. And I just couldn't do it until I got right.

And I think because it's internal, it's a really seductive thought pattern to say, “Oh, I'm a bad person. Oh, I suck. Oh, you know, I lost it,” like, whatever, whatever, whatever because it all exists in your head. And it's not like I'm staggering my way to a finish line on a marathon, and my legs just won't move. So, anyway, that's maybe what I would offer in terms of my experience with burnout.

MIKE: That's fantastic. I love that analogy. I've actually read a lot about this [chuckles]. I've read a lot about this marathon. It's primarily in your liver, I believe. There's glycogen, like, you take glucose, and you stick a whole bunch of them together, and you just have this kind of glob of glucose gunk. It's a sugar that your body can reserve, and your body tends to keep about two hours of reserve. And if you're an absolutely elite marathon runner, you could do it in about two hours. And so, people get close to that finish line, and they hit that...it's gone, right?

WILL: Yeah.

MIKE: They're almost there, and they just...you're done because, literally, there is nothing there. And if you don't carefully replenish that reserve, there's nothing you can do. I do a lot of cycling. And I've experienced this very personally [laughs]. If I don't keep that chug sugar, right [laughs], and it's really sugar, if you don't keep that energy up, then you will hit that wall, and there is nothing you can do. And you will feel terrible. And it doesn't matter how much you want to go. You're not going to. You're not going to go any further. And very much you think, well, if I just push harder. No, that's not how it works. It's just not.

DAVE: It’s like trying to get out of bankruptcy or trying to prevent bankruptcy by spending more.

MIKE: [laughs]

DAVE: You are headed to bankruptcy because you were cashflow negative. And then, you’re white-knuckling through it. You’re just, like trying to pay more. It's like the old joke about Walmart, right? It's like, we're going to sell these for less than we paid for them, but we're going to make it up in volume. No, it doesn't work that way [laughs].

WILL: Yeah, precisely.

MIKE: And you also said that you do that self-judgment, right? There's a lot of self-judgment there. Well, if I were just more capable, if I were a better person, if I cared more, I would get there. But there's physics, right [laughs]? If I care enough, it doesn't make me fly.

EDDY: Talking from someone who came up from QA not knowing much about development, I spent the first year and a half diving in, like, trying to learn everything all at once, drinking from a fire hose. But it's fine. It's fine. It’s fine. It'll eventually click. I don't have to worry about it. I'll get to a point where I'm really, really good and I can slow down, right?

But the problem was I never knew when that slowdown was supposed to happen, right? Like, to me, it was always keep going, keep going, keep going. And oh, if I just learn this, if I just learn this, then I'll be good. Oh, but if I just learn this...and it was a constant hill over and over, and eventually, like, the gruesome hours finally caught up, right? So, I think part of burnout is really trying to understand what your limits are and take a break before you hit that point.

WILL: Absolutely. Absolutely. I am a big fan of metrics and tracking and stuff like that. Like, RescueTime, I got a lot out of because it was like, it’s been tracking everything I've done on the laptop for 10 years plus. It's got a big data set, and you could see where you go bad. And so much of this stuff, you know, if anybody's listening to this podcast, you're going to have to come up with your own blend of herbs and spices. There's going to be things that mess you up. There's going to be things that you can do, things you can't do. And a lot of it is just...what I hope we could get into is sort of small ball tactical things where it's like, do this; don't do that. You can get this big grab bag of things that you could try out to keep yourself on the ball.

Because it's going to be a personal recipe. What works for me won't work for you. What works for you won't work for me although it's going to rhyme. And there's probably stuff that if you can't lift directly from somebody else, you can certainly take inspiration from it. And it's also dynamic, right? That's the thing. Your brain is, I don't know, like, I anthropomorphize, the slack-off part of my brain, the resistance.

To take something from a really good book called “The War of Art,” that deals with working artists, people who have a day job and want to maintain creative pursuits in their lives, like, that concept of resistance is burnout by another name. And it will always come at you, and it will come at you in a thousand different ways. And the thing that worked today might not work tomorrow, might not work the next week. And your tools get dull from overuse, and you'll try. It's always going to be ups and downs, and you just sort of try and make sure the trend line is going up. But you're going to have bad days, you know, you're going to get beat.

MATT: A lot of this comes from culture. There are a number of us on this call who have been around a while, and I think most of us that have come from a world of early startups. That early startup mentality is a constant death march, and it kind of gets instilled in you. And, unfortunately, I still have that in me a lot because I spend a lot of hours outside of working hours doing things related to work because that's just who I am. However, it's not sustainable. It's absolutely not sustainable.

And I think if it comes from the top, that culture of work-life balance, ending death marches, which if you're applying modern software principles, you shouldn't be doing death marches, right? If you're Agile, if you're CI/CD, those types of things help prevent it, but it's still around a bit. And when you feel that pressure, as adults, we want to take the responsibility upon ourselves to see it through, and I think that's oftentimes where we fail.

And it's surprising to me, honestly, Will, that you said the amount of time of your productivity was the same because I find myself less productive. If I'm working 19, 20 hours a day versus a healthy I want to work 8 hours a day, I'm going to get more productivity out of that 8 hours every time.

WILL: I was highly motivated, like, nobody would work those hours if their name wasn't on the door.

MATT: Yeah, I've had a few of those for sure.

WILL: I’d be like, if one of my junior devs was working like that, I'd be like, “Hey.”

DAVE: Go home. I need you here tomorrow.

MATT: Yeah. Slow down.

WILL: Yeah. Go home [laughs].

MATT: And often, things that have motivated me to do these things it's that dangling carrot, right? The thought of equity and the payoff. And if this is successful, what am I getting out of it? And there's always going to be those types of motivations I think that force it. But how many of those things have actually paid off, right? I think in 20, I don't even know how many years anymore, 27 years doing this, twice it's paid off for me. And I–-

EDDY: I think part of that comes from pride in your work. If you're stamping the work that you're doing and it's going out in the wild, right? Like, you want to make sure that it's pristine; it's sustainable; and it's good. That was part of my problem dealing with burnout. It was like, dude, this project will have my name. I'll have the git blame. I’ll be the person who was involved in this project. I want to make sure it gets done correctly. It was my big hurdle.

MATT: Yeah. And I think a lot of that really boils down to trust, right? Trust your peers. Ask for help. Have the confidence to ask for help and reach out and say, “Look, you know what? I need a break. I've been going constantly on this, and I need someone else's eyes on this, and I need to slow down.” And that's really important to have a healthy work-life, right?

WILL: Yeah. It’s hard, too, because I suppose one of the worries is, like, people who have the personality traits that I would say are required to be a good developer, right? You need to have a belief in your work. You need to have self-motivation. You need to have conscientiousness. You need to have, if I'm going to be perfectly honest with you, maybe a little smidgen of anxiety around. Like, most of the really good developers I know they got a little anxious [inaudible 24:57] in them.

MATT: Everybody on this call with anxiety raise your hand, really. I bet you every single person on this call has it [laughs]. Absolutely. Absolutely.

WILL: This is a little bit of a tangent, a little bit of a tangent. I had a buddy who went through some personal stuff, and he went on anti-anxiety meds because he was going through it. He had some life challenges, and he couldn't [laughs] work anymore because without the anxiety, he [laughs] was just like, “I'm not doing this. This is stupid.”

MATT: I experienced the same thing, did the same thing.

WILL: And it’s --

MATT: You need it. It pushes you, right?

WILL: Sure. Sure. But at the same time, when you have those kinds of personality traits, when you find yourself in a burnout situation, what I think is so insidious...and it really depends strongly on leadership to lead, and when I say leadership, I don't mean managers of a certain level. I mean leadership which may or may not exist. If things aren't going good, as a general rule, when you hit burnout, like, things are not going well. And so, when you need these sort of coping strategies for burnout, it’s exactly when there's a lot of pressure on you to perform, if that makes sense.

Like, I didn't do my experiment on working not even half the hours and getting the same amount done until it shipped. I shipped first. This is the last time I'll say it, regular employees should not and will not, and I think, cannot work like that. You shouldn't do it, and you cannot do it long-term working for somebody else because there's no...if you're going to work this hard, man, just get a different job, you know. Just do something else.

DAVE: Yeah. If you're going to drive yourself that hard, you need to be owning your time, not renting it.

WILL: Exactly.

EDDY: I do want to take a moment because I feel like the standard or the norm has always just been 40 hours. I don't know when that was established. It's way beyond my time. But when I started working, it was --

MATT: It was before all our times [laughter] Henry Ford.

EDDY: Gotcha.

WILL: That was the productive amount of time that somebody could bolt a door on a Model T Ford. It's literally that. He had a...not a spreadsheet, right? He had a bunch of people with adding machines tacking up the optimal amount of productivity. And it wasn't 12 hours, and it wasn't 10 hours; it was 8 hours. And it was working the line at the Ford Plant in Dearborn, Michigan in the, what, 1920s. That's it.

EDDY: Wow, Today I Learned.

MIKE: People used to work on farms. It was 12 hours a day, 6, 7 [laughs] days a week, and it was a real shift. 40 was, like, wow, you're dropping that? It was research.

DAVE: From way back.

MIKE: That's what you can actually do effectively.

EDDY: Now, I wanted to just list out just a couple of things from countries that have gone away from that normality, and they have noticed no effect on productivity, right? Just to name a few, because it's kind of extensive, Netherlands, Denmark, Norway, Switzerland, Iceland, France, Belgium, et cetera. Like --

JUSTIN: You're just naming all of Europe, man [laughs].

EDDY: I guess.

MATT: Scandinavia anyway.

[laughter]

JUSTIN: I want to go to Europe now.

MIKE: [laughs]

EDDY: The average is about 30 hours a week, right? And they’re able to be --

MATT: Yeah. They're on a four-day work week generally in those areas.

EDDY: I'm wondering, just by naming these, I'm wondering how prevalent burnout actually is, right? Like, it would be a really awesome comparison. I don't know, but I’m willing --

JUSTIN: You mean burnout in Europe versus burnout here?

EDDY: Yeah. Like, that would be a really nice contrast, right? Just to see if there's a direct correlation between that.

MIKE: There’s an interesting --

WILL: Don't forget the salary difference between Europe and here. A senior dev in Europe is making like, I don't know, like, not a lot, 70, 80. That’s like, upper level.

EDDY: Sure. But, like, cost of living is also less, too, right? So, you're able to make a fraction of what you make in the U.S. living in another country, but you’re --

JUSTIN: It depends on the country.

MIKE: Yeah, it depends on the country.

MATT: And I think that goes into politics, which we do not want to discuss on this podcast.

[laughter]

MIKE: But there’s a contrast, too, because, in China, they have an opposite thing where people will die because they're working so many hours. Like, they've literally been awake too much, working so much their body just stops. And that is widely acclaimed. And I think sometimes there's looking down on, you know, they look negatively at people, like, you're only working 40 hours? And that is true with some businesses as well. And some people make it work. I think maybe a bigger question is, without going into politics, why would some people feel like that's what they need to do, and some people don't? What’s the difference?

WILL: Well, they call it --

MATT: Human conditions.

WILL: They call it 996, 9:00 a.m. to 9:00 at night, six days a week. That is the working thing. And I'll tell you exactly what it is. It's because, like, China's a dog fight. If you are in the top level of China like Beijing, or Shenzhen, or the big cities in China and you got a good job, like, you're living real good. However, most of the country is not like that. And there are a hundred poverty-stricken, rural people gunning for your job. And so, it is a pressure cooker, the likes of which we can't really fathom in the West.

MATT: It's also individual, right? Some people just aren't built for it. Some people are. My grandfather worked full-time until he was 92 years old and died. He didn't have to. He owned the company. The company's been around since the early 1900s, successful. He owned it, but he was at work every day until he was 92 because that's just who he was. And some people are just like that, like, it's what defines them. Is that right? I don't know. I think that's subjective, but --

MIKE: Well, you both just hit on a couple of things. If you have to do it, that sure provides a lot of motivation, right? That lights a fire beneath you. Like, if I don't do this, I lose my job, and I'm going to be out on the street. It gives you a lot of motivation to work hard. Likewise, if it's your own business and you love it, you want to do it, right? Having intrinsic motivation makes a huge difference, and that comes to what feels like work versus what doesn't. Now, it might still feel like work if you have to do it, but you do that to feed yourself, feed your family. You do that because it matters. And it kind of comes down to that, what is what you're valuing and if it aligns with your values, then it sure works a lot better.

EDDY: Well, I do want to reemphasize that it is habitual. Like, my dad recently retired, right? And he was working 6 days a week, you know, 6:00 in the morning to 6:00 p.m., sometimes 7:00. That's also because he took pride in his work. But he did that for years, right, over 80% of his life that he did work in before he retired. But even so, after he retired, he started getting his pension. He was still getting up at the same time he was. He was still doing things around the house. And people told him, “Dude, you're done. Relax. Wind down for a little bit. Take a walk. Sit down on the couch. Be on your phone. Be lazy. You deserve it.” And he's like, “Oh yeah, no, it just feels really weird to just not have to do this anymore.”

[crosstalk 34:10]

JUSTIN: I do have to say one thing about that. A lot of people their work defines their life, and if they were to retire, they have no life outside. Like, they don't have friends. They don't have hobbies. They don't have things they like to do. Maybe they got divorced because of work. All those things can really make you think that, hey, work is my life. And, I don't know, and personally, I think that's pretty unhealthy [laughs]. So, if you look in terms of enjoyment of life and happiness of life, if that was your goal, you got to make sure you cultivate those relationships that are outside of work.

MATT: Yeah. I think...and it's just personality types, right? And I'm by no means...I am not saying I'm a workhorse, anything like that, but my entire life I've just always wanted to be the best at everything I can do: athletics, school, and my job. I want to do the best possible job I can do in anything I do because, at the end, I feel really good about it. Sometimes it's really hard getting there, but it's worth it to me.

I was a wrestler. Anyone who wrestled knows how hard of a sport that is. I played six different sports in high school, wrestling by far the hardest sport there is, training every day for hours upon end for three minutes to get through three minutes just so you can be the best at it. It's that kind of thing, and maybe it was just my family. I came from a family who...my older brothers were the same way. So, I think I was just kind of raised on that, and I saw them doing it. So, I wanted to be that way, too.

But also, not everyone's that way. I have friends who they couldn't care less. They're just like, I'm going to go do what I need to do. I'm going to get it done, and then I'm just going to go about my business and walk away. And I, for one, struggle with that. I think about work when I lay in bed. I think about work when I wake up early in the morning. And I don't necessarily think it's a bad thing, but you're right, it's what defines a lot of people. And my wife calls me out on it often. She's like, “Okay, let it go.” I say, “Okay, I'll try.” But it just doesn't go, you know.

EDDY: You know, I used to have that mindset growing up. Like, I was always competitive, always wanted to be the best. But I found out really quickly that there was always going to be someone better than me. I think it's just inevitable. You can be the GOAT of a player, but you're going to run into someone who's just naturally better than you, or they went at it harder than you did.

MATT: There’ll always be.

EDDY: You’re always going to find someone that’s better.

MATT: There's always someone better.

EDDY: Oh, 100%.

MATT: And there's always going to be, you know. The whole LeBron, Michael Jordan thing, right? Objectively, Michael Jordan is better; we all know this. But, eventually, there's going to be someone else who comes along who is better.

EDDY: Yeah, 100%.

MATT: And who changes the game like Jordan did.

EDDY: 100%

MATT: Wilt Chamberlain did it. Bill Russell did it, Dr. J, Magic Johnson, and then, Jordan came along. And there's always going to be that next person. But, for me, as long as I know that I'm trying my best, that's enough for me. I don't actually have to be the very best person at something. I just want to know that I'm giving it everything I can give, and I just feel good about it.

MIKE: Well, I think you're touching on something there. Not everybody has got the exact same personality type, but we all seek for meaning. We want to have what we're doing mean something. And if work is the main thing we do, going to what Justin was saying, you know, if it becomes everything that defines our life, well, that defines our meaning. It's critical, and this goes to the burnout to find meaning. And it probably means that you're going to have to find some meaning in more than one thing in your life.

MATT: Absolutely.

MIKE: Because one thing is never going to be enough. Think about retirement. It's very common; you've probably heard about this. The people who retire and do nothing they don't tend to last very long. It doesn't work. If you want to be happy and healthy in retirement, you do something else, right? You have that other thing that gives your life meaning. Because if your goal is to do absolutely nothing and have no meaning in your life, well, now you have nothing to live for and nothing you care about. And that's not what anybody wants. Finding meaning in your life, and we've talked about tactics.

Having something else, I think, is absolutely vital. And I know, Matt, you have things outside of work that you care about, too, and if you have those...and maybe there's two things you care about at work. People switch careers, right? But there has to be...you have to define yourself as being valuable outside of just what you accomplish at work. Not that it's not incredibly rewarding to accomplish something at work; it absolutely is. And you can accomplish things elsewhere, too. Your life can be more than that, more than just that. You have to find the places to find meaning.

So, I think if there's one thing...and this goes back to right at the beginning of the call. You have to find personal value out of what you're doing. I am digging a ditch, and [chuckles] it will accomplish something, right? And that is providing meaning to your life. And maybe you've got hobbies. Maybe you've got a family you're doing it for. Maybe you're trying to help your elderly parents in China. So, you're working as much as you can because you care about them. That meaning, I think, is the first thing. Find what you care about and work on that.

People say, “Find the thing you're passionate about, and that's the work you should do.” That's a lot of times misconstrued. Find the thing you're passionate about that you're willing to do stuff that sucks [chuckles]. Like, it doesn't mean you're always going to enjoy it, but you're passionate enough about it that you're willing to do the things you don't enjoy because you find meaning and reward out of it. Oh, if you find that meaning first...and our values are different, right? But if you can find the thing that you value and be working for that, that can help you get into alignment. So, I think that's tactic number one. And we kind of danced around it, but it came at it in several different ways. That meaning is central. I think that's item number one.

WILL: Okay, but expand on that a little bit more because I'll be perfectly honest with you. Like, I have plenty of meaning outside of my work, but that's not helping me. Knowing what it is I'd rather be doing than sitting here [laughter] debugging this preview access token while I'm recording [chuckles] this podcast...

MIKE: [laughs]

WILL: If you see me with my head over, that's not making it...I mean, like, knowing that I would rather go and play with my kids is not...that's not making the medicine go down any smoother, where it's just like, I'd rather be playing Elden Ring.

MIKE: [laughs]

WILL: I'd rather be playing my guitar. I'd rather be X, Y, or Z. It’s like --

MIKE: Well, you mentioned your kids.

WILL: Sure.

MIKE: I'm guessing that some of your income supports those children.

WILL: Baby, without the kids, I would live in a storage container and be happy as a clam [laughs].

MIKE: So, you just answered your own question, right? You said, I am doing this because I care about something. I am earning this income for more than just myself. I have this work that I'm doing because I care about the money because I care where that money goes. It's investing in these people that I love.

WILL: Well, I think that's true. I think that's true, and I actually think...there was another thing that I wanted to bring up, which was...there was a quote from Marissa Mayer, formerly of YouTube and then of Yahoo. And she got drugged through the streets, pilloried for this comment. But in my opinion, she was right. And what she said, effectively, was, I don't believe in burnout. People are not doing the things that they want in life, like, what they want to do in life.

If they want to have dinner with their kids and they can't have dinner with their kids because they're working, or they want to go for a run, or work out, or go on a date with their wife, whatever it is, you're either doing something that you think is stupid, or you're being blocked from doing something that fulfills you. And if you're meeting those goals, right, obviously, you need to maintain your physical and mental health.

Another analogy is...so, I play music. I’m a musician. The standard day in a recording studio, if you're like, “I want to buy, you know what I mean, I want the day rate, right, I want to buy [inaudible 44:13]. Give me the day rate for this recording studio.” The standard day in a recording studio is 12 hours, and that's 12 of the hardest hours you've ever seen anybody work in your entire...When you get done in the studio, you are wrung out. It's hard work. It's 12 hours a day.

Again, people at a regular office job they don't work like that because they don't have that motivation. And you're not going to have that motivation in identifying that stuff is not...it will help you improve your productivity by identifying and protecting it. If you could say, I need this, and you make those boundaries and you guard them assiduously, then you can work hard. If you know what I need to do to stay even and I protect that, the rest of your day will be more effective.

JUSTIN: Going back to what Marissa said, it’s like, if you aren't liking what you're doing, try to change it. And you may have to do the ditch work for a while to get to where you want to be. But, I don't know, it's kind of funny because I used to laugh at life coaches because I'm like, oh, those guys, you know, who needs help from somebody outside themselves to figure out what they want to do. But there is actually some value in there in terms of, I don't know, there's probably a lot of value in it, but this is what I think of now.

If you can identify what you really want to do and then have somebody walk you through the steps to get there from where you are currently, that is worth something. And so, if you're just digging ditches right now and you're not happy, try to figure out what you want to do to be happy. And, hopefully, somebody will pay you for it, and if they won't, figure out what you can do, how much money you need before you can go and do it, you know, whatever. Because if you aren't happy right now, generally, it's not just...it's hard for you to be happy and for the people around you to be happy, too, because that affects how you interact with others and everything.

DAVE: When I was in my 20s, I had to work crap jobs because I was in my 20s, and that's what you get when you’re in your 20s. You work crap jobs. And I hated it. And I was working at Walmart, like, at the electronics counter. And it's an honorable profession, but it wasn't the career that I wanted, and I hated it. And I realized I need to figure out what I want to be when I grow up. I need to figure out a direction for my life. And I'm like, okay, yeah, I love computers. I love programming. I've been doing it forever. I should try and get paid doing this.

And overnight, my job at Walmart went from being this crappy, drag your butt in thing to I'm paying for my education. And it became valuable and meaningful. Being able to do a bait and switch on your brain and basically say, “Hey, this thing that you hate, here's a reason to love it,” you know, it's super-duper powerful to see that when you can.

There's a trick that I learned in my teens. I want to say it was Tony Robbins. Google can fact-check me on this. Tony Robbins’ problem-solving questions. Like, the choices you make about the things that you are dealing with entirely, not dramatically, entirely frame the attitude and opinion that you'll have towards that thing. And so, if you find yourself really, really stuck...now, Tony Robbins is big in NLP, so he likes using specific language to minimize or maximize certain things.

So, you can tell in these questions that he's definitely trying to make you think that the problem is smaller. Just choose to see the problem as smaller and manageable. But the five questions and I can do them really quickly for you, number one is “What's great about this problem?” If you've got a problem, like, if production is down, if that is a problem for you, that means that when production is up, you're making money, so that's great. And the more panicked you are about production being down is probably because the more profitable you were when it was up, right? So, there's actually a great side to it.

Then the next one, and this is Robins’s minimizing language, “What's not perfect about this?” And then, the next three questions are just “What am I willing to do to have it the way I want it?” “What am I willing to stop doing to have it the way I want it?” And then, the magic one, “And what can I do to make it fun while I deal with the thing that I don't like?” And being able to carry that in your back pocket is so powerful because you can literally take, you know, management says, “You have to do this. Here's your death march.” And you're like, okay, I'm going to figure out how to turn this into my choice, into my career, my life, my job, my cathedral.

Mike, you mentioned digging ditches. When I was a kid, I hated digging ditches because I was digging ditches. My dad he didn't love digging ditches, but he wasn't digging ditches. He was investing in the landscaping on the property. He was putting in sprinklers. He was, in his head, enjoying not having to go out and move the sprinkler every day when he was watering the lawn every time he shoved that shovel in. Where I'm just like, man, this sucks. I don't like it. And he had meaning, and I didn't.

EDDY: I do want to just say though, that one thing that's really helped me not get burnout I have obsession disorder a lot of the time. So, when I gravitate to something, I get obsessed, right? And I'm like, I need to know everything I can about whatever it is, and quickly. A passion can quickly become obsession, and you got to know what that limit is before it gets out of hand. For me, I live by that. You got to make sure you know that your passion is not an obsession.

MIKE: There's a word that Will said as he was speaking. He said boundaries. And I'd like to lean into that because we've talked about finding meaning and we've talked about that for a while. Yeah, that matters. What's the next step? You've got to put up a boundary somewhere. If you think that this is what I care about and these are the things I don't care about, well, what are you going to do to say this is where the things I don't care about end and the things I do care about stop? Now you might have to do some of those. There might be some things. But if you don't say, “This is where that stops,” then it's going to eat everything. Whether it's be something that you think is cool and you go down a research rabbit hole and then your whole evening's gone [chuckles], and you don't get to do the thing that you wanted to do.

Or there is the end of your workday, and you just want to finish this project, and so you keep going till it's done. The next thing you know it's seven o'clock, and you're still working on it. You have to draw a line. There has to be a boundary. And if what you care about is that your life is sustainable, and then you reach...and going back to what Will said about it’s the exact same number of hours. You reach the end of the day. This is one thing that works for me. I reach the end of my workday, turn off my computer, and I walk away. Now, maybe before bed because there are some important things, and I'm going to check my messages. And if I get a message from my boss a couple of hours later because something bad's happening, yes, I'll look at that.

But overall, I've got this boundary, right? This is the end of my workday, and I’m going to work hard during that workday [inaudible 52:18] work on something else. Same thing in the morning. I've got the things I care about. I'm going to work on that. And now I'm going to set that aside, and I'm going to go start my workday, and there's an edge. Figuring out where those boundaries are, I think is...and my perception and it's kind of come up from the group as well is we talked about tactics. That is number two. Find boundaries and hold them because if you don't, then you're not in control anymore.

KYLE: So, I've thought about this a little bit as we've been talking. One thing that I've done, and maybe this is a bit of a pessimistic approach, but I will look around and see how much effort I'm putting into something. I'll look around and see how much effort co-workers are putting in. They might be putting in more; they might be putting in less. I think a lot of it, for me, and what will contribute for burnout is not even necessarily how many hours I'm doing. Because at one company or one project, I might be willing to do 60 hours, but another one, it's only about 40.

And a lot of that comes down to is I'm looking and the person that's doing less than me is getting just as much recognition, or the person that's doing more than me is getting just as much recognition. Well, the value, to me, is not that much higher if it's not higher to the company. If the company isn't going to turn around and recognize people, why put in that extra effort and cause that much more burnout to yourself? I feel like that's something I've had to weigh personally.

WILL: I think it's a really interesting thought, right? Because so much of this has been, like, I think implicitly and unconsciously about, like, it's on us. It's on us, right? It's on us, our problem. This is our fault. This is our thing we need to deal with. But with the possible exception of me because I got myself in that. I got myself in that hole.

KYLE: Yeah, self-employed.

WILL: A whole lot of the burnout that we get exposed to it comes from external stuff. If we were some sort of utopian workplace where it's just like, “Everybody, just go out and do your best man. Just do your best,” like, I think that would work a lot better than maybe the business classes would think. But there is external stuff that's coming down that is imposing this burnout on us, and it's not in our power to fix it. If you got a bad manager, you got to fire your manager. That's your job. You have to fire your manager. Sorry. If your manager sucks, you got to fire him. It is incumbent on you and a capitalist free labor market to fire a crappy manager, and that company might have to go with them.

And so, how do you identify a situation where it can only be fixed by a hit in the bricks? Because nobody wants that. I don't think anybody really wants that. But as individual contributors, especially, the lower you are on the totem pole, the more you need to acknowledge that as that must be done. How do you identify a situation where you need to save yourself?

DAVE: Seth Godin wrote a book years back called “The Dip.” He talked about, basically, like, you're going along and then things start to get worse. So, your trend line of happiness is going down, and you have to make this choice. Is it a dip that's going to come back up, or is this the beginning of the cliff? And the interesting thing that he said was a lot of people walk around with this idea that winners never quit, and quitters never win. He says, that's not true. Winners quit all the time. Winners are very, very good at early, early, early identifying this is not a dip. This is a cliff. I'm out. And they bounce as soon as they know that they're throwing good money after bad or good time after bad.

And, for me, that, I think, is the...there's a fantastic question...man, I'm a bookworm today. Martin Seligman's “Authentic Happiness” he gives a fantastic question, which is, if you could do this...there's some tasks that you loathe doing. Ask yourself a question, if you did this perfectly, if you did it absolutely to the best standard of your ability, would you be happy with your work, or would you just feel relieved that it's done? And if it's the latter, you'll never find your happiness there.

Stop trying to get better at the things you're bad at and start investing in the things that you're good at. And, for me, identifying where the dip is I will often look at it and go, if I hit the bricks, will I regret going? You’re in the building. You're trending down. You're like, should I...I don't want to give up what I have. And so, you have to flip it around and say, “If I don't leave, will I regret staying?” And that, for me, can frequently help identify, yeah, this is a dip. I want this to get better, not just for the relief, but because I really want the outcome from this. I really want to achieve what I'm doing here.

WILL: I don't know, man. I don't know. I suppose what I'd say is if I'm being really honest about the arc of my career, I have been pretty darn successful, and I'm really proud of the work that I've done. But at least 80% of it, if not higher, it is just I have a high capacity for frustration, like, I got a high tolerance. And I'll sit down with something for a period of time that is not reasonable. And over the arc of my career, I think by most metrics I've been pretty darn successful. But I don't have any illusions about...like, I'm clever, but I'm no genius. But I will bang my head against that wall until it falls down. And it's been good to me, but it ain't no fun. It ain't never been fun. And it's not going to be fun tomorrow.

EDDY: Yeah, Will, but if you bang enough times, you'll end up with a concussion, then what will you do?

WILL: You should have seen the other guy.

EDDY: [laughs]

DAVE: My plan is to keep doing this until the wall gets a concussion. That's right [laughter].

WILL: It’s worked so far. I got a lot of holes in walls behind me.

KYLE: So on the flip side, I feel like I could definitely relate to some of those points that you're bringing up, Dave, specifically just I go through those trends. You know, when my job is something where I'm just like looking at the clock, waiting for the day to end, that's when I start thinking, oh man, should I be moving on? Should I be looking for something else? And, like, a lot of the time, I actually really enjoy what I'm doing.

And in a way, having a, “salary,” in air quotes, position where I can work more than 40 hours is somewhat nice because sometimes it's just like, okay, cool, I'm working on this. I need to get it done. I'm really enjoying myself, and all of a sudden, I've worked 2, 3, 4 extra hours, randomly, and it's one of those things like, oh yeah, this is cool. And I do feel like, at least in my career, I've gone through phases, right? And it is one of those things where it's just kind of like, ah, I'm waiting for the day to end, but maybe this is just a dip. Will things get better? And over my seven years at Acima, I will say it has had dips.

WILL: [laughs]

KYLE: We have been going through dips, but it has gotten better enough that I've insisted on staying. So, I really resonated with what you were saying.

WILL:  I don't know. I suppose, like, I think it takes different forms for different people. What I would say with my current gig, right, my current gig has been psychologically pretty challenging. So, for people who don't know, I'm a consultant and a fairly high-level consultant. I've been a fairly high-level consultant for going on four years at a very large electronics reseller that doesn't have the greatest technology stack.

And I'm in a psychologically very difficult position in that I have a lot of responsibility, but I'm also like the help, so there's not inconsiderable amount of pressure to be like, what have you done for me lately? But I'm also, like, othered, if it makes any sense. You get a lot of money, but your seat is never secure, and you better be conscious of that. I walked that line for a long time. But in terms of jobs, in terms of job unpleasantness, this is fairly high on the list, but I’m there for a reason that isn't, you know, this is a fun job. This is a good career job. I'm here because I'm here for my family. Flat out. And that's it. That's sort of external motivation.

And I think, bluntly, a lot of people are like that. Like, you can get a full remote well-paid middle to upper-middle-class job that affords your family a pretty nice lifestyle. But you need to be locked, in, and you need to make sure you stay locked in. Because I suppose, I think, this is probably true for you as well, but I can't just sit and wait for the clock to roll over. I have to be fully engaged.

Like, if I finish my work, boom, it's five o'clock somewhere, but it never works out like that. It never works out like that. But I have to be engaged so I'm not working till 10:00 at night. And that's why I worry so much about productivity because I'm not punching the clock. Like, I need to get my work done. If I don't deliver, I am done here. And so, I've got to lock in. I can't just run the meter. It doesn't work like that, I think, for any of us really.

DAVE: So, a lot of what you're saying, Will, is resonating with me. In the late noughties and the first half of the teens, I was running a contracting shop and consulting, freelancing all the time. And I had to drop and get a W2 because my boss was a slave driver. What I found out about myself is that if you hand me a death march, I will push back. Like, I don't care. I've insulted vice presidents in a meeting in front of the entire company and once it cost me my job, so I will mouth off if I need to.

And if somebody says, “You need to be in here 60 hours a week. I want you from here to here on Monday through Saturday and da, da da,” I'll be like, “No, if you've got a project you want done, tell me what you want done, and I'll do that.” And what I found is I will not do a minute of death march. But if you put a rabbit out where I can see it, I will chase that thing until I drop. I'll be up till three in the morning every single night running after that thing. And that's what I ran into when I was consulting was there was always one more rabbit. And so, I couldn't drop the sword at the end of the day.

WILL: Yep. Yep. I definitely identify with that.

DAVE: You were talking, Will, about shoveling up the crap parts of the job. When I was consulting, the thing that I found very quickly is when you are the only employee, everything is your job. I have literally worked on a project where in the morning I was adjusting pixels on a GUI for a Windows app, and that evening I was soldering chips to fix a defect in the hardware that that GUI controlled. And I was like, you call yourself full stack? Go below the server level. When you bust out a soldering iron, we can talk.

And I personally I love electronics, and I love the soldering, but I could easily see how this is just yet another piece of crap that I've got to do. And it's a great skill though, right? You end up being the guy on the team when they go, “There's this one piece that nobody knows how to do because it's completely weird and strange,” and very quickly, people around me are like, “Ask Dave. He's always about doing some weird thing that nobody else is looking at.” And it comes from that everything is my job if I'm the only one here.

WILL: Oh, I know. No, I am well aware. I'm well aware. I have made it through so many...I got five horses shot out from under me last year, like, managers. Like, my managers are gone five times. And here I sit [laughter] working for somebody. Somebody is just like, “No, no,” keep that one. I mean, I don't know. It's something [laughs]. It’s something.

MIKE: It feels like dancing around the same couple of things. If you're willing to put up with the misery because there's something you care about, then you'll keep doing it, and you'll probably be pretty successful. If you're the person who will chase down until you find the weird thing or will just put up with frustration...I've long felt that software development is mostly dealing with frustration, by the way [laughs]. If you're willing to put up with that frustration for a long time, you can be successful. But you've got to value enough to be willing to keep sitting in that seat and keep dealing with the frustration. I have known some people who are like, yeah, I'm getting tired. I'm going to go on social media, and it doesn't work.

You actually have to care about enough to be frustrated for a long time [laughs], but if you're willing to do that, that works. Then also don't let that glycogen run out, right [chuckles]? You have to have those boundaries. So, find the meaning. Remember that meaning. Put the boundaries where they need to be. Is there any other tactics that we should talk about? Because we keep on coming around the same couple, and maybe that's it. Maybe it's as simple as...I didn't say it was easy, but maybe it's as simple as those couple things. Are there any other high-level tactics, or even low-level, or just little things like your Pomodoro or whatever it is?

WILL: Yeah, here's a canary in the coal mine for me. Like, a canary in the coal mine where it's just like, oh, it's starting to creep on. Because I think burnout is a lot like hypoxia, like, where you're not getting enough oxygen in your brain, and you're getting loopy. You're getting sloppy. But it's very difficult to measure your cognition with your cognition, especially if it's starting to go down. Now, what are you going to feel when the grip is starting to slip? What are you going to start seeing? RescueTime is a good thing to get on your laptop if you can get it on your laptop, right?

You can get a corporate laptop now, and RescueTime is, I think, maybe with some merit, like, flagged as malware [laughs], so that's not possible. But look at your screen time on your phone. If that’s not enabled, you should be looking at it. Screen time starts to shoot up. If you can get RescueTime on your computer, you find yourself like, whoa, look at all this. Look at all this stuff that I'm doing on the laptop that's not work. That’s not so good.

Another one, real, real important one, if you are a thought worker, if you are working with your brain, you need to maintain your brain. And the number one with a bullet thing you need to be dealing with with your brain is your sleep. If you are not sleeping...seven hours is an absolute minimum. And so, what you'll see when you're starting to approach burnout is bedtime starts to become a challenge because you didn't get what you needed out of your day.

You need some kind of time to yourself. If I don't have an hour at the end of the day of Will Archer screw-around time to do just whatever dumb crap that I want to do, then I am going to steal it from my sleep. When I steal it from my sleep, then it affects my mood. It affects my flat-out intellectual capacity. Everything starts to go wrong. So, when you start messing with bedtime, think about your day, think about your burnout, think about what's going on, right? So yeah, screen time, like, the quality of your work. What are you doing? Once you find yourself zoning out, you really need to start digging into that.

And I'll give you one more because I want to give other people a chance to contribute to this. Well, how do you figure out what you need to do? I am a giant proponent of journaling, like, reviewing your day at the end of the day, reviewing your day at the beginning of the day. What do you want? What do you need? What are you grateful for? What did you miss? What did you not want to do? Like, where did you feel bad? And then, you got to write it out. You have to scribble it out with a stick in the dirt because that's how your brain processes things.

And, like, those three things, like, four or five things, I guess, if you manage those, that is the check engine soon light, and you can start to unpack what's going on and what you need to do to fix this.

DAVE: You just triggered something in my head, which is having a unit test for your brain, right? You need these things that are alert triggers that, like, if this happens, you're going to think it's a small thing, but you need to treat that like the oddly specific, highly significant thing. This is a ripple in the water, but you're out past the shoals. There shouldn't be any ripples in the water. You're going to run a ground if you don't pay attention to this that kind of –-

EDDY: You need a return statement.

DAVE: Mm-hmm. The moment I discovered unit tests for my brain were a thing was...and I won't tell the whole story here. But I thought it was a Wednesday, and it was Sunday. I was mad at my co-workers for not showing up on time because I needed to talk with people and, like, where is everybody? It's Wednesday. And they're like, “Dude, it's Sunday. They're not coming in today.” I'm like, oh yeah. It was time to change. Like, the timbers in the mind of my brain were creaking and groaning ominously.

But related to that, Ted Woodford runs a fantastic YouTube channel where he repairs guitars. He's awesome. But every day when he gets up to go to work, he plays chess online with a medium-level, the same medium-level AI opponent. And if the AI mops the floor with him, he's like, okay, I'm not firing on all cylinders. I'm going to pick the dull jobs that I don't need to be sharp for. There's a lot of buffing and polishing that needs to get done in the shop. I’ll do that today.

And then, he'll have days where he just absolutely trounces the AI, and he's excited. And he's like, okay, I'm actually sharp today. I think I'm going to pull down that $30,000 guitar and start cutting the inlay for the fretboard. And that's risky, dangerous work where you can cause a $10,000 mistake with just one blink of away from what you're doing.

And so, he actually measures his capacity every day and then tries to adapt his work to suit. We don't always have that freedom, but when you do or when you can learn to, hey, let's move this around, it's super-duper useful because everything just feels hard or easy. We never think, I am coming to this at something other than 100%. You can't...yeah, it's hard to just...sorry, last thought on that. When your judgment becomes impaired, the first judgment to become impaired is your ability to judge your impairment. That's the very first thing that goes.

EDDY: It sounds like drunk drivers, right?

DAVE: Mm-hmm. I will take sleeping pills at night, and then I'll go out in the garage and start working in the shop with power tools. And on the way out, I will tell my Amazon, “Set a timer for 60 minutes,” and where Liz is sitting, my wife, after an hour, the robot pops up and says, “Go tell Dave he's no longer authorized for power tools.”

And I'll be up to my eyeballs building a guitar trying to route out a truss rod pocket, and I'm lost in it. And I'm handling a router, which is one of the most dangerous machines, most easy to hurt yourself. And yeah, Liz will be like, “Honey, it's...” I’m like, “Oh yeah.” And I'm fine. I'm like, no I just want to just cut more. And I've had to learn when she comes out and says, even though I know I'm going to feel like I'm fine, I'm going to down tools anyway. And I still have all 10 fingers.

EDDY: You know what, Dave? There's a couple of times where my wife does come into my room, and she's like, “Hey. Are you done yet?” She’s like, “You know it’s X time. Do you still need to work?” And I look at her and I'm like, “Eh, you know what? No, I think I'm in a good spot,” right? And I just shut off. So, sometimes you do need that external influence to remind you, saying, “Hey, I think you're done [laughs].”

MIKE: So, we're talking about enforcement of boundaries. You can't see it. And then, I think Will brilliantly said, what's your canary in the coal mine? What can you do? Because yeah, your cognition is damaged, so it's not going to pick up. What can you use to measure it and then enforce that you do something to stop?

DAVE: It's the spinning top from Inception, I just realized.

WILL: I'd say a unit test that you can get for your brain, right? It's not the same as playing AI chess, right? Because I think that's great, but I'm just not that good at chess. But one thing that I do to measure is I really...so one of the things that I've found overall productivity-wise is I have a checklist to start your day. I start the day the same way every single day. Every single day, I start the day the same way, and I do the same thing, and I have a spreadsheet for how long it takes me to get through the steps. And the last step is I wrote a line of code, right? One line of code, code is down, whatever it is that starts your day.

Because once I'm in that flow state, I could cook. That's never really been my problem. Like, getting into flow state not a big deal. Getting into flow state hard, staying in the flow, riding that wave, baby, I'll ride all day. And so, write yourself a checklist. What do I need to do today? What's going on today? How am I going to do? I like to put on my instrumental music. I like to do a little stretching, whatever I got to do. And the last thing is write my first line of code. And how long does that take me? From when I sat down to where I got an atom of workout.

MIKE: So, a simple metric.

DAVE: What's your typical variance on that, Will? Do you have some days where it's five minutes and other days where it's three hours?

WILL: No, it's rarely three hours. I'd say it's probably between 10 minutes and 20 minutes because it is a routine. I do things. I plan my day. I reflect on what I'm trying to get done. And so, if I do a good job, 10 minutes, and I got something out. If it's a bad day, it could be 30 before I get something out. If it's a terrible day, honestly, a terrible day, like, it never happens at all, you know, if it's that bad, if I'm really feeling it, yeah, man, sometimes we just never get out of the gate. I've had zero days before. I really don't like them, and they're pretty uncommon.

DAVE: For me, it's always when the schedule gets moved externally. That's the thing that breaks me up. It is like, you're not going to get to write that line of code because you walked in the door, and there's an all-hands meeting going on right now because people dropped in this week, and yeah.

WILL: Mm-hmm. Mm-hmm. Yeah. You get pulled into something else. Yeah, almost always, almost always, like, a zero-day is because something else happened, and usually, it's something important, and I shouldn't count it as a zero. It's just not that. It's not what I wanted.

EDDY: To be fair, if I never type git in my terminal, I count [laughs] that as a non-productive day.

WILL: Well, what do you guys think about things to get out of it, right? How do you get out? How do you get out? Because okay, so I want to set the scenario, right, you've been working really hard, and things are messed up. Like, work is messed up. No blue skies on the horizon, right? But you know that things have gone bad. You're burned out. And I'm not writing you a hall pass to say, “Well, I shifted so now I can take a breather,” because a lot of times when burnout is happening, you're bumped deadline, and your burnout is bumping another deadline. It's like dominoes. And nothing is going to change unless you sort of get right. What are you going to do?

MIKE: I think there's a critical thing with this. You didn't get there in a day, and you're probably not going to get out in one either. Going back to your marathon running, if you hit that wall, it's not going to go away in 10 minutes [chuckles], probably marathon, you're out. Maybe you're going to crawl to that finish line, but you're not going to be effective anymore that day. It's going to take you a while to get out. Part of the reason to catch it early. But I think you have to have a plan that is not looking at the next day, not the next week. What am I doing over the next month? What am I doing over the next year to right this ship? Because if you take short-term fixes, little patches, it's not going to do the job.

EDDY: Well, have you guys seen that picture, like, that meme online where the miner is going through the coal mine, and he's like going at it, going at it, going at it? And then, he's an inch away from finding the gold, and he just gives up and says, “Ah, man, I'm done. This is fruitless. I'm not going to get to where I need to go.” And you just give up. I feel like a lot of people feel like if I just push a little bit more, I'll find that nugget. And I feel like that's like that dangling carrot we were talking about earlier.

WILL: I hate that meme. That's a stupid meme [laughter].

DAVE: Because the miner has no information. It could be a mile of rock. Yeah.

WILL: It’s a stupid thing. That's dumb, and it's destructive. It makes me low-key angry.

EDDY: [laughs]

MIKE: Gold is extremely rare.

EDDY: I was going to say, like, it is a very valuable point.

MIKE: Gold is rare, and you don't just go digging away in rocks and expect to find that nugget. You do research on geology. You do the research and figure out what is going to lead to this vein. And you follow that research, and you use the patterns in the rock. If you're not doing whatever...and I'm not an expert in this, but there's people who are, and you use that expertise. Like you said, there's no information. Unless you're using that information, unless you know that there's that gold there, you're doing the wrong thing. You're just randomly digging a hole. You're doing it wrong.

DAVE: Wow. I just realized when we're death marching, when somebody comes in and says, “Go do this,” it feels like go dig in the mine with no direction, no guidance, no understanding of where we're headed to. Yep.

WILL: I remember, man, I remember, like, what do you call it? It was last weekend. It was last weekend. I had a really gnarly...and we had some features that were backed up, and I had to do a bunch of git surgery because our main branch wasn't going live. But we had some critical bug fixes that need to roll out. And so, I was doing brain surgery with git. It really sucked. We've all done it. It's not fun [laughter]. And it was, like, midnight on a Friday because it just had to get out, and I got it all done, and it was fixed. And I had to reset the branch, and I had to do with git force push, but because I was so tired and I was burned out, I smoked my branch with the force push, and I lost everything.

MIKE: Oh man.

EDDY: There's people on the team that I've heard that have protection against their commits because they want to be really, really sure that it's intentional and that they're going to force push changes.

WILL: That's why I make the big bucks because I got a steady hand of like, you know [laughter], you want to do home surgery on prod? Okay [laughs].

DAVE: Do you know what the ohnosecond is? It's the smallest measurable unit of time. It's the amount of time from when you made a colossal mistake to the moment you realize it's too late [laughter]. For me, it was backing up a database to do some work on it. And while I was backing up on it, I screwed something up, and I dropped to the console, and I was tired. And instead of restoring the database, I deleted it. And this was on Linux, so there was no un-delete. And I literally deleted the production database, and the only thing that saved us was we had a backup from the previous night.

But I literally had to walk through the building, apologizing to the staff saying, “I threw away all your work this morning. I'm so sorry.” Fortunately, it was only, like, two hours’ worth of work. I think they were impressed that I actually came out and apologized, that I owned it, but still, just oh no.

WILL: Yep. I’ve done it.

EDDY: Good thing you had a snapshot because holy cow [laughs].

WILL: Yeah, I smoked a prod database back in the day. It was just me. There was nobody else. It was only me, and I did it. I did it. And it was taking too long, and we were like, oh, why is this taking so long? This was just a simple [laughter]...this was a simple process. And I'm like, [laughter]. They're, like, diligently going line by line and just like, nope, nope. Delete, Delete. Delete. Delete. And I'm like, ahh.

DAVE: Yep. Ow.

WILL: We got it all back. But oh my God, yeah, the [inaudible 01:24:01] factor was really high. Oh, those were times.

DAVE: I don't want to say this while Mike is on the call, but I will say that there are other teams in the building that do not have master branch protection. And the first two weeks that somebody from engineering was working on the data team, there's three commits to master right in a row of, oops, oh no, revert fix. Revert, revert. Yeah. And then, I went back to story branches.

EDDY: [laughs]

WILL: Yep, it is. Got to be done, man. I know [inaudible 01:24:40] do.

DAVE: Do you know what I did though? Again, part of taking control of your environment, I literally rewrote my bash prompt to print the current branch that I'm currently on, and if it's master or main, it's bold white on red. It looks like a screaming claxon error message. I cannot be on the main branch without my bash prompt yelling at me for every command that I type, and that has saved me.

EDDY: There's something to be said about having protection though, on your main branch, right? If you're allowing destructive commands to be making all the way up to your main branch, maybe that's just a symptom of a bigger issue.

MIKE: Well, and I would say --

WILL: I didn't smoke prod, okay? Like, I just screwed up my work. Like, I lost a day's work.

EDDY: No, Will, you were just making sure that the company had a snapshot of the database. You were just testing in prod.

WILL: It wasn’t in prod. It was a release. I had to cherry-pick the release in [laughter], and then there was a whole step. It's a big [laughter] bureaucratic organization. There are guardrails on stuff. But yeah...go ahead. Go ahead. Go ahead.

MIKE: I was going to say, this is actually kind of an example that brings this all in. It kind of summarizes what we've been talking about. And you're tired; you're up late. You don't have the boundaries in place. Bad things happen, and so you learn from it. And the next time around [chuckles], you set up those boundaries to protect yourself. You set up those indicators to catch you early so that you don't get into that trouble.

And remember that if you're up at midnight fixing this, that you're at risk, that right from the beginning, you're at risk, and you're going to do a whole lot better if you maintain yourself so that you're sharp when you go in because once you veer out of that, you're at greater and greater risk of not making things better, but making things worse.

WILL: Yeah, absolutely.

EDDY: My indicator is fog brain. Like, the moment I can't even think straight anymore, like, I'm done.

WILL: Yeah. And I'll leave you with...maybe my last comment is the thing that you can do, like, this is one of my go-to's, when I find myself a little burnt, I like petty rebellion. So, I'm going to get up. Like, I got this feature. And it's going pretty good, but I'm going to get out of my house. Just get up, just change something. It's a quick hit. It's not going to self [SP], like, burnout, but do something different. Do something novel. Make something change. Make something that you're not supposed to do. You're still working but some naughty rebellion, like, some kind of small thing you can do that isn't normal. Get up and go for a walk sort of, like, tiny bits of agency, right? Where you're not completely coloring within the lines.

And I understand that it's like, oh, we are saying like, how can I trick my brain into being productive in this corporate job that is draining the soul from my body [laughter]? Like, I am familiar with the irony in that, but. But nevertheless, one thing to keep in your back pocket is just petty rebellions. Small little things where it's like, we're working outside today. Like, I'm working somewhere else. And I'll tell you where after we stop recording.

[laughter]

MIKE: So, you’re saying, seize your agency; be human. Recognize you're not a machine.

WILL: Yeah. Be human. Be human. Do something not corporate.

EDDY: You know what actually helped my burnout sometimes is working on stuff that I know I'll be productive in, right? Like, for example, if I'm stuck trying to figure out a solution, a bunch of that is imposter syndrome that kind of hits in, and that fuels the fire of my burnout. But if I just pivot, work on something that I know is a simple task and I can get it done quickly, it sort of rejuvenates my fire. And I'm like, oh no, you know what? I'm not dumb. It's just a new concept that's getting the best of me. So, shifting into another task also helps a lot.

DAVE: I call that getting a win, and I think I've said this to Mike in a one-on-one. Literally, I’m just like, dude, I need to get a win right now. I've had three rough ones in a row. I really need to just get a win.

WILL: Absolutely.

MIKE: Well, let's end there. Do something different. Just search your humanity. Go for the walk before you feel like burning down the building [laughter]. Do something that gets you right. Set boundaries. Find meaning. And to Will's point, you may not love the job. You may just be working for the business, but that doesn't mean that you can't find meaning. And it doesn't mean it's not worth getting your head right because maybe you're working for some cute, little kid with an axolotl hat on. Until next time on the Acima Development podcast.

In this episode of the Acima Development Podcast, host Mike and a panel of returning guests—Eddy, Dave, Kyle, and Ramses—engage in a reflective conversation about the importance of engineers and technical professionals understanding the broader business context of their work. Mike opens with metaphorical stories about differing perspectives on the same task and how having a broader view—like knowing you’re building a home, not just hammering wood—can significantly impact the decisions one makes. He compares this awareness to navigating without landmarks, emphasizing that understanding the “map” or business direction helps avoid disorientation in both literal and professional contexts.

The conversation pivots to real-life examples from the panelists’ careers, including Dave’s story about a freelance project where a critical bug caused a dangerous hardware malfunction, reminding him that technical choices can have serious, even life-or-death, consequences. The group discusses how engineers often work in silos, focusing on technical excellence while missing business priorities like revenue, customer needs, and risk management. Mike shares a story of how strategic engineering choices—like creating permanent ad spaces on news sites—prolonged a startup’s life. The discussion illustrates that understanding how a company makes money, or what problems are most pressing, empowers developers to make impactful contributions rather than waste time on low-value tasks.

Throughout the episode, the panelists stress the value of communication between technical teams and business leadership. They discuss the dangers of neglecting foundational issues—using examples like collapsing buildings and broken gondola cables—to highlight the long-term risks of ignoring maintenance and risk mitigation in favor of short-term gains. Eddy and Kyle bring up the role of auxiliary teams (QA, DevOps, IT) in cost-saving and risk reduction, further reinforcing the theme that everyone in a company, not just engineers, benefits from understanding the business landscape. Ultimately, they urge technical professionals to pop their heads up, ask “why,” and align their work with what matters most to the organization.

Transcript:

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting today. I have with me a great panel. We've got Eddy, Dave, Kyle, and Ramses. They've all been here before, so, hopefully, you’ve heard their voices. And we're going to be digging into a topic.

Actually, there's a huge windstorm behind me. We've got a great editor. You probably won't hear it, but if you hear occasionally a weird sound, that's what's going on [laughs]. And let me start with some stories. And I'm not going to introduce the topic. I'd like not to introduce it because I think the stories often speak better than a title.

So, imagine three people, well, three people who are at a Habitat for Humanity site. We actually have... that's a charity that Upbound, our parent company, does a lot of work with. I can't remember how many houses they built last year. It was quite a few. Like, wow, they built that many houses [chuckles]?

DAVE: And Habitat is a charity that builds houses, right?

MIKE: Exactly. Exactly. They build houses for people who need houses. You know, imagine you walk in, you know, you're a reporter. You walk into a Habitat for Humanity site, and you find some people working. You know, yes, you walk up to the first guy and say, “Hey, what you doing?” He says, “Well, I'm nailing this two by four to the floor.” Fair enough, right? You’re like, okay, you know.

So, you go, and you talk to the second person like, “So, what are you working on?” She's like, “Well, I'm raising a wall here.” It's a little different perspective, right? They're working on the same job, but she has a little different perspective. She sees that [laughs] nailing something to the floor is part of a bigger project.

You go and you talk to the third person, you know, “What are you doing?” She says, “Well, I'm building a home for somebody in need.” Those three perspectives are very different. And the third person who, you know, this woman who sees the bigger picture, is going to make different choices because of her awareness of that broader situation. I can tell another story here. So, you know, put a pin in that one because I think it's really important. The choices that she's going to make are going to be affected by her understanding of the bigger picture.

Another story. So [chuckles], I lived in the Midwest for a long time [chuckles], and when I grew up as a kid, I was a landmark navigator. I learned to find places by looking at landmarks, and that actually works okay, right? That works okay, unless you get to a place where you can't see a landmark, and then you end up in a lot of trouble. I [chuckles] actually have, in my neighborhood, we got some friends that...a child who was walking around the neighborhood and couldn't see the water tower anymore, and they were completely lost. They were, like, 12 years old, you know, [chuckles]. They were the point where you think, well, really, they got lost?

Well, if you're used to navigating by looking at this prominent landmark and that goes out of sight, you have no point of reference. And so, you just wander around aimlessly, hoping you'll see the landmark again. It's not a great way [chuckles]. It's not [inaudible 03:29]. I've had to learn to think differently. In my head, I always know which way is north if I am going someplace I haven't been because, that way, I know I'm not going to get lost because I know where I'm at on the map. I know which way is north. I know that, you know, even if something goes off a different way, like, that will probably go the direction I need to go. And it makes a huge difference.

And I've become a much more successful navigator as I've learned that, to learn the, you know, the lay of the map before I even set out because it makes a big difference as to the choices I make. And I do a lot of cycling with my kids; I've mentioned that before. And, you know, we do exploring. Sometimes we go places we've never been before and knowing where we're going makes a big difference, especially, you know, if you have to reroute, you better know where you are. And always knowing which way is north, knowing where you are on the map makes a big difference. It's a huge deal because you don't get that situation where you're untethered. Well, where do I go from next? I have no idea because I can't see the landmark.

DAVE: I love that you said change the way you think, that you became aware that you had to change the way you think. I grew up in Southern Utah with, you know, Canyon red rocks. I grew up in Moab, which is right outside of Arches National Park, so all these gorgeous red rocks. So, I always know that when I'm looking, you know, at the Blue Mountains, I'm looking at the La Sal range and that's east. And so, I grew up with very thick, I mean, geologic landmarks. Nobody's going to move that building.

And then, when I was 19, I went to Puerto Rico, and I spent 7 months living in Ponce. And I've got these mountains, which is on the south side of the island, south coast. So, I've got ocean to the south of me and a mountain on the north. And I was absolutely fine, and I was there for about eight months. And then, I moved to Bayamón, which is on the north side of the island, and I was lost for a month, like, actively disoriented every single time, because the sun was coming up in the west, and I couldn't figure out why. It was, like, genuinely jarring. It took me a surprisingly long time to realize that I have a thinking error that has me 180 degrees turned around.

So, I love that you said you became aware that you had to change your thinking. I love that.

MIKE: And I did, and it was hard, to be honest. It took a while. It took a while. I'm old enough that I used to navigate with paper maps, you know, this is pre-internet navigation days. And you would go to the store, and you would buy a booklet, I say booklet, but it's sometimes more bookish than booklet-ish. And you want to go somewhere? Well, you look up the street name in the back, and it tells you a page number and a letter and a number. You turn to that page, and you find the grid, and within that grid, you find the place that you're looking for. And then, you expand out, and you might have to...and then each map, you know, you see the direction of each side. You had to figure out how to navigate [chuckles] maps that way. It was --

EDDY: But of course, Mike, you did that without driving, right?

MIKE: Yes.

EDDY: Like, you had your own copilot. Okay [inaudible 06:32] listeners.

MIKE: Before you drive.

EDDY: [laughs]

MIKE: You'd see a lot of people in those days with the booklet open [laughs], you know, like, on their knee. Where am I going?

DAVE: Or pulled over on the side of the road with the map unfolded on the steering wheel. Yep.

MIKE: Exactly [laughs]. You know, you’ll learn to do things differently when you have to find things in a different way. So, I'm going to talk about software. About, I don't know, 15 years ago, we'll say, more or less, approximate [laughs], I was working at a place. It was a startup, and there was a big shakeup in leadership, and we basically...we went for about a year where we had no leadership at all for our engineering department. So, what do we work on? And we had to make a choice. We had to make a choice.

And, you know, a lot of times we as engineers don't think about making that choice. We don't think about the broader business perspective, right? So, we think, well, what is going to make this startup successful? In the end, it wasn't successful, so maybe it didn't matter. But we actually prolonged the life of the startup by several years because of what we did. We were building a...it was a website builder.

We worked with publishers like newspapers. And we knew that they needed more flexibility in publishing, and we reworked, well, there are two things we did. We reworked the interface for publishing, which, you know, made that a little nicer. And, honestly, that was the worst choice we made, the worst of the choices we made, because yeah, it made it look better, but the functionality didn't change that much. And when you’re a company that's struggling [chuckles], then a little window dressing is probably not the right answer.

We also worked on a project that allowed you to put a permanent ad, allowed companies to put a permanent ad on a newspaper, which is actually a huge deal because newspapers are one of the most reliable sites in terms of reputation from browsers, from search engines, I'll say, from search engines, not browsers, but search engines. So, if you get linked to from the newspaper, well, that makes you look really good.

And so, what we’d do is we’d allow a business...and this is just taking the newspaper model and putting it online. We'd allow them to have a permanent ad, like a car dealership, you know, car dealership puts an ad in the newspaper. Well, now they have a permanent mini-site within the newspaper that they can go to that not only has their advertisement there but also links back to their website. So, now their website when you search for cars in that area, well, they're the first one that comes up because they're linked to from the local newspaper.

We built a whole system for them to build out those little mini sites and that carried the company for, like, years [laughs] because making that choice, again, we didn't have somebody telling us to do this. We had to make the choice based on feedback from the business. We had to suddenly put on our product hat. I learned a lot from that experience about how important it is to know what's going on in the business.

We could very easily have said, oh yeah, we're just going to make this perfect interface on the inside, make it pretty, which does nothing to make money for our customers. And that would have been very much the wrong choice. We did enough of it that, you know, that there was a cost, and we probably shouldn't have done it at all. Thinking about what we should have worked on made a life-or-death choice for the company.

And that's what we're going to be talking about today is, why we as engineers should know about the business. Why should engineers care about revenue? Why should engineers care about any of these things that the business usually is concerned about: prioritization, money, gathering requirements? I'm going to argue it is a big deal, and one of the key differentiators between engineers who are really successful and those who maybe end up lost quite a bit.

Dave, you've already run with this a little bit. Do you have any more thoughts or anybody else? You know, what are your thoughts about this idea of knowing about business, knowing about the business and its inner workings as an engineer?

DAVE: I have so many stories I could go into, and if you know me personally, you know that that's...I can often not be persuaded to not tell them. I'm going to try to stay concise. What seems to me is very essential is that you need to know, what do we need to succeed? And you want to close that feedback loop. And a lot of times, we just trust. We trust in the society, and the society in this case could be just the ecosystem at your business. I trust that my manager is doing the right thing. It might not be. They might not be.

I lost a job once because the accountant couldn't keep the books straight. You had one job, and that was to save mine [laughter]. And so, the reason I have a ton of these stories is I spent about 15 years as a freelancer, contractor, ran my own shop for a while. And we would go in, and we would fire jump Rails projects like firefighters that jump out of airplanes right in the mess.

I was the guy that you would call if you had a website built by somebody, you know, by your nephew who's, you know, a high school student or whatever. And it kind of works, but it's 90% there, and it doesn't...and it's inefficient, da, da, da. And so, I would jump in and then untangle this and make it work. And I usually...because I was a contractor, I often had an expiration date on my hiring time. I would go into a system like I'm already dead. I'm just going to work really hard while I'm alive, you know, fiscally alive or dead at this company.

And because of that time box and because it was...like, this company they've backed into the meat grinder a little bit, and they're like, “Oh, we have this website that we need to make money.” This is going to sound so obvious that it will sound stupid that I'm mentioning it right up until you realize if you look around the room, nobody is paying attention to this. And that is, if the company doesn't make money, they can't make payroll. If they can't make payroll, you don't have a job. It's absolutely critical that you get that sorted out.

I will append something to this, which is Uncle Dave's career hacks. The most powerful trick I've ever learned from my career was look at the person one level up of you and figure out what does that person need to look good and to succeed? I want to look good to my boss. The best way you can look good to your boss is to make your boss look good to their boss. And if you go high enough up the hierarchy, you're going to reach the C-suite. And, at that point, what's important to them is staying solvent, making money, right? There might be a mission. There might be, you know, noble goals and all that, but if there's no margin, there's no mission.

And so, figure out where the money is going, figure out where your business model is, and be prepared to embrace it and engage and be willing to step down if the boss says, “No, we're not doing that way because you're a technical person. You might not understand the business.” But you need to know the business exists and not shoot everyone in the foot with some bad code.

I'm not sure how perfectly this adapts. I don't know if anybody needs to hear this, but I will add this one bit to it. I worked with an engineer, about 20 years ago, who was amazing. He was a senior in college. And while he was working with us, he got out of college and graduated and [inaudible 13:57]. And this guy had a work ethic that would not quit. If you...well, no, he had a work ethic that would not quit from 8:00 to 5:00. And at 5:01, he was out the door, and he'd go home. But you could trust him to write 1,200 lines of code every single day. You could just about set your watch to this person, very, very smart.

In Dungeons & Dragons terms, he had a very high intelligence, not necessarily a high wisdom or not a very high experience level. He could solve any problem from first principles, but he had to solve every problem from first principles, if that makes sense. And I described him to a co-worker at one point saying, “If you tell him to go plow that field, he will go plow that field. You'll get up in the morning, and you go out and you look and that field will be plowed.” But if you accidentally told him, “Plow that field,” and you pointed at the interstate, you're going to come in tomorrow, and there's going to be a field plowed right through the interstate.

MIKE: [chuckles]

DAVE: And this is that. It's the same thing, right? You need to be an expert at what you're doing. But you also need to be paying attention to, you know, am I hammering a nail correctly, or am I building the house correctly?

MIKE: Yes. You know, I'm going to ask a follow-up question. What sort of choices...so you talked about, you know, plowing the wrong field [chuckles]. What are some of the choices you get into? So, this might take a moment to think about. What are some of these choices where it really matters to understand why you're doing what you're doing?

DAVE: Man, I don't know if I have good examples. I think I've told the story multiple times about the time I got a bug report that began with, “Fortunately no one was killed, but...” that happened.

MIKE: [chuckles]

DAVE: I was building a hardware controller that would shake things to vibrate them. I'll tell the story very, very quickly. It would shake things very, very quickly to do squeak and rattle. And Toyota had one of these machines of ours, and it's like a sound card. You play out waves, and you listen to the feedback from the accelerometer [inaudible 15:43], so you can tell how much you're shaking the car. And we had a bug that caused energy to store up in the system and then suddenly be just released all at once.

And we bent the frame of the car, like, we literally launched it. I say we launched it; this was a car. We launched it, like, up in the air, like, six inches and then over a foot and then back down, which was enough that everyone on the shop floor was wearing their brown trousers at the end of this event, right? Nobody got out of this without, like, holy crap, we could have been hurt, or we could have been killed by this.

Up to that moment, I was just designing data transfer. I had these beautiful acoustic wave patterns, and I knew all about frequency distributions, and they were elegant. And I had ways of generating random data that could then be massaged to fit a pattern but still be random, all this lovely, lovely stuff. And then, we got that bug report, and I just realized there are people's lives at stake. If I screw this up, somebody could die, and that sobered me very, very fast, and, interestingly enough, made me that much more excited about my work because, all of a sudden, what I was doing was important, and was relevant, and there was meaning in it.

I hadn't thought of that. Oh my gosh, the value of knowing where the money's coming and going isn't just that you're going to help protect the money. It's going to get you up in the morning and get you out of bed and excited to go to work. And if you've ever been in a deathmatch slog or had a difficult time at a company, you know that meaning is precious. Finding meaningfulness in your work can be very, very precious. And yeah, knowing where the money is, fantastic. And you won't get pushback from the CEO if you're making them more money.

MIKE: Side story. There was an engineer who worked on a side project for a couple of months, just on the side, and came back and said, “I've got this thing that solves the problem.” I'm not going to go into the details of it, so I’m deliberately being a little vague here. And we looked at it like, um, yeah, that does solve the problem. And it's made, like, tens of millions of dollars in less than a year. It was a critical piece that made an enormous difference. And it's because this engineer looked and said, “Hey, there's a problem here with the thing that is most sensitive in bringing in revenue, and I know how to fix that. I know how to address that problem.”

And so, he looked into the most important problem, came up with a solution, and it made a tremendous difference, a small project with enormous impact because he was thinking, well, what do they need most? That wasn't life and death [chuckles], but it sure makes a huge difference to the trajectory of the company.

DAVE: Oh, yeah.

MIKE: And you're talking about meaning, that is it [chuckles]. Getting up in the morning, like you say, it is...some people may be highly motivated to get up and dig ditches every day. But I think most people want to know that they are building something profound [chuckles], that those ditches are the foundation, right, of some great edifice. And, again, you know, you can be hammering nails, and then you go, and you leave, and you say, “Yeah I nailed some two by fours.” Or you can be the person who built a home and that woman who built a home is going to come back, and she's going to build another one because it meant something.

DAVE: I grew up in a small, rural conservative town in the ‘70s and ‘80s, and I was raised on the general concept that police are your friend, like, Officer Friendly. We literally had skits and movies about Officer Friendly. And as I got older and started getting my first speeding tickets, I was, like, not as happy with the police, at least the traffic enforcement part of it. And a friend of mine was the son of a highway patrolman. And I happened to overhear this officer get up and go...sorry, highway patrol, so it would be a trooper.

This trooper got up and on the way out the door, he said, “Well, I’m going to go save some lives,” and walked out the door. And that's all he was doing. He was speed-trapping on the freeway; making people slow down so that they would stay alive. He didn't see it as like, oh, I can't wait to just confront angry people and take their money. No, that had nothing to do with it. It was literally keeping people safe.

MIKE: I have a friend who was a police officer who ended up in a life-and-death situation [chuckles] and made a choice that ended up sparing the life of the person he was in a confrontation with. And the mother of this young man, you know, just came to him in tears, the officer, in tears afterward. “Thank you for saving the life of my son.” Her son was doing something wrong [laughs], but he managed to address the situation without resorting to lethal force. And he had a mission in mind, which was to keep people safe and alive, and because he followed through on that, lives were changed.

DAVE: Absolutely.

MIKE: Now, in software, as you pointed out, a lot of times it's not life or death stakes. Maybe you're working on pacemakers, yeah, maybe that is critical, right, or anything medical. But, you know, doing things with money is a big deal. And it's very important to not disempower people by, you know, separating them from their money in a way that you shouldn't. And having the meaning where you come in and you’re like, well, you know, I'm providing financial opportunity for people in a fair way really does make a difference. It keeps you going.

Again, I’m going to ask the question again. Dave, you've answered a little bit. What do other people think about? Are there times when you thought, oh, I need to make this choice? And I've got some ideas in mind, and I can throw them out there. But are there others who've had decision points where they had to think about what the business is doing and it made a real difference?

EDDY: Well, I think having the domain knowledge is critical in order to know what you're building, right? If you don't understand the business use case of whatever you're being asked to build, right, you're isolated. And so, you don't have the blueprint of, like, how the user experience is going to look like.

I would argue, though, I kind of ask myself a question, if what I'm working on is going to make the company money, and if it isn't, is it going to provide value in any way, right? And so, trying to find a balance between actively working on something that will pay dividends versus actively working on something that will sustain the software so that it does keep making it money [laughs], you know what I mean? So, I wonder if we can sort of dive into that a little bit. And how are we able to find the balance between understanding that the company that you're working for needs to make profits, but at the same time, right, needs to also maintain its libraries?

MIKE: Absolutely. I was going to dig a little bit further there. So, you're finishing up the ticket. You don't have one [inaudible 22:59] line. What do you do?

EDDY: I think we naturally gravitate to CI or bugs.

MIKE: Bugs. [crosstalk 23:09]

DAVE: What needs doing? Go look at Jira. What needs doing? Yeah, what's in the backlog?

MIKE: Your CI, you know, there's a lot of things that that could mean, continuous integration. I think you're talking about continuous improvement. You're going to go pay down some tech debt, or improve something, or work on a bug. And why is that? Why is that your default? And I think that's the answer probably for most people, but why is that?

EDDY: It's a great question. I don't know. I think I normally gravitate to that to maintain busy, like, busy work.

MIKE: So, I think that the reason we do that...I think there are some reasons; work comes in different sizes, and a lot of times the strategic work we're working on. Well, the business says, “This is the big thing we need to build.” We work on that. It's a big thing, right? We get to the end of that, and we’re like, well, I know there's another big thing coming along in the next sprint or, you know, in the near future, and I'm done with that big thing. Now I need a little thing, right [laughs]? I need something to fill in that space. Because if I start a big thing, I'm not going to get very far because then I'll have the real big thing I need to work on next.

So, we try to size it, and we say, “Okay, well, what's going to fit in that space?” And so, we try to go for something smaller, and I think that's the right thing to do [chuckles]. I think we go and we grab, okay, well yeah, I'm going to grab something small that'll fit. And that's just something really important because there are usually a few things in that size, right?

So, do you do something that shores up the foundation, or do you do something that's kind of cosmetic? Do you do something that solves a customer issue? Those could all have valid reasons to work on them. And if you don't know the reasons, you're just going to pick up something at random. And it suggests something about not just the engineers working on it, but, you know, the leaders, product organization, trying to provide priority here and trying to provide a map. You know, we talked about maps, giving a road map so that you know, well, this is actually what's most important. I need to build up the foundation here because foundation is eroded [chuckles], right? There's a weak spot here, and losing the foundation is not good.

You know, there was a hotel, and it was a hotel or a condominium, I think it was a condominium, in Florida a few years back where the parking garage had been eroding for a number of years and then finally collapsed, which ended up taking down the whole building.

DAVE: Wow.

MIKE: It was tragic. People died. And, you know, they'd known about it for years. They were like, “Wow, that is some exposed rebar there with water running over it. That's probably not a good thing [laughs],” but nobody did anything. And there's another life-or-death situation where it wasn't life or death until it was. Shoring up that foundation really is important. So, that may be the right thing to do, and often is the right thing to do because a lot of times the business can't see it, right? They're looking for the grand aspirations and don't always see that water running over the rebar.

EDDY: So, let's go a little extreme here, right? So, you mentioned, unfortunately, people died because of that restructure. Basically, what you're saying is, okay, if we work on fixing up our foundation for this building, it's going to cost money, right? And it's not going to give us a direct or an immediate profit, right? But your justification is, yeah, but then when this building collapses, you're going to lose more money trying to build a new building. And then, you have all these legal litigations that you have to deal with.

So, I think that's sort of how you can justify working on stuff, where you're like, it's not going to immediately make you money, but it reinforces our framework so that we don't have to go into those critical masses and having to invest more. [inaudible 27:11]

MIKE: Absolutely. And it's hard for the company to keep on that, especially if you've been, like, a startup, because if you're a startup, the answer may be, “You know what, we built that last year. It'll hold for a few years. Don't work on shoring that up because if we don't get revenue, we go out of business, you know, we lose that solvency.” And so, it can be a difficult decision. And then, when you do get solvent, and you've got some money, sometimes the default becomes, “Well, let's just keep building on top.” And you have to make that transition to say, “Well, no, we have to make that investment,” and it's important to voice that. Dave, you were going to say something.

DAVE: I was going to say, I was talking to somebody a little while ago...a long time ago, about 10 years ago. They were talking about how when you get a request from business to say, “Hey, I want to do this,” you should ask them “Why?” and you should follow...it's called popping the why stack. “Oh, it's for that? Okay, well, why do you need that?” “Oh, it's for this.” “Okay, well, why do you need this?” And you keep going until you get an answer that matches one of three categories. It's either going to make us money, or it's going to save us money, or it's going to reduce risk of a catastrophic loss of money.

And that third one is the...because nobody wants to pay to shore up the rebar in the garage when the only money to be made is you get to keep making the money you're already making, so that's a siren song to just, you know, not worry about it, not worry about it, not worry about it. And you can get in a situation where you push your risk off and off and off, and eventually, your risk management strategy becomes hope. And, you know, how hard can you hope? Because you're going to have to hope pretty darn hard because that's the only thing we're doing is just hoping it doesn't break. And it's terrible when that happens.

There was an accident...somebody else will be able to find this better than me. A tram car...one of the gondola things that go up the mountain in Italy fell down the mountain. Like, literally, the cable snapped, and it was near the top. And it was the other side, so the car stayed suspended. But it was now the only weight on the cable, and it flew down the thing. It killed everyone aboard. And the people that maintained it came out and said, “This is a terrible tragedy that nobody could have seen.”

And I remember an engineer sitting down and saying, “Let me give you five reasons why this was criminal negligence, starting with that cable cannot possibly fail until it is visibly frayed in...” Steel cable, you can literally break, like, half the fibers in that cable, and as long as you space them out, the fibers adjacent to them will support. You just about have to cut the thing to ribbons or shred it to rags to get those cables to fail. They're incredibly fault-tolerant. And this cable failed. It literally snapped that cable. And you can't have that happen.

If anybody had just gone up and looked at the cable in the previous five years, they would have immediately said, “Oh man, we have to fix this. This is way too many threads broken per foot, or whatever it was.” There were two or three other systems that also failed, and that engineer then proceeded to say, “And let me tell you why I know they weren't checking the braking system, and why they weren't checking the safety arrest line,” because that also broke. And yeah, just all the way down the line.

It is so tempting to just look at that freight cable and go, “Replacing that cable is going to cost the company a million dollars. I do not want to be the engineer that tells the CEO that we have to replace this cable,” but it was necessary. And some people paid with their lives because they didn't want to save that money, or they wanted to save the money, instead of didn't want to spend it.

EDDY: So, what you're saying is, working on that it's justification for saving the company money in the long run, is what I'm understanding.

DAVE: And there's an art form to that, right? There are times when you're polishing code, and you’re really satisfied. Oh, I want to use this design pattern. I'm going to tuck this in. It's going to be so satisfying to have this. We've all written code that you just sit back and go, “That works, and I like it,” and it makes you very happy. But you can end up polishing rivets, and you think you're doing all this noble good. But what you're really doing is something that makes you feel good but maybe doesn't help the business.

And that's, I think, the point of this discussion is pop your head up and look around and say, what direction is the company going? Is this important? Is this essential? You might find, you know, oh, these classes aren't namespaced correctly. That's a technical debt thing that's going to put us all out of business in 3 weeks. No, it's not. You're fine. We don't need to stop development for 9 months to just change around some scoping rules. That's probably not the severity of risk.

EDDY: What kind of scares me a little bit is when you have, let's say, for example, you're a construction company, and you make a bid. And you're like, “Okay, cool, no, we can make this building for you in three months,” right? And then, you're reaching the deadline, right? You're a few weeks away, and suddenly, you're like, “Okay, we're reaching the end, right? This is going to hit the budget that we have proposed.”

So, suddenly [chuckles], you get murmurs saying, "Eh, you know, this base foundation that's going to keep the building afloat we don't really need that right now.” But it would be nice to kind of go back and say, “Hey, by the way, this building can have tenants, and it's fine for the next couple of years. But if we don't go back there and add this foundation, this is going to crumble.” And you finish, and then, suddenly, you turn the cheek. You work on other stuff, right, you get other contracts. You get other stuff, and then you forget.

Suddenly, it gets forgotten about, and you're like, oh, man, right, suddenly, the building is starting to shake, and you're like, see, I knew that was a problem. We documented it as a problem, and suddenly, we have to go back. And so, I guess my concern is, there is a judgment call at that point, right, on whether it's worth taking the extra days that it would take in order to do it correctly. But what are the repercussions for doing that? And I feel like sometimes business tends to do that, you know, so you have to find that balance as well.

MIKE: Well, and you just suggested something. You said, the business wanted to do something different. Well, the business is made of people, and we're part of the business. We have the obligation to communicate. “Hey, this foundation is unstable,” because somebody in a different place in the business can't see that.

Think about that parking garage, the security guard, right, who's wandering around the garage. They're going to see that every day. The remote owner of the building is not and has no way to see it. And unless the person whose job it is to be looking, right, and maybe their job isn't even to look at the foundation, you know, they're just looking to make sure that there's no thieves. They're in a position to see something. And we're in that situation sometimes, where we have to bring up this uncomfortable fact. You know, there's a security issue, and if we don't deal with this, there's a lot of risk.

It's uncomfortable, and it's easy to just not worry about it. But in the end, you know, the financial viability of the company depends on it. And if you've got leadership that cares about the right things, and I'd like to think that most people do, you know, that always can deviate, right? But assuming that you have capable leadership, they will, I think, in general, appreciate the fact that you told the truth.

EDDY: So, what do you do then if you have to tell your client, right, “Oh, you know what, the projection that we told you about, three months, you know, it's going to actually three and a half months.” What repercussions do those conversations have?

DAVE: Coming in from contracting land, I’ve tried to develop a habit of if something's going to go wrong, I want to tell the customer as quickly as possible. Because if I find a catastrophic error, like, if we literally, you know, we did not consider that this architecture literally cannot accomplish the task that we are doing, if we continue, we have thrown away the money up to this point, if we continue, we're just going to keep throwing good money after bad.

But I also try to present solutions. We had that same problem writ small here at Acima. I had a task to make a database change, and it was just a normalization change, shouldn't have been that hard. But we had left that change for seven years, and so we had this massive First Normal Form problem in some of our data. And trying to extract it, there's all these reports from the data team that they're looking at the data in the wrong place, you know, in that First Normal Form. And to extract it, it's going to put it in another table. That's going to break all the reports. So, I had to sit down and think for a minute about, what can we do to keep the system running the entire time that we're fixing it?

And when we had a, you know, when business comes in and says, “Stop working on that, start working on this,” we had one of those. And I had gotten this refactoring to a point where you can talk to the object like it's in the new pattern, but you can keep talking to it with the old pattern. And the First Normal Form continuously sinks into the third normal and vice versa that if you update this other record, it will update the master.

It's inefficient. It's not great. We shouldn't run this way forever, but if we had to, we could because the system works, and that data is in the right place. And I'd love to be able to circle back, and I might not. I'm not anticipating that I won't be able to. It's near and dear to my heart. I would like to see it finished and, you know, I will love that and adore that topic and push on it with people. But if it never comes up in the budget codes again, then, okay, well, it's probably not as essential as we thought. We went seven years on that First Normal Form thing, and we could have just added two more columns to that table and been done.

It would have been a one-day story and, you know, I turned it into, you know, a two-month tilting at windmills. But, and this is your fault, Mike, I was in the meeting where I said, “This is what happened. This is the problem I ran into. This is how we could fix it.” And I think I actually presented, we could just do it the bad way and just be done, or we can clean it out. And, to this day, I remember Mike saying, “Thank you for doing this the right way.”

I was nervous going into that team meeting of, like, man, I might get yelled at because I'm proposing we blow the budget wide open on this, and that's above my pay grade. And it was nice to have other people in the room saying, “No, I absolutely support. Do this the right way. We need this done the right way. We’ve put this off too long.” But I had to run it. That's my point is, I had to communicate it up the chain because I don't like it when technical people make business decisions, and I don't like it when businesspeople make technical decisions.

If you're a one-man company, you have to do it all yourself, but businesspeople know what the value of something is, and technical people know how much something is going to cost to build. And I don't like it when businesspeople say, “I want you to build this, and I want it to take two days.” Okay, you're welcome to want that as hard as you want. But I'm the technical guy, and I'm telling you this is a seven-day task.

I've also had times where the reverse is true. I come in and I say...Michael on our team I walked in, and I said “Hey, I saw this little problem over here. I can fix it in, like, three hours. Let me just cut a ticket real quick, and I'll turn around and burn it.” And Michael just immediately said, “No. No, that's worthless. This is all wasted money. Please, don't even think about this anymore.” And I'm like, “Oh, okay.”

Because from a technical side, it made a lot of sense, and it was easy, and it was going to make our job a little bit easier, eh, maybe. But I found out that business we're not, you know, that this is supporting an initiative that is no longer important to the company. So, you know, please, there are more valuable things you could be working on. I'd love for you to, even just for 2 or 3 hours, just give them over here to where we need the budget. So, yeah do communicate is what I'm saying.

MIKE: Yeah, that was a fantastic question, right? You know, what do you do when there seems to be misalignment? Going back to our central idea, you can't do that unless you have some awareness of what the business goals are, right, and at least some high levels going on. Maybe you don't know the exact financial status of the company [chuckles], and that may not be important. But you do know that this thing is broken, and I can communicate that and balance that with other concerns.

And you may know that the company said, “Well, right now, we care a lot about revenue,” or “Right now, we care a lot about yield on our existing assets, you know, because we've got good things, and we want to make sure that they work well. We're not going to put in a lot of new investment.” Or, you know, maybe we care about risk management. Or maybe we're a startup, and we're desperate to get a new client, and we need to have a pretty web page [chuckles] that makes the client think that we're great. It can be any of a number of things. And understanding those priorities makes sometimes all the difference as to which choice you make.

DAVE: An adjacent thought to that is I realized that what frustrates me the most about work is when the person in charge of the capacity to make the correct decision [SP] lunches the call, makes it wrong. I hate it when I make a technical call and I'm just flat wrong. I waited that database refactoring. I waited into it fully cavalierly thinking I was going to have this done by Wednesday. And, no, you're not going to have it done by March, think again. And I hated that because it was a technical call, and I made it wrong.

But I've worked at a shop, and some of you people have heard this story. There was a shop that I was at that brought in a VP that said, “I want this. I worked at this other company, and we used this thing, and we want it. We want to give it to our customers because it's fantastic. Everybody in the world is going to want this.” And the whole engineering department dropped what they were doing for a year to go work on this, and we shipped it, and nobody wanted it. It was just DOA absolutely. And the VP lost his job for that. The company took a big hit on the stock price. And, like, when they found out where we'd wasted all that money, nobody wanted it.

Amy Hoy calls it kosher ham, where even if you could make it, there's no market for it. [chuckles] You could say, “Here's some ham, and it's kosher. My Jewish friends, come eat.” And they're going to look at you and go “No, we don't do that, even if it's kosher. I don't care. I don't want that.” So, you'll see the same thing, right? It's like, if you're going to build a report for somebody on the data team and they're saying, “Can I get that as an Excel spreadsheet?” And you're like “No, no, no I can do this. I can do this all for you with a JSON transformed in GraphQL.” Even if it worked, they don't want it. They want a spreadsheet. They want their Excel so they can do it.

And it's the same thing, right? Know what the customer wants so that you're not accidentally...I guess the definition of kosher ham is if you did it perfectly and did the best possible version of this, it would still not be what the customer wanted. And it hurts when the business lunches that call.

MIKE: And we can help. Now, leaders who are listening to this, there's an added burden, right [chuckles]? Communicate early and often. Do that market research. Find out whether that's something that actually will work. Think it through. And sometimes we're going to make wrong choices. But try to minimize the impact of those where we can, right? And get feedback all the way along. Be listening so that when the people who are working with you tell you something, you hear that information and don't just go on your solo mission because it really isn't just you.

Well, I think that we're reaching a good stopping point here. We've shared some stories. We’ve shared some figures of speech and metaphors that connect things to, you know, to other aspects of life to show just how important it is to know where you're going, where you are as you build something. Any final words?

KYLE: Just one thing that I was thinking is because this, I feel like, has been a little bit more specific to developers and new features and how that can bring revenue into the company. Just if you're in an auxiliary position, meaning, like, QA, or, you know, DevOps, roles like that, IT, something like that, always keep in mind, what can you save the company? There's, you know, there's always improvements. There's always, what can you save?

And, you know, you might have a great architect or engineer that maybe doesn't know about a newer technology that's coming out that's cheaper or, you know, there might be newer processes that you can put in place to save the company money. And a lot of the time, especially for these auxiliary roles, that's just as important, if sometimes not more important than some of the work that's being done to bring in additional revenue.

DAVE: That reminds me of the central tenet of, like, agile programming when we were first telling people 25 years ago “How about we don't give you an estimate, how about we don't tell you how long this is going to take. We're just going to tell you what we're working on, and what we think is the most important next step.” And the allegory they use or the analogy that they use is, it's like a deck of cards. And everything you work on, every task you work on on this product should be one of these cards. And you want to stack the deck so that the most important thing is always on top, that way, you know you're not wasting your time padding out, polishing rivets when the engine hasn't been finished.

And there comes a point in your development when you hit kind of, like, around the 80% mark, where you start realizing this is a working MVP. This is a minimum viable product. We could ship this if we had to. And it gets you out of that mindset of, no, we can't let the customer see this until every pixel is perfect, and balanced, and we need the thing, and it's got to adapt. Otherwise...look, we can probably get you 80% of this product in 30% of the time if you just let us work on the most important thing first. But all the other parts have to be there.

You know, the feedback loop, understanding the business, talking to the people that are in charge of that the communication is essential. That goes back, by the way, it goes back to what Eddy was saying that if you're in a shop that's very rigidly working from, like, a story backlog, when you finish the task and you don't know what to work on, well, you do. Go look at the backlog and take the top card. If your product owner is prioritizing, you always know what the next most important thing is, and that is very, very satisfying when you can get that, when you're tired and you don't want to think through the whole project. You know, I could just work on that. We already did this work.

MIKE: Well, thank you. Thank you, everybody here for your input. And, hopefully, our listeners will think a little bit more about what's going on, you know, around them. What am I doing? What am I doing here? And based on that, what should I do next? Until next time on the Acima Development Podcast.

The latest episode of the Acima Development Podcast takes a deep dive into the evolution of DevOps, platform engineering, and the concept of ZeroOps. Host Mike, along with panelists Justin Caldwell, Kyle, Justin Ellis, Will Archer, and Eddy, discuss how infrastructure should function like a well-designed vending machine—providing a seamless interface where developers don’t need to worry about the internal mechanics. The team explores how DevOps has transformed over the years, moving from traditional sysadmin roles to cloud-driven automation, enabling developers to focus on delivering code while minimizing manual operational tasks. The discussion introduces ZeroOps, a modern approach that seeks to eliminate the need for infrastructure management by automating processes and providing a frictionless experience for development teams.

Justin Ellis provides historical context on IT infrastructure, explaining how organizations transitioned from manual punch card systems to today’s highly automated cloud environments. The panel highlights best practices in infrastructure management, emphasizing the importance of standardization, automation, and observability. They discuss the shift from managing infrastructure as “pets”—custom, hand-configured systems—to treating it as “cattle”—standardized, scalable, and disposable resources. Tools like Terraform, Kubernetes, Docker, and Prometheus play a key role in creating resilient systems that support modern software deployment strategies. By implementing these practices, companies like Acima have been able to scale deployments from quarterly releases to multiple times per day, improving reliability and developer efficiency.

The episode wraps up with a discussion on future innovations in DevOps, particularly the pursuit of fully dynamic environments where developers can spin up isolated test environments on demand. The team acknowledges challenges like managing stateful applications, dealing with legacy infrastructure, and ensuring effective observability, but remains optimistic about advances in AI, cloud-native tooling, and automation frameworks. As organizations continue to refine their DevOps strategies, the goal is to build systems that are not only scalable and efficient but also reduce complexity and operational overhead, allowing developers to focus on building great software.

Transcript:

MIKE: Hello and welcome to another episode of the Acima Development Podcast. I'm Mike. I am hosting again today, and I've got a panel with me here today of Justin Caldwell, who's coming to us from the cloud engineering DevOps side. Kyle who has been on here a lot before also from DevOps. We've got Justin, Justin Ellis, who has been here before, and we've got Will Archer. And we've got Eddy.

As usual, I'm going to tell a story to lead us in here to kind of launch our discussion. Oh, I don't know how many years ago, a lot of years ago [laughs], I worked at a sheet metal shop where we made bubble gum machines. You've probably seen a vending machine where you put in a quarter, and the bubble gum ball does a lot of things before it makes it down. We made, like, roller coaster-type things. So, the bubble gum ball would drop in the top, and it would roll around and do some loops and stuff, and it would come out.

And it's interesting to actually make one of those. There's a sheet metal around the side. We'd use the wire, and we'd actually stretch it out so it would bend better [chuckles] and curl it around a jig. Anyway, there was a lot of work that went into it. And then when you run it, you go through all the work to see that bubble gum ball roll all the way down the machine and come out the end. It's exciting to watch. People do it not so much for the bubblegum ball, or maybe, but to watch all of the machinery going on inside because it's fun, right? People like to watch that.

And the interesting thing about this is what you get out at the end is exactly the same as if you didn't see the machinery at all [laughs]. In either case, you got a mechanism that drops a bubblegum ball, and it comes out. And the interface is exactly the same. They probably don't have that many coin-operated [chuckles] vending machines anywhere, you know, a lot of times, you put in a card, but same deal, right?

You have a mechanism. You press the button or turn the dial, and bubblegum ball comes out. It really doesn't matter what happens in the machine. What you want to have, and this is really important, you don't want to have a weird vending machine where you have to put in something other than the quarter, right [laughs? You don't want to have some vending machine where it asks you for something in a foreign currency, for example. Like, well, that's kind of cool, but I don't carry pesos, right [laughs]?

And this is kind of the hard thing about switching over to cards, like a lot of things have done. Well, do I have a card with me? Do I have a coin? And now it's kind of flipped to the other side. Like, who carries cash [laughs]? So, it's flipped. You want to have that common interface because, otherwise, it's awkward, and nobody knows how to use it, and then you have to know what's going on inside.

If you have to know how the machine works, you've failed. The idea is to have this as a simple interface. You don't care what's going on inside. Of course, with a bubblegum machine, you care a lot because that's the main reason you're looking at is to watch the ball come down. But in the end, if it doesn't work inside, then that's a fail. What you want to do is have this simple interface that never changes. You just want that bubblegum ball to come out.

And I say this because it applies directly to how, well, to a lot of things, the interfaces between systems and software. Today we're specifically going to talk about the integration with DevOps. You may call it by different names, whatever interface that you interface with to get your infrastructure. Because there's different ways of handling this. There's times when we have people embedded into the team, so each team has their own person, and they build their own stuff. There's a way of approaching this. It's something that's called ZeroOps, where you just press the button and the thing comes out [chuckles].

DevOps provides a common interface that everybody needs to use, and nobody needs to know how that bubblegum makes it down the track because it simplifies systems so much. And we've got Justin here who's been passionate about this idea for...we were just talking on the pre-call. I've worked with Justin for 15 years [laughs], so he's been passionate about this for a long time and has built this more than once and made multiple companies really successful in terms of infrastructure because of this idea.

And I'd maybe ask Justin first to take over from here and give some thoughts, kind of launching from [chuckles] that introduction. And tell us what you think the interface with a cloud infrastructure, however your infrastructure is, maybe you have an onsite infrastructure, what is that interface supposed to look like? And why is that important?

JUSTIN ELLIS: Yeah. Thanks for the introduction. I have thought about this problem really for the past 15 years about how to build systems better, how to make software development easier. On the system side, you're thinking about availability, reliability. You're thinking about security. You've got to think about a lot of different aspects of it.

But at the end of the day, really, our main job is to enable software developers to be able to deploy code, push code out, manage their code. And they don't really care about the mechanics underneath, right? They have a job to do, and then we have a job to do. And I think about if you go back to the very first instances of kind of our field, in the '70s, people used to write on punch cards. And they'd basically create their punch cards. And then, they would create a huge stack of all these punch cards, and then they would pass these punch cards to someone called an operator.

And that operator had a very specific job. They had to, A, keep those punch cards in a very specific order. And then, they had to feed them to the machine very carefully and keep the order. You couldn't drop them. You couldn't have any of those issues. And then, the program would get executed, and then they would get the output, and then they would help return that output back to the user.

Things have changed quite a bit, and I think there's been some major movements. There's kind of those traditional IT systems where you'd have sysadmins. And then, that kind of moved into the cloud world when AWS was introduced to the world. And that also kicked off this new movement called DevOps, where it was like, hey, instead of passing it to the operator and then having the operator do everything, let's get everybody kind of involved in this process and enable developers to be able to pass this code and then also understand what's happening, what's getting tested, kind of move that forward, right?

And then, since the DevOps kind of movement mindset idea, however you want to call that, that's kind of moved into, you know, SRE is still a big style of managing systems, site reliability engineering. And then, that's kind of moved into now this new model which is very similar and kind of inherits from a lot of the previous models, which is kind of platform engineering.

And the goal on platform engineering is to create kind of a standardized platform that developers can interact with. They can put the gumball machine in. They can put the quarter in, and then their code gets built, tested, make sure everything goes well, and then that gets output to a system that you run.

So, long story short, we've gone through a lot of different movements, and I think that we're still perfecting and getting to a better place. And I think every single iteration has had new innovations and new additions to the systems, things such as orchestration, which we have with Kubernetes and all of that kind of stuff.

And so, through that whole journey, I think we've learned a lot, and I think we're still, not early days, you know, we're well into it. And we're trying to provide kind of a standardized platform the developers can develop against. But it has been interesting to kind of see that evolution over time, and then see kind of new processes, new ideas on how these things get going and get worked.

I will say the one interesting part of our job is that there's not a standard way to do things. I guess that's how software development is, too. There's not one golden path that you can just say, "Hey, just take this golden path and do that up." It all kind of depends on your requirements, and the people involved, and all of those kinds of things.

MIKE: Thank you. That was a very comprehensive answer.

JUSTIN ELLIS: [laughs] Okay.

MIKE: But I think a lot of us…you talked about some of these old days with sysadmins and punch cards. My father-in-law's career actually was a computer operator [laughs]. And he would move the tape reels, you know, the magnetic tape from one place to the other and ran this big computer inside a bank. It was a thing [laughs].

JUSTIN ELLIS: Yeah.

MIKE: And the thing is, having to know about those details, having to know specific details. I remember, back in those days, I would be configuring a server. I remember one time configuring a server, actually, over to my parents-in-law's house (It was, like, a two-hour drive.) banging away on the computer on a remote connection, configuring this server along the way.

And every time you do that, it's going to be a little bit different than the next one because you make some little difference, right? And especially if you've got physical hardware, maybe this one's not exactly like the other one. And so, you have this pool of snowflakes. Every one is a little bit special, and every one is a little bit different. And managing that requires somebody with deep in-depth knowledge, who knows how those systems work. And that person becomes a point of failure. But then every system is hard to work with. It's brittle. It's very challenging to expand. And you're talking about a very different way of thinking about it, Justin.

JUSTIN ELLIS: Yep. Yeah. Yeah. Kind of within the whole DevOps world, we talked a lot about pets versus cattle. And the idea here is that you can have a pet, which is your pet server, that you've got to go in and you've got to care for. You've got to patch. You've got to update. You've got to make sure that everything is going well.

And then, once you have 10 servers, once you have 20 servers, once you have 50 servers, it's the same thing as having 50 cats. It's a lot of work to keep up on it. It's a lot of work to continue that forward. With the cattle idea, it's like, okay, what is an application? What are the types of workloads that we run in our system? And can we standardize, and can we build to a specific type of target?

Now, there's been a lot of tools introduced like Docker containers that kind of allow us to kind of define those targets and define where we're going so that every application that we build, whether that’s an application or a service, that could be kind of bundled up and then pushed out in just the same way as all the other services. And then, we can kind of scale those out independently.

So, you might have one service that has a lot of traffic on it and another service that has very little traffic. This allows us to think about those applications very similarly, build the same tooling for how we do that, but then you can kind of scale those independently. So a lot of power there.

MIKE: You know, I'm going to ask a question for not Justin, or not Justin Caldwell here, and maybe particularly to people who've been around for a while, who've worked with systems that do not treat things like the bubblegum machine, like the cattle, like just hand it off and it works. What are some of the failure modes that you've seen [chuckles]? Because that kind of provides some context as to why this is so important.

JUSTIN CALDWELL: So, I worked for Fidelity Investments for a number of years on their identity and access management team. And we would have a team of DevOps engineers who were very, very familiar with their login infrastructure, and that was all they did, and it was their cat or series of cats.

And we would only be allowed to do an install once a quarter, and it was nuts because we'd have to schedule it out. We could only do it from starting at around midnight, our time, and we only have a window until about two or three o'clock in the morning. And if we didn't make it during that time frame, the install, we would have to start the rollback process and try to get it another night.

But the thing about it was that only they knew all of the arcane secrets. And only they knew, like, where all the servers were, where they physically were, and what version of whatever was on there. And we were at their mercy in terms of an application team. And if any of them, God forbid, if any of them left the company, you knew that that next install was going to be crappy [laughs]. So [chuckles], it was nuts. And that's my little story. Yeah.

JUSTIN ELLIS: And I think it's great to have Kyle here because, you know, I'm trying to tie it back to the gumball machine as you kind of watch it go through. But as a developer kind of pushes some code, they also need to understand what's going on in the system. They don't need to know the fundamentals, but they need to understand, okay, how is my build going? How did the deploy go? Once it's deployed, is it successful? Are there problems kind of going forward?

And so, kind of a big part of the platform, a huge part of the platform and something that Kyle has been focused on for quite a while, has been kind of how we observe it once it's running in the system and everything's going well, but giving feedback to the developers. Because developers are best suited to understand what's going on in their application, best suited to understand what bugs come up, or, you know, maybe that last deploy didn't go as well as we wanted to.

So, I do see our job as kind of also completing that feedback loop so developers can understand what's going on in their system and all of that kind of stuff. And that goes back to also this other idea that's kind of fundamental to continuous deployment, which is application ownership, where engineers are really the ones that are running their applications in production because they know their code best. They're the best person to kind of address those. But having engineers kind of involved in the process, again, they don't need to know the fundamentals, but kind of involved there.

So, that's Kyle. Kyle has built an amazing observability stack that allows us to kind of see that gumball as it moves through the machine, which gives the developers the power and the understanding to see what's going on in the system.

KYLE: I would say, with that, on the DevOps side, that's one of those things where it's almost insulting if the developer can come to me and tell me what's wrong with the gumball machine mechanically, right? I want to know, first, that there's something wrong mechanically with the gumball machine.

If their gumball comes out and it's a result of what they put into the machine and they got a funky gumball out that way, that's fine. They can debug that. But if it's something fundamentally wrong with the gumball machine, that's another side of the coin where that type of information needs to come to us directly so that we can get that addressed so that it is not hindering the engineers.

MIKE: It is true that sometimes the gumballs would literally go off the rails [laughs]. It's a little bit bumpy, and there it goes. And that observability is important. You want to be able to look in there and say, "Okay, it is sitting on the bottom. That did not work." Now, it would be far preferable if the machine could automatically detect that and throw a new gumball down, right [laughs]? It makes a huge difference if you can make a machine that can do that.

JUSTIN CALDWELL: So, one thing I do want to address, classically, DevOps were not viewed as engineers in the classic sense. A lot of them were viewed as, like, oh, point and click like ClickOps. That was the phrase and everything, or back to the mysterious arcane people that have a bunch of bash scripts and things like that.

But in modern day, like, in the last five years or so or maybe even a little bit longer than that, with the introduction of Terraform, with the introduction of other IaC solutions out there, it is nuts to me how much engineering code, like, basically, to me, as a software engineer, I look at what the DevOps teams are doing and I'm like, you guys are just coding [laughs]. You guys are full-on engineers. You have your process. You have automated tools that's checking your coding just like application engineers. And it's, you have your history. You're checking in code. You're doing code reviews.

There is no more ClickOps. And if you are still doing ClickOps, I think people recognize that and they are like, oh, let's move this to Terraform or some other IaC. And it's just like, you're engineering complete systems. You're having data flow diagrams. You have the code that controls all that.

And I look at it like, man, I need to go learn that because you're speaking my language, and it just looks awesome. Rather than putting boxes up on the front end like a web interface or anything new, you're dealing with whole infrastructures and everything. And I think it's so cool to see that work being done that way.

EDDY: I was going to say the appreciation that I have towards DevOps has only been the exposure here at the company. But I hear things, like, terminology tossed around all over the place. Like, that was foreign and still is, right? "Oh yeah, we need to run Terraform," you know, or, like, "Our clusters are weird," and, you know, like, "Oh, that container is really odd or whatever." To me, having a team that understands that at a very deep level and a granular level that's just less things I have to worry about as a developer.

So, when the infrastructure is well built and you have a whole team dedicated specifically for that and it goes smoothly, that's just less things on my plate that I have to worry about. So, I just want to say the value that DevOps provides is enormous.

KYLE: To go along with what Justin was saying there, with this infrastructure as code, I know our team, and I hope other teams are the same way, if you run into something that was done, ClickOps, the entire room shudders. We're all just like, "Oh, that's disgusting. Like, why would you do that? You better have a good reason, right?" Maybe a proof of concept. But if you don't have a good reason, otherwise, it's just like you get chastised. You're like, "Why would you even consider doing it that way?"

So, we have gone completely to this infrastructure as code world. And it is like Justin saying: we have linting; we have formatting; we have code syntax. Some of these are their own specific languages. Like, over here at the company, we use Terraform, and Terraform is historically in HCL. However, now going forward, they're creating support for your popular languages. You can do TypeScript. You can do Python. You can do Ruby. To go along with that, yeah, you are. You're putting Terraform functions and Terraform resources inside your code.

I know here we heavily use, what, Bash, Python, Ruby. We even have some Go. That's a CLI. But the other three, like, those are integrated with our HCL code. It's generating HCL code. It's help functions, you know, those types of things. So, to go along with that, I don't think the idea that DevOps is not engineering anymore really fits. And it is kind of like, so, where do you put them? And I think that's kind of where the community is landing on this idea of we need to call them something else. We need to call them, you know, platform engineers, or SREs, wherever you're landing in that community, but they are engineers now. They're part of the engineering department.

JUSTIN ELLIS: Yeah, we call them SREs at my place. And the other thing we think about a lot is the interface that developers have with us. Now, I feel like we should probably be more mature on the interface side of things, you know, maybe even have a web user interface to allow folks to kind of see what's going on in the system. So, right now, it's all CLI-driven, and then also additional tools like GitHub Actions and Jenkins to kind of provide that interface for us.

But we thought a lot about how do we interface with developers? And one of the things that we've done is thinking about if you're running an application or a service on the platform, how do we allow developers to say, "Hey, I need a new bucket. I need a database. I need a new resource for my thing"? And a lot of times this takes a lot of steps, a lot of coordination between those. And I've always said if developers have to understand Terraform code, or are writing Terraform code, or if they have to know what Kubernetes objects there are, or how the fundamentals of Kubernetes works on a deep level, then we’ve failed our job.

So, we've kind of created what we call kind of an application manifest, which is basically very simple. Well, for some applications, a very simple application manifest that allows developers to kind of list out the resources that they need. And I think that does give feedback to developers, too. So, they can see, okay, what databases do we have? Do we have a bucket? How do we communicate with Kafka? All of those types of things.

So, we kind of want to continue to kind of lean into that style of giving developers that feedback and also empower them to be able to kind of build their applications with components that we've added: security, and guardrails, and all those good things, high availability and everything else. But allow them to say, "Hey, I just need a database. Here's some configuration options for that database," and then we can kind of take it from there. We can make sure that scales properly and kind of goes forward.

So, I think we've done a good job so far, but I think we need to continue to kind of lean into that idea of, like, how do we make that interface for developers as easy as possible to give them legibility to the system, understanding of how their application works? And when the new people onboard, they can kind of see, oh, this is how this application is. This is the database and all the configuration that they have for their application.

MIKE: And you're describing a very proven pattern. You look at the Unix, Linux world; you have tools, and you have a configuration file that's around that tool, and that's the interface that you use. You don't point and click. And if you do point and click, it's going to generate the configuration file. It's a very standard way of communicating. And then, it's configuration, right? It's checked into your version control. It is a form of code, but it's something that is clearly well-structured and that you can learn and fill out the config. Much better than having to speak with the arcane wizards and have them build something out.

EDDY: So, I have a generic question, and this might be a little difficult to answer. But how much control, typically, should you give an engineer, like, a developer, over your configuration file?

JUSTIN ELLIS: Yeah, as much as possible, so as much as possible. But then you still need kind of those checks and balances in there. So, every kind of configuration change that gets made, just like with code that gets reviewed, and PR'd and quality checked and all that kind of stuff, we try to do that same system. So, if people can make a configuration change because they want to make that modification, it doesn't go directly to production. An engineer looks at that, like a PR, understands what the diff is going to be, and then approves it for an application within kind of an environment. So, I always like to empower developers as much as possible.

I've worked places before where, as a software developer, I didn't feel informed, or I didn't understand what other applications there were out there. So, I've always felt we need to empower. But then, obviously, you have other things like security and guardrails and all that kind of stuff and then costs as well. You need to understand like, okay, do you actually need this gigantic database to run this new application? So, there needs to be some back and forth there. But yeah, to answer your question, as much as possible, for sure.

WILL: Listen, I just need 10 more instances, okay? It's the last time I'm coming to you, 10 more instances will do it. This is the last time. Everything's going to be 10 more. And I promise, it's the last time, 10 more, and it'll be cool.

MIKE: [laughs] Nailed it.

JUSTIN CALDWELL: I do want to point out to, Eddy, like the mechanism. I mean, so, from a security point of view, the mechanism for controlling that is through the code owner’s file in your repo. And so, you can go ahead and create a PR that has all of this new infrastructure that Will wants. Will wants 10 more instances, so he goes and modifies that. But the code owners will be, you know, one of the teams will be the SRE team or the DevOps team. And they'll be forced to review that and say, "Sorry, Will, you only get five."

MIKE: And, hopefully, you have somebody...and to go back to Kyle and the observability, they say, "Well, we gave him five, and now it's using half the cluster's worth of resources. Something is wrong here [laughs]." Maybe we should go talk to Will, say, you know, "The problem's not on our side [laughs]. We need to take a look at your code. What is the reason the database is running constantly at 90%? And you've got all this queuing for your requests. Something is not right here." And that conversation can happen. Go ahead.

JUSTIN ELLIS: Oh, no, I was just thinking, you know, so kind of a theme of this is ZeroOps. And just to kind of give you some background, Upbound as a company, generally, we haven't really talked about ZeroOps. This is kind of a new initiative. And I actually have just recently changed roles to kind of focus on ZeroOps specifically. And I think just like anything, naming is hard. ZeroOps doesn't mean that you literally have zero operations. In my mind, it means that you have as few operations as possible. So, that means that we're not running sysadmin-style on servers and keeping those VMs kind of managed. This is more and more moving towards container and orchestration.

So, when I think about kind of ZeroOps across our enterprise today, I've kind of worked on developing that strategy and roadmap for where we're going with ZeroOps. And part of that is understanding where our operations costs are coming from. We kind of have a company where there's different lines of business that have different maturities. As far as kind of platform engineering and infrastructure as code, we have a lot of legacy stuff where things still live on VM. We have a big on-premise structure, right?

So, when I think of ZeroOps, I really do think kind of like increasing our capabilities and moving more towards a platform engineering style, moving towards an interface for developers, for data, for security to allow them to write security rules and then having kind of that same PR process across all of the organizations.

So, just like we've treated developers, kind of start to treat those other folks, whether that's QA or security, kind of moving forward and allow them to kind of go through that same process, where they're able to write new security rules, new QA processes. And then, they have a really clean interface to test out their code in non-production environments and then, eventually, kind of roll that stuff out to production. So, that's kind of ZeroOps. And I'm kind of curious, when you guys hear ZeroOps, what comes to your mind? What is the general feeling about ZeroOps?

WILL: It sounds like ops has been running really good, and the gumball hasn't gotten stuck in a while. And so, some very smart person is looking at a spreadsheet, and thinking like, why do we have all these ops people when ops is running so good? And I feel like it’s people maybe being a victim of their own success, you know what I mean, sort of like, well, gosh, that spare tire is so heavy.

MIKE: [laughs]

WILL: And we don't need it. I mean, the emergency parachute, two parachutes? Why do I have to pack two parachutes? So, I mean, I don't know, I spent a little bit of time last week with the gumball stuck in the machine. It's not that I can't do that kind of stuff, but I'm not as good as you guys are because why would I be if I do it once a year when it's, like, the gumball stuck? Who's got the key? Oh, crap. What's going on?

So, I mean, efficiency is efficiency. One developer could do more work, right? And one ops engineer can do more work. So, you're doing more with fewer people because you have the tools that allow you to have that increased productivity and leverage. But it does give me a little bit of anxiety, a little bit of stomach acid just because I know how good I am at it, and I'll figure it out eventually [laughs], but eventually being the operative word.

JUSTIN ELLIS: Yeah, ServerOps, to me, means...and this might be a way of thinking of it as less SSH-ing into servers, less manual interventions moving towards more automations. So, instead of like, you know, you have a problem that pops up, right? And instead of solving the problem kind of on that local level of just like, hey, let's just get this issue fixed and then move on, it's like, how do we provide a system to prevent all of these errors kind of going forward? So, how do we make sure that this problem kind of never pops up again? How do we build a tool to kind of watch this and intervene, maybe do self-healing? Whatever that may mean, but kind of more sophisticated automation.

And automations are tricky because, especially with interventions and all of that, things can go terribly wrong. So, you have to be very careful and very structured and think about it more of a framework when you do that automation. And then, also, there's systems that are much better automated than others.

We think about kind of stateful systems or stateful applications versus stateless applications, meaning that a stateful application would be something on a VM that requires a disk, and that state is kind of on the disk. And if you need to scale up that server, you've got to go in and change the VM, which is going to require maybe some downtime and all those kinds of issues.

When you move towards more of a stateless model, that means that all the storage is abstracted away from the application itself. And so, that just gives you a lot more tooling and a lot better ability to add automation pieces so that you can scale out. You can address problems, and you can fix those kind of going forward. So, I think that kind of stateful versus stateless distinction and moving more into kind of a stateless direction really does help a lot to add additional automations, and then giving those platform engineers the ability to work within a framework and push our technology forward.

MIKE: There's a couple of questions I'd like to ask, because there's something that you touched on, Justin, that there's legacy systems because that's the reality pretty much everywhere. If you're starting on day one, great, but that's almost nobody, right? Because that only happens on day one. And then, after that, you've got a legacy system that maybe you wish you'd redone on day one. And I'd like to hear about the process. But before we do that, I want to start with a little bit of the goal.

We heard before about a company where you could install or deploy once a quarter, and if you fail that, out of luck, right? Maybe you can do it the next day, but you still get once a quarter. So, I'm going to ask our platform engineer folks here, and I'm putting you on the spot a little bit, but how many times do we deploy a day at Acima? I'm going to ask about Acima line of business specifically. So, how many times do you think about Acima deploys a day? And it doesn't have to be exact, but approximately.

JUSTIN ELLIS: Across all of our applications, I would guess we probably...more than 30 times, sometimes 50 times a day. But somewhere around that range in the multiple dozens of times a day across all of our applications. I would imagine we've had days where we've deployed out a hundred times a day. So yeah, quite a bit.

MIKE: Okay, so up to a hundred times a day. I would argue that that is a much better situation to be in than once a quarter. If I've got a bug, I'd sure like to hear that we're deploying a hundred times a day. I'd much rather hear that than, well, we can get that for you in Q4 [laughs]. It's hard to even express the dramatic difference. And those of us who have kind of seen the industry move here, you saw that happen.

There was a time when we thought, deploying more than once a day, oh, that is my worst nightmare because you knew how bad the deployment process was. But if you actually build the infrastructure, it completely changes the paradigm to where deployment is nothing. And the infrastructure is there; you don't have to think about it. You put in the quarter; you get your gumball. Why would anything else happen? The system just works. And it really is a game changer.

So, you've seen that example of how, I mean, I don't know how many orders of magnitude that is. I'd have to start thinking about it, once per quarter to a hundred times a day. That's a lot of zeros of difference . How do you get there? Because it doesn't start there. How do you take a real company that maybe has some legacy stuff and get them there?

JUSTIN CALDWELL: Yeah, so I would say, if you're in a company where you have a lot of legacy systems; you have a lot of stateful systems; you have a lot of systems that kind of break down, or you may be a place that deploys, once a month, right, and you want to change that. I think you really have to start at the fundamentals and kind of the systems level and start to sketch out how you want to handle your kind of cattle going forward. Start to understand what the patterns are across all the applications.

And then, my recommendation is always start small. Start with maybe one cluster that's kind of a new cluster. This may provide orchestration. But use the mature frameworks and platform tools that we have today, and maybe transfer that into a stateless application that you can run, right? Follow that process. Think about the fundamental abstractions, about what is an application? What's a service? How do we think about scaling this out? How do we secure this right? Take that one application and try to modify it to get to that state. And that's not always possible, right?

But there are always applications that you can do that on, where you can start to think about abstracting it out into its fundamental pieces and fundamental parts. And that may mean, okay, let's get this database off of a VM and maybe move to a managed service like RDS because we're not Postgres experts, or we're not whatever else. But we know enough to handle a Postgres instance on something like Amazon, or AWS RDS or GCP database. So kind of extract out, think of it in pieces, use modern tools.

So, I think that would be my general strategy when tackling those legacy projects. And then, sometimes if you're running like, say, commercial off-the-shelf software, you have no choice. You always have to run a VM, right? Think about you might have to migrate to a new tool to get to this more stateless fast-moving thing.

And then, the last thing I'll say, and I know I'm going long, but it's just, when you do have deploys that are spread out by weeks or maybe even months, what you're going to do is you're going to have a lot of changes into that, you know, if you think about all the changes that a developer does across an entire month, and then you think of all the developers across all the teams making changes to an application, those changes build, and build, and build. And then, you're going to deploy that out, which means you're going to have a really good idea of how this works.

Once you have multiple applications talking to each other, it's a complex system, and you're going to get side effects that you never anticipated and you never thought about. So, when you have that deployment style, you're going to run into more failure modes, and they're going to be larger failure modes, where if you deploy once a day, twice a day, three times a day, you're going to get that quick feedback. You're going to make those changes smaller. So, you just see a lot of benefits by moving quicker and having a solid rollback strategy.

WILL: Yeah, but also, I would say that I think rapid deployment, generally, general best practices, and for sure, better than quarterly or even daily. Like, just run it, but what if I happen to put two quarters in the gumball machine? What about three-quarters? What about five quarters? You know what I mean? What about 30 quarters? And here they go.

And this is maybe less of a, you know, if you have your release strategy for production worked out, I think most people going to production still...it's not the Wild West of no matter how good your pipeline is. But for stuff like your test environment, your staging environment, your QA environment, I have seen many very Wild West kinds of situations where it's an absolute free-for-all.

And you lose a lot of time on that, where it's like, why are my GraphQL queries...just no GraphQL today. Sorry. Oopsie. We made an oopsie. There will be no GraphQL today. It's like, okay, all right. So, it's too much of a good thing. How do you manage access to the gumball machine when you don't have an operator, let's say.

JUSTIN ELLIS: Yeah, you know, this makes me think of application size. So, I know there's this big monolith versus microservices architecture, right? Some styles are better suited for certain environments. And I wouldn't ever advocate a really tiny microservices infrastructure where you have these tiny, tiny microservices.

I think there's a right size for applications. I think once applications get too big and you have too many engineers, say, working on a single application, once you get there, then you're starting to think about queuing, and dependencies, and ordering, and all of those kinds of things.

I think once you get to that point, I think it's probably wise to start thinking about, do we need to break this application up? Is this one application that, say, 20, 30 engineers are working on, right, and the amount of code that they run on? Is it time to start thinking about maybe extracting that out into multiple services? So, if you're starting to queue, I would say maybe it's time to start thinking about separating those concerns. I think it's always smart to start with a monolithic architecture and then extract pieces from there kind of as you go.

WILL: Oh no, no, I'm not talking about a queue although that is a problem. I'm talking about I'm developing my microservice, and you're developing your microservice, and Eddy’s got his microservice. And Ramses and Mike they're working on theirs. So, we've got five microservices. So, we're not going to go to production until we have our ducks in a row.

But we're all on the staging environment, the QA environment, and we're all pushing up changes, and things are getting weird because we're all just trucking along hoping, expecting, right? Because all this microservice, these complex applications where we have successfully split them up. But we still interdepend on each other and if everybody is moving in different directions at the same time, like, things are not always…

We've got a microservice architecture, but we're all pushing our changes up to staging and not for nothing. Like, the data environment is big enough and complex enough so that I can't just...it doesn't just fit on a box. I work for a large retailer, and there's a lot of background knowledge around pricing, and availability, and SKUs, and all this stuff which is way bigger than anybody can have. But I have to have it so that I can get my changes out. And I've seen the end state to a lot of this advanced DevOps stuff, and it can be really challenging in a different way.

MIKE: There's something...and I'm not on the platform engineering team, but I've heard something from them [laughs], and I've talked with them a while about this. You're touching on something, and the problem is the shared environment, right? And you talked about the complex data, absolutely. And this is not an easily solved problem, but it's something that we're actually working on. Because if you can come up with a replicable data environment, create this complex data environment, take a snapshot somehow, right? And that may be through some sort of seeding, or maybe you actually do have an actual snapshot that you copy.

If you can get to the point where you can replicate that data and on the platform side you can replicate the environment, then you can get to the point where there isn't a QA environment, spin up your environment. All the other services are known working based on the most recent build that's in production of the other services. And you're just testing your own in that environment. And that seems like the holy grail.

That's something we've worked for for a while. And kudos to the [laughs] platform engineering team for getting there, because we're actually pretty close at Acima. We're actually just kind of right at the cusp. I think we'll be launching that maybe in the next couple of months to where we can have that and then there isn't a QA environment. You get the environment you want and it's isolated. It's a dream [laughs].

JUSTIN ELLIS: I can't tell you how many times...I've been talking about dynamic environments for a long time, but dynamic environments is still the goal. It would be great for QA to spin up a whole new environment, have just your feature branch in that environment. Everything else is on the main branch with data that's there, that's relevant, that's correlated. I agree with you 100%, that is the holy grail. We're working towards it.

But it is a wicked problem. You have to think about the data side. That's one of the primary things. You have to think about how it interacts with exterior systems, say, maybe, like, a third party that you're interacting with, how that data gets all shuttled through the system, so agree 100%.

MIKE: So, there is a way. There actually is a path through this. There's been kind of a through thread through all this is that things that require interdependencies, coupling, dependencies between each other are hard to deal with. Anytime you have something shared like that, it's hard to deal with. And a shared environment, yeah, you're coupled tightly, and you got to deal with that. If you can figure out how to decouple that, so, no, it's not a shared environment. Everything else is not shared. And then, you just got to put up your own thing. It changes the paradigm, right? It switches the way we think about the problem.

Like, you have, oh yeah, well, deployment is hard, so I'm only going to do it every now and again. Well, now I have all of this entanglement of things and all of the interdependencies that I have to deal with. Well, instead of you disentangling, I just put up one thing at a time, then I get rid of that problem. And you have to rethink the way you approach the problem to eliminate the complexity, and then the problem becomes much easier. What were you going to say, Will?

WILL: Oh, well, you know, I had a question. We were talking about stateful versus stateless systems, right? And I think everybody has bad ops in the beginning for the exact same reason, which is a good reason because you can't afford it. You're just getting up however you can. You don't have money for ops. You've got whoever can get their EC2 instance up and running and rolling, and then you go with that. And then, you get ops, hopefully, when you can afford but probably more commonly when it really starts to exert pain and pressure on the organization.

And so, what makes a service stateful, right? Like, this has to be stateful. This is a stateful thing, where it has to be like that. And it isn't so much of like a well, I'm just going to throw up one instance, and I'll just roll the database in with the instance because I only have one monolith, and there you go.

We'll just put it all up on the same box because that's relatively easy to do. But where it's just like, okay, I'll move it off this box. And now I have two boxes, and then we go and do the thing. But what are the really tough nuts to crack in terms of applications that want to be stateful and they don't want to move into that stateless process?

JUSTIN ELLIS: To me, it's all about the disk, right? So, if that specific application and how it's run and that process needs access to, say, one specific disk, if it's like a regular, say, virtual machine that maybe runs Linux that writes files directly to the disk, that's where you run into problems. Because what you want is, you know, stateless is kind of a misnomer in that you are still storing state. It just happens to be in an external system. And a lot of times we'll use managed services for those external systems.

So, really, it's trying to abstract out and push this stuff into a database, maybe into an object storage and kind of getting that out. Once you've got that stateless, then you can kind of scale up that 1 process into 10 processes, and they can all use those same data sources. They can still all use that same state. They can all reach out to the same Redis instance. And then, that Redis instance is the one that's providing that state going forward. And just a lot of applications just generally depend on the disk. I don't know, Kyle, what else? Can you think of anything else (I'm kind of putting on the spot.) that kind of drives the stateless versus stateful system?

KYLE: The one I'm sitting here thinking of is observability-minded, but I'm thinking of Mimir and how they have that, right? They've tried to get out of being stateful. They use an object store, S3. You can use that as your backend, but you still have some of that that is still very stateful. And that's any of the current time metrics that you're wanting to store. But kind of like Justin is saying here, any of that, as soon as you can just move it off into the object store or anything that's managed, then you kind of get it out of the way or out of the way so that you can become stateless to a sense.

I think the other thing that is helpful is generally a quorum. If not everybody can be talking to it, not everybody can be writing to the same table at the same time, or whatever it happens to be, a quorum to know who's talking to it at the current time, because that's another system, or who's going to be accepting even incoming requests.

Maybe you shouldn't be receiving incoming requests to everybody, load balancing. So, anything like that to where that one source of truth, I guess, is kind of your state, anything not to overload that to allow those frontend replicas or whoever's interacting with that stateful process to allow those to become more stateless to a sense. Hopefully, I made sense on that spiel [laughs].

JUSTIN CALDWELL: Yeah. And so, you mentioned Mimir, and Mimir is basically just a way to manage Prometheus and kind of scale it out. So, I'm kind of curious, on the Mimir side, are they using...I know that we use kind of EFS volume. One of the challenges you have with not writing to disk is that object storage can be slow. Disk I/O you got to go over the network and all that kind of stuff. So, I'm curious, does Mimir use it just for kind of the Disk I/O bandwidth generally, or is it just an architectural consideration?

KYLE: Sorry, I guess I'm not understanding your question.

JUSTIN ELLIS: So, when I think about kind of, you know, back to Will's question of what are those main challenges when you are trying to move from a stateful system to a stateless system, what things are going to be difficult to move into kind of more of a stateless model and getting those into, say, object storage or whatever else?

The one thing I can think of there is that having a disk on hand, having these new SSDs available really does give you a lot more bandwidth. So, data proximity really kind of is a big issue. And when you can write directly to disk, that's going to be so much faster, so much quicker than, say, making a network call out to S3 and writing that file. And you want to read those files as well, and that needs to be quick and fast, right?

So, in an application that really does require to consume a lot of data process, a lot of data in a quick fashion, which I would imagine something like Mimir does, that may be one of the reasons that there are kind of stateful bits throughout their general architecture.

KYLE: Right, right. So, on both the read and the write process, to my understanding, you kind of want that local timeframe, meaning that timeframe that you're going to be using the most of. And it kind of requires a little bit more of an understanding of Prometheus itself. Prometheus data itself gets chunked. And so, you can change that chunk, but I think, by default, it's three hours.

So, when you get that chunked, that chunk can then be sent up to your object store. However, this recent one that has not been chunked, that's the stuff that's being currently written to, so you need quick access to that. So, that's very stateful, whereas that chunk can be up there and be the, quote, unquote "stateless" in S3.

Same thing on the read side. We're going to keep anything that's been queried recently we're going to keep that down, and that's when the long queries are happening is when you're having to go past that short time frame. And you're querying for a day, two weeks, you know, a month or longer, then it's going to have to go and pull and this down. And that's either landing, like Justin is saying, on a disk, or there's also caches in the Mimir stack as well to have that in memory, at least for the time being until that needs to be flushed.

MIKE: So, since there's rapid access memory, you say it’s tricky to send out to something managed [laughs].

JUSTIN ELLIS: Yes, right. Yes. Any applications like AI applications need to read a lot of data at a specific time. So, there are unique systems that do require a lot of disk input, output in kind of a quick fashion, and those ones are going to be a little difficult. You're going to have to think in more of a distributed systems. And distributed systems are hard to manage. And going forward, you're thinking about disk. You're thinking about quorums. You're thinking about ordering and all those kind of complex problems.

Some of those are solved. Some of those are still yet to be solved. But we do really do rely a lot on kind of managed services, especially for distributed systems. But we also can run distributed systems in-house as well, and they are just a little different to manage, a little bit more difficult.

KYLE: And we have kind of a new technology, new-ish technology that's coming down the pipe, where we're also investigating the utilization of streams. Specifically, we're looking at Kafka, you know, what can that provide us? Because you can just assign multiple listeners to a Kafka topic, and then get the stream as you need it, and that can make things stateless as well.

EDDY: I was going to say, I thought maybe as we're inching closer to the end of this...we touched a little bit about holy grails in DevOps, so I want to elaborate a little bit on that. And what are some things that you'd like to have in DevOps that isn't quite possible yet but in an ideal world for a holy grails system?

JUSTIN ELLIS: That's a great question. One thing on my wish list, I think it is possible today, but there has been a lot of movement in what's called an internal developer platform. This could be something like Spotify's Backstage, which is an open-source project, which kind of provides the fundamentals of an internal developer platform.

What that is is, basically, instead of having CLIs and having web services like GitHub Actions and Jenkins, you have kind of, like, a central kind of observability coordinator that can tell you the status of your services and your applications. It can tell you what build is out there today instead of having go to a CLI and kind of integrating all the data from the systems to give developers access to be able to see things and perform basic functions. So, I think that is possible today, but we're not there quite yet. Dynamic environments was the other one that we want to get to. And then, that's all I can think of right now, but I'm sure there's probably 10 more [laughs].

JUSTIN CALDWELL: So, I was going to ask, if you guys could just bullet fast, just list through the tools you guys use every day that you love.

JUSTIN ELLIS: Yeah. So, Terraform that's what we use every day, and we do love it. Terraform really is a great system going forward. Alternatives to Terraform are CloudFormation. And there's some even newer ones that work a little bit differently, maybe within Kubernetes, like Crossplane. But I still think Terraform is great for infrastructure as code, and something that's awesome.

Another tool that we use heavily and we use every day, and this is kind of our orchestration framework, and that's Kubernetes. So, Kubernetes really does have some great fundamentals. Kubernetes comes out of Google who originally how they managed their system was a system called Borg. And Kubernetes is kind of an offshoot of Borg and has a lot of the fundamentals that you want when you're running a system as far as ingresses, deployments, applications, and scaling those out. Those are kind of my two favorites today. Kyle, I'm interested to hear what your favorites are.

KYLE: I'm going to surprise everybody here and say, in general, the Grafana Labs tools, specifically. It's a great product. Most everything that they have is a great product, and they are open source. They serve their open-source community really well for monitoring. It has been needed in my tool belt. I would also throw out Prometheus. Prometheus is something that was needed, a high-performant time series database that was needed. To go along with Justin, tools, Terraform, and Kubernetes.

I also find that I have to throw out my IDE, and that's VS Code. VS Code is something I could not live without. I've tried to use some of the other IDEs out there, and I'm not as comfortable with them. So, there's that. And I'm trying to think if there's anything else. Oh, Lens is a visualization tool to see into Kubernetes. Right now, I'm currently using OpenLens, the open-source version, but any visualization tool to see into Kubernetes and be able to manage that. AWS' offering is getting a bit better. But I still really rely on OpenLens in order to do my day-to-day.

JUSTIN ELLIS: And I'll just add, obviously, Docker and containers just in general.

KYLE: Oh yeah.

JUSTIN ELLIS: That's kind of our bread and butter. I know that developing in Docker sometimes can be a pain. But for what we do when we manage clusters and applications, Docker really just gives us a single target to build towards, and then we can basically put that anywhere. And then, I'll just also say AWS, generally, GCP I really like as well. These cloud platforms have become so mature and so great.

It does require a lot of kind of...this is why I don't want developers to necessarily have to understand AWS. When you go into the console, you see all the tools that they have and all that kind of stuff. So, I've always wanted to create kind of an opinionated framework about what tools we kind of use going forward. But AWS is just phenomenal. I really like GCP as well. Modern cloud platforms are really just pretty awesome and just a game changer. And something so fundamental I don't even think about, you know, AWS has only been around since 2008, or something like that, but really just super fundamental to what we do.

MIKE: And that does bring us, I think, really well to a good conclusion. We've talked about a lot of things, about how to bring legacy systems forward, about the critical nature of having the easy plug and play [laughs]. It does what you expect it to do. You have a configuration file and done, so you don't have that complex interface. And then, the issue of reducing complexity all around and how that applies in a lot of different parts of our work.

Any final words particularly from the platform engineering folks that you'd like to share as people finish this podcast?

JUSTIN ELLIS: I will just say, I think it's a really exciting [laughs]...I know it sounds weird, but it's a really exciting time for infrastructure with this new AI tooling and increased capabilities there. I'm just really excited about the future and how we kind of manage infrastructure going forward.

I think a lot of those fundamental pieces that we were just talking about are now in place. There are some really impressive open-source projects. I'm just really optimistic about the future and how we manage stuff going forward. I think we're just in a great spot in that these ecosystems are going to continue to grow.

Kubernetes has kind of spawned what's called the Cloud Native Computing Foundation, which is kind of very similar to the Linux Foundation. But if you go there and you look at the landscape, and all of those tooling, and SaaS, and frameworks, and open-source projects, it's just really an exciting time in infrastructure, and I think we're in for a lot of improvements in the near future.

MIKE: Great. Well, if you're listening [laughs] and you'd like to get here, it's possible. There's so much opportunity to dramatically improve the experience and return on investment of your infrastructure by approaching with some of these tools that are available. And kudos to the team here for having made that happen.

JUSTIN ELLIS: Thank you.

MIKE: With that, until next time on the Acima Development Podcast.

Cross-team communication is one of the biggest challenges in software development, and the latest episode of the Acima Development Podcast dives deep into why it’s so difficult—and how teams can improve. Host Mike kicks things off with a humorous but insightful story about botching his son’s haircut, using it as a metaphor for what happens when people act alone instead of seeking input from others. The panel, featuring experts from different departments and organizations under the same parent company, shares real-world experiences of miscommunication, siloed teams, and the consequences of not aligning on tools, expectations, and processes. The takeaway? When teams don’t collaborate effectively, things go sideways—fast.

One of the key topics discussed is how different teams use different communication styles and tools, often leading to misunderstandings. Some prefer instant messaging, while others rely on email or ticketing systems, creating friction that slows down progress. The panelists agree that social capital—trust built over time—is crucial for overcoming these obstacles. Successful cross-team collaboration often comes from learning the hard way, through repeated trial and error. They emphasize the importance of setting clear expectations early, aligning documentation processes, and making an intentional effort to break down silos. Without this effort, teams risk wasting time, missing deadlines, and struggling with misalignment between business and engineering goals.

So, what’s the solution? The panel suggests several best practices, including embedding engineers into other teams temporarily to build rapport and understanding, structuring projects around clear milestones, and ensuring leadership actively supports collaboration. They stress that no matter how well-designed a system is, success ultimately depends on human connection and clear communication. Whether working in small teams or across an entire organization, the key is to acknowledge the real cost of collaboration upfront and commit to the effort required to make it work. By embracing transparency, proactive planning, and direct engagement, companies can turn cross-team dysfunction into a productive, high-functioning environment.

Transcript:

MIKE:  Hello and welcome to another episode of the Acima Development Podcast. I am Mike, and I am hosting again today. With me, I've got a great panel, and it's going to be real. I'm going to go a little bit deeper into the introductions than I sometimes have because it's going to matter for this discussion.

First of all, we got Kyle with us from the DevOps team. We've got Dylan from Rent-A-Center, and it's like a sister business, I guess you might call it, you know, we're different lines of business, same parent company. And it's great. We're going to want to do more of this. Our parent company is Upbound, and we'd like to get more participation from across Upbound so we get diversity of opinions.

So, Dylan is with us from the Rent-A-Center side. So, it's great having Dylan. I'm probably going to ask him a lot of questions. We'll get to the topic in a minute, and you'll know why [chuckles]. We've got Molina from application support. We've got Ramses and Dave, who work with lease origination. We've got Eddy who works with our partners, merchants. And we've got Will, Will Archer, who works for an unnamed third-party [laughs] retailer.

WILL: Technically, I’m an independent consultant, but I work for a large retailer.

MIKE: And he’s with us, as he usually is as he usually is [chuckles]. And that’s the group we’ve got today. And I’m going to start, as I like to do, with a story, a story of...in this case, it’s personal humiliation [laughs]. So, a couple of weeks ago, my four-year-old...and I can tell a lot of stories about my four-year-old because four-year-olds provide lots of opportunities for stories. Well, he decided --

DAVID: Content generators.

MIKE: They are. He decided he was going to cut his own hair [laughs], and he did, got a big chunk out of the front of his hair. But it was probably still going to be salvageable, you know, I pulled out the clippers and trimmed around the sides, and it was okay, you know, it was okay. It wasn’t great, but it was okay. And I was talking to my wife about it, and we were working together on this endeavor, this creative endeavor [laughs] of trying to fix the disaster he’d made of his hair.

And we looked at it, and we were like, “You know, it’s okay, but it’s not great [laughs]. We’ve got to do something here.” And my wife suggested, “Well, maybe it should be shorter on top. We need to do something.” And I’m like, “Yeah, you’re right,” then we kind of left it. We kind of left it. Something came up. We got busy. And later that day or the next day, I was trimming my own hair [laughs]. And those of you who haven't seen me, I don't really have hair. So, when I say that, I mean I take off the few scraps that are left around the side so it doesn't look like I'm faking it [laughs] and trim my beard.

I was doing that and in walks my four-year-old. And I've already got the clippers out, and then I said, “Oh, well do you want me to trim your hair?” And see what I did there? I didn't go and ask anybody. I thought, you know what, I'm just going to do this myself. This will be just fine, right? So, I put the comb that you put on the clippers to, you know, make it longer, the one that I usually use for my beard, number four for the longest part, then I use, like, a number two. None of this is important, unless you...but if any of [inaudible 03:40] clippers, you know what I'm talking about.

So, I put all this together, and I started trimming, and I started trimming just a little bit. And I thought, wow, that's a lot shorter than I thought it was going to be [laughs]. And I had a moment there to try to rescue what I had done, and I didn't [laughs]. I didn't. I didn't bring in the team, didn't bring in my wife to say, you know, “Hey, what do you think I should do here” Didn't bring her in initially. I just thought, hey, the situation's here. I'm just going to do something about it, and I just went for it. And I finished, and it wasn't good. In fact, it was terrible [laughs]. It was terrible.

And my wife sees it, and she's like...frankly, she cried [laughs]. It was that bad. I felt so bad. I felt so bad. And that was entirely my fault. There were several places where I could have fixed this problem, and I didn't. If I had brought her in to communicate upfront before I did anything at all, she would have probably given me good feedback. If I had started and saw that there was a problem and communicated, I would have gotten good feedback. But no, I just went ahead alone. I thought, I can do this, and boy was I wrong. Luckily, it's been growing out some, and she came in and fixed it some, and it's better [laughs]. It's okay.

WILL: [inaudible 05:05] give him a Mohawk. Like, a mohawk it always sounds sort of plausibly on purpose.

MIKE: Yeah, it's close. It's almost a mohawk. It’s kind of a mohawk-ish sort of look with a bit of a pointing back. It's growing out. It doesn't look so bad anymore. There's recovery. It was a bad incident all around, and a lot of it is my fault, which is my story of personal humiliation to say, you know, maybe you should involve other people on the team, people on other teams, early and well in your communication process before you do something that's hard to fix. And we're going to talk about cross-team communication today and all of the error cases in that and how to make it better.

That's why this panel is such a great one. We've got people across lines of business. We’ve got people who work in different business, right? And we've got people from various different teams that have to work together in order to talk about this recurring, crucial problem that we have in engineering. And in our pre-call, we talked a lot about this that it’s...this is a [inaudible 06:11] problem, probably in software engineering, and it's not a technical problem. It is a human problem, and it's universal. There is the problem statement [laughs].

And I'm sure you all have some stories about problems that you've run into like this in software, and problems, suggestions, as well as, you know, some things in between, problems, suggestions, and just ideas, right? I'm going to stop talking and let somebody else talk for a little bit and take things in their own direction. So, what are y'all's thoughts?

WILL: You know, for me, I think the most interesting thing, I think, cross-team dysfunction is almost the norm. I think that's almost the default. And so, I'm a lot more interested in hearing about stories when things went well, when things went right. What did that look like? What did that feel like? How were those successes set up? To see if there's through lines and threads that we can sort of, you know, take and put in our back pocket and maybe save for a rainy day, because it's only a matter of time before we're going to have to get some teams together and come up with a good solution.

MIKE: Well said. I've got some suggestions there, but I'm not going to jump in yet.

EDDY: I think it's just important to be honest about what broke and what worked. I think a lot of us like to add sugar saying, “Oh yeah, we met the deadline, you know, it was handed in at the time that we were expected to. There were some bumps, but it's fine. It ended up working.” And that's cool, but I think a lot of the times being able to reflect and do a postmortem and just really be honest, you know, like, what went wrong, how not to replicate that, I think is a really good step.

WILL: Did I misunderstand the question? Are we talking about incident reports, or are we talking about cross-team collaborations that went good? I mean, I'm maybe asking for case studies of, you know, this project and these two teams got together, and it went well, and this is how and maybe why.

MIKE: Yeah, you know, you're right. I think we're talking about cross-team collaboration. I think that Eddy would suggest [chuckles]...well, maybe there is no such thing as a -- [laughter]

DYLAN: I almost have to agree with Eddy here that the best teams for cross-team collaboration often have a long trail of failures behind them, and they've learned to work well together through that.

KYLE: I've got a bit of an example for this. As my organization, we kind of merged together to the greater organization. We both standardly used different tools for communication, we'll go with that, different chat communication tools. And I think this is where a lot of issues come in is people always have preferences on how they use their specific tools. Each tool will breed a different culture. And that was one thing that I wasn't understanding when I was going in on my project with communicating with the other portion of my team over at RAC.

Finally, I sat down with one of the RAC individuals, and I was just like, “Hey, I keep pinging you. I keep pinging your team. I'm not getting any response. What am I doing wrong?” And we had to communicate to say, “Oh, well, we don't use the chat client that way. We use it this way, and that's why we're not seeing any of your communication.” And it wasn't until we had that direct one-on-one communication that I learned how they normally communicate. And I feel like that's how you kind of have to go is one side or the other has to kind of pivot to figure out, like, how does the communication have to happen, and who can pivot, and maybe both sides need to pivot. But who needs to pivot so that that communication channel can stay open?

DYLAN: Absolutely. That's not only with text communications or email communications as well. If two teams are sharing process documents between each other, they may have thought they agreed on some process. And one team says, “Okay, we'll put it in our shared drive for our Confluence,” and the other organization does not use Confluence at all.

KYLE: Oh yeah.

DYLAN: They don't have the same document in the same place, and they don't know what the process is later on.

WILL: Oh, brutal. Man, people from the business side keep on trying to make Teams happen. Teams is never going to happen [laughter]. No one's going to use Teams. It's never going to happen. But they keep on just coming in. They're just like, let’s put...Slack. Why are we paying for two? We get Teams for free.

MIKE: [laughs]

WILL: Don't do it.

MIKE: It's true, and there's a cultural...we should probably do a whole separate podcast on communicating with the business side [laughs] because that cultural divide is deep and vast [chuckles] and hard to bridge.

KYLE: Even the culture of communicating by email that's another one that it’s...engineers, I feel like, and maybe I’m wrong here, but engineers, I feel like, they've developed a culture of communicating over tools like Slack, or even Teams, or Google, even. And there are other lines of the business that prefer email. For engineers, really speaking for myself, that seems to slow the process down to a snail's pace, and it's a hard thing to get over.

MIKE: Exactly [chuckles].

WILL: I've always wondered what it is. Every engineer, I think, I've ever talked to has it: the aversion to email. I don't know, I haven't unpacked it, and I have thought that I should accept it as a given. Like, no engineer will email you, unless extreme pressure has been applied to them [laughs].

EDDY: Well, I think part of that is because, as an engineer, you're expected to hit certain deadlines, and so you need answers pretty quickly. So, leaning towards instant messaging, like, it's even in the word, right? Like, you get messages instantly versus having to wait for emails to come. I'm not saying that one's superior than the other, but I think we work in an environment where we're trying to fetch information as quickly as possible. So, having direct communication with that person via a DM, you know, does wonders.

WILL: Yeah, fair.

MIKE: It violates, like, a psychological contract that we have with each other. When you're talking with somebody in real-time, the timing matters. If somebody pauses for a while, you think, oh, what did I say wrong? It breaks your connection. It breaks the emotional connection badly, and email guarantees that. There's no way you're going to have real-time communication over email. I mean, maybe you could, but it's super awkward. Whereas if you're directly messaging somebody, you feel like, yeah, I'm almost in the same room with them, and it's a totally different dynamic.

KYLE: [crosstalk 13:28] you’re in the same room [laughter].

DAVID: There's a trade-off there that didn't become clear to me until somebody dialed it all the way to 11 in a different direction, which is there's a phrase that came out of agile, back when agile was still called (XP) Extreme Programming. One of the phrases --

WILL: [inaudible 13:50] XP.

DAVID: Oh yeah.

WILL: [inaudible 13:52] to do with that.

DAVID: One of the phrases that they circulated a lot back then was, when you're pair programming, they'll tell you “Never type when you can speak. Never speak when you can point,” right? Literally, it's, like, how much information can you communicate in the smallest amount? Now, pointing is very ambiguous. You have to be in the same context. You have to be physically present. There's all kinds of stuff. But if you have that context, you should use it, right? The last thing you want to do is sit down to pair program with somebody and they pull out their phone and Slack you a message, right?

MIKE: [laughs]

DAVID: Unless they're sending you the shockie to get into something. You’re just like, okay, yeah, I do need that written down, but, yeah, it doesn't make a lot of sense. So, when you see the immediacy and the ceremony...and sometimes you want the ceremony, sometimes you want a Jira ticket documenting that I have opened this issue with you on this date, and this was my request, and y'all better do something about it. Versus just, like, “Hey, we're going to let [inaudible 14:54]. You want to come?” Like, you wouldn't do that with Jira. You wouldn't do it with a ticketing system.

DYLAN: Your Slack example highlights, too, that switching from pair programming to Slack that's a context switch, and it can be jarring. Switching to email is a much bigger context switch because you're not just drafting an email. You're confronted with your whole inbox, and you see a subject line from someone and think, oh, I need to switch gears and reply to this now, too. So, psychologically, it throws you off.

DAVID: Serial access, like, accessing one thing at a time, like, as in messages coming in over time into your inbox, is one of the most inefficient ways to do something. And how is that serial port being used? Every marketer in the world is jamming messages into that serial port. It's...yeah.

EDDY: I want to be completely transparent, and I want to go on out here. I'm hoping other people can agree. But switching between two different applications that have two different interfaces kicks my butt every single time, right? It's crazy. It's crazy.

MIKE: Well, it breaks your flow. Anything that breaks flow prevents effectiveness, right? I think we've talked about flow in another, you know, another podcast before, but getting into that state is hard to get into and if you break it, you lose it for a long time. And anything you add that makes it harder to get into that flow just adds another layer of difficulty.

WILL: You know, for me, honestly, when I think about sending or, God help me, receiving an email, I feel like it is the least likely thing to get me any answer that I need. It's almost always something that I don't want to do that also doesn't matter. And my best-case scenario is I send somebody an email, and I'm like, hey, I have this problem with this vendor tool, and it's doing this thing. And please reach me at an actually productive communications channel so I could work it out [laughs]. The email is sent in getting a Huddle or a Slack message from somebody I don't know well enough to ask directly.

EDDY: I don't fully expect people to really have the answer off the top of their head, but just out of curiosity, like, just a rule of thumb, how productive has the experience we’ve [inaudible 17:27] with emails? Like, have they actually provided any sort of value?

DAVID: I'm going to say yeah.

EDDY: I’m surprised.

DAVID: For me, it's a receipt. You've got something, like, I will pull up my inbox and basically say, “Uh, where's the receipt for this thing? Or where's the confirmation code that I got for this?” And I love that. I've just told you something, though. I've just admitted that I am not an inbox-zero kind of guy. You open my Gmail, and the unread messages is well into five digits. I remember when I passed 20, 000 unread messages and still climbing. And it's because I open up email, I scan, I’m like, yeah none of that's...and then just move on. You know, why bother?

I don't deal and set aside and finish my emails. If that were my working system, email would have a completely different flow for me. For me, it's much more asynchronous. There are times when I will go for a day or three without checking my email, but anybody who knows that I'm living in that asynchronous environment knows that well, we'll send this to him. It will get to him eventually, but it will get to him. Whereas anybody who's tried to Slack me knows that I frequently have my notifications not working the way they need to. And so, I think everyone in this call has gotten a message [inaudible 18:43] “Dang it, I missed this notification. Sorry, where were we?”

MIKE: Well, email, what's right there in its name, replaced postal service delivery, some sort of, you know, human delivery [chuckles] involving somebody grabbing a written document and carrying it or maybe fax [laughs]. And, in that respect, I think, and this may be why it's popular on the business side, it’s like, wow, this is so much better than waiting for the courier [laughs], and maybe it's because people like the paper trail that you get.

But it's not that you don't have that in the other tools. I'm not saying it has no value. There is some value, but we're talking here about collaborating across teams, and we've kind of diverged a bit into the tools. You've got to set the ground rules first. And if your engineer is trying to coordinate on something, email is probably not the way to do it.

WILL: I think email is the industry standard for communicating with people you don't know. I've got a bunch of people from completely different departments. I don't know any of them. I don't know how they work. Are they Teams people, Slack people, whatever people? And we're going to go out and we're going to create, effectively, a document of record, right? So, people you don't know, like, when you're sort of managing up, you're talking to people sort of further up the chain. You can't just Slack, you know, you can't just hit people up like that. You send an email where it’s like, all right, I sent it, and you're going to read it or not. But I sent it, and there it is, and if you have a thing, then you can get me. Anyway, it's not going to be a Slack thing.

And I think that right there kind of segues me into maybe one of my answers in terms of the original question I asked which was, what's a successful collaboration look like and feel like? And, for me, when I have collaborated with teams successfully, and, like, the way I've collaborated with teams successfully, is when one, or preferably both parties, have some sort of social capital with each other.

When I know you, when we've collaborated before, when I know, you know, who you are, and how reliable you are, and whether you are going to, like, whether there is a quid pro quo, whether I'm not just getting bombs thrown into my sprint and then I'm never going to see you again, like, are you a pure [laughter], unadulterated problem for me, or are we collaborators where this is going to go both ways, and I'm going to have a question for you, you know, six weeks or a month down the line and, you know, you're going to scratch my back?

And I think sort of cultivating that is one of the bigger hidden, you know, tidal forces that as you get tenure in an organization, you start to build up those connections where you know people, and you know people who can help you out, who will and who can, versus people who are just going to create problems for you and you must [laughs] be coerced into helping them because you know it's just going to be a time sink. And it's not that you won't. It's not that you won't. If you have a problem, I'm going to fix your problem, but I might solve some other people's problems first, or more thoroughly, or more promptly, because, you know, it is a social capital situation, and not everybody's a good actor.

MIKE: This goes right into what Dylan was saying right at the beginning, which was the collaborations that work well are usually the ones that you've had a few tries first [laughs]. You have to establish that relationship first. And, it goes, you know, if you don't establish some ahead of time, you don't have it. I don't have a relationship with you. I don't trust you. I don't know you. This isn't going to work.

EDDY: Well, it's kind of like I instinctively thought of the analogy of a professional sports team, where, like, you just get a bunch of people who claim they're really good, and they are good in their own right. But if they aren't willing to “play ball,” quote, unquote, things are going to go tough. You have other people who want to do it in one approach. You have another person who wants to do it in a different approach.

They're both valuable approaches, but unless they align, and there's trust between each other, you know, it's going to be rough. It's going to take a really hard time to find that happy ground. But once you do and you've played enough games together, eventually, you start to understand, you know, how other people think, you know, it's easier to come up with game plans when everyone understands the way they play, right?

MIKE: Well said. We don't need a bunch of heroes. We need a team.

DYLAN: That can be both very frustrating and very valuable. Often, when two teams are collaborating, one team is managing software infrastructure, some kind of system that is much more dangerous to mess up than the other team, and often they have more robust processes than the other team. And when the other team joins, it can be very frustrating because they have to go through a ton of red tape to do things. There's a bunch of process they haven't done before. They don't really get why it's important. They haven't been there at the incident management meetings. But being able to integrate that process into themselves also makes them perform better as a team.

WILL: Yeah, and, honestly, a lot of the time, almost all of the time, the collaboration, like, within the teams it's not equal priorities, right? There's a big thing for my team, and I've got a big deliverable. I've got a big project, and this is my thing that I'm going to get out. And it impacts and interfaces with other teams. It's like, oh, I need you to update this API, or I need you to provision some hardware for me, or what have you, right?

This is my main thing, and this is, like, something that you're doing on the side. And that can be a source of some amount of friction because I care a lot, and you care to the extent that, you know what I mean, for better or worse. And that's one of those critical social capital times because I can just be like, “Hey man, can you just get this thing in for me?”

And it's not a big piece of your sprint, you know, no one really is chasing this thing down, unless I go to my manager, and then they go to their director, and then their director goes to the other director, and then the other director goes to the manager. And they're like, okay, come on, let's get it done. There's all those steps, right? That's collaboration across teams, but even as I said it, you can see where it's just like, oh, it sounds like a crime. But if I'm just like, “Mike, do me a solid, you know, help me add this Boolean flag onto my JSON, please, so I can get this thing out, then it's a matter of the day.

MIKE: Well, yeah. You tie it in...I've been thinking about a couple of successful projects I've seen. I'm thinking about a couple of really successful big projects that were related to each other. Well, they weren't related to each other. They're totally separate projects that had a pattern that was related to each other. They followed an almost identical pattern in that they were very successful, and I think for the same reason. And it ties into what you're saying about, you know, you've got to talk to the director. They talk to somebody, you know, that horrible chain of command that makes communication just grind to a standstill and play the telephone game [chuckles].

Where it's really worked, where I've seen this work is when you started with not business stakeholders...business stakeholders, at first, figure out the requirements, and then you have the product team come. You know the requirements really well. That product team that knows the requirements sits down with the engineering leaders who have something to this project, who actually know what they're talking about. And you get the group, small group. I'm not talking 10 people; I'm talking maybe 2 or 3.

The people who are most affected by this project get together with product, and they hash out together the order of operations to get this big project done, so you have a series of milestones. And that gives a roadmap which you can then put timeframes on and then start filling out the specific requirements and teams associated with each milestone. I’m seeing there's a lot of planning involved here, you know. But you've taken this big problem, how do I get from here to the other side of the world, and you've broken it down into smaller pieces. Well, how do I get from here to the edge of my state, or how do I get out of the driveway, right?

You’ve figured out the major milestones along the way, and then you start functioning at the level of those milestones, and you know the teams involved. And then, bring those teams in to solving that milestone, you get that milestone done, move on to your next milestone, and so you have the map right from the beginning that everybody can see. Everybody sees the roadmap. They see the milestones along the way. And then, they pitch in to get each of those milestones done, and then the next milestone. And each of those milestones has the teams clearly identified and the requirements clearly identified to get that milestone done.

So, there's some commonality here, one is the really well-documented and well-created roadmap. You can't just have somebody throw something out there. It has to be agreed on by the engineers who know what they're talking about together with product, so you actually get the right business requirements so we know what you're actually building.

And then, you've broken it into those milestones. I feel like those milestones are critical to make it work because then you're not looking at something so big that nobody cares. You're looking at pieces that everybody can engage with. And then, each of us has a clear deliverable that everybody knows how to get solving. Because then you can get engineers thinking, well, how do I solve this problem? You get each of the teams that care about it engaged with that problem, and it matters to them. Then they can collaborate, reach across teams as necessary.

And before you actually have them collaborate, the teams get together and define the API boundaries. How are we going to communicate with each other? That can only happen...you only get a start with those API boundaries so that they can code to that, rather than not knowing what they're doing, if you've had the road map, and you've identified the teams, and you said, “We're going to get together first.” And that's a lot of overhead up front, and it requires product who's invested and the engineering leaders who are invested in working together. If you can get that, I think you can make it work. But there's lots of places that can fall apart.

WILL: I love it.

DYLAN: The way you put it, setting the groundwork is itself an engineering project.

MIKE: Absolutely [laughs].

WILL: I'd pitch you on it. I mean, I love everything you're saying here. I would pitch you on another thing as well and maybe a tactical, small unit level. I really love engineering embeds, where okay, these two teams are collaborating. And, generally speaking, both teams are not fully tasked. But more often than not, I think 80% of the time, most teams are not fully tasked, but it's like, okay, this team is providing this, you know, subcomponent for this larger project. This is our main baby, and there's this subcomponent. Lend the engineer out. Give an engineer to the other team. They're going to be embedded for as long as their piece needs to be there. They're going to do stand up with them. They're going to sit there. They're going to build that social capital.

And most critically, I mean, maybe two things that I think are going to happen that are really important. One, I think it's very easy when you have a sort of like a [inaudible 31:15] engineer for a couple of days, you know, half of somebody's sprint cycle. It's very easy to underestimate impact to productivity with cross-team friction and collaboration and getting things ready to go. And so, when you're dealing with stuff as an embed, this guy belongs to this team, you know, for such and such, you're going to take that more seriously. And, honestly, I think when you have somebody else's project, it's really easy for that to get kind of short shrift.

And it's also sort of...how do I put it? No, actually, those two things it takes a lot more friction than people get credit for. And this will get people to take it seriously from a planning perspective and fully commit from an engineering perspective, like, I am going to get this thing done. This thing is my thing, and I know these guys. They're not just strangers throwing work over the wall at me, and it's hitting me on the head. They're my team for now. Temporarily, they're my team, and this is our project.

MIKE: Oh, I like that a lot. And it's a little different than setting up with people who are, like, a tiger team, where you say, “Okay, this project, we're going to take somebody from each of these teams and put them together and isolate them from everybody else.” I have seen that not work so well because then you've deliberately isolated people. You've shut them off from the rest. But the way you put it, where you're loaning somebody, where you're sharing rather than isolating, I think that works out really well.

DAVID: I think I've mentioned this before that I've done that on teams proactively without a project, where it's not like...we literally called it the hostage exchange program where we would take a sprint, and we'd basically say, "We want one of yours. We're going to give you one of ours." And after two weeks, we'll go back. And so, literally, the whole point is just to pass this person around the team, let them soak in the environment. They're not going to be super up to speed, but that's okay because your guy over there isn't up to speed either. But after two weeks, you swap back.

And it's so beneficial to be the person who was out on the exchange because now anything that happens with that team's product, everyone in your team will turn and look at you. And you're like, “Hey, you were working on that team, you know, last month.” And the really cool thing is when you want to go drop a ticket on somebody's desk. Like, they've got a ticketing system for how they're going to handle their stuff. But before you do that, you just pop open Slack and you're like, “Are you guys on Kafka 2.14 or 2.15?” It's not worth stopping the whole system.

And the other person, because they've sat next to you for two weeks, they're like, "Oh yeah, 2. 7. We're actually on 17. You're way behind." "Cool. Thank you." And if I need something beyond that, I'll go do the ticket and do the thing. But there's that moment, like, water cooler connection of just, like, "Hey, buddy, where's this at?" and being able to do that.

EDDY: Yeah, Dave, but then you run the risk of becoming the git blame of that, right? And then, everyone just comes up to you and says, "Hey, you touched this code before. What does this do?" You're like, "I was just there. I just --"

DAVID: The reason ticketing systems exist is that people don't have a good defense against becoming the git blame guy, and they're like, I need to get work done. I need you to stop interrupting me. Go talk to the tickets. And that's, unfortunately, that's just replacing one dysfunction with another, but yeah, you're right. That’s a very real problem, and it's bad when it's...

I guess if I'm yelling at you "What version of Kafka are you on? Drop what you're doing. Pay attention to me," I'm not building a team with you. I'm not revisiting a good interaction. It's got to be that water cooler, like, social capital like we've talked about of like, “Hey, how are you with this?” Because half of those conversations are equally likely to begin with, "Hey, is your kid feeling better?" And that's a social capitally valuable, you know what I mean, positive social capital in that communication, because you care about the human on the other side.

WILL: I guess what I'd say, to Eddy's point, when you have that guy, you're the ambassador. You're the liaison between these two teams. You've been over there. You've done the hostage exchange. You have some connections. You have some background. You can get things done. And your job is constantly being looped in to, like, oh, this is going wrong, help me. This is going wrong. Help me. This is going wrong, help me.

A, that is a strong signal that the hostage exchange program needs to not just be continued but expanded, right? And B, think about how badly these projects would be going without you. You're saying, "This is bad. We shouldn't do it.” And I'm saying, "This is good. It's saving your ass, and you need to do a lot more." It only becomes an anti-pattern when you have this unmet demand such that your concierge, your ambassador is just getting dog piled with requests for support from these other teams. That's, I don't know, just another way of looking at that. It's not too much; it's not enough.

DAVID: You could have both.

DYLAN: In a crisis, you end up being very happy that you have someone like that on your team because you don't have to play a game of telephone. When, you know, something goes wrong, support figures out. They call a developer; developer realizes it's a DevOps problem; they call DevOps. If the developers also worked with the DevOps team, then they can start looking at the DevOps side, too, and also call DevOps in parallel, and it saves you time. And if you need to respond to an issue in 5 or 10 minutes, then you don't want to lose that 5 minutes of waiting for someone to respond.

EDDY: So, you're saying when things are on fire, don't use emails.

WILL: No, you don't.

MIKE: [chuckles]

DYLAN: You know, that's important, too.

EDDY: [laughs]

WILL: No, nobody sends an email on fire. They have a guy. And they're like, I have a name, in my head, of somebody who's going to put this fire out. And, you know, from a career development perspective, I'll just say that in my career, like, consistently being that guy has been really good for me. And if you're like, “How do I advance as a developer?” being the name that people can call when things are going bad, it's really good for you.

MIKE: You know, talking about the email, Molina, who was here, had to go, but she does application support. So, she handles the bad stuff. When things are on fire, she's the firefighter [laughs]. And she doesn't instant message me. She doesn't email me. She texts me.

WILL: [laughs]

MIKE: And that matters, right? She knows that she can send me a text message, and I'm going to get immediately on that problem, not just me but a couple of other people that she knows, you know, are critical to solve this problem. And I'm going to jump on that problem because, you know, that's going to happen.

Now, I don't have everybody text me, right [laughs]? But the person who I know is doing that critical firefighting, well, yeah, you're going to get the most direct, and engaging, and immediate sort of communication that I have available, or a phone call even, right? Her and her partner there in application support they'll both do that. Yeah [laughs], that relationship matters.

WILL: Yeah, that's the gig, man.

DAVID: And it's respected, too, though, right? You're not going to get a text message at 3:00 o'clock in the morning saying, "Hey, would you fill out your TPS report sometime in the next week?" [laughter] No. I started work before the internet existed. And we had a saying back then, which was “When they come find you in the bathroom, you know it's serious.” And this is the modern equivalent of that.

WILL: Yeah, what do you call it? I don't know. I keep my phone number on my Slack. Nobody calls me if they don't need me, but it's rung before [laughs]. That's just the gig.

MIKE: So, we've talked about a few things, right? We've talked a lot about channels of communication and how some are better than others. And the more immediate the channel, the better. We’ve talked --

WILL: Not necessarily.

MIKE: Well, that's true. You're right that having a...You're right. I did not state that well, that immediacy is good in many cases, but there are some cases where, I don't know you, asynchronous, keep a record of this is the right answer. So yeah, my summary was a little off.

And we've talked about some successful projects where you have social capital and the deep importance of social capital, both at the, you know, leadership level, at just between team level, any way you can possibly establish it that's critical, and also, the planning. The actually doing the engineering project to build out your milestones, to build out the project plans, and get people engaged before you jump into the project because if you just jump into it and I hope it's going to work, well, it's not [chuckles]. What have we missed?

KYLE: I was just thinking about kind of expanding on what Will had pointed out earlier about integrating a person into the team. And I was thinking...because I think a lot of the examples are, you know, devs from one team going and being on another dev team. And I was thinking about my history in QA, and most of my QA jobs, you had a QA team. And you worked with just all of the developers or, you know, all the support, all the developers. And whenever there was a bug that you needed to have questions answered, you got up, you walked over, and you had to confront that person. Now, sometimes, that person was somebody you knew. Sometimes it was, you know, the grumpy Java developer, and you just didn't want to talk to them that much.

However, I was in another position, which was very jarring, for me, and I didn't understand it when I first got into the job, and this was actually at InsideSales. They distributed the QA individuals. We were on a QA team, but the QA individuals were integrated with the developers. So, now I'm in with the developers. I'm testing this specific product’s app or this specific product. And then, I'm going to their planning meetings. I know all of their requirements. And then, when we're getting together as a group in QA, it's, okay, well, what's happening with this service? And that's kind of how the communications are getting distributed that way.

But that was probably the fastest form of communication in a development style that I'd ever experienced, and that was because as a QA individual, I was part of that development team. Even though I wasn't necessarily directly involved in the code, I was part of the team. And that's something that I don't know that every company gives that kind of credit to their QA or QE engineers, that they're actually part of that development team. And so, I think there's a bit of slower communication because of that. That's just been my experience, but yeah, it's something to add there.

EDDY: I can attest to coming from QA.

DAVID: For me, this all seems to break down to context, right? It's like, if I'm up on stage giving a presentation in front of a thousand people, I'm going to adopt a very general context. I'm not going to tell edgy jokes. I'm not going to make fun of people. I like to tease people. I like to be silly. I like to be fun. But in a universal context, you don't know if I'm absolutely just destroying a person willfully, or if I'm just teasing them because it's funny.

But when you're working with close people, you know, I have a co-worker that we literally did the hostage exchange, and we had a third guy on our team. And halfway through the morning, he was just staring at us, like, mouth open, eyes wide terrified because TJ and I had gone 5 minutes about “Your mom,” “Oh yeah, well, your mom.” And I mean, it sounded like a Call of Duty lobby. And then, TJ had to stop, and he's like, "Matt, just so you know, we are best friends. This is fun for us. We are..."And I'm like, “Oh, yeah, yeah. No, this is totally just a bit. We're just...” And he's like, "Oh my God," just the weight coming off of him.

But that context pays off in both directions. When you have a general context, you have to follow policy. You have to be very strict. It's your ticketing system. You're going to follow these rules. But on the other side of that, if you can move into a higher context with somebody...like with Kyle, if I write you up, "When I test this, I need you to load this app. You need this backing up, da, da, da." I give you these three pages of how to set everything up because I don't know you from Adam. You might not even be in Utah. You're on the other side. You're coming in from the Rent-A-Center side.

So, I've got to give you all the context and all the background, where other times I can hand you maybe a PR and basically say, "This fixes the tax freeze in Montana," and you go, “Got it.” And you know exactly what the problem is. You know the history. You know what to check. You know 90% of the app is going to be untouched by this, so you need to focus here. And if you're in that general context, no, you have to do the test on every single piece because you don't know what the intent was. Having that context, for me, is the key element between stilted receipt-based communication and being able to just very quickly just say, "This is what we want," and being able to strike and move at it.

WILL: Okay, so, I mean, I feel like we're all fairly aligned in that, to the degree that you can, to the degree that it's possible, it's good to have people who know each other. Or, at the very least, you have people who are embedded so that this alien, foreign team they're committed enough to your success, and your project, and your priorities to give you somebody, right? There's a level of trust there.

Like, if you're going to give me a senior developer for a month, okay, all right. I mean, you've taken me a little seriously, at least, and so, that's great. But what about when it's too big to be able to put up a bunch of buddies, right? What happens when it's too big to be, you know, something like that? It's like a cross-organization type thing, you know, where it’s this is a big thing.

MIKE: And that goes down to what I was talking about before. The times when I've seen this successful, you put in the work ahead of time to get the relevant engineering parties together with, you know, the requirements people, you can get it from product, to build out that plan, build out that road map, get the milestones in place, and figure out the teams involved. Take the time to communicate with those teams.

With each milestone, you have a kickoff, right? You collaborate together. You figure out what boundaries you need to establish. What are our APIs going to look like? You know, whatever the case may be. You can't not do that. And I'm going to [laughs]...sometimes you think, oh, I've done this before. I know what I'm doing. No, you don't [laughs]. You don't. It doesn't work that way. It just doesn't. If you don't put in that work ahead of time, it doesn't work. And I've never seen it work, I don't think.

But I have seen it work on the occasions when people really have done it right and put in that time ahead of time and held to it. It helps also if you have a project manager who has the milestones, meet regularly with all of the teams, and talk about how each of those are going. And like, oh, there's a gap here. What do we have to fix?

WILL: You know, I'm over my skis here because I've never done this. I've done, you know, management and team coordination of projects and stuff with developers I could count on my finger. But I would almost pitch people to do the exact same embed thing, except with functional organizational units at whatever level is required. Do I need to have a director and all of their teams embedded in another vice president's thing? Okay, that's that but the full-time commitment to the collaboration.

And so, whoever is overarching this thing, whether it's a VP, or a director, or a CTO, or the CEO, if it's the biggest, baddest thing, maybe, to Mike's point, maybe not on a tiger team, but say, like, okay, this is the thing. We're doing this thing. And I'm going to take you, and you, and you, and you and your organizations, and you're doing this now. You're reporting to this structure for a quarter or maybe a year, depending.

I think based on my, you know, experience with corporate organization and bureaucracy, I think that is a hard order in terms of maybe careerist internal politics. But that doesn't really put me off of it necessarily because you're just saying the quiet part out loud because that's the cost of collaboration on this scale. You're going to need to have somebody applying pressure at that level, with that intensity, or else it's going to grind to a halt, or it's going to take forever, or it's going to fail. And so, I don't know, that's what I would pitch you, you know, the embed philosophy, but you're just embedding bigger pieces. And if you don't want to embed bigger pieces, then you should perhaps question your commitment to an endeavor of this scale.

MIKE: You're saying, if you're not willing to actually create that social structure, then you're saying you don't care about the project because that's how people work is in a social structure.

WILL: Well, exactly, exactly. And so, it's some other VP's thing. And I'm a director, and I want to, you know, move up the thing, or at the very least, I don't want to work any harder. And somebody is applying pressure on me to work. And if no one's going to move me over there for as long as it takes, then you're running into some real serious operational risks, you know?

MIKE: That might be a good place to kind of tie this together. You're saying that...you talked about different scales. It all looks about the same. It's kind of this fractal thing [chuckles]. If you get people working together, it's still people working together. And getting together in the same room or the same virtual room, you know, getting together into the same structure so that they are working together has to happen. And that happens when there's two of you pairing together, or you're putting your stake in the whole company on this huge project. Well, you better have some way to communicate and be working as a team.

WILL: And it's going to have a cost, an intrinsic, undeniable, you know, yeah, it's going to have a cost, and you need to commit to paying that cost. That inter-organization or interpersonal friction it's going to cost you something. And in my view at least, and I know I'm out over my skis here, but whatever, hot take.

MIKE: [laughs]

WILL: I think being transparent and upfront and really, like, grabbing that nettle with both hands is going to get you a better outcome, and it's very easy. And we've all seen it over and over again, where somebody's like, “Oh yeah, this should be fine.

MIKE: [laughs]

WILL: This will be fine. This is no big deal.” I hesitate to think of an example where somebody has overestimated the cost of that functional collaboration. I can't think of a time, ever, you know?

MIKE: Me neither [laughs]. You can't make it go away.

WILL: Yeah, so just, you know, pull the band-aid off fast, and get into it.

MIKE: Cool .Like I said, I think this was a great discussion. Hopefully [chuckles], you walk away with some good ideas as to where to go next. And until next time on the Acima Development Podcast.

This episode highlights how unforeseen career opportunities, such as web development, emerged due to exponential technological growth, which is often difficult for humans to predict. The panel notes that AI is the latest technological frontier, poised to impact software engineering in ways that were previously unimaginable, much like the rapid changes brought by smartphones and the internet. They reflect on how technological tipping points often arrive unexpectedly, transforming society and work in unpredictable ways.

The conversation then shifts to AI’s role in programming, particularly tools like GitHub Copilot, which significantly enhance productivity by automating repetitive coding tasks. While AI accelerates workflow by handling syntax and boilerplate code, the panelists emphasize that human expertise is still required to refine and validate outputs. They explore how AI might evolve from merely assisting developers to potentially managing entire software architectures, leading to shifts in job responsibilities. However, AI’s ability to completely replace developers remains uncertain, as it struggles with higher-level orchestration and nuanced decision-making. The discussion also delves into how AI’s increasing context awareness and rapid iteration capabilities could reshape programming roles into more supervisory, product-management-like positions.

The panelists debate the long-term implications of AI on software engineering careers, raising concerns about the diminishing need for junior developers and the future of technical education. While some anticipate a future where coding is more about directing AI through structured prompts rather than writing syntax, others argue that the demand for precise human understanding will persist. They also touch on potential regulatory concerns, the challenges of AI reliability in mission-critical applications, and the broader economic impact of automation. The conversation concludes with a reminder that while AI-driven transformation is inevitable, adaptability and continuous learning will be crucial for developers to thrive in an evolving landscape.

Transcript:

MIKE:  Hello and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. I've got a great panel with me here today. I've got Justin, Will, Dave, and Kyle. And we are going to talk about the future and let me start by going back to the past.

So, a hundred and fifty years ago-ish, we'll say, more or less [laughs], almost everybody had a farm. It's just what you did. Maybe I could go back a little further, maybe 200 years. But if you go back within a reasonable time period, there were people in cities who had businesses, industries. It's not that industry didn't exist, but it was very much on a...basically, a home scale, right? You couldn't really have factories because nothing had been automated to that point, and they didn't have machines to make factories work. So, you might have a lot of people working together in the same building, but that's about as far as you got in terms of a large-scale industry. And, again, almost everybody was in farms.

And if you had said to somebody in that time, “Well, you know, in the near future, nobody's going to be working on farms, and [chuckles] you're all going to be living in the city. And a lot of people are just going to have jobs like standing at a counter and feeding you food,” they would laugh in your face. It was just patently ridiculous on its face. There's absolutely no way that would happen. And then, a few changes to enable automation led to an industrial revolution and World Wars, [laughs] and to the point now where we’re somewhere between one and two percent of people actually work on farms. And the economy has just completely transformed.

To take that a little bit further into the future, when I graduated from high school and was looking forward to my future careers, you know, what career I might take, I never considered web development because that career didn't exist [laughter]. I couldn't have wanted to work that job because there was no web to work for, you know, there was no popular internet. I couldn't have even conceived of it because nobody had. Maybe a few innovators who thought, oh, maybe these early browsers will accomplish something. Yeah, it just plain didn't exist. And so, most [chuckles] of my career has been in a field that didn't exist when I graduated from high school, which brings us to today.

If you look at areas of exponential growth, things seemed normal for a while, and then, suddenly, they're different. And exponential growth is really hard for us to reason about because we tend to think linearly. You think linear growth, you know, it'll happen gradually over time, but that's not how exponential growth works. And technological change tends to be exponential when you have people in a large group together collaborating. You can go even further back, go to the printing press, which enabled the rise of culture and modern innovation and science because people could share information with each other. And the internet has provided a similar set of growth where we can get access to information so quickly. It enables exponential growth in technology.

And there are new technologies, like AI, that have promised to change software engineering. And the thing about this also is it's really hard to predict what it's going to look like in the future, you know, because you follow a trajectory unexpected things happen, right? You have tipping effects that change things.

I really wouldn't have thought in 2005, in a few years, we would have a phone that would make us all just retreat to ourselves, teenagers learned to stop writing [laughter], and a total change in society where people quit sharing the same reality because they watch different news sources. Like, this is going to happen from a phone in the next decade? What? And these things are really hard to predict, but they happen. So, we are going to go out on a limb today and start doing some speculation, and we're going to be wrong. But [laughs] –-

DAVE: We hope. We hope.

MIKE: [laughs] There are some things that we're probably going to see, and this is a chance to talk about what we see coming up in the future. And there's some value that comes from this because we can start planning some trajectories for what we might want to be doing in the next few years. Because there are some things I think we can see, and then there are some things we're not going to see, and we're going to go back, and we're going to laugh at ourselves [laughs]. But it's a chance to lay something down and give ourselves something to laugh at five years from now when we realize we were totally wrong, or maybe see that we saw some things. That's my lead-in.

So, I could ask a leading question. First of all, any immediate thoughts? But then following that up with a question, what do you think is going to be most impactful from these new technologies? I'm going to talk specifically about AI because that seems to be the big thing right now. Five years ago, everybody thought it was crypto. It's a thing, but it's not ‘the’ thing, right [laughs]? And --

DAVE: That's a separate discussion.

MIKE: That is a separate discussion. Probably not going to be relevant here because probably isn't relevant. That's not the [inaudible 05:30] thing, but this AI thing seems like it might be, even though that's been going on since, like, the ‘50s.

DAVE: I'll give the metaphor that I gave in the pre-show, which is that a year ago, people were asking me, well, programmers, “Is AI going to take over my job?” And I'm like, no there's no way AI can understand at the high orchestration level. You can refactor a function, but you can't hold up the whole app. And a year later, like, I slowly changed over a year. I'm like, ooh, the junior programmers are in trouble, but the seniors are okay. And then, I'm like, oh, the seniors might be struggling as well.

And the metaphor that I gave at the top of the call or in the pre-show was it's 1970, and we are welders watching the car industry wheel robots, welding robots off of the truck, and things are going to change. And one of the welders at the company is going to become the robot manager who is in charge of orchestrating everything and keeping everything, and everyone else is going to get jobs in construction. That's what my dad did when the mine shut down in the 1980s. He'd been doing deconstruction on rock underground, and he's like, well, I can go dig stuff up and tear it up above ground.

But he had to pivot because the uranium industry went away, and our industry isn't going away. Like, everybody still needs their web page. Everybody...well, okay, it's 2025. Everybody needs their podcast, and it's still happening, but I genuinely think we're going to see a lot of industry change.

WILL: Okay. So, I’ll say, all right, I'm hearing you. And what I have seen personally, like, I'm not a [inaudible 07:04] right? Like, I'm a technologist. I'm a future-looking dude. That's why I'm still in the business. And I've used ChatGPT...not ChatGPT, I've used Copilot. Copilot is the thing I've used.

I have used it to great success patching up things that I find tedious and unpleasant, and you know I'm talking about unit tests. And I love it. And I love it. I love it. I could say like, "Listen, I want a test that does this, and this, and this." And it will get my mocks right. And it'll get my imports right. It'll write out the thing. And it will give me what is roughly analogous to, like, if I had a very energetic remote developer on fiber that was putting something out, and they didn't particularly care about the quality. They just wanted their 5 bucks.

And I could take that, and I can give you something good because I know what I'm doing. And I could do this myself; I just don't want to. And I’ve found that to be very useful in a healthy 20%, if I'm being generous and optimistic, which is my nature, 20% productivity bump. I'm getting more done faster. I am not sweating. It seems like you've maybe started the year, started last year, let's say, right, 2024 that's where you were sitting. 2025, you're like, okay, I don't know, right, talk to me about what changed and what you're seeing, the specific terms.

DAVE: I resonate with what you just said about Copilot making things easy. There's an xkcd comic, number 1168, I just Googled it, which, basically, they're sitting over a bomb, and the screen says, "In order to disarm the bomb, enter a tar command with all the switches correct from memory on the first try." And he looks up and he says, "I'm so sorry." [laughter] And nobody knows which switch is the tar, right? I do. I do. I read that, and I'm like, ZXVF to unzip and...anyway, yeah, that's me.

WILL: Yeah, David. Yeah.

DAVE: That's me. Like, I got it. And what I use Copilot all the time for now is give me the Rails generate command to alter this table at a unique index, because you can do this on the command line, right? You can do Rails generate migration, and then you can say, add deleted at to customers or add extra columns to customers. And then, you can type space, and you can do delete it at colon timestamp colon index. If you're going to say customer is going to belong to widget, you can say widget ID colon references. And that literally tells the database to set up a foreign key, and it's going to build the model and build it up.

But I can't ever remember the CLI syntax. I just remember that it exists, and it's super convenient. And if you know the change that you want fits in that little DSL, it's so much faster to just write the command, hit enter, look at the file just real quick to double check it, and then run the migration, and you're done, especially when you're, like, greenfield development, right? It's real fast. And that is what I'm loving. I can sit it down, or I can say, “Give me the command to get the URL of the homepage of the project that I'm currently in the directory of in Git. I want to ask Git, Git remote show origin," that sort of thing.

And I love that Copilot comes back and tells me things because I tore Git all the way down to the bolts in 2010. And I wrote all these scripts to help me manage how Git works on Git version 0.7. There's a lot of development that's happened since then. And Git remote show origin Git was the only way to get the URL of the fetch URL push URL, and you could massage that and turn it into the homepage. And Git remote show will contact the origin and say, "Give me all your data or give me the information and give me the name of every branch because I'm going to have to build the remote's origin thing for this guy."

So, it's like a four-second thing on a big project, on a big server. And I was passing along something, and I asked the AI, "How do I do this da, da, da?" And it mentioned, “Oh yeah, by the way, you want to use Git remote," and it was like, "get URL," or something like that. And I'm like, that's a thing? It is a thing. It only gets the URL, and it's instant because it reads your git config. And I'm like, I needed this 10 years ago, but I solved it back then, and I never circled back because it worked. I love Copilot. It knows the switches, and it knows the latest thing.

WILL: Yeah, okay, sure.

MIKE: But what makes you change your opinion? What makes you change your opinion to think, wow, this is going to do more than giving me Git commands; this is going to be able to architect an app and build it out? What makes you think that we're going to go to that next level? Because there's a substantial jump between those two.

DAVE: Oh, absolutely.

WILL: I'm just going to say, give Mike's answer first, and that's fine. But then when we're done doing [inaudible 12:17] now we can do what separates the real pros and the seniors from the other stuff, which is not greenfield. This field is brown and dirty. There are barrels poking out of the ground. Nobody knows what's in them. There are rocks. There are stones. There's rebar poking up, like –-

DAVE: And it's not hygienic either. It’s got [inaudible 12:43]

MIKE: [laughs]

WILL: [inaudible 12:43] The children must play. And I'm in here to make this into a preschool [laughs].

JUSTIN: So, I got a couple of thoughts on that. One is the context window for all of these tools, you know, the LLM tools have expanded to the point where you can get an entire project into the context window. And that allows you to, you know, a number of things, one of which is, like, it can figure out where to add the changes that you want to change. Like, hey, I want to move this widget 30 pixels to the right on every page. Or you can craft your query like that, and it will give you what it thinks that you should do. So, that's getting better.

But having said that, I read a really cool article, probably two or three weeks ago, about how, to your point, Will, LLMs right now give you 70% of the way, and it's that last 30% that shows whether or not you are, like, a senior engineer or not. And it'll give you a project and everything but resolving all the little issues that pop up because of hallucinations that are hard to track down that’s still --

DAVE: Recognizing it's a hallucination at all, yeah.

JUSTIN: Yeah, and handling the edge cases. So, that, I think, is where we are currently, at least where I'm at currently with the tools. And getting there, you know, here we are 2 years into this revolution, and where are we going to be 5 years into it from now? 70% today and in 2 years, and then, in 5 years, are we leveled off. Or are we going to continue to see steady growth? I don't know. If we see any growth at all, that'll go from 70 to 80 to 85, and that last little bit I'm hesitant to say we'll ever get there. But the people who are doing the work are going to get more and more productive, hopefully.

WILL: One, are we seeing AI development accelerate so that we're seeing more progress faster, or are we seeing plateauing? Because I'll be honest with you, what I'm seeing personally is a plateauing and a refining of all the output.

DAVE: I have some data for you.

WILL: I'm not seeing exponentially...well, I mean, --

JUSTIN: We went from here to here really, really fast. But now it seems like it's just barely increasing.

DAVE: Yeah. So, it’s a lot --

WILL: It feels like it’s slowing down, but I could be wrong. And the second question, and this is just really specific, is like, how many thousand lines of code of context can I get into an LLM? Because, I mean, really? My understanding of the LLM is that it's an exponential growth in complexity as your context size increases because there isn't a semantic understanding of what all this stuff means per se, per se right? It doesn't translate into the symbols like we do. And so, you know, just, like, 2000 lines, 3000 lines, 10, 000 lines that's not, like, a linear model increase. Does anybody what that context is?

MIKE: Yeah. So, Transformers, which is the tech that's been used for all of the LLMs, it was a technology that came out, was it 10 years ago or something? And the way that those work mathematically, that is true. It grows exponentially by the context length. However, however, there are workarounds. Those are moving forward quickly because you can take groups of things and compress them and put them together. And that's really how our brains work anyway, right? We think about things hierarchically. And by approaching it hierarchically, you can actually address larger and larger windows. And now they're into millions of tokens. So, yeah, much longer than just a few thousand.

DAVE: Yeah, ChatGPT I just asked it, “What's your context?” And it's like, “4K,” And I'm like, really? I was playing with Mistral last night, and it's, like, 130k. What are you doing? But that was in the last 12 months. When I was first going out to Hugging Face last February, and it's like 2024, so a year ago, 8K was big. And the LLaMA-30b had just dropped, and everyone was freaking out about it. And now they’re like, 405b, I mean, they're just getting bigger and bigger.

So, I said logarithmic, but it's the same thing as the exponential. We're just on the bad side of it, which is that if you want to double the things you can think about, or if you want AI to go up, like, 1%, you got a double its capacity. And if you want to go up another 1%, you've got to double that capacity. So, we're on the bad side of that. But what we're seeing is, like what Mike said, specific breakthroughs. Like, somebody will attack a specific problem. In the last 12 months, they've solved...in image generation, they've solved the hand problem much better. It still happens a little bit, you know, seven fingers or, three thumbs, or the fingers are on backwards, too many arms, that sort of thing, good time. But I'm not seeing that anymore.

I went out and built a character for a comic. There's an AI that will let you generate, like, graphic novel-type stuff, and you can develop characters. And it had trouble generating you the same face two images in a row. You could do four in a row, but they would all look similar in different poses, but your next slot, completely different all re-randomized. They've solved that. You can literally just say, “I like this one. This is the face. Please make everything look like that.” So, I genuinely think this is coming.

The general trend line is, we just keep adding more and more and more and more. And I think, eventually, we're going to get to a point where the width is big enough and the speed at which it tries and tries again to basically cheat...you can have twice as much AI if you just run it twice, right? And the same idea. Cross-intentional stuff. This blew my mind. I watched this a couple of months ago. The way they made it so that you could say, make me an image of, you know, a whiskey bottle made out of yarn, right, is while they were doing all the Stable Diffusion, like the pixels blurred, now resharpen, now fix it, they were doing that.

At the same time, a completely different text-based AI was being told this image has a girl in a red sweater. This image has an angry woman in it. This image has a dog that is happy. And the genius move is somebody said, let's tie these two together. And so Stable Diffusion when it was blurring and sharpening was also being fed, this is what the English...and, like you said, AIs don't think the way they are. It was not, this tells you what the image looks like. No, it's like, this is the statistical probability of the words that you might hear in sequence correlate to pixels in sequence over here.

But the end result, once you've thrown 24 trillion tokens at it is that you can say, "Show me a whiskey bottle that’s, you know..." And you can tell it crazy, crazy things. And if it knows... “Show me a whiskey bottle with irritable bowel syndrome.” You're going to get an image if you do that.

MIKE: [laughs]

DAVE: I haven't tried it. I think it'll be funny if I do, right? That kind of thing. And it'll do it. And so, it's a trend line. I genuinely think...I guess what I'm saying is I am startled by the amount of things that are now trivially easy to do today that I was certain a year ago we wouldn't be able to do. Are we going to be out of a job? No, absolutely not. There's going to be people managing the AI. Until the robot platforms come out, humanity has a chance, but we'll be the operators running stuff. But yeah, we're going to lose stuff.

Real quick tangent, my father-in-law was a Renaissance man, machine operator, machinist. He built engines in his garage as a hobby, like, literally gasoline engines that were...they would hold pressure, spark, flame. I mean, he would machine things to extremely tight tolerances. He made his own bolts on one project, literally turning them on the lathe.

And he had the entire skill set from woodworking to metalwork to engineering to metallurgy. Like, he knew how to temper steel to different properties, like, he knew the full stack. And when I need something built, I go to Amazon, and I order something from China. And the Renaissance people are going away, people who worked in their garage building engines as a hobby don't exist anymore. And they were shade tree...you could find one on every street 30 years ago, and now they're dying out. And I think that's what we're going to see.

The people like us that are crusty, old farts that we actually know what transistor-transistor logic is, we're going to be surplus to requirements. The next generation of programmers are not going to be able to...they are going to detect hallucinations like we are, but they're going to do it on a completely different basis. They're not going to come to it from knowing everything that the software is built on. They're going to know it by recognizing patterns. And like, oh, yeah, last time I told you to do this, it didn't work. Or, oh, you told me this, but I need to test it, and it'll just be a habit, and it will go along. And then, somebody will write an agent to do it, too.

JUSTIN: Go ahead, Kyle.

KYLE: I was just going to say, even without AI, though, we're hitting that right now. Juniors that I've met they don't necessarily know how a computer works. I mean, we're always having to adapt and expand. And that's where I'm listening to some of this, and is it something that will take our jobs and we'll be replaced? And that's where I have a hard time thinking. Well, no, not unless you let it.

And what I mean by that is we're all going to adapt. We're all going to be doing it differently. Programmers, you know, they were in C, or C++, or any of those languages that were not memory safe. Those are not popular languages now. We have languages that handle that...who a few years ago would have used Stack Overflow. We have Stack Overflow. That has sped things up. I think things are just going --

JUSTIN: Actually, Stack Overflow is going away, but anyway.

KYLE: Well, I'm saying, the progression, right? But I'm just saying, if you adapt to the changes, the world does move faster, but the demand really has not decreased. It's just increased. Like, do things faster, do more. And so, I don't know that the jobs that we have today will exist, but they'll be different and plentiful.

DAVE: That's the way, yes, --

WILL: Well, and --

DAVE: The job you have right now is going to go away no matter what. You're not working on the same stuff today that you were two years ago. We're not going to lose our jobs and starve and die because of AI, just things are changing, right? I don't build WordPress blogs in PHP anymore. I don't solder anymore.

KYLE: Thank God.

DAVE: My job has moved on. Those jobs have gone away, but I am still employed and employable, and that will never change until I die, hopefully.

WILL: Well, I mean, until you fall further and further behind the [inaudible 23:58]. I mean, it is a situation where if we all get 20% more efficient, then some of us are going to, I mean, the weak gazelles in the herd are going to get called out, and we're in the middle of that.

JUSTIN: I just have a question for the team. I mean, we're here in a podcast, a development podcast. What can developers do to not be forced out with the next wave of things? Ideally, we all want jobs, and this wave of more efficiency means that maybe there'll be less jobs. But, ideally, we want to know how to use those tools and be one of the people that is advancing and using it rather than being run over by the car.

MIKE: So, something that was said a minute ago, most people aren't writing in C, I think it was Kyle, aren't writing in C anymore. But it doesn't mean people quit coding. Most people aren't writing in Assembly languages anymore either, which is what...people certainly aren't using punch cards [chuckles] very much to program a computer. There might be somebody out there that has to run a legacy machine. It could exist [chuckles], but it's very limited. But we're all still coding, but the nature of coding may, is, may change.

A couple of years ago, I read an ad...there was an ad for Anthropic, a company that's gotten really prominent in the last couple of years, where they were asking for a prompt engineer. And I think they were offering a 300,000 a year salary, and they couldn't find anybody.

DAVE: Oh wow.

MIKE: Because it didn't exist. They couldn't find anybody who had experience doing that because nobody had done that yet. But, a year from now, two years from now? So, here's a prediction that some people can hold me to and see if I'm totally foolish. Five years from now, if you can't be good at prompting an LLM, I think you're out of a job because I think that becomes the new language.

DAVE: In this industry, yeah.

MIKE: In this industry. I think we may even have new programming languages that arise that are designed to be friendly to AI rather than friendly to whatever the language constraints are right now, and they'll compile down to whatever. That will fundamentally change what we're doing. And it may look a lot more like product management, where we're gathering requirements and expressing them really, really well for a good product manager. It may look a lot more like product management than loops, right?

DAVE: I have a corollary prediction which is going to be really, really interesting. Source code is a thing that you run through a compiler to talk to the computer. But it's written to be read and understood by a human which means good programmers write source code that other humans can read and can interact with.

I would not be surprised in the next five years if the prompt becomes a version of source code, and if we actually start seeing people who start to say, “This prompt and that prompt, this one is more efficient, but this one is more robust. And oh, and by the way, this one, a human can't understand it. They need an agent to do it. So, it's actually bad. Let's use the other one.” You're going to see people...you need to write a prompt so that a human can understand it. The AI will turn it into transforms.

WILL: I'm a hard, hard, hard no on that. I think natural human language, English, spoken English, human language is absolutely terrible, like, atrociously bad for the kind of precise interoperability, for the type of precise design requirements. It is atrocious for that. And we have tried to develop sort of specific language around, like, legal contracts, and it's wretched. It's like we have...we spend billions every year arguing about the specifics of legal contracts, which are not, I mean, they're not on the same level of sort of specificity and complication sort of like source code is. We couldn't do it.

So, I think, I actually think source code for your modern high-level languages like Ruby it's actually an excellent example of something that is source code but is very readable on a human level. I think we'll see something a lot like that.

DAVE: I agree. I think a prompt language might be coming, yeah.

MIKE: Yeah, now, that's exactly what I was suggesting. I think we're going to have a language that's up at the next level. We don't write machine...we don't write Assembly code. Most of us don't write compiled, a lot of us don't anyway, compiled languages anymore; we do interpreted languages. Put another layer on top of that.

Now we've got the prompt language, which is just...will have to be exactly as precise as what the lower-level languages were. But it's a specification of what you want from that project. And the people who are very detail-oriented, just like today, the people who are very detail-oriented, and are able to communicate with precision what exactly they want the system to do will be successful. Because, exactly like you said, Will, if it's not precise, it's going to be wrong. And the need for precision is not going to go away, I don't think.

JUSTIN: Okay, so I want to walk through, like, a typical workflow five years from now. So --

DAVE: Before we jump into that, can I --

JUSTIN: Let's put our imagination caps on and suppose we're working for a web retail company, who could it be, who is selling...they're creating a new form to gather data, or something like that. I don't know. They're doing a know-your-customer data entry, okay? So, you have the product manager, and then you have the engineer. And they're sitting down together, and they both have their laptops in front of them, or maybe their headsets.

And the product manager says, I want to collect, you know, first name, last name, date of birth, driver's license number, SSN. And the other hidden person in the room is the AI, the Figma AI, so suppose Figma is listening, and Figma is listening to that. And what happens next? I imagine that Figma will be listening to this and real time Figma has, up in front of it, the web page with their company's UI.

WILL: Design template, right? And one of you got to style that, right?

JUSTIN: And these things appear on the screen. [crosstalk 30:47] And these fields appear on the screen as you go along. And the engineer is not doing anything at this point.

DAVE: He's steering the UI, though, in real-time, right? Like, make the social security bigger, make this field required.

JUSTIN: Yeah, exactly. And so, it's like, hey, make this field required, a little star shows up next to it. So, in real-time, they're having this conversation, and these things show up on the Figma UI with the company's contextual UI as all part of it. So, they go back and forth with the engineer and the product designer or the product owner. They go back and forth until they reach something that looks good. What happens next?

WILL: Then we're talking about it. It's like, okay, this is what I need. Then you've got the senior engineer, right? And the senior engineer has a good understanding of the topology, right? The various microservices that need to get looped in the sort of ops resources that need to be allocated. And so, he is going to talk through the architecture, right? He's going to talk through the architecture and be like, okay, I need to go, you know, the last [inaudible 31:52]

JUSTIN: So, is copilot listening?

WILL: Yeah, yeah. Let me finish [laughs]. I need to go out to Galactus and get date of birth so we can send everybody a birthday card. And then, I need to go to thus and such service that says, okay, this is the thing. And all the while, boop boop, boop, I need to go here and get this. I need to go here and get this. I need to go here and get this. And the AI is saying, “Okay, you need this thing.” And then, I'm going to create an epic for this sub-team.

It's like, okay, boom, I need this from this. It's going to take the meeting transcript and an understanding of the thing. It goes to the sub-team for that engineer. And then, you've got a, I would say, probably an asynchronous kind of a thing where it's like, okay, we need this. It looks like this. This is how this thing goes for you. There's still going to be people involved in the steps. But all of the incessant meetings and sync-ups and stuff like that it knows what the org chart is. It knows what the subsystems are. It knows who the person in charge of those things is.

And then, you have the follow-up meeting where, like, okay, we take it I'm Galactus, right? And I need the customer information, and then it describes that. And I say, like, okay, this is it. This is the...I think a lot of it is going to be just people are going to look a lot more like...they're going to look a lot more like technical program and product managers where you're the engineering manager for the AI. But you're going to code, and you're going to need to code because there's still going to need to be a body who can say, "This is good, bad or indifferent," you know what I mean?

JUSTIN: I mean, you've got to have somebody who knows it. But when you're at the start of the project, you may not need somebody who is actually coding because you can go in. You have the OpenAI, sorry, OpenAPI specs for all the services you currently have, and they could be represented graphically somehow. And you could, as you're chatting, Copilot is going in and giving you the list of those things. And you say, "Hey, I want that one," and it gets dragged over. And you say, "Hey, I want that service," and it gets dragged over.

And so, these connections that happen, I mean, theoretically, the code between the front end and the back end and then the back end into another back end all of that could be done automatically. You do have to have the engineer to make sure that it’s making sense. But it'll go in. It'll check the parameters that exist already. And then, you'll decide, oh, we don't have a data store for this yet. Go make, based on my GitHub template, go make a new project that has a service that stores these in our standard data store and that will follow all our standards and everything else. This is sounding very hypothetical, right? It's crazy. Maybe not 5 years, but maybe 10.

DAVE: You haven't said anything that I don't think we could build right now with a tiger team, legitimately.

JUSTIN: Great.
DAVE: I genuinely don't feel...To Will's point a minute ago, you were saying because it's fuzzy and we need precision, it's never going to happen. And I see the same problem you do, but I arrive at the opposite conclusion, which is that AI's superpower is that you can give it a typo. You can be imprecise. You can be sloppy. And it's actually smart enough to not just untangle what you said but infer things three leaps away from it, and I think that's going to be a superpower.

I've been playing with storytelling LLMs, and they really, like, Mistral is not going to generate JSON. It can't do it. It can't even keep quotes lined up in English text, but it can tell a compelling emotional story, that kind of thing. And it's really, really good at making inferences. This is the last thing I'll say about [inaudible 36:06], and I’ll let you guys take the floor. But the prompt engineer...the stuff that I'm doing right now is I will write a story where husband and wife are arguing, and he gets mad, and he wants to walk away. And she's like, “You can't walk away from me.” She grabs him by the arm and pulls him back. And I'm like, no, she would not do that. That's wrong to the character.

So, I have to tell the AI, “She's not going to grab him,” and then regenerate. “You can't walk away, grabs him by the arm.” I'm like, why did you ignore that? “Well, the narrative tension needed to escalate.” I'm like, oh, okay. And you end up going to the wife and then changing. Oh, I see she's got vindictive as the first trait on her personality. I'm going to move that down three and move supportive up to the top. Regenerate it. And all of a sudden, it's a completely different story. She's like, "Don't walk away from me. I need you to understand me". And it's a completely different tone to it.

You can give the AI 7500 tokens of this is a young boy, and he's got some interesting exotic powers, and he's gone to a school to learn. But you can take a very small AI with very low context, and you can say, "This guy is Harry Potter," and it just got 10, 000 tokens worth of inferences out of that. That's what prompt engineering I think...it's not headed there. It's there right now, at least in the fiction category. And I think when you marry that to Copilot, it's going to be terrifying because you'll have sloppy. You'll accept slop but emit precise. That's going to be a good day.

WILL: I feel like one thing that I think will really...I think a quantum leap in terms of AI's ability to generate software is the ability to sort of...so, you have a self-training model. I build these things, right? And it's like, well, does it render the webpage, or does it not? So, one of my superpowers...this is something that I do all the time because, sadly, I have progressed at a point in my career where people give me just the worst, just the worst jobs. I get nothing but the absolute worst jobs, worst jobs [laughter].

And it's like one of my models, right? This is a simple model, and I got it when I was developing really wretched stuff, embedded system device drivers, which are hell on earth to debug. But this is just a simple intellectual way to make nearly anything work, and you just debug from the known to the unknown. And what I'll do is I'll have some component. I don't even want to tell you what I had to work on today. It would curl your hair. It would be terrible.

But I had this front-end monstrosity. It wasn't working. And I'm like, okay, listen, we're just going to gut it. We're going to gut it down. I'm going to tear this thing down to the studs, and it's going to print, “Hello, World.” That's literally all it did is, like, print “Hello World,” and I got that working. And then, I started adding things in brick by brick by brick.

DAVE: You got a working piece.

WILL: Until I found the thing that was broken, right? An LLM could do that, an LLM could absolutely do that. And then, you have the model, which is training itself. I'm training myself. It's like, okay, I'll try this. I'll try this. I'll try this. I'll try this. And we could find ourselves in a situation where a lot of my job is just wretched, ditch-digging work where I'll get up into the muck and just stay here until it goes.

Well, an AI could do that job, and I could just be like, this thing doesn't compile. Why doesn't it compile? Make it compile. And the AI could be like, okay, I'll just tear it down to the studs, and I'll add things back in line by line until it works. And that's a slow process, but you could just be like, all right, I'm going to bed. Call me back in the morning. Let me see what you found. And I'll say another thing. You would need this thing. But one of the things I'm grappling with right now at my large, big box retailer, which is not Rent-A-Center. It's not Rent-A-Center. I bet it's better than this [inaudible 40:30]

DAVE: So, you're working for a competitor? How dare you?

WILL: Yeah, anyway. So, I'm [inaudible 40:34] that a big box retailer with a lot of legacy problems. And one of the things that I'm grappling with right now is migrations, where you're migrating one bad service to another thing, right? And so, you migrate it over. You run into real problems, real serious issues when you don't have the operational discipline to finish one migration before starting the next migration, then your problems compound in a truly brutal way. Well, those migrations are extraordinarily painful and the source of a lot of brownfield misery.

But if you had an AI that could generate products and iterate itself and say, “Okay, here's the old one, and here's the new one. Are they the same? No? Fix it.” And that's sort of when you have the template, what you could do is you could execute these massive migrations automagically, close to automagically. Then you could really start to...and it is a dumb ditch-digging miserable process, but that's the kind of thing where you could accelerate development tremendously.

MIKE: So, you're talking about what sounds like reinforcement learning from human feedback.

DAVE: There's an AI listening to this podcast a year from now, as it's ingestion, and it's going, I wonder if I need to be thinking about how the whole system hangs together [laughter].

MIKE: Reinforcement learning from human feedback is what allows ChatGPT to work. They train it, and it's off based on terabytes of data, right? And then, they show it to a human. Here's what I think you meant. And they say, “No, you can go this way,” and it learns from that. And they do that through iterations. So, practices generating the text and then gets human feedback and based on that, over a number of iterations, improves to better match what humans want.

You're talking about taking one system matching it to another one, and it gets it wrong. Get some feedback on that and then keep iterating to get it right. There exist algorithms today to help with that. Also, some of the newer algorithms that exist are no longer bound by a fixed amount of calculation. That is, it comes up with a chain of reasoning where it comes up with the next step, and then the next step, and the next step, which has an indefinite number of steps, right?

DAVE: Oh wow.

MIKE: And it will generate based on that approach. The latest OpenAI models, for example, are built that way. And that changes the way that these problems get approached to be closer to the kind of human-driven algorithm that you're describing. So, I think that we, again, on this one, I'm not making a prediction on because I think it's really hard to say whether that really will be the key. But there is a chance, because there are hundreds of billions of dollars being poured into this, there is a chance with all of that mental energy, all of that compute going into it, that somebody is going to find the algorithm that will continue because they're continuing right now to find new algorithms that push it forward.

JUSTIN: Yeah, we're so early in this.

DAVE: And where we focus our attention it goes fast. Like, the grass is greener where you water it, and that's really, really true right now in the AI side of things. They're taking programmers that are complaining about their jobs and saying, “What are they complaining about? How can we help with this?” I love TDD because it's a completely different way of testing. It needs a different name because people think, oh, I'm just going to write the tests that are at the end I'm just going to write them at the beginning. It's not like that. It's a completely different discipline entirely. It's a completely different experience.

But most programmers don't do that, they prefer to test after, and there's all kinds of anti-patterns that start to happen as a result of this. And the Copilot stuff, and Gemini, and stuff that's out there is optimized to solve that pain. It will write your crappy after-tests for you, but nobody's working on the TDD thing. So, hang on, I just had a business idea. I got to write something down [laughter].

MIKE: So, we've talked a lot about what we see things looking like in five years, higher level languages relying on the AI to do a lot of the grunt work. Potentially, developing some of the processes we use to deal with ugly code, with code that is in bad shape, in a bad environment, bad all-around situation, industry, you name it, it'll help you with those things. And if we do, are jobs going away? Yes? No? It seems like, generally, the answer is no. We still need people building stuff. But our expertise will have to change.

DAVE: Where do we need people right now? That's going to shift. We're just going to be sidling side to side to side to side. And a hundred years from now, only 1% of us will be farmers. And they'll be producing the food for the world but in AI, yeah.

WILL: Yeah, I don't worry about myself so much just because I've been doing this for a long time, and I know a lot of things that I don't even know how I know them. I just know them. I just know. And I don't need to know a language. You could drop me in a Go project tomorrow, and if I had Copilot or something like that, I'd be fine. I'd be fine. It's like, how do I set it? Oh, that's the core routine? Okay, that's fine. How do I do a loop?- How do I do a thread? Okay. How do I make my API calls? Okay. I'm done. Moving, moving. Chop, chop chop. No problem.

I worry more about the junior developers that are coming online right now because I know how I learned it and that sort of, like, old-growth, forest free reign kind of a thing that's not going to be tolerated. I feel like we're eating our seed corn on a large degree in this industry right now. And yeah, Justin, go ahead.

JUSTIN: Yeah, so this hits close to home because my oldest child he is graduating from university in a year from now with a computer science degree. And chatting with him, the degree that he got from Utah State University, which is the northern end of Utah, it was very much a classic computer science degree. It had more to do with my computer science degree than with perhaps the prompt engineering that you see.

So, he's deep in algorithms. He's deep in the classics. And he used C++ of all things, memory allocation, all that, you know, close to the metals type of stuff, not terribly close to the metal. He's not writing in Fortran or things like that. But at the same time, I'm chatting with him and trying to figure out, okay, what are you going to come out and be able to do?

And he's having trouble right now getting an internship, whereas a couple of years ago, he'd be snapped up right away. And not only that, he's, like, at the top of his class. He's keeping his scholarship, so he's got, like, a 3.85 GPA, which is hard to do as a computer science major. But yeah, we're trying to get him an internship, and I'm worried about him getting his first job, so...

MIKE: I have a nephew in the exact same situation. He graduated six, eight months ago, still doesn't have a job.

DAVE: I think we have another podcast topic there because nobody wants to hire the core thing. They're looking for people that are amazing on their side hustles. Side hustle is the wrong word, the other things, right? You don't hire somebody who meets the job requirement and says, "I guess I can do the thing you want to do." You hire the person who comes in and says, "Yes, I can do that. Also, I care about this, and I have a degree in, you know, I have a Juris Doctor because I went to law school, or I have this other thing, or I've worked in finance." That's what you hire for.

And when you're very, very young, the side hustle is completely different. You want to hire the kid who stays late without being asked or cleans up his desk or her desk without being asked to. Or one of the best personality tests I've ever seen...I was in church and one of the senior church officials was standing there, and there was a gum wrapper on the floor of the chapel.

And the senior church rep went over and bent down. And I'm in a church where hierarchy is kind of important. Somebody that high, you're not supposed to make that person pick up litter. And he did. And everyone in the room went, “Oh wow.” And that's not in the job rec for the thing he was doing, right? But that's the difference between getting this job and having a career that will go.

KYLE: There was something else that Justin brought up, too, and that was the C++. Now, I went as a computer engineer, so I did C++ as well, but I'm aware that other universities do C++, but others will do C# Java, which is one step higher, I guess.

JUSTIN: That was in there, too, but yeah.

KYLE: But, yeah, no the point that I'm trying to make here is that we might need to, I mean, this might be an institutional problem, too, because a lot of this is theory. They're teaching theory-based learning and it's all algorithms and stuff like that. We're not actually teaching engineers coding. We're not teaching them prompting and stuff like that. So, it might be a problem at the fundamental, like, schooling level before they can get the jobs.

MIKE: This is not a new concern. For years, people said, “Well, we need to give people more practical, hands-on stuff, trade school stuff for engineers rather than the theory.” And I might push back on that. I think it was Will who said we're eating our own seed corn. I think that's true. And I think that's closer to the problem because, honestly, people who understand what's going on and can understand what happens when the prompt doesn't give you your answer are going to be incredibly valuable, I think, in the coming years.

I think that we need to have that pipeline of people who actually get the things that are happening at a lower level. And they're not going to be that useful right up front. They're going to have to take some time to learn it. They're going to have to learn those skills, but you need both if you want to be really successful. I do think that there is a shortage right now in the industry of companies that are willing to invest in their future. There's some uncertainty.

I think it also has to do with the change in inflation. When you get different interest rates, it changes where the companies want to invest. We had really low interest rates for a long time, so companies would invest in their people. Interest rates go up they say, “Oh, we can make money other ways, so they drop all the people.”

DAVE: [inaudible 51:43] operational, yeah.

MIKE: I think that's a lot of the problem in the computer science industry right now. It's just a lot of big tech is firing because they can make money through investment. However,...go ahead.

WILL: I mean, I'd say having gone pretty far past a regular old bachelor's degree, there are a lot of professional fields where you're not working with a bachelor's degree. You don't work as a doctor. You don't work as a psychologist. You don't work all kinds of things without post-bachelor's education.

KYLE: Post engineering.

WILL: [inaudible 52:21] is a master's degree minimum field to be in. You don't know shit with a bachelor's degree. I'm sorry, you just don't. It's just too much. You can be smart, and you can be trained. And most people with a bachelor's degree and a good GPA are smart, and hardworking, and motivated, and absolutely can be trained. And after they get that sort of finishing school, which we've been dumping off on industry, right? Historically, like, the first...it's just no.

The first year out of bachelor's you kind of suck, and you do what you can, and we give you jobs that you can take on, but we're investing in you with the expectation that you will pay us back later. A lot of managers have exploited that thing, and they underpay their people when they start producing, and then that's kind of a self-defeating cycle. But that was how it was always supposed to...how it was always intended to work. And I think that's just how it's going to go.

But, I mean, honestly, we've already seen them come for the boot campers. First, they came for the boot campers, and I said nothing, for I was not a boot camper. Then they came for the undergraduate interns, and I said nothing. That was not [laughs], you know. And then, they came for the senior engineers, and there was no one left [laughs].

MIKE: No one left because the AIs are already doing everything else [chuckles].

WILL: Listen, you know, if I'm that guy...I got to make it...well, no, that's not true. I got to make it in another 20 years, man [laughs]. I don't know, maybe I'll be the guy shutting the lights off, you know, walking out the door. But I'm consoled by the fact that having seen the incredible leaps and bounds the industry has taken over my career, they just want more. The more we give them, the more they want, the more we give them, the hungrier they get. [inaudible 54:30]

DAVE: This stuff is cool.

WILL: Yeah, this stuff is cool. And it's an arms race. It's an arms race, right? Oh, I have a hundred engineers. I double their productivity, right? Now I'm taking ground on my compiler. So, I've got features they don't have. I'm in markets that they're not in. I'm doing better things, right? And then, your competitor has to hire another hundred engineers to keep up because they just want more and more and more and more and more. I think that if we can get really, really negative and pessimistic, that is something to keep in mind. You know, Rent-A-Center, I don't know who you guys...who's you guys’ number one competitor? You guys got a competitor who you’re worried about.

MIKE: There's a couple of them.

WILL: What's that?

MIKE: Oh, there's a couple of them. I guess we could name them [laughs]. There's –

WILL: Well, no, whoever it is. If you guys do something better, well, then they got to do it, too. And they do something better than you, you got to get...and so, it's one of those things where engineering departments they're pretty durable. Head count is pretty durable.

DAVE: Is it though? Because the AI is taking over call centers. We're seeing call centers get gutted by, like, 90%. And that's right below the interns, and then they're going to come for the interns. The key thing, though, is that if AIs take over the interns, we're no longer taking interns and producing junior programmers and that pipeline of improving people. And that goes back to...a lot of the Renaissance people, I think, are going to go away, but also, we're going to get shot in the foot. We're going to have to find a way to backfill those basic skills or to obviate them so that you don't need them.

KYLE: And does that kind of go back to what Will was pointing out? Does that then mean that future engineers have a master's or a doctorate degree in order to get into the field?

DAVE: I don't know.

MIKE: There may be a cycle. There may be a time when we get greater credentials for a while in this kind of arms race. [inaudible 56:36] I'm good. Eventually, you're going to run out of people. They're going to run out of the junior engineers. And I think it's possible that those requirements come back down, and you get closer to the boot camps where, wow, we need to actually backfill these people. We need people who know what they're doing. These tools work, but we need some people to run them. The pendulum could swing the other way.

DAVE: And all the people that are not going to be working in programming in 15 years are going to be working at something else because their programming needs are going to be solved. So, they're going to have an opportunity to do other things. We couldn't have been programmers 100 years ago, even if we had the computers, because we were too busy farming because we were starving. When you say they're coming for my job, that is the same sentence as they freed me up to do some really cool stuff later. But if you don't know what that cool stuff is, it's a negative statement. Like, they came for my job.

KYLE: I mean, the automated checkout stands were going to take over, right?

DAVE: Yeah.
KYLE: And they haven't fully taken over. People are pushing back on those even, so we don't know where they're going to go. The other thing that we haven't really pointed out, too, is, I don't think we're seeing much of it yet, but policies and regulations. Are those coming, or are we just going to let this go, you know, unyielded?

MIKE: It depends on the political --

WILL: All I could say is, to my mind, it is absolutely stunning. So, my wife works in healthcare, right? She works in healthcare. She's a therapist. And it is stunning the lack of regulation and the power that just a regular, old software developer, high school diploma, maybe [chuckles], is just rocking in out the street. I'm not trying to say, like, you have to have this, and this, and this to be good at your job, or anything of the sort. But there is nothing, nothing, nothing whatsoever [inaudible 58:44] you know. People are doing this stuff in India, where it's just like, is this guy even accountable to United States' criminal codes?

MIKE: Right. [laughs]

DAVE: Nope. We know the answer to that one. Yeah, that's actually a good point.

WILL: Maybe, maybe not, you know.

DAVE: It's something I was going to say at the top of the call, but it's relevant. The problem that I think AI doesn't have, and is a long way away from, is being trustworthy for a mission-critical, low-resolution task. I do not want an AI in my pacemaker trying to decide or hallucinate when it should send that pulse. I don't want an AI in my CPAP. There is an AI in my CPAP, but it's not deciding valvular. It's looking at the entire night going, we're coming up on the time when he breathes easier. Let's take the pressure down so that we're not doing it for him. It is in there.

But the mission-critical part of this, right, where you've got machines that will dispense insulin and if it's the wrong dose, you die. That kind of mission critical stuff I would not toss that to a sketchy LLM without a whole bunch of agent LLMs going, is this reasonable? And even then, I'd still want somebody doing post-mortems on the failures.

WILL: But even in that point…so, I've worked tangentially in certified- software doing static analysis tools for making sure that your avionics controllers and stuff like that are going to do the job that they need to do. And I think AI can contribute tremendously to the effectiveness of that stuff.

DAVE: Oh, absolutely.

WILL: Like, through better static analysis tools, through more robust and durable test suites, just, like, really, really , really, really testing the ever-living snot out of all of this stuff and letting people do better work. And yeah, okay, they could generate some code, but we have run it through, and we have to test it. And you're having smart people be more effective, be more efficient. And I think that's fine.

But, man, I don't know. I think, David, I think you might have sold me, I don't know. Because as I'm walking through the problems, and I'm saying like, okay, this is what I'm grappling with, right? And I didn't spend a lot of time thinking about, and here's how I would build a tool, or a Transformer, like, tooling around shaping this problem into something that the AI could mill down.

MIKE: Well, that's actually probably a good place to end this, talking about avionics. Let's bring this in for a landing.

DAVE: I like it.

MIKE: We asked some questions, where are things going to go? I think we kind of all have a lot of agreement that they're going to change a lot. We don't know exactly what they're going to change like, but there's some suggestions as to the direction things are going, more effective tools, allowing us to do some things more effectively, and maybe even fundamentally transforming our industry.

And there's absolutely a need to stay on top of it. That's never changed. It's been the case all along in this industry, but perhaps even more so now. And it may go in crazy ways we haven't thought of, you know, it's not going to be subtle changes. It may be pretty dramatic ones. That's not necessarily something to be scared of. It's potentially something to be excited about, but we need to be prepared and watching. Keep your finger in the wind, which way is this going? Or else you might end up in trouble.

Thank you. Hopefully, listeners, you've gotten something from this and have some ideas to chew on as to where to take your career. And until next time on the Acima Development Podcast.

This episode of the Acima Development Podcast delves into the challenges and strategies of working with legacy systems in software development. Mike begins by drawing a parallel between his past experience in construction and software engineering, emphasizing how businesses often opt to demolish and rebuild rather than work with existing structures. However, unlike construction, software developers frequently lack the luxury of starting fresh, as applications must remain operational while being updated. This leads to the episode’s central theme: how to effectively manage and modernize legacy systems without causing disruptions. The discussion also highlights how software often evolves in ways that diverge from its original vision, making maintenance and scalability a significant challenge.

The conversation explores various approaches to dealing with legacy systems, from full-scale rewrites to incremental updates. Matt likens outdated software to old pants—once a great fit but no longer suitable for current needs. The panel discusses past examples, such as the rise and fall of ColdFusion, and the complexities of working with outdated languages like COBOL and Fortran. They emphasize that while complete system rewrites are sometimes necessary, companies often benefit from gradually modernizing through a “strangler fig” approach—wrapping legacy components with new systems until they can be fully replaced. They also stress the importance of considering long-term viability when choosing programming languages and architectures, avoiding trends that might fade over time.

The episode concludes with insights into best practices for managing legacy codebases. The panel discusses strategies such as isolating components, minimizing dependencies, and maintaining clear APIs to improve maintainability. They also highlight the importance of version control, automated testing, and structured modernization plans to avoid the pitfalls of outdated software practices. The conversation touches on AI’s potential role in modernizing code, though they agree that current AI tools aren’t ready for fully automating legacy system overhauls. Ultimately, the team underscores the value of continuous improvement—whether through refactoring, gradual replacements, or thoughtful decision-making—to ensure software remains functional and adaptable over time.

Transcript:

MIKE:  Hello, and welcome to another episode of the Acima Development Podcast. I'm Mike. I'm hosting again today. With me, I have Matt, Kyle, and Eddy. We may actually have some other people showing up later. We'll see [laughs]. We've got some delays out there, but, for now, we've got Matt, Kyle, and Eddy. And well, I'm not going to introduce the topic quite yet. Let me lead with some storytelling here, and I think we'll have some good storytelling in today's session.

Years ago, I did construction work, family business. My dad and his brothers they did building, so I did that some growing up, did it in my 20s, did a fair bit of construction work, among other things [laughs], but I did that. One place I worked was an indoor commercial construction, that is, we'd go into office buildings, sometimes high rises, sometimes smaller, you know, I've worked in banks, high rises, and in between [chuckles].

And we would go into spaces that were being leased by a new tenant and build it according to the new client’s specifications. The interesting thing about that is you have these...particularly, like, you have high-rise buildings. It's basically just concrete walls, right, and [chuckles] flat in between other than support beams to hold the whole thing up, and everything else was kind of open to the discretion of the client. And they did things in a lot of different ways.

And what usually happened, and this is interesting, is that it was cheaper for them to just tear the whole thing out rather than work with what was already there. You think, well, maybe, you know, you can keep some of the stuff. That's not what generally happened. The way that, you know, spent a lot of time just completely demolishing [chuckles] stuff that was already there. Actually, the demolishing didn't take that long, but, you know, a lot of time demolishing and then rebuilding from scratch, you know, dropping it down to the concrete.

And that's really interesting because you think it might save you to use the existing building, but that's not what we did. I can't think of a case where we, I mean, that's not entirely true. We did do...some cases we'd leave some stuff, but, you know, drop ceiling torn out, [chuckles] the walls torn out, carpet torn out, wallpaper torn off. A lot of times just tearing things apart before we rebuild it because it's hard; it is hard to work around something that's already there if it's not quite what you want.

You know, in engineering, we a lot of times don't get that luxury. We have to work with what's there because it's in use. It's very different. When the old tenant leaves, you got to strip things out and rebuild it. We don't get that. Like, I can't think of any time I got that luxury, right? You got an application in production. You're trying to make money, and maybe you're desperately trying to make money if you're running a startup. And you can't have downtime. You can't inconvenience your existing customers. You've got to build around what's already there, and what's already there may not fit at all what you want to be doing.

I've heard it described as starting by building a shed and ending up building...you're thinking you're going to build a high rise and ending up building an airport. It's just [laughs] the way that we build because the way business needs change just diverges over time, so we have to work around it. And that's the topic we'd like to talk about today.

We'd like to talk about what it involves working with legacy systems, which is what almost all of us are working in, you know, every system gets older over time. And this is a chance to share stories, and we'll be sharing, doing some of that, but also talking about the principles. What do you need to think about as you're building around things that are already there? Sometimes they fit, sometimes they don't, but you just got to live with that. That's my introduction. And I'm going to stop talking and see what my fellow panelists here think, just off the bat. Any thoughts about this topic? Do you have any stories? Do you have any ideas about what you need to do?

MATT: Well, it's like buying a pair of pants in the ‘90s, right? Here we are, and now 2025, Happy New Year, everyone. Something we bought in the ‘90s, great at the time, in style and fit. Now, here we are in 2024, many years later, gotten older, put on some weight, styles have changed. And those pants, while we still may be able to cover ourselves up with them, they just don't fit, and I think you run into that a lot with software.

Business rules change, direction changes, and teams change, right, code languages come in and out of style, popularity, hard to find support. So, when we're building something, we generally think, oh, what's new and shiny? This would be great to build this in. And anyone who's been in the industry for any lengthy amount of time has probably done this, and that's, oh, there's this hot, new language. Let's take, for instance, ColdFusion, right? I think that's a great example.

MIKE: That’s the one I was thinking of, by the way [laughs], the same one I was thinking of [laughs].

MATT: Yeah, at the time when it came out, a lot of people jumped on that bandwagon and thought, oh, this is going to be awesome, you know, it's Adobe. They're not going anywhere. It's going to be around forever. So, we build applications in it. Then, a few years later, come to find out, nobody knows the language. It's been abandoned, and here we are stuck with a system that's built in ColdFusion. So, what do you do when you need to update it?

MIKE: Well, that's the question, right?

MATT: Right.

MIKE: Assume you're a new tenant now. Throw out the old system, build a new one? If you had a ColdFusion system, what would you do?

MATT: Yeah, I mean, perfect example, we have this case here at Upbound. We have a system that we use to track our capitalizable time, initially written in ColdFusion. And Acima historically has not used that system, and we had a choice, right? Either we adopt it with it not meeting our needs and try and figure out how to adjust it to make our needs, or we build something new. Well, here, we chose to build something new in hopes that it will be able to scale and support our needs for the future, and that's not always the answer.

MIKE: Oh, it's not. There's something to be said that scale matters. And that's why really big systems tend to be built in really old languages. You want to find COBOL? Go look in the airline, right [laughs], or a bank. It's been there for decades, and it just needed to keep running. You're going to find old tech because it was cheaper to keep it running than to build something new, even decades later. Go ahead.

MATT: Yeah. Well, that's a very, very good point. You know, we also didn't really build things in the past in service-based architecture, right? We weren't using microservices. We were building monoliths. And the bigger the application, the more longevity it's had, the harder it is to be able to rewrite that system. So, maybe an answer is, hey, what can we extract out of here? Can we isolate domains, pull things out, and then start to modernize, that word modernization, then we can start to modernize that way, right? Okay, what's a new language that's popular, has had some longevity, so we know that it's going to be around, and how easy are resources to find for it? And then, you start to make decisions that way.

MIKE: You hit right on it. A lot of times, it's not cost-effective to rebuild the whole thing. But there may be a component in that system that, well, a new component, right, that's attached to it that you can build externally. Or there may be a component you want to rewrite wholesale, right? Just rebuild that component. You're not rebuilding the whole thing. You're just extracting that portion of it out to a service, and then you get a new service that wraps the older one. Have you ever seen a strangler fig tree?

MATT: Yes.

MIKE: They're cool, right? They’re interesting. Birds poop the fig seeds into the upper parts of the tree, and it will start growing way up high in the branches of the tree, send down a root down to the ground, and then root itself there to get the nutrients it needs. And that tree will gradually wrap around its host tree until it completely envelops it. And, eventually, the host tree is gone, and you're left only with the strangler fig. And you end up with this big, beautiful tree that is still there, and the original tree has, you know, completely vanished.

You can build software that way, right? You start with just a little piece, make sure it's well grounded, and build it out around the old section until the old section is no longer needed, and it's just gone.

MATT: Yeah, and it's important, too, when looking at these legacy systems, are we sticking with the same language? If so, that makes a big difference, right? It's a lot easier to refactor, make changes than it is to try and work a new language into an existing system, you know, some can be compiled into other languages, you know, if you're running in the JVM, et cetera. But I think that's a big component where you have to start to make these decisions is thinking, okay, what technology is going to be used for it, and what components can we build with this technology?

MIKE: Well, and there's some pressure there. If you're thinking about extracting components, and this is, I think, a healthy thing [chuckles], it provides some pressure to be decoupled because if you're really loosely coupled between systems, then you can completely change the tech stack on one of them, and it doesn't matter, right? They talk to the API. As long as they both talk to the same API, it doesn't matter. Now, this applies at all scales, even within your app, but between them, if you have really loose coupling, it works out great.

There's the famous email that was written by Jeff Bezos at Amazon (Was it 20 years ago?) where he said “Starting today, no applications get to have any internal private APIs to any other. Everything's all public. You can only communicate over these public APIs, and if you violate that rule, you're fired. Thank you [laughs].” And people were kind of panicked by it at first, but it forced teams to have strong APIs that were decoupled from each other, and that's how AWS was born. It matters [laughs].

MATT: It does matter. And it'll help you build things that will have that longevity, right? Just because something's old doesn't mean it's bad. Maybe your business rules haven't changed in 25 years, and you built this initial system in COBOL, but it does exactly what it needs to do. Why would you rebuild it? There's no point in redoing things just for the sake of redoing things.

MIKE: Right. As long as you are still able to get sufficient resources at a reasonable price to maintain the system. And if there's some core business rules that seldom, if ever, change, and, you know, COBOL [chuckles]...I laughed a little about COBOL. It was such a huge technology. There's still a pool of developers out there. They're kind of pricey, but they're available, right? You could get one if you needed to.

MATT: Yeah. One of my very first jobs in the industry was converting COBOL and Fortran over to PERL and PHP. Now, in hindsight, was that a great idea, eh, you know, probably not. Nobody wants to write in PHP or Perl anymore either. But it's interesting that it didn't really need to change. All we were doing was duplicating the exact same logic in a different language, and it really didn't make sense. You know, maybe they were trying to be a little bit forward-thinking in that those languages were going to be dead languages, but they could have lasted for another 10, 15 years and been just fine.

MIKE: Honestly [laughs], Fortran's still got a pretty active community. That's nowhere near a dead language. I haven't looked at language rankings recently, but Fortran might be ahead of Perl [laughs].

MATT: Yeah, you know, and for a while in the early 2000s, Perl was it. Everyone loved Perl, right? CGI scripts and its file handling was exceptional. You could get a lot done with very little code. It still, to this day, wins Code Golf almost every time.

MIKE: Well, and Perl is actively developed. It's not a dead language.

MATT: No, no, not at all. It has its place, and web applications just aren't it.

MIKE: But there's still a lot of shell scripts out there in Perl.

KYLE: I think there's something to be said, too, about some of these older languages. The programs that are written in those they were written in a different time with different constraints. So, we're talking about some of these COBOL or Fortran programs. Sometimes you don't want to get rid of them because they perform. We're not hindered by the same constraints that we were a few years ago, so we don't take into account that we only have half a gig of RAM, right?

And it may not be worth to switch it over to, you know, Ruby, Node, or whatever you're working with that will use two gigs of RAM because of how different the performance metric is. Maybe you need to write an adapter or another layer around that to take advantage of your old applications.

MATT: Yeah, I think, as engineers, we've gotten a little bit complacent in thinking about those types of things, like memory usage, disk space, because it's so inexpensive these days, right?

KYLE: Yeah, you can just throw hardware at it. That's a common solution these days, isn't it?

MATT: Yeah, absolutely. Just spin up another container.

MIKE: It's not necessarily the wrong one, either [laughs].

MATT: No.

MIKE: Hardware costs, while pricey, depending on the hardware, depending, you know, this is conditional, right? But, you know, they can often be less, even if they're somewhat pricey...developers are really pricey. And if you can write something in a language that is easy to think about and easy to communicate about and you're not going to write bad code, it may save you money, even if it uses a lot more resources.

MATT: Absolutely. And, you know, just in this short conversation we've had already, we’ve already listed, what, 10, 12 things that you really have to take into consideration when doing this.

MIKE: We've talked about this idea of wrapping a legacy application, gradually splitting out components, building them in. I'm going to use...you laughed a little bit at modernize [chuckles] because it's a loaded word. What's considered modern is not necessarily a clear concept. But what meets the current business needs, building in some framework that meets the current business needs that, you know, wraps the older one, gradually replacing components of it until, if necessary, the original is entirely replaced.

Sometimes though, we work within the system, right? You're not going to split out a component, but you're working inside legacy code. And maybe it's been there for a decade or more, and many hands have been in there changing different pieces. There's lots of different design styles, lots of different architectural ideas, all kind of crammed in together, like one of those old farmhouses you see that's had additions put on three or four times, and they all look completely different from the other ones [laughs]. It looks like that inside. What do you do when you find yourself in a system like that? How do you stay oriented and stay productive if you have to do all that context-switching between, okay, what's going on here?

MATT: I think that's where you really have to keep design patterns in mind, right, and see what you can isolate. Can you eliminate dependencies, if possible, and really confine your new code so it isn't dependent on other things? I think that's a really good way to start making your application better and more understandable and just make it easier to work in in the end.

MIKE: You said, if I heard you right, make it so it isn't dependent. Can you elaborate on that a little bit?

MATT: Yeah, when we're a little less experienced—and I'm not saying that every legacy system is written by inexperienced programmers, because that's far from the truth—but if, let's say, I don't know Ruby, right, and I just started writing in this language. Likely, if I'm new to this industry, I'm going to couple things together that probably shouldn't be. You know, I'm going to pass dependencies through all my methods, and Class A may rely on something from Class B, et cetera, right?

And I think that's where you need to think about, okay, how can I isolate this? Is this really dependent on this other thing and could I make a superclass? If they require resources to be shared, let's share them through a superclass or whatever, you know, just eliminate dependencies and try and decouple things as much as possible so that code is usable outside of its current form.

MIKE: Got it. So, you're talking about, you know, detangling the spaghetti, decoupling so you don't have mutual dependencies and, you know, single-purpose code [chuckles] that is reusable.

MATT: Exactly. You're much more eloquent than I at stating that, but yes.

MIKE: [laughs] You said it first, just rephrasing [laughs]. Yeah, absolutely. Improving your little corner of the world, right? And you said, specifically around best design principles, getting each component to be serving a single purpose and to have a clear boundary between itself and the rest of the code so that it's reusable and not, you know, like you say, mutually dependent on other code.

MATT: Yeah, and you also said cleaning up your little corner, right? If I have a really messy bedroom, you have to start somewhere, so let's clean the closet. If I have a nice, clean, organized closet, that's a really good start because it makes room. And then, as you start to get that organized, it makes it much easier to organize other things because you've established a pattern on how you're going to do it and what your approach is going to be.

MIKE: That's great, and it reminds me of something my wife does. She borrowed this idea from elsewhere, but it's a brilliant idea. Every week, you focus on just one area, so you don't have to do everything all at once. So, like, this week, we're going to work on making the kitchen clean because there's always the things, right, the things that you put off, maybe the deep cleaning of the stove, or [laughs] cleaning out the junk drawer, right? And if you try to think about all the things that got to get done, you're never going to get them all done. You just throw your hands up. And the tech debt piles up, and [chuckles] you never get to it at all.

But if you allocate some time on a regular basis and a rotating basis to each of the areas, right? And in the home, maybe this week we're all going to work in the kitchen. Next week, we're going to work in the living room, next week bedrooms, whatever the case may be. Like, in your code, maybe we're going to do some front-end cleanup this week or this sprint, right? Next sprint, we're going to do some cleanup in area X, or maybe it's a group of sprints, maybe a quarter, whatever it takes to get some real meaningful work done. That's not everything you're going to be doing, but it's a portion of what you're doing so that you're continuously paying down the tech debt and improving the code.

MATT: Yeah, what's that show? Marie Kondo, I think, her name is, tidying up and about organization. There are so many things that are applicable from life into software engineering and software design and vice versa.

MIKE: Right. Absolutely. Exactly. There is hugely relevant information there, right? That's why, usually, at the beginning of our podcast, I talk about something outside of engineering because getting ourselves out of the mindset of just what's right in front of us, I think, helps us to see those connections.

MATT: Yeah, for sure. Eddy and Ramses, you guys have been in the industry for a while now, but maybe a little less than us old folks. What do you think on this topic? Have you had the opportunity to really get into some legacy code and take a look and say, “Oh, man, what am I looking at, and why is this such a mess?”

RAMSES: Yeah, every day [laughter]. Well, quite a bit. I’ve worked in a few different projects, I guess, now, and some are more than 6 years old, 6 to 10, maybe even 12 now. But, yeah, when you get in those kind of deep pockets, it's, yeah, it's just, you know, no one's touched it for 10 years, and people are scared of touching [laughs] it. It’s brutal. Yeah, that's always just interesting. You just have to kind of, you know, take a step back, figure out what's going on. And, you know, depending on the severity or the complexity of your change, just try and maybe not poke it too much.

MATT: And I think, too, when companies start out, you know, startups, people have a tendency to get a little more cowboy. And when they build software, they're looking to be first to market, right? Let's just make it work, get an MVP out there so we can get to market, and then we'll come back and fix it later. And, you know, often that doesn't happen.

Also, you know, you have a lot of freedom in that kind of a world where you may have seen a podcast or listened to a podcast or seen a video somewhere, video tutorial on some new pattern that you've never tried. And you think, oh, I'm going to try it inside of our application, even though we don't do it anywhere else. And those are the kind of things, I think, when we come across, we just kind of scratch our head and say, why is this here, and why did we do that? That and if I look back at something I've written two, even three, years ago, if I don't look at it and think, what was I thinking? I'm lying to myself, right?

RAMSES: Oh yeah.

MATT: Because we grow every day, and we're always getting better. We're learning from our peers, you know, other ways of doing things. And it's just, there's always room to make things better.

MIKE: How do you navigate a large system that's, you know, got a variety of different styles and approaches? I've got some thoughts on this, you know, on how you approach that. You're in a codebase that you don't fully know, right, and in a large system, nobody does [laughs]. Nobody knows the whole thing. So, what do you do to quickly figure out where you're located? You know, it's like one of those shows where they drop you off without GPS and no obvious landmarks, and you have to find your way out [laughs]. How do you do that in the code?

MATT: I like to start from the outside in, if I can, you know, you find your entry point then it makes it really easy to navigate your way to where you need to be working, or at least find out if you're in the right place, right? If we're in the Ruby world, lots of Prys. I like to throw a Pry in once I get to a comfortable place, you know, maybe it's inside of a service object a couple of layers in, throw a Pry in, and then look at the methods inside of that service and run them inside of my Pry and console just to see if I'm getting the expected results. That's one of the ways that helps me navigate my way through new code. Run the tests, I mean, look at the tests. First thing, right?

MIKE: You know, you said find an entry point. I was thinking something similar. When you have something you need to work on and no context, you've got nothing, right? It's a blank wall. And finding that entry point is everything. So, if you've got an error, see if they've got a stack trace, right, because that can take you to someplace. Now you've found a place in the code. You've got that entry point where you can put your debugging statement, or your print statement, whatever kind of debugger you are, and start tracing things down.

If you have some UI, then you can go to that page and try submitting some things, right, put in some logging on the server side and see what's getting hit. If you're doing heavy front end, you can kind of follow a similar approach. Start with something that you do know because you've got to know something, right? You've got to know something. If you really know nothing, then you probably either don't have a sufficiently written request or you need to go talk to somebody [chuckles] to get them to give you some context. There's always someplace to start. Getting that starting point is more important than almost anything because once you can pry in, right, then you can start digging down and going further.

I find that new developers often get...they get paralyzed. They get paralyzed by all of the unknowns, and that's totally crippling. Instead, if you don't worry about all the unknowns and start by finding something that you do know, then you can start with that seed- and expand that knowledge outward until you know enough to solve the problem or build out the feature that you're looking at. That having an entry point, as you called it, is most of the effort in many cases, in my experience.

MATT: Yeah, and as long as you have...and maybe entry point isn't the right word, reference point, right?

MIKE: Reference point.

MATT: As long as you have a reference point, you can navigate in all directions out from there. If I know I should be working in some, let's say, user class, right? I can also look and see, what is calling that user class? What's instantiating it? What's using its methods? And work your way out from there, and then slowly, but surely, your circle gets bigger. And you start to learn your way around, and you're able to navigate things.

MIKE: You know, thinking about the geography reference I made before, you know, you’re dropped off. You don't know where you are. If you can find a landmark that you recognize, that changes your world [chuckles]. That changes so many things because now you have, again, you have that center that you can navigate out from and start growing that circle. Otherwise, the Cheshire Cat, if you don't know where you're going, it doesn't really matter which way you go [chuckles]. But if you know where you are and where you're going, it changes everything.

So, storytime, maybe for us older folks here especially, what are some times that, you know, some crazy stories that you’ve found in legacy systems? You know what? I'll lead out here. I'm going to start with a story. I'm thinking back to...I think it was my first full-time dev job, although it wasn't right away. I'd been there for a while. We were given a customer management system, and somebody in the company decided it'd be a great idea to buy this. And the guy who sold it to us was just, like, some person out of his garage.

And the whole thing was built inside a SQL Server as stored procedures. And when I say the whole thing, literally, the whole thing. There was, I think, a little bit of UI written in Microsoft Access that gave it a little bit of forms on the front. But then all of the business logic was written in stored procedures, which were stored inside the database that were not under version control.

There was no source control at all, were not searchable [laughs], and had no, you know, there was no rollback. You know, if you make a mistake, it's permanent, and it's live. There were also some calls out to compiled executables for which we didn't have the source code. [inaudible 31:31] here you go, make this work, and we had to make a number of changes to make it work for us [laughs]. It's hard to describe what it was like doing that [chuckles], you know, but --

MATT: That’s the thing nightmares are made of.

MIKE: [laughs] So, say, we make a feature request. Well, I'd start with the form in the UI and then figure out which stored procedure it was calling to, right? And then, in that stored procedure, it would usually call to four or five others, which themselves would call others. They were nested, so, you know, they were nested all over the place. And then, just step through those again.

It's not searchable, so you have to find its reference in the UI [chuckles] for a SQL Server and go...and this was a long time ago. It may be better now. But go find that other stored procedure, find what it linked to, and you could trace your way down eventually. Again, no debugging statements. That was not possible. You just had to try to understand it. And by looking long enough, you started to understand the geography [chuckles] of it and know what was needed.

MATT: Yeah, that, you know, we think today, what did we ever do without tests in our software? And then, you go back further, and no version control, no tests, multiple people working in the same code. And it was like, let's shove this file up via FTP, and then someone else is working in the same file and overrides your changes, and it breaks everything. Oh, man.

MIKE: [laughs]

MATT: Those were the days where you’d get really good at tailing logs.

MIKE: Yes. Or the system I'm describing where there is no test system. Any changes you make are live in production the moment you make them. You get really good at understanding what you're doing before you make a change. So, there's my story.

MATT: I don't know how we navigated that world.

MIKE: [laughs]

MATT: But somehow we survived it. And you're also in charge of keeping the servers up and the network and everything else at the same time.

MIKE: All at the same time [laughs]. Oh, I can tell stories of mistakes I've made.

MATT: I've brought down a number of production systems throughout the years [laughs].

MIKE: So, I gave my legacy system story. What's your legacy system story?

KYLE: What I'm thinking about right now is back when I was an intern for a company that I won't throw them under the bus. But I was an intern and then a full-time employee at one point. And the issue that we had there was we had multiple versions of their software to serve different customers, all written in different versions of Java. And so, what it was is you had different looks and UIs and different functionality bleeding through these different versions of the applications but also had the same functionality.

And my task at that company is I was a manual QA engineer, so I was tasked to test these. And so, we would be provided with specifications, but we had to know how this was implemented in every version of the application for each of these customers. Just how bad that was, like, just the test matrix on those types of systems was just insane just because I was writing bugs.

And half the time, I was right in the sense that, oh yes, it should work this way in this application, but not in this version of the application. And there was times where I misunderstood the requirements, and, no, it shouldn't work this way for this customer, but it should for this other customer. Choices like that where we're creating different versions and getting customers stuck on different versions like that that's one that really stands out to me.

MIKE: There used to be a popular way of doing things some years back, where every time you got a new client, you'd spin up a new version of the app, like, their own complete, hosted copy of the code, rather than having a shared web app. And oh, what a nightmare [laughs]. That is so bad.

KYLE: [inaudible 35:52]

MATT: Yeah. I worked in the lead gen industry for a long time, you know, spammers. And I worked for a place that did that with education. It was all of the for-profit lead gen education. And every time we got a new advertiser, we had to spin up a new instance of the application. But they would share, like, the cost per click, cost per acquisition, all of that.

So, if it changed when spinning up a new advertiser, we had to go in and change 50, 60 different independent applications for every partner every time we made a change, and no version control. It was all FTP. And this would have been, I don't know, I think, maybe 2001 somewhere about. It was just a nightmare. It was an utter nightmare to try and maintain that and keep it up to date.

The application itself was built in a hacked version of Mojave framework, so maybe it was a little later, 2003, 2004. Someone had torn apart Mojave framework and customized it to do this lead gen stuff. And it was just...it was miserable. Just the folder navigation alone was a nightmare.

MIKE: So, I’ve learned a couple of things from these stories. Lack of version control is...are there any systems out there without version control? I hope not. But if there are, if you're thinking about doing it, don't [laughs]. If you're starting in a new project, it's worth it from day one. Get it up in Git. It's worth it. There is no question. There shouldn't even be a hint of a question. If that's a question in your mind, you're doing it wrong [laughs].

MATT: First thing you do, git init in your local directory. It doesn't have to be hosted. You can do it locally if it's only a personal project but do it.

MIKE: Yes. Second lesson is, don't copy your code. Have a shared version. If you need to have customizations, build those in as some sort of feature flagging that you can turn on and off, and you have a configurable system. It's easy. Now you've got a configuration system that users can use to toggle, and it's great, and everybody is happy. Don't build a new system for every client. That is the path to madness.

MATT: And test your code.

MIKE: And test your code. Test your code. You know, things that are best practices sometimes you don't think about because they've just become so ingrained. Have you ever been in a situation where you don't have those? And, you know, a lot of students doing their coursework haven't done unit tests, right, maybe haven't done version control. Maybe just make another copy of their code when they need to make some changes. I mean, there are some things here that are easy to do when you're just getting started because it seems like it makes sense at the time.

There's a reason that these [chuckles] have become standards in the industry because they will save you. It's like autosave in a document. Back in the days when you could write up 30 pages and then the power goes out, and you lose it all, knowing that that would happen, autosave is the thing you do. You just don't write a document unless you have something persistently saving in the background, right? I mean, this is so universal now. It applies from command line editors up to everything, right? But there was a time when that didn't exist. There are some things that you just do from the beginning, and you just don't consider them negotiable because it matters that much.

MATT: Yeah, and it's a little off-topic, but it's just making me think about some of these horror stories. Looking back on the very first time I ever saw Subversion, [chuckles] and, you know, what was it before that, CVS?

MIKE: Yeah, concurrent version, what is the S for?

MATT: Which just stored the entire file, not just changes, and was a huge memory hog, but it worked. And then, one day, I was working for a place, and our manager came over and said, “Hey, as of today, we're going to start using Subversion.” And that's all we were told, no training or anything. Needless to say, about the first 20, 30 merges that happened always broke everything [laughter] because no one knew how to do merges, no one knew how to resolve conflicts, and it was just...it was chaos.

MIKE: Well, there's also something you said there about gradual transitions, meaningful, planned transitions, and not hard cutoffs, training.

MATT: Yeah, and training.

MIKE: Subversion was a great step forward. I loved Subversion back in the day.

MATT: Oh yeah.

MIKE: And then, Git was a great step forward beyond that. There's a reason why the industry...there are a few others out there, Mercurial, that some people use. But Git has just dominated. I can't remember the last time I saw a repo that wasn't in Git.

MATT: And now it's just second nature, right?

MIKE: Yeah.

RAMSES: Yeah, Git is awesome. It's great.

MATT: I wonder if this new AI boom will change the world of repository management.

MIKE: That's interesting. Totally off-topic [laughs], Nvidia recently came out with their new graphics cards, and they get, like, somewhere near 250 frames per minute, or something. And they do it, you talk about the AI, by predicting what the next few frames are going to be via AI. It takes the frames that you've seen in the past, predicts what's going to happen next, and pre-renders those frames, pre-renders those frames, and then just makes any tweaks as necessary, which seems wildly inefficient and crazy, but it makes a tremendous difference. It pushes things forward so much further.

To your point about...I bring this up because you think, well, with version control, what's AI going to do? Well, sometimes, if you have good enough AI, you can do surprising things that you'd never think was possible before.

MATT: And, you know, it's relevant to these legacy systems, right? A lot of these legacy systems have tasks that could be accomplished integrating some of these LLMs and, you know, even machine learning.

MIKE: Now, as of today, in early 2025, I would argue pretty strongly that you can't just take existing AI tools, point an old repo, and say, “Hey, convert this into something new.” It's not going to happen.

MATT: I second that. I've tried.

MIKE: [laughs]

MATT: Just, you know, like, fiddling around with it, seeing how intuitive it really is. I still would not trust it to write my code, though I do trust it to write base tests. I've been doing a lot of Python work lately, and Copilot is great for writing unit tests. You do have to validate them and make sure that it's testing exactly what you want, but it sure saves you a lot of time.

MIKE: Well, that kind of goes to where I see those tools being right now. Think about having an extremely fast but somewhat inexperienced and clumsy junior engineer, and they’re there at your disposal and can do a lot of things, and they don't mind any task you could throw at them. Big tasks, they're probably going to get wrong, and so you shouldn't. But something like writing unit tests, if you can knock those out, well, that's fantastic. That can save you a ton of time.

KYLE: I’m finding it very good for the idea of a rubber ducky. That's what I've been using AI for is it's a very intelligent rubber ducky.

MATT: Oh yeah.

MIKE: Having that rubber duck to talk to is super useful [laughs]. It's super useful. I think that we're kind of reaching a good stopping point here. We've talked about the fact that we're all working with legacy systems, almost all of us, right? Most people don't have the luxury of always working on something new. And the thing you're working on new is not going to be new forever. Eventually, it becomes a legacy system. And there's a variety of techniques you can take to deal with that. Wholesale rewrite, if it's small, might make a lot of sense. Usually, that's not something you have the luxury of doing, but you can split out components and rewrite them in a manner that is better aligned with your current business needs.

And if you have a system that you're just continuing to work in, you can find reference points within the code to navigate your way. And then, as you make changes, you can gradually improve. Pay down the tech debt. Take the time to pay down that tech debt so that your system stays maintainable or becomes more maintainable. Maybe it's not very maintainable, but it becomes more maintainable over time.

And there's some things we can learn from bad systems, too. Follow best practices. Follow best practices. They're there for a reason. Write unit tests. Use version control. Do the things that play well with others because one of those other people is future you, too [chuckles]. It's all worth doing.

Until next time, on the Acima Development Podcast.