Conversation

Before specifics — if a colleague or client had to describe what makes you different from other smart, capable people in your field, what would they say?

¶ honestly, they would probably say I'm the smartest person they have worked with. That's probably the number one thing that sticks out. It doesn't mean I always make the right decision or even I'm the most impactful person they have worked with necessarily, but raw intelligence and processing power is extremely high.

¶ After that they would probably say the consistency of passion, energy, optimism (and/or positive energy) for solving unique problems. I don't know how often this has been spoken directly to me, but I am extremely consistent with this. I show up everyday with a positive attitude ready to attack problems and do my best. I think to a level it's a bit unique. I don't have grumpy days at the office or days I let my standards down.

¶ On the probably negative side, I would also say I might have a tendency to "over engineer" or over think things... which probably more traditional software engineering terms is to build for the abstraction too much. Possibly pushing a bit more towards "idealism" than reality. An example would be a project that isn't too hard for me to maintain, but I probably was a bit too optimistic about an average team with average talent executing it. Things that are not overly complex to me can be overly complex to others, and I don't always do a great job of predicting that. Sometime there might be a simpler path to value.

¶ I would then say last I don't care about what the trends are or what the "consensus" best practice is. I am not overtly contrarian, but I don't buy into the logical fallacies of the "experts' or majority in the slightest. If something doesn't seem to make sense to me, I will think outside the box, come up with my own architectures and standards even if I know many 'respected" people in my field would actively disagree with me.

Two things: first, how do you frame your professional identity in a sentence or two? Second, that passion and optimism you mentioned — what specifically fuels it, even on the hard days?

¶ Related to this, one more note to add to my strengths as I separate myself out SPECIFICALLY in my field / domain is I think I am extremely good at rigorously connecting what I am building to business value, by working backwards from understanding the business. I very quickly can get up to speed in a new department or line of business and understand the key drivers and what we need to impact... and if I don't I'm very good at working with the business stakeholders to extract that information and bridge that gap. I think a lot of smart people build really elegant solutions and stakeholders love them, but that doesn't mean they are providing real business value. I am obsessed with grounding everything I do why it should ACTUALLY drive business results.

¶ As an example, I don't just build a dashboard to support data driven business decisions.. I consider if the data they are asking for is actually the best data to make those business decisions. I understand that just because you are making a decision supported by an exteremely well executed dashboard, if that dashboard is focused on the wrong things or doesn't analyze the problem accurately (proper segmentation, context, correlation causation traps, etc..) the business may FEEL like they are getting value out of that dashboard, and their analysis might claim they are saving or making money, but at the end of the day the business might be no better off than if they made the decision on gut feel and instinct and no dashboard.

¶ The field is data and analytics, end to end. Specific jobs may be more left (ingestion, modeling, data warehousing) or right (consumption, dashboards, AI/LLM solutions) but it's end to end analytics. how do we capture, model, and distribute data to drive business value and solve tough problems.

¶ I think it's a combination. There's a part of me that feels a responsibility to use my gifts to the best of my ability. I understand I am lucky enough to be blessed with an incredibly powerful brain, and I should be using it to best of my ability. In order to do that I have a great desire to maximize human capital. How do we get more out of the human race? How do we keep driving civilization forward? There's also the altruistic desire to earn what's being given to me. My paycheck or the clients check to SDG represents an exchange of money for value, and I feel obligated to fulfill that to the best of my ability.

React to this trend — **"AI is replacing the data analyst"**: LLMs + NL-to-SQL meaning business users just query their data directly.

¶ I would expand this to include chat with data over semantic layer (Ontology, semantic model). This goes back to my point earlier, which is just because you are using a data product to answer a question does not mean you are providing the business value. As it relates to this, not everyone is a data analyst, wants to be a data analyst, or is capable of being a data analyst. If we pretend anything is possible in terms of end users answering their own questions and building their own artifacts (reusable recipes, self made dashboards, push reports, whatever), this does not necessarily mean they will be providing the business any more value than if they were just "winging it". It's very possible they end up making worse decisions because they see false signals and become over confident or run into the problem of "the illusion of an answer is much more dangerous than no answer at all". I think if we are going to be putting this type of tool out there, the business must focus on CPM / BPM, truly deep diving on what metrics ACTUALLY matter and how to think about them..

React to this trend — **The semantic / metrics layer push**: dbt Semantic Layer, MetricFlow, LookML: define metrics once, centrally.

¶ This is related to above. This is important, but doesn't matter if the metrics aren't any good in terms of actual bottom line business value. I think the biggest problem here is what snowflake is attempting to solve with the, not sure the exact name, but open semantic interchange it might be called. Right now there are so many flavors of how to write these semantic models and they all answer certain things better than the others... and I don't know any of them are really that flexible to answer the wide variety of twists and nuances on the metrics without becoming a monster to maintain and not really any value add compared to the alternative. I do think chat with data is very important and some layer of this matters (ontology in general), but I'm not super sold on the benefits of trying to put everything into a semantic layer. If you were to pressure me today I would say some version of lightweight semantic model as a reference point or base for some extremely core and common data marts / metrics... but focus much more on the CPM side, business process specific examples. I'd rather spend a week working with the business, documenting examples and queries and deep dive analysis that resulted in real business impact than building a formal semantic layer. The north star first principles, traps, data nuances in that week long deep dive can give conversational analytics some of the specifics it needs to steer end users towards positive ROI instead of negative ROI. "The answer to your question is this, but this hides a number of real world issues that need broken down or clarified before you make your decision", or something like that.

React to this trend — **Data mesh**: decentralized data ownership, domain teams owning their own pipelines.

¶ I don't think this matters all that much in practice in the sense that each company realistic is limited with exactly how much flexibility they truly have here based on size and scale and scope. That said, I think the best model is generally centralized IT as a center of excellence and data mesh as much as is realistic. For many that may look like IT owned resources that specialize in one or two business functions, but still roll up to IT. That's fine. But the principle is the same as before. You have to understand the business to deliver valuable data products, and in a larger company you generally you have to understand it better than the people making the data requests.. you have to understand it at a c-suite level (or damn near)

React to this trend — **The modern data stack consolidating**: Snowflake vs Databricks as the two poles.

¶ In general I am all in on snowflake for simplicity and favor consolidation and simplicity just to make things as straightforward as possible for your data team. I don't understand stressing over the pennies and ending up with tech sprawl compared to paying more and keeping it simple. 15% markup on token cost for a snowflake managed solution for example to me is a no brainer compared to self hosting on azure.. unless you are talking massive scale. That said, my favorite data stack right now would likely involve snowflake+dbt as the core, motherduck as the gold presentation layer, and feldera for heavy incremental processing. Depending on the business you may be able to adjust that quite a bit... but I do think the landscape is quite different than what snowflake was originally built for. I think distributed, smaller scale, incremental pipelines make a lot of sense and the value and leverage of what snowflake was built for is not as much as it once was. I am not a technical expert on the actual engineering side of databases and the detailed pros and cons of execution.

React to this trend — **Agentic AI / AI data engineers**: agents writing pipelines, transformations, and data-quality monitoring.

¶ AI I think can help an extreme amount on drafting structures, template architectures, Unit Testing, code reviews, smoke tests, and improved CICD particularly... and I think advanced incremental pipelines and column level lineage is pretty vital to that. I think the best data platforms are going to be built and engineered from the ground up to be maintained and enhanced by AI.. but I think we are a far far ways off from AI being able to handle a variety of actual engineering challenges. I always laugh when these topics promote how much time it saves building an "end to end" pipeline because normal data modeling on relative clean data was never the blocker. A data pipelines time to value is not limited by writing joins and bringing together pretty clean dim/fact out of source systems. It's limited by understanding the nuance of the business process and the data and handling that complexity consistently. I do think there are interesting architectures within the realm of reason that could help here. If you have the right environments, test data, and unit tests you could probably set up a "goal driven" AI pipeline that understands the end goal and iterates on solutions until that is met... if it has the right set up input data and unit tests to work with. Writing the unit tests would still be the hardest part in terms of working with the business to understand all the scenarios that are possible (and if you can write off / solve for differently any edge cases). Even there though, if you had the raw data AI can help speed this process as well. Ultimately it ends up back where we started... How do we know all of this is even valuable? That's where AI is still not even close to close, but probably can help smoke out initial pass and expedite the discovery process. I think business will succeed and fail by using AI to leverage either luck or skill presented as great business intuition. The companies that move fast with correct intuitions on what levers really matter and how to think about them and apply them to future (extrapolation is inherently risky) will be able to build analytics processes and CPM that makes their AI assisted workflows and automations provide positive value while others are fiddling with none or even negative ROI. Likely a story of rich get richer.. but also can make disruptors more disruptive faster.

React to this trend — **Vibe coding / AI-first development**: LLMs writing most of the code, humans supervising.

¶ First thought, with a qualification at the end... this is absolutely the way and thinking differently just means you haven't figured out how to do it right yet. Even if no one in the world has yet (this is almost certainly not true) there is absolutely a way to where AI writes the majority of the code with proper north star principles, examples, goals tests standards, etc. That said, I'm flexible on the exact ratio that is optimum here. Would I be extremely shocked if someone said that is 40% AI generated code in mature tech stacks is purely AI code with 30% AI started and heavily modified and 30 % by hand? Probably not. I just think vibe coding and having the AI assisted development offers so much more than just the code output. I think it can empower so much more that we were limited by with a fleet of human developers, while also yes...having some of the same drawbacks as human developers.