Al3x' Tech Blog

My quest for identity in Software Engineering

29 Jan 2020

Keep walking. If I look back I am lost.

― George R.R. Martin, A Dance with Dragons

They say that the sum of all our past choices and experiences makes up most of who we are. Despite that knowledge I never liked to invest much of my free time thinking about the past.

I always thought there’s such a thing as the right to forget and to be forgotten, that the opposite (to look forward, to imagine how the future could be, some might say to dream) is one choice that more consistently brings better outcomes.

If only I could perfectly stabilize between those two opposites… but that’s maybe for another time.

Faulty Random Access Memory

I have to believe that my actions still have meaning, even if I can’t remember them.

― Memento

There’s more. It turns out that our brain is not very accurate at remembering.

Contrary to the common belief, it doesn’t store a perfect snapshot of reality as a calculator might do: immutable, pixel perfect and ready to be retrieved at will. Multiple loads, same exact results.

The human brain apparently doesn’t care much about von Neumann architecture and goes its own way as it did for the latest three hundred thousand years or so.

When remembering, i.e. creating a representation of a past experience in our mind, the hippocampus coordinates the decoding of various media chunks, loading the bits and pieces scattered (should I say sharded?) between different areas of the neocortex into a coherent entity.

There’s consensus between neuroscientists on how incredibly plastic our brain actually is, how it’s constantly and frenetically rewiring itself, creating new pathways and dropping established ones to make room for new models of reality. All of that while running countless maintenance jobs in the background. It is hard to say to what extent this frantic synaptic rewiring activity is messing with the memories of what actually happened.

That’s not the point I want to make with this article though, neither looking forward is a dogma for me. I always liked to keep flexible, leave doors open and go out of the ordinary when it makes sense.

The more I get older, the more I recognize circumstances where it does make sense so, with my 40th birthday knocking at the door, it feels safe and fair enough to look back and indulge in some wrap-ups and where to go from here kind of reflections.

Special Circumstances

Just remember that the things you put into your head are there forever, he said.

― Cormac McCarthy, The Road

If I do look back now I can’t think anything other than “it has been a great ride”. Sometimes harsh. Sometimes exhilarating. Never boring.

The truth is, I was lucky. Not because of some mysterious invisible force keeping me pointed in the right direction, neither because of the average quality of the choices I made.

I was born lucky.

Lucky enough to spend the formative years in a rich economy that gave me, between other things that today for many are still mere luxuries, a solid education paired with early access to new fancy technologies.

On top of that, I had a very timely chance to give a closer look at what back then was the new rising star in the global sky: the “World Wide Web”.

How deep is your learning?

I’m floating around the universe on this giant sphere, suspended here by gravity and going for a ride.

― Anita Harris, Conscious

With a growing stack of knowledge and experiences coming along wherever I’ve been, I was allowed to keep building new skills needed to constantly practice a profession that I still love today as much as I did when moving my first steps, if not more.

Since then I could travel far and abroad, always to find a more than decent occupation virtually everywhere without much of a sweat. Titles have been ranging ever since from Sysadmin to System Engineer to DevOps/SRE Engineer but those labels never truly mattered to me: I just wanted to deliver value doing what the business needed and it turned out their IT infrastructures needed a lot of care.

I met a lot of professionals on the way. Some of them brilliant and ingenuous. Some flamboyant and inspiring. Some pretentious and annoying. Many though, if not all, joined by a common passion for what we crafted.

That multicolored culture within the culture, made up of uncountable people and ideas and cool new technologies and funny but visually ugly memes (…and on and on), soon became fuel for virtuous loops of discover => learn => practice => discover that, almost mindlessly, kept me moving ahead doing what I liked and getting a pretty good living out of it.

Eventually inconsistent

Do not confuse duty with what other people expect of you.

― Robert A. Heinlein, Time Enough for Love

I’m increasingly unsure about how to answer the question “Would you do any different?” though.

I dropped out in the middle of my engineering studies and still forget big O magnitudes 10 minutes after a coding interview is over. I never worked for a BIG nor visited Silicon Valley. I didn’t write a kernel module nor create any widely used (Free and Open Source) software. I was never fully in charge of any product beside when freelancing or working on a personal one.

I could go on indefinitely.

Heck, let’s be honest, I was never the best in anything I did, neither while studying the theory nor putting it into practice. For sure I missed out interesting but invisible (to me) opportunities, and a few times I even dropped already taken good ones while pursuing other more important goals.

Finally, I lost the count of the many pet projects I fell in love with and forgot soon after, a behavior that at this point we can safely call chronic. It’s true that some of those projects left a mark on me, increasing my knowledge and experience, but none of them ever truly conquered my heart.

Maybe it’s true what they say then: the first love you can’t ever forget.

Contacts[0]

Common misconception that; that fun is relaxing. If it is, you’re not doing it right.

― Iain M. Banks, The Player of Games

I first got interested in the web around the late Nineties, when I connected to the Internet with a noisy 28.800 baud modem and upgraded to a PPPoE ADSL soon after.

Broadband connectivity in 1999… that was luxury!

The initial heat soon dissipated though. I started to believe I missed a key natural skill required to become a decent (web) designer, and if you see my hand writing you’d know instantly what I mean.

The first attempts to set up from scratch a decently looking template for a website put me off as an impossible feat and made me wish for something… more ergonomic.

To be truly honest with you, the main reason I gave up is because it wasn’t fun.

Got ROOTs?

Control can sometimes be an illusion. But sometimes you need illusions to gain control.

— Mr. Robot

Sending cryptic commands to an IRC bot while waiting for a recompiling (2.x) Linux kernel. Configuring MTAs to securely relay and deliver root@localhost emails to some mbox. All while jumping between one (hopefully secure) shell to another, palms sweating before hitting enter when commands ended in something like -j DROP.

All that was way too much fun to go in search of any other digital adventure. Even better, I discovered almost without realizing it that businesses were already willing to pay me to do that, full time.

I soon lost interest in the new world of web development while I dived deeper and deeper in the one of networked computer systems. The back, the infrastructure, the invisible end felt like the right place to be: abstract, loosely constrained, wild and unexplored. So there I decided to put my roots (pun intended).

The web, the frontend, was out of my way for good. Or so I thought.

Breaking (out of) DevOps

“Evolution [is] a random walk across a minefield, not a pre-ordained trajectory, onward and upward toward perfection.”

― Greg Egan, Permutation City

Jump forward a couple of decades and here I am, microblogging about Vue.js, discussing about Serverless every chance I get and subscribing to (excellent) web design courses and CSS RSS feeds.

Wait… WHAT?

It looks suspiciously like the classical mid-career/mid-life crisis, right? Just a feral impulse to an easy change: let’s get rid of the past with one clean stroke, reject altogether the idea of self-managing backend systems for good. ssh is dead. Get over it.

I’m afraid the truth is far from that, even though my opinion is that today there’s generally no real need to orchestrate any fleet of servers running SSHd… unless you are deep in legacy territory, a (ti)tty fetishist or some kind of (cloud) service provider.

SSH is dead. Long live SSH

Humans are meant to work and sweat to earn a living. That’s the lesson.

― Cowboy Bebop, Jupiter Jazz (part 1)

I’m not saying server administration is dead. Heck no, long life to Linux & Co. and all the community around it, and most importantly, my sincere compliments to all the System / DevOps / SRE Engineers out there for keeping the Internet alive and kicking, 24/7 365 days per year.

Taming the operational complexities in this era of Global Scales, Spot Instances, Chaos Monkeys and Multi Regions is a very difficult task and one that most of the time gets only little appreciation (to the point of publicly asking for some every new year) because it’s not on the front of IT.

A smarter move might just be to let those brilliant and hard working professionals take care of that, those who have been running the (invisible to the end user) Internet infrastructure for the last decades or so and today most probably work at AWS, Digital Ocean, Netlify and so on. They are the best suited to take care of that anyway, so everybody else can focus on tasks more directly tight to the actual business at hand.

I’m not saying that you absolutely must avoid old style operations involving servers either. I’m just stating that if you’re starting a new software project today, adopting one or more servers (because that’s what it is going to be: you’ll need to nurture them for as long as your business runs) might not be the best way to go.

Simply put, today system administration is most probably a skill you don’t want to internalize unless it’s a core pillar of what you’re building.

Free lunch? Maybe just a cheaper one

The task of a craftsman is not to generate meaning, but rather to cultivate in himself the skill of discerning the meanings that are already there.

― Cal Newport, Deep Work

Unless you operate in some specific and/or constrained context (or if you’re operating at Netflix kind of scale and it might economically make sense to self host), today there are simpler and more effective methodologies that help you build and deliver highly performant (and massively scalable) software products without the need to own nor manage any (virtual or otherwise) server.

Even if you are a strong old school advocate or worry about cloud lock-in, you should at least agree with me that the benefits (auto-management, built-in autoscaling, reduced attack surface, cheaper bills, faster Time To Market… to name just a few) are too many and too appealing for discarding those new trends as just a buzz.

Serverless Computing, Jamstack, Static Site Generators… today all of those technologies are mature and my opinion is not mine alone: many successful enterprises are adopting them as we speak and at an increasingly faster pace. A few of those are kind enough to openly discuss about them so we can make better informed decisions when evaluating adoption of those technologies and adapt them to our context.

To embrace the new a small but sensible paradigm shift is needed and letting go of old familiar practices always comes with some friction. In my opinion though there’s nothing truly dreadful as you might soon realize by yourself once you break the ice with those (only relatively new) concepts.

Sir Chaos, please have a sit

I must not fear. Fear is the mind-killer.

― Frank Herbert, Dune

We’ve been using web services to build our applications for long now. We even survived the global Micro-services invasion that, even though they massively increased the complexity burden on operations, forced us to think about our applications more in terms of fault tolerant, self-contained and independent blocks, orchestrated by asynchronous communication delivered via remote APIs.

I’m oversimplifying here, but I guess you get the point. AWS Lambda today is only 5 years old, Fargate 2, but if you (try to) keep up with the tech landscape it truly feels like those kind technologies have already become a commodity. In time even BOFH kind of sysadmins are learning how not to worry about it… and even love it.

Chaos, unpredictability, is not to be feared.

Maybe it’s even your best ally, the one who keeps your edge sharp: listen to relevant events only, embrace eventual consistency, expect peers unavailability. These are the new kinds of mantras.

At this point should be obvious that personally I bet the new dudes and dudettes are here to stay, so if you have read so far and you’re not thinking yet I’m completely delusional, try to follow me a little longer while I look forward and keep walking.

join(past, present, future)

“Who are you?” he would ask her every day.

“No one”, she would answer.

― George R.R. Martin, A Feast for Crows

I don’t feel there’s any discontinuity, neither practical nor ethical (professionally speaking), between being in operations and developing applications following Jamstack principles.

At this point we’ve been sharing tools and practices between devs and ops for so long that they’re overlapping to the point of fading almost completely. Take this example of a common daily workflow of one of your colleagues:

  1. pick an assigned ticket
  2. analyze its content in search of meaning
  3. identify the relevant repositories and relative file(s)
  4. fire up an IDE to edit these files(s)
  5. implement the requested functionality in a feature branch enforcing best practices
  6. add (unit/E2E/whatever makes sense) tests (move this one up if you prefer)
  7. iterate until all tests are green
  8. push the patch and wait for code review
  9. wait for thumbs up, go back to #2 until done
  10. squash/rebase/push the new commit to master
  11. wait for the CI/CD system to run tests, build artifacts implementing the requested change and deploy them to production
  12. keep an eye on telemetry and alerts to ensure everything is fine
  13. set the ticket state to closed

It might be a bit contrived but it shouldn’t be that far from the real thing. Can you tell his/her job title from this list alone? That old fence line has become quite thin, hasn’t it?

And it might be a very good thing too. We’re left with only engineers working on some task at hand regardless of age or gender or (previous) job titles, everything redundant like labeled hats dropped for good.

Open Change, last revision

Law 25: Re-Create yourself

― Robert Greene, The 48 Laws of Power

With that said, whether you consider yourself a dev or an ops or just a wildcard engineer, it seems to me that the foundations of software development and operations did not change much from my first days as a junior.

Performance, maintainability, availability, data consistency and confidentiality… all those needs are still with us and relatively unchanged. Some might be new (I might be wrong but for example accessibility didn’t feel that important back then), most of them are not.

To sum up with a one-liner, the need to get the work done is with us today as much as it was in the past, virtually untouched. Stakeholders needed to get it done then as much as they need it today, they never cared much how fancy were the names filling up the tech stack as long as they were functional in hitting the mark.

Real Matter

No fear. No distractions. The ability to let that which does not matter truly slide.

― Chuck Palahniuk, Fight Club

I’m sure some of you grins at the idea of a Perl runtime serving traffic via Apache and CGI today. Try to explain why that’s funny to amazon.com or booking.com shareholders, to name just a couple.

Trust me, I don’t reject old and battle tested boring technologies at all, quite the opposite.

I work with the terminal every day and I couldn’t get anything done without the repeated use of grep and Unix pipes. I still love to code in Python even if I feel like everybody else is jumping on Go or Rust or… put your favorite modern language here.

I just say, let’s use the best tools at hand today for the job avoiding all that’s unnecessary.

Sure, the tech landscape evolved in the meanwhile: tools, frameworks, methodologies got new names and restyling. Practices to better manage complexity evolved along with them but nothing ever truly got rid of it completely.

For each new approach that gave us a step forward in simplicity, soon we got more pressure in the opposite direction, pushed by needs like bigger scales of operations, more integrations, more delightful user experiences, smaller time to market… and the list goes on.

Once the two opposed forces have stabilized, new balances emerge making room for bigger amount of business value, delivered faster and to more (hopefully delighted) users.

SQEmpathy injection

Anyone got time for a code review?

― anonymous colleague

Most importantly, the need for empathy is also unchanged. We might write in computer languages, but the ones who read and write in those languages all the time are, as far as I know, still humans.

Empathy for the end user AND for the developer both should drive our efforts, and if you’re reading this article there’s a good chance you are one of the latter… so do yourself a favor today and remember to be empathetic to your future self too.

I don’t refer to the kind of empathy you might feel for someone recently friend-zoned but the kind you feel when you see a colleague struggling with something unnecessarily complex.

Care for user experience is an act of empathy. Why should only the end user benefit from it? Software engineers are software users too and have the same right for a good experience.

That’s at least what I always tried to keep in mind while trying to write maintainable (Configuration as) Code automation. (If you are a former colleague, trust me when I say I truly hope you didn’t have to swear too much when dealing with what I left behind).

I believe that maximizing value delivery to the business should not be the only deciding factor when choosing a technology stack. As the developer tools and practices evolve and become more ergonomic, it’s reasonable to wish for adopting the new and leave the old where it belongs.

In the end people make the product so is reasonable to believe happier developers lead to better products.

Left in the middle. Down to Earth. Up to something?

There are no two words in the English language more harmful than good job.

― Whiplash

So here I find myself today: unable to define home more precisely than planet Earth, neither to define a job title any better than an all encompassing Full Stack Software Engineer, in an evolving polyamory relationship with browser dev tools, APIs, FaaS runtimes and container orchestrators.

In balanced tension between back and front, the old and the new, the boring and the bleeding edge, the server and the (perceived) absence of it.

In time, who knows, I might finally meet that redhead, the kind who keeps telling “you know nothing!” and follow her in yet unexplored wild(lings) territories, leaving all my (techie) legacy behind for good.

Until then I guess I’ll keep telling myself the same thing I’ve been telling myself for the latest 20 or so years: there’s never been a better time to be in IT.

References

I link here some of the media content cited above. The list represents part of what I have read/watched/listened to (either very recently or long ago but still stuck in my head) that inspired me and I find worth recommending:

* hashed with MD5, i.e. never shown nor stored in plain text