• I usually try to not compare PaaS/IaaS providers directly, because they serve different audiences and all have their strengths and weaknesses. That said, Azure’s love of ceremonious ordeals and enigmatic rituals, coupled with poor UX and general lack of sensible design, are sometimes just infuriating. The frustration becomes especially intense when I remind myself just how easily some of these goals are accomplished in AWS (and, I assume, Google cloud as well to some degree - I don’t yet have sufficient expertise in it to draw informed conclusions).

    Allow me to illustrate.

    Today we have a (seemingly) simple task at hand. We’d like to offload TLS termination for our app to the cloud provider’s load balancer, and go about our lives. Will it work? Let’s find out!

    Easy enough, right?

    This is how the two platforms stack up in helping you protect your HTTP services with TLS in this year 2018:

    AWS Azure
    You may request and issue up to 100 of SSL certs for free to use with AWS services, via the web console. You may then reference them with Terraform / boto / aws cli. Certs are signed by AWS’s CA. There’s no free cert option. Azure have partnered with GoDaddy to bring encryption to the masses (at a notable discount of 0%), because clearly running a CA for their customers is just too fucking much to ask of the Redmond giant.
    Such a cert is usable anywhere in AWS, including the application load balancers. The cert that you can buy in Azure is only deployable on Azure app services, i.e. the Web Apps - the Azure PaaS product. The Compute suite of products, such as Load balancers of any kind, are not included. You do have the ability to export the cert for use with other services (after all, you paid for it). The process is complicated and requires PowerShell. If you’re on a Mac or Linux, sucks to be you (no it doesn’t). Also, as part of the process you have to create a Key Vault, for no good reason whatsoever.
    The cert, once provisioned, is immediately available in EC2. Once again, console / Terraform /boto / aws cli / whatever else tickles your fancy. So you either bought a cert from a registrar of your choice, or signed one on your own CA for private use. Either way, the cert, once provisioned, is just that - provisioned. Now you need to upload it to Azure. Yes, even if you bought it from Azure.
    To use it with a load balancer, you get its id and pass it to your scripts. If automation ain’t your thing (it should be), you may pick the cert from a list when setting up a load balancer or another service. You’d expect Azure to have something like Certificate Manager, yes? Well, you’re in luck - Key Vault to the rescue! Remember, you made one earlier? Yes, you do have to create an actual “key vault” resource via either Portal or CLI to store your certs. It’s not just a platform service offered by Azure. It’s an object you have to create and maintain, because you love micromanaging this shit. (Worry not, one vault can hold many keys - thanks, Obama Nadella!).
    You’re done. In fact, you were done after the 2nd step. This was the entirety of setup required to terminate SSL on an AWS load balancer. Feel free to move on to more interesting work. We still want to use the cert with a load balancer^H^H^H^H^H App Gateway[1], right? Guess what. That’s like the only fucking Azure service that CAN NOT get your cert from the goddamned Key Vault.
    you’re done, remember? Yes, you guessed correctly. You need to upload a cert to each App Gateway separately. Yes, we love manual work SO MUCH that will happily do this in Azure and thank them after, even though we pay them good money to have a platform that doesn’t suck balls.

    Any questions?

    Do you think I’m doing it wrong on Azure? Please share your perspective.

    P.S. I haven’t touched on end-to-end SSL here, just for the sake of wrapping up this rant or we’ll be here all night. Suffice to say, it’s super easy in AWS and a huge fucking pain in the ass in Azure. Are you surprised?

    [1] Few conversations can get as confusing as those where you keep saying "load balancer", but really mean "app gateway".

  • Sun, Jan 28, 2018

    I noticed something today: In Chrome, accessing http://127.0.0.1/ will disable some extensions (like AdBlock and Ghostery), while http://localhost/ will not.

    Who knew?!

    Write me a note if this is the same in other browsers. Or alternatively, if I’m missing something obvious and this is totally wrong.

  • Thu, Jan 25, 2018

    So you have an SSH session that’s locked up. Ctrl-C doesn’t work. What to do, aside from closing your terminal?

    Indeed, there is a way. On a newline, type ~. (that is a tilde followed by a period). Instagib.

    The more you know!

  • It’s no secret that web development tooling have been steadily trending towards use of open-source software over the last decade. At conferences and meetups, we’ve seen Lenovos and Toshibas being edged out by Macbooks of every kind, and even the occasional non-Apple laptop is not unlikely to be running some flavour of Linux (as happens to be the case with my Thinkpad).

    I recently started working with a large team of web developers who are - for the most part - Windows-based, and this led me to a surprising discovery: much of the open-source web development tool ecosystem is just paaainful to use on Windows. Open-source projects, which traditionally and practically favour Linux, treat the Microsoft system as an afterthought, if at all. So when a .NET developer attempts to jump into working with Node / Golang / Python, they are immediately disadvantaged by tooling barriers and lack of community content and support.

    Every step of the way, Windows creates friction for web developers who choose to build web products using modern open-source technologies.

    Thanks to its Unix underpinnings, macOS has become an almost de-facto standard in this space. But many developers do prefer to use Windows as their primary desktop OS - for a multitude of very valid reasons, or just due to personal tastes, or simply because gaming.

    The practice of DevOps teaches us to build bridges. And so we shall certainly do whatever is necessary to build as good of a web dev experience possible in the Microsoft ecosystem. But I feel this experience will always lag behind that of working on Linux or macOS, just due to the sheer size of the ecosystem shared by, and targeting, those other two dominant platforms.

    An example:
    • Everyone Most of us either already are, or are soon going to be, using this little tool known as Docker.
    • Also, many devs use VirtualBox, because this free virtualization platform is what a lot of the tooling and community knowledge are centered around.
    • But, on Windows, Docker needs Hyper-V.
    • But! Hyper-V is incompatible with Virtualbox.
    • But!! Virtualbox has the broadest ecosystem of getting shit done in a fun automated way (Vagrant, basically).
    • But!!! Lisa needs braces Docker needs Hyper-V…

    /shrug.

    If you wish, here is another shining example of the hoops one needs to jump through to get some Node.js action on Windows. I won’t post more because the Interwebs are ripe with them (or just talk to some Node developers who happen to use Windows).

    What can we do today to improve this?

    • Automate everything. In place of PowerShell scripts, use task runners. While the venerable make can be installed on Windows, it’s a bit of a ceremony, so try some alternative, cross-platform task runners. We are successfully using go-task.
    • Write tooling around Docker, and try to run as much of your toolchain in containers as possible. Standardise on an image that will work identically across platforms. Encourage yourself and your team to think “container-first”.
    • (reluctantly) use docker-toolbox. Yes, this is a legacy solution, but at least it will get around the Hyper-V requirement, and enable use of Vagrant and Virtualbox. docker-toolbox’s days, however, are probably numbered.

    Other ways exist to ease the path of a Windows web dev, which I will explore in future posts. In the meantime, reach out to @boomstik on twitter with your thoughts, comments and ideas.

  • Wed, Jan 3, 2018

    “Ohai Azure Portal, how I’ve missed you!” – said no one ever.

  • jq is a swiss army knife for working with JSON. It is especially handy for piping output of CLI tools, such as curling JSON APIs, or aws and az CLIs.

    I wanted to get a nice list of public IP addresses of my EC2 instances, together with instance names. I could have used boto for this, but the combo of AWS CLI and jq turned to be a simple and effective one-liner (split for better wrapping).

    aws ec2 describe-instances | jq '.Reservations[].Instances[] |
      {(.Tags[] | select (.Key == "Name") | .Value): .PublicIpAddress}' |
      jq -s add
    

    produces:

    {
      "foo": "54.131.121.177",
      "bar": "52.75.8.58",
      "baz": "34.228.156.28"
    }
    
  • Fri, Oct 13, 2017

    Azure functions can look at blob storage and react to things.

    But actually not really all that well.

    Excerpt from the Documentation:

    When you’re using a blob trigger on a Consumption plan, there can be up to a 10-minute delay in processing new blobs after a function app has gone idle. After the function app is running, blobs are processed immediately. To avoid this initial delay, consider one of the following options:

    Use an App Service plan with Always On enabled.

    Use another mechanism to trigger the blob processing, such as a queue message that contains the blob name. For an example, see Queue trigger with blob input binding.

    Let’s deconstruct this a bit.

    The important parts are the "Consumption Plan" vs "App Service", and how those relate to the Always On mode.

    See, Azure Functions have two methods of operation (“plans”). The “Consumption” plan executes the function only when triggered. So if nothing is calling it, the function will go to sleep. A Function runs ephemerally and you need not think of its underlying resources whatsoever, aside from paying per invocation.

    The App Service plan, on the other hand, launches a VM that will host your functions, and that VM remains running. You don’t need to directly manage it (nor can you), but you are being charged for all the minutes it’s humming away. Also, unlike the Consumption plan, you need to manage autoscaling yourself.

    Only on the App Service plan you are given the option to enable “Always On”, which will prevent your function apps from going to sleep.

    So in contrast to the probably familiar pattern of AWS Lambda being triggered by a change in S3 bucket, the Azure Blob storage doesn’t immediately trigger your function on change in blob storage, unless the function is already running. Otherwise, you are waiting for the scheduled wake-up window (feel free to correct me on Twitter if I am misunderstanding something). I personally find this behaviour to be super confusing, and inferior to what the rest of the cloud has come to to expect of the “serverless” patterns.

  • Sun, Oct 1, 2017

    Good morning. Today we will take the terms “domains”, “fault”, and “update”, and make it sound more sophisticateder because competitive advantage.
    - Azure marketing people, probably

    I mean, it’s good they have thought of this. It’s even on the exam. But really, as the user of Azure, I don’t need to care about how they power their racks and in what order they are restarted. I care about stability of my VMs, but it’s ok to leave the mechanics of fault-tolerance to be a black box. For the most part, it would suffice for me to know that if I launch a group of 3 machines, I’ll have almost 3 machines running most of the time. I don’t have any control over this anyway, so those “domains” are trivia and implementation details.

    That aside, Microsoft’s general aversion to visual presentation of data rears its ugly head here once again. They could have designed the UX around this as a nice grid, with current status of each slot in the fault/update domain, etc. Could’ve even put this next to each VM. But no. Everything must look like a spreadsheet.

    The important takeaway of the entire feature: You should, for best availability vs cost effectiveness, try to horizontally scale your VMs in sets of 5: N % 5 == 0. That’s how many update domains exist. N < 5 - and you’re not utilizing the full fault-tolerance potential. 5 < N < 10 - and you are overprovisioning some of those update domains.

  • Tue, Aug 8, 2017

    Some of these tech vendors need to grow a pair.

    I found an email of this nature in my inbox today:

    Dearest Eugene,

    Our sincerest apologies… (sob)… for previously mistakenly sending you this email about some industry event that we’re hosting. We realize you have not registered, and yet the message was dispatched to you. HOW COULD WE. For the inconvenience that we have caused you, and all the confusion - we are so, so sorry!.. We truly understand the rollercoaster of feels that you’re experiencing at this very moment! We want - nay, need! - to do better. You DESERVE better. We admit, this was a human error. We made a mistakeSUCH a mistake… And, indeed, we do so sincerely hope you still choose to remain on our mailing list. Please, please let’s stay friends! However, if you choose not to (ohnoes!!)…. We will be sad. So very sad, yet understanding of you clicking this unsubscribe link. ( pleeeezdontgoooooooooooo )

    (Of course I’m exaggerating, but only slightly).

    seriously, what IS this shit.

    You’d think this was an apology for poisoning my prized breeding ferret, or another unthinkable evil of similar magnitude. But alas, a mere mis-addressed email message had been the trigger of such profuse expressions of remorse.

    Now I face a dilemma, don’t you see? On one hand, some of the their content is kind of interesting. But on the other, this type of submissive servility is a turn-off of unsubscribeable magnitude.

    Dear tech copywriters of the Web 2.x. We are not children, kk? Most of us don’t need a safe space to cry over your accidental emails. KTHXBAI.

    Though on second thought, if you really feel the need to apologise - nothing short of a phone call shall suffice. Ladies and gentlemen - start your Asterisks!

  • Accidentally discovered that EC2 security groups do not terminate an open connection (like SSH) when the security group rules or membership change. New connections will be prevented, but this will not terminate established ones.

    See for yourself:

    • create an EC2 instance and give it a security group
    • add an ingress rule on port 22
    • SSH into it
    • change the security group; remove the instance from that group altogether, or just change ingress rule.
    • Observe how the SSH connection remains open

    Tested this for N hours and SSH connection did not get terminated. So if someone is in your boxen, you can’t kick them out that way.

    Heed the warning and plan accordingly.


    update 2017-11:

    Apparently Azure NSGs have the same flaw. Not even surprised.

  • Mon, Jan 2, 2017

    So you get an idea, and it’s an amazing one. You’re inspired to start working on it right away…. But what if someone has already made this? Does this mean your idea is dead?

    And so you rush to the interwebs, and prepare to search…. But wait!

    Until you’ve made the search to determine originality of your idea, it is simultaneously both dead and alive.

    Meow.

  • Fri, Dec 30, 2016

    Random ha-ha thoughts:

    • Technical debt doesn’t just accumulate. It incurs interest, in form of features and dependencies that will break and will then need to be refactored when this debt is repaid.

    • Is it even possible to ever repay it completely?

    • I guess if we continue drawing parallels, you can also default on technical debt. That’s when you scrap the whole thing and rewrite it from scratch.

    • And finally, technical bankruptcy is when you lose it completely and start rewriting your API in BASIC.

    10 PRINT "LOL"
    20 GOTO 10
    

    (seriously, get help).

  • Fri, Dec 16, 2016

    It took me 3 years to grok callbacks in JS. Not something I’m proud to admit.

  • Wed, Dec 14, 2016

    Propensity for dogmatic brand loyalty in a person is indicative of their critical thinking skills.

  • Sat, Dec 10, 2016

    Recovering my better half’s system drive. Her OWC 480GB SSD was allowed to reach 100% capacity (only 1.5GB remain)… And all hell broke loose. It’s barely readable (takes 10 minutes just to mount on my laptop), and I can’t even delete anything (any attempt to modify the filesystem just returns an invalid argument).

    I’m currently rsync-ing all the things to a network volume, and will attempt to deal with this after the data is safe.

    That machine was long overdue for a refresh anyway.

    Lessons *:

    • When setting up an SSD, make sure to enable TRIM. Windows | macOS (≥10.10.4).
      • OWC seems to discourage the use of TRIM, citing “garbage collection”. I believe those are different things, but more research is needed.
    • Leave unpartitioned space on the SSD. The accepted guideline seems to be ≈10% of total capacity

    * Disclaimer: I take no responsibility whatsoever for any effects, including but not limited to loss of data, caused directly or indirectly by this blog post.

  • Thu, Dec 8, 2016

    Today I was testing how a SaaS that we use for search handles failover. I clobbered our DNS to make our app use their secondary endpoint. The module in our app handled this very well, and switched over seamlessly.

    But then a thoughts crossed my mind: did this affect the Ops folks on the other end? Did someone see a blip on the charts, and ask the person next to them:

    hey dude, are you seeing weird traffic on node B13? dafuq is that?…

    Or… was it simply missed? unnoticed, lost in the shuffle - because let’s face it: ain’t nobody got time for this.

    I guess the former would be nice, because that would mean those packets of mine were just a little more special than the rest of the internet noise.

    Yet I do hope for the latter…

    Because Karma.

Hosting AWS Docker Microservices Tooling Automation