Posts on ethereal

Newsletter, Summer 2023

punnie@ethereal.io (Pedro Coelho) — Tue, 05 Sep 2023 00:24:19 +0100

Inspired by a friend I’ve decided to put in some effort in order to provide with periodic(-ish) general updates. I predict these will be mostly around personal projects I’m undertaking, and will have no predefined format whatsoever.

This first is about what I’ve been up to in the last 3-4 months, maybe even a bit further back. Quite a bit of life has been happening over the last two years and that has driven predictability away from the amount of spare time I had. Now that things are slowly falling into predictability again it might just happen that I achieve some sort of cadence, if I’m lucky.

It was a rather productive off-season. I feel good about it.

Released Huey into the wild

I have released Huey, the content reader and personal archive application I’m developing. I obviously don’t expect this release will make big waves that rock the world, but it was an important milestone: things only improve in quality with good feedback, and the first step to get that feedback is to make things public.

This was also my first foray into the personal web archiving world. I’m still exploring how to do it. At a first glance, it seems less sophisticated than I anticipated, but I’m making interesting progress. I’ll probably do a couple of write-ups on the matter soon.

Programmed a conversational bot

Based on Huey and ChatGPT, I’ve programmed a small conversational bot that answers questions about the news (e.g. “what’s currently happening with the Spain women’s national football team?”). It will query the news using Huey, and then summarise the articles it finds to articulate a good explanation.

I’ve only added Telegram as a possible way to reach the bot for now, and it is available here.

I know Telegram is problematic in many ways, but between all more widely-known messaging services, it is the only one that has an API you may use free of charge. I’m pondering about writing an adapter for Matrix or Discord.

The code will be released soon. The implementation is a bit naïve, and there are a lot of improvements I’d like to do to create a better experience: the way the bot constructs the queries is too generic, and would benefit from some specifics when we’re talking about news (e.g. responding to “what’s new?”, which is pretty basic in hindsight; or having special parameters for sports, to ask questions like “how’s my team doing this season?”).

I’ve yet to do a small write-up on what the learnings from this first iteration were, and I’ll release the code then.

Nixified the infrastructure

NixOS has been a terrific surprise when it comes to management of assets of a personal infrastructure. NixOS requires to use their configuration format to manage all parts of a server. No exceptions. That format is declarative, and lives an abstraction above the actual configuration files it generates. I’ve found that this approach not only makes servers easily replicable, but also more resilient to upgrades as the added level of abstraction allows the inner workings of Nix to reconfigure itself to achieve the same effect if there are changes in the underlying software.

So far almost all my hardware is infected:

Laptop running MacOS (using the nix package manager only)
Desktop running NixOS
Home automation controller (Raspberry Pi running NixOS)
Cloud VMs at Hetzner

The effect on maintenance has been great: little to no effort and the result is a bunch of servers that automatically update, keeping configurations and with little fuss.

Only thing missing is my NAS server that got queued down to last because of concerns with the effect of NixOS in its ZFS array. Now that I feel more confident I’ll probably do that one next.

Next up

We’ll be moving to a new place as soon as final touches on the house wrap up. I’ll for sure be working on the network setup to make sure everything gels. There’s also a bunch of opportunities there regarding home automation: all curtains and blinds are electrical, and the house runs a central AC system. Excited to start looking at that in due time.

I plan to keep working on Huey at a slow-steady pace to improve readability, and start implementing archival features too. The latter seems like the kind of spelunking that is very enjoyable to do as a hobby.

See you in a bit!

Introducing Huey!

punnie@ethereal.io (Pedro Coelho) — Fri, 01 Sep 2023 10:56:19 +0100

Today I’d like to share with you Huey, already up and running.

Huey is in an ongoing experiment of mine, and it is designed to be a news reader and archiver.

This software is still in its early stages. You get to see it as I’m building it. I’m not aiming at this point to make it a product, but I’d like to get it to a usable and interesting state.

The name takes inspiration from the name of the band “Huey Lewis and The News”.

Sources are available at GitHub, along with a project tracker with next steps. Feedback is more than welcomed.

Another feed reader?

Quite so. Huey was born on the ashes of a previous experiment of mine called LiteRSS, and hopefully captures a bit better on what I learned building it.

Around 5 years ago I found myself commuting in a low cell signal area. Reading JavaScript and asset heavy blogs became impossible, so I thought I’d build a tool that would extract the relevant bits server-side, extracting all the gunk like ads and assets for meaningless features of style and delivering only the essential.

The interface was made to be incredibly simple, mimicking CNN Lite’s. Unlike all other feed readers, there were no categories, distinction between read and unread (safe for blue and purple links) and, what I came to enjoy the most: no unread counts anywhere. Just a list of chronologically sorted news items that I could chose to visit or scroll past.

That made my experience of reading much more pleasurable and less prone to anxiety (still 300 items to read!!). This was a nice surprise, and set me off to start researching a bit on whether anyone else had got to similar conclusions.

I’ve found out that Dave Winer, one of the designers of RSS, has being doing some work around this topic for quite a while now. He calls this approach “Rivers”, and he maintains quite a few. While our ideas around it are not fully identical, they differ in very little. The current implementation of Huey doesn’t yet match what I’m looking for, but we’ll get there.

I also wondered for a while on how RSS readers would have evolved if not drove to almost irrelevance by social networks. Looking back, all readers seemed to mimic the Usenet and BBS experience. Every one of them looked like a slightly odd email client, with folders and unread counts. The thing with electronic correspondence is, though, that there’s a natural incentive to read all of it, or at least to sieve through it. If a reader looks and behaves like an email client it will end up sending the user the same signals, and start flagging news articles as things that require immediate attention. All of them. As the web grew in size so did the number of items to read, and as such the pressure to read everything rose as well, which made this UX paradigm unbearable for me. A commitment free feed made it easier to digest this deluge of items.

Of course that there are still tricks to employ when viewing news articles as a unread-less timeline: some sources output more news articles way more than others, so there’s a need need a way to quickly go through these items without it disrupting flow. Also, there may be sources that are valued a lot and output very little, so those will have to be brought front and centre.

The sorting algorithm is chronological for now, but I’m not against the idea of start using “AI” elements to help sort out the timeline, like clustering and relevance sorting for possibly a more interesting River (still to be determined). I personally don’t think the “algorithm” is the enemy, just the lack of control over it (corporate recommendation algorithms are not actually listening to your signals, but rather serving as profit maximisers for the companies that build them), and the lack of choice there is over using it or not, which is something I’ll look for when building Huey.

Your own personal internet (archive)

I’ve become increasingly and particularly interested in keeping personal records of news articles, blog posts and other internet content I’ve come across. These last years have given us a glimpse on how the internet can be ephemeral, as company owners change enforcing new policies on content (and deleting old content that doesn’t match new guidelines), or producers simply change URL schemes. Old content disappears, either from existence altogether or by simply becoming unresolvable. Other times people just move on with their lives and abandon their blogs. Content disappears, and as this decade moves forward, I fear this effect will only amplify.

While Internet archives already exist, what I’d be interested in would be helping them preserve digital assets while at the same time keeping a copy for my personal records. I plan to have Huey do this as it gathers new URLs from RSS and Atom Feeds, and other sources that will be implemented meanwhile.

I’d like to be able to process and reprocess the data for my own personal usage. The most obvious case is for building and improving a personal content search engine, for that blog post that was read a long time ago but had incredible insights, or to check what were the news on the birthday of someone important, or to do generic research.

A small aside here: I’d love that digital newspapers had some sort of searchable archive of all their news of all time. Not necessarily to scrape and add to my personal archive. I think it would be a researching journalist’s dream come true, especially now with new AI-based tools that detect references to people, places and events more accurately and may help trace timelines and surface interesting stories. But again, I digress.

LiteRSS did a little of this, as it would try to scrape the content of articles it would come across. But this scraping was destructive, keeping only unformatted text and a couple of images. I’ve since changed my approach in Huey: entries in the timeline will lead you to the source of information directly instead of showing you a mangled version of it. After all, form and content go hand in hand. My plans with Huey at present include just adding a reference to the personal archive item for each entry, and use a small subset of text or other elements to make it searchable, thus creating a semantically searchable personal archive of the internet.

What’s next

I’ve got several ideas on what I’d like to do:

Improve the timeline to make reading more enjoyable, as right now the timeline is just a list of links with a preview image
Fetch information from more sources of information, such as ActivityPub feeds or timelines, or specialised APIs such as HN or lobste.rs for added meta-data
- Make use of that information to start building some real-time intel on the articles, by understanding the ones that are posted more frequently or talked about
Archiving and search capabilities

Development shall continue at a hobby-project-like pace. Since this is a side thing, without any commitments to uptime or stability.

More updates coming soon.

A Reproducible Development Environment on MacOS

punnie@ethereal.io (Pedro Coelho) — Tue, 27 Dec 2022 16:10:12 +0000

For long I have fought with the inconsistencies of brew that, while certainly an impressive piece of software, has made several choices in its development that had the effect of distancing it ever far from traditional package managers.

A particular aspect that hurts me the most is its focus on the bleeding edge. Judging from how the community uses different managers to pin versions for the software they have installed, such as asdf or rbenv or pyenv, I don’t think I’m an edge case. But even those workarounds often aren’t enough, and it’s not uncommon to come back to a system with broken software due to linkage problems or a sudden major upgrade of a language runtime. This goes beyond programming languages and libraries, affecting my editor and my tools as well.

I’m not conservative regarding software, but I’m also not fond of rebuilding my system every handful of weeks. My Gentoo days are long past. I have been looking for a way to keeping the system a bit more stable for a while, particularly on the side of development packages.

Up so far I have tried MacPorts as an alternative, but it brings other issues. Mainly: its decreasing usage makes everything in the Mac ecosystem lopsided to brew’s court, which meant costly workarounds to install some packages, and absence of many.

I’ve been wanting to try a different approach: Linux distributions and their package managers usually deal well with these concepts. I could install Linux on my laptop, but there are two strong arguments against it: first, Linux is far from being supported in my personal daily driver—Wi-Fi is still unsupported on my late 2016 MacBook Pro (yes: still this one)—and secondly, I like the desktop ecosystem of MacOS a little better, and would like to keep using it while I can.

The other alternative, and the one I had been pursuing for a while, would be having my development environment in a virtual machine. This would bring the best parts of both worlds at the cost of performance, which is arguably a problem: most of the work I do doesn’t require a beefy computer. So I had a crack at it, or multiple.

A working setup

This has been an journey that has been going on for quite a while, coming from my days working at Cleverly. I have tried many variations alternating between Linux distributions and strategy: from actually using a GUI to keep myself to a terminal or even SSH only. Having some sort of interface made sense early on, so SSH only soon became out of the question. On the other hand, using a GUI was lagged too much to be usable.

I kept going back to the drawing board on this one, and never found anything that fit.

I then found Mitchell Hashimoto’s repository where he tackles this problem with a similar approach. He claimed his setup worked very well, and had minimal lag, so I wondered what were the main differences between his approach and mine were.

Then I realised he used VMWare Fusion instead of VirtualBox. So I tried it out.

It was night and day. Spawned a Debian box with i3 as the WM, and emacs GUI just flew. Gave it half the resources of the laptop (4 cores and 8GiB of RAM) and I managed to do a couple of comfortable programming sessions with both Ruby and Clojure.

I feel obligated to say that the clipboard between both parts also works well, so I can browse in MacOS and do my coding in the virtual environment.

Lastly, I found out that VMWare Fusion Player is free for non-commercial use, which is pretty much my case at the moment.

Reproducibility

So far my approach for reproducible systems would be Debian/Ubuntu with Ansible. That was my first approach to this development environment, and it worked well. Took me about two days to get the system to how I wanted it.

I had my eye on NixOS for quite a while, and found this to be the perfect excuse to try it out, seen that I even had Mitchell’s example and all. After a few hours editing my configuration.nix I had almost everything in place. I was blown away by how fast that was.

And the best part is: due to everything being declarative and written down in a configuration file, now I can go from no VM at all to readiness in minutes, with all missing being user data. Also, because of Nix’s shell environments, all the projects I use them with will have a reproducible environment at the distance of a nix-shell in the terminal. It’s glorious.

I’m stil learning about Nix, and I plan to release my configuration as soon as I learn more about flakes, overlays and packaging.

Other advantages

I’ve found that this approach also gives me some extra nice things:

The development environment functions as a de facto sandbox, so I can do more curl | sudo bash madness with less risk it will affect the environment where I may or not check my bank account.
Memory leaks and other common bleedings from development grade software don’t affect the rest of my system.
Since the environment is running in a VM, I can suspend all my development environment including services, databases, and programs, freeing resources so I can use the computer for other stuff, to later resume work from exactly where I left off.
I can tear it down and back up again in minutes for a fresh install whenever I need to.
I can move it to another computer without losing anything, even user data if I chose to copy a VM snapshot.
Emacs installs without issues, reliably.

GUI Tweaks and Performance

I’ve opted to follow Mitchell’s terminal based approach and use Emacs in its terminal-mode, in a terminal window instead of in a dedicated X window. Although graphical performance was far from being an issue, the complete absence of lag when using the terminal just won me over the nice things coming from the GUI version. I still use i3 to manage all terminal and non-terminal windows, even the occasional browser session.

For vim/nvim users there should be no compromise. I’m even tempted to make an attempt at a vim config later on, if I have the time.

I have no idea about how VSCode behaves. But if anything else fails, VSCode SSH mode works remarkably well, so in that case you may choose to have your development tools and languages inside the VM, but keep your editor out. I’d probably do that if this was my editor of choice.

Final thoughts

I’m happy with this setup. It’s snappy, comfortable to use and stable. It has been a few weeks since I set it up and it’s still behaving predictably.

This left me very curious with the prospect of using NixOS in other places, and Nix as the package manager for MacOS, so it’ll be something I’d like to try out. The tight coupling between system configuration and the system are very appealing, and I’d like to see how this performs in practice.

Escaping from Google

punnie@ethereal.io (Pedro Coelho) — Mon, 19 Dec 2022 15:00:26 +0000

Google has been pulling the plug on G Suite legacy free accounts, after 10 years. To me this meant having to pay a user subscription for each domain that I have there, or resort to weird domain aliasing tricks to get the job done. As such, I’ve decided on finding an alteranative suitable email hosting.

Landed on fastmail.com for now. Heard nice things about them, and tried the service out. The interface is clean, snappy and easy to use, possibly even more featureful than Google’s. Migrating both domains in took less than one hour, and importing existing mail was a breeze. Retained compatibility for all weird features I was using, like routing tricks, and an absurd number of aliases, and since I don’t rely in any of Google’s mobile applications, there was no change in how I use any of my email clients.

I do have some concerns around privacy, though, seen that Fastmail is an Australian company, and that their government has been passing anti-encryption laws. However, the truth is that email is inherently unsafe, and so I will use other methods of communication if I need more privacy. I did check protonmail.com as well, but the usage I make of aliases wasn’t really compatible with any of their plans.

I considered getting back on the wagon of hosting my own email server, but I don’t have that kind of free time anymore. Furthermore, the siege on indie servers seems to have only tightnened in the last few years, so I’d most certainly have to rely on a paid third party SMTP relay to send my email, which would bring us to square one in terms of how costly this would be.

I’m also moving every website account I have using my @gmail.com address to one of my domains. This has been a lenghty, many-month effort I’ve been doing on my spare time. It has also been great to purge some website accounts I haven’t used in a while.

I’m happy with this move. I’d have done it sooner knowing what I do now.

Instrumenting a Rails application

punnie@ethereal.io (Pedro Coelho) — Mon, 22 Jul 2019 00:00:00 +0100

In the 18th of July 2019 I gave a talk at Le Wagon Lisbon about how to instrument a Rails application, using Prometheus.

Since my last talk I grew fond of simple live coding demonstrations to convey practical knowledge as well as awareness that things are often easier than they seem. This time I went about gathering some system and custom metrics from an already existing application into an existing Prometheus server.

The application with the end setup is available here as a docker compose bundle that can easily be run to inspect results and do further experimentation. The instructions in the README should be clear enough to get you a working example, but if they are not then feel free to open an issue in the repository.

Happy hacking!

Email me maybe

punnie@ethereal.io (Pedro Coelho) — Fri, 28 Jun 2019 00:00:00 +0100

Back in the 9th of May I did a little talk/workshop at Porto Codes where I talked about the advantages and perils of owning your own email server, the hassles with spam and protecting your domain from being flagged as an origin, and so forth.

Halfway through I did a live demo where I installed a postfix/dovecot server from a clean Ubuntu installation. The purpose of this exercise was to demonstrate how easily and fast it could be done. Sure enough, I was done in less than 15 minutes, as it is indeed an easy task.

Since then I had some people asking me about the script that I followed for that demo, so it is now available here.

Happy emailing!

The 2016 15" MacBook Pro

punnie@ethereal.io (Pedro Coelho) — Sat, 07 Jan 2017 00:00:00 +0000

I recently purchased the 15" version of late 2016’s MacBook Pro, their last model as of this writing. Touch bar and everything. This came as a necessity as my former laptop—another MacBook Pro—is turning 4 years old in a few months, and in dire need of some repairs and well deserved rest and repurpose.

I’m not the usual review guy, but this model has been receiving a whole lot of hate all around. My expectations were very low when I got it, as I’ve seen almost every updated part of the machine being ripped apart in internet reviews. They say the secret to happiness is low expectations, and they’re probably right: I’m enjoying using this model. Let me tell you why from a software engineer perspective.

Most of my work involves a text editor and a bunch of terminals. And of course the omnipresent browser. In a normal day of work I’ll spend some hours browsing and coding while I’m hosting several background processes running automated test suites, databases and support systems. I also program a bit after work. I haven’t yet played anything in this computer, still waiting for that decent sale on Civilization VI.

Taking that those are my necessities into account, here are my observations.

The screen

The screen is clearly better than its predecessor. Brighter, and with slightly more contrast. It has a higher resolution than the former model as well. It is very comfortable and easy on the eyes at high and low brightness settings.

The keyboard

The keyboard takes some time to get used to, but I ended up liking it. It’s almost flat, so it has quite a lot less travel time than the old ones. On the other hand it’s harder and crisper. I feel it requires more pressure to actuate, so it balances out. Keys are larger, but the spacing between centres is the same.

The clicking noise is different from the old chiclet style, and louder. Sounds rather weird at first, but again, one gets used to it.

The retro illumination looks uneven in some spots. This makes the keyboard look somewhat cheap, in contrast with the 300$ touch bar right above it.

The touchpad

Not to be confused with touch bar. The touchpad is huge, humongous, gargantuan. I get the need to have it larger, but this was way too much in my opinion.

It works wonderfully well though. The force feedback mechanism, while not indistinguishable from the real thing, is very pleasant. The palm rejection calculations also function very well.

The touchbar

Useless at this point, and somewhat cumbersome at first. Took me 1 hour to become fed up with the new “Escape button” and remap it to the Caps Lock key.

I’m not entirely skeptical about it, and I believe a lot of good stuff can be done with it, even in my field. After speaking with some friends we came up with two possible uses: a permanent display of metrics—such as the requests per second your app is receiving, its percentiles, how many users connected to your app, or how much sales you had today—and to be used as your editor’s usual bottom bar which displays the git branch, automated test results, etc…

But for now it just sits there. Drawing power.

The chassis

This worries me a bit. A lot of changes happened to a chassis that I knew was good enough to last me 4 years. The keyboard changed, the hinge changed, and the touchpad changed, along with some overall body modifications here and there.

Everything looks sturdy enough, but lets see how it fares against the test of time.

Lighter and thinner, so good on my back.

Battery life

Better than my old computer. But then again, my old computer is 4 years old. I haven’t measured anything yet, but it looks like it’s worse than what my old computer could do 4 years ago.

Ports

I had to buy some adapters for standard USB and Thunderbolt 2/Mini Display Port. So yeah, I jumped on the adapter train.

I’m kind of divided in this. USB-C is great and also the future. However, a couple of standard ports would be nice as well. All that talk about adapters and lack of ports makes an awful lot of sense for most people—although truth being said, it doesn’t affect me that much. So lucky me.

The lack of a MagSafe power supply is a bummer. It is very nice, however, that you can plug in the power from both sides of the laptop. It’s also nice that you can plug in more than one power adapter, even though it’s plainly useless.

The price

Terribly expensive. I got it under very special conditions which made the deal attractive to me, but I probably wouldn’t have gotten it otherwise.

The fact is that including a mandatory 300$ touch bar into a laptop with such an universal use as the MBP hurts a lot of people. It is a nice shenanigan but at this point it is nothing beyond that: a shenanigan.

What about the specs?

Those are fine. Performs better than my last computer when doing CPU intensive work. The SSD is also faster. The internet is ripe with benchmarks regarding this.

I’ve gone with Apple because of good build quality, a decent screen and a comfortable experience. These things are important to me as I can easily spend at least 8 hours a day using a computer. I really don’t care for that extra fast CPU if it means having my palm rests burning all the time.

These are the things I expect from this computer. That and some durability.

You don’t go to Apple if you’re pursuing specs. Stop kidding yourself.

Final thoughts

So as I lay in my sofa typing this in a brand new MacBook Pro I realise the criticism against this release of Apple’s professional laptops was greatly exaggerated. There is still some merit to them, but the computer is good, albeit very expensive.