Archive

How To REALLY Use Cron To Run Scheduled Jobs: Some techniques for using cron in the modern era.

Testing What Scene a Body is in Godot: Using the class_name and is to detect if a body entering an Area2D is of a particular scene.

Specify Field To Sort By Using The Sort Command: The sort command allows one to specify which field to sort by. These fields need to be separated by a character, such as a colon or comma, which is specified using the -t option. To select the actual field to sort, specify the field position using the -k option, which is one based: $ cat eg.csv 111, …

Using Input Filename In JQ Pipeline: One other JQ thing: there’s an operator which returns the filename of the file currently being processed by JQ, called input_filename. This can be used alongside the concat operator to reproduce something similar to grep -H: $ jq -r 'input_filename + ": " + .payload.data' *.json …

Using Recursive Decent In JQ: Yesterday, I discovered that JQ has a recursive descent operator — .. — which allows one to go through each field of a JSON structure. When used, it will emit every value of a JSON structure, doing so in pre-order (i.e. any nested objects or arrays will be written to the output before they’re …

Dump Protobuf Messages Without A Schema: For anyone working with gRPC or Protobuf messages that need to decode the binary message format, but don’t have or don’t want to find the actual schema, this invocation of protoc works great: protoc --decode_raw It takes a binary Protobuf message from STDIN and produces a text-based output showing …

Anyone scanning or querying DynamoDB with a projection, you may want to know that if you plan to get the LastEvaluatedKey for paging, you will want to make sure the primary key attribute is included in the projection. Leaving it out may produce an error much like the following: dynamo: failed to …

Local Values In Fiber: For much of my use of Fiber, whenever I needed to tack on a value that’s scoped to the request, I used the standard approach in Go whenever a Context is available: define a new key and call context.WithValue() type userKeyType struct{} var userKey = userKeyType{} func setUser(c *fiber.Ctx) error { …

Stimulus Outlet Names Must Match The Target Controller: Learnt a very import thing about Stimulus outlets this evening: the outlet name must match the controller name of the outlet target. From the docs: The outlet identifier in the host controller must be the same as the target controller’s identifier. That is, if you have a controller with the name …

A Way To Resolve Redelivered Messages Stuck In A NATS Jetstream: Fair warning, this is probably an approach that’ll only work for me, since the services I’m working on upserts1 NATS Jetstream consumers on startup. We’re using NATS Jetstream to dispatch jobs to a pool of workers. Jobs will come in via the job-inbox stream, and the workers publish results to a …

Link: Probably Avoid Relying On Error Codes To Optimistically Insert In Postgres: 🔗 Understanding the difference between INSERT and INSERT..ON CONFLICT Okay, I admit I was considering this. My Go-like treatment of errors as “just another value” had me wondering if I could avoid writing an INSERT with an ON CONFLICT DO UPDATE clause, and instead do this in code: // This is bad err …

Don’t Use ‘T’ RDS Instance Types For Production: Note to self: when choosing an instance type for a production PostgreSQL database in Amazon RDS or Amazon Aurora, don’t choose any of the ‘T’ classes. The reason is that the ‘T’ classes work on CPU credits, and if you have a sustained performance for a long period of time, those credits will be …

Feedback Of Conditional Updates In PostgreSQL: Imagine, if you will, a service responsible for dispatching jobs to a pool of workers. The worker reports when a job has started, and will send updated while the job is in progress, and when it’s finished. The dispatcher tracks this in a PostgreSQL database and forwards the completion message …

Slow NATS Go Subscribers: Heads-up for anyone using Golang NATS client: setting up a subscription using Subscribe or QueueSubscribe will not setup a worker pool. As far as I can tell, using either Subscribe or QueueSubscribe will only setup a handler backed by a single goroutine. What’s Your Evidence? I setup a small …

psql Techniques: Some techniques with using psql. Supplying Password Non-interactively To supply a password in a non-interactive way, use the PGPASSWORD environment variable: PGPASSWORD=xxx psql -U username database Source: Stack Overflow Describing Customer Types These are the enum types created using CREATE TYPE …

HTMX And POST Redirects: I tend to be stuck in my old ways when writing POST handlers. When I accept a POST request from a HTML page, I send a 303 See Other redirect to force the browser to get the resulting page with a GET. This keeps the path routing relatively clean, and saves me from having multiple handlers return the …

Working With JSON In PostgreSQL: A collection of useful operations for working with JSON fields I wish to remember. Others would be added here when I encounter them (maybe). Selecting JSON Fields Imaging a table with the following schema: CREATE TABLE data ( id INT PRIMARY KEY, json_props jsonb ); INSERT INTO data (id, json_props) …

Get Or Create In PostgreSQL: One operation I find myself occasionally doing with anything involving a database is “get or create”: return an object with a particular ID or unique field, and if it doesn’t exist, create it. Yesterday, in the project I’m working on now, I saw a need for this. Being backed by a PostgreSQL database, …

Links: Working With OpenSSL: I’m sure I’m not alone in thinking that OpenSSL is a bit of a dark art, what with all the terminology and strange CLI invocations and such. I suppose one may think that documenting this process would be better, but there’s already a lot out there that just needs to be surfaced. So, here’s a …

Don’t Forget Your Launch Template Version: Hey. Yeah, you. The one trying to create that new EC2 instance. I see you’re trying to use a launch template with that. Did you remember to specify the correct version number? EC2 will use the default version number in the launch template which may not be what you want. So make sure you’ve got the …

Force Deleting AWS Secrets: If you were to log into AWS and try to delete a Secrets Manager secret, you will be told that the secret will not be removed straight away. Instead, you’ll be asked to specify a “recovery window” in which the deleted secret can be recovered. The shortest recovery window you can …

Disabling Imported Jobs In Gitlab CI/CD Pipeline: Small one today. I’ve got a Gitlab CI/CD file that imports a much larger one defining a standard set of jobs for a build. But I needed to disable one of those jobs as it didn’t apply to the project I’m working on. So after trying a few things, the approach I found that worked was …

Packaging Services With Systemd: This is going to be of a rambling post as I try to find my way to best configure a long-running service that’s managed with Systemd. The motivation here is to install the service on a Linux machine, then configure it to run using Systemd such that: It launches on startup It restarts when the …

AWS Secrets Manager Cached Credentials Error: Spent around two hours trying to diagnose why one of our containers couldn’t read secrets from AWS Secrets Manager. I was seeing errors of the following form when I was trying to call GetSecretValue: operation error Secrets Manager: GetSecretValue, get identity: get credentials: failed to …

Disabling Parallel Test Runs In Go: Go has had parallel tests enabled by default for a while now. But if you can’t run your tests in parallel, what should you do? Most people probably know of the -p 1 option to disable parallel execution of tests: go test -p 1 . But this only works if you have access to the command itself. If …

PostgreSQL, pgx, sqlc and bytea: Subtitle: if you know what these four words mean, then this post is for you This is a quick one as I’m not really in a blogging mood. But I couldn’t for the life of me find a way to decode a PostgreSQL bytea value (which is what PostgreSQL uses for BLOB values) using pgx and sqlc so that …

Obsidian PDF Styling Improvements: I’ve been writing a bunch of technical documents in Obsidian recently: complete with code-blocks and Mermaid diagrams. And I must say, it’s been a pretty good writing experience. Certainly much nicer than writing in Confluence. But when I made PDF exports of these documents, I found a …

PostgreSQL BIGSERIAL “Type”: The same person that taught me about LATERIAL SQL queries also showed me the BIGSERIAL type, which is an automatically incrementing integer column. I don’t know why I didn’t see this before. The AUTOINCREMENT option in MySQL was one of the things I missed when I started using PostgreSQL. …

Formatting Javascript/Typescript Code With Prettier: You’ve got a bunch of Typescript files with both .ts and.tsx. You need to format them using prettier. Your code is managed with source control so you can back out of the changes whenever you need to. Here’s a quick command line invocation to do it: npx prettier -w '**/*.ts' …

Counting In DynamoDB: Does getting a count in DynamoDB return the total across the entire table, or just the total for the current page? While a brief search online didn’t give any conclusive result, it did show that it’s possible to get a count for a scan or a query without fetching the items themselves, …

Custom Import Paths In Go: One of the craziest ideas I had recently was to move all my code from Github to a self-hosted SCM system. The impetus for this was to have import paths with a custom domain name for all my Go packages, rather than have all them start with github.com/lmika/(something). Fortunately, this proved to be …

PostgreSQL LATERIAL Joins: Someone shared with me the LATERAL join type supported by PostgreSQL. He described it as a “for each” built into SQL: When a FROM item contains LATERAL cross-references, evaluation proceeds as follows: for each row of the FROM item providing the cross-referenced column(s), or set of …

Changing gRPC Schemas: What changes can you make to a gRPC schema that’s already in use? Honestly, I always forget these rules, or get paranoid when I’m touching messages that are already deployed, so it was time to find out what these rules are. I came across this Medium post by Toan Hoang which listed them, …

Go Unit Test Naming Conventions: You’re working on a Go project (yay! 🎉) and you need to write a unit test. You decide to go with a table-driven approach. What names should you use for your variables? For a long while, I was writing tests using names like these: func TestSomething(t *testing.T) { scenarios := []struct { …

List And Delete All Docker Containers: This will list all Docker containers, and delete each one regardless of whether it’s running or not. Good if you use Docker for dev containers and need to reset your state. docker ps -a --format '{{.ID}}' | xargs -I{} docker rm -v {} Another way to do this (thanks to @sonicrocketman): …

Dealing With Large Docker Pulls Over Slow Connections: If you’ve got a slow internet connection and are trying to pull a large collection of images, you may encounter retries causing the entire pull to fail, not to mention multiple concurrent downloads sucking up all your bandwidth. I found that the following Docker daemon configuration changes …

Equivalent Linux Commands In MacOS: Even though I’ve been using MacOS for a while, there are certain things that I just still remember doing in Linux. Probably it got burned into my mind when I was under pressure to debug something in work. Whatever the reason, I can never remember how to do similar in MacOS. So I’m noting them down …

Working With CSVTK: There’s a small tool called csvtk which can be used to do various things with CSV files on the terminal. It’s… fine. If I had my own way, I would’ve made different decisions. But one thing going for it is that it exists, and my fantasy tool does not, so it’ll do for now. Anyway, here’s a collection …

Working With Buffalo: Here are some random notes about working with Buffalo (may it rest in peace). Cleaning Up Assets In Buffalo Buffalo doesn’t clean up old versions of bundled JavaScript files. This means that the public/asset directory can grow to gigabytes in size, eventually reaching the point where Go will …

Go: Random Language Notes: Some random notes about the Go language. Commas in Expression Case Statements The use of comma in case statements work for expression: x := 2 switch { case x == 1, x == 2: fmt.Println("x < 3") default: fmt.Println("x >= 3") } If x is 1 or 2, the x < 3 will be printed out. …

Building a MacOS .app From A Go Project: I spent the better half of last week trying to build a MacOS .app bundle for one of my Go projects. I want to eventually put this online for others to download, so it was time to work out how to go from a simple binary to a .app bundle. I want something that is repeatable, automated, and could be …

Posts Only RSS Feed in Hugo: This blog was once a static website generated using Hugo. Like most other static website generators, Hugo provides a lot of features out of the box, one of them being the generation of RSS feeds. I wanted to add a link to an RSS feed of the blog posts to the nav-bar. After trying the /index.xml, I …

Git: Submodules: Git cheatsheet for working with submodules.

Git: Diff And History: Git cheatsheet for working with diffs and history.

Git: Committing: Git cheatsheet for working with commits

Sentinel Errors In Go: The errors.New() function in emperror.dev/errors package includes context information, like stack traces. This means that if you were to use this for sentinel error types, i.e. error types like EOL, you’ll be including unnecessary stack information. This is especially useless when these are …

Set Variables In Go Build: Go allows the injection of build-time variables, such as version numbers or Git hashes, directly into the code using the linker option -X.

MacOS, CLI, and Colour Scheme: To get the current colour scheme from the CLI, use the following command: defaults read -g AppleInterfaceStyle If MacOS is in dark mode, this will print Dark. But if MacOS is in light mode, this key won’t be set at all, and the command will print return an error of the form: 2022-10-04 …

Updating Stripe Subscription Items: It looks like it’s possible to replace the items of a subscription while it’s in progress. This can be useful for upgrades, etc. The general technique is as follows: Take the existing subscription items. Compare with the desired subscription items based of the price ID: If the …

Turning Off Shared History in ZSH: To turn off shared history in Zhs, simply by adding the following to your .zshrc file. unsetopt share_history

Stripe: Product vs. Price: How to determine whether to create a new product for something, or a new price for an existing product for something? It looks like the way to do so is determine whether what your giving the customer will be different. If it would be, it probably should be a new product. If it’s not, it …

Stripe Zero Decimal Currencies: List of Zero Decimal Currencies Stripe Supports

Quickly Starting A Dev Server: Here are various ways to start a HTTP dev server on the terminal. Python 2 $ python -m SimpleHTTPServer Python 3 $ python -m http.server

Go Generate with Go Run: Using go:generate to run tools over your Go application is super useful, but how can you know you’ve got the tool installed? Simply having something like: //go:generate mockery --name=Provider --case=snake --with-expecter would only work if you’ve already got mockery installed. If the …

Parametrising Your BDD Tests In Go: Parametrising Cucumber tests using Go and Go’s templating language.

Test Helpers and Packages: A technique for using helper functions that can access the private symbols of a package, yet are only available to tests.

Go And WASM: How to load and launch a Go WASM app in the browser.