I am cramped

@kjuulh

2023-09-09

Cramped is an experiment at building a fully open source data platform built on both new and old components. This project is heavily biased towards my ideals, this generally means no java (even if extremely illogical in the data world, I will dig into this in a later post) where it can be avoided, the system should scale, but should be able to run on somewhat resource constrained environments, like a normal developer workstation.

My primary ideals are:

  • Opinionated: I want to get 80% of the way there with 20% of the effort, if your use-case is in the 20 remaining, then this project is not for you
  • Efficient: The project should feel fast, in execution, but also to develop on.
  • Minimal: The project should feel minimal, even if it has a lot of complexities, what is done here is trying to create actual abstractions, and let cross talk between components be as limited as possible
  • Immutable and compose-able architecture: The architecture should feature immutable steps, and present a consistent api of which real applications can use as its bedrock. Each component should be developed in of itself, this doesn't mean that components shouldn't communicate, but that they will have a valid migration or use pattern in of itself.

The name is "Cramped"

I like to use words that described a feature I'd like to avoid, I usually call the project exactly what I am trying to do to avoid doing while developing it. Sometimes it succeeds other times it doesn't but it is my ideal. For example in World of Warcraft, I played a Warrior in which my play style would best be described as Coked out if its mind Squirrel, aptly named Reckless, I gave it that name to try to better my behavior, and actually play like a sane person.

Cramped is the feeling of being stuck in a small space, where even if you can make room to do stuff, you feel constrained, annoyed and uncomfortable. I'd like exactly the opposite, a platform where you feel like you can make the changes you need, and iterate as you'd like, while having fun working on new and old parts of the system, and generally trust that the changes you do makes sense.

The goal

As mentioned Cramped is first and foremost an experiment. I've maintained an existing data platform single-handily (to mixed success) since January (we're in September now), as such I've not had the experience of actually building a data platform from scratch, and with a lot of dependent engineers and analytics, I can't really make as much experimentation as I'd like. Which gave rise to the feeling associated with the project name.

I hope this project will help me work out some of the frustrations I've had with our existing system, as well as get some knowledge on how to improve it, what is current and best practice atm.

I am a platform engineer first and foremost, so definitely not a Data engineer or Data Ops engineer. I am doing this for fun, but also go get some more experience, so please go easy on me, as I work through and share my decisions.

Approach

I will start by defining my top level architecture components, and simply start small on developing each part. Each part should provide inside into which decisions I am going for and why I am doing so. I will use esoteric languages and tools. However, most of them have alternatives that may or may not be useful for your platform.

Each piece will be developed by itself, and be deployed independently. I will also use my own tools cuddle and churn for respectively Development Experience Platform and Deployment and Orchestration. I don't know why I have a fascination with Cs... Each of these tools aren't ready to see the light of day yet, but they're incredibly useful for my own development.

Next

I will add a Table of Context of the different parts as we go.

  • TBA