Rust, D4 and Nix

Lazying and recuperating time over, work on D4 has started.

But for now, just an update on Rust and D4, with a touch on nix.

This post will mostly be about abstractions and where the world seems to be going.

Nix

Let’s start with Nix(OS).
Great distribution, love the concept, however:

• nixops must improve to handle different architectures
• there are very few flags, nothing like emerge of gentoo
• there is no standard way to hide secrets in the conf
• everyone can access /nix, which can be a security problem

So while I loved the system, and I am starting to use it on my raspberry for a lot of testing, I think it will be a long while since it realizes its full potential.

Currently the containers are not usable due to security concerns. Containers also prove that nix (the package manager) and nixops should (somehow) better integrate together.

Currently I suggest keeping nixos for the servers base system, and to manage the containers via podman

nixos inside the containers is doable too, but it’s a bit tricky to keep the read-only guarantees of podman with deploying only what is needed via nixos.

Still, I think I found my new server operating system

I think nixos is starting to get the right abstractions in managing an operating system, although they got the language wrong, and they still are looking the wrong way on the integration with deployment.

I will expand further on what I mean on next posts, but this looks like another big project, and with D4 and Fenrir I think I have my hands full :)

Rust

I have been learning rust, and I have to say I really love it!

The compilation times could be better, right, but all the typesafety, the tooling and the clean implementation of a new programming language is something I only dreamed of in C++

The quirk to work out now seem to e more focused on external libraries, crates, and not on the language itself anymore.

Macros

Just wonderful. syn and quote are extremely powerful and bring macros to a whole new level than the few things you could do in C++. Derivation of code with a comment on a structure is fantastic, it simplifies and opens so many possibilities.

The SerDe crate (the basis of almost all SERialization and DEserialization in rust is more complicated than I thought, and I need more time to dive into it, but luckily I only wanted to play with that for some advanced stuff…that it turns out I can’t really do anyway due to serde design.

I just maybe might have gone a tiny little bit too much in the rabbit hole by trying to start programming in rust directly with macros, but the concept was too beautiful not to explore.

I wanted to use serde to parse a toml configuration file (easy with existing crates) and then automatically add cmdline and environment overrides for all options.

Turns out I can’t since serde is designed to give back a struct with your configuration, it is not designed to operate on existing ones, so no overrides :(

I will come back to it eventually, since this is a feature I am missing from Go’s viper library, but I have to admit I can live without it a while longer.

Non-macro

I can’t really say how much I love the rust typesafety. In C++ it was not integrated in the whole language and you could only do part of what rust gives, with worse syntax and a lot more attention devoted to that.

Coming from the C++ world, it is not easy to immediately spot when a variable is copied or moved, and I still occasionally have relapses of not remembering the rules.

I have not had much to do with lifetimes yet, but I am getting close to needing them, though they don’t seem as though as macros at first glance.

rustfmt is a must-have, just add the 80-column rule and the formatting is already perfect.

cargo-watch is something else that I miss in the cli world of C++.

The error messages of the compiler practically tell you where the problems are and how to fix them.

Overall I really can not be happier with the language.
I didn’t think I would love it this much, honestly

D4 status

Currently D4 is definitely not working. I am still working on the architecture that comes before the actual working code.

However, in about 800 lines of code I already have a solid, typesafe configuration handling, privilege dropping, thread pool with fine grained thread management, mini database via sqlite.

D4 will work directly on drives, but since some uses preclude having direct access to disks or partitions, It will still support using raw files as backends. This is really the only reason why D4 needs the sqlite db right now. I am planning on adding some kind of auto-discovery of the disk drives, like mdadm does for all non-file related disks

Correct abstractions and io_uring

I am really impressed with the work on this API by the kernel folks.

I am really starting to see a pattern here. And I think this will be important for new technologies.
D4 will follow this pattern. What is it? Kinda simple, really: Abstractions should simplify, but not oversimplify.

We have seen this in the DirectX and OpenGL/CL world, where projects like vulkan put the hardware in the hands of the developers. If you want, you still have the higher level abstractions (and multiple companies are doing now building GL/CL/DX on top of vulkan now), but the power users developers will need access patterns that are as close to the hardware as possible.

io_uring is now fixing this in the kernel by correctly exposing the right interface between kernel and user space. Sure, probably projects like DPDK and SPDK might still give you better performance for now, but those are a complete bypass of the kernel.
So while DPDK and SPDK will still make sense in high-performance scenarios, their design make them less useful in standard usages.

D4 will be designed initially with io_uring due to the already very high performance achievable, and only much later on we will look at DPDK and SPDK.

Coming back to abstractions, the same reasoning can be done with filesystems. Object storages somehow are a step in the right direction, but they are not quite there yet.

The object storage protocol by de-facto standard (S3) however went too far and oversimplified. D4 will aim to fix that, and bring and expose the correct abstractions for what I hope will be a new global standard for object storage.

While the API of D4 is still in very high flux (to the point it’s pointless to publish it), and I still have to fully look at the backblaze one, I am confident the right direction is clear.

Just by looking at OpenGL->vulkan and filesystem vs object storage you should get an Idea of what I am talking about. If not, wait for the next posts :P

Conclusion

The D4 architecture is coming along nicely, and I am trying to make everything as cpu-pinnable and lockless as possible, though much remains to be seen yet.

Rust is a wonderful language and I wished I had discovered it earlier. Congratulations and my thanks to anyone who worked on it. It will be the base of all my projects from no on

The minor performance hit compared to C++ is greatly compensated by the added safety, features and tooling. Switch to rust if you can.

-Luker