Announcing EmbassyOS 0.3.0!
Sovereign Computing
Two years ago, Start9 released EmbassyOS 0.1.0 — the first ever, general purpose, mass market operating system for a personal server. It enabled anyone, regardless of technical expertise, to self-host their own software services on their own personal server inside their own home, thereby eliminating trusted third parties from select areas of their digital lives.
In September of 2020, we released EmbassyOS 0.2.0, which touted a generalized dependency management system and unparalleled service uptime and reliability, establishing Start9 not only as the pioneer of the self-hosting revolution, but as its unequivocal leader. 0.2.x was, until today, the most advanced, secure, and reliable personal server OS in the world.
Now, we are proud to announce the release of EmbassyOS 0.3.0! From base architecture, to UI design, and everything in between — 0.3.0 improves significantly upon its predecessor and marks a milestone in the history of personal computing, an inflection point when the tide of decentralization will begin to swell.
We are also excited to show off a brand new website, including an improved User Manual and all new Developer Documentation!
Below is a detailed enumeration of the changes and improvements from 0.2.x to 0.3.0.
Highlights
SSD Support
Operating system and service data are now stored on an SSD instead of the microSD, massively improving storage capacity, as well as overall performance and reliability. It is now possible to run an archival Bitcoin node, and Initial Block Download (syncing from the genesis block) has been reduced from ~11 days down to ~2–3 days.
Unified Backend Codebase
EmbassyOS 0.2.x was a composition of three components: Lifeline, Agent, and Appmgr. Lifeline was written in Rust and was responsible for system initialization and disaster recovery. Agent was written in Haskell and provided a client API as well as managed server-specific business logic such as power, WiFi, SSH keys, health metrics, and SSL certificates. Appmgr was written in Rust and was Start9’s novel container management system, responsible for installing, configuring, and running services.
EmbassyOS 0.3.0 removes all three components in favor of a single, unified backend codebase written entirely in Rust.
PatchDB
Start9’s own, purpose-built PatchDB is a data storage and transport system that implements the JSON Patch protocol to facilitate real-time state synchronization between server and clients. Whenever a state change occurs on the server, the change is communicated to one or more clients via websocket in the form of a nonced “Patch”, which itself consists of one or more “Operations”. You can think of a Patch as a “block”, and Operations as “transactions”. Patches must be ingested in sequence, and Operations direct state changes to be made atomically. In this way, state can be updated incrementally, maximizing efficiency without sacrificing reliability. If a client ever falls out of sync or falls sufficiently behind as to negate the efficiency of incremental updates, the client can request a database dump, essentially wiping the slate clean and starting fresh with the latest state.
Websocket Support
As explained above, server-side state changes are communicated to web clients in real time via websockets. This provides an exceptional experience. Despite physical distance, the user feels “plugged in”, as if their browser screen were directly attached to their Embassy.
OS/Data Bifurcation
With EmbassyOS 0.2.x, OS data, service data, and the OS itself were all stored together on a single microSD card. Amongst other drawbacks, this meant that a user could not re-flash the microSD with a new OS version without losing their data. This made both recovery procedures and standard OS updates challenging.
EmbassyOS 0.3.0 bifurcates the operating system from its data. Now, the microSD card stores a read-only copy of the operating system, some metadata, and enough space for a 2nd whole copy of EmbassyOS, enabling red/green updates and rollbacks!
OS and service data are stored on an attached SSD, separate from the OS itself.
Alternative Marketplaces
Beginning with EmbassyOS 0.3.0, users can access alternative service marketplaces. Anyone can host their own marketplace and make it available to others, and connecting to a new marketplace is a simple matter of entering its clearnet or Tor URL inside the settings menu. This is an important feature for ensuring Start9 does not come to occupy a central point of failure for service availability going forward.
Session Management
It is now possible to view your active sessions across various devices and log out remotely.
Revamped User Interface
The user interface has been rewritten and redesigned to be more performant, understandable, and beautiful :)
Comprehensive Backups
Creating backups in EmbassyOS 0.2.x required going service-to-service and backing them up one at a time. With 0.3.0, creating backups is now a single button. Additionally, you can continue to use your Embassy during the backup process and monitor progress in real time.
LAN Shared Folders (for backups)
EmbassyOS now supports creating encrypted backups “over the air” to a LAN Shared Folder — that is, a folder on your laptop/desktop (or an external drive) that you “share” with the Local Area Network (LAN). Because the Raspberry Pi cannot sufficiently power multiple physical drives without additional hardware, LAN Shared Folders are the recommended way to create backups. This flow is also the groundwork for remote backups.
Streamlined OS Updates
Previously, updating EmbassyOS was a blocking procedure. Meaning, while an OS update was taking place, Embassy was unusable. Now, updates take place in the background, and Embassy continues to run as usual. This is important for achieving maximum uptime while also providing real time progress reports during the update process.
Self-Contained Setup Flow
In 0.2.x, setting up an Embassy involved downloading a native Setup App from the App Store or Play Store. With 0.3.0, we are proud to eliminate this dependency on Apple and Google in favor of a self-contained web application served by EmbassyOS itself. Now, setting up an Embassy is a matter of plugging it in and visiting embassy.local from your browser of choice. EmbassyOS then guides you through a few simple steps, ultimately resulting in three artifacts: (1) a unique Tor (.onion) address, (2) a unique LAN (.local) address, and (3) a self-signed Root Certificate Authority for SSL (https) encrypted communication over LAN. After setup, embassy.local ceases to exist, and Embassy is thereafter reachable on either its unique Tor address or its unique LAN address. By necessity, the initial setup flow is facilitated over unencrypted http, but under the hood, communications are encrypted using your Embassy’s unique product key; so as long as you do not expose your product key, even someone naively snooping on your LAN during setup cannot hijack information.
Form-Based Service Config
EmbassyOS 0.2.x touted a graphical method for configuring services. Instead of requiring users to use SSH and edit config files from the command line, which can be daunting and prone to error, we developed a graphical user interface that enabled configuration through the Embassy dashboard. In 0.3.0, we made service config even better, granting it a full face lift, including a cleaner aesthetic, better indicators, and more guided error messaging.
Improved Logging
Embassy and service-specific logs are now paginated, such that users can scroll back through time to view older logs. This is highly useful for system insight, including and especially debugging. They are also now brought to you in beautiful technicolor :)
Diagnostic UI
This is a new user interface that will show in the event that EmbassyOS is unable to discover or access an expected data drive. This might happen if you unplug your SSD or plug in the wrong SSD. Embassy will boot to the diagnostic UI, explain clearly what the issue is and provide suggestions for fixing it.
Enhanced Dev Tools
EmbassyOS package developers use creativity and skill to shape the end user experience for a given service. EmbassyOS translates their choices and code into standard UI elements that anyone can understand. They are, in effect, the always-present, helping hand that makes running a personal server accessible to non-technical users. With EmbassyOS 0.3.0, we are giving developers even more power with an unified, turing complete API for things such as backup and restore, GUI configuration, server metrics, server properties, notifications, health checks, Actions, and dependency requirements and interactions.
Arbitrary Inputs for Actions
In 0.2.x, package developers could define Actions, arbitrary scripts for users to execute, such as resyncing Bitcoin from genesis. So rather than requiring users to use SSH and the command line, they could simply click a button that says “resync blockchain”, and the script would be executed under the hood. With EmbassyOS 0.3.0, Scripts can now receive user inputs of arbitrary complexity, such as a string value or a deeply nested form.
Progress Reporting
With EmbassyOS 0.3.0, we introduce visual progress reporting for service installs/updates, Embassy updates, service backups, and full Embassy recoveries.
Multiple Interfaces
In EmbassyOS 0.2.0, each installed service received its own Tor Address (.onion URL). For some services, such as File Browser, the URL represented a website that could be visited in the browser; for other services, such as Bitcoin, the URL had to be input into a native client wallet such as Fully Noded or Specter. Certain services, such as Bitcoin, actually have multiple interfaces. Bitcoin has an RPC interface, a P2P interface, a ZeroMQ interface, and could potentially even have a graphical User interface, such as a dashboard displaying important node information. Using the same URL for these various interfaces is not only confusing, it could potentially pose a security vulnerability. For example, a user may want to share their P2P interface address with someone for peering but not want to give out their UI address, which is for private use only. As such, EmbassyOS 0.3.0 assigns a unique URL for every interface. Users can view and access all interfaces for a given service inside the new “Interfaces’’ section of the service dashboard.
Advanced Dependency Management
EmbassyOS 0.2.x introduced a generalized dependency management system, such that services would be aware of services they depend on. For example, C-Lightning depends on Bitcoin. If Bitcoin is not installed, configured properly, or actively running, C-Lightning can inform the user of the unmet dependency and refuse to start up until the dependency is satisfied.
In 0.3.0, we redesigned the dependency management system to be even smarter and more robust. Now, services can depend on any aspect of another service. For example, a service might require another service to have been running for at least 10 minutes before starting up itself.
Health Checks
One of the most critical duties of a server operator is to monitor the health of your services. For example, a simple health check that monitors the availability of an LND node could mean the difference between that node having a poor reputation or a great one. Sometimes, it is not obvious when a service is unhealthy, especially since “health” is a subjective term. For example, is your Bitcoin node “healthy” if it is not fully synced? In EmbassyOS 0.3.0, package developers define what constitutes health and implement health checks according to subjective criteria that are then displayed to the user in easily digestible messages, complete with icons and colors. Health checks are completely arbitrary and turing complete, meaning they can include anything, including config options and internal or external dependencies! For example, a Lightning wallet package developer could say “this service is only healthy if (1) it is fully synced, (2) Bitcoin is fully synced, (3) LND is fully synced, and (4) if and only if the user has opted for real-time pricing from a third party website, that third party website must be reachable.” Enormous power.
