The thesis is, in general, excellent and well worth a read if you're interested in software reliability even outside of Erlang. The first few chapters are just a great read for any developer I think.
When he sends the "{become, F}" message, and F is a function pointer or closure, how is that dispatched across a network? Do the code definitions need to be present remotely too? If so, I'm not sure that quite qualifies as a "universal" server, only a server for locally defined functions. Still cool though.
BTW, I've used a similar technique to create a pool of worker goroutines in Golang. Had good performance from it too.
That function's code is serialized and sent to remote node along with necessary data.
In erlang, if you want, you can compile whole modules from source constructed at runtime and they behave just like normal code. Typical usage is hot-loading new modules from disk or from other nodes, you load your patch on one node and distribute it over network and if you don't make something stupid, your code is updated in whole system without dropping any tcp connections or the like. Everything happens automatically and every process using old code is updated as soon as possible to new version when possible.
If you've ever looked at Erlang's data model briefly and gone "what the heck is with the immutability and lack of user-defined structures, etc.", have another look at it from this point of view of distribution. It's the secret decoder ring of "why did Erlang do it that way?"
With another ~25 years of collective experience, if somebody took a shot at it today I think we could make some improvements, but it was pretty far ahead of its time, and can still pull some stunts almost nothing else can, or at least not without creating "an ad-hoc, informally-specified, bug-ridden, slow implementation of half of [Erlang]", to tweak Greenspun a bit.
if you call a function in the form module:function-name then erlang detect and use the latest compiled version of the module, if you use function-name without the module then the old version is used. Is advised to use the module:function-name in tail call position.
Sure, but if you send fun Module:function-name/arity over the network, that will be sent as EXPORT_EXT [1], and only work if the module is available on the receiver.
I wouldn't use function sending to deploy new code in a productionish environment. It's fun to play with, but it's easier to understand a production environment if you push the code/compiled beam files as traditional files through whatever means you like (rsync/packages/whatever), and then load the updated beam files.
You could do the code loading by sending a function to the node that calls code:load(Module) and the like; code:soft_purge(Module) first is a good idea to make sure you don't kill lingering processes by accident.
You could go all in and have (most of) your Erlang nodes be diskless Erlang, requesting the code to run from another Erlang server when they start, and that seems like a fun adventure, but I don't know if it adds a lot of business value. I'd be happy to play with that on your dime, contact in profile ;)
Honestly, in most scenarios, you kill the vm and let it restart with new code (just like you would in a kubernetes pod, and in fact using kubernetes pods with BEAM languages is also a valid strategy).
If you must keep some state preserved, there's even ways to do it without erlang's hot code reloading strategies, I would recommend watching Dan Azuma's fantastic video (stick around for the live demo): https://www.youtube.com/watch?v=nLApFANtkHs
If you need to install module on other node, you make a fun which installs that module (typically one function call) and execute it on other node, that fun is automatically sent and executed where you want it. It's like using remote shell (ssh), but your program is automatically sent to remote server.
In erlang the code is compiled to an intermmediate representation for a virtual machine, BEAM (erlang virtual machine), usually the functions you define are very short and in this case the BEAM code is sent over the network, so the universal server receives the compiled or transpiled code and executed it using the BEAM. You can compare it to a lisp server in which you send the program as a list, in lisp code is data.
Can these functions be closures that close over a bunch of data that isn't hard-coded into the functions? I don't know erlang, but imagine the erlang version of this python code:
def adder(v):
return lambda x: v + x
Can I then send the result of (say) "adder(5)" over the network and have it work?
It's also crazy insecure, so you shouldn't use this sort of stuff unless you really know what you're doing and you have full control over the inputs and outputs, that includes owning your network in depth.
I don't want to discourage you from using erlang and Elixir, because the beam is amazing, plenty of very lucrative companies (like bet365) do, just, be aware of things you might versus might not run in an actual prod situation.
I mean, it's not that it's crazy insecure. It's that the security model is totally different. In distributed erlang, access to any node is effectively access to all of the nodes; it's best to consider the distributed system as one computer from a security point of view.
Do not run distributed Erlang with nodes you don't trust. Do not run distributed Erlang on a network you don't trust (VPN/TLS can help).
well specfically `binary_to_term` on arbitrary untrusted data is insecure, but you could of course pass `[safe]` in. Just, read the docs and heed the warnings.
You could, of course, send that file over the network (via email, tcp/ip, carrier pigeon, whatever), and it's even easier if you use built-in erlang distribution primitives, which indeed do this under the hood.
Keep in mind that by default values are immutable, so when you're building that lambda it takes out a fixed-in-time reference to that 5 (or whatever, you could even make it user input). In other languages, if that variable changes... Something strange might happen, depending on what that underlying datatype is. In particular, side effects can exist for lists but not scalars in python.
In elixir, you can rebind that variable 5 to something else but it won't change the lambda (erlang doesn't allow you to do that at all, except in the repl, sort of). You could also do that with, say, a list datatype, and you get the same deal.
for example:
list = [1,2,3]
lambda = fn -> length(list) end
lambda.() #==> 3
list = list ++ [4, 5]
list #==> [1,2,3,4,5]
lambda.() #==> 3
which is what lets you have a lambda with no expectation on the mutability of the bound-in values. What you get is what you expect, no matter how far away or disconnected or even dead the source is. And that respect for the physics of causality and relativity, at the core, is what makes the BEAM languages so great for distributed systems.
In erlang you can serialize anything. I can write a lambda and email it to you and you can put it in your VM and assuming you have the correct dependencies, it will probably work.
Dependencies are also just atom (universal, global, enum) references, so there's no pointers involved.
Unfortunately, Erlang does not directly implement Actor behavior change.
Consequently, Erlang cannot directly implement Actor behavior change. A factorial server is implemented below:
FactorialServer[ ] *implements* ProcedureServer<[NaturalNumber], NaturalNumber>
[i] |-> // received message [i]
Factorial.[i] // return result of sending [i] to Factorial
In order to implement Actor behavior change, Erlang must resort to using helper processes that trampoline back so that helper processes can return the desired change.
See the following for direct implementation of Actor behavior change:
https://papers.ssrn.com/abstract=3418003
PS. The type ProcedureServer above can be defined as follows:
ProcedureServer<t1, t2> ≡ // Procedure server with parameters types t1 and t2 is defined to be
t1 -> t2 // type of procedure from t1 into t2
By "Actor" I see that you're referring to a particular theory named "Actor", you're not using the term in a colloquial sense.
You seem to be saying that Erlang can implement "Actor behaviour change", just not directly. In what sense is this a concern? i.e., what value does direct implementation add here?
Yes, Actor is used in the technical sense axiomatized up to a unique isomorphism in the linked article and not in the common usage of a thespian ;-)
Erlang not being able to directly express Actor behavior change means that Erlang programs are more obscure and convoluted.
For example, consider the following program:
ProcessorController[pk:PublicKey] *implements* controller
// initialize processor controller with a public key pk
currrentVersion ≔ 1 // processor current version number is initially 1
running ← Crowd [1] // running is a crowd of at most 1 running activities
boot[*itemIs* i *whichIs* Package[*imageIS* code, *versionIs* packageVersion],
signedIs s] ↦ // received boot request with item i and signed s
*IsEmpty* running *cases* // Check if the processor is running
True ⇾ // If not running, then
SigningChecker.[*itemIs* i, *signedIs* s, *publicKeyIs* pk] *cases*
// check that i was signed by s with private key for pk
True ⇾ // If signing check succeeds, then
currentVersion⩽packageVersion *cases*
// check processor current version is
// less or equal than package version
True ⇾ // If version check succeeds, then
(currentVersion ≔ packageVersion; // first update current version number,
code.run thru running)
// afterward run code in a hole of mutual exclusion passing
// thru running then returning a termination report
False ⇾ // If version check fails,
*Throw* BadVersionException[] // throw bad version exception
False ⇾ *Throw* BadSignatureException[] // If signature check fails,
False ⇾ // If already running, then
*Throw* AlreadyRunningException[] // throw already running exception
shutDown ↦ // received shut down request
*IsEmpty* running *cases* // Check if the processor is running
True ⇾ // If not running, then
*Throw* NotRunningException[] //throw not running exception
False ⇾ // If already running, then
*Cancel* running // cancel running returning Void
The challenge in Erlang is to implement the hole in the region of mutual exclusion in the above implementation.
How about something like this? I don't understand the code you wrote exactly, and I smooshed the processor into the controller, because it simplifies things.
fun Self (PublicKey) ->
CurrentVersion = 1,
receive
{upgrade, Function, Version, Signature} ->
case sign_checker(PublicKey, Function, Version, Signature) of
true when Version > CurrentVersion -> Function(PublicKey);
true -> throw(bad_version);
false -> throw(bad_signature)
end;
shutdown -> ok;
_Other ->
% here's where you handle work
% But I don't know what work you wanted to do,
% so I'm just looping
Self(PublicKey)
end end.
You probably don't actually want to throw for these errors (why should the server stop if it got an invalid request?), and you probably want your messages to include a reply tag, so you can get the results (like gen_server:gen_call/2,3 does), but you know, examples on the interwebs; don't run them as-is.
The signature for a controller to implement is as follows:
Controller *interface*
boot[*itemIs* Item, *signedIs* Signed] → TerminationReport
// boot request with an item and signed returns
// a termination report from the boot request
shutDown → Void // shut down request returns Void
The challenge in Erlang is to allow the shutDown request to be received even though the boot request has not yet returned with a termination report.
The cancellation exception is caught within the implementation of the run request and turned into a termination report.
fun Self (State) ->
{PublicKey, Ref, Function} = case State of
{P, R, F} when is_ref(R), is_function(F) -> {P, R, F};
Other -> {Other, false, false}
end,
CurrentVersion = 1,
receive
{upgrade, Function, NewVersion, Signature} when Ref == false ->
SignRef = make_ref(),
async_sign_checker({self(), SignRef}, PublicKey, Function, NewVersion, Signature),
Self({PublicKey, SignRef, Function, NewVersion);
{upgrade, _, _, _} -> throw (no_concurrent_upgrades);
{sign_check, SignRef, true} when Version > CurrentVersion -> Function(PublicKey);
{sign_check, SignRef, true} -> throw(bad_version);
{sign_check, SignRef, false} -> throw(bad_signature);
{sign_check, _, _} -> throw(bad_sign_check);
shutdown -> ok;
_Other ->
% here's where you handle work
% But I don't know what work you wanted to do,
% so I'm just looping
Self(PublicKey)
end end.
assuming async_sign_check sends back something sensible {sign_check, SignRef, true}. async_sign_check would send the work to another process, if you want it to stop checking the signature in case of early shutdown, the processes could be linked.
Although, I'm still not quite sure I understand the system you're trying to get to. This in the context of {become, F}, so I'm assuming you want the process receiving the boot request to become the requested function. But you also want a termination report; and I'm assuming the function isn't expected to terminate, it's expected to receive and process requests. But, you didn't tell us where you want these reports sent?
In my updated code, there is certainly a possible window where the signature checker finished near the same time as a shutdown request was sent. In that case, if the shutdown request is received first, the process would shutdown; if the signature result is received first, that would be processed, and if the signature result and versioning was good, the process would become the new function, and that function would receive the shutdown request. If the new function has some complicated setup, and you wanted to shut it down in the middle of that, you would most likely need to use erlang:exit(Pid, kill); although, a carefully written function could periodically check for a shutdown message in the middle of startup. I don't find that a terribly useful usecase --- if you started something, let it finish starting, or just brutally kill it; architecting a clean shutdown from a partial start is most likely to be wasted effort, instead, one should architect their start so it doesn't perturb the state of the world until it is fully started and can be expected to shutdown cleanly.
Looks like you are still not implementing the signature of a Controller, which as follows:
Controller *interface*
boot[*itemIs* Item, *signedIs* Signed] → TerminationReport
// boot request with an item and signed returns
// a termination report from the boot request
shutDown → Void // shut down request returns Void
A boot request checks the item which is a Package with code and packageVersion and then runs the code. The boot request returns a termination report that results from running the code.
However, when the boot request is still operating, the controller might receive a shutDown request, which will cause the cancellation of the running code that will caught within the run request and turned into a termination report.
Note that there is no timing error in the Actor implementation because, a shutDown request cannot be received until code.run has released the region of mutual exclusion.
OK, so this isn't really {become, F}, this is {run, F}.
The function F, is expected to return results and terminates, so it is no longer running (but, it could have spawned a process or otherwise changed the world state; Erlang is generous with respect to side effects)
I don't really understand the version number means in this case? You can only run functions with the same or greater versions than had previously been run?
I think you're somehow asking for this to be both synchronous, in the sense that the result of running the function is returned directly and asynchronous in that you'd like to be able to cancel it. If you want the return directly, I can't also have previously returned a cancellation id.
If you just want to run a function, and get the results back, that's not that hard; but before I write the code for that, I want to really be sure what you're asking for.
With respect to your question: "I don't really understand the version number means in this case?" An item can be booted only if its version number has not regressed in to prevent replay attacks.
There are no cancellation ids in the Actor implementation, however a cancellation exception can be thrown when an activity is cancelled.
Function(PublicKey) runs the code, which presumably knows its new version number. As mentioned otherwise, I had assumed this was in the context of {become, F}; but perhaps you had something else in mind.
The Erlang version doesn't need to become to run code either. It is just a fun thing to do. It's straightforward to just run the code as well; if you get a function F, you just do F(), assuming the function has no arguments, because you didn't include any; you can also use erlang:apply(F, Args), if you had passed a list of arguments.
Honestly, I think we're going back and forth here, because we're not speaking the same language. I don't know what a crowd is, and I don't know what cancelled means. Those aren't concepts in Erlang. I'm not sure if you mean to have a distinction between 'boot' and 'run', you're using them both, and I think to mean the same thing you send a boot request when you want to run a piece of signed code?
Erlang is foundationally about independent processes with ordered mailboxes. The two ways to interpret cancel in Erlang would be a) send a message 'cancel', and hope the process receives it before completing the work, or b) kill the process. A process can set a flag so that most kills will be turned into sending a message, but there's also an untrappable kill. Processes can also be linked and/or monitored for propagation of kills.
You would typically model something with state like this as a process. The process would essentially be looping on a single function, passing any state it needed to keep to itself. In this case, you'd send it a boot request with the code to run etc, and your process id and a reference to get a response, and potentially a cancellation request, and maybe a get current version, and get running status (you didn't spec it, but they're both useful).
The state in this case, that would be the current version, the public key, the current boot request (if any), and tracking details for the asynchronous signature checking, and tracking the running of the code.
If you want to be able to kill the running code, without killing this process that has the state, then you'd need to spawn a new process to run the code, and you'd track that. If you don't really need that, you could just run the code in this process, and when it returns send back a result.
Erlang can kill a process, but does not have cancellation, which causes a process to be halted and throw a cancellation exception from the point at which it was halted so that clean up can be performed while unwinding.
Crowds are a way to keep track of processes performing activities so that they can be cancelled, etc.
A boot request is sent to a controller to boot up a processor returning a termination report. A run request is sent to code to run returning a value.
The Erlang approach to implementing behavior change by recursing on a single procedure passing the variables in a tail call has issues implementing holes in regions of mutual exclusion similar to ones encountered in JavaScript. Because tail calls with state variables do not always work and Erlang does not have an assignment command, Erlang programs often use helper processes to temporarily hold state.
The above Actor implementation illustrates how the above interact in a way that can make Erlang implementations more complicated. Such interactions are common in implementing intelligent systems.
> Erlang can kill a process, but does not have cancellation, which causes a process to be halted and throw a cancellation exception from the point at which it was halted so that clean up can be performed while unwinding.
Typically you implement the callback terminate/2 in a genserver. If the process is hopelessly locked, you don't get to do the cleanup you want. It's usually not a big deal, though, because your processes are usually "owner in name only" for any given contentious resource, and a well-designed interface to a contentious resource uses what erlang aptly calls "resources", that lets the VM itself manage cleanup when the owning process dies.
In the end, this leads to cleaner code, because you don't even have to worry about cleaning up (I almost never clean up tcp sockets, SSH connections, or file descriptors, because the VM does it for me and knows exactly when to do it). I am writing a native code interface, and the same can easily go from a non-erlang OS thread, by hooking in the correct "resource" adapters, I can have OS thread lifecycle well-managed by the already robust OTP system.
> Erlang does not have an assignment command, Erlang programs often use helper processes to temporarily hold state.
It actually kind of does. For the sorts of things you seem to be talking about, there is a process-bound k/v store that you can use to sneak some statefulness in. It's totally not talked about much because it's dangerous, especially for beginners, but it's useful when you know that the lifetime of your value is specifically bound to the lifetime of the process. In Elixir, this process k/v store is used, for example, to register in a side channel a "callers" and "ancestors" tree; when you're running tests, this lets you for example, run concurrent database tests transparently associated with the test process, that are also associated with a database transaction, so you can have concurrent database actions which are sandboxed to their own view of the universe. Some libraries even let you encode this into an http request so that your request can leave the VM, return back into the VM, and maintain its sandbox id and database association on the other side of the request. All of this is managed with about two lines of code provided out of the box in the gold standard database library.
Prof Hewitt: I don't think erlang is a "true actor system" but I think it's much easier to code without errors than you may think.
> Erlang can kill a process, but does not have cancellation, which causes a process to be halted and throw a cancellation exception from the point at which it was halted so that clean up can be performed while unwinding.
Indeed, Erlang does not have a way to induce a catchable exception into a process stack from outside the process. Processes are isolated, and the way that processes interact with the world is through their mailbox.
The Erlang way (TM) is to let your processes crash (or be killed), but be supervised and restarted, and the new process will handle whatever state the world is in, and bring it to a good state (or it won't, and it will die, and the supervisor will die, and the system will restart, eventually).
You could certainly pepper you code with something like
receive cancel -> throw cancel after 0 -> ok end
In places where you might want to be cancelled, and send a cancel message as appropriate, to get something like what you're asking. But it wouldn't feel very Erlangy, and looking at the results of being explicit about everywhere you might want to be cancelled might lead you towards the premise of Let it Crash. There are a million and one ways that the system might fail during activity, and for most of them, the right thing to do is restart, pick up the pieces, and go forward from there if you can. You're going to have to handle that anyway in case of unexpected restarts due to hardware or software faults.
> A boot request is sent to a controller to boot up a processor returning a termination report. A run request is sent to code to run returning a value.
What do you mean by a processor here. Like a separate CPU, or is this a process / thread?
What is the relationship between this processor and the code, and the run request? What behavior is changing in this example?
> Because tail calls with state variables do not always work and Erlang does not have an assignment command, Erlang programs often use helper processes to temporarily hold state.
That Erlang programs often use processes to hold state is not a deficiency; that's how Erlang models the world. State is intended to be held in processes, and the way that they do that is through tail call recursion on the explicit state. Although, sometimes, some state is stored in the process dictionary (which is essentially a key-value store) or in ETS/Mnesia, which can be modeled as a process (but is implemented as shared memory), or in state held outside of Erlang (such as the filesystem, or other programs).
If you don't like the world model where state is held in processes, Erlang clearly isn't for you; but I find it's a model that fits well with distributed systems, and allows for complex systems to be built with small teams.
> Code should not be peppered something like the following:
receive cancel -> throw cancel after 0 -> ok end
Instead, cancellation should trigger cleanup to make it easier to recover. If no clean up is needed, then it doesn't have to be done.
Actors were designed for massively concurrent systems that are often physically distributed. Behavior change (including state change) is essential for Actors, e.g., security, performance, etc.
Hah, of course I meant the "techie" colloquial of Actor, not the thepsian one. But I suppose that my comment is true in both senses. :)
Thanks for clarifying. I'm not familiar with your theory, but at least I understand why you raised the point -- some Erlang programs are needlessly complicated due to the language's (colloquial!) actor semantics.
> Some Erlang programs are needlessly complicated due to the language's (colloquial!) actor semantics.
Maybe you're referring to junior-to-BEAM programmers reaching to prove they can use gen_servers everywhere?
There are also obscure corners of OTP that are a bit of a mess, but I'd say that's because it's inherited a lot of C-style folkways and doesn't have hierarchical module aliasing in the same way that say Elixir has. And there's a substantial amount of using modules-like-factories that got picked up in certain corners of the ecosystem, probably from Java habits, but I don't see that as much in Elixir, which suggests that it's not the actor semantics so much as, just outdated programming trends that have accumulated in the erlang ecosystem by virtue of it being older, and a "less popular" language that other people try to shove their round pegs into (arguably elixir is the latest iteration of that but they seem to have made some really good choices, at least in my opinion).
It's incredible how someone with so much knowledge and programming experience still came down to help newbies in the Elixir forum. He was one of the big reasons why I started learning Elixir and owe the career pivot to him. God bless you Mr. Armstrong.
https://erlang.org/download/armstrong_thesis_2003.pdf