http://canhekick.it/project/52464fcc8f64d1fd5512a6c7

I use the term "Pure Dataflow Programming" in reference to the original idea put forth by Bert Sutherland's paper "The On-line Graphical Specification of Computer Procedures" and others. It is characterized by fine grain operations (small operations like add or subtract) and the assumption that dataflow concepts will be employed for all levels of an application.

Flow-Based Programming takes the Dataflow concepts but says that the nodes (aka components) should be programmed using our current, sequential, control-flow programming languages. The Dataflow graph is a layer above and it controls the execution of the nodes.

Reactive Programming is a newer term. I use it to mean Dataflow using the push model (nodes only execute when data is pushed to it). It is "reactive" to changing data.

There is a tower of babble in respect to Dataflow terminology because we haven't yet defined a common meaning for all the terms. These are my definitions for the terms but everyone seems to define things slightly different.

I will have a chapter in the book that explains all the terms and their common synonyms.

As Dataflow concepts are just now starting to come to the attention of the masses, we don't have any consensus on the best, or most common, Dataflow implementation architecture. So the book will cover all the tiny variations of Dataflow architectures and compare them to each other for their pros and cons.

In Arc, my immediate approach would be to define 'push as a macro, but also to give it a procedure version that's just a little more verbose. Then we can extend the procedure version and leave the macro version alone.

Here's some working Arc 3.1 code, which calls the macro "my-push", rather than overwriting Arc's own 'push:

  (mac place (p)
    (w/uniq new-val
     `(make-place (fn () ,p) (fn (,new-val) (= ,p ,new-val)))))
  
  (def make-place (getter setter)
    (annotate 'place (list getter setter)))
  
  (def place-get (place)
    (rep.place.0))
  
  (def place-set (place new-val)
    rep.place.1.new-val)
  
  (defset place-get (place)
    (w/uniq g-place
      `((,g-place ,place)
        (place-get ,g-place)
        [place-set ,g-place _])))
  
  
  (mac my-push (elem stack)
    `(fn-push ,elem (place ,stack)))
  
  (def fn-push (elem stack)
    (zap [cons elem _] place-get.stack))

  foo <- (fn (n) (eval `(+ ,n 1) caller_scope))

  foo <- (fn '(n) (eval ((fn () `(+ ,n 1))) caller_scope))

---

"I'm still not convinced I fully understand why eval and fn each add an indirection to caller_scope ."

(For the moment, I'm going to overlook eval here.)

That's just how you chose to build it, right? Isn't the whole point of caller_scope that it's specific to the current call?

I think the main issue you're encountering is that you always use the same name, "caller_scope", resulting in shadowing. Kernel uses an explicit parameter, as in ($vau () env ...), but Wart doesn't. This isn't necessarily a problem, because you can do something like this to work around the shadowing when necessary:

  (fn (x)
    (let my_scope caller_scope
      (fn (y)
        ...)))

I think this code will evaluate the ... as though it's just inside the (fn (x) ...), so that it can see the correct binding of caller_scope:

  (fn (x)
    (let fnx_scope ((fn () caller_scope))
      (fn (y)
        (eval '... fnx_scope))))

If you tweak the language a bit, you could make the implicit caller_scope parameter get shadowed by the explicit parameters. That is, if a function is of the form (fn (caller_scope) ...), you could let it ignore its implicit caller_scope binding altogether. (Currently Wart treats this just like any other duplicate argument name, and it throws an error.) In that case, you could use this technique:

  (fn (x)
    (let my_scope caller_scope
      (fn (y)
        (let caller_scope my_scope
          ...))))

---

"I'm still not convinced I fully understand why eval and fn each add an indirection to caller_scope ."

(Now to tackle eval.)

Are you sure 'eval adds a layer of indirection? I think the only indirection you're encountering is caused by 'fn. I just tried it out in Wart to make sure:

  ((fn () ((fn () (caller_scope = caller_scope)))))
  => ''1
  ((fn () ((fn () (caller_scope = (eval 'caller_scope caller_scope))))))
  => nil
  ((fn () ((fn () (caller_scope = ((fn () (eval 'caller_scope caller_scope))))))))
  => ''1

Within this second layer, we get a true value for (caller_scope = caller_scope). This is just a sanity check.

We false value for (caller_scope = (eval 'caller_scope caller_scope)), presumably because we're comparing the two different environments we've established.

Finally, it seems we can successfully use ((fn () (eval '<foo> caller_scope)))) in place of <foo>, because although it introduces a new environment, it evals in the old one.

Here's another way we can verify that eval doesn't introduce its own binding for caller_scope:

  ((fn () (eval 'caller_scope caller_scope)))
  020lookup.cc:28 no binding for caller_scope
  => nil
  ((fn () (eval '2 caller_scope)))
  => 2

---

"But I needed it to expand to this:"

  let $old push
    mac (push x s)
      `(if (isa stack ,s)
         (push ,x (rep ,s))
         (,$old ,x ,s))

  mac (push x s) :case `(isa stack ,s)
    `(push ,x (rep ,s))
  
  -->
  
  let $old push
    mac (push x s)
      `(if ,`(isa stack ,s)
         ,`(push ,x (rep ,s))
         (,$old ,x ,s))

In between the , and the ` you have "room to operate on the names before starting on the backquote":

  mac (foo n) :case (let a .. `(isa foo-type ,a))
    with (a ..
          b ..)
      `(bar ,a ,b)
  
  -->
  
  let $old foo
    mac (foo n)
      `(if ,(let a .. `(isa foo-type ,n))
         ,(with (a ..
                 b ..)
            `(bar ,a ,b))
         (,$old ,n))

---

"But even if there isn't, I'm starting to question if lisp is indeed the hundred-year programming model. I'm starting to feel at the end of my tether, at the limits of what macroexpansion can cleanly support."

  (def foo (a) `(list ,a))

  arc> (foo 3)
  (list 3)

---

"Perhaps it's impossible to do if commas don't expand to unquote ?"

From what you're saying, it sounds like you're using some custom data structures, but that you just haven't given yourself all the primitives you need to manipulate them.

  > foo <- (make-unquote 12)
  ,12
  
  > (unquote? foo)
  t
  
  > (unquote-get foo)
  12
  
  > foo <- (make-quasiquote 12)
  `12
  
  > (quasiquote? foo)
  t
  
  > (quasiquote-get foo)
  12

  # Simplified expansion of:
  #   mac (foo x)
  #     `(+ ,x 1)
  #
  #   mac (foo x) :case nil
  #     `(list ,car.x)

  mac (foo x)
    `(if nil
       ,`(list ,(car x))
       ,`(+ ,x 1))

  (foo 3)

The case itself should be a piece of generated code:

  -#   mac (foo x) :case nil
  +#   mac (foo x) :case `nil
  
  -  `(if nil
  +  `(if ,`nil

Also, this seems to be unrelated to the issue you're having, but I just realized you need to be careful about variable scope. What if the original definition said "mac (foo x)" but the extension said "mac (foo y)"? I recommend putting in a form that binds the right variables:

  -  `(if ,`nil
  +  `(if ,(with (x x)
  +          `nil)
  -    ,`(list ,(car x))
  +    ,(with (x x)
  +       `(list ,(car x)))
       ,`(+ ,x 1))

---

To address the issue you're encountering, yes, every case will expand at every call site, even if there's no intuitive need for it to expand there. It can't branch on a decision that hasn't been made until run time, unless you're willing to use the full power of fexprs. (I assume you're trying to maintain a semblance of stage separation, or you wouldn't distinguish macros from functions.)

Keeping this stuff in mind, you can write your code so that it doesn't give you that error:

  # Simplified expansion of:
  #   mac (foo x)
  #     `(+ ,x 1)
  #
  #   mac (foo x) :case `nil
  #     if acons.x
  #       `(list ,car.x)
  #       `(err "I hope this code will never be reached.")

  mac (foo x)
    `(if ,(with (x x)
            `nil)
       ,(with (x x)
          (if (acons x)
           `(list ,(car x))
           `(err "I hope this code will never be reached.")))
       ,`(+ ,x 1))

Whoops, that (with ...) stuff isn't ideal because it leaves the outer variables in scope. That is, if the original definition has a parameter called "list", an extension will see that binding even if it uses a different name for the same parameter, so it might try to call the global 'list and be surprised.

I suppose I recommend doing this instead, where g123-args is a gensym:

  mac (foo . g123-args)
    `(if ,(apply (fn (x)
                   `nil)
                 g123-args)
       ,(apply (fn (x)
                 (if (acons x)
                   `(list ,(car x))
                   `(err "I hope this code will never be reached.")))
               g123-args)
       ,(apply (fn (x)
                 `(+ ,x 1))
               g123-args))

  let g1-old foo
    (mac (foo . g2-args)
      `(if ,(apply (fn (x)
                     `nil)
                   g2-args)
         ,(apply (fn (x)
                   (if (acons x)
                     `(list ,(car x))
                     `(err "I hope this code will never be reached.")))
                 g2-args)
         
         (,g1-old ,@g2-args)
         ; or in Arc, ",(apply rep.g1-old g2-args)"
         
         )

https://github.com/Pauan/nulan (project page)

http://pauan.github.io/nulan/doc/tutorial.html (interactive tutorial in the browser)

https://github.com/arclanguage/rainbow-js (project page)

http://arclanguage.github.io/rainbow-js/test/ (interactive REPL)

It's a direct port of Rainbow (https://github.com/conanite/rainbow), but with a hand-rolled parser and a hand-rolled implementation of asynchronous stream I/O. Like Rainbow, it "compiles" Arc code to an intermediate representation and interprets that. Rainbow was already a fast Arc implementation (faster than official Arc!) so Rainbow.js is pretty speedy as well. I used to think it was even faster than Rainbow itself, but then I realized that was due to a bug. :-p

I haven't been motivated to use Rainbow.js myself, because it has many quirks: Some quirks of its own (e.g. no threads), some inherited from Rainbow (e.g. no bignums, and weird call/cc behavior during macroexpansion), and of course all the quirky features of Arc (e.g. ssyntax). It also has some quirks which are truly necessary to make it usable; for instance, its (load ...) implementation loads from the Web, because there is no filesystem. While this (load ...) fix makes some sense, it's not always clear to me what the best behavior would be. Instead of pondering these small design questions and imposing my own decisions on this project, I've been working on other language designs from the ground up instead.