Yielding functions proposal

Hi all!


While swift 2 is not open source yet, there isn't a specific place for discussing new feature proposals. Maybe for now this is the most appropriate place? I want to propose swift support for "yielding functions" a concept similar to coroutines, generators, fibers, etc in other languages. These constructs provide an excellent framework for working with asynchronous code, making it feel more like synchronous code, and making the job of handling errors much easier than with callbacks.


My proposal is something along the lines of:


func createYieldingFunction(foo: String) -> String yields -> String {
    return { bar in      
        let baz = yield foo + bar
        return foo + baz
    }
}

do {
    let yieldingFunction = createYieldingFunction(foo: "foo") // yieldingFunction will be: String yields -> String
    let foobar = try yieldingFunction("bar")                  // foobar will be: "foobar"

    if yieldingFunction.isAlive {
        print("this will print")
    }

    let foobaz = try yieldingFunction("baz")                  // foobaz will be: "foobaz"

    if yieldingFunction.isAlive {
        print("this will not print")
    }

    try yieldingFunction("blah")                              // will throw error
} catch {
    print(error)                                              // will print something like: Dead yielding function called
}


The function createYieldingFunction is a regular function that returns a "yielding function". We use a function that generates a "yielding function" because these kinds of functions have a limited lifetime. If the "yielding function" returns, it turns into a "dead" state, meaning that it can't execute code anymore. So, if we have a dead yielding function, we can just create another one by calling createYieldingFunction again.


Yielding functions are very similar to "throwing functions", actually, every yielding function is implicitly marked as throws (or act like they're marked as throws) and should be called with the "try", "try!" or "try?" keywords. It "throws" by default because if you try to call a dead yielding function it wouldn't be capable of honoring the call, so it should throw a standard error like DeadYieldingFunctionError. If we want to know at runtime if the function is still alive we can call "isAlive" on it, which in turns returns true if the yielding function is still alive or false if it is dead.


The first time you call a yielding function you pass the required parameters like a regular function. When the function yields, it returns the yielded value like a regular return and passes the control back to the caller. The next time the caller calls the yielding function passing the required parameters, the code continues execution from where it left off. The "yield" keyword inside the yielding function returns the passed parameters and the execution goes on untill a new yield or a return appears.


Another important feature of yielding functions is the ability to throw errors from the caller scope into the yielding function. This is done by calling throw before the yielding function with the specified error.


func createYieldingFunction() -> Void yields -> Void {
    return {
        do {
            yield
        } catch {
            print(error) // will print Error(description: "Error that will be thrown into the yielding function")
        }
    }
}

do {
    let yieldingFunction = createYieldingFunction()
    try yieldingFunction() // executes untill the yield
    throw yieldingFunction Error(description: "Error that will be thrown into the yielding function")
} catch {
    print(error) // will never be called
}


If the the yielding function catches the error, it can deal with the error inside the yielding function. It's important to know that even if you catch errors inside the yielding function the yielding function itself will still throw errors if it is called in a dead state.


func createYieldingFunction() -> Void yields -> Void {
    return {
        do {
            yield
        } catch {
            print(error) // will print Error(description: "Error that will be thrown into the yielding function")
        }
    }
}

do {
    let yieldingFunction = createYieldingFunction()
    try yieldingFunction() // executes untill the yield
    throw yieldingFunction Error(description: "Error that will be thrown into the yielding function")
    try yieldingFunction() // throws DeadYieldingFunctionError
} catch {
    print(error) // will print something like: Dead yielding function called
}


If the yielding function doesn't catch the error, the error will be retrown back to the caller.


func createYieldingFunction() -> Void yields -> Void {
    return {
        yield
    }
}

do {
    let yieldingFunction = createYieldingFunction()
    try yieldingFunction() // executes untill the yield
    throw yieldingFunction Error(description: "Error that will be thrown into the yielding function")
} catch {
    print(error) // will print Error(description: "Error that will be thrown into the yielding function")
}


TL;DR


With yielding functions, instead of this pyramid of doom:


getString { string, error in
    if let error = error {
        print(error)
    } else {
        let upperCaseString = string!.uppercaseString
        getOtherString(upperCaseString) { anotherString, anotherError in
            if let error = anotherError {
                print(error)
            } else {
                print(anotherString!)
            }
        }
    }
}


We can write this:


async { _ in
    try {
        let string = yield getString()
        let upperCaseString = string.uppercaseString
        let anotherString = yield getOtherString(upperCaseString)
        print(anotherString)
    } catch {
        print(error)
    }
}


We basically turned asynchronous code into synchronous looking code, solving the readability and easing error handling. (getString() and getOtherString() are still async)

Replies

OK, if I have to be completely explicit...


Create an OS X command line application with this code.


import Foundation


struct Error : ErrorType, CustomStringConvertible {
    let description: String
}

func getString(completion: (String?, ErrorType?) -> Void) {
    let delay = dispatch_time(DISPATCH_TIME_NOW, Int64(1 * Double(NSEC_PER_SEC)))
    let queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)
  
    dispatch_after(delay, queue) {
        if arc4random_uniform(2) == 0 {
            completion("hello", nil)
        } else {
            completion(nil, Error(description: "Error in getString"))
        }
    }
}

func getOtherString(string: String, completion: (String?, ErrorType?) -> Void) {
    let delay = dispatch_time(DISPATCH_TIME_NOW, Int64(1 * Double(NSEC_PER_SEC)))
    let queue = dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_DEFAULT, 0)

    dispatch_after(delay, queue) {
        if arc4random_uniform(2) == 0 {
            completion("\(string) world!", nil)
        } else {
            completion(nil, Error(description: "Error in getOtherString"))
        }
    }
}

getString { string, error in
    if let error = error {
        print(error)
    } else {
        let upperCaseString = string!.uppercaseString
        getOtherString(upperCaseString) { anotherString, anotherError in
            if let error = anotherError {
                print(error)
            } else {
                print(anotherString!)
            }
        }
    }
}

dispatch_main()


Can you guess what it does? Of course I'm using a stupid example to represent when you have code that depend on the result of a previous async operation. You could be waiting on a image to download so you can apply a filter and them upload it back to the server, etc..


Do you see how ugly this code looks with this nested callbacks?

As far as I can tell, you're proposing a structure that's way too cumbersome to actually be used, that exposes the caller to all of the implementation specific issues like "How long is the platform going to let this method execute before it suspends?"


1. The caller has to have lines of code to unwrap/process the optional portion of the yield.

2. The caller has to have lines of code to ensure that the yielding method is called again if it's not yet complete.

3. How many times the yeilding method needs to be called is arbitrary and under the control of neither the caller of the yielding method or the person who wrote the yielding method.

4. The caller has to have lines of code to catch the exception caused by calling the yielding method more times than it was supposed to be called.

5. If the caller calls the yielding method too many times, it's now lost the actual return value.


So every times someone uses a yielding function in practice, they're going to swamp the call site in boiler plate code, and complaint #5 means that they're going to have to spend more time debugging the code.


Edit: Note that your sample code in your proposal is wrong. The commend on line 9 should be

// foobaz _might_ be "foobaz", or it could be a yielded function value.


  1. do {
  2. let yieldingFunction = createYieldingFunction(foo: "foo") // yieldingFunction will be: String yields -> String
  3. let foobar = try yieldingFunction("bar") // foobar will be: "foobar"
  4. if yieldingFunction.isAlive {
  5. print("this will print")
  6. }
  7. let foobaz = try yieldingFunction("baz") // foobaz will be: "foobaz"
  8. if yieldingFunction.isAlive {
  9. print("this will not print")
  10. }
  11. try yieldingFunction("blah") // will throw error
  12. } catch {
  13. print(error) // will print something like: Dead yielding function called
  14. }

Actually you can use the yielding function like this at the call site:


async { _ in
    try {
        let string = yield getString()
        let upperCaseString = string.uppercaseString
        let anotherString = yield getOtherString(upperCaseString)
        print(anotherString)
    } catch {
        print(error)
    }
}


The yielding function is passed as a parameter to the async function. This looks cumbersome to you? All the complexity is hidden inside the async function. You should check generators in ES6 to really get what this is all about. I rewrote my example above, to be veeeery explicit. Check this:

http://jlongster.com/A-Study-on-Solving-Callbacks-with-JavaScript-Generators


It's from 2013 so a lot has changed since, but you can see the basic idea. Also I recommend reading the other post I linked before:


https://medium.com/@tjholowaychuk/callbacks-vs-coroutines-174f1fe66127

I don't think you really understood the concept. The yield only returns control to the caller or get back control from the caller. There's no "platform" involved, no scheduling, or something like that. When the caller calls the function, it is passing control to the function. When the function yields, it gives back control to the caller, this can be done a lot of times, until the function returns, and "dies". If you know what you're doing you can write library functions, like the one I used in the example, to take care of all of the complexity, making the call site extremely simple. After all, this is the goal.

If you are having trouble understaning the concept. Just think about how this:


getString { string, error in
    if let error = error {
        print(error)
    } else {
        let upperCaseString = string!.uppercaseString
        getOtherString(upperCaseString) { anotherString, anotherError in
            if let error = anotherError {
                print(error)
            } else {
                print(anotherString!)
            }
        }
    }
}


can become this:


async { _ in
    try {
        let string = yield getString()
        let upperCaseString = string.uppercaseString
        let anotherString = yield getOtherString(upperCaseString)
        print(anotherString)
    } catch {
        print(error)
    }
}


That's all you need to know, to profit from this feature.

But where is the profit in using this? Very similar syntax is possible already by just using throwing functions.

You can't throw with async functions. You have to pass the error to the callback, or throught the "future". This is exactly with this is useful, because it makes it possible to use async functions like synchronous functions that throw errors.

As people have to understand the basic problem and something about the various different existing solutions in order to give meaningful feedback to your particular proposal, maybe it would help if you or someone else could provide a broad overview of the problem and some comparisons to already established related techniques.


You already mentioned ES6 generators, which I guess is somewhat similar to the approaches taken in Google's Go and Clojure. It might also be relevant to include comparisons to things like React, RAC, Reactive Extensions, etc., and maybe even NSOperations with dependencies? But most importantly, a broad overview of the problem(s) that all these techniques aim to solve.


I'm not conversant enough with this topic, but I think it's very interesting/important.

But why would I want to do that? Just let linear programm logic go - it died ages ago when event based programming started.

I guess my main problem is that I don't get why one should use that completion handler nonsense in the first place. So it follows that a solution to replace them seems kinda strange, too.

I do like the idea of yielding, but I feel like this version is too complicated, and as someone mentioned above, exposes too many implimentation details to the caller.


Yeild is similar to return, except instead of ending the execution of the function, it pauses it in place. Then the function resumes from where it was when it is called again. For example: I would like to be able to run through a list of items, and "yeild" to return the first item meeting some criteria. Then at some later point, I would like to resume running through that list (picking up where I left off). This would make building a packrat-style parser much easier.


Perhaps a simpler way to approach this would be to have a rule that once it "returns", it forever returns that same value when called (as opposed to having "live" and "dead" states). That way, if I want it to make it optional and have it return nil after it is "dead", I can do that. Or I can throw an error if I prefer. Or I could just have it return an empty string if I want.


The trickiest bit to understand for me in your code above is how you basically need a "yeilding function factory" to create these things. Not sure how to avoid that, but it is confusing (and boilerplate). I guess it is similar to the relationship between sequences and generators though.


Maybe yeilding functions could return a struct (or tuple) with the value yeilded and a "remainder" which you would call like a closure to continue execution. Calling the original function again would start at the beginning like any other function. Calling the remainder would continue execution from where it had left off, and return another value/remainder pair.


A contrived and innefficient example which shows the behavior:

func chunkString(str:String, chunkLength:Int) yeilds -> String {
     var chunk = ""
     for c in str.characters {
          chunk += String(c)
          if chunk.characters.length >= chunkLength {
               yeild chunk
               chunk = ""
          }
     }
     yeild chunk
     return ""
}


You would then call it like so:

let (chunk1,rem1) = chunkString("ABCDEFGH",chunkLength:3)
print(chunk1) //"ABC"

let (chunk2,rem2) = rem1()
print(chunk2) //"DEF"

let (newChunk,_) = chunkString("ABCDEFGH",chunkLength:3)
print(newChunk) //"ABC"

let (chunk3,rem3) = rem2()
print(chunk3) //"GH"

let (chunk4,rem4) = rem3()
print(chunk4) //""


If you don't care about the remainder, you should be able to do this as well

let chunk = chunkString("ABCDEFGH",chunkLength:3)
print(chunk) //"ABC"


Possible alternate notation:

let chunk yeilding rem = chunkString("ABCDEFGH",chunkLength:3)
print(chunk) //"ABC"

//Leave off the "yeilding" if we don't care about continuing later
let chunk = chunkString("ABCDEFGH",chunkLength:3)


Maybe someone else has a better way to simplify?

FWIW:


Considering the impasse we have on this topic, the lack of a suitable example (in my opinion), I wonder if it would be possible to create a working example using SwiftGo quoted in this new topic.

https://forums.developer.apple.com/thread/20572


I'm very interested in new ways to create asynchronous code, but after a long time using GCD i almost never can structure my mind to think in a different way.

https://github.com/Zewo/SwiftGo