BREAKPOINT

Coming soon ...

Preamble

from __future__ import division

Using Generator to Break Long-Running Functions

Basic operation:

result = compute(result)

Encapsulate instead and call the function ?

import time

def compute(value):
    time.sleep(0.1)
    return value + 1

That's our simulation of compute computation. Long-running operations are sequences of such operations.

def count_to_three():
    result = 0
    result = compute(result)
    result = compute(result)
    result = compute(result)
    return result

Using it:

>>> count_to_three():
3

Intermediate version ? With yield and yield None, result at the end ? That seems artificial (the end), could we accept yield result without ambiguity ? Unfortunately, no, we would need an extra option in the decorator and it's not worth it. State that it's a convention we use that is going to get explained shortly.

Generator version (minimal): Is the support for the bare yield interesting of misleading ? Present something that COULD NOT be wrapped with breakpoint, but that lays down the main idea. N.B.: with the future progress argument of breakpoint, it actually is valid code to be wrapped.

def count_to_three():
    result = 0
    yield
    result = compute(result)
    yield
    result = compute(result)
    yield
    result = compute(result)
    yield result

Using it:

>>> counter = count_to_three()
>>> counter.next()
>>> counter.next()
>>> counter.next()
>>> counter.next()
3

Version not very useful: we have split the info, but no criteria to actually do something useful in the steps. Hence progress and partial result.

Generator version: progress and partial result

def count_to_three():
    result = 0
    yield 0 / 3, result
    result = compute(result)
    yield 1 / 3, result
    result = compute(result)
    yield 2 / 3, result
    result = compute(result)
    yield 3 / 3, result

generator version with progress and partial result: explain 3 ops, hence progress computation. Explain partial result. Those two results are optional (replacable by None). If you don't care at all, yield (aka yield None) is OK too.

State that progress is optional (None if you don't know)

Using it:

>>> counter = count_to_three()
>>> counter.next()
(0.0, 0)
>>> counter.next()
(0.3333333333333333, None)
>>> counter.next()
(0.6666666666666666, None)
>>> counter.next()
(1.0, None)

Loop version, explicit progress, partial result

def count_to(n):
    result = 0
    for i in range(n):
        result = compute(result)
    return result

Using it:

>>> count_to(3)
3

def count_to(n):
    result = 0
    for i in range(n):
        progress = i / n
        yield progress, result 
        result = compute(result)
    return result

Breakpoint Decorators and Handlers

Restore the original function behavior (reverse the split) from the generator:

Question: rename breakpoint decorator function ?

@breakpoint()
def count_to(n):
    result = 0
    for i in range(n):
        progress = i / n
        yield progress, result 
        result = compute(result)
    return result

Using it:

>>> count_to(3)
3

So why the hell did we tweak the original function ? Because we have new options now such as: because now we can track the evolution of the computation. For example, display partial results:

def partial():
    def handler(**kwargs):
        print kwargs["progress"], kwargs["result"]
    return handler

Decorate count_to like this:

@breakpoint(partial)
def count_to(n):
    result = 0
    for i in range(n):
        progress = i / n
        yield progress, result 
        result = compute(result)
    return result

Use it like this:

>>> count_to(3)
(0.0, 0)
(0.3333333333333333, 1)
(0.6666666666666666, 2)
(1.0, 3)
3

Available information for the handler: includes what you returned in the yield, that is progress, result. There is more, so make sure to use **kwargs.

Explain handler factory (closure): every time your run, a new instance of the handler is generated. For example:

def partial():
   start = True
   def handler(**kwargs):
       nonlocal start # not in Python 2.7 !
       if start:
           print "first handler call"
           start = False
       print kwargs["progress"], kwargs["result"]
   return handler

Alternate implementation (class-based):

class partial(object):
    def __init__(self):
        self.start = True
    def __call__(self, **kwargs):
        if self.start:
            print "first handler call"
            self.start = False
        print kwargs["progress"], kwargs.get["result"]

Time Target

Explain that you can always track the elapsed time: demo

Then the remaining time would be nice too ! We can estimate it if you gave some progress information.

Now, the issue is that you may be calling the breakpoint handler too frequently or to little. How to cope with that ?

New assumption if you use dt: you call yield at regular frequency.

Explain elapsed and result.

Advanced Handlers

Good Enough, Thank you: Partial Results

When it's convienent to externalize stop conditions. But .... is it not what progress is for ? Can't we just simply STOP when progress is met or overreached ? Probably ! Compare the current schemes that do not use progress and the one that do ... progress is probably simpler ...

Fibionacci Sequence

Fibionacci generator:

def generator():
    a, b = 0, 1
    while True:
        yield None, a
        a, b = b, a + b

Use it like that:

... 

Modify it to return the list of computed values and the progress.

def fib_generator(n):
    a, b = 0, 1
    results = []
    while True:
        results.append(a)
        progress = a / n
        yield progress, results
        a, b = b, a + b

then use the simpler handler that makes sure that you do not overshoot your target

def track_progress():
    def handler(progress, result, **extra):
        if progress > 1.0:
            return result[:-1]
    return handler

def fibionacci(n):
    function = breakpoint(track_progress)(lambda: fib_generator(n))
    return function()

But you may actually be willing to keep the original, simple and clean, version of the fibionacci generator (hey, maybe you cannot change that code !) and externalize the termination condition. It's actually possible!

def fib_generator():
    a, b = 0, 1
    while True:
        yield None, a
        a, b = b, a + b

def max_result(n):
    def handler_factory():
        results = []
        def handler(**kwargs):
            number = kwargs["result"]
            if number > n:
                return results
            results.append(number)
        return handler
    return handler_factory

def fibionacci(n):
    function = breakpoint(max_result(n))(fib_generator)
    return function()

Root Finding

Example on how to deal with numerical precision.

TODO: example with gradient algorithm that does not progress anymore ? Search the solution of x**2 - 2 = 0 ? Rk: we could also use progress here ... Compare the two schemes ? The "not-progress" stuff externalizes more of the code ... wheter this is good or bad is yet to be seen. Rk: progress SHOULD be a linear measure of time, the algorithm complexity should be taken into account if we want the time measurements to be stable.

\[ f(x) = x^2 - 2.0 \]

We have \(f'(x) = 2x\) ... describe Newton's algorithm.

def generator():
    x0 = 2.0
    while True:
        yield None, x0
        x0 = x0 - 0.5 * (x0 - 2.0 / x0)

def min_step(eps):
    def handler_factory():
        x = [None]
        def handler(**kwargs):
            x0, x1 = x[0], kwargs["result"]
            if x0 is not None:
                step = abs(x1 - x0)
                if step < eps:
                    return x1
            x[0] = x1
        return handler
    return handler_factory

def find_root(eps=1e-15):
    function = breakpoint(handler=min_step(eps))(generator)
    return function()

TODO: add a failure time ? Add a management of convergenve to the NEGATIVE solution ?


This takes too much time ...

Handler Factories:

def max_wait(seconds):
    def handler_factory():
        def handler(**kwargs):
            remaining = kwargs["remaining"]
            if remaining > seconds:
                raise AbortException(**kwargs)
    return handler_factory

def max_yields(n):
    def handler_factory():
        yields = [0]
        def handler(**kwargs):
            yields[0] += 1
            if yields[0] > n:
                raise AbortException(**kwargs)
        return handler
    return handler_factory