Preamble
from __future__ import divisionUsing Generator to Break Long-Running Functions
Basic operation:
result = compute(result)Encapsulate instead and call the function ?
import time
def compute(value):
time.sleep(0.1)
return value + 1That's our simulation of compute computation. Long-running operations are sequences of such operations.
def count_to_three():
result = 0
result = compute(result)
result = compute(result)
result = compute(result)
return resultUsing it:
>>> count_to_three():
3Intermediate version ? With yield and yield None, result at the end ? That seems artificial (the end), could we accept yield result without ambiguity ? Unfortunately, no, we would need an extra option in the decorator and it's not worth it. State that it's a convention we use that is going to get explained shortly.
Generator version (minimal): Is the support for the bare yield interesting of misleading ? Present something that COULD NOT be wrapped with breakpoint, but that lays down the main idea. N.B.: with the future progress argument of breakpoint, it actually is valid code to be wrapped.
def count_to_three():
result = 0
yield
result = compute(result)
yield
result = compute(result)
yield
result = compute(result)
yield resultUsing it:
>>> counter = count_to_three()
>>> counter.next()
>>> counter.next()
>>> counter.next()
>>> counter.next()
3Version not very useful: we have split the info, but no criteria to actually do something useful in the steps. Hence progress and partial result.
Generator version: progress and partial result
def count_to_three():
result = 0
yield 0 / 3, result
result = compute(result)
yield 1 / 3, result
result = compute(result)
yield 2 / 3, result
result = compute(result)
yield 3 / 3, resultgenerator version with progress and partial result: explain 3 ops, hence progress computation. Explain partial result. Those two results are optional (replacable by None). If you don't care at all, yield (aka yield None) is OK too.
State that progress is optional (None if you don't know)
Using it:
>>> counter = count_to_three()
>>> counter.next()
(0.0, 0)
>>> counter.next()
(0.3333333333333333, None)
>>> counter.next()
(0.6666666666666666, None)
>>> counter.next()
(1.0, None)Loop version, explicit progress, partial result
def count_to(n):
result = 0
for i in range(n):
result = compute(result)
return resultUsing it:
>>> count_to(3)
3
def count_to(n):
result = 0
for i in range(n):
progress = i / n
yield progress, result
result = compute(result)
return resultBreakpoint Decorators and Handlers
Restore the original function behavior (reverse the split) from the generator:
Question: rename breakpoint decorator function ?
@breakpoint()
def count_to(n):
result = 0
for i in range(n):
progress = i / n
yield progress, result
result = compute(result)
return resultUsing it:
>>> count_to(3)
3So why the hell did we tweak the original function ? Because we have new options now such as: because now we can track the evolution of the computation. For example, display partial results:
def partial():
def handler(**kwargs):
print kwargs["progress"], kwargs["result"]
return handlerDecorate count_to like this:
@breakpoint(partial)
def count_to(n):
result = 0
for i in range(n):
progress = i / n
yield progress, result
result = compute(result)
return resultUse it like this:
>>> count_to(3)
(0.0, 0)
(0.3333333333333333, 1)
(0.6666666666666666, 2)
(1.0, 3)
3Available information for the handler: includes what you returned in the yield, that is progress, result. There is more, so make sure to use **kwargs.
Explain handler factory (closure): every time your run, a new instance of the handler is generated. For example:
def partial():
start = True
def handler(**kwargs):
nonlocal start # not in Python 2.7 !
if start:
print "first handler call"
start = False
print kwargs["progress"], kwargs["result"]
return handlerAlternate implementation (class-based):
class partial(object):
def __init__(self):
self.start = True
def __call__(self, **kwargs):
if self.start:
print "first handler call"
self.start = False
print kwargs["progress"], kwargs.get["result"]Time Target
Explain that you can always track the elapsed time: demo
Then the remaining time would be nice too ! We can estimate it if you gave some progress information.
Now, the issue is that you may be calling the breakpoint handler too frequently or to little. How to cope with that ?
New assumption if you use dt: you call yield at regular frequency.
Explain elapsed and result.
Advanced Handlers
Good Enough, Thank you: Partial Results
When it's convienent to externalize stop conditions. But .... is it not what progress is for ? Can't we just simply STOP when progress is met or overreached ? Probably ! Compare the current schemes that do not use progress and the one that do ... progress is probably simpler ...
Fibionacci Sequence
Fibionacci generator:
def generator():
a, b = 0, 1
while True:
yield None, a
a, b = b, a + bUse it like that:
... Modify it to return the list of computed values and the progress.
def fib_generator(n):
a, b = 0, 1
results = []
while True:
results.append(a)
progress = a / n
yield progress, results
a, b = b, a + bthen use the simpler handler that makes sure that you do not overshoot your target
def track_progress():
def handler(progress, result, **extra):
if progress > 1.0:
return result[:-1]
return handler
def fibionacci(n):
function = breakpoint(track_progress)(lambda: fib_generator(n))
return function()But you may actually be willing to keep the original, simple and clean, version of the fibionacci generator (hey, maybe you cannot change that code !) and externalize the termination condition. It's actually possible!
def fib_generator():
a, b = 0, 1
while True:
yield None, a
a, b = b, a + b
def max_result(n):
def handler_factory():
results = []
def handler(**kwargs):
number = kwargs["result"]
if number > n:
return results
results.append(number)
return handler
return handler_factory
def fibionacci(n):
function = breakpoint(max_result(n))(fib_generator)
return function()Root Finding
Example on how to deal with numerical precision.
TODO: example with gradient algorithm that does not progress anymore ? Search the solution of x**2 - 2 = 0 ? Rk: we could also use progress here ... Compare the two schemes ? The "not-progress" stuff externalizes more of the code ... wheter this is good or bad is yet to be seen. Rk: progress SHOULD be a linear measure of time, the algorithm complexity should be taken into account if we want the time measurements to be stable.
\[ f(x) = x^2 - 2.0 \]
We have \(f'(x) = 2x\) ... describe Newton's algorithm.
def generator():
x0 = 2.0
while True:
yield None, x0
x0 = x0 - 0.5 * (x0 - 2.0 / x0)
def min_step(eps):
def handler_factory():
x = [None]
def handler(**kwargs):
x0, x1 = x[0], kwargs["result"]
if x0 is not None:
step = abs(x1 - x0)
if step < eps:
return x1
x[0] = x1
return handler
return handler_factory
def find_root(eps=1e-15):
function = breakpoint(handler=min_step(eps))(generator)
return function()TODO: add a failure time ? Add a management of convergenve to the NEGATIVE solution ?
This takes too much time ...
Handler Factories:
def max_wait(seconds):
def handler_factory():
def handler(**kwargs):
remaining = kwargs["remaining"]
if remaining > seconds:
raise AbortException(**kwargs)
return handler_factory
def max_yields(n):
def handler_factory():
yields = [0]
def handler(**kwargs):
yields[0] += 1
if yields[0] > n:
raise AbortException(**kwargs)
return handler
return handler_factory