tensorpack.utils package

Common utils. These utils should be irrelevant to tensorflow.

tensorpack.utils.change_env(name, val)

Parameters:

• name (str) – name of the env var

• val (str or None) – the value, or set to None to clear the env var.

Returns:

a context where the environment variable name being set to val. It will be set back after the context exits.

tensorpack.utils.get_rng(obj=None)

Get a good RNG seeded with time, pid and the object.

Parameters:obj – some object to use to generate random seed. Returns:np.random.RandomState – the RNG.

tensorpack.utils.fix_rng_seed(seed)

Call this function at the beginning of program to fix rng seed within tensorpack.

Parameters:seed (int) –

Note

See https://github.com/tensorpack/tensorpack/issues/196.

Example

Fix random seed in both tensorpack and tensorflow.

seed = 42
utils.fix_rng_seed(seed)
tesnorflow.set_random_seed(seed)
# run trainer

tensorpack.utils.get_tqdm(*args, **kwargs)

Similar to tqdm.tqdm(), but use tensorpack’s default options to have consistent style.

tensorpack.utils.execute_only_once()

Each called in the code to this function is guaranteed to return True the first time and False afterwards.

Returns:bool – whether this is the first time this function gets called from this line of code.

Example

if execute_only_once():
    # do something only once

tensorpack.utils.humanize_time_delta(sec)

Humanize timedelta given in seconds

Parameters:sec (float) – time difference in seconds. Must be positive. Returns:str - time difference as a readable string

Example:

print(humanize_time_delta(1))                                   # 1 second
print(humanize_time_delta(60 + 1))                              # 1 minute 1 second
print(humanize_time_delta(87.6))                                # 1 minute 27 seconds
print(humanize_time_delta(0.01))                                # 0.01 seconds
print(humanize_time_delta(60 * 60 + 1))                         # 1 hour 1 second
print(humanize_time_delta(60 * 60 * 24 + 1))                    # 1 day 1 second
print(humanize_time_delta(60 * 60 * 24 + 60 * 2 + 60*60*9 + 3)) # 1 day 9 hours 2 minutes 3 seconds

tensorpack.utils.argtools module

tensorpack.utils.argtools.map_arg(**maps)

Apply a mapping on certain argument before calling the original function.

Parameters:maps (dict) – {argument_name: map_func}

tensorpack.utils.argtools.memoized(user_function)

Alias to functools.lru_cache() WARNING: memoization will keep keys and values alive!

tensorpack.utils.argtools.memoized_method(func)

A decorator that performs memoization on methods. It stores the cache on the object instance itself.

tensorpack.utils.argtools.graph_memoized(func)

Like memoized, but keep one cache per default graph.

tensorpack.utils.argtools.shape2d(a)

Ensure a 2D shape.

Parameters:a – a int or tuple/list of length 2 Returns:list – of length 2. if a is a int, return [a, a].

tensorpack.utils.argtools.shape4d(a, data_format='NHWC')

Ensuer a 4D shape, to use with 4D symbolic functions.

Parameters:a – a int or tuple/list of length 2 Returns:list –

of length 4. if a is a int, return [1, a, a, 1]

or [1, 1, a, a] depending on data_format.

tensorpack.utils.argtools.memoized_ignoreargs(func)

A decorator. It performs memoization ignoring the arguments used to call the function.

tensorpack.utils.argtools.log_once

Log certain message only once. Call this function more than one times with the same message will result in no-op.

Parameters:

• message (str) – message to log

• func (str) – the name of the logger method. e.g. “info”, “warn”, “error”.

tensorpack.utils.argtools.call_only_once(func)

Decorate a method or property of a class, so that this method can only be called once for every instance. Calling it more than once will result in exception.

tensorpack.utils.concurrency module

class tensorpack.utils.concurrency.StoppableThread(evt=None)

Bases: threading.Thread

A thread that has a ‘stop’ event.

__init__(evt=None)

Parameters:evt (threading.Event) – if None, will create one.

queue_get_stoppable(q)

Take obj from queue, but will give up when the thread is stopped

queue_put_stoppable(q, obj)

Put obj to queue, but will give up when the thread is stopped

stop()

Stop the thread

stopped()

Returns:bool – whether the thread is stopped or not

class tensorpack.utils.concurrency.LoopThread(func, pausable=True)

Bases: tensorpack.utils.concurrency.StoppableThread

A pausable thread that simply runs a loop

__init__(func, pausable=True)

Parameters:func – the function to run

pause()

Pause the loop

resume()

Resume the loop

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

class tensorpack.utils.concurrency.ShareSessionThread(th=None)

Bases: threading.Thread

A wrapper around thread so that the thread uses the default session at “start()” time.

__init__(th=None)

Parameters:th (threading.Thread or None) –

default_sess()

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

start()

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

tensorpack.utils.concurrency.ensure_proc_terminate(proc)

Make sure processes terminate when main process exit.

Parameters:proc (multiprocessing.Process or list) –

class tensorpack.utils.concurrency.OrderedResultGatherProc(data_queue, nr_producer, start=0)

Bases: multiprocessing.context.Process

Gather indexed data from a data queue, and produce results with the original index-based order.

__init__(data_queue, nr_producer, start=0)

Parameters:

• data_queue (mp.Queue) – a queue which contains datapoints.

• nr_producer (int) – number of producer processes. This process will terminate after receiving this many of DIE sentinel.

• start (int) – the rank of the first object

get()

run()

Method to be run in sub-process; can be overridden in sub-class

class tensorpack.utils.concurrency.OrderedContainer(start=0)

Bases: object

Like a queue, but will always wait to receive item with rank (x+1) and produce (x+1) before producing (x+2).

Warning

It is not thread-safe.

__init__(start=0)

Parameters:start (int) – the starting rank.

get()

has_next()

put(rank, val)

Parameters:

• rank (int) – rank of th element. All elements must have different ranks.

• val – an object

class tensorpack.utils.concurrency.DIE

Bases: object

A placeholder class indicating end of queue

tensorpack.utils.concurrency.mask_sigint()

Returns:If called in main thread, returns a context where SIGINT is ignored, and yield True. Otherwise yield False.

tensorpack.utils.concurrency.start_proc_mask_signal(proc)

Start process(es) with SIGINT ignored.

Parameters:proc – (mp.Process or list)

Note

The signal mask is only applied when called from main thread.

tensorpack.utils.fs module

tensorpack.utils.fs.mkdir_p(dirname)

Like “mkdir -p”, make a dir recursively, but do nothing if the dir exists

Parameters:dirname (str) –

tensorpack.utils.fs.download(url, dir, filename=None, expect_size=None)

Download URL to a directory. Will figure out the filename automatically from URL, if not given.

tensorpack.utils.fs.recursive_walk(rootdir)

Yields:str – All files in rootdir, recursively.

tensorpack.utils.fs.get_dataset_path(*args)

Get the path to some dataset under $TENSORPACK_DATASET.

Parameters:args – strings to be joined to form path. Returns:str – path to the dataset.

tensorpack.utils.loadcaffe module

tensorpack.utils.loadcaffe.load_caffe(model_desc, model_file)

Load a caffe model. You must be able to import caffe to use this function. :param model_desc: path to caffe model description file (.prototxt). :type model_desc: str :param model_file: path to caffe model parameter file (.caffemodel). :type model_file: str

Returns:dict – the parameters.

tensorpack.utils.loadcaffe.get_caffe_pb()

Get caffe protobuf. :returns: The imported caffe protobuf module.

tensorpack.utils.logger module

The logger module itself has the common logging functions of Python’s logging.Logger. For example:

from tensorpack.utils import logger
logger.set_logger_dir('train_log/test')
logger.info("Test")
logger.error("Error happened!")

tensorpack.utils.logger.set_logger_dir(dirname, action=None)

Set the directory for global logging.

Parameters:

• dirname (str) – log directory

• action (str) –

  an action of [“k”,”d”,”q”] to be performed when the directory exists. Will ask user by default.

  ”d”: delete the directory. Note that the deletion may fail when the directory is used by tensorboard.

  ”k”: keep the directory. This is useful when you resume from a previous training and want the directory to look as if the training was not interrupted. Note that this option does not load old models or any other old states for you. It simply does nothing.

tensorpack.utils.logger.auto_set_dir(action=None, name=None)

Use logger.set_logger_dir() to set log directory to “./train_log/{scriptname}:{name}”. “scriptname” is the name of the main python file currently running

tensorpack.utils.logger.get_logger_dir()

Returns:The logger directory, or None if not set. The directory is used for general logging, tensorboard events, checkpoints, etc.

tensorpack.utils.serialize module

tensorpack.utils.serialize.loads(buf)

Parameters:buf – the output of dumps.

tensorpack.utils.serialize.dumps(obj)

Serialize an object.

Returns:Implementation-dependent bytes-like object.

class tensorpack.utils.serialize.NonPicklableWrapper(obj)

Bases: object

TODO

https://github.com/joblib/joblib/blob/master/joblib/externals/loky/cloudpickle_wrapper.py

tensorpack.utils.compatible_serialize module

tensorpack.utils.stats module

class tensorpack.utils.stats.StatCounter

Bases: object

A simple counter

average

count

feed(v)

Parameters:v (float or np.ndarray) – has to be the same shape between calls.

max

min

reset()

samples()

Returns all samples.

sum

class tensorpack.utils.stats.BinaryStatistics

Bases: object

Statistics for binary decision, including precision, recall, false positive, false negative

false_negative

false_positive

feed(pred, label)

Parameters:

• pred (np.ndarray) – binary array.

• label (np.ndarray) – binary array of the same size.

precision

recall

reset()

class tensorpack.utils.stats.RatioCounter

Bases: object

A counter to count ratio of something.

count

int: the total

Type:Returns

feed(count, total=1)

Parameters:

• cnt (int) – the count of some event of interest.

• tot (int) – the total number of events.

ratio

reset()

total

int: the total

Type:Returns

class tensorpack.utils.stats.Accuracy

Bases: tensorpack.utils.stats.RatioCounter

A RatioCounter with a fancy name

accuracy

class tensorpack.utils.stats.OnlineMoments

Bases: object

Compute 1st and 2nd moments online (to avoid storing all elements).

See algorithm at: https://www.wikiwand.com/en/Algorithms_for_calculating_variance#/Online_algorithm

feed(x)

Parameters:x (float or np.ndarray) – must have the same shape.

mean

std

variance

tensorpack.utils.timer module

tensorpack.utils.timer.total_timer(msg)

A context which add the time spent inside to TotalTimer.

tensorpack.utils.timer.timed_operation(msg, log_start=False)

Surround a context with a timer.

Parameters:

• msg (str) – the log to print.

• log_start (bool) – whether to print also at the beginning.

Example

with timed_operation('Good Stuff'):
    time.sleep(1)

Will print:

Good stuff finished, time:1sec.

tensorpack.utils.timer.print_total_timer()

Print the content of the TotalTimer, if it’s not empty. This function will automatically get called when program exits.

class tensorpack.utils.timer.IterSpeedCounter(print_every, name=None)

Bases: object

Test how often some code gets reached.

Example

Print the speed of the iteration every 100 times.

speed = IterSpeedCounter(100)
for k in range(1000):
    # do something
    speed()

__init__(print_every, name=None)

Parameters:

• print_every (int) – interval to print.

• name (str) – name to used when print.

reset()

class tensorpack.utils.timer.Timer

Bases: object

A timer class which computes the time elapsed since the start/reset of the timer.

is_paused()

pause()

Pause the timer.

reset()

Reset the timer.

resume()

Resume the timer.

seconds()

Returns:float –

the total number of seconds since the start/reset of the timer, excluding the

time in between when the timer is paused.

tensorpack.utils.viz module

tensorpack.utils.viz.interactive_imshow(img, lclick_cb=None, rclick_cb=None, **kwargs)

Parameters:

• img (np.ndarray) – an image (expect BGR) to show.

• rclick_cb (lclick_cb,) – a callback func(img, x, y) for left/right click event.

• kwargs – can be {key_cb_a: callback_img, key_cb_b: callback_img}, to specify a callback func(img) for keypress.

Some existing keypress event handler:

• q: destroy the current window

• x: execute sys.exit()

• s: save image to “out.png”

tensorpack.utils.viz.stack_patches(patch_list, nr_row, nr_col, border=None, pad=False, bgcolor=255, viz=False, lclick_cb=None)

Stacked patches into grid, to produce visualizations like the following:

Parameters:

• patch_list (list[ndarray] or ndarray) – NHW or NHWC images in [0,255].

• nr_row (int), nr_col(int) – rows and cols of the grid. nr_col * nr_row must be no less than len(patch_list).

• border (int) – border length between images. Defaults to 0.05 * min(patch_width, patch_height).

• pad (boolean) – when patch_list is a list, pad all patches to the maximum height and width. This option allows stacking patches of different shapes together.

• bgcolor (int or 3-tuple) – background color in [0, 255]. Either an int or a BGR tuple.

• viz (bool) – whether to use interactive_imshow() to visualize the results.

• lclick_cb – A callback function f(patch, patch index in patch_list) to get called when a patch get clicked in imshow.

Returns:

np.ndarray – the stacked image.

tensorpack.utils.viz.gen_stack_patches(patch_list, nr_row=None, nr_col=None, border=None, max_width=1000, max_height=1000, bgcolor=255, viz=False, lclick_cb=None)

Similar to stack_patches() but with a generator interface. It takes a much-longer list and yields stacked results one by one. For example, if patch_list contains 1000 images and nr_row==nr_col==10, this generator yields 10 stacked images.

Parameters:

• nr_row (int), nr_col(int) – rows and cols of each result.

• max_width (int), max_height(int) – Maximum allowed size of the stacked image. If nr_row/nr_col are None, this number will be used to infer the rows and cols. Otherwise the option is ignored.

• border, viz, lclick_cb (patch_list,) – same as in stack_patches().

Yields:

np.ndarray – the stacked image.

tensorpack.utils.viz.dump_dataflow_images(df, index=0, batched=True, number=1000, output_dir=None, scale=1, resize=None, viz=None, flipRGB=False)

Dump or visualize images of a DataFlow.

Parameters:

• df (DataFlow) – the DataFlow.

• index (int) – the index of the image component.

• batched (bool) – whether the component contains batched images (NHW or NHWC) or not (HW or HWC).

• number (int) – how many datapoint to take from the DataFlow.

• output_dir (str) – output directory to save images, default to not save.

• scale (float) – scale the value, usually either 1 or 255.

• resize (tuple or None) – tuple of (h, w) to resize the images to.

• viz (tuple or None) – tuple of (h, w) determining the grid size to use with gen_stack_patches() for visualization. No visualization will happen by default.

• flipRGB (bool) – apply a RGB<->BGR conversion or not.

tensorpack.utils.viz.intensity_to_rgb(intensity, cmap='cubehelix', normalize=False)

Convert a 1-channel matrix of intensities to an RGB image employing a colormap. This function requires matplotlib. See matplotlib colormaps for a list of available colormap.

Parameters:

• intensity (np.ndarray) – array of intensities such as saliency.

• cmap (str) – name of the colormap to use.

• normalize (bool) – if True, will normalize the intensity so that it has minimum 0 and maximum 1.

Returns:

np.ndarray – an RGB float32 image in range [0, 255], a colored heatmap.

tensorpack.utils.viz.draw_boxes(im, boxes, labels=None, color=None)

Parameters:

• im (np.ndarray) – a BGR image in range [0,255]. It will not be modified.

• boxes (np.ndarray) – a numpy array of shape Nx4 where each row is [x1, y1, x2, y2].

• labels – (list[str] or None)

• color – a 3-tuple BGR color (in range [0, 255])

Returns:

np.ndarray – a new image.

tensorpack.utils.gpu module

tensorpack.utils.gpu.change_gpu(val)

Parameters:val – an integer, the index of the GPU or -1 to disable GPU. Returns:a context where CUDA_VISIBLE_DEVICES=val.

tensorpack.utils.gpu.get_num_gpu()

Returns:int – #available GPUs in CUDA_VISIBLE_DEVICES, or in the system.