tensorpack.utils package¶
Common utils. These utils should be irrelevant to tensorflow.
-
tensorpack.utils.
change_env
(name, val)[source]¶ Parameters: name (str), val(str) – Returns: a context where the environment variable name
being set toval
. It will be set back after the context exits.
-
tensorpack.utils.
get_rng
(obj=None)[source]¶ Get a good RNG seeded with time, pid and the object.
Parameters: obj – some object to use to generate random seed. Returns: np.random.RandomState – the RNG.
-
tensorpack.utils.
fix_rng_seed
(seed)[source]¶ Call this function at the beginning of program to fix rng seed within tensorpack.
Parameters: seed (int) – Example
Fix random seed in both tensorpack and tensorflow.
import tensorpack.utils.utils as utils seed = 42 utils.fix_rng_seed(seed) tesnorflow.set_random_seed(seed) # run trainer
-
tensorpack.utils.
get_tqdm
(*args, **kwargs)[source]¶ Similar to
tqdm.tqdm()
, but use tensorpack’s default options to have consistent style.
-
tensorpack.utils.
execute_only_once
()[source]¶ Each called in the code to this function is guaranteed to return True the first time and False afterwards.
Returns: bool – whether this is the first time this function gets called from this line of code. Example
if execute_only_once(): # do something only once
-
tensorpack.utils.
humanize_time_delta
(sec)[source]¶ Humanize timedelta given in seconds
Parameters: sec (float) – time difference in seconds. Must be positive. Returns: str - time difference as a readable string Example:
print(humanize_time_delta(1)) # 1 second print(humanize_time_delta(60 + 1)) # 1 minute 1 second print(humanize_time_delta(87.6)) # 1 minute 27 seconds print(humanize_time_delta(0.01)) # 0.01 seconds print(humanize_time_delta(60 * 60 + 1)) # 1 hour 1 second print(humanize_time_delta(60 * 60 * 24 + 1)) # 1 day 1 second print(humanize_time_delta(60 * 60 * 24 + 60 * 2 + 60*60*9 + 3)) # 1 day 9 hours 2 minutes 3 seconds
tensorpack.utils.argtools module¶
-
tensorpack.utils.argtools.
map_arg
(**maps)[source]¶ Apply a mapping on certain argument before calling the original function.
Parameters: maps (dict) – {argument_name: map_func}
-
tensorpack.utils.argtools.
memoized
(user_function)¶ Alias to
functools.lru_cache()
WARNING: memoization will keep keys and values alive!
-
tensorpack.utils.argtools.
memoized_method
(func)[source]¶ A decorator that performs memoization on methods. It stores the cache on the object instance itself.
-
tensorpack.utils.argtools.
graph_memoized
(func)[source]¶ Like memoized, but keep one cache per default graph.
-
tensorpack.utils.argtools.
shape2d
(a)[source]¶ Ensure a 2D shape.
Parameters: a – a int or tuple/list of length 2 Returns: list – of length 2. if a
is a int, return[a, a]
.
-
tensorpack.utils.argtools.
shape4d
(a, data_format='channels_last')[source]¶ Ensuer a 4D shape, to use with 4D symbolic functions.
Parameters: a – a int or tuple/list of length 2 Returns: list – - of length 4. if
a
is a int, return[1, a, a, 1]
- or
[1, 1, a, a]
depending on data_format.
- of length 4. if
-
tensorpack.utils.argtools.
memoized_ignoreargs
(func)[source]¶ A decorator. It performs memoization ignoring the arguments used to call the function.
tensorpack.utils.concurrency module¶
-
class
tensorpack.utils.concurrency.
StoppableThread
(evt=None)[source]¶ Bases:
threading.Thread
A thread that has a ‘stop’ event.
-
__init__
(evt=None)[source]¶ Parameters: evt (threading.Event) – if None, will create one.
-
-
class
tensorpack.utils.concurrency.
LoopThread
(func, pausable=True)[source]¶ Bases:
tensorpack.utils.concurrency.StoppableThread
A pausable thread that simply runs a loop
-
run
()[source]¶ Method representing the thread’s activity.
You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.
-
Bases:
threading.Thread
A wrapper around thread so that the thread uses the default session at “start()” time.
Parameters: th (threading.Thread or None) –
Method representing the thread’s activity.
You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.
Start the thread’s activity.
It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.
This method will raise a RuntimeError if called more than once on the same thread object.
-
tensorpack.utils.concurrency.
ensure_proc_terminate
(proc)[source]¶ Make sure processes terminate when main process exit.
Parameters: proc (multiprocessing.Process or list) –
-
class
tensorpack.utils.concurrency.
OrderedResultGatherProc
(data_queue, nr_producer, start=0)[source]¶ Bases:
multiprocessing.context.Process
Gather indexed data from a data queue, and produce results with the original index-based order.
-
class
tensorpack.utils.concurrency.
OrderedContainer
(start=0)[source]¶ Bases:
object
Like a queue, but will always wait to receive item with rank (x+1) and produce (x+1) before producing (x+2).
Warning
It is not thread-safe.
-
class
tensorpack.utils.concurrency.
DIE
[source]¶ Bases:
object
A placeholder class indicating end of queue
tensorpack.utils.fs module¶
-
tensorpack.utils.fs.
mkdir_p
(dirname)[source]¶ Like “mkdir -p”, make a dir recursively, but do nothing if the dir exists
Parameters: dirname (str) –
-
tensorpack.utils.fs.
download
(url, dir, filename=None, expect_size=None)[source]¶ Download URL to a directory. Will figure out the filename automatically from URL, if not given.
tensorpack.utils.loadcaffe module¶
-
tensorpack.utils.loadcaffe.
load_caffe
(model_desc, model_file)[source]¶ Load a caffe model. You must be able to
import caffe
to use this function. :param model_desc: path to caffe model description file (.prototxt). :type model_desc: str :param model_file: path to caffe model parameter file (.caffemodel). :type model_file: strReturns: dict – the parameters.
tensorpack.utils.logger module¶
The logger module itself has the common logging functions of Python’s
logging.Logger
. For example:
from tensorpack.utils import logger
logger.set_logger_dir('train_log/test')
logger.info("Test")
logger.error("Error happened!")
-
tensorpack.utils.logger.
set_logger_dir
(dirname, action=None)[source]¶ Set the directory for global logging.
Parameters: dirname (str) – log directory
action (str) –
an action of [“k”,”d”,”q”] to be performed when the directory exists. Will ask user by default.
”d”: delete the directory. Note that the deletion may fail when the directory is used by tensorboard.
”k”: keep the directory. This is useful when you resume from a previous training and want the directory to look as if the training was not interrupted. Note that this option does not load old models or any other old states for you. It simply does nothing.
tensorpack.utils.serialize module¶
-
tensorpack.utils.serialize.
loads
(buf)¶ Parameters: buf – the output of dumps.
-
tensorpack.utils.serialize.
dumps
(obj)¶ Serialize an object.
Returns: Implementation-dependent bytes-like object.
tensorpack.utils.compatible_serialize module¶
-
tensorpack.utils.compatible_serialize.
loads
(buf)¶ Parameters: buf – the output of dumps.
-
tensorpack.utils.compatible_serialize.
dumps
(obj)¶ Serialize an object.
Returns: Implementation-dependent bytes-like object.
tensorpack.utils.stats module¶
-
class
tensorpack.utils.stats.
StatCounter
[source]¶ Bases:
object
A simple counter
-
average
¶
-
count
¶
-
max
¶
-
min
¶
-
sum
¶
-
-
class
tensorpack.utils.stats.
BinaryStatistics
[source]¶ Bases:
object
Statistics for binary decision, including precision, recall, false positive, false negative
-
false_negative
¶
-
false_positive
¶
-
feed
(pred, label)[source]¶ Parameters: pred (np.ndarray) – binary array.
label (np.ndarray) – binary array of the same size.
-
precision
¶
-
recall
¶
-
-
class
tensorpack.utils.stats.
RatioCounter
[source]¶ Bases:
object
A counter to count ratio of something.
-
count
¶ Returns – int: the total
-
ratio
¶
-
total
¶ Returns – int: the total
-
-
class
tensorpack.utils.stats.
Accuracy
[source]¶ Bases:
tensorpack.utils.stats.RatioCounter
A RatioCounter with a fancy name
-
accuracy
¶
-
tensorpack.utils.timer module¶
-
tensorpack.utils.timer.
total_timer
(msg)[source]¶ A context which add the time spent inside to TotalTimer.
-
tensorpack.utils.timer.
timed_operation
(msg, log_start=False)[source]¶ Surround a context with a timer.
Parameters: Example
with timed_operation('Good Stuff'): time.sleep(1)
Will print:
Good stuff finished, time:1sec.
-
tensorpack.utils.timer.
print_total_timer
()[source]¶ Print the content of the TotalTimer, if it’s not empty. This function will automatically get called when program exits.
-
class
tensorpack.utils.timer.
IterSpeedCounter
(print_every, name=None)[source]¶ Bases:
object
Test how often some code gets reached.
Example
Print the speed of the iteration every 100 times.
speed = IterSpeedCounter(100) for k in range(1000): # do something speed()
tensorpack.utils.viz module¶
-
tensorpack.utils.viz.
interactive_imshow
(img, lclick_cb=None, rclick_cb=None, **kwargs)[source]¶ Parameters: img (np.ndarray) – an image (expect BGR) to show.
rclick_cb (lclick_cb,) – a callback
func(img, x, y)
for left/right click event.kwargs – can be {key_cb_a: callback_img, key_cb_b: callback_img}, to specify a callback
func(img)
for keypress.
Some existing keypress event handler:
q: destroy the current window
x: execute
sys.exit()
s: save image to “out.png”
-
tensorpack.utils.viz.
stack_patches
(patch_list, nr_row, nr_col, border=None, pad=False, bgcolor=255, viz=False, lclick_cb=None)[source]¶ Stacked patches into grid, to produce visualizations like the following:
Parameters: patch_list (list[ndarray] or ndarray) – NHW or NHWC images in [0,255].
nr_row (int), nr_col(int) – rows and cols of the grid.
nr_col * nr_row
must be no less thanlen(patch_list)
.border (int) – border length between images. Defaults to
0.05 * min(patch_width, patch_height)
.pad (boolean) – when patch_list is a list, pad all patches to the maximum height and width. This option allows stacking patches of different shapes together.
bgcolor (int or 3-tuple) – background color in [0, 255]. Either an int or a BGR tuple.
viz (bool) – whether to use
interactive_imshow()
to visualize the results.lclick_cb – A callback function
f(patch, patch index in patch_list)
to get called when a patch get clicked in imshow.
Returns: np.ndarray – the stacked image.
-
tensorpack.utils.viz.
gen_stack_patches
(patch_list, nr_row=None, nr_col=None, border=None, max_width=1000, max_height=1000, bgcolor=255, viz=False, lclick_cb=None)[source]¶ Similar to
stack_patches()
but with a generator interface. It takes a much-longer list and yields stacked results one by one. For example, ifpatch_list
contains 1000 images andnr_row==nr_col==10
, this generator yields 10 stacked images.Parameters: max_width (int), max_height(int) – Maximum allowed size of the stacked image. If
nr_row/nr_col
are None, this number will be used to infer the rows and cols. Otherwise the option is ignored.border, viz, lclick_cb (patch_list,) – same as in
stack_patches()
.
Yields: np.ndarray – the stacked image.
-
tensorpack.utils.viz.
dump_dataflow_images
(df, index=0, batched=True, number=1000, output_dir=None, scale=1, resize=None, viz=None, flipRGB=False)[source]¶ Dump or visualize images of a
DataFlow
.Parameters: df (DataFlow) – the DataFlow.
index (int) – the index of the image component.
batched (bool) – whether the component contains batched images (NHW or NHWC) or not (HW or HWC).
number (int) – how many datapoint to take from the DataFlow.
output_dir (str) – output directory to save images, default to not save.
scale (float) – scale the value, usually either 1 or 255.
resize (tuple or None) – tuple of (h, w) to resize the images to.
viz (tuple or None) – tuple of (h, w) determining the grid size to use with
gen_stack_patches()
for visualization. No visualization will happen by default.flipRGB (bool) – apply a RGB<->BGR conversion or not.
-
tensorpack.utils.viz.
intensity_to_rgb
(intensity, cmap='cubehelix', normalize=False)[source]¶ Convert a 1-channel matrix of intensities to an RGB image employing a colormap. This function requires matplotlib. See matplotlib colormaps for a list of available colormap.
Parameters: Returns: np.ndarray – an RGB float32 image in range [0, 255], a colored heatmap.
-
tensorpack.utils.viz.
draw_boxes
(im, boxes, labels=None, color=None)[source]¶ Parameters: im (np.ndarray) – a BGR image in range [0,255]. It will not be modified.
boxes (np.ndarray) – a numpy array of shape Nx4 where each row is [x1, y1, x2, y2].
labels – (list[str] or None)
color – a 3-tuple (in range [0, 255]). By default will choose automatically.
Returns: np.ndarray – a new image.