Restoring GUI state... qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 13018, resource id: 9132931, major code: 40 (TranslateCoords), minor code: 0 Saving config: /home/shuyu/.sleap/1.2.9/preferences.yaml qt.qpa.xcb: QXcbConnection: XCB error: 3 (BadWindow), sequence: 29841, resource id: 9229676, major code: 40 (TranslateCoords), minor code: 0 Using already trained model for centroid: /home/shuyu/SLEAP/2023-01-20-sleap-topdown/models/exp20_230120_193352.centroid.n=1855/training_config.json Using already trained model for centered_instance: /home/shuyu/SLEAP/2023-01-20-sleap-topdown/models/exp20_230121_041700.centered_instance.n=1855/training_config.json Command line call: sleap-track /home/shuyu/SLEAP/2023-01-23-sleap-topdown-test/labels.v000.e13-22_11_20-SR-MF3-reunion.topdown.slp --video.index 0 --frames 0,-110499 -m /home/shuyu/SLEAP/2023-01-20-sleap-topdown/models/exp20_230120_193352.centroid.n=1855/training_config.json -m /home/shuyu/SLEAP/2023-01-20-sleap-topdown/models/exp20_230121_041700.centered_instance.n=1855/training_config.json --tracking.tracker simple --tracking.target_instance_count 2 --tracking.pre_cull_to_target 1 --tracking.pre_cull_iou_threshold 0.8 --tracking.similarity centroid --tracking.match hungarian --tracking.track_window 5 --tracking.post_connect_single_breaks 0 -o /home/shuyu/SLEAP/2023-01-23-sleap-topdown-test/predictions/labels.v000.e13-22_11_20-SR-MF3-reunion.topdown.slp.230123_095540.predictions.slp --verbosity json --no-empty-frames INFO:numexpr.utils:NumExpr defaulting to 6 threads. Started inference at: 2023-01-23 09:55:43.601522 Args: 2023-01-23 09:55:43.688342: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero { 2023-01-23 09:55:43.780057: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-01-23 09:55:43.780845: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero │ 'data_path': '/home/shuyu/SLEAP/2023-01-23-sleap-topdown-test/labels.v000.e13-22_11_20-SR-MF3-reunion.topdown.slp', │ 'models': [ │ │ '/home/shuyu/SLEAP/2023-01-20-sleap-topdown/models/exp20_230120_193352.centroid.n=1855/training_config.json', │ │ '/home/shuyu/SLEAP/2023-01-20-sleap-topdown/models/exp20_230121_041700.centered_instance.n=1855/training_config.json' │ ], │ 'frames': '0,-110499', │ 'only_labeled_frames': False, │ 'only_suggested_frames': False, │ 'output': '/home/shuyu/SLEAP/2023-01-23-sleap-topdown-test/predictions/labels.v000.e13-22_11_20-SR-MF3-reunion.topdown.slp.230123_095540.predictions.slp', │ 'no_empty_frames': True, │ 'verbosity': 'json', │ 'video.dataset': None, │ 'video.input_format': 'channels_last', │ 'video.index': '0', │ 'cpu': False, │ 'first_gpu': False, │ 'last_gpu': False, │ 'gpu': 'auto', │ 'max_edge_length_ratio': 0.25, │ 'dist_penalty_weight': 1.0, 2023-01-23 09:55:44.823238: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2023-01-23 09:55:44.823636: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-01-23 09:55:44.823842: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-01-23 09:55:44.823977: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero │ 'batch_size': 4, │ 'open_in_gui': False, │ 'peak_threshold': 0.2, │ 'tracking.tracker': 'simple', │ 'tracking.target_instance_count': 2, │ 'tracking.pre_cull_to_target': 1, │ 'tracking.pre_cull_iou_threshold': 0.8, │ 'tracking.post_connect_single_breaks': 0, │ 'tracking.clean_instance_count': None, │ 'tracking.clean_iou_threshold': None, │ 'tracking.similarity': 'centroid', │ 'tracking.match': 'hungarian', │ 'tracking.track_window': 5, │ 'tracking.min_new_track_points': None, │ 'tracking.min_match_points': None, 2023-01-23 09:55:45.629833: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-01-23 09:55:45.630051: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-01-23 09:55:45.630225: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero 2023-01-23 09:55:45.630590: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1510] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 9265 MB memory: -> device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5 │ 'tracking.img_scale': None, │ 'tracking.of_window_size': None, │ 'tracking.of_max_levels': None, │ 'tracking.save_shifted_instances': None, │ 'tracking.kf_node_indices': None, │ 'tracking.kf_init_frame_count': None } INFO:sleap.nn.inference:Auto-selected GPU 0 with 10620 MiB of free memory. 2023-01-23 09:55:46.560667: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 84934656 exceeds 10% of free system memory. 2023-01-23 09:55:46.603809: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 63700992 exceeds 10% of free system memory. 2023-01-23 09:55:47.477995: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2) 2023-01-23 09:55:50.532031: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -67 } dim { size: -68 } dim { size: -69 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 2080 Ti" frequency: 1545 num_cores: 68 environment { key: "architecture" value: "7.5" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 5767168 shared_memory_size_per_multiprocessor: 65536 memory_size: 9715646464 bandwidth: 616000000 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -70 } dim { size: -71 } dim { size: 1 } } } 2023-01-23 09:55:50.532288: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_UINT8 } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_UINT8 shape { dim { size: 4 } dim { size: 1536 } dim { size: 2048 } dim { size: 3 } } } inputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -2 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "CPU" vendor: "GenuineIntel" model: "110" frequency: 3699 num_cores: 6 environment { key: "cpu_instruction_set" value: "AVX SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2" } environment { key: "eigen" value: "3.3.90" } l1_cache_size: 32768 l2_cache_size: 262144 l3_cache_size: 9437184 memory_size: 268435456 } outputs { dtype: DT_FLOAT shape { dim { size: -2 } dim { size: -75 } dim { size: -76 } dim { size: 3 } } } 2023-01-23 09:55:50.535714: W tensorflow/core/grappler/costs/op_level_cost_estimator.cc:690] Error in PredictCost() for the op: op: "CropAndResize" attr { key: "T" value { type: DT_FLOAT } } attr { key: "extrapolation_value" value { f: 0 } } attr { key: "method" value { s: "bilinear" } } inputs { dtype: DT_FLOAT shape { dim { size: -125 } dim { size: -126 } dim { size: -127 } dim { size: 1 } } } inputs { dtype: DT_FLOAT shape { dim { size: -6 } dim { size: 4 } } } inputs { dtype: DT_INT32 shape { dim { size: -6 } } } inputs { dtype: DT_INT32 shape { dim { size: 2 } } } device { type: "GPU" vendor: "NVIDIA" model: "NVIDIA GeForce RTX 2080 Ti" frequency: 1545 num_cores: 68 environment { key: "architecture" value: "7.5" } environment { key: "cuda" value: "11020" } environment { key: "cudnn" value: "8100" } num_registers: 65536 l1_cache_size: 24576 l2_cache_size: 5767168 shared_memory_size_per_multiprocessor: 65536 memory_size: 9715646464 bandwidth: 616000000 } outputs { dtype: DT_FLOAT shape { dim { size: -6 } dim { size: -128 } dim { size: -129 } dim { size: 1 } } } 2023-01-23 09:55:51.331438: I tensorflow/stream_executor/cuda/cuda_dnn.cc:369] Loaded cuDNN version 8201 Versions: SLEAP: 1.2.9 TensorFlow: 2.6.3 Numpy: 1.19.5 Python: 3.7.12 OS: Linux-5.4.0-132-generic-x86_64-with-debian-buster-sid System: GPUs: 1/1 available Device: /physical_device:GPU:0 Available: True Initalized: False Memory growth: True 2023-01-23 09:57:01.771408: W tensorflow/core/common_runtime/bfc_allocator.cc:338] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2023-01-23 09:57:01.907331: W tensorflow/core/common_runtime/bfc_allocator.cc:272] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.42GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2023-01-23 09:57:01.907397: W tensorflow/core/common_runtime/bfc_allocator.cc:272] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.42GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2023-01-23 09:57:02.020997: W tensorflow/core/common_runtime/bfc_allocator.cc:272] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.15GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2023-01-23 09:57:02.021021: W tensorflow/core/kernels/gpu_utils.cc:49] Failed to allocate memory for convolution redzone checking; skipping this check. This is benign and only means that we won't check cudnn for out-of-bounds reads and writes. This message will only be printed once. 2023-01-23 10:03:18.690853: W tensorflow/core/common_runtime/bfc_allocator.cc:272] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.68GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2023-01-23 10:03:18.690957: W tensorflow/core/common_runtime/bfc_allocator.cc:272] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.68GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2023-01-23 10:03:18.815421: W tensorflow/core/common_runtime/bfc_allocator.cc:272] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.39GiB with freed_by_count=0. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available. 2023-01-23 10:10:58.532401: W tensorflow/core/common_runtime/bfc_allocator.cc:338] Garbage collection: deallocate free memory regions (i.e., allocations) so that we can re-allocate a larger region to avoid OOM due to memory fragmentation. If you see this message frequently, you are running near the threshold of the available device memory and re-allocation may incur great performance overhead. You may try smaller batch sizes to observe the performance impact. Set TF_ENABLE_GPU_GARBAGE_COLLECTION=false if you'd like to disable this feature. 2023-01-23 10:11:09.032026: W tensorflow/core/common_runtime/bfc_allocator.cc:457] Allocator (GPU_0_bfc) ran out of memory trying to allocate 4.15GiB (rounded to 4459069440)requested by op top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu If the cause is memory fragmentation maybe the environment variable 'TF_GPU_ALLOCATOR=cuda_malloc_async' will improve the situation. Current allocation summary follows. Current allocation summary follows. 2023-01-23 10:11:09.032115: I tensorflow/core/common_runtime/bfc_allocator.cc:1004] BFCAllocator dump for GPU_0_bfc 2023-01-23 10:11:09.032170: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (256): Total Chunks: 100, Chunks in use: 99. 25.0KiB allocated for chunks. 24.8KiB in use in bin. 3.0KiB client-requested in use in bin. 2023-01-23 10:11:09.032213: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (512): Total Chunks: 14, Chunks in use: 13. 8.0KiB allocated for chunks. 7.2KiB in use in bin. 6.3KiB client-requested in use in bin. 2023-01-23 10:11:09.032248: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1024): Total Chunks: 11, Chunks in use: 11. 14.5KiB allocated for chunks. 14.5KiB in use in bin. 13.4KiB client-requested in use in bin. 2023-01-23 10:11:09.032283: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2048): Total Chunks: 8, Chunks in use: 8. 19.8KiB allocated for chunks. 19.8KiB in use in bin. 19.5KiB client-requested in use in bin. 2023-01-23 10:11:09.032321: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4096): Total Chunks: 6, Chunks in use: 5. 33.0KiB allocated for chunks. 28.5KiB in use in bin. 21.1KiB client-requested in use in bin. 2023-01-23 10:11:09.032357: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8192): Total Chunks: 3, Chunks in use: 2. 35.0KiB allocated for chunks. 20.8KiB in use in bin. 15.0KiB client-requested in use in bin. 2023-01-23 10:11:09.032390: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16384): Total Chunks: 2, Chunks in use: 2. 47.2KiB allocated for chunks. 47.2KiB in use in bin. 38.2KiB client-requested in use in bin. 2023-01-23 10:11:09.032431: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (32768): Total Chunks: 3, Chunks in use: 3. 132.8KiB allocated for chunks. 132.8KiB in use in bin. 112.5KiB client-requested in use in bin. 2023-01-23 10:11:09.032467: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (65536): Total Chunks: 3, Chunks in use: 2. 295.0KiB allocated for chunks. 214.0KiB in use in bin. 153.0KiB client-requested in use in bin. 2023-01-23 10:11:09.032499: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (131072): Total Chunks: 4, Chunks in use: 4. 594.0KiB allocated for chunks. 594.0KiB in use in bin. 558.0KiB client-requested in use in bin. 2023-01-23 10:11:09.032536: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (262144): Total Chunks: 4, Chunks in use: 4. 1.34MiB allocated for chunks. 1.34MiB in use in bin. 1.34MiB client-requested in use in bin. 2023-01-23 10:11:09.032569: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (524288): Total Chunks: 4, Chunks in use: 3. 2.83MiB allocated for chunks. 2.19MiB in use in bin. 1.76MiB client-requested in use in bin. 2023-01-23 10:11:09.032600: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (1048576): Total Chunks: 5, Chunks in use: 5. 6.99MiB allocated for chunks. 6.99MiB in use in bin. 6.29MiB client-requested in use in bin. 2023-01-23 10:11:09.032637: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (2097152): Total Chunks: 5, Chunks in use: 4. 14.75MiB allocated for chunks. 11.27MiB in use in bin. 10.83MiB client-requested in use in bin. 2023-01-23 10:11:09.032673: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (4194304): Total Chunks: 4, Chunks in use: 4. 26.38MiB allocated for chunks. 26.38MiB in use in bin. 21.38MiB client-requested in use in bin. 2023-01-23 10:11:09.032707: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (8388608): Total Chunks: 4, Chunks in use: 3. 45.70MiB allocated for chunks. 30.52MiB in use in bin. 28.12MiB client-requested in use in bin. 2023-01-23 10:11:09.032745: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (16777216): Total Chunks: 5, Chunks in use: 5. 109.97MiB allocated for chunks. 109.97MiB in use in bin. 82.69MiB client-requested in use in bin. 2023-01-23 10:11:09.032782: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (33554432): Total Chunks: 8, Chunks in use: 6. 316.62MiB allocated for chunks. 247.00MiB in use in bin. 236.25MiB client-requested in use in bin. 2023-01-23 10:11:09.032815: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (67108864): Total Chunks: 1, Chunks in use: 1. 81.00MiB allocated for chunks. 81.00MiB in use in bin. 81.00MiB client-requested in use in bin. 2023-01-23 10:11:09.032848: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin. 2023-01-23 10:11:09.032883: I tensorflow/core/common_runtime/bfc_allocator.cc:1011] Bin (268435456): Total Chunks: 4, Chunks in use: 1. 8.46GiB allocated for chunks. 4.15GiB in use in bin. 4.15GiB client-requested in use in bin. 2023-01-23 10:11:09.032917: I tensorflow/core/common_runtime/bfc_allocator.cc:1027] Bin for 4.15GiB was 256.00MiB, Chunk State: 2023-01-23 10:11:09.032960: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Size: 415.25MiB | Requested Size: 132.89MiB | in_use: 0 | bin_num: 20, prev: Size: 36.00MiB | Requested Size: 36.00MiB | in_use: 1 | bin_num: -1 2023-01-23 10:11:09.032998: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Size: 867.50MiB | Requested Size: 2.15GiB | in_use: 0 | bin_num: 20, prev: Size: 4.15GiB | Requested Size: 4.15GiB | in_use: 1 | bin_num: -1 2023-01-23 10:11:09.033030: I tensorflow/core/common_runtime/bfc_allocator.cc:1033] Size: 3.05GiB | Requested Size: 531.56MiB | in_use: 0 | bin_num: 20 2023-01-23 10:11:09.033057: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 5368709120 2023-01-23 10:11:09.033088: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7090000000 of size 4459069440 next 193 2023-01-23 10:11:09.033116: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f7199c80000 of size 909639680 next 18446744073709551615 2023-01-23 10:11:09.033142: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 3275292672 2023-01-23 10:11:09.033168: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f71d0000000 of size 3275292672 next 18446744073709551615 2023-01-23 10:11:09.033194: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 536870912 2023-01-23 10:11:09.033220: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f75f4000000 of size 63700992 next 139 2023-01-23 10:11:09.033247: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f75f7cc0000 of size 37748736 next 213 2023-01-23 10:11:09.033273: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f75fa0c0000 of size 435421184 next 18446744073709551615 2023-01-23 10:11:09.033298: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 268435456 2023-01-23 10:11:09.033324: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7614000000 of size 42467328 next 121 2023-01-23 10:11:09.033350: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7616880000 of size 84934656 next 135 2023-01-23 10:11:09.033376: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f761b980000 of size 21233664 next 143 2023-01-23 10:11:09.033401: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f761cdc0000 of size 15925248 next 145 2023-01-23 10:11:09.033428: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f761dcf0000 of size 26542080 next 142 2023-01-23 10:11:09.033453: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f761f640000 of size 37748736 next 192 2023-01-23 10:11:09.033479: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7621a40000 of size 39583744 next 18446744073709551615 2023-01-23 10:11:09.033506: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 134217728 2023-01-23 10:11:09.033532: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7624000000 of size 37748736 next 72 2023-01-23 10:11:09.033558: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7626400000 of size 37748736 next 70 2023-01-23 10:11:09.033584: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7628800000 of size 1327104 next 148 2023-01-23 10:11:09.033609: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f7628944000 of size 3649536 next 153 2023-01-23 10:11:09.033635: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7628cbf000 of size 2985984 next 123 2023-01-23 10:11:09.033661: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7628f98000 of size 27426816 next 77 2023-01-23 10:11:09.033687: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f762a9c0000 of size 23330816 next 18446744073709551615 2023-01-23 10:11:09.033714: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 67108864 2023-01-23 10:11:09.033741: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f762c000000 of size 5308416 next 144 2023-01-23 10:11:09.033767: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f762c510000 of size 3981312 next 146 2023-01-23 10:11:09.033793: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f762c8dc000 of size 10616832 next 149 2023-01-23 10:11:09.033820: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f762d2fc000 of size 11943936 next 130 2023-01-23 10:11:09.033846: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f762de60000 of size 35258368 next 18446744073709551615 2023-01-23 10:11:09.033872: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 33554432 2023-01-23 10:11:09.033899: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7630000000 of size 1769472 next 80 2023-01-23 10:11:09.033923: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f76301b0000 of size 7667712 next 62 2023-01-23 10:11:09.033950: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7630900000 of size 9437184 next 60 2023-01-23 10:11:09.033976: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7631200000 of size 7077888 next 75 2023-01-23 10:11:09.034002: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f76318c0000 of size 7602176 next 18446744073709551615 2023-01-23 10:11:09.034027: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 8388608 2023-01-23 10:11:09.034053: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7632000000 of size 589824 next 79 2023-01-23 10:11:09.034078: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7632090000 of size 442368 next 81 2023-01-23 10:11:09.034104: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f76320fc000 of size 165888 next 102 2023-01-23 10:11:09.034129: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7632124800 of size 331776 next 109 2023-01-23 10:11:09.034155: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7632175800 of size 681984 next 84 2023-01-23 10:11:09.034181: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f763221c000 of size 1327104 next 51 2023-01-23 10:11:09.034206: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7632360000 of size 2359296 next 50 2023-01-23 10:11:09.034232: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f76325a0000 of size 2490368 next 18446744073709551615 2023-01-23 10:11:09.034257: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 16777216 2023-01-23 10:11:09.034283: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f7632800000 of size 16777216 next 18446744073709551615 2023-01-23 10:11:09.034308: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 2097152 2023-01-23 10:11:09.034334: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600000 of size 1280 next 1 2023-01-23 10:11:09.034360: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600500 of size 256 next 2 2023-01-23 10:11:09.034386: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600600 of size 256 next 3 2023-01-23 10:11:09.034427: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600700 of size 256 next 4 2023-01-23 10:11:09.034450: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600800 of size 256 next 5 2023-01-23 10:11:09.034475: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600900 of size 256 next 8 2023-01-23 10:11:09.034499: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600a00 of size 256 next 9 2023-01-23 10:11:09.034521: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600b00 of size 256 next 10 2023-01-23 10:11:09.034543: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600c00 of size 256 next 13 2023-01-23 10:11:09.034567: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600d00 of size 256 next 14 2023-01-23 10:11:09.034592: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600e00 of size 256 next 17 2023-01-23 10:11:09.034617: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f600f00 of size 256 next 18 2023-01-23 10:11:09.034642: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601000 of size 256 next 19 2023-01-23 10:11:09.034666: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601100 of size 256 next 20 2023-01-23 10:11:09.034691: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601200 of size 256 next 23 2023-01-23 10:11:09.034716: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601300 of size 256 next 24 2023-01-23 10:11:09.034742: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601400 of size 256 next 27 2023-01-23 10:11:09.034767: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601500 of size 256 next 6 2023-01-23 10:11:09.034791: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601600 of size 256 next 98 2023-01-23 10:11:09.034818: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601700 of size 512 next 34 2023-01-23 10:11:09.034845: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601900 of size 1024 next 7 2023-01-23 10:11:09.034894: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601d00 of size 256 next 28 2023-01-23 10:11:09.034935: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601e00 of size 256 next 29 2023-01-23 10:11:09.034960: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f601f00 of size 256 next 32 2023-01-23 10:11:09.034984: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602000 of size 256 next 33 2023-01-23 10:11:09.035010: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602100 of size 512 next 36 2023-01-23 10:11:09.035034: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602300 of size 256 next 37 2023-01-23 10:11:09.035058: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602400 of size 256 next 38 2023-01-23 10:11:09.035084: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602500 of size 512 next 42 2023-01-23 10:11:09.035109: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602700 of size 256 next 43 2023-01-23 10:11:09.035134: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602800 of size 256 next 44 2023-01-23 10:11:09.035159: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602900 of size 1024 next 45 2023-01-23 10:11:09.035184: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602d00 of size 256 next 48 2023-01-23 10:11:09.035207: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602e00 of size 256 next 49 2023-01-23 10:11:09.035231: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f602f00 of size 1024 next 52 2023-01-23 10:11:09.035256: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f603300 of size 256 next 53 2023-01-23 10:11:09.035281: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f603400 of size 256 next 54 2023-01-23 10:11:09.035307: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f603500 of size 2048 next 56 2023-01-23 10:11:09.035332: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f603d00 of size 256 next 57 2023-01-23 10:11:09.035357: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f603e00 of size 256 next 59 2023-01-23 10:11:09.035382: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f603f00 of size 2048 next 63 2023-01-23 10:11:09.035406: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f604700 of size 256 next 64 2023-01-23 10:11:09.035431: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f604800 of size 256 next 65 2023-01-23 10:11:09.035455: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f604900 of size 7168 next 12 2023-01-23 10:11:09.035482: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f606500 of size 2048 next 58 2023-01-23 10:11:09.035507: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f606d00 of size 7168 next 11 2023-01-23 10:11:09.035531: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f608900 of size 256 next 67 2023-01-23 10:11:09.035555: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f608a00 of size 256 next 68 2023-01-23 10:11:09.035580: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f608b00 of size 2048 next 73 2023-01-23 10:11:09.035605: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609300 of size 1024 next 39 2023-01-23 10:11:09.035630: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609700 of size 512 next 47 2023-01-23 10:11:09.035653: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609900 of size 256 next 83 2023-01-23 10:11:09.035677: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609a00 of size 256 next 89 2023-01-23 10:11:09.035700: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609b00 of size 256 next 85 2023-01-23 10:11:09.035723: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609c00 of size 256 next 88 2023-01-23 10:11:09.035746: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609d00 of size 256 next 82 2023-01-23 10:11:09.035770: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609e00 of size 256 next 76 2023-01-23 10:11:09.035793: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f609f00 of size 256 next 22 2023-01-23 10:11:09.035816: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a000 of size 256 next 15 2023-01-23 10:11:09.035838: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a100 of size 256 next 100 2023-01-23 10:11:09.035862: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a200 of size 256 next 101 2023-01-23 10:11:09.035885: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a300 of size 256 next 104 2023-01-23 10:11:09.035908: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a400 of size 256 next 106 2023-01-23 10:11:09.035931: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a500 of size 256 next 107 2023-01-23 10:11:09.035954: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a600 of size 512 next 108 2023-01-23 10:11:09.035978: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60a800 of size 512 next 111 2023-01-23 10:11:09.036001: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60aa00 of size 256 next 113 2023-01-23 10:11:09.036024: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60ab00 of size 256 next 114 2023-01-23 10:11:09.036050: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60ac00 of size 768 next 115 2023-01-23 10:11:09.036073: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60af00 of size 768 next 118 2023-01-23 10:11:09.036096: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60b200 of size 256 next 119 2023-01-23 10:11:09.036119: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60b300 of size 256 next 120 2023-01-23 10:11:09.036144: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60b400 of size 1792 next 90 2023-01-23 10:11:09.036167: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60bb00 of size 256 next 92 2023-01-23 10:11:09.036190: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60bc00 of size 256 next 93 2023-01-23 10:11:09.036213: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60bd00 of size 256 next 94 2023-01-23 10:11:09.036236: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60be00 of size 256 next 95 2023-01-23 10:11:09.036259: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60bf00 of size 256 next 96 2023-01-23 10:11:09.036282: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60c000 of size 1792 next 97 2023-01-23 10:11:09.036307: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f60c700 of size 36864 next 16 2023-01-23 10:11:09.036332: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f615700 of size 94720 next 21 2023-01-23 10:11:09.036355: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62c900 of size 256 next 162 2023-01-23 10:11:09.036378: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62ca00 of size 512 next 110 2023-01-23 10:11:09.036402: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62cc00 of size 768 next 112 2023-01-23 10:11:09.036425: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62cf00 of size 1280 next 69 2023-01-23 10:11:09.036448: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62d400 of size 256 next 134 2023-01-23 10:11:09.036471: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62d500 of size 256 next 137 2023-01-23 10:11:09.036495: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62d600 of size 3072 next 140 2023-01-23 10:11:09.036520: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62e200 of size 1536 next 128 2023-01-23 10:11:09.036543: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62e800 of size 512 next 116 2023-01-23 10:11:09.036567: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62ea00 of size 4352 next 154 2023-01-23 10:11:09.036592: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62fb00 of size 256 next 182 2023-01-23 10:11:09.036615: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62fc00 of size 256 next 183 2023-01-23 10:11:09.036638: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62fd00 of size 256 next 184 2023-01-23 10:11:09.036662: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62fe00 of size 256 next 185 2023-01-23 10:11:09.039811: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f62ff00 of size 256 next 186 2023-01-23 10:11:09.039866: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f765f630000 of size 256 next 188 2023-01-23 10:11:09.039891: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f630100 of size 256 next 199 2023-01-23 10:11:09.039913: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f630200 of size 256 next 200 2023-01-23 10:11:09.039940: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f765f630300 of size 4608 next 208 2023-01-23 10:11:09.039964: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f631500 of size 512 next 201 2023-01-23 10:11:09.039988: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f631700 of size 512 next 207 2023-01-23 10:11:09.040012: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f765f631900 of size 768 next 156 2023-01-23 10:11:09.040037: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f631c00 of size 256 next 157 2023-01-23 10:11:09.040060: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f631d00 of size 256 next 158 2023-01-23 10:11:09.040084: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f631e00 of size 256 next 159 2023-01-23 10:11:09.040115: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f631f00 of size 2816 next 160 2023-01-23 10:11:09.040141: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f632a00 of size 1536 next 117 2023-01-23 10:11:09.040165: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f633000 of size 4352 next 161 2023-01-23 10:11:09.040190: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634100 of size 256 next 151 2023-01-23 10:11:09.040214: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634200 of size 256 next 150 2023-01-23 10:11:09.040238: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634300 of size 256 next 122 2023-01-23 10:11:09.040263: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634400 of size 256 next 147 2023-01-23 10:11:09.040287: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634500 of size 256 next 141 2023-01-23 10:11:09.040311: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634600 of size 256 next 164 2023-01-23 10:11:09.040336: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634700 of size 256 next 165 2023-01-23 10:11:09.040360: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634800 of size 256 next 166 2023-01-23 10:11:09.040384: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634900 of size 256 next 167 2023-01-23 10:11:09.040409: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634a00 of size 256 next 168 2023-01-23 10:11:09.040433: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634b00 of size 256 next 169 2023-01-23 10:11:09.040458: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634c00 of size 256 next 170 2023-01-23 10:11:09.040482: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634d00 of size 256 next 171 2023-01-23 10:11:09.040506: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634e00 of size 256 next 172 2023-01-23 10:11:09.040530: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f634f00 of size 256 next 173 2023-01-23 10:11:09.040554: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635000 of size 256 next 174 2023-01-23 10:11:09.040579: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635100 of size 256 next 175 2023-01-23 10:11:09.040603: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635200 of size 256 next 176 2023-01-23 10:11:09.040627: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635300 of size 256 next 177 2023-01-23 10:11:09.040652: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635400 of size 256 next 178 2023-01-23 10:11:09.040676: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635500 of size 256 next 179 2023-01-23 10:11:09.040700: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635600 of size 256 next 180 2023-01-23 10:11:09.040724: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635700 of size 256 next 181 2023-01-23 10:11:09.040749: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635800 of size 256 next 91 2023-01-23 10:11:09.040774: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f635900 of size 9216 next 99 2023-01-23 10:11:09.040800: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f637d00 of size 27648 next 26 2023-01-23 10:11:09.040825: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f63e900 of size 36864 next 86 2023-01-23 10:11:09.040850: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f647900 of size 1536 next 124 2023-01-23 10:11:09.040874: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f647f00 of size 256 next 125 2023-01-23 10:11:09.040898: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f648000 of size 256 next 126 2023-01-23 10:11:09.040923: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f648100 of size 3072 next 127 2023-01-23 10:11:09.040947: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f648d00 of size 3072 next 129 2023-01-23 10:11:09.040971: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f649900 of size 256 next 131 2023-01-23 10:11:09.040995: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f649a00 of size 256 next 132 2023-01-23 10:11:09.041021: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f649b00 of size 12032 next 41 2023-01-23 10:11:09.041046: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f64ca00 of size 6144 next 136 2023-01-23 10:11:09.041071: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f765f64e200 of size 14592 next 74 2023-01-23 10:11:09.041097: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f651b00 of size 20736 next 163 2023-01-23 10:11:09.041122: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f656c00 of size 62208 next 103 2023-01-23 10:11:09.041147: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f665f00 of size 124416 next 105 2023-01-23 10:11:09.041172: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f765f684500 of size 82944 next 31 2023-01-23 10:11:09.041197: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f698900 of size 147456 next 30 2023-01-23 10:11:09.041222: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f6bc900 of size 147456 next 87 2023-01-23 10:11:09.041247: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f6e0900 of size 147456 next 35 2023-01-23 10:11:09.041273: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f704900 of size 1029888 next 18446744073709551615 2023-01-23 10:11:09.041300: I tensorflow/core/common_runtime/bfc_allocator.cc:1040] Next region of size 4194304 2023-01-23 10:11:09.041327: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f800000 of size 294912 next 25 2023-01-23 10:11:09.041352: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f848000 of size 1769472 next 78 2023-01-23 10:11:09.041377: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765f9f8000 of size 331776 next 152 2023-01-23 10:11:09.041401: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] Free at 7f765fa49000 of size 663552 next 155 2023-01-23 10:11:09.041427: I tensorflow/core/common_runtime/bfc_allocator.cc:1060] InUse at 7f765faeb000 of size 1134592 next 18446744073709551615 2023-01-23 10:11:09.041451: I tensorflow/core/common_runtime/bfc_allocator.cc:1065] Summary of in-use Chunks by size: 2023-01-23 10:11:09.041488: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 99 Chunks of size 256 totalling 24.8KiB 2023-01-23 10:11:09.041519: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 10 Chunks of size 512 totalling 5.0KiB 2023-01-23 10:11:09.041546: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 3 Chunks of size 768 totalling 2.2KiB 2023-01-23 10:11:09.041572: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 4 Chunks of size 1024 totalling 4.0KiB 2023-01-23 10:11:09.041599: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 1280 totalling 2.5KiB 2023-01-23 10:11:09.041626: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 3 Chunks of size 1536 totalling 4.5KiB 2023-01-23 10:11:09.041652: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 1792 totalling 3.5KiB 2023-01-23 10:11:09.041679: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 4 Chunks of size 2048 totalling 8.0KiB 2023-01-23 10:11:09.041706: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 2816 totalling 2.8KiB 2023-01-23 10:11:09.041733: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 3 Chunks of size 3072 totalling 9.0KiB 2023-01-23 10:11:09.041760: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 4352 totalling 8.5KiB 2023-01-23 10:11:09.041787: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 6144 totalling 6.0KiB 2023-01-23 10:11:09.041814: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 7168 totalling 14.0KiB 2023-01-23 10:11:09.041842: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 9216 totalling 9.0KiB 2023-01-23 10:11:09.041869: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 12032 totalling 11.8KiB 2023-01-23 10:11:09.041897: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 20736 totalling 20.2KiB 2023-01-23 10:11:09.041924: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 27648 totalling 27.0KiB 2023-01-23 10:11:09.041952: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 36864 totalling 72.0KiB 2023-01-23 10:11:09.041979: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 62208 totalling 60.8KiB 2023-01-23 10:11:09.042007: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 94720 totalling 92.5KiB 2023-01-23 10:11:09.042035: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 124416 totalling 121.5KiB 2023-01-23 10:11:09.042063: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 3 Chunks of size 147456 totalling 432.0KiB 2023-01-23 10:11:09.042091: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 165888 totalling 162.0KiB 2023-01-23 10:11:09.042118: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 294912 totalling 288.0KiB 2023-01-23 10:11:09.042145: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 331776 totalling 648.0KiB 2023-01-23 10:11:09.042174: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 442368 totalling 432.0KiB 2023-01-23 10:11:09.042201: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 589824 totalling 576.0KiB 2023-01-23 10:11:09.042228: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 681984 totalling 666.0KiB 2023-01-23 10:11:09.042256: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 1029888 totalling 1005.8KiB 2023-01-23 10:11:09.042284: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 1134592 totalling 1.08MiB 2023-01-23 10:11:09.042311: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 1327104 totalling 2.53MiB 2023-01-23 10:11:09.042338: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 2 Chunks of size 1769472 totalling 3.38MiB 2023-01-23 10:11:09.042365: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 2359296 totalling 2.25MiB 2023-01-23 10:11:09.042391: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 2490368 totalling 2.38MiB 2023-01-23 10:11:09.042417: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 2985984 totalling 2.85MiB 2023-01-23 10:11:09.042444: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 3981312 totalling 3.80MiB 2023-01-23 10:11:09.042471: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 5308416 totalling 5.06MiB 2023-01-23 10:11:09.042497: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 7077888 totalling 6.75MiB 2023-01-23 10:11:09.042524: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 7602176 totalling 7.25MiB 2023-01-23 10:11:09.042550: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 7667712 totalling 7.31MiB 2023-01-23 10:11:09.042578: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 9437184 totalling 9.00MiB 2023-01-23 10:11:09.042606: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 10616832 totalling 10.12MiB 2023-01-23 10:11:09.042634: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 11943936 totalling 11.39MiB 2023-01-23 10:11:09.042662: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 16777216 totalling 16.00MiB 2023-01-23 10:11:09.042689: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 21233664 totalling 20.25MiB 2023-01-23 10:11:09.042717: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 23330816 totalling 22.25MiB 2023-01-23 10:11:09.042745: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 26542080 totalling 25.31MiB 2023-01-23 10:11:09.042773: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 27426816 totalling 26.16MiB 2023-01-23 10:11:09.042802: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 3 Chunks of size 37748736 totalling 108.00MiB 2023-01-23 10:11:09.042830: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 39583744 totalling 37.75MiB 2023-01-23 10:11:09.042858: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 42467328 totalling 40.50MiB 2023-01-23 10:11:09.042908: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 63700992 totalling 60.75MiB 2023-01-23 10:11:09.042938: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 84934656 totalling 81.00MiB 2023-01-23 10:11:09.042965: I tensorflow/core/common_runtime/bfc_allocator.cc:1068] 1 Chunks of size 4459069440 totalling 4.15GiB 2023-01-23 10:11:09.042991: I tensorflow/core/common_runtime/bfc_allocator.cc:1072] Sum Total of in-use chunks: 4.66GiB 2023-01-23 10:11:09.043017: I tensorflow/core/common_runtime/bfc_allocator.cc:1074] total_region_allocated_bytes_: 9715646464 memory_limit_: 9715646464 available bytes: 0 curr_region_allocation_bytes_: 34359738368 2023-01-23 10:11:09.043053: I tensorflow/core/common_runtime/bfc_allocator.cc:1080] Stats: Limit: 9715646464 InUse: 5001944320 MaxInUse: 6817332480 NumAllocs: 737626 MaxAllocSize: 4459069440 Reserved: 0 PeakReserved: 0 LargestFreeBlock: 0 2023-01-23 10:11:09.043149: W tensorflow/core/common_runtime/bfc_allocator.cc:468] **********************************************__________________________________________***___****** 2023-01-23 10:11:09.043237: W tensorflow/core/framework/op_kernel.cc:1692] OP_REQUIRES failed at conv_ops_fused_impl.h:846 : Resource exhausted: OOM when allocating tensor with shape[35,24,1152,1152] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/home/shuyu/anaconda3/envs/sleap/bin/sleap-track", line 33, in sys.exit(load_entry_point('sleap==1.2.9', 'console_scripts', 'sleap-track')()) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 4660, in main labels_pr = predictor.predict(provider) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 440, in predict self._make_labeled_frames_from_generator(generator, data) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 2315, in _make_labeled_frames_from_generator for ex in generator: File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 371, in _predict_generator ex = process_batch(ex) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 316, in process_batch preds = self.inference_model.predict_on_batch(ex, numpy=True) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/sleap/nn/inference.py", line 965, in predict_on_batch outs = super().predict_on_batch(data, **kwargs) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/keras/engine/training.py", line 1947, in predict_on_batch outputs = self.predict_function(iterator) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 885, in __call__ result = self._call(*args, **kwds) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 924, in _call results = self._stateful_fn(*args, **kwds) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 3040, in __call__ filtered_flat_args, captured_inputs=graph_function.captured_inputs) # pylint: disable=protected-access File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1964, in _call_flat ctx, args, cancellation_manager=cancellation_manager)) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 596, in call ctx=ctx) File "/home/shuyu/anaconda3/envs/sleap/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute inputs, attrs, num_outputs) tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found. (0) Resource exhausted: OOM when allocating tensor with shape[35,24,1152,1152] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu (defined at /lib/python3.7/site-packages/sleap/nn/inference.py:1868) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. [[top_down_inference_model/find_instance_peaks/PartitionedCall/Cast/_204]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. (1) Resource exhausted: OOM when allocating tensor with shape[35,24,1152,1152] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[node top_down_inference_model/find_instance_peaks/model/stack0_enc0_act1_relu/Relu (defined at /lib/python3.7/site-packages/sleap/nn/inference.py:1868) ]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode. 0 successful operations. 0 derived errors ignored. [Op:__inference_predict_function_3887] Function call stack: predict_function -> predict_function Process return code: 1