mainXPK Start: Sat Apr 18 18:19:44 UTC 2026 2026-04-18 18:20:01.778247: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) I0418 18:20:05.368672 140522888795968 max_utils.py:273] Attempting to initialize the jax distributed system... INFO:2026-04-18 18:20:14,407:jax._src.distributed:149: Starting JAX distributed service on [::]:8482 I0418 18:20:14.407491 140522888795968 distributed.py:149] Starting JAX distributed service on [::]:8482 INFO:2026-04-18 18:20:14,409:jax._src.distributed:166: Connecting to JAX distributed service on mt-07-distill-smoke-zzr31-slice-job-0-0.mt-07-distill-smoke-zzr31:8482 I0418 18:20:14.409733 140522888795968 distributed.py:166] Connecting to JAX distributed service on mt-07-distill-smoke-zzr31-slice-job-0-0.mt-07-distill-smoke-zzr31:8482 I0418 18:20:15.501716 140522888795968 max_utils.py:284] Jax distributed system initialized! Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/deps/src/maxtext/trainers/post_train/distillation/train_distill.py", line 765, in <module> app.run(main) File "/usr/local/lib/python3.12/site-packages/absl/app.py", line 316, in run _run_main(main, args) File "/usr/local/lib/python3.12/site-packages/absl/app.py", line 261, in _run_main sys.exit(main(argv)) ^^^^^^^^^^ File "/deps/src/maxtext/trainers/post_train/distillation/train_distill.py", line 740, in main student_config = pyconfig.initialize(argv, **student_overrides) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/deps/src/maxtext/configs/pyconfig.py", line 294, in initialize pydantic_config = initialize_pydantic(argv, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/deps/src/maxtext/configs/pyconfig.py", line 343, in initialize_pydantic validate_no_keys_overridden_twice(model_loaded_cfg.keys(), overrides_cfg.keys()) File "/deps/src/maxtext/configs/pyconfig.py", line 99, in validate_no_keys_overridden_twice raise ValueError( ValueError: Keys ['vocab_size'] are overridden by both model config and CLI/kwargs.This is not allowed, unless setting `override_model_config=True`. XPK End: Sat Apr 18 18:20:30 UTC 2026 EXIT_CODE=1
XPK Start: Sat Apr 18 18:44:48 UTC 2026 2026-04-18 18:45:04.967443: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303) I0418 18:45:08.540647 132874013833024 max_utils.py:273] Attempting to initialize the jax distributed system... INFO:2026-04-18 18:45:17,579:jax._src.distributed:149: Starting JAX distributed service on [::]:8482 I0418 18:45:17.579999 132874013833024 distributed.py:149] Starting JAX distributed service on [::]:8482 INFO:2026-04-18 18:45:17,582:jax._src.distributed:166: Connecting to JAX distributed service on mt-07-distill-smoke-o6xh3-slice-job-0-0.mt-07-distill-smoke-o6xh3:8482 I0418 18:45:17.582274 132874013833024 distributed.py:166] Connecting to JAX distributed service on mt-07-distill-smoke-o6xh3-slice-job-0-0.mt-07-distill-smoke-o6xh3:8482 I0418 18:45:18.439979 132874013833024 max_utils.py:284] Jax distributed system initialized! Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/deps/src/maxtext/trainers/post_train/distillation/train_distill.py", line 765, in <module> app.run(main) File "/usr/local/lib/python3.12/site-packages/absl/app.py", line 316, in run _run_main(main, args) File "/usr/local/lib/python3.12/site-packages/absl/app.py", line 261, in _run_main sys.exit(main(argv)) ^^^^^^^^^^ File "/deps/src/maxtext/trainers/post_train/distillation/train_distill.py", line 740, in main student_config = pyconfig.initialize(argv, **student_overrides) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/deps/src/maxtext/configs/pyconfig.py", line 294, in initialize pydantic_config = initialize_pydantic(argv, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/deps/src/maxtext/configs/pyconfig.py", line 343, in initialize_pydantic validate_no_keys_overridden_twice(model_loaded_cfg.keys(), overrides_cfg.keys()) File "/deps/src/maxtext/configs/pyconfig.py", line 99, in validate_no_keys_overridden_twice raise ValueError( ValueError: Keys ['vocab_size'] are overridden by both model config and CLI/kwargs.This is not allowed, unless setting `override_model_config=True`. XPK End: Sat Apr 18 18:45:33 UTC 2026 EXIT_CODE=1