XPK Start: Wed Apr 22 10:06:30 UTC 2026
PyTorch was not found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2026-04-22 10:06:55.195041: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)
I0422 10:06:55.375699 133487101175616 max_utils.py:273] Attempting to initialize the jax distributed system...
I0422 10:07:04.417870 133487101175616 distributed.py:149] Starting JAX distributed service on [::]:8482
I0422 10:07:04.420285 133487101175616 distributed.py:172] Connecting to JAX distributed service on mt-10-shardy-false-uby0v-slice-job-0-0.mt-10-shardy-false-uby0v:8482
F0422 10:12:09.422635 10 client.h:80] Terminating process because the JAX distributed service detected fatal errors. This most likely indicates that another task died; see the other task logs for more details. Disable Python buffering, i.e. `python -u`, to be sure to see all the previous output. absl::Status: DEADLINE_EXCEEDED: Deadline Exceeded
RPC: /tensorflow.CoordinationService/RegisterTask
*** Check failure stack trace: ***
@ 0x796734d58464 absl::lts_20250814::log_internal::LogMessage::SendToLog()
@ 0x796734d583e6 absl::lts_20250814::log_internal::LogMessage::Flush()
@ 0x796734dafa3e std::_Function_handler<>::_M_invoke()
@ 0x7967315989d5 std::_Function_handler<>::_M_invoke()
@ 0x7967315b2e31 xla::CoordinationServiceAgent::SetError()
@ 0x7967315b27a2 xla::CoordinationServiceAgent::Connect()
@ 0x79673159743a xla::DistributedRuntimeCoordinationServiceClient::Connect()
@ 0x796734dab413 nanobind::detail::func_create<>()::{lambda()#1}::__invoke()
@ 0x7967314d600d nanobind::detail::nb_func_vectorcall_simple_1()
@ 0x7967e394f501 PyObject_Vectorcall
Fatal Python error: Aborted
Current thread 0x00007967e33f6740 (most recent call first):
File "/usr/local/lib/python3.12/site-packages/jax/_src/distributed.py", line 173 in initialize
File "/usr/local/lib/python3.12/site-packages/jax/_src/distributed.py", line 331 in initialize
File "/deps/src/maxtext/utils/max_utils.py", line 275 in maybe_initialize_jax_distributed_system
File "/deps/src/maxtext/configs/pyconfig.py", line 420 in initialize_pydantic
File "/deps/src/maxtext/configs/pyconfig.py", line 294 in initialize
File "/deps/src/maxtext/trainers/pre_train/train.py", line 768 in initialize
File "/deps/src/maxtext/trainers/pre_train/train.py", line 839 in main
File "/usr/local/lib/python3.12/site-packages/absl/app.py", line 312 in _run_main
File "/usr/local/lib/python3.12/site-packages/absl/app.py", line 367 in run
File "/deps/src/maxtext/trainers/pre_train/train.py", line 847 in <module>
File "<frozen runpy>", line 88 in _run_code
File "<frozen runpy>", line 198 in _run_module_as_main
Extension modules: numpy._core._multiarray_umath, numpy._core._multiarray_tests, numpy.linalg._umath_linalg, jaxlib.cpu_feature_guard, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, zstandard.backend_c, uvloop.loop, msgpack._cmsgpack, yaml._yaml, simplejson._speedups, google._upb._message, charset_normalizer.md, charset_normalizer.cd, requests.packages.charset_normalizer.md, requests.packages.chardet.md, requests.packages.charset_normalizer.cd, requests.packages.chardet.cd, _cffi_backend, grpc._cython.cygrpc, multidict._multidict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenlist, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.utils, h5py.h5t, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5r, h5py._npystrings, h5py._proxy, h5py._conv, h5py.h5z, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5g, h5py.h5i, h5py.h5o, h5py.h5f, h5py.h5fd, h5py.h5pl, h5py.h5l, h5py._selector, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, PIL._imaging, psutil._psutil_linux, pyarrow.lib, pandas._libs._cyutility, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.properties, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pyarrow._compute, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, kiwisolver._cext, markupsafe._speedups, regex._regex, sentencepiece._sentencepiece, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._batched_linalg, scipy.linalg._decomp_lu_cython, scipy.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.special._ufuncs_cxx, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.spatial._hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._propack, scipy.optimize._group_columns, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.linalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.optimize._direct, scipy.integrate._odepack, scipy.integrate._quadpack, scipy.integrate._vode, scipy.integrate._dop, scipy.interpolate._fitpack, scipy.interpolate._dfitpack, scipy.interpolate._dierckx, scipy.interpolate._ppoly, scipy.interpolate._interpnd, scipy.interpolate._rbfinterp_pythran, scipy.interpolate._rgi_cython, scipy.special.cython_special, scipy.stats._stats, scipy.stats._biasedurn, scipy.stats._stats_pythran, scipy.stats._levy_stable.levyst, scipy.stats._ansari_swilk_statistics, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.stats._sobol, scipy.stats._qmc_cy, scipy.stats._rcont.rcont, scipy.stats._qmvnt_cy, scipy.ndimage._nd_image, scipy.ndimage._rank_filter_1d, _ni_label, scipy.ndimage._ni_label (total: 191)
bash: line 5: 10 Aborted (core dumped) python3 -m maxtext.trainers.pre_train.train src/maxtext/configs/base.yml pure_nnx=False pure_nnx_decoder=False enable_nnx=False base_output_directory=gs://lance-maxtext/linen_ckpt_xpk_feat_nnx_trainstate_and_training_loop_20260422_093106 dataset_type=synthetic steps=10 learning_rate_schedule_steps=10 enable_checkpointing=True checkpoint_period=10 enable_goodput_recording=False per_device_batch_size=1 ici_fsdp_parallelism=32 log_config=False tokenizer_path=src/maxtext/assets/tokenizers/tokenizer.llama2 use_vertex_tensorboard=False vertex_tensorboard_project='' vertex_tensorboard_region='' enable_pathways_goodput=False enable_gcp_goodput_metrics=False monitor_goodput=False attention=dot_product profiler=xplane debug_sharding=True run_name=linen_xpk_feat_nnx_trainstate_and_training_loop_20260422_093106_10_shardy_false shardy=False
XPK End: Wed Apr 22 10:12:15 UTC 2026
EXIT_CODE=134