ijCdZddlZddlZddlmZddlmZddlmZddZ ddZ dZ d Z d Z d Zd Zd ZdZdZdZy)z5Utilities for distribution strategy with JAX backend.N)seed_generator) jax_utils) rng_utilsc|r|jnd}tj|}|Dcgc]}|jd|jc}Scc}w)aReturn all the available devices based on the device type. Note that this should return the global devices in a distributed setting. Args: device_type: string of `"cpu"`, `"gpu"` or `"tpu"`. Defaults to `"gpu"` or `"tpu"` if available when device_type is not provided. Otherwise will return the `"cpu"` devices. Return: List of devices that are available for distribute computation. Nbackend:)lowerjaxdevicesplatformid) device_type jax_devicesdevices {/var/www/html/emotional.easysim.app/public_html/venv/lib/python3.12/site-packages/keras/src/backend/jax/distribution_lib.py list_devicesr sL*5+##%$K++k2K;F Gvq , GG Gs"AcT|r|jnd}tj|S)aReturns the number of available JAX devices. Args: device_type: Optional device type to count (e.g., "cpu", "gpu", "tpu"). If `None`, it defaults to counting "gpu" or "tpu" devices if available, otherwise it counts "cpu" devices. It does not return the sum of all device types. Returns: int: The total number of JAX devices for the specified type. N)r r device_count)rs rget_device_countrs'*5+##%$K   K ((c4ddlm}t||r |j}t j |r t jj||St|t jrt|t jjr2|jj|t|jr|St|drt!|dd}||k(r"|St|drt!|dd}||k(r|St j"||S)aDistribute the tensor based on the layout. Note that this function can be used both in eager context, or within a jitted function. Args: tensor: `jax.Array` that need to be distributed. layout: `TensorLayout` for the created variable, or a JAX-supported layout instance (e.g. `jax.sharding.Sharding`). Returns: Distributed value. r TensorLayout)ndimlayoutNformat)keras.src.distributionr isinstancebackend_layoutris_in_jax_tracing_scoper laxwith_sharding_constraintArrayshardingShardingis_equivalent_tolenshapehasattrgetattr device_put)tensorrrcurrent_layouts rdistribute_tensorr/+s4&,'&&((0ww//??&#))$  CLL)) oo..vC >&& ))rcjddlm}t||r |j}t j ||S)aDistribute the input data with the corresponding layout. Note that the inputs here is a local worker batch. Within the local worker, the data need to be further partitioned to map to each of the devices. Args: inputs: `jax.Array` that is already sharded to a local process size. layout: `TensorLayout` for the distribution information, or a `jax.sharding.Sharding` instance. Returns: A global batch distributed according to `layout`. rr)rrrr r "make_array_from_process_local_data)per_process_batchrbatch_dim_namers rdistribute_data_inputr4Vs04&,'&&  1 1&:K LLrctj}|tjd}tjd}tj j tjg|ztj j}tjdd||jd}tj|yy)zInitializes the global random number generator across processes. This is required for consistent initialization in multi-host settings. Ncpu)dtypecBtjj|dS)Nall)r r"psum)xs rz initialize_rng..scggll1e,rr9) axis_namer r) rget_random_seedr r local_device_countnumpyasarrayrmake_default_seeduint32pmapitemset_random_seed) global_seed cpu_devicesnum_local_cpu_devices local_seeds rinitialize_rngrKms ++-Kkk%( # 6 6u =YY&&  - - / 03H H))""'  chh ,   d1g  !!+.%rc|rHd|vrD|jd}|+|t|k7rtd|dt|d||d}n|}tjj |||t y)N,zThe provided job_addresses z has z jobs, but num_processes is r)coordinator_address num_processes process_id)splitr( ValueErrorr distributed initializerK) job_addressesrOrPrNs rrTrTs -&++C0   $#m:L)L-m_E}%&&B /#  ,A.+OO/#rc*tjS)zDReturn the number of processes for the current distribution setting.)r process_countrrrOrO    rc*tjS)z;Return the current process ID for the distribution setting.)r process_indexrXrrrPrPrYrc8t|tjr|St|}d|vr|d}}n|j d\}}tj |}|D].}|j |k(s|jt|k(s,|cStd|)Nr rrzDevice not found: ) rr DevicestrrQr r rintrR) device_namer device_idr rs r_to_backend_devicerbs+szz*k"K +!,aY !,!2!23!7 Ykk+.G ??k )fii3y>.IM )+7 88rc4|jj}|jjDcgc] }t|}}t j |j |}tjj||jScc}w)zConvert the DeviceMesh to JAX backend specific Mesh. Args: device_mesh: DeviceMesh instance to convert. Returns: A `jax.sharding.Mesh` instance. ) r r)flattenrbnparrayreshaper r%Mesh axis_names) device_meshr)dr s r_to_backend_meshrlsy    % %E.9.A.A.I.I.KL!!$LGLhhw''.G <<  Wk&<&< ==MsBc|j tdtjj|j }|jj }tjj||S)zConvert the TensorLayout to JAX backend specific Sharding. Args: tensor_layout: TensorLayout instance to convert. Returns: A `jax.sharding.NamedSharding` instance. zDCannot create sharding when device mesh is not set for TensorLayout.)rjrRr r% PartitionSpecaxes backend_mesh NamedSharding) tensor_layoutpartition_specjax_meshs r_to_backend_layoutrusf  (   \\//1C1CDN((55H << % %h ??r)N)__doc__r r@rekeras.src.randomrkeras.src.utilsrrrrr/r4rKrTrOrPrbrlrurXrrrysR; +%%H$ )(*VM./:6  9 >@r