ijddlZddlZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZejdk(r!ej6j,j8ZndZdZe dGddeZe dGddeZ y)N)backend)tree) keras_export)is_float_dtype)standardize_dtype)Layer draw_seed)serialization_lib) jax_utils)tracking)jax) tensorflowrc|SN)fns n/var/www/html/emotional.easysim.app/public_html/venv/lib/python3.12/site-packages/keras/src/utils/jax_layer.py#tf_no_automatic_dependency_trackingrs ctjdk(r'tj|tjdS|S)Nrr)rtfbitcastuint32)tensors r_convert_to_jax_keyrs0L(zz&")),Q// Mrzkeras.layers.JaxLayerceZdZdZ dfd ZdZdZdZdZe je dZ dZ d Zd Zd Zd Zfd ZddZfdZefdZxZS)JaxLayera!Keras Layer that wraps a JAX model. This layer enables the use of JAX components within Keras when using JAX as the backend for Keras. ## Model function This layer accepts JAX models in the form of a function, `call_fn`, which must take the following arguments with these exact names: - `params`: trainable parameters of the model. - `state` (*optional*): non-trainable state of the model. Can be omitted if the model has no non-trainable state. - `rng` (*optional*): a `jax.random.PRNGKey` instance. Can be omitted if the model does not need RNGs, neither during training nor during inference. - `inputs`: inputs to the model, a JAX array or a `PyTree` of arrays. - `training` (*optional*): an argument specifying if we're in training mode or inference mode, `True` is passed in training mode. Can be omitted if the model behaves the same in training mode and inference mode. The `inputs` argument is mandatory. Inputs to the model must be provided via a single argument. If the JAX model takes multiple inputs as separate arguments, they must be combined into a single structure, for instance in a `tuple` or a `dict`. ## Model weights initialization The initialization of the `params` and `state` of the model can be handled by this layer, in which case the `init_fn` argument must be provided. This allows the model to be initialized dynamically with the right shape. Alternatively, and if the shape is known, the `params` argument and optionally the `state` argument can be used to create an already initialized model. The `init_fn` function, if provided, must take the following arguments with these exact names: - `rng`: a `jax.random.PRNGKey` instance. - `inputs`: a JAX array or a `PyTree` of arrays with placeholder values to provide the shape of the inputs. - `training` (*optional*): an argument specifying if we're in training mode or inference mode. `True` is always passed to `init_fn`. Can be omitted regardless of whether `call_fn` has a `training` argument. ## Models with non-trainable state For JAX models that have non-trainable state: - `call_fn` must have a `state` argument - `call_fn` must return a `tuple` containing the outputs of the model and the new non-trainable state of the model - `init_fn` must return a `tuple` containing the initial trainable params of the model and the initial non-trainable state of the model. This code shows a possible combination of `call_fn` and `init_fn` signatures for a model with non-trainable state. In this example, the model has a `training` argument and an `rng` argument in `call_fn`. ```python def stateful_call(params, state, rng, inputs, training): outputs = ... new_state = ... return outputs, new_state def stateful_init(rng, inputs): initial_params = ... initial_state = ... return initial_params, initial_state ``` ## Models without non-trainable state For JAX models with no non-trainable state: - `call_fn` must not have a `state` argument - `call_fn` must return only the outputs of the model - `init_fn` must return only the initial trainable params of the model. This code shows a possible combination of `call_fn` and `init_fn` signatures for a model without non-trainable state. In this example, the model does not have a `training` argument and does not have an `rng` argument in `call_fn`. ```python def stateless_call(params, inputs): outputs = ... return outputs def stateless_init(rng, inputs): initial_params = ... return initial_params ``` ## Conforming to the required signature If a model has a different signature than the one required by `JaxLayer`, one can easily write a wrapper method to adapt the arguments. This example shows a model that has multiple inputs as separate arguments, expects multiple RNGs in a `dict`, and has a `deterministic` argument with the opposite meaning of `training`. To conform, the inputs are combined in a single structure using a `tuple`, the RNG is split and used the populate the expected `dict`, and the Boolean flag is negated: ```python def my_model_fn(params, rngs, input1, input2, deterministic): ... if not deterministic: dropout_rng = rngs["dropout"] keep = jax.random.bernoulli(dropout_rng, dropout_rate, x.shape) x = jax.numpy.where(keep, x / dropout_rate, 0) ... ... return outputs def my_model_wrapper_fn(params, rng, inputs, training): input1, input2 = inputs rng1, rng2 = jax.random.split(rng) rngs = {"dropout": rng1, "preprocessing": rng2} deterministic = not training return my_model_fn(params, rngs, input1, input2, deterministic) keras_layer = JaxLayer(my_model_wrapper_fn, params=initial_params) ``` ## Usage with Haiku modules `JaxLayer` enables the use of [Haiku](https://dm-haiku.readthedocs.io) components in the form of [`haiku.Module`](https://dm-haiku.readthedocs.io/en/latest/api.html#module). This is achieved by transforming the module per the Haiku pattern and then passing `module.apply` in the `call_fn` parameter and `module.init` in the `init_fn` parameter if needed. If the model has non-trainable state, it should be transformed with [`haiku.transform_with_state`]( https://dm-haiku.readthedocs.io/en/latest/api.html#haiku.transform_with_state). If the model has no non-trainable state, it should be transformed with [`haiku.transform`]( https://dm-haiku.readthedocs.io/en/latest/api.html#haiku.transform). Additionally, and optionally, if the module does not use RNGs in "apply", it can be transformed with [`haiku.without_apply_rng`]( https://dm-haiku.readthedocs.io/en/latest/api.html#without-apply-rng). The following example shows how to create a `JaxLayer` from a Haiku module that uses random number generators via `hk.next_rng_key()` and takes a training positional argument: ```python class MyHaikuModule(hk.Module): def __call__(self, x, training): x = hk.Conv2D(32, (3, 3))(x) x = jax.nn.relu(x) x = hk.AvgPool((1, 2, 2, 1), (1, 2, 2, 1), "VALID")(x) x = hk.Flatten()(x) x = hk.Linear(200)(x) if training: x = hk.dropout(rng=hk.next_rng_key(), rate=0.3, x=x) x = jax.nn.relu(x) x = hk.Linear(10)(x) x = jax.nn.softmax(x) return x def my_haiku_module_fn(inputs, training): module = MyHaikuModule() return module(inputs, training) transformed_module = hk.transform(my_haiku_module_fn) keras_layer = JaxLayer( call_fn=transformed_module.apply, init_fn=transformed_module.init, ) ``` Args: call_fn: The function to call the model. See description above for the list of arguments it takes and the outputs it returns. init_fn: the function to call to initialize the model. See description above for the list of arguments it takes and the outputs it returns. If `None`, then `params` and/or `state` must be provided. params: A `PyTree` containing all the model trainable parameters. This allows passing trained parameters or controlling the initialization. If both `params` and `state` are `None`, `init_fn` is called at build time to initialize the trainable parameters of the model. state: A `PyTree` containing all the model non-trainable state. This allows passing learned state or controlling the initialization. If both `params` and `state` are `None`, and `call_fn` takes a `state` argument, then `init_fn` is called at build time to initialize the non-trainable state of the model. seed: Seed for random number generator. Optional. native_serialization_platforms: Sequence of platforms ('cpu', 'cuda', 'rocm', 'tpu') to compile for when using `jax2tf.convert` with native serialization. This is only used when the Keras backend is `tensorflow`. If `None`, the function is compiled for the default backend. If multiple platforms are specified, the exported module will be device-polymorphic. c <tjdvr6t|jjdtjt |di|||_||_|j|d|_ |j|d|_ |j |j|j|j|dhddh|_d |jv|_d |jv|_d |jv}|r%tj$j'||_nd|_|'|%|#|j s |j"r td |r|j|d hddh|_d|_d|_||_y)N)rrzH is only supported with the JAX or Tensorflow backend. Current backend: T trainableFcall_fn>rngstateinputsparamstrainingr%r&r$r#zk`init_fn`, `params` and `state` cannot all be `None` when `call_fn` takes a `params` or a `state` argument.init_fn>r#r%r'r)r ValueError __class____name__super__init__r"r(_create_variablestracked_params tracked_stater&r$_build_at_init_validate_signaturecall_fn_argumentscall_fn_has_paramscall_fn_has_staterandom SeedGeneratorseed_generatorinit_fn_argumentsjax2tf_training_false_fnjax2tf_training_true_fn%jax2tf_native_serialization_platforms) selfr"r(r&r$seednative_serialization_platformskwargscall_fn_has_rngr*s rr-zJaxLayer.__init__s ?? $9 9>>**+,99@9J8KM  "6"  "44Vt4L!33EU3K ;; "djj&<    !!%!9!9   < J "  #+d.D.D"D!(D,B,B!B4#9#99 ")..">">t"DD "&D  O ((D,B,BD  %)%=%=$AH:&D " )-%'+$ * 2rc Ztj|j}|D]}||vstd|d|dg}|j D]Y}|j |vr.td|d|j ddj |d|j|j [|S)NzMissing required argument in `z`: ``zUnsupported argument in `z`, supported arguments are `z`, `)inspect signature parametersr)valuesnamejoinappend) r=rfn_nameallowedrequired fn_parametersparameter_nameparameter_names parameters rr2zJaxLayer._validate_signature+s))"-88 & N]2 4WI>&'q* &--/ 3I~~W, /yY^^> 2  3rc tjtjtjdtj tjdfd}t j||}|S)aConvert input shape in a format suitable for `jax2tf`. `jax2tf` expects a letter for each unknown dimension, which allows correlated dimensions. Since correlated dimensions are not supported by Keras, we simply use 'a', 'b', 'c'..., for each unknown dimension. We however use 'batch' for dimension 0 if not defined to correlate the batch size across inputs. Example (spaces added for readability): ``` input_shape: (None , 4 , None, None, 5 ) result: "(batch, 4 , a , b , 5 )" ``` Args: input_shape: a single shape or a structure of shapes for the inputs. Returns: the shape or shapes structure in the `jax2tf` format as strings. c ||zSrr)abs rz2JaxLayer._get_jax2tf_input_shape..Vs QUr)repeatcg}t|D]S\}}||jt|#|dk(r|jd:|jtUddj |zdzS)Nrbatch(z, )) enumeraterJstrnextrI)shape jax2tf_shapeindexdim dim_namess rget_single_jax2tf_shapezAJaxLayer._get_jax2tf_input_shape..get_single_jax2tf_shape[syL'. 9 s? ''C1aZ ''0 ''Y8  9<0036 6r) itertoolschainstringascii_lowercasestarmapproductrmap_shape_structure)r= input_shapereresrds @r_get_jax2tf_input_shapez JaxLayer._get_jax2tf_input_shape?sb(OO  " "   "!!&"8"8C   7&&'> L rcddlm}d|i}|j|j|d<|j|fi|}tj j j|}|S)Nr)jax2tfpolymorphic_shapesr?)jax.experimentalrqr<convertr autograph experimentaldo_not_convert)r=rrrrq jax2tf_kwargs converted_fns r_jax2tf_convertzJaxLayer._jax2tf_convertksg+-/AB  5 5 A:: : ;&v~~b:M: ||00?? M rcJtjfd}|S)aReturn a new partial with one positional argument set to a value. This is needed because `jax2tf` only supports positional arguments and `functools.partial` only supports setting positional arguments starting from the left. Our use case is the `training` argument which is typically the righmost argument. Args: fn: the function to wrap. index: the index of the positional argument to set to `value`. value: the value for the positional argument at `index`. c,|dfz|dz}|S)Nrr)argsrrbvalues rwrapperz2JaxLayer._partial_with_positional..wrappers*%=E8+d56l:Dt9 r) functoolswraps)r=rrbr~rs ``` r_partial_with_positionalz!JaxLayer._partial_with_positionalys'     rcfd}tjj||}r|_n|_tjj |\}}|S)aCreate a structure of variables from a structure of JAX arrays. `values` is traversed via JAX's `tree_map`. When a leaf is a JAX array or a tensor-like object, a corresponding variable is created with it as the initial value. The resulting structure of variables is assigned to `self.params` or `self.state` depending on `trainable`. Then, a flattened version of the variables is returned for tracking. `self.params` or `self.state` are intentionally not tracked because structures like `TrackedList` interfere with `jax.tree_utils`. Note that leaf objects that are not JAX arrays and not tensor-like are left intact as they are assumed to be configuration used by the model. Args: values: the structure of values to traverse. trainable: whether to create trainable variables. Returns: flat list of variables initialized with `values` for tracking. ctj|s9t|tjtj t jfrK|j}t|rd}j|jtj||St|tttfrIt!t#|}t|rd}jdtj||S|S)N) initializerdtyper!r)r is_tensor isinstancenpndarraygenericrArrayrr add_weightr`convert_to_tensorboolintfloatrtype)r~rr=r!s rcreate_variablez3JaxLayer._create_variables..create_variables  ': BJJ :, !%( EKK ' 9 9% @' ' ED#u#56)$u+6!%( E ' 9 9% @' ' r)r tree_utiltree_mapr&r$ tree_flatten)r=rGr!r variablesflat_variables_s` ` rr.zJaxLayer._create_variablessP. 6MM**?FC #DK"DJMM66yArctdS)z Returns a single seed as a tensor of shape [2]. Call this within `_get_init_rng()` to obtain a new seed. Returns: A native tensor of shape [2] and the backend dtype for seeds. Nr r=s r_get_init_seedzJaxLayer._get_init_seedsrc"|jS)a Returns a seed or seeds to pass as the `rng` argument of `init_fn`. By default, this returns a single seed. Override this to return a different structure. Overrides should use `self._get_init_seed()` to obtain new seeds. Returns: RNG key or structure of keys as tensors of shape [2] and the backend dtype for seeds. rrs r _get_init_rngzJaxLayer._get_init_rngs""$$rc6|jjS)z Returns a single seed as a tensor of shape [2]. Call this within `_get_call_rng()` to obtain a new seed. Returns: A native tensor of shape [2] and the backend dtype for seeds. )r8r_rs r_get_call_seedzJaxLayer._get_call_seeds""''))rc(|r|jSy)a Returns a seed or seeds to pass as the `rng` argument of `call_fn`. By default, this returns a seed when `training` is `True`, and `None` when `training` is `False`. Override this to return a different structure or to pass seeds in inference mode too. Overrides should use `self._get_call_seed()` to obtain seeds. Returns: RNG key or structure of keys as tensors of shape [2] and the backend dtype for seeds. Nrr=r's r _get_call_rngzJaxLayer._get_call_rngs &&( (rctjr tdd}tj||}t j dk(r6t jtj|dr tdg}|jD]t}|dk(r?|jtjjd|jG|dk(r|j|^|d k(sd|jd v|j|}|j r|\}}n|d}}|j#|d |_|j#|d |_y) Nz-'JaxLayer' cannot be built inside tf functioncp|Dcgc]}||nd }}tjj|Scc}w)N)rnumpyones)r`ds r create_inputz2JaxLayer._initialize_weights..create_inputs68=>1!-QQ.>E>99>>%( (?s 3rrz-'JaxLayer' cannot be built in a tracing scoper#cRtjjt|Sr)rrarrayr)xs rrVz.JaxLayer._initialize_weights..s#))//2Ea2H"Irr%r'Tr F)rinside_functionr)rrlrr is_in_jax_tracing_scopeflattenr9rJrrrrr(r5r.r/r0) r=rmr init_inputs init_args argument_name init_result init_params init_states r_initialize_weightszJaxLayer._initialize_weightssR   LM M )..|[I ??  %)*K*K LL %a (+ LM M !33 'M%  MM**I**, (*  -*,  & '#dllI.  ! !&1 #K&14K"44 45 "33J%3Prc|j5|j)|js |jr|j |t j dk(rg}|j D]?}|dk(r!|j|j|)|dk7s/|jdAd|j vr|j jd}|j|j|j|d||_ |j|j|j|d||_n(|j|j||_ d|_t|A|yy)Nrr%r'z...FT)r&r$r4r5rrr3rJrorbrzrr"r:r;r,build)r=rmrrargumenttraining_argument_indexr*s rrzJaxLayer.build)si KK  "((D,B,B  $ $[ 1 ??  ,!#  22 5x'&--44[A+&--e4 5T333*.*@*@*F*F+'150D0D11 &=u' 1- 04/C/C11 &=t' 0,150D0DLL&1-04, GM+ &C -rc Jd}gjD]}|dk(r:jtjj |j C|dk(r:jtjj |j |dk(rCjtjj tj||dk(rj||dk(stjdk(sj|dfd }tjdk(r|jStjd k(r2|rj|jS|jSy) Nc"|dS|jSr)r~)variables runwrap_variablez&JaxLayer.call..unwrap_variableUs#+4 ? ?rr&r$r#r%r'rcTt|ds td|j|y)NassignzStructure mismatch: the structure of the state returned by `call` does not match the structure of the state at initialization time.)hasattrr)r)r~rs rassign_state_to_variablez/JaxLayer.call..assign_state_to_variablens,8X. + OOE "rcjr5|\}}tjj|j|S|Sr)r5rrrr$)r predictions new_stater call_argsr=s r call_with_fnz#JaxLayer.call..call_with_fnxsI%%)+Y& Y &&,i#"9~%rr) r3rJrrrr&r$rrrr"r;r:)r=r%r'rrrrrs` @@rcallz JaxLayer.callTsh @ !33 /M(  MM**?DKKH')  MM**?DJJG%'  MM**+T-?-?-I (*  (*,??$-$$X.' /* # & ??  % - - __ , .D88D#D$@$@AA#D$A$ABB /rc,tj|jtj|j|jd}t |}tt|jt|jzS)N)r"r(r?) r serialize_keras_objectr"r(r<r, get_configdictlistitems)r=config base_configr*s rrzJaxLayer.get_configso(?? M(?? M::  g(* D**,-V\\^0DDEErctj|d}tj|d}||d<||d<t| |S)Nr"r()r deserialize_keras_objectr, from_config)clsrr"r(r*s rrzJaxLayer.from_configsQ#<.apply_with_trainingsB;;$$33FEB{{%! % rcxjjj||||jS)N)rrrr)r&r$r#r%rr=s rapply_without_trainingz2FlaxLayer.__init__..apply_without_trainings?;;$$33FEB{{% % rctjjj||j|S)N)rr'_variables_to_params_and_staterinitr)r#r%r'r=s rinit_with_trainingz.FlaxLayer.__init__..init_with_trainings=66   ;;% ! rcrjjj||jS)N)rr)r#r%r=s rinit_without_trainingz1FlaxLayer.__init__..init_without_training%s:66   ;;! rr')r"r(r&r$r) flax.linenrrrrDrE__call__rFrr,r-)r=rrrr@rrrrrr"r(r&r$rr*s` @rr-zFlaxLayer.__init__s (   (,        !:6??;FF G 34FWG57LWG;;IF       rc&|r |ri||S|S|r|SiSrr)r=r&r$s rrz(FlaxLayer._params_and_state_to_variables@s* *&*E** L rc|yd|vri|fSt|dk(r|ifSd|di}|jDcic]\}}|dk7s ||}}}||fScc}}w)NNNr&r)lenr)r=rr&kvr$s rrz(FlaxLayer._variables_to_params_and_stateJsz   9 $y= y>Q b= Ih/0"+//"3E$!QqH}AEEu}Fs AAcD|j|jdS)N)r&dropoutrrs rrzFlaxLayer._get_init_rngYs$))+**,  rc.|rd|jiSiS)Nrrrs rrzFlaxLayer._get_call_rng_s t2245 5Irc|j}t|jdr9|jj|jk(r|jj}t j |jt j |d}t|!}|jd|jdtt|jt|jzS)N__self__)rrr"r() rrrrr+r rr,rpoprrr)r= config_methodrrr*s rrzFlaxLayer.get_configes DKK , $$ 3!KK00M'>>t{{K'>>}M g(*  " "D**,-V\\^0DDEErctj|d}tj|d}t|dtr t ||}||d<||d<|di|S)Nrrr)r rrr^getattr)rrrrs rrzFlaxLayer.from_configwsg";;F8rs -=@(5.%$,97?? $   AA(  %&u +uu +'u +p&'aa(ar