ij2lddlmZddlmZddlmZddlmZddlmZedGddeZ y ) )backend)ops) keras_export) KerasTensor)Layerzkeras.layers.AttentionceZdZdZ d fd ZdZdZddZdZ ddZ ddZ d Z dd Z dd Z fd ZxZS) Attentiona Dot-product attention layer, a.k.a. Luong-style attention. Inputs are a list with 2 or 3 elements: 1. A `query` tensor of shape `(batch_size, Tq, dim)`. 2. A `value` tensor of shape `(batch_size, Tv, dim)`. 3. A optional `key` tensor of shape `(batch_size, Tv, dim)`. If none supplied, `value` will be used as a `key`. The calculation follows the steps: 1. Calculate attention scores using `query` and `key` with shape `(batch_size, Tq, Tv)`. 2. Use scores to calculate a softmax distribution with shape `(batch_size, Tq, Tv)`. 3. Use the softmax distribution to create a linear combination of `value` with shape `(batch_size, Tq, dim)`. Args: use_scale: If `True`, will create a scalar variable to scale the attention scores. dropout: Float between 0 and 1. Fraction of the units to drop for the attention scores. Defaults to `0.0`. seed: A Python integer to use as random seed in case of `dropout`. score_mode: Function to use to compute attention scores, one of `{"dot", "concat"}`. `"dot"` refers to the dot product between the query and key vectors. `"concat"` refers to the hyperbolic tangent of the concatenation of the `query` and `key` vectors. Call arguments: inputs: List of the following tensors: - `query`: Query tensor of shape `(batch_size, Tq, dim)`. - `value`: Value tensor of shape `(batch_size, Tv, dim)`. - `key`: Optional key tensor of shape `(batch_size, Tv, dim)`. If not given, will use `value` for both `key` and `value`, which is the most common case. mask: List of the following tensors: - `query_mask`: A boolean mask tensor of shape `(batch_size, Tq)`. If given, the output will be zero at the positions where `mask==False`. - `value_mask`: A boolean mask tensor of shape `(batch_size, Tv)`. If given, will apply the mask such that values at positions where `mask==False` do not contribute to the result. return_attention_scores: bool, it `True`, returns the attention scores (after masking and softmax) as an additional output argument. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). use_causal_mask: Boolean. Set to `True` for decoder self-attention. Adds a mask such that position `i` cannot attend to positions `j > i`. This prevents the flow of information from the future towards the past. Defaults to `False`. Output: Attention outputs of shape `(batch_size, Tq, dim)`. (Optional) Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. c t|di|||_||_||_||_|jdkDr%t jj||_ |jdvrtd|y)Nrseed)dotconcatz_Invalid value for argument score_mode. Expected one of {'dot', 'concat'}. Received: score_mode=) super__init__ use_scale score_modedropoutr rrandom SeedGeneratorseed_generator ValueError)selfrrrr kwargs __class__s y/var/www/html/emotional.easysim.app/public_html/venv/lib/python3.12/site-packages/keras/src/layers/attention/attention.pyrzAttention.__init__Bs "6""$  <">D">"ID  ??"3 3((2|5  4c|j|d|_d|_|jr%|j ddd|j d|_|j dk(r&|j ddd|j d|_yy)NscaleronesT)nameshape initializerdtype trainablerconcat_score_weight)_validate_inputsrr&r add_weightr$r)r input_shapes rbuildzAttention.buildYs k* #' >>"jj )DJ ??h &'+*"jj (7(D $ 'rc|jdk(rYtj|tj|dd}|j tj ||j}|S|jdk(rtj |d}tj |d}|jI|jtjtj|j||zzdz}|S|jtjtj||zdz}|Std)aCalculates attention scores as a query-key dot product. Args: query: Query tensor of shape `(batch_size, Tq, dim)`. key: Key tensor of shape `(batch_size, Tv, dim)`. Returns: Tensor of shape `(batch_size, Tq, Tv)`. r raxiszscores not computed) rrmatmulswapaxesrmultiply expand_dimsr&sumtanhr)rquerykeyscores q_reshaped k_reshapeds r_calculate_scoreszAttention._calculate_scoresns ??e #ZZs||CR'@AFzz%fdjj9$ #__ (R8J26Jzz%11CGGHHTZZ: +BCD25 11CGGHHZ*45B5  23 3rc|qtj|}|jdk(rdnd}t|jdk(rtj |d}tj |||z |}tj|d}|rE|jdkDr6tjj||j|j }tj|||fS) aApplies attention scores to the given value tensor. To use this method in your attention layer, follow the steps: * Use `query` tensor of shape `(batch_size, Tq)` and `key` tensor of shape `(batch_size, Tv)` to calculate the attention `scores`. * Pass `scores` and `value` tensors to this method. The method applies `scores_mask`, calculates `attention_distribution = softmax(scores)`, then returns `matmul(attention_distribution, value). * Apply `query_mask` and return the result. Args: scores: Scores float tensor of shape `(batch_size, Tq, Tv)`. value: Value tensor of shape `(batch_size, Tv, dim)`. scores_mask: A boolean mask tensor of shape `(batch_size, 1, Tv)` or `(batch_size, Tq, Tv)`. If given, scores at positions where `scores_mask==False` do not contribute to the result. It must contain at least one `True` value in each line along the last dimension. training: Python boolean indicating whether the layer should behave in training mode (adding dropout) or in inference mode (no dropout). Returns: Tensor of shape `(batch_size, Tq, dim)`. Attention scores after masking and softmax with shape `(batch_size, Tq, Tv)`. float16g@geAr,r.r-rr ) r logical_notr$lenr"r4wheresoftmaxrrrrr1)rr9value scores_masktraining padding_mask max_valueweightss r _apply_scoreszAttention._apply_scoress<  "??;7L$*<<9#<%I<%%&!+"|"E YY|Vi-?HF++f2.  q(nn,, ((-G zz'5)722rcb|rtj|}d|d|df}tj|d}tj|d}tj|d}tj||} |-tj |d}tj || S| S|S)Nr,r-int32)r"r$r.)rr"r cumsum greater_equalr4 logical_and) rr9v_maskuse_causal_mask score_shape mask_shape ones_mask row_index col_index causal_masks r_calculate_score_maskzAttention._calculate_score_masks ))F+K[_k"o>JzAI 926I 926I++IyAK!b9v{;; Mrcv|j|||d}|d}t|dkDr|dn|}|r|dnd} |r|dnd} |j||} |j| | |} |j | || |\} }| .t j | d} t j| | d} |r| |fS| S) NinputsmaskrrLr?)r7r8)r9rDrErFr-r.)r'rAr<rYrJrr4rB)rr\r]rFreturn_attention_scoresrRqvkq_maskrQr9rEattention_outputattention_scoress rcallzAttention.calls V$7 1I 1IVqF1Ia ad ad''aQ'700 FO .2-?-? h.@. **  __V"5F"yy1A1E  "$&67 7# #rch|j||||dytj|dS)Nr[r)r'rconvert_to_tensor)rr\r]s r compute_maskzAttention.compute_masks: V$7 <47?$$T!W--rc0|d}|d}g|dd|dS)NrrLr-r)rr) query_shape value_shapes rcompute_output_shapezAttention.compute_output_shapes/!!n !!n 3Sb!3;r?33rcr|j|||d}|d}t|dkDr|dn|}|j|j|j|jg} t | |j } |r:g|jdd|jd} | t | |j fS| S)NrrLr?)r$r-r,)r'rArlr"r compute_dtype) rr\r]r^rFrRr7rDr8 output_shape output_spec scores_shapes rcompute_output_speczAttention.compute_output_specs fd+q q v;?fQi00 [[%++syy 1 ",d6H6HI "Sb! " L D$6$6! rc|jj}t|tst |d|dt |dkst |dkDrt |dt |d|Rt|tst |d|dt |dkst |dkDrt |d|d |dyy) z'Validates arguments of the call method.zj layer must be called on a list of inputs, namely [query, value] or [query, value, key]. Received: inputs=.r?zl layer accepts inputs list of length 2 or 3, namely [query, value] or [query, value, key]. Received length: NzL layer mask must be a list, namely [query_mask, value_mask]. Received: mask=z< layer accepts mask list of length 2 or 3. Received: inputs=z, mask=)r__name__ isinstancelistrrA)rr\r] class_names rr'zAttention._validate_inputss^^,, &$',$$*81.  v;?c&kAo,$$'K=3   dD) !l#GGKfAO4y1}D A  !l#((.xwtfA?!. rct|}|j|j|j|j d}i||S)N)rrrr )r get_configrrrr )r base_configconfigrs rr{zAttention.get_config7sEg(* //||II   )+(((r)Fr gN)NF)NFFF)N)NFNF)rv __module__ __qualname____doc__rr*r<rJrYrerhrlrrr'r{ __classcell__)rs@rr r sq6t  .*B.3`4 % $<. 4 % :6))rr N) keras.srcrrkeras.src.api_exportrkeras.src.backendrkeras.src.layers.layerrr rrrrs8-)(&'v)v)(v)r