ijE8ddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl m Z  ddlZeej dsd ej _d Zed Gd d e Zy#e$rY!wxYw)N)backend)tree) keras_export)MonitorCallback) saving_lib)ocp record_scalarcyN)argskwargss y/var/www/html/emotional.easysim.app/public_html/venv/lib/python3.12/site-packages/keras/src/callbacks/orbax_checkpoint.pyrsctjdk(r|j}d}n|jd}d}|rd}tj||S|S)z8Get the complete model state as a nested tree structure.jaxF numpy_array) value_formatTct|tjr|jdk(r|j St|tj r|j S|S)Nr) isinstancenpndarrayndimitemgeneric)objs rconvert_scalarsz(_get_state_tree..convert_scalars*sE#rzz*sxx1}xxz!C,xxz! r)rget_state_treer map_structure)model state_treedid_numpy_conversionrs r_get_state_treer$seE!))+ $))})E # !!/:>>rzkeras.callbacks.OrbaxCheckpointceZdZdZ dfd ZfdZdZddZdZdZ dZ dd Z dd Z dd Z dd Zd ZxZS)OrbaxCheckpointa`Callback to save and load model state using Orbax with a similar API to ModelCheckpoint. This callback saves the model's weights and optimizer state asynchronously using Orbax, allowing training to continue without blocking for I/O. **Multi-host Support**: When running in a multi-host distributed training environment with JAX backend, this callback automatically coordinates checkpointing across all hosts to ensure consistency and proper synchronization. Multi-host checkpointing is only supported on JAX. Example: ```python model.compile(loss=..., optimizer=..., metrics=['accuracy']) EPOCHS = 10 checkpoint_dir = '/tmp/ckpt' orbax_checkpoint_callback = keras.callbacks.OrbaxCheckpoint( directory=checkpoint_dir, monitor='val_accuracy', mode='max', save_best_only=True) # Model is saved at the end of every epoch, if it's the best seen so far. model.fit(epochs=EPOCHS, callbacks=[orbax_checkpoint_callback]) # Alternatively, save checkpoints every N batches - orbax_checkpoint_callback = keras.callbacks.OrbaxCheckpoint( directory=checkpoint_dir, save_freq=100) # Save every 100 batches model.fit(epochs=EPOCHS, callbacks=[orbax_checkpoint_callback]) ``` Args: directory: path to the directory where to save the checkpoints. monitor: The metric name to monitor (e.g., 'val_loss'). verbose: Verbosity mode, 0 or 1. save_best_only: if `save_best_only=True`, it only saves when the model is considered the "best" based on the monitored quantity. mode: one of {'auto', 'min', 'max'}. Used with `save_best_only`. save_freq: `'epoch'` or integer. Frequency to save checkpoints. max_to_keep: Integer, maximum number of recent checkpoints to keep. If None, keeps all. Defaults to 1. save_on_background: Boolean, whether to save asynchronously in the background. Defaults to True. initial_value_threshold: Floating point initial "best" value for the monitor, used with `save_best_only`. c .tjt | |||||_||_||_||_||_| |_ | |_ d|_ d|_ d|_ g|_|j|_|jdk7r3t#|jt$st'd|jdg} |8| j)tj*j,j/|d} | r=t1| dk(r| d} n)tj*j,j3| } tj*j5|| tj*j6j9d|_y)NrepochzUnrecognized save_freq: z2. Expected save_freq are 'epoch' or integer values) directorypreservation_policysave_decision_policy)r initializesuper__init__r*verbosesave_best_only save_freq max_to_keepsave_on_backgroundsave_weights_only_batches_seen_since_last_saving_last_batch_seen_total_batches_seen_async_futures_is_multihost_initialized_multihost_initializedrint ValueErrorappendtrainingpreservation_policiesLatestNlenAnyPreservationPolicy Checkpointersave_decision_policiesFixedIntervalPolicy checkpointer)selfr*monitorr0r1moder2initial_value_thresholdr3r4r5policiesr+ __class__s rr/zOrbaxCheckpoint.__init__ns|  $(?@" ,"&"4!2/0, $#$  '+&D&D&F# >>W $Z-L*4>>*:;CC    " OO 22::;G  # 8}!&.qk#LL66LL $ LL55 3!$!D!D!X!X"6 rct||t|dr2|j%t |jj |_yyy)N optimizer)r. set_modelhasattrrOr< iterationsr8)rHr!rMs rrPzOrbaxCheckpoint.set_modelsC % 5+ &5??+F'*5??+E+E'FD $,G &rcrtjdk7rytj}|jS)z/Check if multi-host environment is initialized.rF)rr multihost%is_jax_distributed_client_initializedrHrTs rr:z)OrbaxCheckpoint._is_multihost_initializeds0 ??  %MM >>@@rcl|jsytj}|xsd}|j|y)z'Synchronize all processes across hosts.Norbax_checkpoint_sync)r;rrTsync_global_processes)rHkeyrTsync_keys r_sync_processeszOrbaxCheckpoint._sync_processess0** MM 11''1rc|jS)akReturn True if multi-host checkpointing is enabled and initialized. This method can be used to check if the callback is operating in a multi-host distributed training environment. Multi-host checkpointing is only supported on JAX backend. Returns: bool: True if multi-host support is active, False otherwise. )r;)rHs ris_multihost_enabledz$OrbaxCheckpoint.is_multihost_enableds***rc`|jsytj}|jdS)aReturn True if this process is the primary host in multi-host setup. In multi-host environments, only the primary host typically handles logging and coordination tasks. Multi-host checkpointing is only supported on JAX backend. Returns: bool: True if this is the primary host, False otherwise. Always returns True in single-host environments. Tr) primary_host)r;rrTis_primary_hostrVs rrazOrbaxCheckpoint.is_primary_hosts-**MM ((a(88rc(|jdk(ry|j||jkr|dz}n||jz }|xj|z c_||_|xj|z c_|j|jk\rd|_yy)z&Check if we should save on this batch.r(Fr)rT)r2r7r6r8)rHbatch add_batchess r_should_save_on_batchz%OrbaxCheckpoint._should_save_on_batchs >>W $  (ET5J5J,J!)K$"7"77K ,, ;, %   K/  / /4>> A34D 0rc8t|j}|jr |d|dd}n|}d|i}|jsOtj|j\}}d|i|d<tj |j}|||d<t j5|j } | r|jj||n7|jj||} |jj| dddy#1swYyxYw) zASave a checkpoint at the given step with multi-host coordination.trainable_variablesnon_trainable_variables)rgrhpytreeconfig model_configNassets)r$r!r5r_serialize_model_as_json_save_assets_to_dictrContextr4rGsave_checkpointablessave_checkpointables_asyncr9r>) rHsteplogsr"composite_statepayload config_json_ assets_dictuse_syncfutures r_save_checkpointz OrbaxCheckpoint._save_checkpoints%TZZ0   ! !'12G'H+5-,O)O_-%%'@@LNK'/&=GN #$99$**EK&$/![[] 3222H!!66tWE**EE'##**62 3 3 3s #A$DDcl|j|rd}|jrr|r|j|jnd}|+t j d|jd|ddd}n&|j ||jsd}n||_|r |j}|j||yyy) NTCan save best model only with z# available, skipping save at batch . stacklevelFrrrs) rer1getrIwarningswarn_is_improvementbestr8r{)rHrcrs should_savecurrentrrs ron_train_batch_endz"OrbaxCheckpoint.on_train_batch_end-s  % %e ,K""48$((4<<0d?MM8G==BG1F#$ #(K--gtyyA"'K!(DI//%%4d%;% -rc|j|j|jdk(}|r~|jrr|r|j |j nd}|+t jd|j d|ddd}n&|j||jsd}n||_ |r|j||yy) Nr(r}z# available, skipping save at epoch r~rrFr) monitor_op_set_monitor_opr2r1rrIrrrrr{)rHr(rsrrs r on_epoch_endzOrbaxCheckpoint.on_epoch_endEs ?? "  "nn/  4..04dhht||,$G 4T\\NC..3WA7  $ ))'499=# $   ! !u4 ! 8 rcz |jj|jdy#t$rYwxYw)Ncheckpoint_cleanup)rGclose Exceptionr\)rHrss r on_train_endzOrbaxCheckpoint.on_train_end`s>     # # % 12    s . ::c|jD]}|j|jj|jj |j dy)zWait for any in-progress checkpoint operations to complete. This method blocks until all asynchronous checkpoint save operations have completed across all hosts in a multi-host setup. checkpoint_wait_completeN)r9resultclearrGwaitr\)rHrzs rwait_until_finishedz#OrbaxCheckpoint.wait_until_finishedksY )) F MMO  !!#   78r) val_lossrFautor(Nr)TFr )__name__ __module__ __qualname____doc__r/rPr:r\r^rarer{rrrr __classcell__)rMs@rr&r&9sg1l  $G RG A2 +9 &+3Z<096 39rr&)rnumpyr keras.srcrrkeras.src.api_exportr$keras.src.callbacks.monitor_callbackrkeras.src.savingrkeras.src.utils.module_utilsrrrQ monitoringr ImportErrorr$r&r rrrs-(,   3>>? 3'C$ </09o919G  s)A11A98A9