+ }iPE`^RIHt^RIt^RIHtHtHt^RIHt^RIH t H t H t ^RI H t ^RIHtHtHtHtHt^RIHt^R IHt^R IHt^R IHt^R IHtHt^R IH t H!t!] 'd ^RI"H#t#^RI$H%t%]!]&4t'] !R]R7t(!RR])](,4t*]!R4!RR44t+R#)) annotationsN)datetime timedeltatimezone) getLogger) TYPE_CHECKINGTypeVarcast)service_locator)ClientSnapshot CpuSnapshotEventLoopSnapshotMemorySnapshotSnapshot)ByteSize)ensure_context) docs_group) RecurringTask) MemoryInfoget_memory_info)EventEventSystemInfoData) TracebackType) ConfigurationT)boundc&]tRt^tRtRRltRtR#)SortedSnapshotListzRA list that maintains sorted order by `created_at` attribute for snapshot objects.c V^8dQhRRRR/#)itemrreturnNone)formats"r/Users/ahmad/.openclaw/workspace/my-crawler/.venv/lib/python3.14/site-packages/crawlee/_autoscaling/snapshotter.py __annotate__SortedSnapshotList.__annotate__ sDDDdDc 8\P!WRR7R#)zUAdd an item to the list maintaining sorted order by `created_at` using binary search.cVP#N created_at)r!s&r&(SortedSnapshotList.add.."s4??r))keyN)bisectinsort)selfr!s&&r&addSortedSnapshotList.add s d&BCr)r$N)__name__ __module__ __qualname____firstlineno____doc__r5__static_attributes__r$r)r&rrs\DDr)r Autoscalingc]tRt^%tRt]!RR7t]!RR7t]!^R7tRt ]!RR7t ^t RR lt ] R,R R ll4t]R Rl4t]RRl4tRRltRRlt]R,RRll4t]R,RRll4t]R,RRll4t]R,RRll4t]R,RRll4tRR ltR!R"ltR#R$ltR%R&ltR'R(ltR)R*lt R+t!R #)- SnapshotteraMonitors and logs system resource usage at predefined intervals for performance optimization. The class monitors and records the state of various system resources (CPU, memory, event loop, and client API) at predefined intervals. This continuous monitoring helps in identifying resource overloads and ensuring optimal performance of the application. It is utilized in the `AutoscaledPool` module to adjust task allocation dynamically based on the current demand and system load. i) millisecondsisecondsg?i'c 0V^8dQhRRRRRRRRRR R R /#) r max_used_cpu_ratiofloatmax_used_memory_ratiomax_event_loop_delayrmax_client_errorsintmax_memory_sizerr"r#r$)r%s"r&r'Snapshotter.__annotate__AsH**"* % * ( *  *"* *r)c WnW nW0nW@nWPnVP \ \,!44VnVP \ \,!44Vn VP \ \,!44Vn VP \ \,!44Vn\VP VP"4Vn\VP&VP(4Vn\,P.!\0P24\5^R7, VnRVnR#)aInitialize a new instance. In most cases, you should use the `from_config` constructor to create a new instance based on the provided configuration. Args: max_used_cpu_ratio: Sets the ratio, defining the maximum CPU usage. When the CPU usage is higher than the provided ratio, the CPU is considered overloaded. max_used_memory_ratio: Sets the ratio, defining the maximum ratio of memory usage. When the memory usage is higher than the provided ratio of `max_memory_size`, the memory is considered overloaded. max_event_loop_delay: Sets the maximum delay of the event loop. When the delay is higher than the provided value, the event loop is considered overloaded. max_client_errors: Sets the maximum number of client errors (HTTP 429). When the number of client errors is higher than the provided number, the client is considered overloaded. max_memory_size: Sets the maximum amount of system memory to be used by the `AutoscaledPool`. )hoursFN)_max_used_cpu_ratio_max_used_memory_ratio_max_event_loop_delay_max_client_errors_max_memory_size_get_sorted_list_by_created_atlistr _cpu_snapshotsr_event_loop_snapshotsr_memory_snapshotsr _client_snapshotsr_snapshot_event_loop_EVENT_LOOP_SNAPSHOT_INTERVAL_snapshot_event_loop_task_snapshot_client_CLIENT_SNAPSHOT_INTERVAL_snapshot_client_taskrnowrutcr!_timestamp_of_last_memory_warning_active)r4rDrFrGrHrJs&$$$$$r&__init__Snapshotter.__init__As2$6 &;#%9""3 /"AA${BSBUV%)%H%HN_I`Ib%c"!%!D!DT.EYE[!\!%!D!DT.EYE[!\)6t7P7PRVRtRt)u&%243H3H$JhJh%i";C<< ;UXahiXj;j. r)Nc V^8dQhRRRR/#)r configzConfiguration | Noner"r?r$)r%s"r&r'rKns  !5  r)c T;'g\P!4pVP'd!\P!VP4M@\\ \ 4PPVP,44pV!VPVPVPVPVR7#)zInitialize a new instance based on the provided `Configuration`. Args: config: The `Configuration` instance. Uses the global (default) one if not provided. )rDrFrGrHrJ)r get_configuration memory_mbytesrfrom_mbrIr total_sizebytesavailable_memory_ratiorDrFrGrH)clsrfrJs&& r& from_configSnapshotter.from_configms>>?<<>###   V11 2#o/::@@6C`C``ab  %88"(">">!'!>QRS S '99; u004;M;MNu004;P;PQ &&,,. ""((* sCC c(V^8dQhRRRRRRRR/#) r exc_typeztype[BaseException] | None exc_valuezBaseException | None exc_tracebackzTracebackType | Noner"r#r$)r%s"r&r'rKs2,(,   r)c "VP'g$\RVPP R24h\P !4pVP \PVPR7VP \PVPR7VPP4GRjxL VPP4GRjxL RVnR#L/L5i)aStop all resource capturing. This method stops capturing snapshots of system resources (CPU, memory, event loop, and client information). It should be called to terminate resource capturing when it is no longer needed. Raises: RuntimeError: If the context manager is not active. r~z is not active.rNF)rbrrr7r roffrrrrr[stopr^)r4rrrrs&&&& r& __aexit__Snapshotter.__aexit__s|||dnn&=&=%>oNO O'99;  1 1DC4C0!C4"C2#C42C4c V^8dQhRRRR/#r durationtimedelta | Noner"list[Snapshot]r$)r%s"r&r'rK 5 5*: 5n 5r)c P\RVP4pVPW!4#)zReturn a sample of the latest memory snapshots. Args: duration: The duration of the sample from the latest snapshot. If omitted, it returns a full history. Returns: A sample of memory snapshots. r)r rW _get_sampler4r snapshotss&& r&get_memory_sampleSnapshotter.get_memory_sample()4+A+AB  44r)c V^8dQhRRRR/#rr$)r%s"r&r'rKs 5 5.> 5. 5r)c P\RVP4pVPW!4#)zReturn a sample of the latest event loop snapshots. Args: duration: The duration of the sample from the latest snapshot. If omitted, it returns a full history. Returns: A sample of event loop snapshots. r)r rVrrs&& r&get_event_loop_sample!Snapshotter.get_event_loop_samples()4+E+EF  44r)c V^8dQhRRRR/#rr$)r%s"r&r'rKs 5 5'7 5> 5r)c P\RVP4pVPW!4#)zReturn a sample of the latest CPU snapshots. Args: duration: The duration of the sample from the latest snapshot. If omitted, it returns a full history. Returns: A sample of CPU snapshots. r)r rUrrs&& r&get_cpu_sampleSnapshotter.get_cpu_samples()4+>+>?  44r)c V^8dQhRRRR/#rr$)r%s"r&r'rKrr)c P\RVP4pVPW!4#)zReturn a sample of the latest client snapshots. Args: duration: The duration of the sample from the latest snapshot. If omitted, it returns a full history. Returns: A sample of client snapshots. r)r rXrrs&& r&get_client_sampleSnapshotter.get_client_samplerr)c$V^8dQhRRRRRR/#)r rrrrr"r$)r%s"r&r'rKs( e e~ e9I eUc er)c V'gV#V'g.#VR,PpVUu.uFq2VP, V8:gKVNK up#uupi)zPReturn a time-limited sample from snapshots or full history if duration is None.r-)rr latest_timesnapshots&& r&rSnapshotter._get_samplesO Im.. )2dXHDWDW6W[c6cddds A Ac V^8dQhRRRR/#r event_datarr"r#r$)r%s"r&r'rKs**(;**r)c 2\VPPVPVPPR7p\ RVP 4pVPW1PP4VP PV4R#)a,Capture a snapshot of the current CPU usage. This method does not perform CPU usage measurement. Instead, it just reads the data received through the `event_data` parameter, which is expected to be supplied by the event manager. Args: event_data: System info data from which CPU usage is read. ) used_ratiomax_used_ratior.rN) r cpu_inforrNr.r rU_prune_snapshotsr5)r4rrrs&& r&rSnapshotter._snapshot_cpusx!**5533!**55  )4+>+>?  i)<)<)G)GH )r)c V^8dQhRRRR/#rr$)r%s"r&r'rKskk+>k4kr)c  4\VPPVPVPVPP RRR7p\ VP;p\4'd#VPVnVPVn \RVP4pVPWBP 4VPPV4VPVPPVPP 4R#)a5Capture a snapshot of the current memory usage. This method does not perform memory usage measurement. Instead, it just reads the data received through the `event_data` parameter, which is expected to be supplied by the event manager. Args: event_data: System info data from which memory usage is read. N) current_sizerJrFr.system_wide_used_sizesystem_wide_memory_sizer)r memory_inforrRrOr. isinstancerrrkrr rWrr5_evaluate_memory_load)r4rrrrs&& r&rSnapshotter._snapshot_memorys"#//<< 11"&"="=!--88"&$(   Z%;%;;kZ H H-8-N-NH */:/E/EH ,)4+A+AB  i)<)<= ""8, "":#9#9#F#F H^H^HiHijr)cV^8dQhRR/#r r"r#r$)r%s"r&r'rK-s11d1r)c \VP\^R7R7pVP'dVPR,MRpV'd5VPVP, VP , pW1n\RVP4pVPWAP4VPPV4R#)aCapture a snapshot of the current event loop usage. This method evaluates the event loop's latency by comparing the expected time between snapshots to the actual time elapsed since the last snapshot. The delay in the snapshot reflects the time deviation due to event loop overhead - it's calculated by subtracting the expected interval between snapshots from the actual time elapsed since the last snapshot. If there's no previous snapshot, the delay is considered zero. rA) max_delaydelayNrr) rrPrrVr.rZrr rr5)r4rprevious_snapshotevent_loop_delayrs& r&rY Snapshotter._snapshot_event_loop-s%t/I/IQZcdQef>B>X>X>XD66r:^b '225F5Q5QQTXTvTvv -N)4+E+EF  i)<)<= ""&&x0r)cV^8dQhRR/#rr$)r%s"r&r'rK@s--$-r)c \P!4pVP4pVPVP^4pVP 'dVP R,P M^p\VW4, VPR7p\RVP 4pVPWeP4VP PV4R#)a3Capture a snapshot of the current API state by checking for rate limit errors (HTTP 429). Only errors produced by a 2nd retry of the API call are considered for snapshotting since earlier errors may just be caused by a random spike in the number of requests and do not necessarily signify API overloading. ) error_countnew_error_countmax_error_countrNr) r get_storage_clientget_rate_limit_errorsget$_CLIENT_RATE_LIMIT_ERROR_RETRY_COUNTrXrr rQr rr.r5)r4clientrate_limit_errorsrprevious_error_countrrs& r&r\Snapshotter._snapshot_client@s !335,2,H,H,J'++D,U,UWXY IMI_I_I_t55b9EEef!#'> 33  )4+A+AB  i)<)<= ""8,r)c$V^8dQhRRRRRR/#)r rrr_rr"r#r$)r%s"r&r'rKVs!.xDr)c Rp\V4F*wrEW%P, VP8:gK(TpM VeVRV1R#VP4R#)ahRemove snapshots that are older than the `self._snapshot_history`. This method modifies the list of snapshots in place, removing all snapshots that are older than the defined snapshot history relative to the `now` parameter. Args: snapshots: List of snapshots to be pruned in place. now: The current date and time, used as the reference for pruning. N) enumerater._SNAPSHOT_HISTORYclear)r4rr_keep_from_indexirs&&& r&rSnapshotter._prune_snapshotsVsY$Y/KA(((D,B,BB"#0  &*?*+ OO r)c$V^8dQhRRRRRR/#)r current_memory_usage_sizersnapshot_timestamprr"r#r$)r%s"r&r'rKos(HHxH]eHjnHr)c  W PVP,8dR#VPVP,pVP^VP, ,VP,pW4,pW8di\ VP VPP , ^d,4p\PRV RVP RV R24W nR#R#)zEvaluate and logs critical memory load conditions based on the system information. Args: current_memory_usage_size: The current memory usage. snapshot_timestamp: The time at which the memory snapshot was taken. Nz'Memory is critically overloaded. Using z of z (z)%). Consider increasing available memory.) ra_MEMORY_WARNING_COOLDOWN_PERIODrOrR_RESERVE_MEMORY_RATIOroundrlloggerwarning)r4rrthreshold_memory_sizebuffer_memory_sizeoverload_memory_threshold_sizememory_usage_percentages&&& r&r!Snapshotter._evaluate_memory_loados  F FImIm m m  $ ; ;d>S>S S!22a$:U:U6UVY]YsYss)>)S& % E&+-F-L-LtOdOdOjOj-jnq,q&r # NN9:S9TTX((),C+DE88  6H 2 Fr)) rbrXrUrVrQrPrRrNrOrWr^r[rar,)"r7r8r9r:r;rrZr]rrrrrc classmethodro staticmethodrSpropertyrzrrrrrrrrrrrYr\rrr<r$r)r&r?r?%s+%.3$?!: )t <6!"-=c&/U&C#]+,(f*X  2"0 5 5 5 5 5 5 5 5 e e*&k61&-,2HHr)r?), __future__rr2rrrloggingrtypingrr r crawleer crawlee._autoscaling._typesr r rrrcrawlee._utils.byte_sizercrawlee._utils.contextrcrawlee._utils.docsrcrawlee._utils.recurring_taskrcrawlee._utils.systemrrcrawlee.events._typesrrtypesrcrawlee.configurationrr7rrrTrr?r$r)r&rs# 22//#pp-1*7=<#3 8  Cx DaD M`H`H`Hr)