+ }i3D^RIHt^RIt^RIHt^RIHt^RIHtH t ^RI H t ^RI H t Ht^RIHt^R IHt^R IHt^R IHt^R IHt]'d'^R IHt^RI H t ^RIHt^RIHt^RIH t ^RI!H"t"H#t#]!]$4t%] !R4t&]!R4!RR]]44t'R#)) annotationsN) timedelta) getLogger) TYPE_CHECKINGTypeVar)override)Requestservice_locator) docs_group)wait_for_all_tasks_for_finish)RequestManager)Storage)validate_storage_name)Sequence)r ) Configuration) StorageClient)RequestQueueClient)ProcessedRequestRequestQueueMetadataTStoragesc ]tRt^ tRtRRlt]]RRl44t]]RRl44t ]RR l4t ]R R l4t ]R R l4t ]] RRRRRRRRRR/RRll44t]RRl4t]RRl4t]RR/RRll4t]RRRRR ]!^R!7R"RR#R/R$R%ll4tR&R'ltR(R)ltR*R+ltRR/R,R-lltR.R/ltR0R1ltR2^RR/R3R4lltR5tR#)6 RequestQueueaRequest queue is a storage for managing HTTP requests. The request queue class serves as a high-level interface for organizing and managing HTTP requests during web crawling. It provides methods for adding, retrieving, and manipulating requests throughout the crawling lifecycle, abstracting away the underlying storage implementation details. Request queue maintains the state of each URL to be crawled, tracking whether it has been processed, is currently being handled, or is waiting in the queue. Each URL in the queue is uniquely identified by a `unique_key` property, which prevents duplicate processing unless explicitly configured otherwise. The class supports both breadth-first and depth-first crawling strategies through its `forefront` parameter when adding requests. It also provides mechanisms for error handling and request reclamation when processing fails. You can open a request queue using the `open` class method, specifying either a name or ID to identify the queue. The underlying storage implementation is determined by the configured storage client. ### Usage ```python from crawlee.storages import RequestQueue # Open a request queue rq = await RequestQueue.open(name='my_queue') # Add a request await rq.add_request('https://example.com') # Process requests request = await rq.fetch_next_request() if request: try: # Process the request # ... await rq.mark_request_as_handled(request) except Exception: await rq.reclaim_request(request) ``` c(V^8dQhRRRRRRRR/#) clientridstrname str | NonereturnNone)formats"q/Users/ahmad/.openclaw/workspace/my-crawler/.venv/lib/python3.14/site-packages/crawlee/storages/_request_queue.py __annotate__RequestQueue.__annotate__Js0@@1@s@*@QU@c \V4WnW nW0n\\ P ,!4VnR#)zInitialize a new instance. Preferably use the `RequestQueue.open` constructor to create a new instance. Args: client: An instance of a storage client. id: The unique identifier of the storage. name: The name of the storage, if available. N)r_client_id_namelistasyncioTask_add_requests_tasks)selfrrrs&&&&r%__init__RequestQueue.__init__Js2 d#  #' #5#7 ?r(cV^8dQhRR/#)rr!rr#)r$s"r%r&r'_sCr(c VP#N)r+r1s&r%rRequestQueue.id]s xxr(cV^8dQhRR/#)rr!r r#)r$s"r%r&r'dsjr(c VP#r6)r,r7s&r%rRequestQueue.namebszzr(cV^8dQhRR/#)rr!rr#)r$s"r%r&r'hs11$81r(c R"VPP4GRjxL #L5ir6)r* get_metadatar7s&r%r>RequestQueue.get_metadatags\\..0000 '%'cV^8dQhRR/#rr!intr#)r$s"r%r&r'ls...r(c j"VPP4GRjxL pVP#L5ir6)r*r>handled_request_countr1metadatas& r%get_handled_countRequestQueue.get_handled_countks+2244---5 313cV^8dQhRR/#rBr#)r$s"r%r&r'qs,,s,r(c j"VPP4GRjxL pVP#L5ir6)r*r>total_request_countrFs& r%get_total_countRequestQueue.get_total_countps+2244+++5rJrNralias configurationstorage_clientc 0V^8dQhRRRRRRRRRRR R /#) rrr rrPrQzConfiguration | NonerRzStorageClient | Noner!rr#)r$s"r%r&r'wsH        ,  -   r(c "Vf\P!4MTpVf\P!4MTpVPWW4R7pVP VR7p\P P VVVVVVR7GRjxL #L5i)N)rrrPrQ)rQ)rrrPclient_opener_corostorage_client_cache_key)r get_configurationget_storage_clientcreate_rq_clientget_storage_client_cache_keystorage_instance_manageropen_storage_instance)clsrrrPrQrRrUadditional_cache_keys&$$$$$ r%openRequestQueue.openus@M?T99;Zg AOAW;;=]k+<<UZ<x-JJYfJg$==SS 1%9 T    sBB B B cV^8dQhRR/#rr!r"r#)r$s"r%r&r's""D"r(c "\PpVPV4VPP 4GRjxL R#L5ir6)r r[remove_from_cacher*drop)r1r[s& r%reRequestQueue.drops7$3#K#K  2248ll!!!s?A AA cV^8dQhRR/#rbr#)r$s"r%r&r's##T#r(c V"VPP4GRjxL R#L5ir6)r*purger7s&r%riRequestQueue.purgesll  """s )') forefrontFc$V^8dQhRRRRRR/#)rrequestz str | Requestrkboolr!rr#)r$s"r%r&r's(... .  .r(c "VPV4pVPPV.VR7GRjxL pVP^,#L5i)rkN)_transform_requestr*add_batch_of_requestsprocessed_requests)r1rmrkresponses&&$ r% add_requestRequestQueue.add_requestsM))'2;;WIQZ;[[**1--\s3AA A batch_sizeiwait_time_between_batches)seconds!wait_for_all_requests_to_be_added)wait_for_all_requests_to_be_added_timeoutc4V^8dQhRRRRRRRRR RR R R R /#)rrequestszSequence[str | Request]rkrnrwrCrxrrzr{ztimedelta | Noner!r"r#)r$s"r%r&r'sQ00)0 0  0 $- 0,004D0 0r(c aaaaa a a "SPV4o SP4o S RSpV'dSPVSSR7GRjxL RVVVV VV 3Rllp\P!V!4RR7o SP P S 4S PV V3Rl4V'd\S 3\VR7GRjxL R#R#LL 5i)Nbase_retry_waitrkcV^8dQhRR/#rbr#)r$s"r%r&/RequestQueue.add_requests..__annotate__s 8 8$ 8r(c<"\S\S4S4FapSWS,pSPVSSR7GRjxL VS,\S48gKC\P!S4GRjxL Kc R#LAL 5i)rN)rangelen_process_batchr.sleep)ibatchrwrkr1transformed_requestsrxwait_time_secss r%_process_remaining_batches=RequestQueue.add_requests.._process_remaining_batchess:s+?'@*M,QZ@))$='* z>C(<$==!--777N 8s(+RequestQueue.add_requests..s4;S;S;Z;Z[q;rr()loggertimeout) _transform_requests total_secondsrr. create_taskr0appendadd_done_callbackr r) r1r}rkrwrxrzr{ first_batchrrrrs f&ddd$$ @@@r% add_requestsRequestQueue.add_requestss $77A2@@B+;J7 %% 9#&   8 8")!4!4 & (?"    ''(>?001rs -/')A   -7 8 s+ACCA'C8CC CCcV^8dQhRR/#)rr!Request | Noner#)r$s"r%r&r's77.7r(c R"VPP4GRjxL #L5i)aReturn the next request in the queue to be processed. Once you successfully finish processing of the request, you need to call `RequestQueue.mark_request_as_handled` to mark the request as handled in the queue. If there was some error in processing the request, call `RequestQueue.reclaim_request` instead, so that the queue will give the request to some other consumer in another call to the `fetch_next_request` method. Note that the `None` return value does not mean the queue processing finished, it means there are currently no pending requests. To check whether all requests in queue were finished, use `RequestQueue.is_finished` instead. Returns: The next request to process, or `None` if there are no more pending requests. N)r*fetch_next_requestr7s&r%rRequestQueue.fetch_next_requests \\446666r@c V^8dQhRRRR/#)r unique_keyrr!rr#)r$s"r%r&r's : :C :N :r(c T"VPPV4GRjxL #L5i)zRetrieve a specific request from the queue by its ID. Args: unique_key: Unique key of the request to retrieve. Returns: The request with the specified ID, or `None` if no such request exists. N)r* get_request)r1rs&&r%rRequestQueue.get_requests"\\--j9999 (&(c V^8dQhRRRR/#)rrmr r!ProcessedRequest | Noner#)r$s"r%r&r's C CW CAX Cr(c T"VPPV4GRjxL #L5i)aMark a request as handled after successful processing. This method should be called after a request has been successfully processed. Once marked as handled, the request will be removed from the queue and will not be returned in subsequent calls to `fetch_next_request` method. Args: request: The request to mark as handled. Returns: Information about the queue operation. N)r*mark_request_as_handled)r1rms&&r%r$RequestQueue.mark_request_as_handleds"\\99'BBBBrc$V^8dQhRRRRRR/#)rrmr rkrnr!rr#)r$s"r%r&r's-PPP P ! Pr(c V"VPPWR7GRjxL #L5i)aReclaim a failed request back to the queue for later processing. If a request fails during processing, this method can be used to return it to the queue. The request will be returned for processing again in a subsequent call to `RequestQueue.fetch_next_request`. Args: request: The request to return to the queue. forefront: If true, the request will be added to the beginning of the queue. Otherwise, it will be added to the end. Returns: Information about the queue operation. rpN)r*reclaim_request)r1rmrks&&$r%rRequestQueue.reclaim_requests%(\\11'1OOOOs )')cV^8dQhRR/#rr!rnr#)r$s"r%r&r's - - -r(c R"VPP4GRjxL #L5i)atCheck if the request queue is empty. An empty queue means that there are no requests currently in the queue, either pending or being processed. However, this does not necessarily mean that the crawling operation is finished, as there still might be tasks that could add additional requests to the queue. Returns: True if the request queue is empty, False otherwise. N)r*is_emptyr7s&r%rRequestQueue.is_emptys \\**,,,,r@cV^8dQhRR/#rr#)r$s"r%r&r'&s4r(c "VP'd\PR4R#VP4GRjxL 'd\PR4R#R#L#5i)aCheck if the request queue is finished. A finished queue means that all requests in the queue have been processed (the queue is empty) and there are no more tasks that could add additional requests to the queue. This is the definitive way to check if a crawling operation is complete. Returns: True if the request queue is finished (empty and no pending add operations), False otherwise. z4Background add requests tasks are still in progress.FNzThe request queue is empty.T)r0rdebugrr7s&r% is_finishedRequestQueue.is_finished&sJ  # # # LLO P LL6 7 !s=A%A# A% A%attemptc ,V^8dQhRRRRRRRRR R /#) rrzSequence[Request]rrrrCrkrnr!r"r#)r$s"r%r&r':s< #      r(c H"^pVPPWR7GRjxL pVP'Ed\P RVP R24W58d'\P RV RVP 24M\P R4VPUu0uFqwP kK ppVUu.uFqwP V9gKVNK p p\P!W#,P44GRjxL VPWV^,R7GRjxL \V4\VP4, p V 'd(\P R V R VP 24R#R#ELmuupiuupiLLi5i) z;Process a batch of requests with automatic retry mechanism.rpNz'Following requests were not processed: .z1Following requests were not processed even after z attempts: zRetry to add requests.)rrzAdded z, requests to the queue. Processed requests: ) r*rrunprocessed_requestsrrwarningrr.rrrrrs) r1rrrrk max_attemptsrtrm unprocessed_requests_unique_keys retry_batch request_counts &&$$$ r%rRequestQueue._process_batch:sh ;;E;WW  ( ( ( LLB8C`C`Baabc d%G ~Ua4457  56V^VsVs3tVs74F4FVs03t6;ve7?Q?QUu?uwwe vmm_%>$M$M$OPPP))+`gjk`k)lllE S)F)F%GG  LL'ST\ToToSpq  %X4uvPlsX"F"FF"A2F".FF"FF%/F"F!F"6F 7AF" F" F")r0r*r+r,)__name__ __module__ __qualname____firstlineno____doc__r2propertyrrrr>rHrN classmethodr_rerirurrrrrrrrr__static_attributes__r#r(r%rr s&P@&  11..,,    !  /3  04  0""##. ..0 0  0 09/C 0380GK00d7" : CP P, -(    r(r)( __future__rr.datetimerloggingrtypingrrtyping_extensionsrcrawleer r crawlee._utils.docsr crawlee._utils.waitr crawlee.request_loadersr _baser_utilsrcollections.abcrcrawlee.configurationrcrawlee.storage_clientsrcrawlee.storage_clients._basercrawlee.storage_clients.modelsrrrrrrr#r(r%rsv")&,*=2)(35@U 8  CL Jx7Nxxr(