*SrSSKrSSKrSSKrSSKJr SSKJr SSKJr SSK J r SSK J r SSK Jr SSKJr SSKJr SS KJr SS KJr SSKrS rS r\R2R4R6R8R;\5 \R2R4R6R<R;\5 SS jrSSjr Sr!SSjr"SSjr#SSjr$Sr%"SS\&5r'"SS\&5r(g)zHelpers for accessing GCS. Bulk object uploads and downloads use methods that shell out to gsutil. Lightweight metadata / streaming operations use the StorageClient class. N) exceptions)transfer) storage_api) storage_util)apis)log) properties) resourcesgs<c[RRRR 5nU(a [ X5 g[ XUS9 g)N)storage_client)r VALUESstorage use_gsutilGetBool _UploadGsutil_UploadStorageClient)files destinationrrs 6lib/googlecloudsdk/api_lib/dataproc/storage_helpers.pyUploadr5s9  ((33;;=*%%NKc U=(d [R"5nUHrn[RR U5n[RR X5n[ RRU5nURXG5 Mt g![Ra:n[R"SRSR U5X55eSnAff=f)zUpload a list of local files to GCS. Args: files: The list of local files to upload. destination: A GCS "directory" to copy the files into. storage_client: Storage api client used to copy files to gcs. z)Failed to upload files ['{}'] to '{}': {}', 'N)r StorageClientospathbasenamejoinrObjectReferenceFromUrl CopyFileToGCSrBadFileException dp_exceptionsFileUploadErrorformat) rrrclientfile_to_upload file_namedest_url dest_objecterrs rrr?s  8[668&n  0Iww||K3H..66x@K5 >7   & &5  ) ) 5 < <kk% + 4 555sBC&,5C!!C&cUnX!/- n[R"SU5nUS:wa5[R"SR SR U5U55eg)zUpload a list of local files to GCS. Args: files: The list of local files to upload. destination: A GCS "directory" to copy the files into. cprz5Failed to upload files ['{0}'] to '{1}' using gsutil.rN)rRunGsutilCommandr%r&r'r )rrargs exit_codes rrrTs` $-$++D$7)!^  ' '?FF KK   - ..rcU=(d [R"5nURU5$![Ra gf=f)zGets a bucket if it exists. Args: bucket: The bucket name. storage_client: Storage client instance. Returns: A bucket message, or None if it doesn't exist. N)rr GetBucketBucketNotFoundError)bucketrr(s rr4r4dsE  8[668&   F ##  ( ( s1AAc`U=(d [R"5nURXUS9 g)a[Creates a bucket. Creates a bucket in the specified region. If the region is None, the bucket will be created in global region. Args: bucket: Name of bucket to create. region: Region to create bucket in. storage_client: Storage client instance. project: The project to create the bucket in. If None, current Cloud SDK project is used. )locationprojectN)rrCreateBucketIfNotExists)r6regionrr9r(s rr:r:vs*  8[668&  ' JrcdU=(d [R"5n[RR U5nUR U5n[ R"USS9nUR5$![Ra& [R"SRU55ef=f)a9Reads an object's content from GCS. Args: object_url: The URL of the object to be read. Must have "gs://" prefix. storage_client: Storage api client used to read files from gcs. Raises: ObjectReadError: If the read of GCS object is not successful. Returns: A str for the content of the GCS object. zutf-8)encodingzFailed to read file '{0}'.)rrrr!r" ReadObjectio TextIOWrapperreadrr$r%ObjectReadErrorr') object_urlrr( object_refbytes_iowrappers rr>r>s  8[668&++33J?*9  ,Hx':G <<>  $ $9  ' '$++J7 999s 5A55:B/c[RRU5nURURUR S9$)z.Build an Object proto message from a GCS path.)r6name)r REGISTRYParseStorageURLObjectr6object)rmessagesresources r GetObjectRefrOs3    / / 5( hoo FFrc4\rSrSrSrSrS SjrSrSrSr g) rzMicro-client for accessing GCS.ct[R"SS5Ul[R"SS5Ulg)Nrv1) core_apisGetClientInstancer(GetMessagesModulerMselfs r__init__StorageClient.__init__s*--i>DK// 4@DMrNcURRURURS9nURR R X2S9$![Ra gf=f)N)r6rL)requestdownload) rMStorageObjectsGetRequestr6rHr(objectsGetapitools_exceptionsHttpNotFoundError)rXrDr]r\s r _GetObjectStorageClient._GetObjectsemm44  5:G [[ $ $W $ HH  0 0 s"AA*)A*c$URU5$)a Get the object metadata of a GCS object. Args: object_ref: A proto message of the object to fetch. Only the bucket and name need be set. Raises: HttpError: If the responses status is not 2xx or 404. Returns: The object if it exists otherwise None. )rc)rXrDs r GetObjectStorageClient.GetObjects ??: &&rct[RRXRSS9nUR X#S9 U$)aBuild an apitools Download from a stream and a GCS object reference. Note: This will always succeed, but HttpErrors with downloading will be raised when the download's methods are called. Args: stream: An Stream-like object that implements write() to write into. object_ref: A proto message of the object to fetch. Only the bucket and name need be set. Returns: The download. F) total_size auto_transfer)r])rDownload FromStreamsizerc)rXstreamrDr]s rBuildObjectStreamStorageClient.BuildObjectStreams<  ++??%,AHOOJO2 Or)r(rMN) __name__ __module__ __qualname____firstlineno____doc__rYrcrfro__static_attributes__rrrrs'A' rrcd\rSrSrSrS Sjr\S5rSrSr Sr \ R4S jr S rg) StorageObjectSeriesStreamzFI/O Stream-like class for communicating via a sequence of GCS objects.NcjXlU=(d [5UlSUlSUlSUlg)aConstruct a StorageObjectSeriesStream for a specific gcs path. Args: path: A GCS object prefix which will be the base of the objects used to communicate across the channel. storage_client: a StorageClient for accessing GCS. Returns: The constructed stream. TrN) _base_pathr_gcs_open_current_object_index_current_object_pos)rXrrs rrY"StorageObjectSeriesStream.__init__s2O1-/DIDJ"#D !DrcUR$)zWhether the stream is open.rrWs ropenStorageObjectSeriesStream.opens ::rcSUlg)zClose the stream.FNrrWs rCloseStorageObjectSeriesStream.Closes DJrc<UR(d [S5eg)NzI/O operation on closed stream.)r ValueErrorrWs r _AssertOpen%StorageObjectSeriesStream._AssertOpens 99 8 99 rcSRURU5nURR[ X RR 55$)z!Get the ith object in the series.z {0}.{1:09d})r'r}r~rfrOrM)rXirs rrc$StorageObjectSeriesStream._GetObjects<    3D 99  |D))2D2DE FFrcUR5 SnSnUnX5:GazURURS-5nU(aU(a&URUR5nU(dU$URUR- nUS:a$[SRUR55eURS:XaUR5 U$XS- n [X5n U S:a`URRX5n U R!URURU -S- 5 U=RU - slX:- nU=(a URUR:Hn U (a!UnU=RS- slSUlGM}U$U$![Ra"n[ R "SU5 SnAU$SnAff=f)aORead from this stream into a writable. Reads at most n bytes, or until it sees there is not a next object in the series. This will block for the duration of each object's download, and possibly indefinitely if new objects are being added to the channel frequently enough. Args: writable: The stream-like object that implements write() to write into. n: A maximum number of bytes to read. Defaults to sys.maxsize (usually ~4 GB). Raises: ValueError: If the stream is closed or objects in the series are detected to shrink. Returns: The number of bytes read. rNzFailed to fetch GCS output: %szObject [{0}] shrunk.)rrcrra HttpErrorrwarningrmrrr'rHrminr~roGetRange) rXwritablen bytes_read object_infomax_bytes_to_readnext_object_infoerrornew_bytes_availablebytes_left_to_readnew_bytes_to_readr]object_finisheds rReadIntoWritable*StorageObjectSeriesStream.ReadIntoWritables* JK  ()C)Ca)GH, (B(BC+ L I(,,t/G/GG q /66{7G7GHII   Q   8 5-90F Q 99..xE  $ $  $ $'8 81 < >   $55 '  Kt77;;K;KK &  ""a'"#$   :W#,,  ++7 ? R W s FG-G  G)r}rrr~rrq)rrrsrtrurvrYpropertyrrrrcsysmaxsizerrwrxrrrzrzs=N!*  :G *-Lrrzrq)NN))rvr?rrapitools.base.pyrrargooglecloudsdk.api_lib.dataprocr%googlecloudsdk.api_lib.storagerrgooglecloudsdk.api_lib.utilrrTgooglecloudsdk.calliopegooglecloudsdk.corerr r six.moves.urllib.parsesixSTORAGE_SCHEME HTTP_TIMEOUTmovesurllibparse uses_relativeappend uses_netlocrrrr4r:r>rOrLrrzrxrrrs  >%G679.#*)  $$++N; "")).9L5* . $K"92G 4F4nvvr