S3(3tcl)

📦
version 2.0 · 2 versions
, in debian trixie, ubuntu questing, ubuntu resolute, ubuntu stonking

💖 S3 — Amazon S3 Web Service Interface 💖

okay bestie, welcome to S3(3tcl), a.k.a. the Amazon S3 Web Service Utilities page. this is the whole tour of the Tcl package that lets you talk to Amazon's Simple Storage Solution, and it is long, so hydrate 💅

✨ SYNOPSIS ✨

package require Tcl 8.5 9

package require S3 ?1.0.5?

package require sha1 1.1

package require md5 2.1

package require base64 2.4

package require xsxp 1.1

S3::Configure ?-reset boolean? ?-retries integer? ?-accesskeyid idstring? ?-secretaccesskey idstring? ?-service-access-point FQDN? ?-use-tls boolean? ?-default-compare always|never|exists|missing|newer|date|checksum|different? ?-default-separator string? ?-default-acl private|public-read|public-read-write|authenticated-read|keep|calc? ?-default-bucket bucketname?

S3::SuggestBucket ?name?

S3::REST dict

S3::ListAllMyBuckets ?-blocking boolean? ?-parse-xml xmlstring? ?-result-type REST|xml|pxml|dict|names|owner?

S3::PutBucket ?-bucket bucketname? ?-blocking boolean? ?-acl {}|private|public-read|public-read-write|authenticated-read?

S3::DeleteBucket ?-bucket bucketname? ?-blocking boolean?

S3::GetBucket ?-bucket bucketname? ?-blocking boolean? ?-parse-xml xmlstring? ?-max-count integer? ?-prefix prefixstring? ?-delimiter delimiterstring? ?-result-type REST|xml|pxml|names|dict?

S3::Put ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-file filename? ?-content contentstring? ?-acl private|public-read|public-read-write|authenticated-read|calc|keep? ?-content-type contenttypestring? ?-x-amz-meta-* metadatatext? ?-compare comparemode?

S3::Get ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-compare comparemode? ?-file filename? ?-content contentvarname? ?-timestamp aws|now? ?-headers headervarname?

S3::Head ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-dict dictvarname? ?-headers headersvarname? ?-status statusvarname?

S3::GetAcl ?-blocking boolean? ?-bucket bucketname? -resource resourcename ?-result-type REST|xml|pxml?

S3::PutAcl ?-blocking boolean? ?-bucket bucketname? -resource resourcename ?-acl new-acl?

S3::Delete ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-status statusvar?

S3::Push ?-bucket bucketname? -directory directoryname ?-prefix prefixstring? ?-compare comparemode? ?-x-amz-meta-* metastring? ?-acl aclcode? ?-delete boolean? ?-error throw|break|continue? ?-progress scriptprefix?

S3::Pull ?-bucket bucketname? -directory directoryname ?-prefix prefixstring? ?-blocking boolean? ?-compare comparemode? ?-delete boolean? ?-timestamp aws|now? ?-error throw|break|continue? ?-progress scriptprefix?

S3::Toss ?-bucket bucketname? -prefix prefixstring ?-blocking boolean? ?-error throw|break|continue? ?-progress scriptprefix?

🌸 DESCRIPTION 🌸

so this package is your all-access pass to Amazon's Simple Storage Solution web service. periodt.

quick rundown, bestie: Amazon Simple Storage Solution is a for-fee web service that lets you store arbitrary data as "resources" inside "buckets" online. you want the deets on that whole system, go look at http://www.amazonaws.com/. you reach the service over HTTP, either SOAP or REST. and ngl, a whole lot of this documentation is gonna be giving ✨confusion✨ if you're not already familiar with the terms and functionality of the Amazon S3 service. iykyk.

this package handles reading and writing your data items through the REST interface, and it also throws in some higher-level operations for you. and there are other packages in the same distribution that do even more, so stay tuned 🫶

Copyright 2006 Darren New. All Rights Reserved. NO WARRANTIES OF ANY TYPE ARE PROVIDED. COPYING OR USE INDEMNIFIES THE AUTHOR IN ALL WAYS. This software is licensed under essentially the same terms as Tcl. See LICENSE.txt for the terms.

💀 ERROR REPORTING 💀

the way this package reports errors?? it's giving extra detail. it uses $errorCode so you get way more than just a thrown error and a shrug. any error the S3 package catches (and bestie, it tries to catch them all) comes back with an $errorCode that's a list of at least three elements. here's the tea:

  • the first element is always "S3".
  • the second element is one of six values, and whichever one it is decides what the third and following elements mean.

one exception, and it matters: S3::REST does not throw an error. instead it returns a dictionary with the keys "error", "errorInfo", and "errorCode" set. this is what makes reliable background use possible, no cap.

so here are the six possible second elements:

  • usage — you used the package wrong. like, a command got invoked that needs the library to be configured first, but it wasn't configured yet, or you passed an invalid combination of options. the third element of $errorCode gives you the name of the parameter that was wrong, and the fourth usually gives you the arguments that were actually supplied to the throwing proc — unless the usage error isn't confined to a single proc, in which case, shrug 🙈
  • local — something happened on your system that threw an error. for example, you asked to upload or download a file and the file permissions said "absolutely not". the third element of $errorCode is the original $errorCode.
  • socket — something went sideways with the socket. it closed prematurely, or some other condition of failure-to-communicate-with-Amazon got detected. the third element of $errorCode is the original $errorCode, or sometimes the message from fcopy, or ...? (yes, the docs literally say "or ...?" 💀)
  • remote — the Amazon web service returned an error code outside the 2xx range in the HTTP header. meaning: everything went exactly as documented, except this particular case was documented not to work. the third element is the dictionary returned from ::S3::REST. note that S3::REST itself never throws this error — it just returns the dictionary. most of the higher-level commands do throw it, for convenience, unless an argument says they shouldn't. so if something is documented as "not throwing an S3 remote error", that means a status return gets set instead of throwing when Amazon hands back a non-2XX HTTP result code.
  • notyet — you followed the documentation perfectly, and the author just hasn't gotten around to implementing this feature yet 😭 (right now, only TLS support and sophisticated permissions fall into this category, plus the S3::Acl command.)
  • xml — the service returned invalid XML, or XML whose schema is unexpected. for the high-level commands that take service XML as input for parsing, this one can get thrown too.

👑 COMMANDS 👑

this package comes in several separate levels of complexity, and honestly, it's a whole ladder:

  • the lowest level just takes arguments to send to the service, sends them, grabs the result, and hands it back to you. heads up: this layer allows both synchronous and event-driven processing. it depends on the MD5 and SHA1 and base64 packages from Tcllib (available at http://core.tcl.tk/tcllib/). and note that S3::Configure is required for S3::REST to work, because of the authentication portion — which is why we count it as part of the "lowest level."
  • the next layer parses the results of calls, which unlocks functionality like uploading only changed files, synchronizing directories, and so on. this layer depends on the TclXML package plus the included xsxp package. these packages are package required only when these more-sophisticated routines get called, so nothing breaks if they aren't correctly installed. we love a graceful degrade 💅
  • also included is a separate program that uses the library. it provides code to parse $argv0 and $argv from the command line, so you can invoke it as a tclkit, etc. (Not yet implmented.)
  • another separate program provides a GUI interface allowing drag-and-drop and other such functionality. (Not yet implemented.)
  • also built on this package is the OddJob program. it's a separate program designed to allow distribution of computational work units over Amazon's Elastic Compute Cloud web service.

the goal is to have at least the bottom-most layers implemented in pure Tcl, using only stuff that comes from widely-available sources, such as Tcllib. 🎀

🦋 LOW LEVEL COMMANDS 🦋

these commands don't require any packages beyond the ones listed above. they talk directly to the service, or they're utility or configuration routines. also, fun fact: the xsxp package was written specifically to support this package, so it should be available wherever you got this package from. slay.

⭐ S3::Configure

S3::Configure ?-reset boolean? ?-retries integer? ?-accesskeyid idstring? ?-secretaccesskey idstring? ?-service-access-point FQDN? ?-use-tls boolean? ?-default-compare always|never|exists|missing|newer|date|checksum|different? ?-default-separator string? ?-default-acl private|public-read|public-read-write|authenticated-read|keep|calc? ?-default-bucket bucketname?

there's exactly one command for configuration, and bestie, this is it: S3::Configure. here's how it behaves:

  • call it with no arguments → it returns a dictionary of key/value pairs listing all the current settings.
  • call it with one argument → it returns the value of that single argument.
  • call it with two or more arguments → it must be called with pairs of arguments, and it applies the changes in order.

and important: there is only one set of configuration information per interpreter. 👑

the following options are accepted:

  • -reset boolean — default is false. if true, any previous changes and any changes made on the same call before the reset option get returned to default values. so it's like a lil undo button for everything before it 🫶
  • -retries integer — default value is 3. if Amazon returns a 500 error, a retry after an exponential backoff delay gets tried this many times before it finally throws the 500 error. this applies to each call to S3::REST made from the higher-level commands, but not to S3::REST itself. meaning: S3::REST will always return httpstatus 500 if that's what it receives. functions like S3::Put will retry the PUT call, and will also retry the GET and HEAD calls used to do content comparison. changing this to 0 will prevent retries and their associated delays. plus, socket errors (i.e., errors whose errorCode starts with "S3 socket") get similarly retried after backoffs. ✨
  • -accesskeyid idstring — your access key ID. Defaults to an empty string.
  • -secretaccesskey idstring — your secret access key. Defaults to an empty string.

Both of these have to be set before any calls are made, periodt. This is literally your S3 ID. Once you sign up for an account, go to http://www.amazonaws.com/, sign in, hit the "Your Web Services Account" button, pick "AWS Access Identifiers", and boom — your access key ID and secret access keys will be available. Every single S3::REST call is authenticated. And yes, blame Amazon for the poor choice of names 🙈 — not our fault, bestie.

  • -service-access-point FQDN — the fully-qualified domain name of the server to contact for S3::REST calls. Defaults to "s3.amazonaws.com".

Honestly you should probably never need to touch this 💅 — unless someone else implements a compatible service, or you wanna test something by pointing the library at your own service. Main character experimentation energy, we love it.

  • -slop-seconds integer — when comparing dates between Amazon and the local machine, two dates within this many seconds of each other are considered the same. Useful for clock drift correction, processing overhead time, and so on. ✨
  • -use-tls boolean — defaults to false. This is not yet implemented. If true, S3::REST will negotiate a TLS connection to Amazon. If false, unencrypted connections are used.
🚨 bestie hear me — the true branch is literally not implemented yet, so don't be surprised when it doesn't do the thing.
  • -bucket-prefix string — defaults to "TclS3". This string is used by S3::SuggestBucket if that command is passed an empty string as an argument. It's there to distinguish different applications using the Amazon service. Your application should always set this so you don't interfere with the buckets of other users of Amazon S3, or with other buckets of the same user. fr fr, set it. 🙏
  • -default-compare always|never|exists|missing|newer|date|checksum|different — defaults to "always". If no -compare is specified on S3::Put, S3::Get, or S3::Delete, this comparison is used. See those commands for a description of the meaning. 👑
  • -default-separator string — defaults to "/". This is currently unused. It might make sense to use this for S3::Push and S3::Pull, but allowing resources to have slashes in their names that aren't marking directories would be problematic. Hence, this currently does nothing. 💀 iconic of it honestly.
  • -default-acl private|public-read|public-read-write|authenticated-read|keep|calc — defaults to an empty string. If no -acl argument is provided to S3::Put or S3::Push, this string is used (given as the x-amz-acl header if not keep or calc). If this is also empty, no x-amz-acl header is generated. This is not used by S3::REST.
  • -default-bucket bucketname — if no bucket is given to S3::GetBucket, S3::PutBucket, S3::Get, S3::Put, S3::Head, S3::Acl, S3::Delete, S3::Push, S3::Pull, or S3::Toss, and if this configuration variable is not an empty string (and not simply "/"), then this value will be used for the bucket. This is useful if one program does a large amount of resource manipulation within a single bucket. 🎀

🌸 S3::SuggestBucket 🌸

S3::SuggestBucket ?name?

The S3::SuggestBucket command accepts an optional string as a prefix and returns a valid bucket containing the name argument and the Access Key ID. This makes the name unique to the owner and to the application (assuming the application picks a good name argument, so like... choose wisely bestie 🫶). If no name is provided, the name from S3::Configure -bucket-prefix is used. If that too is empty (which is not the default), an error is thrown. 😭

👑 S3::REST 👑

S3::REST dict

The S3::REST command takes as an argument a dictionary and returns a dictionary. The return dictionary has the same keys as the input dictionary, and includes additional keys as the result. The presence or absence of keys in the input dictionary can control the behavior of the routine. It never throws an error directly, but includes keys "error", "errorInfo", and "errorCode" if necessary. Some keys are required, some optional. The routine can run either in blocking or non-blocking mode, based on the presense of resultvar in the input dictionary. This requires the -accesskeyid and -secretaccesskey to be configured via S3::Configure before being called.

The possible input keys are these:

  • verb GET|PUT|DELETE|HEAD — this required item indicates the verb to be used.
  • resource string — this required item indicates the resource to be accessed. A leading / is added if not there already. It will be URL-encoded for you if necessary. Do not supply a resource name that is already URL-encoded.
  • ?rtype torrent|acl? — this indicates a torrent or acl resource is being manipulated. Do not include this in the resource key, or the "?" separator will get URL-encoded. 🙈
  • ?parameters dict? — this optional dictionary provides parameters added to the URL for the transaction. The keys must be in the correct case (which is confusing in the Amazon documentation 💀) and the values must be valid. This can be an empty dictionary or omitted entirely if no parameters are desired. No other error checking on parameters is performed.
  • ?headers dict? — this optional dictionary provides headers to be added to the HTTP request. The keys must be in lower case for the authentication to work. The values must not contain embedded newlines or carriage returns. This is primarily useful for adding x-amz-* headers. Since authentication is calculated by S3::REST, do not add that header here. Since content-type gets its own key, also do not add that header here.
  • ?inbody contentstring? — this optional item, if provided, gives the content that will be sent. It is sent with a tranfer encoding of binary, and only the low bytes are used, so use [encoding convertto utf-8] if the string is a utf-8 string. This is written all in one blast, so if you are using non-blocking mode and the inbody is especially large, you may wind up blocking on the write socket. 😬
  • ?infile filename? — this optional item, if provided, and if inbody is not provided, names the file from which the body of the HTTP message will be constructed. The file is opened for reading and sent progressively by [fcopy], so it should not block in non-blocking mode even if the file is very large. The file is transfered in binary mode, so the bytes on your disk will match the bytes in your resource. Due to HTTP restrictions, it must be possible to use [file size] on this file to determine the size at the start of the transaction. ✨
  • ?S3chan channel? — this optional item, if provided, indicates the already-open socket over which the transaction should be conducted. If not provided, a connection is made to the service access point specified via S3::Configure, which is normally s3.amazonaws.com. If this is provided, the channel is not closed at the end of the transaction.
  • ?outchan channel? — this optional item, if provided, indicates the already-open channel to which the body returned from S3 should be written. That is, to retrieve a large resource, open a file, set the translation mode, and pass the channel as the value of the key outchan. Output will be written to the channel in pieces so memory does not fill up unnecessarily. The channel is not closed at the end of the transaction.
  • ?resultvar varname? — this optional item, if provided, indicates that S3::REST should run in non-blocking mode. The varname should be fully qualified with respect to namespaces and cannot be local to a proc. If provided, the result of the S3::REST call is assigned to this variable once everything has completed; use trace or vwait to know when this has happened. If this key is not provided, the result is simply returned from the call to S3::REST and no calls to the eventloop are invoked from within this call.
  • ?throwsocket throw|return? — this optional item, if provided, indicates that S3::REST should throw an error if throwmode is throw and a socket error is encountered. It indicates that S3::REST should return the error code in the returned dictionary if a socket error is encountered and this is set to return. If throwsocket is set to return or if the call is not blocking, then a socket error (i.e., an error whose error code starts with "S3 socket") will be returned in the dictionary as error, errorInfo, and errorCode. If a foreground call is made (i.e., resultvar is not provided), and this option is not provided or is set to throw, then error will be invoked instead. 💥

Once the call to S3::REST completes, a new dict is returned, either in the resultvar or as the result of execution. This dict is a copy of the original dict with the results added as new keys. The possible new keys are these:

  • error errorstring
  • errorInfo errorstring
  • errorCode errorstring

If an error is caught, these three keys will be set in the result. Note that S3::REST does not consider a non-2XX HTTP return code as an error. The errorCode value will be formatted according to the ERROR REPORTING description. If these are present, other keys described here might not be. 🦋

  • httpstatus threedigits — the three-digit code straight outta the HTTP transaction. 2XX means we ate, 5XX means the server is having a whole crisis, etc.
  • httpmessage text — the textual result that shows up right after the status code. Like "OK" or "Forbidden" or whatever it's giving.
  • outbody contentstring — if outchan was not specified, this key holds a reference to the (unencoded) contents of the body that came back. And bestie, if Amazon returned an error (as in, the httpstatus is not a 2XX value), that error message will be sitting in outbody or written to outchan, whichever applies.
  • outheaders dict — this one holds a dictionary of headers returned by Amazon. The keys are always lower case. It's mostly the move for finding the x-amz-meta-* headers if there are any, though stuff like last-modified and content-type is also useful. The keys of this dictionary are always lower case (yes, saying it twice, it matters 💅). Both keys and values get trimmed of extraneous whitespace.

👑 HIGH LEVEL COMMANDS 👑

Okay so the routines in this section all make use of one or more calls to S3::REST to do their thing, then they parse and manage the data in a way that's actually convenient. All of these commands throw errors exactly as described in ERROR REPORTING unless it says otherwise.

In every single one of these commands, all arguments are given as name/value pairs, in any order. And every argument name starts with a hyphen.

There are a few options that are common to a bunch of these commands, and those common ones are documented right here so we don't have to repeat ourselves forty times.

  • -blocking boolean — if you pass this and set it to false, then any calls to S3:REST will be non-blocking, and internally these routines will call [vwait] to get the results. Translation: the routines still return the same value, but there'll be event loops running the whole time they're waiting on Amazon. Multitasking queen behavior. ✨
  • -parse-xml xmlstring — if you pass this, the routine skips actually talking to Amazon entirely, and instead acts like the XML string you handed it was the body of the call. Since several of these routines let you get data back in different formats, this argument is how you parse existing XML to pull out just the bits you need. Also super helpful for testing, no cap.
  • -bucket bucketname — basically every high-level command needs to know which bucket the resources live in, and this is where you tell it. (Only the command that lists available buckets doesn't need this one.) You do not need to URL-encode it, even if it's got special or non-ASCII characters. It may or may not have leading or trailing spaces — the commands normalize the bucket for you. If you don't supply it, the value comes from S3::Configure -default-bucket if that string isn't empty. Heads up: spaces and slashes are always trimmed off both ends, and what's left has to be a valid bucket.
  • -resource resourcename — this is the resource you care about inside the bucket. It may or may not start with a slash — both cases are handled, we're not picky. And no, you don't need to URL-encode it, even with special or non-ASCII characters in there.
  • -compare always|never|exists|missing|newer|date|checksum|different — when commands copy resources to files or files to resources, you get to say the copy should be skipped if the contents are the same. This argument sets the conditions under which the files get copied. If you don't pass it, the result of S3::Configure -default-compare is used, which itself defaults to "always." Here's what all the values actually mean:
    • always — always copy the data. This is the default.
    • never — never copy the data. This is basically a no-op, except in S3::Push and S3::Pull where the -delete flag might still make a difference.
    • exists — copy the data only if the destination already exists.
    • missing — copy the data only if the destination doesn't already exist.
    • newer — copy the data if the destination is missing, or if the date on the source is newer than the date on the destination by at least S3::Configure -slop-seconds seconds. If the source is Amazon, the date comes from the Last-Modified header. If the source is local, it's the mtime of the file. And if the source data is given as a string instead of a file, it's treated as right now, via [clock seconds].
    • date — like newer, except it copies if the date is newer or older.
    • checksum — calculate the MD5 checksum on the local file or string, ask Amazon for the eTag of the resource, and copy the data if they're different. Also copies if the destination is missing. Heads up though: this can be slow with big local files unless the C version of the MD5 support is available. 🐌
    • different — copy the data if the destination doesn't exist. If the destination does exist and an actual file name was specified (not a content string), and the date on the file differs from the date on the resource, copy the data. If the data is provided as a content string, the "date" is treated as "right now", so it'll probably always differ unless slop-seconds is large. And if the dates are the same, the MD5 checksums get compared, and the data is copied if the checksums differ.
Quick vibe check on all that: "newer" and "date" don't care about the contents, and "checksum" doesn't care about the dates, but "different" checks both. 💅

S3::ListAllMyBuckets ?-blocking boolean? ?-parse-xml xmlstring? ?-result-type REST|xml|pxml|dict|names|owner?

This routine performs a GET on the Amazon S3 service, which is defined to return a list of buckets owned by the account identified by the authorization header. (Blame Amazon for the dumb names, not us.)

  • -blocking boolean — see above for the standard definition.
  • -parse-xml xmlstring — see above for the standard definition.
  • -result-type REST — the dictionary returned by S3::REST is the return value of S3::ListAllMyBuckets. In this case, a non-2XX httpstatus will not throw an error. You may not combine this with -parse-xml.
  • -result-type xml — the raw XML of the body is returned as the result (with no encoding applied).
  • -result-type pxml — the XML of the body as parsed by xsxp::parse is returned.
  • -result-type dict — a dictionary of the interesting portions of the XML is returned. That dictionary contains these keys:
    • Owner/ID — the Amazon AWS ID (in hex) of the owner of the bucket.
    • Owner/DisplayName — the Amazon AWS ID's Display Name.
    • Bucket/Name — a list of names, one for each bucket.
    • Bucket/CreationDate — a list of dates, one for each bucket, in the same order as Bucket/Name, in ISO format (as returned by Amazon).
  • -result-type names — a list of bucket names is returned with all the other information stripped out. This is the default result type for this command. 🎀
  • -result-type owner — a list containing two elements is returned. The first element is the owner's ID, and the second is the owner's display name.

S3::PutBucket ?-bucket bucketname? ?-blocking boolean? ?-acl {}|private|public-read|public-read-write|authenticated-read?

This command creates a bucket if it doesn't already exist. Bucket names are globally unique, so you may get a "Forbidden" error from Amazon even if you literally cannot see the bucket in S3::ListAllMyBuckets. Check out S3::SuggestBucket for ways to minimize that risk. The x-amz-acl header comes from the -acl option, or from S3::Configure -default-acl if you didn't specify one.

S3::DeleteBucket ?-bucket bucketname? ?-blocking boolean?

This command deletes a bucket if it's empty and you have the permission to do so. Okay so real talk: Amazon's list of buckets is a global resource, which means it needs far-flung synchronization. So if you delete a bucket, it may be quite a few minutes (or hours 😭) before you can recreate it, and you'll get "Conflict" errors until then. Patience, bestie.

S3::GetBucket ?-bucket bucketname? ?-blocking boolean? ?-parse-xml xmlstring? ?-max-count integer? ?-prefix prefixstring? ?-delimiter delimiterstring? ?-result-type REST|xml|pxml|names|dict?

This lists the contents of a bucket. As in, it gives you a directory listing of the resources inside a bucket — it does not transfer any user data. Just the receipts. 🧾

  • -bucket bucketname — the standard bucket argument.
  • -blocking boolean — the standard blocking argument.
  • -parse-xml xmlstring — the standard parse-xml argument.
  • -max-count integer — if you supply this, it's the max number of records to be returned. If you don't supply it, the code will iterate until all records have been found. Not compatible with -parse-xml. Note that if this is supplied, only one call to S3::REST will be made. Otherwise, enough calls get made to exhaust the listing, buffering results in memory — so take care if you might have absolutely massive buckets. 💀
  • -prefix prefixstring — if present, restricts the listing to resources with a particular prefix. One leading / gets stripped if it's there.
  • -delimiter delimiterstring — if present, specifies a delimiter for the listing. Having one of these summarizes multiple resources into one entry, as if S3 actually supported directories. See the Amazon documentation for the details.
  • -result-type REST|xml|pxml|names|dict — this indicates the format of the return result of the command.
  • REST — if -max-count is specified, you get back the dictionary that S3::REST returned. If -max-count is not specified, you get a list of all the dictionaries returned from the one-or-more calls to S3::REST. one call, many calls, whatever, you're covered 💖
  • xml — if -max-count is specified, you get the body that S3::REST returned. If -max-count is not specified, you get a list of all the bodies returned from the one-or-more calls to S3::REST.
  • pxml — if -max-count is specified, the body returned from S3::REST gets passed through xsxp::parse and then handed back to you. If -max-count is not specified, you get a list where every one of the bodies returned from the one-or-more calls to S3::REST has been passed through xsxp::parse. parsed and slayed ✨
  • names — returns a list of all the names found in either the Contents/Key fields or the CommonPrefixes/Prefix fields. And if there's no -delimiter specified and no -max-count specified, this hands you a list of all the resources with the -prefix you asked for. just names, no extra lore 🎀
  • dict — returns a dictionary. And yes, it returns only one dictionary even if -max-count wasn't specified. The keys of that dictionary are:
    • Name — the name of the bucket (from the final call to S3::REST).
    • Prefix — from the final call to S3::REST.
    • Marker — from the final call to S3::REST.
    • MaxKeys — from the final call to S3::REST.
    • IsTruncated — from the final call to S3::REST, which means it's always false if -max-count is not specified.
    • NextMarker — always provided if IsTruncated is true, and it gets calculated if Amazon does not provide it. May be empty if IsTruncated is false.
    • Key — a list of names of resources in the bucket matching the -prefix and -delimiter restrictions.
    • LastModified — a list of times of resources in the bucket, in the same order as Key, in the format returned by Amazon. (I.e., it is not parsed into a seconds-from-epoch. raw amazon time, we don't touch it 🙈)
    • ETag — a list of entity tags (a.k.a. MD5 checksums) in the same order as Key.
    • Size — a list of sizes in bytes of the resources, in the same order as Key.
    • Owner/ID — a list of owners of the resources in the bucket, in the same order as Key.
    • Owner/DisplayName — a list of owners of the resources in the bucket, in the same order as Key. These are the display names.
    • CommonPrefixes/Prefix — a list of prefixes common to multiple entities. This one is present only if -delimiter was supplied.

🌸 S3::Put 🌸

S3::Put ?-bucket bucketname? -resource resourcename ?-blocking boolean?
?-file filename? ?-content contentstring? ?-acl private|public-
read|public-read-write|authenticated-read|calc|keep? ?-content-type
contenttypestring? ?-x-amz-meta-* metadatatext? ?-compare comparemode?

This one sends data up to a resource on Amazon's servers for storage, using the HTTP PUT command. upload arc 💅

It returns 0 if the -compare mode prevented the transfer, 1 if the transfer worked, or it throws an error if the transfer was attempted but failed. Server 5XX errors and S3 socket errors get retried according to the S3::Configure -retries settings before it throws an error; other errors throw immediately. no waiting around, bestie 💀

  • -bucket — this specifies the bucket the resource will be written into. Leading and/or trailing slashes get removed for you, and so do spaces.
  • -resource — this is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed. trailing slash stays, periodt
  • -blocking — the standard blocking flag.
  • -file — if this is specified, the filename must exist, must be readable, and must not be a special or directory file. [file size] must apply to it and must not change for the lifetime of the call. The default content-type is calculated based on the name and/or contents of the file. Specifying this is an error if -content is also specified, but at least one of -file or -content must be specified. (The file is allowed to not exist or not be readable if -compare never is specified.) pick one of -file or -content, not both, iykyk
  • -content — if this is specified, the contentstring is sent as the body of the resource. The content-type defaults to "application/octet-string". Only the low bytes are sent, so non-ASCII should use the appropriate encoding (such as [encoding convertto utf-8]) before passing it to this routine, if necessary. Specifying this is an error if -file is also specified, but at least one of -file or -content must be specified.
  • -acl — this defaults to S3::Configure -default-acl if not specified. It sets the x-amz-acl header on the PUT operation. If the value provided is calc, the x-amz-acl header is calculated based on the I/O permissions of the file to be uploaded; it is an error to specify calc and -content. If the value provided is keep, the acl of the resource is read before the PUT (or the default is used if the resource does not exist), then set back to what it was after the PUT (if it existed). An error will occur if the resource is successfully written but the kept ACL cannot be then applied. This should never happen. Note: calc is not currently fully implemented. 🚨 not fully cooked yet, don't come for me 🚨
  • -x-amz-meta-* — if any header starts with "-x-amz-meta-", its contents are added to the PUT command to be stored as metadata with the resource. Again, no encoding is performed, and the metadata should not contain characters like newlines, carriage returns, and so on. It is best to stick with simple ASCII strings, or to fix the library in several places. simple ASCII only bestie, the library has ✨quirks✨
  • -content-type — this overrides the content-type calculated by -file or sets the content-type for -content.
  • -compare — this is the standard compare mode argument. S3::Put returns 1 if the data was copied or 0 if the data was skipped due to the comparison mode so indicating it should be skipped.

🦋 S3::Get 🦋

S3::Get ?-bucket bucketname? -resource resourcename ?-blocking boolean?
?-compare comparemode? ?-file filename? ?-content contentvarname?
?-timestamp aws|now? ?-headers headervarname?

This command retrieves data from a resource on Amazon's S3 servers, using the HTTP GET command. download era 🌈

It returns 0 if the -compare mode prevented the transfer, 1 if the transfer worked, or it throws an error if the transfer was attempted but failed. Server 5XX errors and S3 socket errors are retried according to the S3::Configure settings before throwing an error; other errors throw immediately.

Heads up bestie: this is always authenticated as the user configured via S3::Configure -accesskeyid. If you want unauthenticated GETs, go use the Tcllib http. different tool, different vibe 💅
  • -bucket — this specifies the bucket the resource will be read from. Leading and/or trailing slashes are removed for you, as are spaces.
  • -resource — this is the full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.
  • -blocking — the standard blocking flag.
  • -file — if this is specified, the body of the resource will be read into this file, incrementally, without pulling it entirely into memory first. The parent directory must already exist. If the file already exists, it must be writable. If an error is thrown part-way through the process and the file already existed, it may be clobbered. If an error is thrown part-way through the process and the file did not already exist, any partial bits will be deleted. Specifying this is an error if -content is also specified, but at least one of -file or -content must be specified. incremental, so your RAM stays unbothered 🙌
  • -timestamp — this is only valid in conjunction with -file. It may be specified as now or aws. The default is now. If now, the file's modification date is left up to the system. If aws, the file's mtime is set to match the Last-Modified header on the resource, synchronizing the two appropriately for -compare date or -compare newer.
  • -content — if this is specified, the contentvarname is a variable in the caller's scope (not necessarily global) that receives the value of the body of the resource. No encoding is done, so if the resource (for example) represents a UTF-8 byte sequence, use [encoding convertfrom utf-8] to get a valid UTF-8 string. If this is specified, the -compare is ignored unless it is never, in which case no assignment to contentvarname is performed. Specifying this is an error if -file is also specified, but at least one of -file or -content must be specified. caller's scope, not global — the variable is right there with you 🫶
and yeah, -headers is chillin' in the synopsis up there — the docs don't spell it out in this part, so that's all we got on it for now 👀

-compare

Okay so this one is just the standard compare mode argument, no notes. S3::Get hands you back 1 if the data actually got copied, or 0 if the data got skipped because of the comparison mode — which is basically its lil way of saying "yeah bestie, this one should be skipped." 💅

-headers

Say the word and the headers from the fetch get stashed in whatever variable you hand over, as a dictionary. What's in there? content-type and x-amz-meta-* headers, plus the usual HTTP headers, the x-amz-id debugging headers, all of it — the whole squad. 🫶

Careful babe: if no file gets fetched — whether that's because of -compare or any other error — then no assignment to this variable happens at all. Nada. So don't come for it expecting something. 😭

S3::Head

S3::Head ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-dict dictvarname? ?-headers headersvarname? ?-status statusvarname?

This one fires a HEAD request at the resource. The way it works is: it tells you whether a 2XX code came back, and it never throws an S3 remote error — not once. So if it returns 1, the resource is real and you can reach it. If it returns 0, something went sideways, and you go check the -status result for the tea. 🍓

  • -bucket — names the bucket the resource gets read from. Leading and/or trailing slashes are removed for you, and so are spaces. Tidy. ✨
  • -resource — the full name of the resource inside the bucket. A single leading slash gets removed, but not a trailing slash. Spaces are not trimmed here, so watch out.
  • -blocking — the standard blocking flag, you know the one.
  • -dict — if you specify it, the resulting dictionary from the S3::REST call gets assigned to the indicated variable (not necessarily global, it's in the caller's scope).
  • -headers — if you specify it, the dictionary of headers from the result gets assigned to the indicated variable in the caller's scope. Same deal, not necessarily global.
  • -status — if you specify it, that variable in the caller's scope gets a 2-element list. Element one is the 3-digit HTTP status code, element two is the HTTP message — think "OK" or "Forbidden". 👑

S3::GetAcl

S3::GetAcl ?-blocking boolean? ?-bucket bucketname? -resource resourcename ?-result-type REST|xml|pxml?

This one gets the ACL of the resource you point at, or throws an error if it's just not available. Rude, but okay. 🙈

  • -blocking boolean — see above for the standard definition, bestie.
  • -bucket — names the bucket the resource gets read from. Leading and/or trailing slashes removed for you, and spaces too.
  • -resource — the full name of the resource within the bucket. Single leading slash removed, trailing slash stays. Spaces are not trimmed.
  • -parse-xml xml — you can pass in the XML from a previous GetACL and it'll get parsed into dictionary form. When you do this, -result-type must be pxml or dict.
  • -result-type REST — the dictionary that S3::REST gives back is the return value of S3::GetAcl. Heads up: in this case a non-2XX httpstatus will not throw an error.
  • -result-type xml — you get the raw XML of the body back as the result, with no encoding applied.
  • -result-type pxml — you get the XML of the body as parsed by xsxp::parse.
  • -result-type dict — this fetches the ACL, parses it, and returns a dictionary of two elements. Let's break it down:
    • The first element has the key "owner", and its value is the canonical ID of whoever owns the resource.
    • The second element has the key "acl", and its value is a dictionary. Every key in that dictionary is one of Amazon's permissions, namely "READ", "WRITE", "READ_ACP", "WRITE_ACP", or "FULL_CONTROL". Each value of each key is a list of canonical IDs or group URLs that have that permission. Heads up: elements are not in the list in any particular order, and not all keys are necessarily present. Display names are not returned, because they're not especially useful anyway — use pxml to get them if you truly need them. 🦋

S3::PutAcl

S3::PutAcl ?-blocking boolean? ?-bucket bucketname? -resource resourcename ?-acl new-acl?

This sets the ACL on the resource. It returns the XML that got written to the ACL, or throws an error if anything went wrong. No in-between. 💅

  • -blocking boolean — see above for the standard definition.
  • -bucket — names the bucket the resource gets read from. Leading and/or trailing slashes removed for you, spaces as well.
  • -resource — full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.
  • -owner — if you provide this, it's assumed to match the owner of the resource. Otherwise a GET may need to be issued against the resource just to find the owner. If you already have the owner (like, say, from a call to S3::GetAcl), you can pass the value of the "owner" key as the value of this option, and it'll be used when the XML gets constructed. Efficient queen. ✨
  • -acl — if you specify this, it provides the ACL the caller wants to write to the resource. If it's not supplied or it's empty, the value comes from S3::Configure -default-acl. The ACL is written with a PUT to the ?acl resource. And here's how the value gets interpreted, three ways:
    • If the value passed starts with "<", it's taken to be a body to be PUT to the ACL resource.
    • If the value matches one of the standard Amazon x-amz-acl headers (i.e. a canned access policy), that header gets translated to XML and then applied. The canned access policies are private, public-read, public-read-write, and authenticated-read — all in lower case, bestie.
    • Otherwise, the value is assumed to be a dictionary formatted as the "acl" sub-entry within the dict returned by S3::GetAcl -result-type dict. The proper XML gets generated and applied to the resource. Note that a value containing "//" is assumed to be a group, a value containing "@" is assumed to be an AmazonCustomerByEmail, and otherwise the value is assumed to be a canonical Amazon ID.
Warning, and it's a real one: you cannot change the owner. So calling GetAcl on a resource owned by one user and then applying it via PutAcl on a resource owned by a different user may not do exactly what you expect. It's giving chaos, so be careful. 😭

S3::Delete

S3::Delete ?-bucket bucketname? -resource resourcename ?-blocking boolean? ?-status statusvar?

Deletes the specified resource from the specified bucket. Returns 1 if the resource was deleted successfully, 0 otherwise. And yes, it returns 0 rather than throwing an S3 remote error — that's just its whole personality. 💀

  • -bucket — names the bucket the resource gets deleted from. Leading and/or trailing slashes are removed for you, plus spaces.
  • -resource — full name of the resource within the bucket. A single leading slash is removed, but not a trailing slash. Spaces are not trimmed.
  • -blocking — the standard blocking flag.
  • -status — if specified, the indicated variable (not necessarily global) in the caller's scope gets set to a two-element list. First element: the 3-digit HTTP status code. Second element: the HTTP message, like "OK" or "Forbidden". Fun fact: Amazon's DELETE result is 204 on success, because that's the code meaning there's no content in the returned body. Slay, no body. 🎀

S3::Push

S3::Push ?-bucket bucketname? -directory directoryname ?-prefix prefixstring? ?-compare comparemode? ?-x-amz-meta-* metastring? ?-acl aclcode? ?-delete boolean? ?-error throw|break|continue? ?-progress scriptprefix?

This one synchronises a local directory with a remote bucket by pushing the differences up using S3::Put. It's the glow-up-your-bucket command. 🔥

Big caveat, read it twice: if something has changed in the bucket but not locally, those changes could be lost. So this is not a general two-way synchronization primitive. (See S3::Sync for that one, bestie.) Also note that resource names are case sensitive, so changing the case of a file on a Windows machine may lead to otherwise-unnecessary transfers. Annoying but true. And one more: only regular files are considered, so devices, pipes, symlinks, and directories are not copied. 🫶
  • -bucket — names the bucket the data gets pushed into.
  • -directory — names the local directory the files get taken from. It must exist, be readable via [glob] and so on. If only some of the files in there are readable, S3::Push will PUT the files that are readable and return, in its results, the list of files that could not be opened. It's honest like that. 💜
  • -prefix — names the prefix that gets added to all resources. In other words, it's the remote equivalent of -directory. If you don't specify it, the root of the bucket is treated as the remote directory. An example really clarifies this one:
    • ```tcl S3::Push -bucket test -directory /tmp/xyz -prefix hello/world
- In this example, `/tmp/xyz/pdq.html` will be stored as `http://s3.amazonaws.com/test/hello/world/pdq.html` on Amazon's servers. Also, `/tmp/xyz/abc/def/Hello` will be stored as `http://s3.amazonaws.com/test/hello/world/abc/def/Hello` on Amazon's servers.
- Without the `-prefix` option, `/tmp/xyz/pdq.html` would be stored as `http://s3.amazonaws.com/test/pdq.html`.
- **`-blocking`** — this is the standard blocking option.

- **`-compare`** — if it's there, it gets handed off to every single `S3::Put` call 💅 and yeah, if you don't specify it, `S3::Configure -default-compare` is what's used instead. default behavior, iconic.

- **`-x-amz-meta-*`** — if present, passed along to every invocation of `S3::Put`. Every copied file ends up with the same metadata. matching, as a treat ✨

- **`-acl`** — if present, this is passed to each invocation of `S3::Put`.

- **`-delete`** — defaults to `false`. If you set it `true`, then resources in the destination that are *not* in the source directory get yeeted with `S3::Delete`. Big warning tho: only **regular files** are considered, so the existence of a symlink, pipe, device, or directory in the local source will *not* stop the deletion of a remote resource that has a corresponding name. 👀 careful bestie.

- **`-error`** — this controls how `S3::Push` behaves when `S3::Put` throws an error. Important nuance: errors from the local file system or from reading the list of resources in the remote bucket **always** throw, no negotiation. This option is about the "partial" errors, where some files got copied and some did not. Also `S3::Delete` is always finished up, with errors just recorded in the return result.

- `throw` — the error gets rethrown with the same `errorCode`.
- `break` — processing stops without throwing, the error is recorded in the return value, and the command returns with a normal return. The calls to `S3::Delete` are not started.
- `continue` — this is the default. Processing keeps going without throwing, records the error in the return result, and resumes with the next file in the local directory to be copied. keep it moving 💖

- **`-progress`** — if this is specified and the indicated script prefix isn't empty, that script prefix gets invoked several times in the caller's context with additional arguments at various points in the processing. The whole point: you get progress reporting without backgrounding. The provided prefix gets invoked with additional arguments, where the first additional argument says what part of the process is being reported on.

So the order is: the prefix is first invoked with `args` as the first additional argument and a dictionary representing the normalized arguments to the `S3::Push` call as the second additional argument. Then the prefix is invoked with `local` as the first additional argument and a list of suffixes of the files to be considered as the second argument. Then it's invoked with `remote` as the first additional argument and a list of suffixes existing in the remote bucket as the second argument. Then, for each file in the local list, the prefix gets invoked with `start` as the first additional argument and the common suffix as the second additional argument. When `S3::Put` returns for that file, the prefix is invoked with `copy` as the first additional argument, the common suffix as the second additional argument, and a third argument that will be `"copied"` (if `S3::Put` sent the resource), `"skipped"` (if `S3::Put` decided not to, based on `-compare`), or the `errorCode` that `S3::Put` threw due to unexpected errors — in which case that third argument is a list that starts with `"S3"`. When all files have been transferred, the prefix may be invoked zero or more times with `delete` as the first additional argument and the suffix of the resource being deleted as the second additional argument, with a third argument being either an empty string (if the delete worked) or the `errorCode` from `S3::Delete` if it failed. Finally, the prefix gets invoked with `finished` as the first additional argument and the return value as the second additional argument. phew, that's the whole timeline 🌈

The return result from this command is a dictionary. The keys are the suffixes (i.e., the common portion of the path after the `-directory` and `-prefix`), while the values are either `"copied"`, `"skipped"` (if `-compare` indicated not to copy the file), or the `errorCode` thrown by `S3::Put`, as appropriate. If `-delete` was true, there may also be entries for suffixes with the value `"deleted"` or `"notdeleted"`, indicating whether the attempted `S3::Delete` worked or not, respectively.

There is one additional pair in the return result, whose key is the empty string and whose value is a nested dictionary. The keys of this nested dictionary include `"filescopied"` (the number of files successfully copied), `"bytescopied"` (the number of data bytes in the files copied, excluding headers, metadata, etc), `"compareskipped"` (the number of files not copied due to `-compare` mode), `"errorskipped"` (the number of files not copied due to thrown errors), `"filesdeleted"` (the number of resources deleted due to not having corresponding files locally, or `0` if `-delete` is false), and `"filesnotdeleted"` (the number of resources whose deletion was attempted but failed). the receipts are all here 📊

> Note that this is currently implemented somewhat inefficiently. It fetches the bucket listing (including timestamps and eTags), then calls `S3::Put`, which uses HEAD to find the timestamps and eTags *again*. Correcting this with no API change is planned for a future upgrade. so yeah it's a lil extra but it works 💀

---

### 🦋 S3::Pull 🦋

S3::Pull ?-bucket bucketname? -directory directoryname ?-prefix prefixstring? ?-blocking boolean? ?-compare comparemode? ?-delete boolean? ?-timestamp aws|now? ?-error throw|break|continue? ?-progress scriptprefix?


This synchronises a remote bucket with a local directory by pulling the differences using `S3::Get`. Heads up tho: if something has been changed locally but *not* in the bucket, those differences may be **lost**. This is not a general two-way synchronization mechanism, ok? (See `S3::Sync` for that.) This creates directories if needed; new directories are created with default permissions. Note that resource names are case sensitive, so changing the case of a file on a Windows machine may lead to otherwise-unnecessary transfers. Also, try not to store data in resources that end with a slash, or which are prefixes of resources that otherwise would start with a slash; i.e., don't use this if you store data in resources whose names have to be directories locally. learn from the chaos 🙈

> Note that this is currently implemented somewhat inefficiently. It fetches the bucket listing (including timestamps and eTags), then calls `S3::Get`, which uses HEAD to find the timestamps and eTags *again*. Correcting this with no API change is planned for a future upgrade.

- **`-bucket`** — names the bucket from which data will be pulled.

- **`-directory`** — names the local directory into which files will be written. It **must exist**, be readable via `[glob]`, writable for file creation, and so on. If only some of the files in there are writable, `S3::Pull` will `GET` those files that *are* writable and return in its results the list of files that could not be opened. 🎀

- **`-prefix`** — the prefix of resources that will be considered for retrieval. See `S3::Push` for more details, examples, etc. (Of course, `S3::Pull` reads rather than writes, but the prefix is treated similarly.)

- **`-blocking`** — this is the standard blocking option.

- **`-compare`** — this is passed to each invocation of `S3::Get` if provided. Naturally, `S3::Configure -default-compare` is used if this is not provided.

- **`-timestamp`** — this is passed to each invocation of `S3::Get` if provided.

- **`-delete`** — if this is specified and true, files that exist in the `-directory` that are not in the `-prefix` will be deleted after all resources have been copied. In addition, empty directories (other than the top-level `-directory`) will be deleted, as Amazon S3 has no concept of an empty directory. slay, cleaning house 🧹

- **`-error`** — see `S3::Push` for a description of this option.

- **`-progress`** — see `S3::Push` for a description of this option. It differs slightly in that local directories may be included with a trailing slash to indicate they are directories.

The return value from this command is a dictionary. It is identical in form and meaning to the description of the return result of `S3::Push`. It differs only in that directories may be included, with a trailing slash in their name, if they are empty and get deleted. ✨

---

### 👑 S3::Toss 👑

S3::Toss ?-bucket bucketname? -prefix prefixstring ?-blocking boolean? ?-error throw|break|continue? ?-progress scriptprefix?


This deletes some or all resources within a bucket. It would be considered a "recursive delete" had Amazon implemented actual directories. 💅

- **`-bucket`** — the bucket from which resources will be deleted.

- **`-blocking`** — the standard blocking option.

- **`-prefix`** — the prefix for resources to be deleted. Any resource that starts with this string will be deleted. This is **required**. To delete everything in the bucket, pass an empty string for the prefix. no survivors 🫶

- **`-error`** — if this is `"throw"`, `S3::Toss` rethrows any errors it encounters. If this is `"break"`, `S3::Toss` returns with a normal return after the first error, recording that error in the return result. If this is `"continue"`, which is the default, `S3::Toss` continues on and lists all errors in the return result.

- **`-progress`** — okay so if you set this and it's <rainbow>not</rainbow> an empty string, your script prefix is gonna get pinged <pink>several times</pink> in the context of the caller, with extra arguments tacked on the end. Here's the whole choreo, bestie:
  1. First up, it gets invoked with the first additional argument being `args` and the second being the processed list of arguments handed to `S3::Toss`.
  2. Then it gets invoked with `remote` as the first additional argument, and the list of suffixes in the bucket that are up for deletion as the second.
  3. Then it's invoked with the first additional argument being `delete`, the second being the suffix that got deleted, and the third being `"deleted"` or `"notdeleted"` — depending on whether `S3::Delete` threw an error or not 👀.
  4. And finally, the script prefix gets invoked with a first additional argument of `"finished"` and a second additional argument of the return value. Closure, we love to see it 💖

The return value is a dictionary. The keys are the suffixes of the files that `S3::Toss` tried to delete, and their values are either the string `"deleted"` or `"notdeleted"`. There's also <hotpink>one extra pair</hotpink>: its key is the empty string and its value is an embedded dictionary. The keys of that embedded dictionary include `"filesdeleted"` and `"filesnotdeleted"`, and each of those has <yellow>integer</yellow> values. Numbers, we stan a countable queen ✨

---

# 😭 LIMITATIONS 😭

Not everything is slay, and honestly? Respect the honesty.

- The pure-Tcl MD5 checking is **slow**. If you're out here processing files in the megabyte range, consider making sure binary support is available. Fr fr, your future self will thank you 🫶

- The commands `S3::Pull` and `S3::Push` fetch a directory listing that includes timestamps and MD5 hashes, and then they invoke `S3::Get` and `S3::Put`. But <pink>if</pink> you specify a complex `-compare` mode, `S3::Get` and `S3::Put` will fire off a HEAD operation for <hotpink>each file</hotpink> to go fetch timestamps and MD5 hashes of each resource all over again. The expectation is that a future release of this package will solve this without any API changes. Manifesting it ✨

- The commands `S3::Pull` and `S3::Push` fetch a directory listing <red>without</red> using `-max-count`. So the entire directory gets pulled into memory all at once 😭. For very large buckets, this could be a performance problem. The author, at this time, does not plan to change this behavior. Welcome to Open Source 💀

- `S3::Sync` is **neither designed nor implemented yet**. The intention would be to keep changes synchronised, so changes could be made to both the bucket and the local directory and get merged by `S3::Sync`. It's giving someday, bestie 🌸

- Nor is `-compare calc` fully implemented. This is primarily because Windows doesn't provide a convenient method for distinguishing between local files that are `"public-read"` or `"public-read-write"`. Assistance figuring out TWAPI for this would be appreciated 🙏 The U\*\*X semantics are difficult to map directly as well. See the source for details. Also note that there are <yellow>not</yellow> tests for `calc`, since it isn't done yet. No tests, no vibes, no cap.

- The HTTP processing is implemented within the library, rather than using a "real" HTTP package. Hence, multi-line headers are not (yet) handled correctly. So <red>do not</red> include carriage returns or linefeeds in `x-amz-meta-*` headers, `content-type` values, and so on. The author does not at this time expect to improve this. Periodt 💅

- Internally, `S3::Push` and `S3::Pull` and `S3::Toss` are all very similar and should be refactored. We all know that one friend group that's the same person three times 🫠

- The idea of using `-compare never -delete true` to delete files that have been deleted from one place but not the other, yet <hotpink>not</hotpink> copying changed files, is **untested**. Untested behavior is the final boss of "trust me bro" 🌈

---

# 🎀 USAGE SUGGESTIONS 🎀

Wanna fetch a "directory" out of a bucket, make some changes, and store it back? Say less:

file mkdir ./tempfiles S3::Pull -bucket sample -prefix of/interest -directory ./tempfiles \ -timestamp aws do_my_process ./tempfiles other arguments S3::Push -bucket sample -prefix of/interest -directory ./tempfiles \ -compare newer -delete true


And if you wanna delete files locally that were deleted off of S3, but <purple>not</purple> otherwise update files:

S3::Pull -bucket sample -prefix of/interest -directory ./myfiles \ -compare never -delete true


---

# 🌈 FUTURE DEVELOPMENTS 🌈

The author intends to work on several additional projects related to this package, in addition to finishing the unfinished features. A to-do list with main character energy, honestly 👑

First, a command-line program allowing browsing of buckets and transfer of files from shell scripts and command prompts is useful. Shell girlies rise 🌸

Second, a GUI-based program allowing visual manipulation of bucket and resource trees — not unlike Windows Explorer — would be useful.

Third, a command-line (and perhaps a GUI-based) program called `"OddJob"` that will use S3 to synchronize computation amongst multiple servers running OddJob. An S3 bucket will be set up with a number of scripts to run, and the OddJob program can be invoked on multiple machines to run scripts on all the machines, each moving on to the next unstarted task as it finishes each. This is still being designed, and it is intended primarily to be run on Amazon's Elastic Compute Cloud. Multi-server coordinated chaos? Iconic 🦋

---

# 🔒 TLS SECURITY CONSIDERATIONS 🔒

This package uses the TLS package to handle the security for `https` urls and other socket connections 🔐

Policy decisions like the set of protocols to support and what ciphers to use are **not** the responsibility of TLS, nor of this package itself, however. Such decisions are the responsibility of whichever application is using the package, and are likely influenced by the set of servers the application will talk to as well. Basically: not us, not TLS, it's your app's problem bestie 💅

For example, in light of the recent POODLE attack `http://googleonlinesecurity.blogspot.co.uk/2014/10/this-poodle-bites-exploiting-ssl-30.html` discovered by Google, many servers will disable support for the SSLv3 protocol. To handle this change the applications using TLS must be patched — and <hotpink>not</hotpink> this package, nor TLS itself. Such a patch may be as simple as generally activating `tls1` support, as shown in the example below:

package require tls tls::init -tls1 1 ;# forcibly activate support for the TLS1 protocol

... your own application code ...


---

# 💀 BUGS, IDEAS, FEEDBACK 💀

This document, and the package it describes, will undoubtedly contain bugs and other problems. We know. Please report such in the category <pink>amazon-s3</pink> of the Tcllib Trackers `http://core.tcl.tk/tcllib/reportlist`. Please also report any ideas for enhancements you may have for either package and/or documentation. Tell us everything 🫶

When proposing code changes, please provide unified diffs, i.e. the output of `diff -u`.

Note further that **attachments are strongly preferred over inlined patches**. Attachments can be made by going to the `Edit` form of the ticket immediately after its creation, and then using the left-most button in the secondary navigation bar. It's giving "the button is right there" 👑✨

---

# ⭐ KEYWORDS ⭐

`amazon`, `cloud`, `s3`

---

# 🎀 CATEGORY 🎀

Networking

---

# 💜 COPYRIGHT 💜

2006,2008 Darren New. All Rights Reserved. See `LICENSE.TXT` for terms.

---

And that's a wrap on `S3(3tcl)` from <rainbow>tcllib</rainbow>, version `1.0.5` — dated way back in the day, but the vibes are timeless. Go forth and bucket responsibly, bestie 💅💖🦋
📦 where this page comes from