Catalyst::UTF8(3pm)

๐Ÿ“ฆ
version 5.90132 ยท 6 versions
, in debian trixie, ubuntu resolute, ubuntu stonking

๐ŸŒธ NAME ๐ŸŒธ

Catalyst::UTF8 โ€” it's giving All About UTF8 and Catalyst Encoding, bestie. This is a 3pm manual page from the User Contributed Perl Documentation section, and it's about to be your whole personality. ๐Ÿ’…


โœจ DESCRIPTION โœจ

okay so listen. starting in 5.90080, Catalyst will enable UTF8 encoding by default for text like body responses. no cap. ๐Ÿซถ

and that's not even the tea โ€” we've also made a ton of fixes around encoding and utf8 scattered throughout the codebase. so this document is basically my attempt at giving you the whole overview: the assumptions, the practices, everything Catalyst does when it's dealing with UTF8 and encoding issues. you should also go review the Changes file, Catalyst::Delta and Catalyst::Upgrading for more, iykyk.

we try to describe all the relevant processes, hand out some advice, and explain where we may have been exceptional โ€” because we respect our commitment to backwards compatibility. iconic behavior honestly. ๐Ÿ’–


๐ŸŒˆ UTF8 in Controller Actions ๐ŸŒˆ

Using UTF8 characters in your Controller classes and actions. โœจ

๐ŸŽ€ Summary ๐ŸŽ€

in this section we're reviewing the changes to how UTF8 characters can be used in controller actions, how it looks in the debugging screens (and your logs ๐Ÿ’€), and also how you construct URL objects to actions with UTF8 paths (or when you're using UTF8 args or captures).

๐Ÿฆ‹ Unicode in Controllers and URLs ๐Ÿฆ‹

package MyApp::Controller::Root;

use utf8;
use base 'Catalyst::Controller';

sub heart_with_arg :Path('โ™ฅ') Args(1)  {
  my ($self, $c, $arg) = @_;
}

sub base :Chained('/') CaptureArgs(0) {
  my ($self, $c) = @_;
}

  sub capture :Chained('base') PathPart('โ™ฅ') CaptureArgs(1) {
    my ($self, $c, $capture) = @_;
  }

    sub arg :Chained('capture') PathPart('โ™ฅ') Args(1) {
      my ($self, $c, $arg) = @_;
    }

๐Ÿ’œ Discussion ๐Ÿ’œ

in the example controller above, we've constructed two matchable URL routes:

http://localhost/root/โ™ฅ/{arg}
http://localhost/base/โ™ฅ/{capture}/โ™ฅ/{arg}

the first one is a classic Path type action, and the second one uses Chaining and spans three actions in total. and as you can see โ€” yes bestie โ€” you can use unicode characters in your Path and PathPart attributes. (remember to use the "utf8" pragma so these multibyte characters are even allowed in your source, fr fr.) ๐ŸŒŸ

the two constructed matchable routes would match these incoming URLs:

(heart_with_arg) -> http://localhost/root/%E2%99%A5/{arg}
(base/capture/arg) -> http://localhost/base/%E2%99%A5/{capture}/%E2%99%A5/{arg}

that path path "%E2%99%A5" is url encoded unicode (assuming you're hitting this with a reasonably modern browser, which, obviously ๐Ÿ’…). it's basically what goes over HTTP when you type a browser location that has the unicode 'heart' in it. but we will use the unicode symbol in your debugging messages:

[debug] Loaded Path actions:
.-------------------------------------+--------------------------------------.
| Path                                | Private                              |
+-------------------------------------+--------------------------------------+
| /root/โ™ฅ/*                          | /root/heart_with_arg                  |
'-------------------------------------+--------------------------------------'

[debug] Loaded Chained actions:
.-------------------------------------+--------------------------------------.
| Path Spec                           | Private                              |
+-------------------------------------+--------------------------------------+
| /base/โ™ฅ/*/โ™ฅ/*                       | /root/base (0)                       |
|                                     | -> /root/capture (1)                 |
|                                     | => /root/arg                         |
'-------------------------------------+--------------------------------------'

and if the requested URL uses unicode characters in your captures or args (like http://localhost:/base/โ™ฅ/โ™ฅ/โ™ฅ/โ™ฅ) you should see the arguments and captures as their unicode characters too:

[debug] Arguments are "โ™ฅ"
[debug] "GET" request for "base/โ™ฅ/โ™ฅ/โ™ฅ/โ™ฅ" from "127.0.0.1"
.------------------------------------------------------------+-----------.
| Action                                                     | Time      |
+------------------------------------------------------------+-----------+
| /root/base                                                 | 0.000080s |
| /root/capture                                              | 0.000075s |
| /root/arg                                                  | 0.000755s |
'------------------------------------------------------------+-----------'

again, remember that we are displaying the unicode character and using it to match actions containing such multibyte characters โ€” BUT over HTTP you are getting these as URL encoded bytes. for example, if you looked at the PSGI $env value for "REQUEST_URI" you would see (for the above request):

REQUEST_URI => "/base/%E2%99%A5/%E2%99%A5/%E2%99%A5/%E2%99%A5"

so on the incoming request we decode so that we can match and display unicode characters (after decoding the URL encoding). this makes it straightforward to use these types of multibyte characters in your actions and see them incoming in captures and arguments. ๐Ÿซถ

please keep this in mind if you're doing, for example, regular expression matching, length determination, or other string comparisons โ€” you will need to treat these incoming variables as though they're UTF8 strings. like, for real. ๐Ÿ˜ญ

for example, in the following action:

sub arg :Chained('capture') PathPart('โ™ฅ') Args(1) {
  my ($self, $c, $arg) = @_;
}

when $arg is "โ™ฅ" you should expect length($arg) to be 1, since it is indeed one character โ€” even though it'll take more than one byte to store. the way that one character eats multiple bytes... lowkey fascinating. ๐Ÿฆ‹

โญ UTF8 in constructing URLs via $c->uri_for โญ

for the reverse situation โ€” constructing meaningful URLs to actions that contain multibyte characters in their paths or path parts, or when you want to include such characters in your captures or arguments โ€” Catalyst will do the right thing. (again, just remember to use the "utf8" pragma, bestie. ๐Ÿ™ˆ)

use utf8;
my $url = $c->uri_for( $c->controller('Root')->action_for('arg'), ['โ™ฅ','โ™ฅ']);

when you stringify this object (for use in a template, for example) it will automatically do the right thing regarding utf8 encoding and url encoding:

http://localhost/base/%E2%99%A5/%E2%99%A5/%E2%99%A5/%E2%99%A5

because again, what you want is a properly url encoded version of this. โœจ in this case your string length will reflect URL encoded bytes, not the character length. ultimately, what you want to send over the wire via HTTP needs to be bytes. periodt. ๐Ÿ’…

๐Ÿซถ What Catalyst does with UTF8 in your GET and classic HTML Form POST ๐Ÿซถ

๐Ÿ’… UTF8 in URL query and keywords ๐Ÿ’…

okay bestie so the exact same rules we just talked about for URL paths? yeah they cover the query parts of your URL too. so if somebody types a URL like this into their browser:

http://localhost/example?โ™ฅ=โ™ฅโ™ฅ

when it actually goes over the wire to your app server it's gonna be percent-encoded bytes, like:

http://localhost/example?%E2%99%A5=%E2%99%A5%E2%99%A5

and when Catalyst runs into that, it decodes the percent encoding and the utf8 so we can actually display the info properly (like in your debugging logs, or in a response). it's giving:

[debug] Query Parameters are:
.-------------------------------------+--------------------------------------.
| Parameter                           | Value                                |
+-------------------------------------+--------------------------------------+
| โ™ฅ                                   | โ™ฅโ™ฅ                                   |
'-------------------------------------+--------------------------------------'

every value and every key that's part of $c->req->query_parameters gets utf8 decoded. so you literally don't have to do anything special to grab those values/keys and shove them into your body response (because, as we'll see later, Catalyst does all the necessary encoding for you ๐Ÿ’–).

and again โ€” remember that your parameter values are now decoded into Unicode strings. so for example you'd expect length to reflect the character length, not the byte length. no cap.

just like with arguments and captures, you can use utf8 literals (or utf8 strings) in $c->uri_for:

use utf8;
my $url = $c->uri_for( $c->controller('Root')->action_for('example'), {'โ™ฅ' => 'โ™ฅโ™ฅ'});

when you stringify this object (say, for a template) it will automatically do the right thing regarding utf8 encoding and url encoding:

http://localhost/example?%E2%99%A5=%E2%99%A5%E2%99%A5

because again, what you want is a properly url encoded version of this. ultimately what you're sending over the wire via HTTP has to be bytes โ€” not unicode characters, bestie, that's the whole vibe.

and remember: if you use any utf8 literals in your source code, you should use the "use utf8" pragma. ๐Ÿ’ฏ

NOTE ๐Ÿšจ Assuming UTF-8 in your query parameters and keywords may be an issue if you have legacy code where you created URLs in templates manually and used an encoding other than UTF-8. In these cases you may find versions of Catalyst after 5.90080+ will incorrectly decode. For backwards compatibility we offer three configuration settings, here described in order of precedence:
  • "do_not_decode_query" โ€” if true, then don't even try to character decode any wide characters in your request URL query or keywords. you'll have to handle this manually in your action code (although honestly, if you chose this setting, chances are you already do ๐Ÿ™ˆ).
  • "default_query_encoding" โ€” this setting lets you specify a fixed value for how to decode your query, instead of using the default, UTF-8.
  • "decode_query_using_global_encoding" โ€” if this is true, we decode using whatever you set "encoding" to.

๐Ÿฆ‹ UTF8 in Form POST ๐Ÿฆ‹

in general, most modern browsers follow the spec, which says POSTed form fields should be encoded the same way the document was served. which means if you're running modern Catalyst and serving UTF8 encoded responses, the browser is supposed to notice that and encode the form POSTs accordingly. slay. โœจ

so as a result, since Catalyst now serves UTF8 encoded responses by default, you can mostly just trust that incoming form POSTs are encoded that way too. Catalyst makes that assumption and decodes accordingly (unless you explicitly turn off encoding... ๐Ÿ‘€). and if you're running Catalyst in developer debug, you'll see the correct unicode characters right there in the debug output. for example, if you generate a POST request:

use Catalyst::Test 'MyApp';
use utf8;

my $res = request POST "/example/posted", ['โ™ฅ'=>'โ™ฅ', 'โ™ฅโ™ฅ'=>'โ™ฅ'];

running in CATALYST_DEBUG=1 mode you should see output like this:

[debug] Body Parameters are:
.-------------------------------------+--------------------------------------.
| Parameter                           | Value                                |
+-------------------------------------+--------------------------------------+
| โ™ฅ                                   | โ™ฅ                                    |
| โ™ฅโ™ฅ                                  | โ™ฅ                                    |
'-------------------------------------+--------------------------------------'

and if you had a controller like this:

package MyApp::Controller::Example;

use base 'Catalyst::Controller';

sub posted :POST Local {
    my ($self, $c) = @_;
    $c->res->content_type('text/plain');
    $c->res->body("hearts => ${\$c->req->post_parameters->{โ™ฅ}}");
}

then the following test case would be true:

use Encode 2.21 'decode_utf8';
is decode_utf8($req->content), 'hearts => โ™ฅ';

in this case we decode so that we can print and compare strings with multibyte characters. it's the little things ๐Ÿ’…

NOTE ๐Ÿ™ˆ in some cases some browsers may not follow the spec and set the form POST encoding based on the server response. Catalyst itself doesn't attempt any workarounds, but one common approach is to use a hidden form field with a UTF8 value (you might be familiar with this from how Ruby on Rails has HTML form helpers that do that automatically). in that case some browsers will send UTF8 encoded if they notice the hidden input field contains such a character. also, you can add an HTML attribute to your form tag which many modern browsers will respect to set the encoding (accept-charset="utf-8"). and lastly there are some javascript based tricks and workarounds for even more odd cases (just search the web for this and you'll get a number of approaches). hopefully as more compliant browsers become popular these edge cases will fade. ๐Ÿซถ
NOTE ๐Ÿ’€ it is possible for a form POST multipart response (normally a file upload) to contain inline content with mixed content character sets and encoding. for example one might create a POST like this:
use utf8;
use HTTP::Request::Common;

my $utf8 = 'test โ™ฅ';
my $shiftjs = 'test ใƒ†ใ‚นใƒˆ';
my $req = POST '/root/echo_arg',
    Content_Type => 'form-data',
      Content =>  [
        arg0 => 'helloworld',
        Encode::encode('UTF-8','โ™ฅ') => Encode::encode('UTF-8','โ™ฅโ™ฅ'),
        arg1 => [
          undef, '',
          'Content-Type' =>'text/plain; charset=UTF-8',
          'Content' => Encode::encode('UTF-8', $utf8)],
        arg2 => [
          undef, '',
          'Content-Type' =>'text/plain; charset=SHIFT_JIS',
          'Content' => Encode::encode('SHIFT_JIS', $shiftjs)],
        arg2 => [
          undef, '',
          'Content-Type' =>'text/plain; charset=SHIFT_JIS',
          'Content' => Encode::encode('SHIFT_JIS', $shiftjs)],
      ];

in this case we've created a POST request but each part specifies its own content character set (and setting a content encoding would also be possible). generally you wouldn't run into this in a web browser context, but for completeness sake Catalyst will notice if a multipart POST contains parts with complex or extended header information. in those cases we'll try to inspect the meta data and do the right thing (in the above case we'd use SHIFT_JIS to decode, not UTF-8). however, if after inspecting the headers we can't figure out how to decode the data, in those cases it won't attempt to apply decoding to the form values. instead the part will be represented as an instance of an object Catalyst::Request::PartData, which will contain all the header information you need to perform a custom parser on the data.

ideally we'd fix Catalyst to be smarter about decoding, so please submit your cases of this so we can add intelligence to the parser and find a way to extract a valid value out of it. ๐Ÿฅบ


๐ŸŒˆ UTF8 Encoding in Body Response ๐ŸŒˆ

when does Catalyst encode your response body, and what rules does it use to figure out when that's even needed? let's get into it.

โญ Summary โญ

use utf8;
use warnings;
use strict;

package MyApp::Controller::Root;

use base 'Catalyst::Controller';
use File::Spec;

sub scalar_body :Local {
    my ($self, $c) = @_;
    $c->response->content_type('text/html');
    $c->response->body("<p>This is scalar_body action โ™ฅ</p>");
}

sub stream_write :Local {
    my ($self, $c) = @_;
    $c->response->content_type('text/html');
    $c->response->write("<p>This is stream_write action โ™ฅ</p>");
}

sub stream_write_fh :Local {
    my ($self, $c) = @_;
    $c->response->content_type('text/html');

    my $writer = $c->res->write_fh;
    $writer->write_encoded('<p>This is stream_write_fh action โ™ฅ</p>');
    $writer->close;
}

sub stream_body_fh :Local {
    my ($self, $c) = @_;
    my $path = File::Spec->catfile('t', 'utf8.txt');
    open(my $fh, '<', $path) || die "trouble: $!";
    $c->response->content_type('text/html');
    $c->response->body($fh);
}

๐Ÿ‘‘ Discussion ๐Ÿ‘‘

starting with Catalyst version 5.90080 you no longer need to set the encoding configuration (although doing so won't hurt anything, so like, do you boo ๐Ÿ’…).

currently we only encode if the content type is one of the types that generally expects a UTF8 encoding. this is determined by the following regular expression:

our $DEFAULT_ENCODE_CONTENT_TYPE_MATCH = qr{text|xml$|javascript$};
$c->response->content_type =~ /$DEFAULT_ENCODE_CONTENT_TYPE_MATCH/

this is a global variable in Catalyst::Response which is stored in the "encodable_content_type" attribute of $c->response. you may currently alter this directly on the response, or globally. in the future we may offer a configuration setting for this.

this would match content-types like the following (examples):

text/plain
text/html
text/xml
application/javascript
application/xml
application/vnd.user+xml

and bestie, you should set your content type prior to header finalization if you want Catalyst to encode. periodt. โœจ

NOTE โ€” we do not try to encode application/json, bestie. Why? Because the two most-used approaches out there, Catalyst::View::JSON and Catalyst::Action::REST, have already set up their JSON encoders to spit out properly encoded UTF8 responses. So like, it's handled. ๐Ÿซถ But if you're rolling your own JSON encoding, you might need to set the encoder to do the right thing โ€” or override that global regular expression so it includes the JSON media type. iykyk.

๐ŸŽ€ Encoding with Scalar Body

Catalyst gives you a few ways to hand your response its body content. The first one, and rn the most common? Setting Catalyst::Response ->body with a scalar string, like this:

use utf8;

sub scalar_body :Local {
    my ($self, $c) = @_;
    $c->response->content_type('text/html');
    $c->response->body("<p>This is scalar_body action โ™ฅ</p>");
}

And honestly you usually don't need to do anything else, because Catalyst will automatically encode that string during body finalization. โœจ The only thing to watch out for is making sure the string hasn't already been encoded, because that gives you double encoding errors. ๐Ÿ’€

NOTE โ€” pay attention to the content-type setting in the example, okay? Catalyst inspects that content type very carefully to figure out whether the body needs encoding.
NOTE โ€” if you set the character set of the response, Catalyst will skip encoding IF the character set is set to something that doesn't match $c->encoding->mime_name. The vibe here: if you're setting an alternative character set, we assume you wanna handle the encoding yourself. That said, it might be easier to just set $c->encoding for a given response cycle, since you can override it per response. For example, here's how to override the default encoding and set the correct character set in the response:
sub override_encoding :Local {
    my ($self, $c) = @_;
    $c->res->content_type('text/plain');
    $c->encoding(Encode::find_encoding('Shift_JIS'));
    $c->response->body("ใƒ†ใ‚นใƒˆ");
}

That will use the alternative encoding for a single response. slay. ๐Ÿ’…

NOTE โ€” if you manually set the content-type character set to whatever $c->encoding->mime_name is set to, we STILL encode. We do not assume your manual setting is a flag to override. Why so stubborn? Backward compatible assumptions, babes โ€” in particular Catalyst::View::TT has set a utf-8 character set in its default content-type for AGES, even though it doesn't itself do any encoding on the body response. So if you're gonna handle encoding manually, you may set $c->clear_encoding for a single request response cycle, or โ€” like in the example above โ€” set an alternative encoding.

๐ŸŒŠ Encoding with streaming type responses

Catalyst offers two approaches to streaming your body response. Again: you must remember to set your content type before streaming, since invoking a streaming response will automatically finalize and send your HTTP headers (and your content type MUST be one that matches the regular expression given above). ๐Ÿ™ˆ

Also, if you're going to override $c->encoding (or invoke $c->clear_encoding), do that before anything else! Like, first thing. fr fr.

The first streaming method is to use the write method on the response object. This one allows 'inlined' streaming and is generally used with blocking style servers.

sub stream_write :Local {
    my ($self, $c) = @_;
    $c->response->content_type('text/html');
    $c->response->write("<p>This is stream_write action โ™ฅ</p>");
}

You can call the write method as many times as you need to finish streaming all your content. Catalyst will encode each line in turn, as long as the content-type meets the 'encodable types' requirement and $c->encoding is set (and it is set, as long as you didn't change it). ๐Ÿ‘‘

NOTE โ€” if you try to change the encoding after you start the stream, that invokes an error response. But since you've already started streaming, it will not show up as an HTTP error status code โ€” instead you get error information in your body response and an error in your logs. ๐Ÿ˜ญ
NOTE โ€” if you use ->body AFTER using ->write (say, because you wanna write your HTML HEAD information as fast as possible), we expect the contents of the body to be encoded just like they normally would be if you never called ->write at all. Generally speaking, unless you're doing weird custom stuff with encoding, this will probably already just do the correct thing.

The second way to stream a response is to grab the response writer object and invoke methods on that directly:

sub stream_write_fh :Local {
    my ($self, $c) = @_;
    $c->response->content_type('text/html');

    my $writer = $c->res->write_fh;
    $writer->write_encoded('<p>This is stream_write_fh action โ™ฅ</p>');
    $writer->close;
}

You can use this just like the write method, but typically you reach for this object when you wanna do a nonblocking style response, since the writer object can be closed over or sent to a model that'll invoke it in a non blocking manner. For more on using the writer object for non blocking responses, you should review the Catalyst documentation โ€” and also you can look at several articles from last year's advent, in particular:

The main difference this year is that previously, calling ->write_fh would return the actual Plack writer object supplied by your Plack application handler, whereas now we wrap that object in a lightweight decorator object that proxies the write and close methods and supplies an additional write_encoded method. write_encoded does the exact same thing as write, except it will first encode the string when necessary. In general, if you're streaming encodable content such as HTML, this is the method to use. ๐ŸŒธ If you're streaming binary content, you should just use the write method (although if the content type is set correctly we'd skip encoding anyway โ€” but you may as well avoid the extra noop overhead).

The last style of content response that Catalyst supports is setting the body to a filehandle like object. In this case the object is passed down to the Plack application handler directly and currently we do nothing to set encoding.

sub stream_body_fh :Local {
    my ($self, $c) = @_;
    my $path = File::Spec->catfile('t', 'utf8.txt');
    open(my $fh, '<', $path) || die "trouble: $!";
    $c->response->content_type('text/html');
    $c->response->body($fh);
}

In this example we create a filehandle to a text file that contains UTF8 encoded characters. We pass this down without modification, which I think is correct since we don't want to double encode. But heads up: this may change in a future development release, so please be sure to double check the current docs and changelog. It's possible a future release will require you to set an encoding on the IO layer level so that we can be sure to properly encode at body finalization. So this is still an edge case we're writing test examples for. But for now, if you're returning a filehandle like response, you're expected to make sure you're following the PSGI specification and return raw bytes. ๐Ÿงš

๐Ÿ”„ Override the Encoding on Context

As already noted, you may change the current encoding (or remove it) by setting an alternative encoding on the context:

$c->encoding(Encode::find_encoding('Shift_JIS'));

Please note you can keep changing encoding UNTIL the headers have been finalized. The last setting always wins. ๐Ÿ‘‘ Trying to change encoding after header finalization is an error.

๐Ÿ“ฎ Setting the Content Encoding HTTP Header

In some cases you may set a content encoding on your response. Like, if you're encoding your response with gzip. In that case, you are again on your own. If we notice the content encoding header is set when we hit finalization, we skip automatic encoding:

use Encode;
use Compress::Zlib;
use utf8;

sub gzipped :Local {
    my ($self, $c) = @_;

    $c->res->content_type('text/plain');
    $c->res->content_type_charset('UTF-8');
    $c->res->content_encoding('gzip');

    $c->response->body(
      Compress::Zlib::memGzip(
        Encode::encode_utf8("manual_1 โ™ฅ")));
}

If you're using Catalyst::Plugin::Compress, you need to upgrade to the most recent version in order to be compatible with changes introduced in Catalyst 5.90080. Other plugins may require updates too (please open bugs if you find them ๐Ÿ™).

NOTE โ€” content encoding may be set to 'identify' and we will still perform automatic encoding if the content type is encodable and an encoding is present for the context.

๐Ÿฆ‹ Using Common Views

The following common views have been updated so that their tests pass with default UTF8 encoding for Catalyst:

  • Catalyst::View::TT
  • Catalyst::View::Mason
  • Catalyst::View::HTML::Mason
  • Catalyst::View::Xslate

See Catalyst::Upgrading for additional information on Catalyst extensions that require upgrades.

In general, for the common views you shouldn't need to do anything special. If your actual template files contain UTF8 literals, you should set configuration on your View to enable that. For example in TT, if your template has an actual UTF8 character in it, you should do the following:

MyApp::View::TT->config(ENCODING => 'utf-8');

However, Catalyst::View::Xslate wants to do the UTF8 encoding for you. (We assume the authors of that view did this as a workaround to the fact that until now encoding was not core to Catalyst.) So if you use that view, you either need to tell it not to encode, or you need to turn off encoding for Catalyst:

MyApp::View::Xslate->config(encode_body => 0);

or

MyApp->config(encoding=>undef);

Preference is to disable it in the View. ๐Ÿ’œ

Other views may be similar. You should review View documentation and test during upgrading. We tried to make sure most common views worked properly and noted every workaround, but if we missed something please alert the development team โ€” instead of introducing a local hack into your application that will mean nobody will ever upgrade it... ๐Ÿ˜ญ

๐ŸŒˆ Setting the response from an external PSGI application.

Catalyst::Response allows one to set the response from an external PSGI application. If you do this, and that external application sets a character set on the content-type, we clear_encoding for the rest of the response. This is done to prevent double encoding. ๐Ÿซถ

๐Ÿ’– NOTE: even if the character set of the content type is literally the same as the encoding set in $c->encoding, we still skip encoding. Yep, it's a regrettable little difference from the general rule up above โ€” the one where if the current character set matches the current encoding, we encode anyway. And yet? I think this is the correct behavior, bestie. Because that earlier rule exists only to keep backward compatibility with Catalyst::View::TT. That's it. That's the whole reason. ๐Ÿซถ

So in general: if you want Catalyst to handle encoding, avoid setting the content type character set, because Catalyst will set it automatically based on the requested response encoding. Let the app cook. ๐Ÿ”ฅ

The move is to request alternative encodings by setting $c->encoding. And if you really want manual control of encoding, always $c->clear_encoding so that the programmers who come after you are very clear about your intentions. Leave them a note, not a mystery. ๐Ÿ’…

Disabling default UTF8 encoding ๐ŸŒธ

Okay so you might run into issues with legacy code running under the default UTF8 body encoding. It happens! If that's you, you can disable it with this config setting:

MyApp->config(encoding=>undef);

Where "MyApp" is your Catalyst subclass. Slay. โœจ

But what if you don't want to nuke all the Catalyst encoding features? Fair. You can disable specific features via two additional configuration options: 'skip_body_param_unicode_decoding' and 'skip_complex_post_part_handling'.

  • 'skip_body_param_unicode_decoding' โ€” skips any attempt to decode POST parameters when creating body parameters.
  • 'skip_complex_post_part_handling' โ€” skips creation of instances of Catalyst::Request::PartData in the case where the multipart form upload contains parts with a mix of content character sets.

And now the part where I get on my soapbox ๐ŸŽค: if you believe you've discovered a bug in UTF8 body encoding, I strongly encourage you to report it โ€” and not try to hack a workaround into your local code. We also recommend you regard such a workaround as a temporary solution. It is ideal if Catalyst extension authors can start to count on Catalyst doing the right thing for encoding. Let the maintainers fix it, babe. ๐Ÿ™


Conclusion ๐Ÿ‘‘

This document has attempted to be a complete review of how UTF8 and encoding works in the current version of Catalyst, and also to document known issues, gotchas and backward compatible hacks. Please report issues to the development team. No gatekeeping here โ€” we want the bug reports. ๐Ÿฆ‹

Author ๐ŸŽ€

John Napiorkowski jjnapiork@cpan.org <mailto:jjnapiork@cpan.org>


And that's a wrap on Catalyst::UTF8(3pm), bestie! ๐ŸŒˆ This one came from perl v5.40.0, dated 2024-11-15 โ€” go forth and encode responsibly. Periodt. ๐Ÿ’–โœจ๐ŸŽ€

๐Ÿ“ฆ where this page comes from