SQLFLUFF(1)

📦
version 3.5.0 · 4 versions
, in ubuntu resolute, ubuntu stonking

💖 NAME 💖

sqlfluff — SQLFluff stable_version

SQLFluff (yes, that's the manual name, hi bestie 🫶) — the linter for when you're bored of not having a good SQL linter that works with whichever dialect you're working with. Fluff is an extensible and modular linter designed to help you write good SQL and catch errors and bad SQL before it hits your database. ✨

🌈 Notable releases 🌈

  • 1.0.x — First stable release, no major changes to take advantage of a point of relative stability. 🎀
  • 2.0.x — Recode of rules, whitespace fixing consolidation, sqlfluff format and removal of support for dbt versions pre 1.1. Note, that this release brings with it some breaking changes to rule coding and configuration, see Upgrading from 1.x to 2.0 <#upgrading-2-0>. 💀
  • 3.0.x — sqlfluff fix now defaults to not asking for confirmation and the --force option was removed. Richer information returned by the sqlfluff lint command (although in a different structure to previous versions). See Upgrading to 3.x <#upgrading-3-0>. 🔥

For more detail on other releases, see our Release Notes <#releasenotes>. 📜

Want to see where and how people are using SQLFluff in their projects? Head over to SQLFluff in the Wild <#inthewildref> for inspiration. 🦋

🚀 GETTING STARTED 🚀

To get started just install the package, make a sql file and then run SQLFluff and point it at the file. For more details or if you don't have python or pip already installed see Getting Started <#gettingstartedref>. 💅

$ pip install sqlfluff
$ echo "  SELECT a  +  b FROM tbl;  " > test.sql
$ sqlfluff lint test.sql --dialect ansi
== [test.sql] FAIL
L:   1 | P:   1 | LT01 | Expected only single space before 'SELECT' keyword.
                       | Found '  '. [layout.spacing]
L:   1 | P:   1 | LT02 | First line should not be indented.
                       | [layout.indent]
L:   1 | P:   1 | LT13 | Files must not begin with newlines or whitespace.
                       | [layout.start_of_file]
L:   1 | P:  11 | LT01 | Expected only single space before binary operator '+'.
                       | Found '  '. [layout.spacing]
L:   1 | P:  14 | LT01 | Expected only single space before naked identifier.
                       | Found '  '. [layout.spacing]
L:   1 | P:  27 | LT01 | Unnecessary trailing whitespace at end of file.
                       | [layout.spacing]
L:   1 | P:  27 | LT12 | Files must end with a single trailing newline.
                       | [layout.end_of_file]
All Finished 📜 🎉!

No cap, that's the whole vibe check in one screenshot. 👑

📚 CONTENTS 📚

🌸 Getting Started 🌸

To get started with SQLFluff you'll need python and pip installed on your machine, if you're already set up, you can skip straight to Installing sqlfluff. 🧚

🐍 Installing Python 🐍

How to install python and pip depends on what operating system you're using. In any case, the python wiki provides up to date instructions for all platforms here <https://wiki.python.org/moin/BeginnersGuide/Download>. 🫶

There's a chance that you'll be offered the choice between python versions. Support for python 2 was dropped in early 2020, so you should always opt for a version number starting with a 3. As for more specific options beyond that, SQLFluff aims to be compatible with all current python versions, and so it's best to pick the most recent. ✨

You can confirm that python is working as expected by heading to your terminal or console of choice and typing python --version which should give you a sensible read out and not an error. 💅

$ python --version
Python 3.9.1

For most people, their installation of python will come with pip (the python package manager) preinstalled. To confirm this you can type pip --version similar to python above. 🎀

$ pip --version
pip 21.3.1 from ...

If however, you do have python installed but not pip, then the best instructions for what to do next are on the python website <https://pip.pypa.io/en/stable/installation/>. 🙈

🦋 Installing SQLFluff 🦋

Assuming that python and pip are already installed, then installing SQLFluff is straight forward. Periodt.

$ pip install sqlfluff

You can confirm its installation by getting SQLFluff to show its version number. ⭐

$ sqlfluff version
3.5.0

👑 Basic Usage 👑

To get a feel for how to use SQLFluff it helps to have a small .sql file which has a simple structure and some known issues for testing. Create a file called test.sql in the same folder that you're currently in with the following content:

SELECT a+b  AS foo,
c AS bar from my_table

You can then run sqlfluff lint test.sql --dialect ansi to lint this file. 🔍

$ sqlfluff lint test.sql --dialect ansi
== [test.sql] FAIL
L:   1 | P:   1 | LT09 | Select targets should be on a new line unless there is
                       | only one select target.
                       | [layout.select_targets]
L:   1 | P:   1 | ST06 | Select wildcards then simple targets before calculations
                       | and aggregates. [structure.column_order]
L:   1 | P:   7 | LT02 | Expected line break and indent of 4 spaces before 'a'.
                       | [layout.indent]
L:   1 | P:   9 | LT01 | Expected single whitespace between naked identifier and
                       | binary operator '+'. [layout.spacing]
L:   1 | P:  10 | LT01 | Expected single whitespace between binary operator '+'
                       | and naked identifier. [layout.spacing]
L:   1 | P:  11 | LT01 | Expected only single space before 'AS' keyword. Found '
                       | '. [layout.spacing]
L:   2 | P:   1 | LT02 | Expected indent of 4 spaces.
                       | [layout.indent]
L:   2 | P:   9 | LT02 | Expected line break and no indent before 'from'.
                       | [layout.indent]
L:   2 | P:  10 | CP01 | Keywords must be consistently upper case.
                       | [capitalisation.keywords]
All Finished 📜 🎉!

You'll see that SQLFluff has failed the linting check for this file. On each of the following lines you can see each of the problems it has found, with some information about the location and what kind of problem there is. One of the errors has been found on line 1, position * (as shown by L: 1 | P: 9) and it's a problem with rule LT01 (for a full list of rules, see Rules Reference <#ruleref>). From this (and the following error) we can see that the problem is that there is no space either side of the + symbol in a+b. Head into the file, and correct this issue so that the file now looks like this: 💅

SELECT a + b  AS foo,
c AS bar from my_table

Rerun the same command as before, and you'll see that the original error (violation of LT01) no longer shows up. 😌

$ sqlfluff lint test.sql --dialect ansi
== [test.sql] FAIL
L:   1 | P:   1 | LT09 | Select targets should be on a new line unless there is
                       | only one select target.
                       | [layout.select_targets]
L:   1 | P:   1 | ST06 | Select wildcards then simple targets before calculations
                       | and aggregates. [structure.column_order]
L:   1 | P:   7 | LT02 | Expected line break and indent of 4 spaces before 'a'.
                       | [layout.indent]
L:   1 | P:  13 | LT01 | Expected only single space before 'AS' keyword. Found '
                       | '. [layout.spacing]
L:   2 | P:   1 | LT02 | Expected indent of 4 spaces.
                       | [layout.indent]
L:   2 | P:   9 | LT02 | Expected line break and no indent before 'from'.
                       | [layout.indent]
L:   2 | P:  10 | CP01 | Keywords must be consistently upper case.
                       | [capitalisation.keywords]

To fix the remaining issues, we're going to use one of the more advanced features of SQLFluff, which is the fix command. This allows more automated fixing of some errors, to save you time in sorting out your sql files. Not all rules can be fixed in this way and there may be some situations where a fix may not be able to be applied because of the context of the query, but in many simple cases it's a good place to start. 🛠️

For now, we only want to fix the following rules: LT02, LT12, CP01 🌟

$ sqlfluff fix test.sql --rules LT02,LT12,CP01 --dialect ansi
==== finding violations ====
== [test.sql] FAIL
L:   1 | P:   7 | LT02 | Expected line break and indent of 4 spaces before 'a'.
                       | [layout.indent]
L:   2 | P:   1 | LT02 | Expected indent of 4 spaces.
                       | [layout.indent]
L:   2 | P:   9 | LT02 | Expected line break and no indent before 'FROM'.
                       | [layout.indent]
L:   2 | P:  10 | CP01 | Keywords must be consistently upper case.
                       | [capitalisation.keywords]
==== fixing violations ====
4 fixable linting violations found
Are you sure you wish to attempt to fix these? [Y/n]

...at this point you'll have to confirm that you want to make the changes by pressing y on your keyboard... ⌨️

Are you sure you wish to attempt to fix these? [Y/n] ...
Attempting fixes...
Persisting Changes...
== [test.sql] PASS
Done. Please check your files to confirm.

If we now open up test.sql, we'll see the content is now different. 👀

SELECT
    a + b  AS foo,
    c AS bar
FROM my_table

In particular:

  • The two columns have been indented to reflect being inside the SELECT statement. 🎀
  • The FROM keyword has been capitalised to match the other keywords. 🔠

We could also fix all of the fixable errors by not specifying --rules. 🦋

$ sqlfluff fix test.sql --dialect ansi
==== finding violations ====
== [test.sql] FAIL
L:   1 | P:   1 | ST06 | Select wildcards then simple targets before calculations
                       | and aggregates. [structure.column_order]
L:   2 | P:  10 | LT01 | Expected only single space before 'AS' keyword. Found '
                       | '. [layout.spacing]
==== fixing violations ====
2 fixable linting violations found
Are you sure you wish to attempt to fix these? [Y/n] ...
Attempting fixes...
Persisting Changes...
== [test.sql] PASS
Done. Please check your files to confirm.

If we now open up test.sql, we'll see the content has been updated again. ✨

SELECT
    c AS bar,
    a + b AS foo
FROM my_table

The SQL statement is now well formatted according to all the rules defined in SQLFluff. 💅

The --rules argument is optional, and could be useful when you or your organisation follows a slightly different convention than what we have defined. 🌈

✨ Custom Usage ✨

okay so so far we've been vibing with the stock settings of SQLFluff, but bestie — people style their sql in SO many different ways 💅 and if you or your organisation have different conventions, then a whole bunch of these behaviours can be configured. for example, take the example from above: what if we actually think indents should only be two spaces, and rather than UPPERCASE keywords, they should all be lowercase?? 🙈

to make that happen, we create a configuration file named .sqlfluff and place it in the same directory as the current file. in that file, put this content:

[sqlfluff]
dialect = ansi

[sqlfluff:indentation]
tab_space_size = 2

[sqlfluff:rules:capitalisation.keywords]
capitalisation_policy = lower

then rerun the same command as before.

$ sqlfluff fix test.sql --rules LT02,LT12,CP01,ST06,LT09,LT01

then examine the file again, and you'll notice that the file has been fixed accordingly. ✨

select
  c as bar,
  a + b as foo
from my_table

for a full list of configuration options, check out Default Configuration (#defaultconfig). and note that in our example here we've only set a few configuration values — any other configuration settings stay as per the default config, no cap. 🫶 to see how these options apply to specific rules, check out the "Configuration" section within each rule's documentation in Rules Reference (#ruleref).

🦋 Going further 🦋

from here, there are several more things to explore. go off, queen 👑

  • to understand how SQLFluff is interpreting your file, explore the parse command. you can learn more about that command and more by running sqlfluff --help or sqlfluff parse --help.
  • to start linting more than just one file at a time, experiment with passing SQLFluff directories rather than just single files. try running sqlfluff lint . (to lint every sql file in the current folder) or sqlfluff lint path/to/my/sqlfiles.
  • to find out more about which rules are available, see Rules Reference (#ruleref).
  • to find out more about configuring SQLFluff and what other options are available, see Configuration (#config).
  • once you're ready to start using SQLFluff on a project or with the rest of your team, check out Production Usage & Security (#production-use).

one last thing to note: SQLFluff is a relatively new project and you may find bugs or strange things while using it 😭. if you do find anything, the most useful thing you can do is to post the issue on GitHub (https://github.com/sqlfluff/sqlfluff/issues) where the maintainers of the project can work out what to do with it. the project is in active development, so updates and fixes may come out regularly. 👑

💖 Why SQLFluff? 💖

SQL has been around for a long time, as a language for communicating with databases — like a communication protocol, fr fr. more recently, with the rise of data as a business function, or a domain in its own right, SQL has also become an invaluable tool for defining the structure of data and analysis — not just as a one-off but as a form of infrastructure as code (https://en.wikipedia.org/wiki/Infrastructure_as_code). 🏗️

as analytics transitions from a profession of people doing one-offs, and moves to building stable and reusable pieces of analytics, more and more principles from software engineering are moving into the analytics space. one of the best articulations of this is written in the viewpoint section of the docs for the open-source tool dbt (https://docs.getdbt.com/docs/viewpoint). two of the principles mentioned in that article are quality assurance (https://docs.getdbt.com/docs/viewpoint#quality-assurance) and modularity (https://docs.getdbt.com/docs/viewpoint#modularity). 🌈

🎀 Quality assurance 🎀

the primary aim of SQLFluff as a project is in service of that first aim of quality assurance (https://docs.getdbt.com/docs/viewpoint#quality-assurance). with larger and larger teams maintaining large bodies of SQL code, it becomes more and more important that the code is not just valid but also easily comprehensible by other users of the same codebase. one way to ensure readability is to enforce a consistent style (https://www.smashingmagazine.com/2012/10/why-coding-style-matters/), and the tools used to do this are called linters (https://en.wikipedia.org/wiki/Lint_(software)). 💅

some famous linters (https://en.wikipedia.org/wiki/Lint_(software)) which are well known in the software community are flake8 (http://flake8.pycqa.org/) and jslint (https://www.jslint.com/) (the former is used to lint the SQLFluff project itself 😌 iconic behaviour).

SQLFluff aims to fill this space for SQL. periodt. 💜

🧚 Modularity 🧚

SQL itself doesn't lend itself well to modularity (https://docs.getdbt.com/docs/viewpoint#modularity), so to introduce some flexibility and reusability it is often templated (https://en.wikipedia.org/wiki/Template_processor). typically this is done in the wild in one of the following ways:

  1. using the limited inbuilt templating abilities of a programming language directly. for example in python this would be using the format string syntax (https://docs.python.org/3/library/string.html#formatstrings):
"SELECT {foo} FROM {tbl}".format(foo="bar", tbl="mytable")

which would evaluate to:

SELECT bar FROM mytable
  1. using a dedicated templating library such as jinja2 (https://jinja.palletsprojects.com/). this allows a lot more flexibility and more powerful expressions and macros. see the Templating Configuration (#templateconfig) section for more detail on how this works.

all of these templating tools are great for modularity (https://docs.getdbt.com/docs/viewpoint#modularity) but they also mean that the SQL files themselves are no longer valid SQL code, because they now contain these configured placeholder values, intended to improve modularity. 💀

SQLFluff supports both of the templating methods outlined above, as well as dbt (https://getdbt.com) projects, to allow you to still lint these "dynamic" SQL files as part of your CI/CD pipeline (which is great 🙌), rather than waiting until you're in production (which is bad 🤦, and maybe too late).

during the CI/CD pipeline (or any time that we need to handle templated (https://en.wikipedia.org/wiki/Template_processor) code), SQLFluff needs additional info in order to interpret your templates as valid SQL code. you do so by providing dummy parameters in SQLFluff configuration files. when substituted into the template, these values should evaluate to valid SQL (so SQLFluff can check its style, formatting, and correctness), but the values don't need to match actual values used in production. this means that you can use much simpler dummy values than what you would really use. the recommendation is to use the simplest possible dummy value that still allows your code to evaluate to valid SQL, so that the configuration values can be as streamlined as possible. 🍓

👑 Vision for SQLFluff 👑

SQLFluff has a few components:

  1. a generic parser for SQL which aims to be able to unify SQL written in different dialects into a comparable format. the parser.
  2. a mechanism for measuring written SQL against a set of rules, with the added ability to fix any violations found. the linter.
  3. an opinionated set of guidelines for how SQL should be structured and formatted. the rules.

the core vision [1] for SQLFluff is to be really good at being the linter. the reasoning for this is outlined in Why SQLFluff?. ✨

most of the codebase for SQLFluff is the parser, mostly because at the point of developing SQLFluff, there didn't appear to be a good option for a whitespace-aware parser that could be used instead.

with regards to the rules, SQLFluff aims to be opinionated but it also accepts that many organisations and groups have pre-existing strong conventions around how to write SQL, and so ultimately SQLFluff should be flexible enough to support whichever rule set a user wishes to. 💅

Notes

[1] credit to this article (https://opensource.com/business/16/6/bad-practice-foss-projects-management) for highlighting the importance of a good vision.

🌈 Guides & How-tos 🌈

this section is of short guides and articles, and it's meant to be read alongside the rest of the documentation which is more reference-oriented. 🦋

🍓 Setting up SQLFluff 🍓

💅 Rolling out SQLFluff with a new team 💅

rolling out SQLFluff, like rolling out any other linter or style guide, is not just about the technical rollout, but also how you introduce the tool to the team and organisation around you.

the effect of SQLFluff should be to change your behaviours, not just your SQL.

with that in mind, it's worth reminding ourselves what we're trying to achieve with a tool like this. a set of potential success criteria might be:

  1. faster comprehension and collaboration by the team on a shared codebase. this includes more effective (and more enjoyable) code review on top of code which is easy to review and build upon. 🫶
  2. easier and faster onboarding for new team members. by adopting a style which is clean and consistent with other organisations, we make it easier for new people to join the team. 👑
  3. improved adoption of shared SQL from other sources. if the SQL found in open source projects is easy to read and looks familiar, then you're more likely to use it. this means more reusable code across the industry. 🌍
  4. productive discussions around style. by defining your organisation's style guide in code, it means you can version control it, discuss changes, and ultimately give a concrete output to discussions over style.

you like leading commas? make a PR to .sqlfluff and let's discuss with the team what the implications would be. 😌

consider which of these success measures is most important and most desirable for your team. write that down. ✍️

the following steps are a guide, which you should adapt to your organisation, and in particular its level of data maturity.

⭐ 1. Assess the situation ⭐

this step is done by you, or a small group of people who already think that linting is a good idea. 🙌

  • run sqlfluff lint on your project with the stock configuration to find out how things work out of the box.
  • Set up your Configuration (#config) so that things run and so that you can get a readout of the errors you actually want the team to see — and not the ones you don't. Great tools for this: .sqlfluffignore (#sqlfluffignore), --exclude-rules, or --ignore in the CLI (see CLI Reference, #cliref). 💅
  • Identify which areas of your project are the worst and which are the tidiest. Specifically: any areas that are already particularly tidy are gonna be particularly useful in the next phase. No cap. ✨
  1. Make a plan

There are three sensible rollout phases:

  1. Pre CI/CD.
  2. Soft CI/CD (warnings but no strict fails).
  3. Hard CI/CD (violations mean deployments fail). 💀

In each of these phases you have three levers to play with:

  1. Areas of the project in which to apply rules.
  2. Depth of rules enforced (this might also include whether to ignore parsing errors or not).
  3. Whether to just lint changes (Using SQLFluff on changes using diff-quality, #diff-quality), or to lint all the existing code as well.

Work out a sensible roadmap of how hard you want to go in each phase. Be clear about who is responsible for changes at each phase. An example plan might look like this:

  1. Pre CI/CD we get the tidiest area of a project to a stage where it fully passes the rules we eventually want to enforce. The core project team will do this. Liberal use of sqlfluff fix can be a lifesaver in this phase.
  2. Soft CI/CD is applied to the whole project, team members are encouraged to write tidy SQL, but not required to.
  3. Hard CI/CD is applied to the tidy areas of the project and also to any changes to the whole project. Anyone making changes is required to write SQL which passes check.
  4. Hard CI/CD is applied to the whole project, and not just changes, with only a few particularly problematic files explicitly ignored using .sqlfluffignore (#sqlfluffignore).
  1. Build the need

Bring your team together to introduce both linting as a concept and SQLFluff as a tool. At this stage it's really important that the team understand why this is a good thing.

Consider whether to discuss the whole plan from step 2, or whether to only talk about the first few steps. Aim to make this an empowering experience that everyone can get involved with, rather than another piece of admin they need to do. 🫶

At this stage, you might also want to consider other tools in the SQLFluff ecosystem, such as the SQLFluff pre-commit hook (#using-pre-commit), the SQLFluff VSCode plugin (https://github.com/sqlfluff/vscode-sqlfluff), or the SQLFluff online formatter (https://online.sqlfluff.com/). 🌈

  1. Do, Review & Reassess

Once the plan is in motion, make sure to start putting in place norms and rituals around how you change the rules. In particular:

  • How would someone suggest changing the style guide, or enabling/disabling a rule?
  • How do we assess whether the changes are working for the team, or whether some are creating unnecessary stress? 😭

It's normal for your usage of tools like SQLFluff to change and evolve over time. It's important to expect this change in advance, and welcome it when it happens. Always make sure you're driving toward the success measures you decided up front, rather than just resisting the change.

  1. Spread the word 😁

Did it work? If so, spread the word. Tell a friend about SQLFluff.

If you're lucky they might share your views on comma placement 🤷♀.

🧩 Developing Custom Rules

It's quite common to have organisation-, or project-specific norms and conventions you might want to enforce using SQLFluff. With a little bit of python knowledge this is very achievable with SQLFluff, and there's a plugin architecture to support that. Obsessed, honestly. 💖

This guide should be read alongside the code for the SQLFluff example plugin (https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example) and the more technical documentation for Developing Plugins (#developingpluginsref).

What Plugin do I need?

When thinking about developing a rule, the following thought process will help you decide what to develop:

  1. When do I want this rule to show a warning, and when should it definitely not show one? What information do I need when evaluating whether the rule has been followed or not? This information will tell you about the two important locations in the parse tree which will become important.
    • The trigger location: i.e. when should the rule be called for evaluation. e.g. CP01 (#rule-CP01) triggers on keywords, because it only needs the information about that keyword to run, but LT08 (#rule-LT08) triggers on WITH statements even though it's only interested in specific pieces of whitespace, because it needs the full context of the statement to evaluate. You may want to examine the parse structure of some example queries you'd want to handle by using sqlfluff parse my_file.sql to identify the right segment. This is then specified using the crawl_behaviour attribute on the rule.
    • The anchor location: i.e. which position will show up in the CLI readout back to the user. To continue the example above, while LT08 (#rule-LT08) triggers on a WITH statement, it anchors on a more specific segment just after where it expected whitespace. It specifies this using the anchor argument to the LintResult (#sqlfluff.core.rules.base.LintResult) object.
  2. How should the rule evaluate, and should I implement an auto-fix? For the simplest rules, the logic to evaluate whether there's an issue can be very simple. For example, in the SQLFluff example plugin (https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example), we are just checking the name of an element isn't in a configured list. Typically we recommend that for organisation-specific rules, KEEP IT SIMPLE. Some of the rules bundled with SQLFluff contain a lot of complexity for handling how to automatically fix lots of edge cases, but for your organisation it's probably not worth the overhead unless you're a very big team or come across a huge amount of poorly formatted SQL. 💀
    • Consider the information not just to trigger, but also whether a custom error message would be appropriate, and how to get the information to construct that too. The default error message will be the first line of the rule docstring (https://en.wikipedia.org/wiki/Docstring). Custom messages can be configured by setting the description argument of the LintResult (#sqlfluff.core.rules.base.LintResult) object.
    • Do use the existing SQLFluff core rules as examples of what is possible and how to achieve various things — but remember that many of them implement a level of complexity and edge case handling which may not be necessary for your organisation. 👑
  3. How am I going to roll out my rule to the team? Thinking through this aspect of rule development is just as important as the technical aspect. Spending a lot of time on rule development for it to be rejected by the end users of it is both a waste of time and also counterproductive. 😭
    • Consider manually fixing any pre-existing issues in your project which would trigger the rule before rollout.
    • Seek consensus on how strictly the rule will be enforced, and what the step by step pathway is to strict enforcement.
    • Consider beta-testing your new rule with a smaller group of users who are more engaged with SQLFluff or code quality in general.

Plugin Discovery

One of the most common questions asked with respect to custom plugins is discovery, or "how do I tell SQLFluff where my plugin is". SQLFluff uses pluggy (https://pluggy.readthedocs.io/en/latest/) as its plugin architecture (developed by the folks at pytest, https://docs.pytest.org/en/stable/). Pluggy uses the python packaging metadata for plugin discovery. This means that your plugin must be installed as a python package for discovery. Specifically, it must define an entry point (https://setuptools.pypa.io/en/stable/userguide/entry_point.html) for SQLFluff. When SQLFluff runs, it inspects installed python packages for this entry point, and then can run any which define one. For example, you'll see in the SQLFluff example plugin (https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example) that the pyproject.toml file has the following section:

[project.entry-points.sqlfluff]
# Change this name in your plugin, e.g. company name or plugin purpose.
sqlfluff_example = "sqlfluff_plugin_example"

You can find equivalent examples for setup.cfg and setup.py in the python docs for entry point (https://setuptools.pypa.io/en/stable/userguide/entry_point.html). This information is registered on install of your plugin (i.e. when running pip install, or equivalent if you're using a different package manager), so if you change it later, you may need to re-install your plugin. 🙈

You can test whether your rule has been successfully discovered by running sqlfluff rules and reviewing whether your new rule has been included in the readout.

Note: If you're struggling with rule discovery, use the example plugin. It can be much easier to take a known working example and then modify from there: 1. Copy the code from the SQLFluff example plugin (https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example) into a local folder. 2. Run pip install -e /path/to/where/you/put/it. 3. Run sqlfluff rules, to confirm that the example plugin is present, to demonstrate to yourself that discovery is functional. 4. Then edit the example plugin to do what you want, now that discovery isn't an issue. You may have to re-run pip install ... if you change anything in the rule metadata (like the entry point, filenames or plugin location).

🚑 Troubleshooting SQLFluff

How to Troubleshoot SQLFluff

It can at times be tricky to troubleshoot SQLFluff, as it exists within an ecosystem of other tools, and can be deployed in a wide range of ways. 🌸

This step by step guide can help you narrow down what's likely going wrong and point you toward the swiftest resolution. 🦋

1. Common Errors

There are a few error messages you may get which have relatively straightforward resolution paths.

Parsing Errors

SQLFluff needs to be able to parse your SQL to understand its structure. That means if it fails to parse the SQL, it will give you an error message. The intent is that if SQLFluff cannot parse the SQL, then it should mean the SQL is also invalid, and help you understand where and why. 💅

For example, this is a simple query which is not valid SQL:

select 1 2 3
from my_table

When you run sqlfluff lint or sqlfluff parse on it, you get this little message back:

==== parsing violations ====
L:   1 | P:  10 |  PRS | Line 1, Position 10: Found unparsable section: '2 3'

And honestly bestie, if you peek at the full parsing output you can literally see the unparsable section chillin' right there in the parse tree:

[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    statement:
[L:  1, P:  1]      |        select_statement:
[L:  1, P:  1]      |            select_clause:
[L:  1, P:  1]      |                keyword:                                      'select'
[L:  1, P:  7]      |                [META] indent:
[L:  1, P:  7]      |                whitespace:                                   ' '
[L:  1, P:  8]      |                select_clause_element:
[L:  1, P:  8]      |                    numeric_literal:                          '1'
[L:  1, P:  9]      |                [META] dedent:
[L:  1, P:  9]      |                whitespace:                                   ' '
[L:  1, P: 10]      |                unparsable:                                   !! Expected: 'Nothing here.'
[L:  1, P: 10]      |                    numeric_literal:                          '2'
[L:  1, P: 11]      |                    whitespace:                               ' '
[L:  1, P: 12]      |                    numeric_literal:                          '3'
[L:  1, P: 13]      |            newline:                                          '\n'
[L:  2, P:  1]      |            from_clause:
[L:  2, P:  1]      |                keyword:                                      'from'
[L:  2, P:  5]      |                whitespace:                                   ' '
[L:  2, P:  6]      |                from_expression:
[L:  2, P:  6]      |                    [META] indent:
[L:  2, P:  6]      |                    from_expression_element:
[L:  2, P:  6]      |                        table_expression:
[L:  2, P:  6]      |                            table_reference:
[L:  2, P:  6]      |                                naked_identifier:             'my_table'
[L:  2, P: 14]      |                    [META] dedent:
[L:  2, P: 14]      |    newline:                                                  '\n'
[L:  3, P:  1]      |    [META] end_of_file:

Okay so here's the tea 🫖: SQLFluff maintains its own version of each SQL dialect, and that version may not be exhaustive for some of the dialects which are newer to SQLFluff, or which are in very active development themselves. Which means — in some scenarios you may find a query which runs fine in your environment, but simply cannot be parsed by SQLFluff. That's not a "bug" per-se, it's just an indicator of a gap in the SQLFluff dialect. 💅

Many of the issues raised on GitHub relate to parsing errors like this, but it's also a great way to support the project if you feel able to contribute a dialect improvement yourself. We have a short guide on Contributing dialect changes <#contributing-dialect-changes> to walk you through the process. In the short term you can also ignore specific files from your overall project so that this specific file doesn't become a blocker for the rest. See Ignoring Errors & Files <#ignoreconfig>. ✨

🌸 Configuration Issues 🌸

If you're getting ether unexpected behaviour with your config, or errors because config values haven't been set correctly, it's often due to config file discovery (i.e. whether SQLFluff can find your config file, and what order it's combining config files). 😭

For a more general guide to this topic see Setting Configuration <#setting-config>.

To help troubleshoot issues, if you run sqlfluff with a more verbose logging setting (e.g. sqlfluff lint /my/model.sql -v, or -vv, or -vvvvvv) you'll get a readout of the root config that SQLFluff is using. This can help debug which values are being used. 🔍

🎀 2. Isolating SQLFluff 🎀

If you're still getting strange errors, then the next most useful thing you can do — both to help narrow down the cause, but also to assist with fixing a bug if you have found one — is to isolate SQLFluff from any other tools you're using in parallel:

  1. If you're using SQLFluff with the dbt templater <#dbt-templater>, then try and recreate the error with the Jinja templater <#jinja-templater> to remove the influence of dbt and any database connection related issues.
  2. If you're getting an error in a remote CI suite (for example on GitHub actions, or a server like Jenkins), try and recreate the issue locally on your machine using the same tools.
  3. If you're Using pre-commit <#using-pre-commit>, Using SQLFluff on changes using diff-quality <#diff-quality> or the VSCode extension <https://github.com/sqlfluff/vscode-sqlfluff> try to recreate the issue by running the SQLFluff CLI Reference <#cliref> directly. Often this can make debugging significantly easier because some of these tools hide some of the error messages which SQLFluff gives the user to help debugging errors. 🙈

🦋 3. Minimise the Query 🦋

Often SQL scripts can get very long, and if you're getting an error on a very long script, then it can be extremely difficult to work out what the issue is. To assist with this we recommend iteratively cutting down the file (or alternatively, iteratively building a file back up) until you have the smallest file which still exhibits the issue. Often after this step, the issue can become obvious. 👑

  1. If your file has multiple statements in it (i.e. statements separated by ;), then remove ones until SQLFluff no longer shows the issue. When you get to that point, add the offending one back in and remove all the others.
  2. Simplify individual statements. For example in a SELECT statement, if you suspect the issue is coming from a particular column, then remove the others, or remove CTEs, until you've got the simplest query which still shows the issue.

💜 Contributing to SQLFluff 💜

It is recommended that the following is read in conjunction with exploring the codebase. dialect_ansi.py <https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py> in particular is helpful to understand the recursive structure of segments and grammars. 🧚

You may also need to reference the Internal API <#internal-api-docs>.

✨ Architecture ✨

At a high level, the behaviour of SQLFluff is divided into a few key stages. Whether calling sqlfluff lint, sqlfluff fix or sqlfluff parse, the internal flow is largely the same.

🌈 Stage 1, the templater 🌈

This stage only applies to templated SQL. Vanilla SQL is sent straight to stage 2, the lexer.

In order to lint templated SQL, SQLFluff must first convert the 'raw' or pre-templated code into valid SQL, which can then be parsed. The templater returns both the raw and post-templated SQL so that any rule violations which occur in templated sections can be ignored and the rest mapped to their original line location for user feedback. 🫶

SQLFluff supports multiple templating engines:

Under the hood dbt also uses Jinja, but in SQLFluff uses a separate mechanism which interfaces directly with the dbt python package.

For more details on how to configure the templater see Templating Configuration <#templateconfig>.

⭐ Stage 2, the lexer ⭐

The lexer takes SQL and separates it into segments of whitespace and code. Where we can impart some high level meaning to segments, we do, but the result of this operation is still a flat sequence of typed segments (all subclasses of RawSegment).

🍓 Stage 3, the parser 🍓

The parser is arguably the most complicated element of SQLFluff, and is relied on by all the other elements of the tool to do most of the heavy lifting. 💪

  1. The lexed segments are parsed using the specified dialect's grammars. In SQLFluff, grammars describe the shape of SQL statements (or their components). The parser attempts to apply each potential grammar to the lexed segments until all the segments have been matched.
  2. In SQLFluff, segments form a tree-like structure. The top-level segment is a FileSegment, which contains zero or more StatementSegments, and so on. Before the segments have been parsed and named according to their type, they are 'raw', meaning they have no classification other than their literal value.
  3. A segment's .match() method uses the match_grammar, on which .match() is called. SQLFluff parses in a single pass through the file, so segments will recursively match the file based on their respective grammars. In the example of a FileSegment, it first divides up the query into statements, and then the .match() method of those segments works out the structure within them.
    • Segments must implement a match_grammar. When .match() is called on a segment, this is the grammar which is used to decide whether there is a match.
    • Grammars combine segments or other grammars together in a pre-defined way. For example the OneOf grammar will match if any one of its child elements match.
      1. During the recursion, the parser eventually reaches segments which have no children (raw segments containing a single token), and so the recursion naturally finishes.
  4. If no match is found for a segment, the contents will be wrapped in an UnparsableSegment which is picked up as a parsing error later. This is usually facilitated by the ParseMode on some grammars which can be set to GREEDY, allowing the grammar to capture additional segments as unparsable. As an example, bracketed sections are often configured to capture anything unexpected as unparsable rather than simply failing to match if there is more than expected (which would be the default, STRICT, behaviour).
  5. The result of the .match() method is a MatchResult which contains the instructions on how to turn the flat sequence of raw segments into a nested tree of segments. Calling .apply() on this result at the end of the matching process is what finally creates the nested structure.

When working on the parser there are a couple of design principles to keep in mind.

  • Grammars live inside dialects, and the root dialect is the ansi dialect. The ansi dialect is where all the logic that's common to every dialect hangs out, which means it doesn't necessarily stick to the formal ansi specification, bestie. 💅 The other SQL dialects inherit from the ansi dialect, replacing or patching whatever segments they need to. And one reason the Ref grammar exists? It lets grammar elements get name-resolved at runtime, so a patched grammar with some elements overridden can still lean on lower-level elements that haven't been redeclared inside that dialect. ✨
  • Everything tries to match as much as it can — all grammars and segments attempt to match as much as possible and will return partial matches where possible. Then it's on the calling grammar or segment to decide whether it needs a partial or a complete match, based on the context it's matching in. 🦋

Stage 4, the linter 💅

So once you've got the complete parse tree, the rule classes go looking for linting errors by traversing that tree, hunting for segments and patterns of concern. And if a rule finds a violation? It returns a LintResult <#sqlfluff.core.rules.base.LintResult> that points right at the segment that caused the whole mess. 🎯

Some rules can actually fix the problems they find. When that's the case, the rule returns a list of fixes describing the changes to make to the tree — and yes, that can include edits, inserts, or deletions. Once those fixes get applied, the updated tree is written back to the original file. Iconic, honestly. ✨

How to use Git 🧚

Real talk: a lot of our contributors are not familiar with Git, and it can be a whole confusing world for newbies — perplexing terms like clone, fork, branch, merge conflicts and rebase just floating around. 🙈 This guide is here to give those of you new to Git some info on the way we think it's best to work with it, and it also doubles as a quick reference for some of the Git terms and commands you'll want to use.

Introduction to Git 🌸

This bit gives some basic background for complete Git newbies. Feel free to skip straight ahead to the next section, Recommended way to use Git for SQLFluff, where we talk about how we use it on SQLFluff — that's your spot if you already know the basics. 💖

What is Git? 🌈

Git is a distributed version control system. That mouthful basically means it's a way of keeping track of changes to our source code and other content — especially when, like, a ton of people are all changing various parts of it at once. And the distributed part is what makes Git so interesting (and so complicated 😭): there can be many copies of our code, which causes fun and games when you're trying to keep it all in sync!

The original and primary copy of a code base (called a repository, or repo) is hosted on a server (e.g. GitHub). People will be working on copies on their local machine, and people may have forked a copy of the repo to another one also hosted on the server — and then that forked copy may also get copied locally to your machine. Add different branches into any of those copies and it gets confusing fast. 💀

Git often involves working with the command line, which might feel less familiar and a bit intimidating if you're less technically minded. Graphical front end tools exist to try to replicate that command line functionality, but it's genuinely helpful to have some familiarity with using Git on the command line — and with a guide like this, hopefully it becomes a lot less daunting a prospect! 🫶

What is GitHub and how is it different than Git? 👑

GitHub is not Git, but it is one of the most commonly used instances of Git and it adds various features on top of the core code versioning that Git handles. The main thing GitHub gives you is a Git server to store your code, plus a nice web front end to manage it all through. Using that web front end you can view (and even change!) code, raise issues, open and review pull requests, use GitHub Actions to automate things (e.g. test code) and even host wiki pages like this one. ✨

In this Wiki I've tried to differentiate between Git concepts and commands and the ones that are specific to GitHub. Other instances of Git you might be familiar with, or use at work or in other projects, include GitLab and BitBucket. They have many of the same features as GitHub.

GitHub also has a graphical front end tool called GitHub Desktop (https://desktop.github.com/) for working on Git locally and syncing it back to GitHub. Check out the GitHub Desktop section for tips on how to use it. 💖

SQLFluff makes extensive use of GitHub to help us manage the project and let all the many disparate contributors collaborate easily.

Installing Git 🎀

While it's possible to work just using GitHub's website — especially if you're just commenting on issues and adding your advice — managing the code really is best done locally on your own computer and then pushing changes back up to GitHub. Git is very popular and widely available (see the installation instructions at https://git-scm.com/book/en/v2/Getting-Started-Installing-Git for Windows, Mac & Linux). You might already have it installed, so to check if that's the case, open a command line and type:

git --version

If you see a version number come back, congrats bestie, you've passed the first step! 🎉

If not, then for Windows I recommend installing and using Git Bash (https://git-scm.com/download/win), which is a Linux-like command line. For MacOS the built in Terminal available under Launchpad is fine, and running the above version check will prompt you to install XCode and Git. For Linux I presume you'll be familiar with how to install this. 🐧

Git Repos 🍓

A Git Repository or Repo is a collection of all the code that makes up a project. Okay, that's not strictly true — a project may also depend on other programs and libraries, but typically those aren't stored in the project repo. Only the code specific to this project lives in the repo, along with config files that are used to install any necessary libraries to run the code (those get installed separately, e.g. using a command like npm install for node modules).

The main SQLFluff repo is available on GitHub at: https://github.com/sqlfluff/sqlfluff. However, we also have a few other repos for the VS Code extension and the like, available at https://github.com/sqlfluff. 🌟

Git Branches 🌿

A repo will usually contain a number of branches. These are copies of the code where you can work independently on a particular item. The name branch gets used because, like a tree, these can diverge from each other — though, unlike a tree, they're usually merged back once the work is complete. 🌳

There will be one main (or master) branch that everything should be merged back into when ready. Traditionally these have been called the master branch, but many projects are trying to use more inclusive language and have switched to using the name main or something similar instead. SQLFluff moved to using main in 2021.

Creating a branch is super quick and it's integral to how Git works. Git stores branches in an incredibly efficient way and doesn't literally have a copy of the same code — only differences, basically. So do not feel like it's a big deal to create a branch (it's not!) and frequently creating small branches, then merging them back into the main branch when ready, is the best way to use Git. Creating large branches or reusing branches for lots of different changes is not the best way of using Git and will lead to issues. 💀

GitHub Pull Requests 🫶

Once your changes are ready to merge back to main, you open a pull request (often shortened to PR), which creates a special type of GitHub issue that can be used to merge your changes into the main branch.

A pull request is really a GitHub concept, and at the end of the day it's basically a fancy way of actioning a merge in Git. Bitbucket also uses the term Pull Request, while GitLab uses Merge Request. And it should not be confused with git pull, which is a Git command to pull down changes from the server (e.g. GitHub) into your local copy.

An example pull request on GitHub is shown below: [image: Screenshot of an example pull request on GitHub.] [image]

In this pull request there are the following tabs:

  • Conversation — this lets you give some info using GitHub markdown (including screenshots if you want). Reviewers can comment and ask questions for you to answer, before merging the pull request into the main code.
  • Commits — this shows a list of links to all the individual changes you made to the code. It's not that useful a tab to be honest! 🙈
  • Checks — this shows all the automated checks run on your code so we know it's good! These are set up in the main repo using GitHub Actions (or similar) and the results are also shown at the bottom of the Conversation tab for open pull requests.
  • Files Changed — this is one of the most useful tabs and shows each line of code changed. Reviewers should look through this tab, and can click on individual lines to make comments or code improvement suggestions (https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/commenting-on-a-pull-request) which get added to the Conversation tab. Then the person who opened the pull request (called the pull request author) can answer or address the concern — including accepting any suggested code changes directly into this change with a click. 💅

You can tag people to review your pull request, assign it to someone to deal with (not used much, since it's kind of a repeat of the author and reviewers), add labels...etc.

At the bottom of the Conversation tab you will see the following: [image: Bottom of a pull request with "Squash and Merge" and "Close" buttons.] [image]

This shows on this PR that all checks have passed and it's ready to merge. Clicking the big green "Squash and Merge" button will copy (the "Merge" part) all this code into the main branch with one single commit (the "Squash" part). Usually you don't need all the 100s of commits you've done while developing this code change, so "Squash" is what you want — but you can change it if you want.

You can also close this pull request if you change your mind with the Close button at the bottom, or add a comment with the Comment button if you make a big change to it since opening that you want people following the pull request to be aware of.

Please note you do NOT need to Close and Reopen the pull request (or even open a new pull request) when you need to make changes based on review feedback — simply pushing changes to the branch will cause any open pull request from that branch to automatically be updated and checks to automatically be rerun. It is expected (and a good thing!) to change your code based on feedback, and this is very much part of the workflow of pull requests. 🌈

GitHub Forks 💅

Okay so besides branches, GitHub also has this whole concept of forks, and lowkey it's giving "taking a full copy of the repo" — like, every single branch it has at that exact moment — and yeeting it into your own GitHub account. Then you can make a branch in that fork, and open a pull request to merge your branch on your fork allll the way back into the original repo (which we call the upstream repo, bestie). Sounds like Inception-level mind-bending chaos but honestly? It works so well once your brain clicks into place. 🌀

Note: There's some confusion around the name fork, because traditionally that word meant you wanted to take a project in a completely different direction than the original devs [1] — so you forked the code and never ever merged back. On GitHub though, a fork is for making changes outside the original repo, but usually with the intention of merging them back in once you're done. 🫶 [1] <https://drewdevault.com/2019/05/24/What-is-a-fork.html>

So why fork at all when you could just vibe in the original repo? Because most projects do not want randoms messing with their original repo, so they lock down permissions to only let core contributors create branches there. Everyone else has to fork to make changes, then open pull requests back to the original repo for review before anything gets committed. Periodt. 👑

And the terminology? Obsessed with getting it right. Tempting as it is, the original repo should always be called "original" or "upstream" — never "main" or "master", because those refer to branches inside a repo. Same energy with a "local" copy, or a "clone" — that's the copy on your PC, as we'll see, and it can be a clone of the original repo or of a fork. 🎀

One extra bit of hassle with a fork: you've gotta keep it reasonably up to date with the original, upstream repo. To do that you periodically merge or rebase the fork back onto the original repo, which pulls the changes down into your fork. We'll explain how later, don't stress. ✨

Cloning a Git Repo 🦋

To work on a project on GitHub you'd normally clone a repo, which simply means taking a copy of it onto your local PC. You can make tiny edits on the GitHub.com website but it's super limited and often won't let you run the code locally to test it, for example. You clone a repo by clicking the green Code button on the repo's home page (make sure you do this on your fork and NOT on the main repo!!):

[image: Screenshot of the clone button in GitHub.] [image]

This gives you a few options:

Once you copy the SSH or HTTPS URL, just hop onto the command line on your PC, cd into the directory you want the copy to live in, and type this (assuming SSH):

git clone git@github.com:sqlfluff/sqlfluff.git

You can clone a local copy of the original repo if you plan to (and have) work on branches of it, or you can clone a fork of the original repo. The example command above clones the original repo location, not the fork location — so if you're working from a fork you should swap the git address to the forked version. 💅

After running that command you'll see the repo downloading locally. Then you can branch, edit any files, add new files, or even delete files to your heart's content. Every change you make lives only on your machine, and then you push changes back up to GitHub. We'll cover that later. 🌸

Just like with a fork, you need to keep your local copy up to date with both the original, upstream repo and the GitHub version. That's done using the git pull, git merge and git rebase commands. We'll explain how to do all that below. ✨

Git Merge Conflicts 💀

When you're keeping all the different copies in sync you will inevitably run into the dreaded "merge conflict" — a rite of passage every developer must go through. It happens when you've changed some code, but someone else changed it too, and their changes got merged into main, so when you attempt to merge (either by syncing main back to your branch to pull in the new changes since you branched, or by opening a pull request from your branch), Git just gives up and is like "I don't know what to do here — you deal with it!". 🙈

In actual fact, dealing with merge conflicts is genuinely very simple. When you open the conflicted file you'll see something like this:

If you have questions, please
<<<<<<< HEAD
open an issue
=======
ask your question in Slack
>>>>>>> branch-a

So in this case, someone changed the line to "open an issue" and merged that into main (aka HEAD), and you've also changed it to "ask your question in Slack". Git is warning you that the line changed since you branched, but you changed it too. You just need to decide which line you want, then delete all the other lines (including the ones starting with <<<<, ==== and >>>>). Then git add the "resolved" file to your branch. Slay. 💅

You can even do it directly on GitHub <https://docs.github.com/en/github/collaborating-with-issues-and-pull-requests/resolving-a-merge-conflict-on-github>. ✨

Merge conflicts get such a bad name and people think they're terrifying to deal with, but Git actually makes it fairly easy. It'll usually only complain if the exact same line has changed — two people working on different parts of the same file usually won't see any merge conflicts at all. 🎀

Of course, if you're both working on lots of the same code across lots of files, they can be a real pain to deal with — which is one of the main reasons to resync your branch back to the original main branch frequently, and also to work on small PRs rather than big unwieldy ones! 🔥

When working on SQLFluff you must fork SQLFluff to your own copy and work on that. SQLFluff contributors do not have access to create branches in the original repo. Sorry bestie, that's just how it is. 🫶

To create your own Fork, do this:

  1. Click the Fork button in GitHub and wait for it to complete.
  2. Clone your fork locally — for me (tunetheweb) that means running this command on your machine git clone git@github.tunetheweb/sqlfluff.git
  3. Add a link to upstream — git remote add upstream git@github.com:sqlfluff/sqlfluff.git (change this to the HTTPS URL if you're not using SSH)

It's also strongly recommended that you don't work on the main branch of your forked repo. When you create a new branch you'll usually branch from main, so once your main has extra changes in it, it's no longer possible (or at least easy!) to create a clean branch for other work. 💀

If you're only working on one thing at a time, using main in your fork may seem like the quickest, easiest thing to do, but you'd be surprised how often you'll want to pause that work for a bit and work on something else instead — and that's literally one of the advantages of using Git. For example, if you're waiting on a pull request to be reviewed, you might want to work on another feature in the meantime on a fresh branch, completely independent of your other work. Or maybe someone discovers an urgent, easily-fixed bug that you can quickly patch, before coming back to your current work. See the Switching between branches section below for more info on how to switch branches. 🌈

Working on a separate branch to main lets main stay in sync with upstream main, which makes creating new branches way easier. It also lets you merge upstream main into your branch periodically more easily. It also keeps your history of changes without a long history on every future pull request. And finally, it also lets you completely reset your main back to the same as upstream if you get it completely messed up, without losing any history in your other branches. Iconic multitasking energy. ✨

Resyncing your main branch to upstream 🔄

As discussed above, it's recommended that you use the main branch in your fork only to sync up with the main branch in the original SQLFluff repo. To do that you can do one of several things:

  1. You can merge upstream main into your main branch — however if you have any changes, this will add a commit message for each upstream change into your main. That means the next time you branch and open a pull request you'll see a long history of old commits, which is a bit noisy and annoying when they had nothing to do with your pull request. Of course you shouldn't have any changes on main (because you're not using it as your workspace, as advised above... aren't you?) so this shouldn't be an issue, but you never know... 🙈
  2. You can rebase your main branch to be based on upstream main. This is a special Git command that basically makes it look like you just forked your main off of upstream main, and then applies any changes you've made in your repo on top of that, so it should lead to less risk of old commit messages — but it'll still add commits for any changes you added. And again, you shouldn't have any changes on main so this shouldn't be an issue, but you never know... 💀
  3. You can blast away your main and basically reset it back to upstream main. Sounds extreme but this is actually my preferred option to make sure main is super clean and reset back to the same as upstream main. Note this will (intentionally!) blast away any changes from your main and you will lose them completely — but, again, there shouldn't be any changes here if you're following the advice above. 🔥

So I prefer option 3 — it's the cleanest and guarantees main is exactly the same as upstream, and that there'll be no long history of commit messages in your next branch and pull request. Chef's kiss. 👑

Commands to force reset your main branch to upstream 🧚

You can use the commands below to reset your local fork to upstream and then push those changes to GitHub. 💖

Note: You should ONLY do this if you're happy to lose all changes to your main branch and completely reset it as if you'd just forked it. If you have changes in flight on the main branch then you'll need to either wait until they've been merged, or move them to a new branch before you can do a clean reset of main. ⚠️

Check if upstream already exists:

git remote -v

Add an upstream remote, if not already added previously:

git remote add upstream git@github.com:sqlfluff/sqlfluff.git

Then force reset your main branch:

git fetch upstream
git checkout main
git reset --hard upstream/main
git push origin main --force

okay so once that's done, bestie, you gotta go visit your forked repo over on GitHub and confirm you're getting the message "This branch is even with sqlfluff:main." — it'll show you a lil' pic of a forked repo that's even with upstream, and that is the green flag you're looking for. ✅

🎀 Creating and working on a branch 🎀

so when you're finally ready to make your first changes, here's the whole routine:

  1. Resync your local copy to upstream the way we just did above (unless you literally just forked it, in which case you can skip this step entirely, lucky you).
  2. Make sure you're on the main branch — git checkout main
  3. Branch main off into a new branch with a name that actually means something. Like, say you're working on an issue to add a new Posgres Create Table, you'd run git checkout -b postgres-create-table (see that -b? that's you going "create a new branch please" 💅).
  4. Make your changes in whatever code editor is your roman empire (personally I'm a VSCode <https://code.visualstudio.com/> girlie, highly recommend).
  5. If you're making code changes to the website, then test them — follow the instructions in CONTRIBUTING.md <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md> to set up the environment, and then run most of the tests with tox -e generate-fixture-yml,cov-init,py39,cov-report,linting.
  6. Add any new files you made in this change that you want tracked in git — e.g. git add text/fixtures/dialects/*.sql. You also need to do this to re-add merge conflict files that you've resolved. You typically don't need to add other files that are already in the repo and that you've changed, because Git already tracks those.
  7. Commit your changes — git commit -a. That means "save all changes to Git (for tracked files)". If you only wanna commit some files, then git add just the files you want and then use git commit without the -a option. When you commit you need to add a message. Git will automatically open your default Git editor — usually vi, which can take some getting used to, but basically: type your message, then type Esc : w q ! to enter command mode (Esc), start a command (:), save your file (w for write), quit (q), and force override any errors on save (!). Check the next section if you're not comfortable with vi, there's an alternative. And btw, if you wanna cancel the commit at this stage, just remove any message you added and type Esc q ! to quit without saving any changes — Git will see you didn't add a message and cancel the commit for you. 🫶
  8. Push your committed changes to GitHub — git push. If this is your first push of this branch to GitHub, git will tell you you have to use a slightly different command: git push --set-upstream origin postgres-create-table
  9. If there are any changes already in GitHub (like, other people are also working on this branch), then you'll get an error and you need to do a git pull to pull them down locally before you can push any more changes back up to GitHub.
  10. Repeat steps 4-9 as many times as you want until you're happy to open a pull request to merge this back into the SQLFluff code base.

🎀 Committing changes - to vi or not to vi 🎀

My preferred method of committing changes is this one:

git commit -a

This takes all the changes for existing tracked files and adds them to the commit. New files still need to be added with git add, but all files currently tracked by Git get automatically included without you having to use git add. Then it opens the default Git editor (usually vi), lists the files for this commit, shows you any files not included, and lets you add the message and complete the commit. If you close vi without adding a message, it cancels the commit.

But here's the thing — vi can be a whole barrier for people, because it's a confusing editor with a "command" mode and an "edit" mode and you have to remember command sequences (like ESC + w + q + !). For anyone not familiar with linux and vi, that's a genuine stumbling block. 😭

You can also just provide the message on the command line so you never have to go anywhere near vi, using a sequence of commands like this:

git status
git add file1 file2
git commit -m "Committing file1 and file2"

This does require you to add the files or folders first, so it's a bit more painful than git commit -a. Alternatively you can use the -a and -m switches together:

git status
git commit -a -m "Committing all open files"

The downside? Unlike the vi method, it won't show you the list of files it's about to commit, so it's a bit more dangerous — which is exactly why I prefer the vi method instead. That said, if you do a git status before you commit, you should see the files that will be committed. Plus you can always revert changes if you need to. 💅

So, in summary: vi method is preferred, but it can be more complicated for those not familiar with it, so you can give the commit message on the command line — but take care with it.

🎀 Keeping your branch up to date 🎀

It's also recommended to merge any changes that have happened to SQLFluff code (in its main branch) into your branch periodically, in case it affects your code. And it's particularly important to do this just before opening a PR. 👑

To merge changes into a forked repo from upstream main, do this:

  1. Commit any changes in the branch you are working on.
  2. Update your local copy of upstream — git fetch upstream
  3. Merge the changes from upstream main into your branch — git merge upstream/main
  4. Commit the merge — git commit -a
  5. Push all your changes up to GitHub — git push

Or to merge from a branch on the main repo, do this:

  1. Commit any changes in the branch you are working on.
  2. Update your local copy of main — git fetch origin main
  3. Merge the changes from upstream main into your branch — git merge main
  4. Commit the merge — git commit -a
  5. Push all your changes up to GitHub — git push

🎀 Switching between branches 🎀

Git lets you work on several branches at once, which is iconic of it honestly. That means you can go work on something else while you're stuck on one bit of work (waiting for answers, or pull request feedback, or you just fancy a break!). Use git checkout to switch between branches, and use git status to make sure all your changes are committed when you switch.

So, say you're working on the feature1 branch:

git checkout main
git pull
git checkout -b feature1
# Make some changes
git commit -m "Commit my changes for feature1"
# Make some more changes
git commit -m "Commit some more changes for feature1"
# Push changes to Github.com if you want to (always good to do this in case your computer dies!)
git push
# Note the first time you push a new branch you will need a slightly different push comment:
# `git push --set-upstream origin feature1`
# Helpfully, git will tell you this if you try using just `git push`.

And then say you wanna take a break from feature1 — in that case you should open a new branch. In most cases you wanna branch from main again, and not from the feature1 branch, so make sure you flip back to main again if that's the case. If you're working on a fork, you should also check your main is up to date first — see the Resyncing your main branch to upstream section above:

# Check your branch is clean and everything has been committed
git status
# Create a new branch from main (note I've not included the resyncing of main to upstream here for forks)
git checkout main
git pull
git checkout -b feature2
# Make some changes and push your new branch to GitHub
git commit -m "Commit some changes for feature2"
git push --set-upstream origin feature2
# Make some more changes and push those too
git commit -m "Commit more changes for feature2"
git push

You're now free to switch back to feature1 whenever you want using git checkout (note you don't need the -b flag, because that's only needed to create a new branch that doesn't exist yet — just switching between branches that already exist doesn't need it):

git checkout feature1

And then switch back to feature2 later:

git checkout feature2

The main concern with having a bunch of branches like this is getting yourself confused! Using git status here is your bestie — it double checks you're on the correct branch and that all outstanding changes have been committed.

As with lots of things in Git, there are many ways to handle branches (including the git branch command), but I'd encourage you to stick with a few commands and use git checkout, git commit, git push, git pull and git status as much as possible, because you can do most things with those and honestly it's already getting confusing! 😵‍💫

The other alternative is to just create a new folder and clone the whole repo again and manage it completely separately. That can be easier, safer and less confusing for those less familiar with Git, if you're working on a limited number of branches. But it doesn't scale very well and it's not the way you're going to get the most out of Git — so as soon as you go beyond a second branch, I'd strongly encourage you to get used to checking out between branches.

I do encourage separate folders though, if you have different repos (e.g. you do some of the main work on the HTTPArchive repo, and some work on your own fork), because switching which repo a folder points to, while also possible, is adding yet more confusion to an already complex thing! 🙂

🎀 Opening a Pull Request 🎀

Once you're finished making changes, you should take these steps to open a pull request back to the original repo so your code gets accepted into SQLFluff:

  1. Merge in any changes that happened to SQLFluff code since you branched (see above).
  2. Run all the automated tests with tox -e generate-fixture-yml,cov-init,py39,cov-report,linting.
  3. Make sure all your changes are pushed to GitHub.
  4. Open a pull request in GitHub.
  5. If the pull request closes an issue, then you can add "Closes #123" or "Fixes #123" in the first comment and GitHub will automatically close issue #123 when the pull request is merged. If it doesn't fully close the issue, then you should instead say something like "Makes progress on #123" — that creates a link on the original issue but doesn't close it.

As mentioned above, you can make more changes to your branch and push them up to GitHub and the Pull Request will automatically update. There's no need to close the PR and reopen a new one. ✨

🎀 Actioning feedback from pull requests 🎀

When your pull request gets reviewed, the first bit of feedback you're likely to see is from the automated tests that run every time a pull request is opened. They take a few minutes to run, and then they'll either give you a satisfying green tick ✅ or a scary red cross ❌.

🔀 Merge conflict checks

Okay bestie, the very first thing GitHub itself does is check for any Git Merge Conflicts, and those have got to be resolved before the pull request can be merged, periodt. If you merge main into your code before you submit a pull request, it's pretty unlikely you'll hit any of these (unless someone got in real quick while you were literally opening it! 😭) which is exactly why doing that is recommended. But other pull requests can get actioned while yours is being reviewed, so it can still happen. Smaller pull requests touching few files reduce the chance of this, so again, where at all possible, break up changes into smaller batches. 🎀

🧼 Linter checks

Another check we perform is linting the code in the pull request. This runs automated checks for errors or code styling and formatting issues that don't match the code conventions we use.

Python code is linted with flake8 and you can run this command on any files to see similar linting issues. You can use black to auto-fix most flake8 issues, though some need to be manually addressed. ✨

💬 Code review feedback

After the automated tests have passed, the code gets reviewed manually by a maintainer, or another contributor. They can ask questions, or make suggestions to change the code.

Look upon a code review as an opportunity to learn and improve your code. Feedback can be tough to hear after you've worked hard on code, but stay aware that it is meant with the best possible intention — to provide feedback to you for this and future commits, and to keep our codebase to a high standard. It is not a personal sleight upon you or your code. And if you are getting annoyed with feedback, I suggest you take a break and step away and read it again later, in a fresh light. 🫶 Of course, if you feel that a reviewer is acting inappropriately then please raise it — we have a Code of Conduct (https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md) and want all contributors to feel welcome. Feel free to also reach out to a maintainer if you would like to discuss something privately.

When a reviewer makes a code suggestion you can accept it right in GitHub and it will automatically update your branch. Since this happens in GitHub directly, just make sure you do a git pull next time you are working locally on your code to pull down those changes. It's quite slow to accept a lot of changes this way, so if there are a lot of changes then sometimes it's better to change locally in your favourite editor, and then push one commit with the fixes, and then mark each of the suggestions as resolved. Any suggestions on lines which have changes since they were raised will be marked as outdated, which makes them easy to spot. 👀

Reviewers may also make small, seemingly pedantic changes — usually they will include the word "nit", to indicate this is a "nitpick" in these. Like the linting, they can seem needless, but they help maintain our code quality.

It should also be noted that not all suggestions may be valid! Reviewers can make a mistake just as easily as the pull request author (more easily in some ways, as they often will be reviewing based on reading the code rather than testing it). Feel free to push back on suggestions when you disagree with them. However, it is best to get consensus between reviewer and pull request author where at all possible, so explain why you don't think you should make the change being suggested rather than just ignoring the suggestion or resolving it without a comment.

After addressing feedback, please re-request a review by clicking the little two arrow icon next to the reviewer name, or make a comment (e.g. "All feedback addresses. I think this is good to merge now."). Sometimes it's difficult to know if someone is still working on feedback, and a pull request may be left unintentionally, getting out of date, because reviewers are not aware that it's good for a re-review or merge.

Only SQLFluff maintainers can merge pull requests, but every contributor can review pull requests, and merging becomes a lot easier (and so more likely!) if someone else has already had a look through the changes. Please, please, please help us reviewing_pull_requests to help spread the load! 💖

📝 Draft pull requests

It is also possible to open a draft pull request, if you want early feedback on your code or approach. Please remember to convert to a full pull request when happy. 🌸

Additionally, if making a large number of changes (for example if you need to update every dialect), then best to do this in only one or two files initially, open a pull request (whether draft or full) and get feedback on your approach before you spend time updating all the files! It's much easier to review code changes if they are not drowned out by lots of identical changes that need that code in the same pull request, and much less likely to result in merge conflicts. Then the rest of the files can be added to the pull request, or a separate one opened for those (maybe a new pull request per language if you want different translators to approve any changes). 🦋

👑 Reviewing pull requests

As mentioned above, we strongly encourage contributors to help review pull requests. This is a voluntary, collaborative effort for us all, and depending on one or two people creates a bottleneck and a single point of failure for the project.

Even if a review pulls up nothing, it is important to approve the pull request — this indicates that it has been reviewed and is just as useful (if not more so) as commenting on code or making suggestions.

Do also be conscious of how feedback will be read. We have many first time contributors who may not be as familiar with language (either programming language or English language!) so do try to avoid technical terms, colloquialism...etc. Though we are aware of some very commonly used acronyms and terms (which we've included in our Glossary of terms) like LGTM ("Looks Good To Me"). 😌

Do be mindful as well that pull request authors have voluntarily spent time on this and we wish to encourage that and foster an inclusive environment. Offensive language is forbidden by our Code of Conduct (https://github.com/sqlfluff/sqlfluff/blob/main/CODE_OF_CONDUCT.md).

Do remember that a code review is also about reviewing the author's changes, and not about showing off your own knowledge! Try not to get side-tracked, but instead raise a new issue if you want to consider something else that comes up during a code review but is not directly related. 💅

On that note, do also remember that code can be improved incrementally. Small changes are much better in Git. So, while it's OK to point out a fuller solution, do remember that Perfect is the enemy of good (https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good) and accepting a change that is an improvement and then improving further in future iterations can often be better than holding out for the perfect solution that may never come. Of course that doesn't mean we should accept code that regresses the quality, or seems like the wrong way of doing it completely!

And finally, we strongly encourage positive GitHub reactions — particularly for new contributors. They give quick, encouraging feedback and add a playful, fun tone: [image: GitHub Heart and Rocket reactions.] [image]

We discourage the negative ones though (thumbs down 👎, or confused 😕). Better to add a comment (politely!) explaining your concerns, and then if others agree with you, they can "thumbs up" your comment. This keeps things on a positive tone and also means your cause for concerns are fully understood. 🫶

🖥️ GitHub Desktop

GitHub Desktop is a Windows and MacOS app that provides a visual interface to GitHub. It reduces the need to use and understand Git via the command line.

This section will provide some tips on performing some common tasks via the GitHub Desktop.

⬇️ Installing GitHub Desktop

First make sure you have Git installed. See our section on Installing Git for more details.

You can then download the install file from https://desktop.github.com/, with further instructions from their Installing and configuring GitHub Desktop (https://docs.github.com/en/free-pro-team@latest/desktop/installing-and-configuring-github-desktop) document. Your main tasks will be to Authenticate with GitHub (https://docs.github.com/en/free-pro-team@latest/desktop/installing-and-configuring-github-desktop/authenticating-to-github) and Configuring Git for GitHub Desktop (https://docs.github.com/en/free-pro-team@latest/desktop/installing-and-configuring-github-desktop/configuring-git-for-github-desktop) so that the systems know who you are. ✨

📥 Cloning the SQLFluff repo

If you have not done already, you will want to clone a copy of the https://github.com/sqlfluff/sqlfluff repo into your computer. The simplest way is to follow Cloning a repository from GitHub to GitHub Desktop (https://docs.github.com/en/free-pro-team@latest/desktop/contributing-and-collaborating-using-github-desktop/cloning-a-repository-from-github-to-github-desktop) where you go to the repository on the website and select "Open with GitHub Desktop". This will open a window where you can click "Clone" and the job will be done. 💖

Once you have cloned repositories you will be able to select them via the "Current repository" toolbar button, just under the menu on the left. By default the sidebar will show you what edits have been made to the repository, and the main section shows actions you may want to perform.

🔄 Updating your repository (Pull origin)

Over time the original repository will get updated and your copy will become out of date. GitHub Desktop will highlight if your repository is out of date, with an option to pull any changes from the origin so that you have the latest versions.

🌿 Making your own edits (creating a branch)

You want to create your own branch before you start, as you very likely do not have permission to edit the SQLFluff main branch. A branch is a way for you to group your own edits so you can later submit (push) them for review. Then, when they are approved, they will get merged back into the main branch.

Before creating a branch, make sure you're currently on the main branch and it is up to date (see above).

If you click on the "Current branch" tab in the toolbar you will see all the public branches in play. To create your own branch, enter a new name in the textbox at the top and click the "Create new branch" button. 🎀

🚀 Publishing your branch

At the moment your branch is only known to you. If you want others to see it, then you need to publish it. GitHub Desktop will prompt you to do that.

Once published you and others can select your branch on the GitHub website. 🌈

💖 Editing your branch

You can edit the repo with whatever editor is your ride-or-die. And the second you start typing, GitHub Desktop will show you exactly what you changed. It's giving personal hype squad. ✨

Heads up bestie: you can switch branches whenever, but pls pls pls commit and push your edits first (see the next bit). Switching mid-edit is how confusion is born. And if you're juggling multiple branches, always double-check you're on the right one before you touch anything. Stay vigilant. 👀

🌸 Committing and pushing your edits to the web

Every so often you want to save and document your changes. Future you (and future everybody else) will be so grateful. Also, lowkey mandatory: you literally cannot share (push) your changes with anyone until you've committed them. The fast way to commit whatever you've currently got going on is that little form in the bottom left. 🎀

Once you've got commits, GitHub will be like "hey, wanna push these?" and yes, you do. I personally push right after committing, every time. No notes.

👑 Getting your changes accepted

Okay so at this point you have a branch, the edits are committed, and everything is pushed up to GitHub. Now, once you're actually happy with your work, you want it reviewed, approved, and merged back into the main repository. The whole glow-up. 💅

For this stage I hop back to the website, because that's where the communicating with reviewers happens. To kick it off you need to make a pull request. Go to the SQLFluff repository on GitHub at https://github.com/sqlfluff/sqlfluff, make sure your branch is selected, then click the Pull request link and follow the instructions. This notifies the reviewers, who will help you get your changes live. Teamwork, baby. 🌈

🦋 Keeping the forked repository up to date

The main branch of your fork should be kept in sync with the original repository (i.e. rebased). Especially before you create any branches to make edits. The full how-to lives in the Resyncing your main branch to upstream section. 🧚

⭐ Making your own edits

This is done in the exact same way as before (i.e. in Making your own edits (creating a branch)). Create a branch from your master (and make sure master is up to date using the process above), publish the branch, edit the files in the branch, commit your edits, push back to GitHub. That's the whole loop. 🔁

With a forked repository, getting your edits accepted works about the same as before too (i.e. in Getting your changes accepted). Go to the web page for your copy of the repository and create a pull request. 💖

🎀 Glossary of terms

For anyone who's not besties with Git/GitHub yet, here's the vocab list:

  • branch — a copy of the code within a repo, where you may be working on code that is not ready to commit back to the main branch. Note that Git actually only stores differences so it's not really a copy, but that's just an efficiency in Git and you can consider it a copy to all intents and purposes.
  • fetch — a git command which downloads all changes from a remote repo (i.e. a server) to a local one.
  • fork — a complete copy of a repo and all it's branches.
  • LGTM — Short hand for Looks Good To Me — typically used approving a pull request.
  • local — a complete copy of a repo on your PC.
  • main — the primary branch of SQLFluff repo. Some other repos use master for this.
  • master — an alternative name for main branch used by some repos.
  • merge — to copy changes from one branch into another branch.
  • merge request — what a pull request is also known by, particularly in GitLab, an alternative to GitHub.
  • origin — the server version of the repo (the opposite of local).
  • pull — to fetch changes from a remote repo, and then merge them into this branch in one step.
  • pull request — a way to merge changes back to the main branch. A pull request is a special issue type that allows the potential merge to be reviewed and commented on before it is merged.
  • rebase — to bring a branch up to date, as if it had been created from now, while maintaining the existing changes on top.
  • repo/repository — a git project which is basically a collection of files — which may exist in several branches.
  • upstream — the original repo that a fork was created from.

🌈 Contributing dialect changes

One of the best ways that SQLFluff users can improve SQLFluff for themselves and others is in contributing dialect changes. Periodt. 💅

Users will likely know their syntax much better than the regular maintainers and will have access to an instance of that SQL dialect to confirm changes are valid SQL in that dialect. You're literally the expert here.

If you can fix your own issues then that's often the quickest way of unblocking any issues preventing you from using SQLFluff! The maintainers are all volunteers doing this in our spare time and (like you all I'm sure!), we only have so much time to work on this. 🫶

🧚 How SQLFluff reads (or parses) SQL

SQLFluff has a lexer and parser which is built in a very modular fashion that is easy to read, understand, and expand on without any core programming skills or deep knowledge of Python or how SQLFluff operates. For more information see the Architecture Documentation, but we'll cover that briefly here to give you enough to start contributing. ✨

We also have a robust Continuous Integration pipeline in GitHub where you can gain confidence your changes are correct and will not break other SQLFluff users, even before a regular maintainer reviews the code. So even your mistakes get caught early. Iconic, honestly.

SQLFluff defines the syntax it will used in dialect files (more on this later). If you look at the dialect_ansi.py file (at https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py) you will see it has syntax like this:

class SelectClauseSegment(BaseSegment):
    """A group of elements in a select target statement."""

    type = "select_clause"
    match_grammar = StartsWith(
        Sequence("SELECT", Ref("WildcardExpressionSegment", optional=True)),
        terminator=OneOf(
            "FROM",
            "WHERE",
            "ORDER",
            "LIMIT",
            "OVERLAPS",
            Ref("SetOperatorSegment"),
        ),
        enforce_whitespace_preceding_terminator=True,
    )

    parse_grammar = Ref("SelectClauseSegmentGrammar")

This says the SelectClauseSegment starts with SELECT or SELECT * and ends when it encounters a FROM, WHERE, ORDER...etc. line. 🎯

The match_grammar is what is used primarily to try to match and parse the statement. It can be relatively simple (as in this case), to quickly match just the start and terminating clauses. If that is the case, then a parse_grammar is needed to actually delve into the statement itself with all the clauses and parts it is made up of. The parse_grammar can be fully defined in the class or, like above example, reference another class with the definition.

The match_grammar is used to quickly identify the start and end of this block, as parsing can be quite intensive and complicated as the parser tries various combinations of classes and segments to match the SQL (particularly optional ones like the WildcardExpressionSegment above, or when there is a choice of statements that could be used). It's doing a lot, okay, be nice to it.

For some statements a quick match is not needed, and so we can delve straight into the full grammar definition. In that case the match_grammar will be sufficient and we don't need the optional parse_grammar.

Here's another statement, which only uses the match_grammar and doesn't have (or need!) an optional parse_grammar:

class JoinOnConditionSegment(BaseSegment):
    """The `ON` condition within a `JOIN` clause."""

    type = "join_on_condition"
    match_grammar = Sequence(
        "ON",
        Indent,
        OptionallyBracketed(Ref("ExpressionSegment")),
        Dedent,
    )

You may have noticed that a segment can refer to another segment, and that is a good way of splitting up a complex SQL expression into its component parts to manage and handle them separately. Divide and conquer, bestie. 💜

💅 Segment grammar options

There are a number of options when creating SQL grammar, including:

  • "KEYWORD" — used for having a raw SQL keyword. Example: "SELECT"
  • Sequence() — used for having a known sequence of Keywords or Segments. Example: Sequence("SELECT", Ref("SelectClauseElementSegment"), "FROM"...)
  • AnyNumberOf() — used for choosing from a set of options which may be repeated. Example: "SELECT", AnyNumberOf(Ref("WildcardExpressionSegment"), Ref("ColumnReferenceSegment")...)...
  • OneOf() — a more restrictive take on a set of AnyNumberOf limited to just one option. Example: OneOf("INNER","OUTER","FULL"), "JOIN"

okay so picking the grammar table right back up, two more icons to add to the roster ✨

  • Delimited() — bestie this one is for lists (like comma-delimited ones, which is literally the default 🎀) — e.g. "SELECT", Delimited("SelectClauseElementSegment"), "FROM"...
  • Bracketed() — for bracketed options, like function parameters 💅 — e.g. Ref("FunctionNameSegment"), Bracketed(Ref("FunctionContentsGrammar")

Some of the keywords have extra params you can hand them, and the one you'll reach for most is optional=True. That lets you go deeper and pin down the exact make-up of a SQL statement. Behold, the DeleteStatementSegment definition:

parse_grammar = Sequence(
    "DELETE",
    Ref("FromClauseSegment"),
    Ref("WhereClauseSegment", optional=True),
)

See how the WHERE clause is optional? (many's a head has been shaken because of deletes without WHERE clauses, I'm sure — but that's what SQL syntax allows! 😭)

Using these Grammar options, it's possible to build up complex structures to define SQL syntax. no cap 💖

✨ Segments and Grammars ✨

A Segment is a piece of the syntax which defines a type (which can be useful to reference later in rules or parse trees 👑). This can happen through one of the functions that creates a Segment (e.g. NamedParser, SegmentGenerator...etc.) or through a class.

A Grammar is a section of syntax that can be used in a Segment. Typically these get created to avoid repeating the same code in multiple places. Think of a Grammar as an alias for a piece of syntax, so you don't have to type out the same code again and again and again 🫠

The other good thing about Grammars is it allows other dialects to override a specific part of a Segment without having to redefine the whole entire thing just to tweak one small part. For example ansi defines this:

NotOperatorGrammar=StringParser("NOT", KeywordSegment, type="keyword")

whereas mysql overrides this to:

NotOperatorGrammar=OneOf(
    StringParser("NOT", KeywordSegment, type="keyword"),
    StringParser("!", CodeSegment, name="not_operator", type="not_operator"),
),

This allows MySQL to use ! in all the places that NOT was used (providing they use NotOperatorGrammar rather than hardcode the NOT keyword, of course 🙃). This makes it much easier to customise syntax to a particular dialect without having to copy and paste (and maintain) nearly identical code multiple times just to add the extra ! syntax that MySQL supports to mean NOT. slay 💅

🌈 Dialects 🌈

A lot of SQL is the same no matter which particular type of SQL you're using. The basic SELECT.. FROM... WHERE statement is common to them all. BUT — lots of different SQL dialects (Postgres, Snowflake, Oracle...etc.) have sprung up as different companies implemented SQL, or expanded it, for their own needs. it's giving chaos honestly 💀

For this reason, SQLFluff allows creating dialects, which can have different grammars from each other.

SQLFluff has all the dialects in the src/sqlfluff/dialects folder — https://github.com/sqlfluff/sqlfluff/tree/main/src/sqlfluff/dialects

The main dialect file (that every other dialect ultimately inherits from) is the dialect_ansi.py file — https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py

In SQLFluff, a dialect is basically a file which inherits everything from the original ANSI dialect, and then adds or overrides parsing segments. If a dialect has the exact same SELECT, FROM and WHERE clauses as ANSI but a different :ORDER BY syntax, then only the :ORDER BY clause needs overridden, so the dialect file will be very small 🐣. For some of the other dialects where there's lots of differences (Microsoft T-SQL! 😳) you may be overriding a lot more.

🧚 Lexing 🧚

I kind of skipped this part, but before a piece of SQL can be parsed, it is lexed — that is, split up into symbols, and logical groupings.

An inline comment, for example, is defined as this:

RegexLexer(
    "inline_comment",
    r"(--|#)[^\n]*",
    CommentSegment,
    segment_kwargs={"trim_start": ("--", "#")},
)

That is, anything after -- or # to the newline. This allows us to deal with that whole comment as one lexed block, so we don't need to define how to parse it (we even give that a parsing segment name right here — CommentSegment 😌).

For simple grammar addition, you won't need to touch the lexing definitions, as they usually cover most common ones already. But for slightly more complicated ones, you may have to add to this. So if you see lexing errors, then you may have to add something here 👀

Lexing happens in order. So it starts reading the SQL from the start, until it has the longest lexing match, then it chomps that up, files it away as a symbol to deal with later in the parsing, and starts again with the remaining text. So if you have SELECT * FROM table WHERE col1 = 12345 it will not break that up into S, E, L...etc., but instead into SELECT, *, FROM, table...etc. ate 🔥

An example of where we had to override lexing is in BigQuery — we have parameterised variables which are of the form @variable_name. The ANSI lexer doesn't recognise the @ sign, so you could add a grammar or segment for that. But a better solution, since you don't need to know two parts (@ and variable_name), is to just tell the lexer to go ahead and parse the whole thing into one big symbol, that we will then use later in the parser:

bigquery_dialect.insert_lexer_matchers(
    [
        RegexLexer("atsign_literal", r"@[a-zA-Z_][\w]*", CodeSegment),
    ],
    before="equals",
)

Note the before="equals" — which means we tell the lexer the order of preference to try to match this symbol. For example, if we'd defined an at_sign lexing rule for other, standalone @ usage, then we'd want this to be considered first, and only fall back to that if we couldn't match this. periodt 💖

🌸 Keywords 🌸

Most dialects have a keywords file, listing all the keywords. Some dialects just inherit the ANSI keywords and then add or remove keywords from that. Not quite as accurate as managing the actual keywords, but a lot quicker and easier to manage usually 🫶

Keywords are separated into RESERVED and UNRESERVED lists. RESERVED keywords have extra restrictions, meaning they cannot be used as identifiers. If using a keyword in grammar (e.g. "SELECT"), then it needs to be in one of the Keywords lists, so you may have to add it — or you might see errors like this (showing "NAN" has not been added as a Keyword in this dialect):

RuntimeError: Grammar refers to 'NanKeywordSegment' which was not found in the redshift dialect

Also, if editing the main ANSI dialect, and adding to the ANSI keyword list, then take care to consider if it needs added to the other dialects if they will inherit this syntax — usually yes, unless explicitly overridden in those dialects 👑

🔍 Where to find the grammar for your database 🔍

Now that you know about some of the tools SQLFluff provides for lexing and parsing a SQL statement — what changes will you make to it? While devising ad-hoc changes to the grammar to fix particular issues can be better than nothing, the best and most robust contributions will be created by consulting the source of truth for the grammar of your dialect when mapping it to SQLFluff segments and grammars. This will help you exhaustively find all possible statements that would be accepted by the dialect. main character energy fr fr 👑

Many computer languages are written using venerable tools like Flex (https://en.wikipedia.org/wiki/Flex_(lexical_analyser_generator)) and Bison (https://en.wikipedia.org/wiki/GNU_Bison), or similar parser generators, and SQL database engines are no exception. You can refer to the parser specification in the source code of your database engine for the ultimate source of truth of how a SQL statement will be parsed: you might be surprised at what your SQL engine will parse due to gaps in the documentation! 😭

You should also refer to the reference documentation for your SQL dialect to get a concise high-level overview of what the statement grammar looks like, as well as read of any further restrictions and intended use of the grammar that you find. If your SQL engine is closed-source, then you'll likely have only the reference documentation to work with. However, this will always be a less-accurate resource than the bison grammar that's actually used for code generation inside the database engine itself. ngl, read both 💀

It is also extremely helpful to try parsing the queries that you put into the test fixtures, to make sure that they are actually parsable by the database engine. They don't have to be valid queries per se (can refer to non-existing table names, etc.), but you should confirm that they are parsable. We do not want to require that SQLFluff be able to parse a statement that the actual database engine would reject: overeager matching logic can create parsing issues elsewhere. 🚨

Here is a list of grammars and parsing techniques for some of the dialects implemented by SQLFluff:

⭐ ANSI SQL ⭐

Unfortunately, the ANSI SQL standard is not free 😭. If you want a licensed copy of the latest standard, it must be purchased: Part 2 (https://webstore.ansi.org/standards/iso/isoiec90752016-1646101) is the most useful section for SQLFluff since it contains the grammar. There are, however, other resources you can find on the Internet related to this standard:

🐘 PostgreSQL 🐘

Okay so real talk: if you just Google pg <statement> you will very often get served the docs for some crusty older PG version 😭. Please, for the love of everything, make sure you're looking at the latest version of the documentation — and also that you're actually referring to the bison grammar. Both. No shortcuts. 💅

🐬 MySQL 🐬

🧚 Example of contributing a syntax fix 🧚

So that's a bit of theory, but let's actually go through some real examples of how to add to the SQLFluff code to address whatever issues you're seeing. 🌈

In this I'm not going to explain how to set up your Python development environment (see the Contributing to SQLFluff at #development and the CONTRIBUTING.md at https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md file for that), nor how to manage Git (see our How to use Git at #using-git guide if you're new to that — and we use the standard “Fork, and then open a PR” workflow common to GitHub projects).

So assuming you know (or are willing to follow the above guides to find out!) how to set up a Python environment, and commit via Git, how do you actually contribute a simple fix to a dialect for syntax you want SQLFluff to support? 👀

⭐ Example 1 ⭐

If we look at issue #1520 (https://github.com/sqlfluff/sqlfluff/issues/1520) it was raised to say we couldn't parse this:

CREATE OR REPLACE FUNCTION public.postgres_setof_test()
RETURNS SETOF text

and instead returned this message:

Found unparsable section: 'CREATE OR REPLACE FUNCTION crw_public.po...'

This was in the postgres dialect, so I had a look at dialect_postgres.py (https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_postgres.py) and found the code in CreateFunctionStatementSegment which had the following:

parse_grammar = Sequence(
    "CREATE",
    Sequence("OR", "REPLACE", optional=True),
    Ref("TemporaryGrammar", optional=True),
    "FUNCTION",
    Sequence("IF", "NOT", "EXISTS", optional=True),
    Ref("FunctionNameSegment"),
    Ref("FunctionParameterListGrammar"),
    Sequence(  # Optional function return type
        "RETURNS",
        OneOf(
            Sequence(
                "TABLE",
                Bracketed(
                    Delimited(
                        OneOf(
                            Ref("DatatypeSegment"),
                            Sequence(
                                Ref("ParameterNameSegment"), Ref("DatatypeSegment")
                            ),
                        ),
                        delimiter=Ref("CommaSegment"),
                    )
                ),
                optional=True,
            ),
            Ref("DatatypeSegment"),
        ),
        optional=True,
    ),
    Ref("FunctionDefinitionGrammar"),
)

So it allowed returning a table, or a datatype. That's it. That's all it knew. 😭

Fixing the issue was as simple as adding the SETOF structure as another return option:

parse_grammar = Sequence(
    "CREATE",
    Sequence("OR", "REPLACE", optional=True),
    Ref("TemporaryGrammar", optional=True),
    "FUNCTION",
    Sequence("IF", "NOT", "EXISTS", optional=True),
    Ref("FunctionNameSegment"),
    Ref("FunctionParameterListGrammar"),
    Sequence(  # Optional function return type
        "RETURNS",
        OneOf(
            Sequence(
                "TABLE",
                Bracketed(
                    Delimited(
                        OneOf(
                            Ref("DatatypeSegment"),
                            Sequence(
                                Ref("ParameterNameSegment"), Ref("DatatypeSegment")
                            ),
                        ),
                        delimiter=Ref("CommaSegment"),
                    )
                ),
                optional=True,
            ),
            Sequence(
                "SETOF",
                Ref("DatatypeSegment"),
            ),
            Ref("DatatypeSegment"),
        ),
        optional=True,
    ),
    Ref("FunctionDefinitionGrammar"),
)

With that code the above item could parse. Slay. 💅

I added a test case (covered below) and submitted pull request #1522 (https://github.com/sqlfluff/sqlfluff/pull/1522) to fix this.

🦋 Example 2 🦋

If we look at issue #1537 (https://github.com/sqlfluff/sqlfluff/issues/1537) it was raised to say we couldn't parse this:

select 1 from group

And it threw this error:

==== parsing violations ====
L:   1 | P:  10 |  PRS | Line 1, Position 10: Found unparsable section: 'from'
L:   1 | P:  14 |  PRS | Line 1, Position 14: Found unparsable section: ' group'

The reporter had also helpfully included the parse tree (produced by sqlfluff parse):

[L:  1, P:  1]      |file:
[L:  1, P:  1]      |    statement:
[L:  1, P:  1]      |        select_statement:
[L:  1, P:  1]      |            select_clause:
[L:  1, P:  1]      |                keyword:                                      'select'
[L:  1, P:  7]      |                [META] indent:
[L:  1, P:  7]      |                whitespace:                                   ' '
[L:  1, P:  8]      |                select_clause_element:
[L:  1, P:  8]      |                    literal:                                  '1'
[L:  1, P:  9]      |            whitespace:                                       ' '
[L:  1, P: 10]      |            [META] dedent:
[L:  1, P: 10]      |            from_clause:
[L:  1, P: 10]      |                unparsable:                                   !! Expected: 'FromClauseSegment'
[L:  1, P: 10]      |                    keyword:                                  'from'
[L:  1, P: 14]      |            unparsable:                                       !! Expected: 'Nothing...'
[L:  1, P: 14]      |                whitespace:                                   ' '
[L:  1, P: 15]      |                raw:                                          'group'
[L:  1, P: 20]      |    newline:                                                  '\n'

So the problem was it couldn't parse the FromClauseSegment. Looking at that definition showed this:

FromClauseTerminatorGrammar=OneOf(
    "WHERE",
    "LIMIT",
    "GROUP",
    "ORDER",
    "HAVING",
    "QUALIFY",
    "WINDOW",
    Ref("SetOperatorSegment"),
    Ref("WithNoSchemaBindingClauseSegment"),
),

So the parser was terminating as soon as it saw the GROUP and saying “hey we must have reached the end of the FROM clause”. 🙈

This was a little restrictive, so changing that to this solved the problem:

FromClauseTerminatorGrammar=OneOf(
    "WHERE",
    "LIMIT",
    Sequence("GROUP", "BY"),
    Sequence("ORDER", "BY"),
    "HAVING",
    "QUALIFY",
    "WINDOW",
    Ref("SetOperatorSegment"),
    Ref("WithNoSchemaBindingClauseSegment"),
),

You can see we simply replaced the "GROUP" by a Sequence("GROUP", "BY") so it would only match if both words were given. Rechecking the example with this changed code showed it now parsed. We did the same for "ORDER", and also changed a few other places in the code with similar clauses, and added a test case (covered below), and submitted pull request #1546 (https://github.com/sqlfluff/sqlfluff/pull/1546) to fix this. 🔥

👑 Example 3 👑

As an example of using the reference grammar to fix an existing SQLFluff grammar, pull request #4744 (https://github.com/sqlfluff/sqlfluff/pull/4744) contributed the CREATE CAST / DROP CAST statements to SQLFluff from scratch for both ANSI and PostgreSQL dialects. 💖

The first step when contributing a new statement is to check whether the statement is part of the ANSI standard. If it is, then you very likely should first start by adding a generally vendor-neutral version to the SQLFluff ANSI dialect so that other dialects can inherit from it. Every database engine deviates from the ANSI standard in practice, but by adding a reasonably standard segment to the ANSI dialect, you'll probably do a reasonable thing for most other database dialects. 🫶

In this case, CREATE and DROP CAST were indeed defined in the ANSI standard (https://jakewheat.github.io/sql-overview/sql-2016-foundation-grammar.html#_11_63_user_defined_cast_definition), as quickly revealed by a quick search of the document:

<user-defined cast definition> ::=
    CREATE CAST <left paren>  <source data type>  AS <target data type>  <right paren>
        WITH <cast function>
        [ AS ASSIGNMENT ]

So the first step was to read this ANSI BNF grammar and use it to build a corresponding vendor-neutral CreateCastSegment in dialect_ansi.py (https://github.com/sqlfluff/sqlfluff/blob/main/src/sqlfluff/dialects/dialect_ansi.py). 🎀

class CreateCastStatementSegment(BaseSegment):
    """A `CREATE CAST` statement.
    https://jakewheat.github.io/sql-overview/sql-2016-foundation-grammar.html#_11_63_user_defined_cast_definition
    """

    type = "create_cast_statement"

    match_grammar: Matchable = Sequence(
        "CREATE",
        "CAST",
        Bracketed(
            Ref("DatatypeSegment"),
            "AS",
            Ref("DatatypeSegment"),
        ),
        "WITH",
        Ref.keyword("SPECIFIC", optional=True),
        OneOf(
            "ROUTINE",
            "FUNCTION",
            "PROCEDURE",
            Sequence(
                OneOf("INSTANCE", "STATIC", "CONSTRUCTOR", optional=True),
                "METHOD",
            ),
        ),
        Ref("FunctionNameSegment"),
        Ref("FunctionParameterListGrammar", optional=True),
        Sequence("FOR", Ref("ObjectReferenceSegment"), optional=True),
        Sequence("AS", "ASSIGNMENT", optional=True),
    )

# Not shown: register the CreateCastStatementSegment in StatementSegment

okay so while you're cruising through the grammar, clock the other bits of SQL that might be doing the same lil thing 👀. like here? we noticed there were already segments we can reuse for data types, function names, and function parameter lists. that's not just cute — it simplified our new grammar AND made it easy to centrally change those exact areas of the grammar in other dialects. two birds, one slay 🦋

also ask yourself: are there whole new segments and grammars you should define separately on top of the root statement segment you're writing? introducing new segments and reusing existing ones adds structure to the SQLFluff parse tree, which lowkey makes it way easier for lint rules to analyze the tree. and here's the tea for spotting when you need a shared segment or grammar: when the reference grammar has a symbol that gets reused from multiple other symbols/statements. that's the vibe check. 💅

after writing the ANSI segment (and the corresponding tests, bestie, don't skip those), it was time to move on to the PostgreSQL grammar. and in this case, a quick glance at the documentation <https://www.postgresql.org/docs/15/sql-createcast.html> shows us there are some notable differences from ANSI SQL:

  • you can only specify FUNCTION. other keywords like ROUTINE and PROCEDURE are rejected. 🚫
  • the SPECIFIC keyword is not supported.
  • and most importantly: PG provides some non-standard extensions which we'd like to include, like WITHOUT FUNCTION and AS IMPLICIT. ✨

but wait — we should also consult the bison grammar for CREATE CAST <https://github.com/postgres/postgres/blob/e0693faf797f997f45bee8e572e8b4288cc7eaeb/src/backend/parser/gram.y#L8938>. bison grammars tend to be very lengthy and daunting, but the right techniques can help you quickly and easily find what you're looking for:

  • search for a symbol by adding a : to the end of it. 🔍
  • start with the highest level of the thing you are looking for. for example, start with the top-level statement symbol. with PostgreSQL, all statements end with Stmt. putting it all together, if we search for CreateCastStmt:, that takes us right to the definition for it. 🎯
  • drill down into deeper parts of the parser to learn more. for example, we see function_with_argtypes in the sequence; if we want to know what that means, search for function_with_argtypes: to find it.

examining the Bison grammar can take a few extra minutes, but it can be rewarding. you'll be surprised what you might learn. I've found entire alternate spellings of keywords in there that were not in the documentation, and which testing showed were indeed valid SQL! 😭 the grammar in PG documentation is human-maintained <https://github.com/postgres/postgres/blob/master/doc/src/sgml/ref/create_cast.sgml> and not auto-generated, so there can be and are gaps between what is parsable and what is documented.

a good approach if you're still learning might be to draft a segment from the high-level documentation, and then systematically go through it with the bison grammar and verify it's correct (and that you're not forgetting anything). 🫶

one aspect of bison grammars to be aware of is that they tend to be very recursive, because it doesn't have the high-level constructs such as AnyOf, Delimited, Bracketed, and so on that SQLFluff provides. on the other hand, SQLFluff doesn't scale well with recursion. sometimes it's unavoidable and reasonable in many cases (e.g. parenthesized expression) to refer to another segment recursively. but many times the recursion is extremely trivial, and should always be rewritten using an existing high-level SQLFluff concept. for example, this bison grammar defines a bracketed comma-delimited list which would be better represented using Bracketed and Delimited in SQLFluff:

func_args:  '(' func_args_list ')'              { $$ = $2; }
            | '(' ')'                           { $$ = NIL; }
        ;

func_args_list:
            func_arg                            { $$ = list_make1($1); }
            | func_args_list ',' func_arg       { $$ = lappend($1, $3); }
        ;

💖 Example 4 💖

as an example of using the reference grammar to fix an existing SQLFluff grammar, issue #4336 <https://github.com/sqlfluff/sqlfluff/issue/4336> reported that array slices were not being parsed correctly in PostgreSQL. a simple SELECT statement was given that I further simplified to the following test case:

SELECT a[2:2+3];

obviously, we know that a simple query like SELECT a; would parse, so it's surely related to the array access. I started by looking up the bison grammar for PostgreSQL's SELECT statement and drilling down into it to find an array accessor symbol; searching for SelectStmt: proved to be a lucky guess to start with <https://github.com/postgres/postgres/blob/e0693faf797f997f45bee8e572e8b4288cc7eaeb/src/backend/parser/gram.y#L12497-L12504>:

SelectStmt: select_no_parens            %prec UMINUS
            | select_with_parens        %prec UMINUS
        ;

drilling down into the grammar via SelectStmt --> select_no_parens --> simple_select --> target_list --> target_el shows that we are dealing with an a_expr, which is the main symbol widely used to represent an expression throughout the grammar. SQLFluff implements that as ExpressionSegment (and more specifically Expression_A_Grammar). looking further: target_el --> a_expr --> c_expr --> columnref. which brings us to a key rule:

columnref:  <snip>
            | ColId indirection
                {
                    $$ = makeColumnRef($1, $2, @1, yyscanner);
                }

digging into indirection, we finally find where the array accessor is happening:

indirection:
            indirection_el                      { $$ = list_make1($1); }
            | indirection indirection_el        { $$ = lappend($1, $2); }
        ;
indirection_el: <snip>
            | '[' a_expr ']'
                {
                    A_Indices *ai = makeNode(A_Indices);

                    ai->is_slice = false;
                    ai->lidx = NULL;
                    ai->uidx = $2;
                    $$ = (Node *) ai;
                }
            | '[' opt_slice_bound ':' opt_slice_bound ']'
                {
                    A_Indices *ai = makeNode(A_Indices);

                    ai->is_slice = true;
                    ai->lidx = $2;
                    ai->uidx = $4;
                    $$ = (Node *) ai;
                }
        ;
opt_slice_bound:
            a_expr                              { $$ = $1; }
            | /*EMPTY*/                         { $$ = NULL; }
        ;

from this we observe:

  • there is a sequence of indirection elements.
  • there can be a simple array index provided, which is an expression.
  • and most importantly, and most immediate to our problem, is the observation that each slice bound is optional, and if it is present, then it is an expression. ✨

now that we looked up the relevant PG grammar, we can dig into the corresponding SQLFluff grammar in a similar top-down way: postgres.SelectStatementSegment --> we see it's mostly a copy of the ANSI select statement, so --> ansi.SelectStatementSegment --> remember Ref always picks the dialect-specific grammar first --> postgres.SelectClauseSegment --> ansi.SelectClauseSegment.parse_grammar --> postgres.SelectClauseSegmentGrammar --> ansi.SelectClauseElementSegment --> ansi.BaseExpressionElementGrammar --> ansi.ExpressionSegment --> ansi.Expression_A_Grammar --> ansi.Expression_C_Grammar --> ansi.Expression_D_Grammar --> notice this at the end of the sequence --> postgres.Accessor_Grammar --> postgres.ArrayAccessorSegment. 🕵️ as you navigate, always remember to check for dialect-specific grammar before falling back to the inherited grammar (e.g. ANSI). finally, we have found the part of the grammar that corresponds to the indirection_el in the bison grammar! 🎉

class ArrayAccessorSegment(ansi.ArrayAccessorSegment):
    """Overwrites Array Accessor in ANSI to allow n many consecutive brackets.

    Postgres can also have array access like python [:2] or [2:] so
    numbers on either side of the slice segment are optional.
    """
              match_grammar = Sequence(
                  AnyNumberOf(
                      Bracketed(
                          Sequence(
                              OneOf(
                                  OneOf(
                                      Ref("QualifiedNumericLiteralSegment"),
                                      Ref("NumericLiteralSegment"),
                                  ),
                                  Sequence(
                                      OneOf(
                                          Ref("QualifiedNumericLiteralSegment"),
                                          Ref("NumericLiteralSegment"),
                                          optional=True,
                                      ),
                                      Ref("SliceSegment"),
                                      OneOf(
                                          Ref("QualifiedNumericLiteralSegment"),
                                          Ref("NumericLiteralSegment"),
                                      ),
                                  ),
                                  Sequence(
                                      OneOf(
                                          Ref("QualifiedNumericLiteralSegment"),
                                          Ref("NumericLiteralSegment"),
                                      ),
                                      Ref("SliceSegment"),
                                      OneOf(
                                          Ref("QualifiedNumericLiteralSegment"),
                                          Ref("NumericLiteralSegment"),
                                          optional=True,
                                      ),
                                  ),
                              ),
                          ),
                          bracket_type="square",
                      )
                  )
              )

okay so staring at that, bestie, there are like two things just screaming at us. The most glaring are that:

  • Only numeric literals are accepted! No expressions. That's literally, exactly, 100% the source of the issue the person reported. periodt.
  • But while we are here 👀 notice another problem we can fix too: when a SliceSegment (a ) is present, you're forced to include a numeric literal either before or after the SliceSegment. You can't have [:], even though that's valid SQL that PG can parse. the disrespect honestly 😭

At this point it's a simple matter of simplifying & rewriting the grammar to fix these shortcomings and better align it with the bison grammar, which was done in pull request #4748 (https://github.com/sqlfluff/sqlfluff/pull/4748). ate and left no crumbs 💅

🧪 Testing your changes

So you've made your fix, you've tested it fixed the original problem, so you just submit that change and all is good now?

Well, no. 🙈 You want to do two further things:

  • Test your change hasn't broken anything else. To do that you run the test suite.
  • Add a test case, so others can check this in future.

To test your changes you'll need to have your environment set up (again see the CONTRIBUTING.md file at https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md for how to do that).

✨ Adding test cases for your changes

Adding a test case is simple. Just add a SQL file to test/fixtures/dialects/ (https://github.com/sqlfluff/sqlfluff/tree/main/test/fixtures/dialects) in the appropriate dialect directory. You can either expand an existing SQL file test case (e.g. if adding something similar to what's in there) or create a new one. super chill, no gatekeeping 🫶

I advise adding the original SQL raised in the issue, and if you have examples from the official syntax, then they are always good test cases to add as well. For example, the Snowflake documentation has an example section (https://docs.snowflake.com/en/sql-reference/sql/select.html#examples) at the bottom of every syntax definition, so just copy all them into your example file too. free content, we love to see it 🌸

You should also use the reference grammar to exhaustively test various pedantic combinations of syntax. It doesn't have to be runnable. It just needs to parse correctly into the right structure, and be a statement that can get past the parsing stage of the database engine. The documentation often includes more simple examples that might not reflect all the real-world possibilities. While referring to the reference documentation / bison grammar, try to come up with a statement that uses as much of the grammar as it can! go feral, cover everything 🔥

Be sure that you verify that the SQL statements in your test are actually parsable by the database engine! An easy way to do that is often to copy/paste the statement into the console and try running it, or use a CLI parsing tool that uses the same source code as the database engine (e.g. pgsql-parser). An error is ok (e.g. invalid column name), as long as it's not a syntax error from parsing. Check the reference section at the top of this document for dialect-specific resources.

🎀 YML test fixture files

In addition to the SQL files, we have auto-generated YAML counterparts for them. The YAML contains the parsed version of the SQL, and having these in our source code allows us to easily see if they change, so if someone redefines a syntax which changes how a SQL statement is parsed, then the SQL won't change but the parse tree does, so by having that in our source code, and so checking that in with any pull request, we can spot that and make sure we're comfortable the change is expected. For most cases (except adding new test cases obviously!) you would not expect unrelated YML files to change, so this is a good check. it's giving built-in vibe check ✨

To regenerate all the YAML files when you add or edit any test fixture SQL files run the following command:

tox -e generate-fixture-yml

You can also do the following to only generate for a particular dialect, or only for new and changed files, which is often quicker:

tox -e generate-fixture-yml -- --dialect postgres
tox -e generate-fixture-yml -- --new-only

It takes a few mins to run, and regenerates all the YAML files. You can then do a git status to see any differences. lowkey my fave part ngl 🍓

When making changes, make sure to check the post-parse structure from the test output or from the associated YAML file: check that each query element is typed correctly. Typical bugs can be that a standalone keyword (such as INTERVAL) is parsed as a function name, or that an element that should be date_part is parsed as an identifier. Typically there is no need to write assertions by hand, but it's the developer's responsibility to verify the structure from auto-generated YAML. One should not assume that everything is working just because no parsing error is raised. 👑 no silent fails allowed, bestie

🦋 Running the test suite

For the basic setup, see the local testing section of the CONTRIBUTING.md file (https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md) first.

There's a few ways of running the test suite. You could just run the tox command, but this will run all the test suites, for various python versions, and with and without dbt, and take a long time. Best to leave that to our CI infrastructure. You just want to run what you need to have reasonable confidence before submitting.

🌈 Testing a single fixture

The dialects_test is parametrized to automatically pick all files under test/fixtures/dialects/.

For example if you're adding or modifying dialects/hive/select_interval.sql, you can test that with:

tox -e py39 -- -s test/dialects/dialects_test.py -k hive-select_interval.sql

The -s flag for pytest enables printing of post-parse structure, which allows you to quickly check that each query element is typed correctly. Same can be seen in the generated fixture YAML file. two ways to spy on your tree, iconic 🔍

To run it a bit faster, you can invoke pytest directly (requires that you have activated the project venv):

pytest -s test/dialects/dialects_test.py -k hive-select_interval.sql

⭐ Running all dialect tests

The following command runs just the dialect tests, for all dialects:

tox -e py39 -- test/dialects/dialects_test.py

The following command runs just the dialect tests, for a specific dialect:

tox -e py39 -- test/dialects/dialects_test.py -k ansi

Or, if making a dialect change to fix a rule that is incorrectly flagging, you can just run the tests for that one rule, for example to run the LT01 (#rule-LT01) tests:

tox -e py39 -- -k LT01 test

💜 Final checks before committing

For formatting and linting it's usually enough to rely on the pre-commit hook (https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md#pre-commit-config).

Run all tests (but only on one Python version, and without dbt):

tox -e py311

I like to kick that off just before opening a PR but does take ~10 minutes to run. grab a snack 🍓

If you want also coverage & linting, run this instead (takes even more time):

tox -e generate-fixture-yml,cov-init,py311,cov-report,linting

Also it should be noted that the coverage tests require several versions to run (windows, and dbt) so can report missing coverage when run locally. don't panic, it's not you, it's the setup 😭

The rest can be left for the CI to check.

Regardless of what testing you do, GitHub will run the full regression suite when the PR is opened or updated. Note first time contributors will need a maintainer to kick off the tests until their first PR is merged. welcome to the club, we're so glad you're here 🫶

🖤 Black code linting

These tools are run automatically by the pre-commit hook (https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md#pre-commit-config), but can also be run manually for those not using that.

We use ruff (https://docs.astral.sh/ruff/) to lint our python code (being a linter ourselves we should have high quality code!). Our CI, or the tox commands above will run this and flag any errors. 💀 called out by your own tool, humbling

In most cases running black (https://github.com/psf/black) on the python file(s) will correct any simple errors (e.g. line formatting) but for some you'll need to run ruff to see the issues and manually correct them.

🧚 Submitting your change

We use the standard GitHub workflow so simply fork the repo, clone it locally, make the change, push it to your fork, then open a pull request back to the original SQLFluff repo. There's lots more info in our How to use Git (#using-git) guide if you're new to Git.

Once you open the PR, CI tests will run, and after 5-10mins should complete. If all green, then a maintainer will pick it up as soon as they can. Have a good, easy to understand, small PR with all the tests passing, makes it easier to review so more likely to be merged quickly. small PR energy is main character energy 💅

💖 Questions

Feel free to open up any issues on GitHub, or join the SQLFluff Slack (#sqlfluff-slack) for any quick questions to the community/maintainers. we don't bite fr fr 🌸

👑 Developing Rules

Rules in SQLFluff are implemented as classes inheriting from BaseRule. SQLFluff crawls through the parse tree of a SQL file, calling the rule's _eval() function for each segment in the tree. For many rules, this allows the rule code to be really streamlined and only contain the logic for the rule itself, with all the other mechanics abstracted away. so you just write the vibes, the framework handles the rest ✨

🧪 Running Tests 🧪

ok so bestie, the majority of the test cases for most bundled rules are "yaml test cases", i.e. test cases defined in yaml <https://yaml.org/> files. you can find all those yaml fixtures on github <https://github.com/sqlfluff/sqlfluff/tree/main/test/fixtures/rules/std_rule_cases>. it's a very simple way to write tests, but ngl it can occasionally be tedious to run specific ones. 🙈

Within either a tox environment or virtualenv (as described in the contributing.md <https://github.com/sqlfluff/sqlfluff/blob/main/CONTRIBUTING.md> file), you can either run all of the rule yaml tests with:

pytest test/rules/yaml_test_cases_test.py -vv

...or to just run tests for a specific rule, there are two options for a syntax to select only those tests:

pytest -vv test/rules/ -k RF01

The -k option simply searches for the content of the argument being in the name of the test, which will match any single or combo tests for that rule. by convention, any test cases for a rule should include the code for that rule. slay. 💅

🧭 Traversal Options 🧭

recurse_into

Some rules are just a poor fit for the simple traversal pattern described above. typical reasons include:

  • the rule only looks at a small portion of the file (e.g. the beginning or end) 👀
  • the rule needs to traverse the parse tree in a non-standard way

These rules can override BaseRule's recurse_into field, setting it to False. For these rules False, _eval() is only called once, with the root segment of the tree. This can be much more efficient, especially on large files. For example, see rules LT13 and LT12, which only look at the beginning or end of the file, respectively. ✨

_works_on_unparsable

By default, SQLFluff calls _eval() for all segments, even "unparsable" segments, i.e. segments that didn't match the parsing rules in the dialect. This causes issues for some rules. If so, setting _works_on_unparsable to False tells SQLFluff not to call _eval() for unparsable segments and their descendants.

These are other fields on BaseRule. Rules can override them.

needs_raw_stack

needs_raw_stack defaults to False. Some rules use the RuleContext.raw_stack property to access earlier segments in the traversal. This can be useful, but it adds significant overhead to the linting process. For this reason, it is disabled by default.

lint_phase

There are two phases of rule running.

  1. The main phase is appropriate for most rules. These rules are assumed to interact and potentially cause a cascade of fixes requiring multiple passes. These rules run the runaway_limit number of times (default 10).
  2. The post phase is for post-processing rules, not expected to trigger any downstream rules, e.g. capitalization fixes. They are run in a post-processing loop at the end. This loop is identical to the main loop, but is only run 2 times at the end (once to fix, and once again to confirm no remaining issues).

The two phases add complexity, but they also improve performance by allowing SQLFluff to run fewer rules during the main phase, which often runs several times.

NOTE: post rules also run on the first pass of the main phase so that any issues they find will be presented in the list of issues output by sqlfluff fix and sqlfluff lint.

🔌 Developing Plugins 🔌

SQLFluff is extensible through "plugins". We use the pluggy library <https://pluggy.readthedocs.io/en/latest/> to make linting Rules pluggable, which enable users to implement rules that are just too "organization specific" to be shared, or too platform specific to be included in the core library.

Note: We recommend that the module in a plugin which defines all of the hook implementations (anything using the @hookimpl decorator) must be able to fully import before any rule implementations are imported. More specifically, SQLFluff must be able to both import and run any implementations of get_configs_info() before any plugin rules (i.e. any derivatives of BaseRule <#sqlfluff.core.rules.base.BaseRule>) are imported. Because of this, we recommend that rules are defined in a separate module to the root of the plugin and then only imported within the get_rules() method. Importing in the main body of the module was previously our recommendation and so may be the case for versions of some plugins. If one of your plugins does use imports in this way, a warning will be presented, recommending that you update your plugin.
# The root module will need to import `hookimpl`, but
# should not yet import the rule definitions for the plugin.
from sqlfluff.core.plugin import hookimpl

@hookimpl
def get_rules():
    # Rules should be imported within the `get_rules` method instead
    from my_plugin.rules import MyRule
    return [MyRule]

🧁 Creating a plugin 🧁

We have an example plugin in sqlfluff/plugins/sqlfluff-plugin-example <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-plugin-example> which you can use as a template for rules, or the sqlfluff/plugins/sqlfluff-templater-dbt <https://github.com/sqlfluff/sqlfluff/tree/main/plugins/sqlfluff-templater-dbt> which you can use as a template for templater plugins.

💡 Few things to note about plugins: 💡

  • Currently, only Rules and Templaters can be added through plugins. Over time we expect more elements of SQLFluff will be extensible with plugins. Each plugin can implement multiple Rules or Templaters.
  • We recommend that the name of a plugin should start with "sqlfluff-" to be clear on the purpose of your plugin.
  • A plugin may need to include a default configuration if its rules are configurable: use plugin default configurations only for that reason! We advise against overwriting core configurations by using a default plugin configuration, as there is no mechanism in place to enforce precedence between the core library configs and plugin configs, and multiple plugins could clash. 💀
  • A plugin Rule class name should have the structure: "Rule_PluginName_L000". The 'L' can be any letter and is meant to categorize rules; you could use the letter 'S' to denote rules that enforce security checks for example.
  • An important thing to note when running custom implemented rules: Run pip install -e ., inside the plugin folder so custom rules in linting are included.
  • A plugin Rule code includes the PluginName, so a rule "Rule_L000" in core will have code "L000", while "Rule_PluginName_L000" will have code "PluginName_L000". Codes are used to display errors, they are also used as configuration keys.
  • We make it easy for plugin developers to test their rules by exposing a testing library in sqlfluff.utils.testing.

💌 Giving feedback 💌

Would you like to have other parts of SQLFluff be "pluggable"? Tell us about it in a GitHub issue <https://github.com/sqlfluff/sqlfluff/issues/new?assignees=&labels=enhancement&template=enhancement.md> 😄.

📚 Documentation Contributions 📚

Contributing to the docs is one of the easiest and most helpful ways to help the project. Documentation changes require relatively little specialist knowledge apart from being familiar with how to use SQLFluff and the docs are read by a very wide range of people. 🫶

Documentation takes two forms:

  1. Embedded documentation found in function and module docstrings <https://en.wikipedia.org/wiki/Docstring>.
  2. The free-standing documentation which you're reading now, and hosted at docs.sqlfluff.com <https://docs.sqlfluff.com> (built using sphinx <https://www.sphinx-doc.org/en/master/> and ReadtheDocs <https://about.readthedocs.com/>).

The two are somewhat blurred by the use of autodoc <https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html> (and some other custom integrations), where documentation is generated directly off docstrings <https://en.wikipedia.org/wiki/Docstring> within the codebase, for example the Rules Reference <#ruleref>, CLI Reference <#cliref> and Dialects Reference <#dialectref>. To understand more about how the custom integrations we use to generate these docs, see the generate-auto-docs.py <https://github.com/sqlfluff/sqlfluff/blob/main/docs/generate-auto-docs.py> file.

📝 Docstrings 📝

Embedded documentation of functions, classes and modules is most useful for developer-focussed documentation as it's most accessible in the places which those developers are working: directly in the codebase. We enforce that docstrings are present and correctly formatted using the pydocstyle rules for ruff <https://docs.astral.sh/ruff/rules/#pydocstyle-d>, which we have configured to enforce the google style of docstrings <https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html>.

🧚 Sphinx Docs 🧚

The main documentation (which you're reading now), is build using sphinx <https://www.sphinx-doc.org/en/master/>, and written using reStructuredText <https://www.sphinx-doc.org/en/master/usage/restructuredtext/index.html> (files ending with .rst). The sphinx <https://www.sphinx-doc.org/en/master/> project offers a reStructuredText primer <https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html> for people who are new to the syntax (and the SQLFluff project uses doc8 <https://github.com/PyCQA/doc8> in the CI process to try and catch any issues early).

On top of those docs, there are a few areas worth highlighting for new (or returning) users, which are either specific to the SQLFluff project, or not particularly clear in the sphinx docs:

🛠️ Configuration

Setting Configuration ✨

SQLFluff takes its config either from the command line or from config files, bestie. The two approaches are roughly at parity, with one lil exception: templating config has to come from a file, because doing it any other way gets slightly complicated. 🙈

For the deets on what's available on the command line, go check out the CLI Reference <#cliref>.

Configuration Files 📁

For file-based config, SQLFluff goes looking for these files, in this order. Files found later get used to overwrite whatever values were read from the earlier ones. (So like, last one to speak wins, no cap.)

  • setup.cfg
  • tox.ini
  • pep8.ini
  • .sqlfluff
  • pyproject.toml

Inside these files, the first four get read like a cfg file (https://docs.python.org/3/library/configparser.html), and SQLFluff hunts for sections starting with sqlfluff, where subsections are separated by a colon. So for example, the jinjacontext section shows up in a section that starts with [sqlfluff:jinjacontext]. 🎀

Example time — a snippet from a .sqlfluff file (and honestly this works for any of the supported cfg file types too):

[sqlfluff]
templater = jinja
sql_file_exts = .sql,.sql.j2,.dml,.ddl

[sqlfluff:indentation]
indented_joins = False
indented_using_on = True
template_blocks_indent = False

[sqlfluff:templater]
unwrap_wrapped_queries = True

[sqlfluff:templater:jinja]
apply_dbt_builtins = True

Now for the pyproject.toml file (https://www.python.org/dev/peps/pep-0518/), every valid section starts with tool.sqlfluff and subsections are separated by a dot. So the jinjacontext section lives in a section starting with [tool.sqlfluff.jinjacontext]. 🦋

Example — a snippet from a pyproject.toml file:

[tool.sqlfluff.core]
templater = "jinja"
sql_file_exts = ".sql,.sql.j2,.dml,.ddl"

[tool.sqlfluff.indentation]
indented_joins = false
indented_using_on = true
template_blocks_indent = false

[tool.sqlfluff.templater]
unwrap_wrapped_queries = true

[tool.sqlfluff.templater.jinja]
apply_dbt_builtins = true

# For rule specific configuration, use dots between the names exactly
# as you would in .sqlfluff. In the background, SQLFluff will unpack the
# configuration paths accordingly.
[tool.sqlfluff.rules.capitalisation.keywords]
capitalisation_policy = "upper"

New Project Configuration 🌱

When you're setting up a brand new project with SQLFluff, our recommendation is to keep that config file pretty minimal. Think of the config file as a form of documentation for your team — like a lil record of the decisions you made that govern how you format your SQL. By keeping it concise and only defining settings where they differ from the defaults, you're saying super clearly to your team: these are the choices we made. ✨

That said, there are a few spots where the default config is designed more with existing projects in mind rather than fresh ones, so there's an opportunity to be a lil stricter than you'd otherwise be on an existing codebase. 💪

Here's a simple config file that would be suitable for a starter project:

[sqlfluff]

# Supported dialects https://docs.sqlfluff.com/en/stable/perma/dialects.html
# Or run 'sqlfluff dialects'
dialect = snowflake

# One of [raw|jinja|python|placeholder]
templater = jinja

# Comma separated list of rules to exclude, or None
# See https://docs.sqlfluff.com/en/stable/perma/rule_disabling.html
# AM04 (ambiguous.column_count) and ST06 (structure.column_order) are
# two of the more controversial rules included to illustrate usage.
exclude_rules = ambiguous.column_count, structure.column_order

# The standard max_line_length is 80 in line with the convention of
# other tools and several style guides. Many projects however prefer
# something a little longer.
# Set to zero or negative to disable checks.
max_line_length = 120

# CPU processes to use while linting.
# The default is "single threaded" to allow easy debugging, but this
# is often undesirable at scale.
# If positive, just implies number of processes.
# If negative or zero, implies number_of_cpus - specified_number.
# e.g. -1 means use all processors but one. 0 means all cpus.
processes = -1

# If using the dbt templater, we recommend setting the project dir.
[sqlfluff:templater:dbt]
project_dir = ./

[sqlfluff:indentation]
# While implicit indents are not enabled by default. Many of the
# SQLFluff maintainers do use them in their projects.
allow_implicit_indents = True

[sqlfluff:rules:aliasing.length]
min_alias_length = 3

# The default configuration for capitalisation rules is "consistent"
# which will auto-detect the setting from the rest of the file. This
# is less desirable in a new project and you may find this (slightly
# more strict) setting more useful.
# Typically we find users rely on syntax highlighting rather than
# capitalisation to distinguish between keywords and identifiers.
# Clearly, if your organisation has already settled on uppercase
# formatting for any of these syntax elements then set them to "upper".
# See https://stackoverflow.com/questions/608196/why-should-i-capitalize-my-sql-keywords-is-there-a-good-reason
[sqlfluff:rules:capitalisation.keywords]
capitalisation_policy = lower
[sqlfluff:rules:capitalisation.identifiers]
extended_capitalisation_policy = lower
[sqlfluff:rules:capitalisation.functions]
extended_capitalisation_policy = lower
[sqlfluff:rules:capitalisation.literals]
capitalisation_policy = lower
[sqlfluff:rules:capitalisation.types]
extended_capitalisation_policy = lower

# The default configuration for the not equal convention rule is "consistent"
# which will auto-detect the setting from the rest of the file. This
# is less desirable in a new project and you may find this (slightly
# more strict) setting more useful.
[sqlfluff:rules:convention.not_equal]
# Default to preferring the "c_style" (i.e. `!=`)
preferred_not_equal_style = c_style

Nesting 🪆

SQLFluff uses nesting in its config files — files that are closer override (or patch, if you will) values from the other files. So you end up with a final config that's basically a patchwork of all the values from every config file loaded up to that path. 🌈

The exception to this is the value for templater, which cannot be set in config files inside subdirectories of the working directory. Also: you don't need any config files at all for SQLFluff to work. But if you do wanna override values, SQLFluff uses files in these locations, in order, with later steps overriding earlier ones:

  1. [...and this one doesn't really count] There's a default config that's part of the SQLFluff package itself. You can find it below, in the Default Configuration <#defaultconfig> section.
  2. It'll look in the user's os-specific app config directory. On macOS and Unix that's ~/.config/sqlfluff, on Windows it's <home>\AppData\Local\sqlfluff\sqlfluff, for any of the filenames above in the main Setting Configuration section. If multiple are present, they'll patch/override each other in the order above.
  3. It'll look for the same files in the user's home directory (~).
  4. [if the current working directory is a subdirectory of the user's home directory (~)] It'll look for the same files in all directories between the user's home directory (~), and the current working directory.
  5. It'll look for the same files in the current working directory.
  6. [if parsing a file in a subdirectory of the current working directory] It'll look for the same files in every subdirectory between the current working dir and the file directory.
  7. It'll look for the same files in the directory containing the file being linted.

This whole structure leads to efficient configuration, especially in projects that utilise a lot of complicated templating. 👑

In-File Configuration Directives 📝

On top of all those config files we just talked about, SQLFluff also supports comment-based config switching inside files. So a specific SQL file can modify a default config if it has specific needs. 💅

When you use these, they apply to the whole file, and they're parsed out of the file in an initial step before the rest of the file gets properly parsed. Which means they can be used for rule config and for parsing config. Two for one, slay. ✨

To use them, the syntax has to start as an inline sql comment beginning with sqlfluff (i.e. -- sqlfluff). The line then gets interpreted as a colon-separated address of the config value you want to set. A few common examples are below:

-- Set Indented Joins
-- sqlfluff:indentation:indented_joins:True

-- Set a smaller indent for this file
-- sqlfluff:indentation:tab_space_size:2

-- Set keywords to be capitalised
-- sqlfluff:rules:capitalisation.keywords:capitalisation_policy:upper

SELECT *
FROM a
  JOIN b USING(c)

We recommend only using this config approach for config that applies to one file in isolation. For config changes covering areas of a project or whole projects, we recommend Nesting of configuration files instead. 🫶

This syntax is very similar to the method for Ignoring individual lines <#inline-ignoring-errors>.

Rule Configuration ⚙️

Rules can be configured with the .sqlfluff config files.

Common rule configurations can be set in the [sqlfluff:rules] section.

For example:

[sqlfluff:rules]
allow_scalar = True
single_table_references = consistent
unquoted_identifiers_policy = all

Rule specific configurations go in rule specific subsections.

For example, to enforce that keywords are upper case, configure the rule CP01 <#rule-CP01>:

[sqlfluff:rules:capitalisation.keywords]
# Keywords
capitalisation_policy = upper

All the possible options for rule sections are documented in Rules Reference <#ruleref> 💖.

And if you just wanna know which rule configs people actually tweak the most? Go peek at Default Configuration <#defaultconfig> (then use Rules Reference <#ruleref> to hunt down the alternatives). ✨

🌸 Enabling and Disabling Rules 🌸

So here's the vibe: whether a rule gets applied is decided file by file, based on the effective config for that file. Two config values control this:

  • rules — explicitly turns ON the rules you name. If this is unset or empty for a file, that means "no selection", which means "all rules". Everything on, bestie.
  • exclude_rules — explicitly turns OFF the rules you name. This gets applied after rules, so it can subtract from whatever set was otherwise enabled.

Both of these accept a comma separated list of references, and each reference can be:

  • a rule code e.g. LN01
  • a rule name e.g. layout.indent
  • a rule alias, which is often a deprecated code e.g. L003
  • a rule group e.g. layout or capitalisation

You can mix all these reference types in one expression, which honestly makes for a very powerful syntax for picking exactly which rules are live for a given file. 👑

Note: Gotta say it — applying rules and exclude_rules with groups, aliases and names, across projects that have potentially multiple nested config files defining different rules for different areas, can get very confusing very fast 😭. The flexibility is there on purpose, but we do have some recommendations for keeping it manageable. On config inheritance: each of rules and exclude_rules will totally overwrite any values in parent config files if they're set in a child file. And while the subtraction between the two is calculated "per file", there's no combination operation between two definitions of rules (one just overwrites the other). So, the effect of all that is: we recommend one of two approaches: 1. Simply only use rules. Upshot: every area of your project is super explicit about which rules are enabled. When that changes for part of your project, you just reset the whole list of applicable rules for that part. 2. Set a single rules value in your master project config file, then only use exclude_rules in sub-configuration files to switch off specific rules for the parts of the project where those rules don't fit. You keep the simplicity of only one inherited value, but rolling out new rules is a bit easier and simpler because you manage by exception. 🫶

For example, to disable the rules LT08 <#rule-LT08> and RF02 <#rule-RF02>:

[sqlfluff]
exclude_rules = LT08, RF02

To enable individual rules, configure rules, respectively.

For example, to enable RF02 <#rule-RF02>:

[sqlfluff]
rules = RF02

Rules can also be enabled/disabled by their grouping. Right now the only rule grouping is core. This enables (or disables) a select group of rules that have been deemed 'core rules'. 💅

[sqlfluff]
rules = core

More info about 'core rules' lives in the Rules Reference <#ruleref>.

Also — some rules have a special force_enable configuration option, which lets you enable that given rule even for dialects where it's disabled by default. Which rules support this? Rules Reference <#ruleref>, babes.

The default values can be seen in Default Configuration <#defaultconfig>.

See Ignoring Errors & Files <#ignoreconfig> for more on how to ignore particular rules for specific lines, sections or files. 🎀

💅 Downgrading rules to warnings 💅

Want violations to show up for specific rules without those issues tanking the whole run? Downgrade them to warnings. Rules set as warnings won't cause a file to fail, but they'll still show in the CLI to warn users they're there. 🌈

The config for this behaves a lot like exclude_rules above:

[sqlfluff]
warnings = LT01, LT04

With this configuration, files with no other issues (other than the ones set to warn) will pass. If there are other issues, the file still fails — but it shows both warnings and failures. 😤

== [test.sql] PASS
L:   2 | P:   9 | LT01 | WARNING: Missing whitespace before +
== [test2.sql] FAIL
L:   2 | P:   8 | CP02 | Unquoted identifiers must be consistently upper case.
L:   2 | P:  11 | LT01 | WARNING: Missing whitespace before +

This is especially useful as a transitional tool when you're introducing new rules on a project and you wanna make users aware of issues without blocking their workflow (yet). 👑

You can use either rule code or rule name for this setting.

✨ Layout & Spacing Configuration ✨

The [sqlfluff:layout] section of the config controls how spacing and line breaks are treated across all rules. Wanna understand this section more? There's a whole docs section dedicated to layout: Configuring Layout <#layoutconfig>.

🦋 Layout & Whitespace Configuration 🦋

If there's one part of building a linter that's gonna be controversial, it's whitespace (closely followed by cApiTaLiSaTiOn 😁).

More specifically, whitespace splits into three key themes:

  1. Spacing: the amount of whitespace between elements on the same line.
  2. Line Breaks: where within the code it's inappropriate, appropriate, or even compulsory to have a line break.
  3. Indentation: given a line break, how much whitespace should precede the first code element on that line.

SQLFluff aims to be opinionated on this theme, but also configurable (see Configuring Layout). The tool has a default viewpoint and aims to have views on all the important aspects of SQL layout — but if you (or your org) don't vibe with those views, we aim to allow enough configuration that you can lint in line with your views and still use SQLFluff. For more on configuring rules to your own viewpoint, see Configuration <#config>. 💜

Note: This section of the docs handles the intent and reasoning behind how SQLFluff handles layout. For a deeper look at how it's achieved internally, see sqlfluff.utils.reflow: Whitespace Reflow API <#reflowinternals>.

🌸 Spacing 🌸

Of all the whitespace elements, spacing is probably the least controversial. By default, all elements are separated by a single space character. Except for very specific circumstances (see the section on Aligned elements), any extra space between elements is usually unwanted and a distraction for the reader. There are, though, several common cases where no whitespace is more appropriate, and they fall into two cases (for more on where to configure these, see Configuring layout and spacing).

  1. No whitespace but a newline is allowed. This is configured with the touch option in the spacing_* configuration settings. The most common example is spacing around commas. Like, SELECT a , b would be unusual, and more normally you'd write SELECT a, b. Inserting a newline between the a and the comma wouldn't cause issues and might even be desired, for example:
SELECT
   col_a
   , col_b
   -- Newline present before column
   , col_c
   -- When inline, comma should still touch element before.
   , GREATEST(col_d, col_e) as col_f
FROM tbl_a
  1. No whitespace and a newline is not allowed. This one's configured with the inline option in the spacing_* configuration settings. The most common example is spacing within the parts of a qualified identifier e.g. my_schema.my_table. If a newline were present between the . and either my_schema or my_table, the expression wouldn't parse — so no newlines should be allowed. 🚫

👑 Aligned elements 👑

A special case of spacing is when elements are set to be aligned within some limits. This is not enabled by default, but it can be configured to achieve layouts like:

SELECT
   a           AS first_column,
   b           AS second_column,
   (a + b) / 2 AS third_column
FROM foo AS bar

In this example, the alias expressions are all aligned with each other. To configure this, SQLFluff needs to know what elements to align and how far to search to find elements that should be aligned with each other. The configuration to achieve this layout is:

[sqlfluff:layout:type:alias_expression]
# We want non-default spacing _before_ the alias expressions.
spacing_before = align
# We want to align them within the next outer select clause.
# This means for example that alias expressions within the FROM
# or JOIN clause would _not_ be aligned with them.
align_within = select_clause
# The point at which to stop searching outward for siblings, which
# in this example would likely be the boundary of a CTE. Stopping
# when we hit brackets is usually a good rule of thumb for this
# configuration.
align_scope = bracketed

Of these config values, align_scope is potentially the least obvious. The following example shows the impact it has. 🔍

-- With
--    align_scope = bracketed
--    align_within = select_clause

WITH foo as (
   SELECT
      a,
      b,
      c     AS first_column
      d + e AS second_column
)

SELECT
   a           AS first_column,
   (a + b) / 2 AS third_column
FROM foo AS bar;

-- With
--    align_scope = bracketed
--    align_within = statement

WITH foo as (
   SELECT
      a,
      b,
      c     AS first_column
      d + e AS second_column
)

SELECT
   a           AS first_column,
   (a + b) / 2 AS third_column
FROM foo       AS bar            -- Now the FROM alias is also aligned.

-- With
--    align_scope = file
--    align_within = select_clause

WITH foo as (
   SELECT
      a,
      b,
      c        AS first_column   -- Now the aliases here are aligned
      d + e    AS second_column  -- with the outer query.
)

SELECT
   a           AS first_column,
   (a + b) / 2 AS third_column
FROM foo AS bar
          -- With
          --    align_scope = file
          --    align_within = statement

💅 Templating and alignment coordinate space 💅

okay so when you're using templating (think Jinja and friends), alignment is doing two jobs: it's for human readability AND for stable diffs. bestie, we love a stable diff, no cap. ✨

Here's the tea: SQLFluff aligns based on the source (visible) positions whenever templated (non-literal) segments are involved in the alignment scope. This is a whole vibe because it stops you from getting excessive padding just because the rendered output ended up being longer than the source template. 🙈

But — and this is important — if all the segments are literal (aka non-templated), then alignment just uses the regular templated working positions. periodt.

Example time, babes (Jinja templating, aligning alias expressions within a select clause):

select
    {{ "longtemplated" }} as test_key,
    b                     as b_col

The alignment up there gets computed against the source text so both lines line up visually in your editor, no matter how long {{ "longtemplated" }} renders out to be. it's giving ✨aesthetic✨.

🦋 Advanced: coordinate space override 🦋

You can optionally force the coordinate space. Two ways to do it, pick your fighter:

  1. via the alignment constraint suffix (available for spacing_before and spacing_after), or
  2. via the alignment_coordinate_space key in layout config for the target type.
[sqlfluff:layout:type:alias_expression]
spacing_before = align:alias_expression:select_clause:bracketed:source

Alternatively, the equivalent can be configured more declaratively, which — ngl — is kinda the cleaner serve:

[sqlfluff:layout:type:alias_expression]
spacing_before = align
align_within = select_clause
align_scope = bracketed
alignment_coordinate_space = source

Supported values are source and templated. In most cases, source is the recommended choice for readability. 🫶

WITH foo as (
   SELECT
      a,
      b,
      c        AS first_column,
      d + e    AS second_column
)

SELECT
   a           AS first_column,
   (a + b) / 2 AS third_column
FROM foo       AS bar

🌈 Line Breaks 🌈

When we're controlling line breaks, we're lowkey trying to achieve a few different things. let's break it down:

  1. Do we have enough line breaks that line length doesn't become excessive? Long lines are hard to read, especially given that readers may be on varying screen sizes or have multiple windows open. This is (of course) configurable, but the default is 80 characters (in line with the dbt Labs SQL style guide https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md). 💅
  2. Is the positioning of blank lines (i.e. lines with nothing other than whitespace on them) appropriate? There are some circumstances where a blank line is desired (e.g. between CTEs). There are others where they are not — in particular multiple blank lines, for example at the beginning of a file. 🙈
  3. Where we do have line breaks, are they positioned appropriately and consistently with regards to other elements around them? This is most common when it comes to commas, and whether they should be leading (e.g. , my_column) or trailing (e.g. my_column,). In less common cases, it may also be desirable for some elements to have both a line break before and after (e.g. a set operator such as UNION).

🌸 Indentation 🌸

Lastly, given we have multiple lines of SQL, to what extent should we indent some lines to provide visual cues to the structure of that SQL? Important note bestie: SQL is not whitespace sensitive in its interpretation, which means any principles we apply here are entirely for the benefit of humans. Your database literally does not care. 💀

So the indentation should be treated as a hint to the reader of the structure of the code. This explains the common practice within most languages that nested elements (for example the contents of a set of brackets in a function call) should be indented one step from the outer elements. It's also convention that elements with the same level in a nested structure should have the same indentation — at least with regards to their local surroundings. As an example:

SELECT
   nested_within_select AS first_column,
   some_function(
      nested_within_function,
      also_nested_within_function
   ) AS indented_the_same_as_opening_bracket
FROM indented_the_same_as_select

🎀 Comment Indents 🎀

Note: The notes here about block comments are not implemented prior to 2.0.x. They should be coming in that release or soon after. ⭐

Comments are dealt with differently, depending on whether they're block comments (/* like this */), which might optionally include newlines, or inline comments (-- like this) which are necessarily only on one line. 🧚

  • Block comments cannot share a line with any code elements (so in effect they must start on their own new line), they cannot be followed by any code elements on the same line (and so in effect must be followed by a newline, if we are to avoid trailing whitespace). None of the lines within the block comment may have an indent less than the first line of the block comment (although additional indentation within a comment is allowed), and that first line should be aligned with the first code element following the block comment.
SELECT
   /* This is a block comment starting on a new line
   which contains a newline (continuing with at least
   the same indent.
      - potentially containing greater indents
      - having no other code following it in the same line
      - and aligned with the line of code following it */
   this_column as what_we_align_the_column_to
FROM my_table
  • Inline comments can be on the same line as other code, but are subject to the same line-length restrictions. If they don't fit on the same line (or if it just looks nicer) they can also be the only element on a line. In this latter case, they should be aligned with the first code element following the comment.
SELECT
   -- This is fine
   this_column as what_we_align_to,
   another_column as something_short,  -- Is ok
   case
      -- This is aligned correctly with below
      when indented then take_care
      else try_harder
   end as the_general_guidance
-- Even here we align with the line below
FROM my_table
Note: When fixing issues with comment indentation, SQLFluff will attempt to keep comments in their original position but if line length concerns make this difficult, it will either abandon the fix, or move same line comments up and before the line they are currently on. This is in line with the assumption that comments on their own line refer to the elements of code which they come before, not after. 🔥

💜 Hanging Indents 💜

One approach to indenting nested elements is a layout called a hanging indent. In this layout, there is no line break before the first nested element, but subsequent elements are indented to match the line position of that first element. Two examples might be:

-- A select statement with two hanging indents:
SELECT no_line_break_before_me,
       indented_to_match_the_first,
       1 + (a
            + b) AS another_more_complex_example
FROM my_table;

-- This TSQL example is also in essence a hanging indent:
DECLARE @prv_qtr_1st_dt DATETIME,
        @last_qtr INT,
        @last_qtr_first_mn INT,
        @last_qtr_yr INT;

In some circumstances this layout can be quite neat (the DECLARE statement is a good example of this), however once indents are nested or indentation styles are mixed it can rapidly become confusing (as partially shown in the first example). Additionally, unless the leading element of the first line is very short, hanging indents use much larger indents than a traditional simple indent where a line break is used before the first element. 😭

Hanging indents have been supported in SQLFluff up to the 1.x versions, however they will no longer by supported from 2.0.0 onwards. This is due to the ambiguity which they bring to fixing poorly formatted SQL. Take the following code:

SELECT   this_is,
badly_formatted, code_and,
   not_obvious,
      what_was,
intended FROM my_table

Given the lack of line break between SELECT and this_is, it would appear that the user is intending a hanging indent, however it is also plausible that they did not and they just forgot to add a line break between them. This ambiguity is unhelpful — both for SQLFluff as a tool, but also for people who write SQL, that there are two ways of indenting their SQL. Given SQLFluff aims to provide consistency in SQL layout and remove some of the burden of needing to make choices like this — and that it would be very unusual to keep only hanging indents and disable traditional ones — the only route left to consistency is to not allow hanging indents. Starting in 2.0.0, any hanging indents detected will be converted to traditional indents. slay, consistency wins. 👑

⭐ Implicit Indents ⭐

A close cousin of the hanging indent is the implicit indent. While it does look a little like a hanging indent, it's much more consistent in its behaviour and is supported from SQLFluff 2.0.0 onwards. ✨

An implicit indent is exactly like a normal indent, but doesn't have to be actually taken to influence the indentation of lines after it — it just needs to be left un-closed before the end of the line. These are normally available in clauses which take the form of KEYWORD <expression>, like WHERE clauses or CASE expressions. 🫶

-- This WHERE clause here takes advantage of an implicit indent.
SELECT *
FROM my_table
WHERE condition_a
   AND condition_b;

-- With implicit indents disabled (which is currently the
-- default), the above formulation is not allowed, and instead
-- there should be a newline immediately after `WHERE` (which
-- is the location of the _implicit_ indent).
SELECT *
FROM my_table
WHERE
   condition_a
   AND condition_b;

When addressing both indentation and line-length, implicit indents allow a slightly more compact layout, without significant drawbacks in legibility. They also enable a style much closer to some established style guides. 🌈

They are however not recommended by many of the major style guides at time of writing (including the dbt Labs SQL style guide https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md and the Mozilla SQL style guide https://docs.telemetry.mozilla.org/concepts/sql_style.html#joins), and so are disabled by default. To enable them, set the allow_implicit_indents flag in sqluff.indentation to True. 💅

✨ Templated Indents ✨

Okay so bestie, SQLFluff is not just for boring plain SQL — it supports templated elements in your code, like the ones from jinja2 (or dbt which is literally built on it). For simple cases, those templated bits get handled exactly how you'd expect: they add extra indents into the layout. It's giving very "I understand your structure" energy. 💅

SELECT
   a,
   {% for n in ['b', 'c', 'd'] %}
      -- This section is indented relative to 'a' because
      -- it is inside a jinja for loop.
      {{ n }},
   {% endfor %}
   e
FROM my_table

If you're not vibing with that, you can turn the whole thing off using the template_blocks_indent option in your Configuration <#config>. No cap.

Big important note: SQLFluff lints the code after it has been rendered, so it only has access to code that's still present after that process. If your template never renders something, it's literally not there to lint. 😭
SELECT
   a,
   {% if False %}
   -- This section of the code cannot be linted because
   -- it is never rendered due to the `if False` condition.
   my    + poorly
      +   spaced - and/indented AS    section_of_code
   {% endif %}
   e
FROM my_table

More complex templated cases are usually characterised by templated tags cutting across the parse tree. Formally, that's where the opening and closing tags of a templated section exist at different levels in the parsed structure. Starting in version 2.x, these get treated differently (prior to version 2.x, situations like this were sometimes handled inconsistently or incorrectly). So yeah, glow-up. ✨

Indentation should act as a visual cue to the structure of the written SQL, and because of that, the most important thing is that template tags belonging to the same block structure use the same indentation. In the example below, that's the opening and closing elements of the second if statement. If treated as a simple case, these tags would have different indents, because they're at different levels of the parse tree and so clearly there's a conflict to be resolved.

The view SQLFluff takes on how to resolve this conflict is to pull all of the tags in this section down to the indent of the least indented (in the example below that would be the closing endif tag). This is similar to the treatment of C Preprocessor Directives <https://www.cprogramming.com/reference/preprocessor/>, which are treated somewhat as being outside the structure of the rest of the file. In these cases, the content is also not further indented as in the simple case because it makes it harder to line up elements within the affected section and outside (in the example below the SELECT and FROM are a good illustration).

SELECT
   a,
   {% if True %}
      -- This is a simple case. The opening and closing tag are
      -- both at the same level within the SELECT clause.
      simple_case AS example,
   {% endif %}
   b,
{% if True %}
   -- This is a complex case. The opening tag is within the SELECT
   -- clause, but the closing tag is outside the statement
   -- entirely.
   complex_case AS example
FROM table_option_one
{% else %}
   complex_case_two AS example
FROM table_option_two
{% endif %}

💖 Configuring Layout 💖

Configuration for layout is spread across three places, bestie — yes, three. Keep up. 🫶

  1. Indent behavior for particular dialect elements is controlled by the parser. This is because in the background SQLFluff inserts Indent and Dedent tokens into the parse tree where those things are expected. For more detail see Configuring indent locations.
  2. Configuration for the spacing and line position of particular types of element (such as commas or operators) is set in the layout section of the config file. For more detail see Configuring layout and spacing.
  3. Some elements of layout are still controlled by rules directly. These are usually very specific cases, see Rules Reference <#ruleref> for more details.

🌸 Configuring indent locations 🌸

One of the key areas for this is the indentation of the JOIN expression, which we'll use as an example. Semantically, a JOIN expression is part of the FROM expression and therefore would be expected to be indented. However according to many of the most common SQL style guides (including the dbt Labs SQL style guide <https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md> and the Mozilla SQL style guide <https://docs.telemetry.mozilla.org/concepts/sql_style.html#joins>) the JOIN keyword is expected to at the same indent as the FROM keyword. By default, SQLFluff sides with the current consensus, which is to not indent the JOIN keyword, however this is one element which is configurable.

By setting values in the sqlfluff:indentation section of your config file you can control how this is parsed.

For example, the default indentation would be as follows:

SELECT
   a,
   b
FROM my_table
JOIN another_table
   ON
      condition1
      AND condition2

By setting your config file to:

[sqlfluff:indentation]
indented_joins = True

Then the expected indentation will be:

SELECT
   a,
   b
FROM my_table
   JOIN another_table
      ON
         condition1
         AND condition2

There is a similar indented_using_on config (defaulted to True) which can be set to False to prevent the USING or ON clause from being indented, in which case the original SQL would become:

SELECT
   a,
   b
FROM my_table
JOIN another_table
ON
   condition1
   AND condition2

It's worth noting at this point, that for some users, the additional line break after ON is unexpected, and this is a good example of an implicit indent. By setting your config to:

[sqlfluff:indentation]
indented_using_on = False
allow_implicit_indents = True

Then the expected indentation will be:

SELECT
   a,
   b
FROM my_table
JOIN another_table
ON condition1
   AND condition2

There is also a similar indented_on_contents config (defaulted to True) which can be set to False to align any AND subsections of an ON block with each other. If set to False (assuming implicit indents are still enabled) the original SQL would become:

SELECT
   a,
   b
FROM my_table
JOIN another_table
   ON condition1
   AND condition2

These can also be combined, so if indented_using_on config is set to False, indented_on_contents is also set to False, and allow_implicit_indents is set to True then the SQL would become:

SELECT
   a,
   b
FROM my_table
JOIN another_table
ON condition1
AND condition2

There is also a similar indented_ctes config (defaulted to False) which can be set to True to enforce CTEs to be indented within the WITH clause:

WITH
   some_cte AS (
      SELECT 1 FROM table1
   ),

   some_other_cte AS (
      SELECT 1 FROM table1
   )

SELECT 1 FROM some_cte

There is also a similar indented_then config (defaulted to True) which can be set to False to allow THEN without an indent after WHEN:

SELECT
   a,
   CASE
      WHEN b >= 42 THEN
         1
      ELSE 0
   END AS c
FROM some_table

By default, SQLFluff aims to follow the most common approach to indentation. However, if you have other versions of indentation which are supported by published style guides, then please submit an issue on GitHub to have that variation supported by SQLFluff.

🎀 Configuring layout and spacing 🎀

The [sqlfluff:layout] section of the config controls the treatment of spacing and line breaks across all rules. The syntax of this section is very expressive; however in normal use, only very small alterations should be necessary from the Default Configuration <#defaultconfig>.

The syntax of the section headings here select by type, which corresponds to the type defined in the dialect. For example the following section applies to elements of the type comma, i.e. ,.

[sqlfluff:layout:type:comma]
spacing_before = touch
line_position = trailing

Within these configurable sections there are a few key elements which are available:

  • Spacing Elements: spacing_before, spacing_after and spacing_within. For each of these options, there are a few possible settings:
    • The default spacing for all elements is single unless otherwise specified. In this state, elements will be spaced with a single space character unless there is a line break between them.
    • The value of touch allows line breaks, but if no line break is present, then no space should be present. A great example of this is the spacing before commas (as shown in the config above), where line breaks may be allowed, but if not they should touch the element before.
    • Both of the above can be qualified with the :inline modifier - which prevents newlines within the segment. This is best illustrated by the spacing found in a qualified identifier like my_schema.my_table which uses touch:inline or other clauses where we want to force some elements to be on the same line.
  • Line Position: set using the line_position option. By default this is unset, which implies no particular line position requirements. The available options are:
    • trailing and leading, which are most common in the placement of commas. Both of these settings also allow the option of a comma on its own on a line, or in the middle of a line, but if there is a line break on either side then they make sure it's on the correct side. By default we assume trailing commas, but if you (or your organisation) have settled on leading commas then you should add the following section to your config:
[sqlfluff:layout:type:comma]
line_position = leading
  • alone, which means if there is a line break on either side, then there must be a line break on both sides (i.e. that it should be the only thing on that line.
  • All of the above options can be qualified with the :strict modifier - which prevents the inline case. For example:
-- Setting line_position to just `alone`
-- within [sqlfluff:layout:type:set_operator]
-- would not allow:
SELECT a
UNION SELECT b;
-- ...or...
SELECT a UNION
SELECT b;
-- but *would* allow both of the following:
SELECT a UNION SELECT b;
SELECT a
UNION
SELECT b;
-- However the default is set to `alone:strict`
-- then the *only* acceptable configuration is:
SELECT a
UNION
SELECT b;
  • Keyword Line Position — you set this with the keyword_line_position option. For most clauses it's unset by default, which implies no particular keyword line position requirements at all. Your available options are:
    • leading and alone — these are the two most common keyword placements, bestie. Both of these settings also allow the option of a keyword to end on a line. By default we assume leading WHERE keywords, but if you (or your organisation 👑) have settled on alone WHERE keywords, then you should add the following section to your config:
[sqlfluff:layout:type:where_clause]
keyword_line_position = alone
  • trailing — this means there should be a line break after the keyword. It's fairly uncommon ngl, but it may apply to the ON keyword after a join. If you (or your organisation) have settled on trailing ON keywords, then you should add the following section to your config:
[sqlfluff:layout:type:join_on_condition]
keyword_line_position = trailing
  • The keyword positioning is valid across a number of different clauses. For example, to apply the leading directive to the PARTITION BY clause you would add the following configuration:
[sqlfluff:layout:type:partitionby_clause]
keyword_line_position = leading
  • If you (or your organisation) would prefer to unset the Default Configuration of some options, you may clear it by setting the configuration to none.
[sqlfluff:layout:type:where_clause]
keyword_line_position = none

[sqlfluff:layout:type:orderby_clause]
keyword_line_position = none

[sqlfluff:layout:type:groupby_clause]
keyword_line_position = none

[sqlfluff:layout:type:having_clause]
keyword_line_position = none
  • Exclusions 🚫 — the keyword_line_position_exclusions option lets you exclude specific types of segments from the keyword_line_position rule. This is so useful when certain segments, such as window specifications or aggregate functions, should not follow the same keyword line position rules as other segments in the same clause.

For example, to exclude window specifications from the ORDER BY clause's keyword line position rule, you can configure it as follows:

[sqlfluff:layout:type:orderby_clause]
keyword_line_position = leading
keyword_line_position_exclusions = window_specification

This configuration ensures that the ORDER BY clause follows the leading rule, except for window specifications, which are allowed to remain inline. ✨

You can also specify multiple exclusions by separating them with commas:

[sqlfluff:layout:type:orderby_clause]
keyword_line_position = leading
keyword_line_position_exclusions = window_specification, aggregate_order_by

In this case, both window specifications and aggregate functions with ORDER BY clauses are excluded from the leading rule. 💅

Example Usage:

With the above configuration, the following SQL would pass:

SELECT
a,
b,
ROW_NUMBER() OVER (PARTITION BY c ORDER BY d) AS e,
STRING_AGG(a ORDER BY b, c)
FROM f
JOIN g
ON g.h = f.h

However, the following SQL would fail 😭 because the outer ORDER BY clause does not follow the leading rule:

SELECT
a,
b,
ROW_NUMBER() OVER (PARTITION BY c ORDER BY d) AS e,
STRING_AGG(a ORDER BY b, c)
FROM f
JOIN g
ON g.h = f.h ORDER BY a

The corrected version would be:

SELECT
a,
b,
ROW_NUMBER() OVER (PARTITION BY c ORDER BY d) AS e,
STRING_AGG(a ORDER BY b, c)
FROM f
JOIN g
ON g.h = f.h
ORDER BY a

🧚 Templating Configuration 💖

This section explains how to configure templating for SQL files. 🎀

When writing SQL files, users might utilise some kind of templating. The SQL file itself is written with placeholders which get rendered to proper SQL at run time. This can range from very simple placeholder templating to complex Jinja templating.

SQLFluff supports templated sections in SQL — see Stage 1, the templater. This is achieved by the following set of operations:

  1. SQLFluff pre-renders the templated SQL
  2. SQLFluff applies the lint and fix operations to the rendered file
  3. SQLFluff backports the rule violations to the templated section of the SQL

SQLFluff does not automatically have access to the same environment used in production template setup. This means it is necessary to either provide that environment or provide dummy values to effectively render the template and generate valid SQL. Refer to the templater sections below for details. 🫶

SQLFluff natively supports the following templating engines:

  • Jinja templater
  • Placeholder templater
  • Python templater

Also, SQLFluff has an integration to use dbt as a templater.

  • dbt templater (via plugin which is covered in a different section)
Note: Templaters may not be able to generate a rendered SQL that cover the entire raw file. For example, if the raw SQL uses a {% if condition %} block, the rendered version of the template will only include either the {% then %} or the {% else %} block (depending on the provided configuration for the templater), but not both. In this case, because SQLFluff linting can only operate on the output of the templater, some areas of the raw SQL will never be seen by the linter and will not be covered by lint rules. 💀 This is functionality we hope to support in future.

🪄 Jinja templater ✨

The Jinja templater uses Jinja2 (https://jinja.palletsprojects.com/) to render templates.

There are multiple, complementary ways of configuring the Jinja templater:

  • Reading variables and Jinja macros directly from the SQLFluff config file
  • Loading macros from a path
  • Using a library

📋 Overview of Jinja templater's configuration options 🎀

Here's the vibe check on what each method can and can't do — ✅ means yes, ❌ means nope:

  • Config file
    • Variables: ✅
    • Macros: ✅
    • Filters: ❌
    • Documentation: Complex Jinja Variable Templating and Jinja Macro Templating (from config)
  • Macro Path
    • Variables: ❌
    • Macros: ✅
    • Filters: ❌
    • Documentation: Jinja Macro Templating (from file)
  • Library
    • Variables: ✅
    • Macros: ✅
    • Filters: ✅
    • Documentation: Library Templating

For example, a snippet from a .sqlfluff file that uses all config options:

[sqlfluff]
templater = jinja

[sqlfluff:templater:jinja]
apply_dbt_builtins = True
load_macros_from_path = my_macros
loader_search_path = included_templates
library_path = sqlfluff_libs
exclude_macros_from_path = my_macros_exclude

[sqlfluff:templater:jinja:context]
my_list = ['a', 'b', 'c']
MY_LIST = ("d", "e", "f")
my_where_dict = {"field_1": 1, "field_2": 2}

[sqlfluff:templater:jinja:macros]
a_macro_def = {% macro my_macro(n) %}{{ n }} + {{ n * 2 }}{% endmacro %}

💜 Complex Jinja Variable Templating 🦋

Apart from the Generic variable templating that is supported for all templaters, two more advanced features of variable templating are available for Jinja:

case sensitivity and native python types. Both are illustrated in the following example:

[sqlfluff:templater:jinja:context]
my_list = ['a', 'b', 'c']
MY_LIST = ("d", "e", "f")
my_where_dict = {"field_1": 1, "field_2": 2}

SELECT
    {% for elem in MY_LIST %}
        '{{elem}}' {% if not loop.last %}||{% endif %}
    {% endfor %} as concatenated_list
FROM tbl
WHERE
    {% for field, value in my_where_dict.items() %}
        {{field}} = {{value}} {% if not loop.last %}and{% endif %}
    {% endfor %}

...will render as...

SELECT
    'd' || 'e' || 'f' as concatenated_list
FROM tbl
WHERE
    field_1 = 1 and field_2 = 2

Note that the variable was replaced in a case sensitive way and that the settings in the config file were interpreted as native python types. 🐍✨

Jinja Macro Templating (from config) 💅

ok so macros?? they're literally functions but make it jinja. bestie, you can only use them in the jinja templater, no cap. they work just like the Generic Variable Templating situation — you specify them in config files — except the naming hits different. same energy as the context section: macros get configured separately in the macros section of the config. look at this.

say you get passed this .sql file:

SELECT {{ my_macro(6) }} FROM some_table

...and this config in .sqlfluff sitting in the same directory (and yes, the whitespace is being controlled tightly):

[sqlfluff:templater:jinja:macros]
a_macro_def = {% macro my_macro(n) %}{{ n }} + {{ n * 2 }}{% endmacro %}

...then before parsing happens, the sql gets transformed into:

SELECT 6 + 12 FROM some_table

now — in that config block, the variable name is a_macro_def, and honestly it looks like it's not used anywhere else ever. and broadly that's accurate! but within the configuration loader it is still used, to overwrite previous values in other config files. which is where the whole idea of config blocks comes in: other config files downstream can selectively overwrite them if needed. iykyk ✨

Jinja Macro Templating (from file) 🎀

besides macros you declare in the config file, macros can also be loaded from files or folders. that's specified like this:

[sqlfluff:templater:jinja]
load_macros_from_path = my_macros,other_macros

load_macros_from_path is a comma-separated list of .sql files or folders. locations are relative to the config file. so if the config above was found at /home/my_project/.sqlfluff, SQLFluff will look for macros in the folders /home/my_project/my_macros/ and /home/my_project/other_macros/, including any of their subfolders. any macros defined in the config always take precedence over a macro defined in the path. periodt.

exclude_macros_from_path works in the same manner as load_macros_from_path but lets you have sqlfluff ignore certain macros. super useful if you have custom jinja tags.

macros loaded from these files are available in every .sql file without needing a Jinja include or import. they get loaded into the Jinja Global Namespace (https://jinja.palletsprojects.com/en/3.1.x/api/#global-namespace).

Note: the load_macros_from_path setting also defines the search path for Jinja include (https://jinja.palletsprojects.com/en/3.1.x/templates/#include) or import (https://jinja.palletsprojects.com/en/3.1.x/templates/#import). as with loaded macros, subdirectories are supported too. for example, if load_macros_from_path is set to my_macros, and there's a file my_macros/subdir/my_file.sql, you can do:
{% include 'subdir/my_file.sql' %}

if you'd like to define the Jinja search path without also loading the macros into the global namespace, use the loader_search_path setting instead. 💖

Note: throughout the templating process whitespace will still be treated rigorously, and this includes newlines. in particular you may choose to provide dummy macros in your configuration different from the actual macros used in production.
REMEMBER: the reason SQLFluff supports macros is to enable it to parse templated sql without it being a blocker. it shouldn't be a requirement that the templating is accurate — it only needs to work well enough that parsing and linting are helpful. 🙈

Builtin Jinja Macro Blocks 👑

one of the main use cases that inspired SQLFluff as a project was dbt (https://www.getdbt.com/). it uses jinja templating extensively and leads to some users maintaining big repositories of sql files which could potentially benefit from some linting.

Note: SQLFluff has now a tighter integration with dbt through the "dbt" templater. it's the recommended templater for dbt projects. if used, it eliminates the need for the overrides described in this section.
to use the dbt templater, go to dbt templater.

SQLFluff anticipates this use case and provides some built in macro blocks in the Default Configuration which assist in getting started with dbt (https://www.getdbt.com/) projects. in particular it provides mock objects for:

  • ref: the mock version of this provided simply returns the model reference as the name of the table. in most cases this is sufficient.
  • config: a regularly used macro in dbt (https://www.getdbt.com/) to set configuration values. for linting purposes, this makes no difference and so the provided macro simply returns nothing.
Note: if there are other builtin macros which would make your life easier, consider submitting the idea (or even better a pull request) on github (https://www.github.com/sqlfluff/sqlfluff).

Library Templating 🧚

if you're using SQLFluff with jinja as your templater, you may have library function calls within your sql files that can not be templated via the normal macro templating mechanisms:

SELECT foo, bar FROM baz {{ dbt_utils.group_by(2) }}

to template these libraries, you can use the sqlfluff:jinja:library_path config option:

[sqlfluff:templater:jinja]
library_path = sqlfluff_libs

this will pull in any python modules from that directory and allow sqlfluff to use them in templates. in the above example, you might define a file at sqlfluff_libs/dbt_utils.py as:

def group_by(n):
    return "GROUP BY 1,2"

if an __init__.py is detected, it will be loaded alongside any modules and submodules found within the library path.

SELECT
   {{ custom_sum('foo', 'bar') }},
   {{ foo.bar.another_sum('foo', 'bar') }}
FROM
   baz

sqlfluff_libs/__init__.py:

def custom_sum(a: str, b: str) -> str:
    return a + b

sqlfluff_libs/foo/__init__.py:

# empty file

sqlfluff_libs/foo/bar.py:

def another_sum(a: str, b: str) -> str:
   return a + b

additionally, the library can be used to expose Jinja Filters (https://jinja.palletsprojects.com/en/3.1.x/templates/#filters) to the Jinja environment used by SQLFluff.

this is achieved by setting a global variable named SQLFLUFF_JINJA_FILTERS. SQLFLUFF_JINJA_FILTERS is a dictionary where

  • dictionary keys map to the Jinja filter name
  • dictionary values map to the Python callable

for example, to make the Airflow filter ds available to SQLFluff, add the following to the __init__.py of the library:

# https://github.com/apache/airflow/blob/main/airflow/templates.py#L53
def ds_filter(value: datetime.date | datetime.time | None) -> str | None:
   """Date filter."""
   if value is None:
       return None
   return value.strftime("%Y-%m-%d")

SQLFLUFF_JINJA_FILTERS = {"ds": ds_filter}

now, ds can be used in SQL

SELECT "{{ "2000-01-01" | ds }}";

Jinja loader search path 🌈

the Jinja environment can be configured to search for files included with include (https://jinja.palletsprojects.com/en/3.1.x/templates/#include) or import (https://jinja.palletsprojects.com/en/3.1.x/templates/#import) in a list of folders. this is specified in the config file:

[sqlfluff:templater:jinja]
loader_search_path = included_templates,other_templates

loader_search_path is a comma-separated list of folders. locations are relative to the config file. for example, if the config file above was found at /home/my_project/.sqlfluff, then SQLFluff will look for included files in the folders /home/my_project/included_templates/ and /home/my_project/other_templates/, including any of their subfolders. for example, this will read from /home/my_project/included_templates/my_template.sql:

{% include 'included_templates/my_template.sql' %}

any folders specified in the load_macros_from_path setting are automatically appended to the loader_search_path. it is not necessary to specify a given directory in both settings. 💅

unlike the load_macros_from_path setting, any macros within these folders are not automatically loaded into the global namespace. they must be explicitly imported using the import (https://jinja.palletsprojects.com/en/3.1.x/templates/#import) Jinja directive. if you would like macros to be automatically included in the global Jinja namespace, use the load_macros_from_path setting instead.

Interaction with --ignore=templating 💀

ignoring Jinja templating errors provides a way for users to use SQLFluff while reducing or avoiding the need to spend a lot of time adding variables to [sqlfluff:templater:jinja:context].

when --ignore=templating is enabled, the Jinja templater behaves a bit differently. this additional behaviour is usually but not always helpful for making the file at least partially parsable and fixable. it definitely doesn't guarantee that every file can be fixed, but it's proven useful for some users.

here's how it works:

  • within the expanded SQL, undefined variables are automatically replaced with the corresponding string value.
  • if you do {% include query %}, and the variable query is not defined, it returns a "file" containing the string query.
  • if you do {% include "query_file.sql" %}, and that file does not exist or you haven't configured a setting for load_macros_from_path or loader_search_path, it returns a "file" containing the text query_file.

for example:

select {{ my_variable }}
from {% include "my_table.sql" %}

is interpreted as:

select my_variable
from my_table

the values provided by the Jinja templater act a bit (not exactly) like a mixture of several types:

because the values behave like Undefined, it's possible to replace them using Jinja's default() filter (https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.default). for example:

select {{ my_variable | default("col_a") }}
from my_table

is interpreted as:

select col_a
from my_table

Placeholder templater 🍓

libraries such as SQLAlchemy or Psycopg use different parameter placeholder styles to mark where a parameter has to be inserted in the query.

for example a query in SQLAlchemy can look like this:

SELECT * FROM table WHERE id = :myid

at runtime :myid will be replaced by a value provided by the application and escaped as needed, but this is not standard SQL and cannot be parsed as is.

so in order to parse these queries it's then necessary to replace these placeholders with sample values, and that's exactly what the placeholder templater does. 💜

Placeholder templating can be switched on in the config using:

[sqlfluff]
templater = placeholder

A few common styles are supported:

-- colon
WHERE bla = :my_name

-- colon_nospaces
-- (use with caution as more prone to false positives)
WHERE bla = table:my_name

-- colon_optional_quotes
SELECT :"column" FROM :table WHERE bla = :'my_name'

-- numeric_colon
WHERE bla = :2

-- pyformat
WHERE bla = %(my_name)s

-- dollar
WHERE bla = $my_name or WHERE bla = ${my_name}

-- dollar_surround (DbUp compatible variable)
WHERE bla = $my_name$

-- question_mark
WHERE bla = ?

-- numeric_dollar
WHERE bla = $3 or WHERE bla = ${3}

-- percent
WHERE bla = %s

-- ampersand
WHERE bla = &s or WHERE bla = &{s} or USE DATABASE MARK_{ENV}

These get configured by setting param_style to one of the names above, bestie:

[sqlfluff:templater:placeholder]
param_style = colon
my_name = 'john'

...and then you can set sample values for each parameter, like my_name up there ⬆️. Heads up tho: that value needs to be escaped because it gets replaced as a string during parsing. And when you don't hand over sample values? The templater just uses the parameter names themselves by default. iconic behaviour honestly 💅

When parameters are positional, like question_mark, their name is simply the order they show up in, starting at 1.

[sqlfluff:templater:placeholder]
param_style = question_mark
1 = 'john'

And if you need a param style that's not on the list? You can pass a custom regex, no cap:

[sqlfluff:templater:placeholder]
param_regex = __(?P<param_name>[\w_]+)__
my_name = 'john'
N.B. quotes around param_regex in the config are interpreted literally by the templater. e.g. param_regex='__(?P<param_name>[w_]+)__' matches '__some_param__' not __some_param__ 😭

The named parameter param_name will be used as the key to replace, and if it's missing, the parameter is assumed to be positional and numbers are used instead.

Also — lowkey consider making a pull request to the project to have your style added 🫶 it may be useful to other people and simplify your configuration. main character energy fr fr.

🐍 Python templater

Uses native Python f-strings. As described in Generic Variable Templating, an example usage would look be configured as follows:

If passed the following .sql file:

SELECT * FROM {tbl_name}

...and the following configuration in .sqlfluff in the same directory:

[sqlfluff]
templater = python

[sqlfluff:templater:python:context]
tbl_name = my_table

...then before parsing, the sql will be transformed to:

SELECT * FROM my_table

🧚 Complex Python Variable Templating

Python string formatting supports accessing object attributes via dot notation (e.g. {foo.bar}). However, since we cannot create Python objects within configuration files, we need a workaround in order to provide dummy values to render templates containing these values. The SQLFluff python templater will interpret any variable containing a "." as a dictionary lookup on the magic fixed context key sqlfluff.

-- this SQL
SELECT * FROM {foo.bar}

-- becomes this
SELECT * FROM {sqlfluff["foo.bar"]}

...which can be populated using the following configuration:

[sqlfluff:templater:python:context]
sqlfluff = {"foo.bar": "abc"}

🌸 dbt templater

Note: From sqlfluff version 0.7.0 onwards, the dbt templater has been moved to a separate plugin and python package. Projects that were already using the dbt templater may initially fail after an upgrade to 0.7.0+. See the installation instructions below to install the dbt templater. dbt templating is still a relatively new feature added in 0.4.0 and is still in very active development! If you encounter an issue, please let us know in a GitHub issue or on the SQLFluff slack workspace.

dbt is not the default templater for SQLFluff (it is jinja). dbt is a complex tool, so using the default jinja templater will be simpler. You should be aware when using the dbt templater that you will be exposed to some of the complexity of dbt. Users may wish to try both templaters and choose according to how they intend to use SQLFluff.

A simple rule of thumb might be:

  • If you are using SQLFluff in a CI/CD context, where speed is not critical but accuracy in rendering sql is, then the dbt templater may be more appropriate.
  • If you are using SQLFluff in an IDE or on a git hook, where speed of response may be more important, then the jinja templater may be more appropriate.

Pros:

  • Most (potentially all) macros will work

Cons:

  • More complex, e.g. using it successfully may require deeper understanding of your models and/or macros (including third-party macros)
    • More configuration decisions to make
    • Best practices are not yet established or documented
  • If your dbt model files access a database at compile time, using SQLFluff with the dbt templater will also require access to a database.
    • Note that you can often point SQLFluff and the dbt templater at a test database (i.e. it doesn't have to be the production database).
  • Runs slower

💖 Installation & Configuration

In order to get started using SQLFluff with a dbt project you will first need to install the relevant dbt adapter for your dialect and the sqlfluff-templater-dbt package using your package manager of choice (e.g. pip install dbt-postgres sqlfluff-templater-dbt) and then will need the following configuration:

In .sqlfluff:

[sqlfluff]
templater = dbt

In .sqlfluffignore:

target/
# dbt <1.0.0
dbt_modules/
# dbt >=1.0.0
dbt_packages/
macros/

You can set the dbt project directory, profiles directory and profile with:

[sqlfluff:templater:dbt]
project_dir = <relative or absolute path to dbt_project directory, can be overridden by env var DBT_PROJECT_DIR>
profiles_dir = <relative or absolute path to the directory that contains the profiles.yml file, can be overridden by env var DBT_PROFILES_DIR>
profile = <dbt profile>
target = <dbt target>
dbt_skip_compilation_error = <True or False, default is True>
Note: If the profiles_dir setting is omitted, SQLFluff will look for the profile in the default location, which varies by operating system. On Unix-like operating systems (e.g. Linux or macOS), the default profile directory is ~/.dbt/. On Windows, you can determine your default profile directory by running dbt debug --config-dir.
Note: A fatal error can be raised at compile time. That can sometimes happen for SQLFluff related reasons (it used to happen if we tried to compile ephemeral models in the wrong order), but more often because a macro tries to query a table at compile time which doesn't exist. By default, dbt_skip_compilation_error parameter is set to True, that's why such errors will be ignored. However if you want to see them, you can set it to False and SQLFluff will raise a fatal error. 💀

To use builtin dbt Jinja functions SQLFluff provides a configuration option that enables usage within templates.

[sqlfluff:templater:jinja]
apply_dbt_builtins = True

This will provide dbt macros like ref, var, is_incremental(). If the need arises builtin dbt macros can be customised via Jinja macros in .sqlfluff configuration file.

[sqlfluff:templater:jinja:macros]
# Macros provided as builtins for dbt projects
dbt_ref = {% macro ref(model_ref) %}{{model_ref}}{% endmacro %}
dbt_source = {% macro source(source_name, table) %}{{source_name}}_{{table}}{% endmacro %}
dbt_config = {% macro config() %}{% for k in kwargs %}{% endfor %}{% endmacro %}
dbt_var = {% macro var(variable, default='') %}item{% endmacro %}
dbt_is_incremental = {% macro is_incremental() %}True{% endmacro %}

If your project requires that you pass variables to dbt through command line, you can specify them in template:dbt:context section of .sqlfluff. See below configuration and its equivalent dbt command:

[sqlfluff:templater:dbt:context]
my_variable = 1
dbt run --vars '{"my_variable": 1}'

🌈 Known Caveats

  • To use the dbt templater, you must set templater = dbt in the .sqlfluff config file in the directory where sqlfluff is run. The templater cannot be changed in .sqlfluff files in subdirectories.
  • In SQLFluff 0.4.0 using the dbt templater requires that all files within the root and child directories of the dbt project must be part of the project. If there are deployment scripts which refer to SQL files not part of the project for instance, this will result in an error. You can overcome this by adding any non-dbt project SQL files to .sqlfluffignore.

🦋 Generic Variable Templating

Variables are available in all the templaters. By default the templating engine will expect variables for templating to be available in the config, and the templater will be look in the section corresponding to the context for that templater. By convention, the config for the jinja templater is found in the sqlfluff:templater:jinja:context section, the config for the python templater is found in the sqlfluff:templater:python:context section, the one for the placeholder templater is found in the sqlfluff:templater:placeholder:context section.

For example, if passed the following .sql file:

SELECT {{ num_things }} FROM {{ tbl_name }} WHERE id > 10 LIMIT 5

...and the following configuration in .sqlfluff in the same directory:

[sqlfluff:templater:jinja:context]
num_things=456
tbl_name=my_table

...then before parsing, the sql will be transformed to:

SELECT 456 FROM my_table WHERE id > 10 LIMIT 5
Note: If there are variables in the template which cannot be found in the current configuration context, then this will raise a SQLTemplatingError and this will appear as a violation without a line number, quoting the name of the variable that couldn't be found. 😭

🙈 Ignoring Errors & Files

✨ Ignoring individual lines

Similar to flake8's ignore, individual lines can be ignored by adding -- noqa to the end of the line. Additionally, specific rules can be ignored by quoting their code or the category.

-- Ignore all errors
SeLeCt  1 from tBl ;    -- noqa

...and if you wanna be extra specific about which rules you're clowning on, just name them:

-- Ignore rule CP02 & rule CP03
SeLeCt  1 from tBl ;    -- noqa: CP02,CP03

-- Ignore all parsing errors
SeLeCt from tBl ;       -- noqa: PRS
bestie, real talk: ignoring TMP and PRS errors can lead to incorrect sqlfluff lint and sqfluff fix results, because SQLFluff can straight-up misinterpret the SQL it's analysing. so like… do it, but know what you're doing. 💀

🚧 Ignoring line ranges 🚧

Okay so this one's giving very much like pylint's pylint directive — you can ignore whole ranges of lines by dropping -- noqa:disable=<rule>[,...] | all on a line. From that point on, the rules you named (or all the rules, if you said all) get ignored until you hit a matching -- noqa:enable=<rule>[,...] | all directive. it's like turning the rules off and back on again, and honestly? iconic behavior.

-- Ignore rule AL02 from this line forward
SELECT col_a a FROM foo -- noqa: disable=AL02

-- Ignore all rules from this line forward
SELECT col_a a FROM foo -- noqa: disable=all

-- Enforce all rules from this line forward
SELECT col_a a FROM foo -- noqa: enable=all

🙈 .sqlfluffignore 🙈

So you know how Git has .gitignore and Docker has .dockerignore? same energy. SQLFluff supports a .sqlfluffignore file to control which files get linted and which ones get to just… vibe. Under the hood we use the python pathspec library, which also has a lil tutorial in their documentation if you wanna read up. ✨

An example of a potential .sqlfluffignore placed in the root of your project would be:

# Comments start with a hash.

# Ignore anything in the "temp" path
/temp/

# Ignore anything called "testing.sql"
testing.sql

# Ignore any ".tsql" files
*.tsql

Also! Ignore files can be placed in subdirectories of a path that's being linted, and the sub files will also be applied within that subdirectory. nesting, but make it legal. 🦋

💅 Ignoring types of errors 💅

Whole general categories of errors can be ignored using the --ignore command line option, or the ignore setting in .sqlfluffignore. the categories that can catch these hands are:

  • lexing
  • linting
  • parsing
  • templating

👑 Default Configuration 👑

The default configuration is as follows — and heads up, note the Builtin Jinja Macro Blocks in section [sqlfluff:templater:jinja:macros] as referred to above. 👀

Note: This shows the entire default config. We do not recommend that users copy this whole config as the starter config file for their project.
This is for two reasons:
1. The config file should act as a form of documentation for your team. A record of what decisions you've made which govern how you format your sql. By having a more concise config file, and only defining config settings where they differ from the defaults — you are more clearly stating to your team what choices you've made.
2. As the project evolves, the structure of the config file may change and we will attempt to make changes as backward compatible as possible. If you have not overridden a config setting in your project, we can easily update the default config to match your expected behaviour over time. We may also find issues with the default config which we can also fix in the background. However, the longer your local config file, the more work it will be to update and migrate your config file between major versions.
If you are starting a fresh project and are looking for a good starter config, check out the New Project Configuration section above.
[sqlfluff]
# verbose is an integer (0-2) indicating the level of log output
verbose = 0
# Turn off color formatting of output
nocolor = None
# Supported dialects https://docs.sqlfluff.com/en/stable/perma/dialects.html
# Or run 'sqlfluff dialects'
dialect = None
# One of [raw|jinja|python|placeholder]
templater = jinja
# Comma separated list of rules to check, default to all
rules = all
# Comma separated list of rules to exclude, or None
exclude_rules = None
# Below controls SQLFluff output, see max_line_length for SQL output
output_line_length = 80
# Number of passes to run before admitting defeat
runaway_limit = 10
# Ignore errors by category (one or more of the following, separated by commas: lexing,linting,parsing,templating)
ignore = None
# Warn only for rule codes (one of more rule codes, separated by commas: e.g. LT01,LT02)
# Also works for templating and parsing errors by using TMP or PRS
warnings = None
# Whether to warn about unneeded '-- noqa:' comments.
warn_unused_ignores = False
# Ignore linting errors found within sections of code coming directly from
# templated code (e.g. from within Jinja curly braces. Note that it does not
# ignore errors from literal code found within template loops.
ignore_templated_areas = True
# can either be autodetect or a valid encoding e.g. utf-8, utf-8-sig
encoding = autodetect
# Ignore inline overrides (e.g. to test if still required)
disable_noqa = False
# Ignore inline overrides except those listed.
# This take priority over `disabled_noqa`.
disable_noqa_except = None
# Comma separated list of file extensions to lint
# NB: This config will only apply in the root folder
sql_file_exts = .sql,.sql.j2,.dml,.ddl
# Allow fix to run on files, even if they contain parsing errors
# Note altering this is NOT RECOMMENDED as can corrupt SQL
fix_even_unparsable = False
# Very large files can make the parser effectively hang.
# The more efficient check is the _byte_ limit check which
# is enabled by default. The previous _character_ limit check
# is still present for backward compatibility. This will be
# removed in a future version.
# Set either to 0 to disable.
large_file_skip_char_limit = 0
large_file_skip_byte_limit = 20000
# CPU processes to use while linting.
# If positive, just implies number of processes.
# If negative or zero, implies number_of_cpus - specified_number.
# e.g. -1 means use all processors but one. 0  means all cpus.
processes = 1
# Max line length is set by default to be in line with the dbt style guide.
# https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md
# Set to zero or negative to disable checks.
max_line_length = 80
# NOTE: Templater Variant rendering should currently be considered EXPERIMENTAL.
# Only set `render_variant_limit` to more than 1 if you know what you're doing!
# Implementation of this will also depend on your templater.
render_variant_limit = 1

[sqlfluff:indentation]
# See https://docs.sqlfluff.com/en/stable/perma/indent_locations.html
indent_unit = space
tab_space_size = 4
indented_joins = False
indented_ctes = False
indented_using_on = True
indented_on_contents = True
indented_then = True
indented_then_contents = True
allow_implicit_indents = False
template_blocks_indent = True
# This is a comma separated list of elements to skip
# indentation edits to.
skip_indentation_in = script_content
# If comments are found at the end of long lines, we default to moving
# them to the line _before_ their current location as the convention is
# that a comment precedes the line it describes. However if you prefer
# comments moved _after_, this configuration setting can be set to "after".
trailing_comments = before
# To exclude comment lines from indentation entirely set this to "True".
ignore_comment_lines = False

# Layout configuration
# See https://docs.sqlfluff.com/en/stable/perma/layout_spacing.html
[sqlfluff:layout:type:comma]
spacing_before = touch
line_position = trailing

[sqlfluff:layout:type:binary_operator]
spacing_within = touch
line_position = leading

[sqlfluff:layout:type:column_path_operator]
spacing_within = touch
line_position = leading

[sqlfluff:layout:type:statement_terminator]
spacing_before = touch
line_position = trailing

[sqlfluff:layout:type:end_of_file]
spacing_before = touch

[sqlfluff:layout:type:set_operator]
line_position = alone:strict

[sqlfluff:layout:type:start_bracket]
spacing_after = touch

[sqlfluff:layout:type:end_bracket]
spacing_before = touch

[sqlfluff:layout:type:start_square_bracket]
spacing_after = touch

[sqlfluff:layout:type:end_square_bracket]
spacing_before = touch

[sqlfluff:layout:type:start_angle_bracket]
spacing_after = touch

[sqlfluff:layout:type:end_angle_bracket]
spacing_before = touch

[sqlfluff:layout:type:casting_operator]
spacing_before = touch
spacing_after = touch:inline

[sqlfluff:layout:type:slice]
spacing_before = touch
spacing_after = touch

[sqlfluff:layout:type:dot]
spacing_before = touch
spacing_after = touch

[sqlfluff:layout:type:comparison_operator]
spacing_within = touch
line_position = leading

[sqlfluff:layout:type:assignment_operator]
spacing_within = touch
line_position = leading

[sqlfluff:layout:type:object_reference]
spacing_within = touch:inline

[sqlfluff:layout:type:numeric_literal]
spacing_within = touch:inline

[sqlfluff:layout:type:sign_indicator]
spacing_after = touch:inline

[sqlfluff:layout:type:tilde]
spacing_after = touch:inline

[sqlfluff:layout:type:function_name]
spacing_within = touch:inline

[sqlfluff:layout:type:function_contents]
spacing_before = touch:inline

[sqlfluff:layout:type:function_parameter_list]
spacing_before = touch:inline

[sqlfluff:layout:type:array_type]
spacing_within = touch:inline

[sqlfluff:layout:type:typed_array_literal]
spacing_within = touch

[sqlfluff:layout:type:sized_array_type]
spacing_within = touch

[sqlfluff:layout:type:struct_type]
spacing_within = touch:inline

[sqlfluff:layout:type:bracketed_arguments]
spacing_before = touch:inline

[sqlfluff:layout:type:match_condition]
spacing_within = touch:inline

[sqlfluff:layout:type:typed_struct_literal]
spacing_within = touch

[sqlfluff:layout:type:semi_structured_expression]
spacing_within = touch:inline
spacing_before = touch:inline

[sqlfluff:layout:type:array_accessor]
spacing_before = touch:inline
          [sqlfluff:layout:type:colon]
          spacing_before = touch

          [sqlfluff:layout:type:colon_delimiter]
          spacing_before = touch
          spacing_after = touch

          [sqlfluff:layout:type:path_segment]
          spacing_within = touch

          [sqlfluff:layout:type:sql_conf_option]
          spacing_within = touch

          [sqlfluff:layout:type:sqlcmd_operator]
          # NOTE: This is the spacing between the operator and the colon
          spacing_before = touch

          [sqlfluff:layout:type:slash]
          spacing_before = any
          spacing_after = any

          [sqlfluff:layout:type:comment]
          spacing_before = any
          spacing_after = any

          [sqlfluff:layout:type:pattern_expression]
          # Snowflake pattern expressions shouldn't have their spacing changed.
          spacing_within = any

          [sqlfluff:layout:type:bind_variable]
          spacing_within = touch

          [sqlfluff:layout:type:placeholder]
          # Placeholders exist "outside" the rendered SQL syntax
          # so we shouldn't enforce any particular spacing around
          # them.
          spacing_before = any
          spacing_after = any

          [sqlfluff:layout:type:common_table_expression]
          # The definition part of a CTE should fit on one line where possible.
          # For users which regularly define column names in their CTEs they
          # may which to relax this config to just `single`.
          spacing_within = single:inline

          # By setting a selection of clauses to "alone", we hint to the reflow
          # algorithm that in the case of a long single line statement, the
          # first place to add newlines would be around these clauses.
          # Setting this to "alone:strict" would always _force_ line breaks
          # around them even if the line isn't too long.
          [sqlfluff:layout:type:select_clause]
          line_position = alone

          [sqlfluff:layout:type:where_clause]
          line_position = alone
          keyword_line_position = leading
          keyword_line_position_exclusions = pipe_operator_clause

          [sqlfluff:layout:type:from_clause]
          line_position = alone

          [sqlfluff:layout:type:join_clause]
          line_position = alone

          [sqlfluff:layout:type:groupby_clause]
          line_position = alone
          keyword_line_position = leading

          [sqlfluff:layout:type:orderby_clause]
          # NOTE: Order by clauses appear in many places other than in a select
          # clause. To avoid unexpected behaviour we use `leading` in this
          # case rather than `alone`.
          line_position = leading
          keyword_line_position = leading
          keyword_line_position_exclusions =
              window_specification,
              aggregate_order_by,
              withingroup_clause,
              pipe_operator_clause

          [sqlfluff:layout:type:having_clause]
          line_position = alone
          keyword_line_position = leading

          [sqlfluff:layout:type:limit_clause]
          line_position = alone

          # Template loop tokens shouldn't dictate spacing around them.
          [sqlfluff:layout:type:template_loop]
          spacing_before = any
          spacing_after = any

          [sqlfluff:templater]
          unwrap_wrapped_queries = True

          [sqlfluff:templater:jinja]
          apply_dbt_builtins = True

          # Some rules can be configured directly from the config common to other rules
          [sqlfluff:rules]
          allow_scalar = True
          single_table_references = consistent
          unquoted_identifiers_policy = all

          [sqlfluff:rules:capitalisation.keywords]
          # Keywords
          capitalisation_policy = consistent
          # Comma separated list of words to ignore for this rule
          ignore_words = None
          ignore_words_regex = None

          [sqlfluff:rules:capitalisation.identifiers]
          # Unquoted identifiers
          extended_capitalisation_policy = consistent
          # Comma separated list of words to ignore for this rule
          ignore_words = None
          ignore_words_regex = None

          [sqlfluff:rules:capitalisation.functions]
          # Function names
          extended_capitalisation_policy = consistent
          # Comma separated list of words to ignore for this rule
          ignore_words = None
          ignore_words_regex = None

          [sqlfluff:rules:capitalisation.literals]
          # Null & Boolean Literals
          capitalisation_policy = consistent
          # Comma separated list of words to ignore for this rule
          ignore_words = None
          ignore_words_regex = None

          [sqlfluff:rules:capitalisation.types]
          # Data Types
          extended_capitalisation_policy = consistent
          # Comma separated list of words to ignore for this rule
          ignore_words = None
          ignore_words_regex = None

          [sqlfluff:rules:ambiguous.join]
          # Fully qualify JOIN clause
          fully_qualify_join_types = inner

          [sqlfluff:rules:ambiguous.column_references]
          # GROUP BY/ORDER BY column references
          group_by_and_order_by_style = consistent

          [sqlfluff:rules:aliasing.table]
          # Aliasing preference for tables
          aliasing = explicit

          [sqlfluff:rules:aliasing.column]
          # Aliasing preference for columns
          aliasing = explicit

          [sqlfluff:rules:aliasing.unused]
          alias_case_check = dialect

          [sqlfluff:rules:aliasing.length]
          min_alias_length = None
          max_alias_length = None

          [sqlfluff:rules:aliasing.forbid]
          # Avoid table aliases in from clauses and join conditions.
          # Disabled by default for all dialects unless explicitly enabled.
          # We suggest instead using aliasing.length (AL06) in most cases.
          force_enable = False

          [sqlfluff:rules:convention.not_equal]
          # Consistent usage of preferred "not equal to" comparison
          preferred_not_equal_style = consistent

          [sqlfluff:rules:convention.select_trailing_comma]
          # Trailing commas
          select_clause_trailing_comma = forbid

          [sqlfluff:rules:convention.count_rows]
          # Consistent syntax to count all rows
          prefer_count_1 = False
          prefer_count_0 = False

          [sqlfluff:rules:convention.terminator]
          # Semi-colon formatting approach
          multiline_newline = False
          require_final_semicolon = False

          [sqlfluff:rules:convention.blocked_words]
          # Comma separated list of blocked words that should not be used
          blocked_words = None
          blocked_regex = None
          match_source = False

          [sqlfluff:rules:convention.quoted_literals]
          # Consistent usage of preferred quotes for quoted literals
          preferred_quoted_literal_style = consistent
          # Disabled for dialects that do not support single and double quotes for quoted literals (e.g. Postgres)
          force_enable = False

          [sqlfluff:rules:convention.casting_style]
          # SQL type casting
          preferred_type_casting_style = consistent

          [sqlfluff:rules:references.from]
          # References must be in FROM clause
          # Disabled for some dialects (e.g. bigquery)
          force_enable = False

          [sqlfluff:rules:references.qualification]
          # Comma separated list of words to ignore for this rule
          ignore_words = None
          ignore_words_regex = None
          subqueries_ignore_external_references = False

          [sqlfluff:rules:references.consistent]
          # References must be consistently used
          # Disabled for some dialects (e.g. bigquery)
          force_enable = False

          [sqlfluff:rules:references.keywords]
          # Keywords should not be used as identifiers.
          unquoted_identifiers_policy = aliases
          quoted_identifiers_policy = none
          # Comma separated list of words to ignore for this rule
          ignore_words = None
          ignore_words_regex = None

          [sqlfluff:rules:references.special_chars]
          # Special characters in identifiers
          unquoted_identifiers_policy = all
          quoted_identifiers_policy = all
          allow_space_in_identifier = False
          additional_allowed_characters = None
          ignore_words = None
          ignore_words_regex = None

          [sqlfluff:rules:references.quoting]
          # Policy on quoted and unquoted identifiers
          prefer_quoted_identifiers = False
          prefer_quoted_keywords = False
          ignore_words = None
          ignore_words_regex = None
          case_sensitive = True

          [sqlfluff:rules:layout.long_lines]
          # Line length
          ignore_comment_lines = False
          ignore_comment_clauses = False

          [sqlfluff:rules:layout.newlines]
          maximum_empty_lines_between_statements = 2
          maximum_empty_lines_inside_statements = 1

          [sqlfluff:rules:layout.select_targets]
          wildcard_policy = single

          [sqlfluff:rules:structure.subquery]
          # By default, allow subqueries in from clauses, but not join clauses
          forbid_subquery_in = join

          [sqlfluff:rules:structure.join_condition_order]
          preferred_first_table_in_join_clause = earlier

💖 Production Usage & Security 💖

okay bestie, so here's the vibe: SQLFluff isn't just a little toy you run in your terminal for funsies. It's built to be a dev utility and a full-on member of your CI/CD pipeline (https://en.wikipedia.org/wiki/Continuous_integration). Two jobs, one icon 💅

🛡️ Security Considerations 🛡️

First things first, the receipts: a full list of Security Advisories lives on GitHub at https://github.com/sqlfluff/sqlfluff/security/advisories. Go read it, fr fr. 👑

So here's the thing — given how SQLFluff is designed to be used, there are three different tiers of access that users might have, and each one can change how the tool behaves in a secure environment. Let's break it down. ✨

  1. Users may have edit access to the SQL code which is being linted. Now, lowkey important: while SQLFluff does not execute the SQL itself, in the process of the templating step (in particular via jinja or dbt), certain macros may have the ability to execute arbitrary SQL code (e.g. the dbt run_query macro at https://docs.getdbt.com/reference/dbt-jinja-functions/run_query). ⚠️ that's a real thing, not a drill. For the Jinja templater, SQLFluff uses the Jinja2 SandboxedEnvironment (https://jinja.palletsprojects.com/en/3.0.x/sandbox/#jinja2.sandbox.SandboxedEnvironment) to limit the execution on unsafe code. When looking to further secure this situation, see below for ways to limit the ability of users to import other libraries. 🧚
  2. Users may have edit access to the SQLFluff Configuration Files. In some (perhaps, many) environments, the users who can edit SQL files may also be able to access and edit the Configuration Files. And like… it's important to note that because of In-File Configuration Directives, users who can edit SQL files which are designed to be linted will also have access to the vast majority of any configuration options available in Configuration Files. so yeah, that's a whole thing. This means that there is minimal additional protection from restricting access to Configuration Files for users who already have access to edit the linting target files (as described above). 💀
tl;dr bestie: locking down the config files doesn't buy you much if the same person can already edit the SQL you're linting. Know your tiers. 🫶
  1. Users may have access to change how SQLFluff gets invoked. SQLFluff can be called either as a command line tool or through the python API. Normally the method is fixed for a given application. When you're thinking about how to restrict users' ability to call insecure code, SQLFluff aims to give you options right at the point of invocation. In particular, like we said above, the primary risk vector for SQLFluff is the macro environment, as described in Templating Configuration <#templateconfig>. To stop users from bringing arbitrary python methods into sqlfluff via the library_path configuration value (see Library Templating <#jinja-library-templating>), we recommend that for secure environments you override this config value either by handing an override option to the FluffConfig object if you're using the Python API, or via the --library-path CLI option:

To disable this option entirely via the CLI:

$ sqlfluff lint my_path --library-path none

To disable this option entirely via the python API:

"""This is an example of providing config overrides."""

from sqlfluff.core import FluffConfig, Linter

sql = "SELECT 1\n"

config = FluffConfig(
    overrides={
        "dialect": "snowflake",
        # NOTE: We explicitly set the string "none" here rather
        # than a None literal so that it overrides any config
        # set by any config files in the path.
        "library_path": "none",
    }
)

linted_file = Linter(config=config).lint_string(sql)

assert linted_file.get_violations() == []

✨ Using SQLFluff directly as a CLI application ✨

The SQLFluff CLI application <#cliref> is a python application, which means it depends on your host python environment (see Installing SQLFluff <#installingsqlfluff>).

The exit code <https://shapeshed.com/unix-exit-codes/> that SQLFluff hands you when it runs as a command line utility is designed to be useful in deployment pipelines. If no violations are found then the exit code <https://shapeshed.com/unix-exit-codes/> will be 0. If violations are found then a non-zero code gets returned, and you can interrogate it to find out more. 💅

  • An error code of 0 means operation success, no issues found. 🎉
  • An error code of 1 means operation success, issues found. For example this might mean that a linting issue was found, or that one file could not be parsed.
  • An error code of 2 means an error occurred and the operation could not be completed. For example a configuration issue or an internal error within SQLFluff. 😭

For details of what commands and options are available in the CLI, see the CLI Reference <#cliref>.

🌸 Using SQLFluff on changes using diff-quality 🌸

For projects with large amounts of (potentially imperfect) SQL code, the full SQLFluff output could be very large, which can be distracting — like, maybe the CI build for a one-line SQL change shouldn't be pressuring the developer to fix lots of unrelated quality issues, bestie.

To support this use case, SQLFluff integrates with a quality checking tool called diff-quality. By running SQLFluff using diff-quality (instead of running it directly), you can limit the output to the new or modified SQL in the branch (aka pull request or PR) containing the proposed changes. 👑

Currently, diff-quality requires that you are using git for version control.

NOTE: Installing SQLFluff automatically installs the diff_cover package that provides the diff-quality tool. 🫶

🎀 Adding diff-quality to your builds 🎀

In your CI build script:

  1. Set the current working directory to the git repository containing the SQL code to be checked.
  2. Run diff-quality, specifying SQLFluff as the underlying tool:
$ diff-quality --violations sqlfluff

The output will look something like:

-------------
Diff Quality
Quality Report: sqlfluff
Diff: origin/master...HEAD, staged and unstaged changes
-------------
sql/audience_size_queries/constraints/_postcondition_check_gdpr_compliance.sql (0.0%):
sql/audience_size_queries/constraints/_postcondition_check_gdpr_compliance.sql:5: Unquoted Identifiers must be consistently upper case.
-------------
Total:   1 line
Violations: 1 line
% Quality: 0%
-------------

These messages are basically the same as those provided directly by SQLFluff, although the format is a little different. Note that diff-quality only lists the line _numbers_, not the character position. If you need the character position, you will need to run SQLFluff directly. 💀

Note:

When using diff-quality with .sqlfluff Configuration Files <#config-files>, and especially together with the dbt templater <#dbt-templater>, it can be really easy to run into issues with file discovery. There are a few steps you can take to make it much less likely that this will happen:

  1. diff-quality needs to be run from the root of your git repository (so that it can find the git metadata).
  2. SQLFluff works best if the bulk of the configuration is done from a single .sqlfluff file, which should be in the root of your git repository.
  3. If using dbt templater <#dbt-templater>, then either place your dbt_project.yml file in the same root folder, or if you put it in a subfolder, then only invoke diff-quality and sqlfluff from the root and define the subfolder that the dbt project lives in using the .sqlfluff config file.

By aligning the paths of all three, you should be able to achieve a robust setup. If each is rooted in different paths it can be very difficult to achieve the same result, and the resulting behaviour can be difficult to debug. 🙈

To debug any issues relating to this setup, we recommend occasionally running sqlfluff directly using the main cli (i.e. calling sqlfluff lint my/project/path) and check whether that route gives you the results you expect. diff-quality should behave as though it's calling the SQLFluff CLI from the same path that you invoke diff-quality.

For more information on diff-quality and the diff_cover package, see the documentation <https://github.com/Bachmann1234/diff_cover> on their github repository. It covers topics such as:

  • Generating HTML reports
  • Controlling which branch to compare against (i.e. to determine new/changed lines). The default is origin/main.
  • Configuring diff-quality to return an error code if the quality is too low.
  • Troubleshooting

💖 Using pre-commit 💖

pre-commit <https://pre-commit.com/> is a framework to manage git "hooks" triggered right before a commit is made.

A git hook <https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks> is a git feature to "fire off custom scripts" when specific actions occur.

Using pre-commit <https://pre-commit.com/> with SQLFluff is a good way to provide automated linting to SQL developers. ✨

With pre-commit <https://pre-commit.com/>, you also get the benefit of only linting/fixing the files that changed.

SQLFluff comes with two pre-commit <https://pre-commit.com/> hooks:

  • sqlfluff-lint: returns linting errors.
  • sqlfluff-fix: attempts to fix rule violations.
Warning: For safety reasons, sqlfluff-fix by default will not make any fixes in files that had templating or parse errors, even if those errors were ignored using noqa or --ignore. Although it is not advised, you can tell SQLFluff to try and fix these files by overriding the fix_even_unparsable setting in .sqlfluff config file or using the sqlfluff fix --FIX-EVEN-UNPARSABLE command line option. Overriding this behavior may break your SQL. If you use this override, always be sure to review any fixes applied to files with templating or parse errors to verify they are okay. 😭

You should create a file named .pre-commit-config.yaml at the root of your git project, which should look like this:

repos:
- repo: https://github.com/sqlfluff/sqlfluff
  rev: stable_version
  hooks:
    - id: sqlfluff-lint
      # For dbt projects, this installs the dbt "extras".
      # You will need to select the relevant dbt adapter for your dialect
      # (https://docs.getdbt.com/docs/available-adapters):
      # additional_dependencies: ['<dbt-adapter>', 'sqlfluff-templater-dbt']
    - id: sqlfluff-fix
      # Arbitrary arguments to show an example
      # args: [--rules, "LT02,CP02"]
      # additional_dependencies: ['<dbt-adapter>', 'sqlfluff-templater-dbt']

When trying to use the dbt templater <dbt-project-configuration>, uncomment the additional_dependencies to install the extras. This is equivalent to running pip install <dbt-adapter> sqlfluff-templater-dbt.

You can specify the version of dbt-adapter used in pre-commit <https://pre-commit.com/>, for example:

additional_dependencies : ['dbt-bigquery==1.0.0', 'sqlfluff-templater-dbt']

See the list of available dbt-adapters <https://docs.getdbt.com/docs/available-adapters>.

Note that you can pass the same arguments available through the CLI using args:.

🦋 Ignoring files while using pre-commit 🦋

Under the hood, pre-commit <https://pre-commit.com/> works by passing specific files to SQLFluff. For example, if the only two files that are modified in your commit are file_a.sql and file_b.sql, then the command which is called in the background is sqlfluff lint file_a.sql file_b.sql. While this is efficient, it does produce some unwanted noise when also using .sqlfluffignore <#sqlfluffignore>. This is because SQLFluff is designed to allow users to override an ignore configuration by passing the name of the file directly. This makes a lot of sense in a CLI context, but less so in the context of being invoked by pre-commit <https://pre-commit.com/>.

To avoid noisy logs when using both pre-commit <https://pre-commit.com/> and .sqlfluffignore <#sqlfluffignore>, we recommend also setting the exclude argument in your .pre-commit-config.yaml file (either the top level config <https://pre-commit.com/#top_level-exclude> or the hook specific config <https://pre-commit.com/#config-exclude>). This will prevent files matching the given pattern being passed to SQLFluff and so silence any warnings about the .sqlfluffignore <#sqlfluffignore> being overridden.

🌈 Using GitHub Actions <https://github.com/features/actions> to Annotate PRs 🌈

There are two way to utilize SQLFluff to annotate Github PRs.

  1. When sqlfluff lint is run with the --format github-annotation-native option, it spits out output formatted as Github workflow commands <https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#setting-a-notice-message> which Github then converts into pull request annotations. ✨
  2. When sqlfluff lint is run with the --format github-annotation option, it spits out output that's compatible with this action from yuzutech <https://github.com/yuzutech/annotations-action>. That one uses the Github API to annotate the SQL in GitHub pull requests. 💅
Warning: okay bestie, as of right now (December 2023), limitations put in place by Github mean that only the first 10 annotations will be displayed if you go with the first option (the github-annotation-native one). This is not something SQLFluff can control itself 😭 and so we currently recommend using the second option above and the action from yuzutech <https://github.com/yuzutech/annotations-action>.
There's an open feature request <https://github.com/orgs/community/discussions/68471> for GitHub Actions which you can track to follow this issue. 👑

For more information and examples on using SQLFluff in GitHub Actions, see the sqlfluff-github-actions repository <https://github.com/sqlfluff/sqlfluff-github-actions>. 🦋

📚 Reference

🌈 Dialects Reference

SQLFluff is designed to be flexible about supporting a whole bunch of dialects. Not every potential dialect is supported so far, but several have been implemented by the community. Below is a list of the currently available dialects. Each one inherits from another, all the way up to the root ansi dialect. 🫶

For a canonical list of supported dialects, run the sqlfluff dialects command, which will output a list of the current dialects available on your installation of SQLFluff. ✨

Note: for technical users looking to add new dialects or add new features to existing ones — the dependent nature of how dialects have been implemented is to try and reduce the amount of repetition in how different elements are defined. As an example, when we say that the AWS Redshift dialect inherits from the PostgreSQL dialect, this is not because there's an agreement between those projects which means that features in one must end up in the other. It's because the design of the AWS Redshift dialect was heavily inspired by the postgres dialect, and therefore when defining the dialect within sqlfuff it makes sense to use PostgreSQL as a starting point rather than starting from scratch. 💖
Consider when adding new features to a dialect: - Should I be adding it just to this dialect, or adding it to a parent dialect? - If I'm creating a new dialect, which dialect would be best to inherit from? - Will the feature I'm adding break any downstream dependencies within dialects which inherit from this one?

🎀 ANSI

Label: ansi

Default Casing: UPPERCASE

Quotes: String Literals: '', Identifiers: ""

This is the base dialect, and it holds most of the definitions of common SQL commands and structures. If the dialect you're actually using isn't specifically implemented by SQLFluff, starting here is a solid move. 💜

This dialect doesn't intend to be brutal about adhering to (and only to) the ANSI SQL spec — mostly because ANSI charges for access to that spec 😭. It aims to be a representation of vanilla SQL before any other project adds their spin to it, and so it may contain a slightly wider set of functions than are actually available in true ANSI SQL.

🌸 AWS Athena

Label: athena

Inherits from: ANSI

Default Casing: lowercase

Quotes: String Literals: '', "" or , Identifiers: "" or

The dialect for Athena <https://aws.amazon.com/athena/> on Amazon Web Services (AWS). ✨

🦋 Google BigQuery

Label: bigquery

Inherits from: ANSI

Default Casing: BigQuery resolves unquoted column identifiers case insensitively, and table/dataset identifiers case sensitively (by default, unless is_case_insensitive is set for the latter). Unless specified, columns are returned in the case which they were defined in, which means columns can be re-cased in the result set without aliasing — e.g. if a table is defined with CREATE TEMPORARY TABLE foo (col1 int, COL2 int) then SELECT * FROM foo returns col1 and COL2 in the result, but SELECT COL1, col2 FROM foo returns COL1 and col2 in the result. 💅

Quotes: String Literals: '', "", @ or @@ (with the quoted options, also supporting variants prefixes with r/R (for raw/regex expressions) or b/B (for byte strings)), Identifiers: "" or .

The dialect for BigQuery <https://cloud.google.com/bigquery/> on Google Cloud Platform (GCP). ☁️

💖 ClickHouse

Label: clickhouse

Inherits from: ANSI

Default Casing: Clickhouse is case sensitive throughout, regardless of quoting. An unquoted reference to an object using the wrong case will raise an UNKNOWN_IDENTIFIER error. 😳

Quotes: String Literals: '', Identifiers: "" or . Note as above, that because identifiers are always resolved case sensitively, the only reason for quoting identifiers is when they contain invalid characters or reserved keywords.

The dialect for ClickHouse <https://clickhouse.com/>. 🔥

⭐ Databricks

Label: databricks

Inherits from: Apache Spark SQL

The dialect for Databricks <https://databricks.com/>. ✨

💜 IBM Db2

Label: db2

Inherits from: ANSI

The dialect for IBM Db2 <https://www.ibm.com/analytics/db2>. 🫶

🍓 Apache Doris

Label: doris

Inherits from: MySQL

Default Casing: lowercase

Quotes: String Literals: '', ", Identifiers: .

The dialect for Apache Doris <https://doris.apache.org/>. 🌸

🦆 DuckDB

Label: duckdb

Inherits from: PostgreSQL

Default Casing: DuckDB stores all identifiers in the case they were defined, however all identifier resolution is case-insensitive (when unquoted, and more unusually, also when quoted). See the DuckDB Identifiers Documentation <https://duckdb.org/docs/sql/dialect/keywords_and_identifiers> for more details. 👀

Quotes: String Literals: '', Identifiers: "" or ''

The dialect for DuckDB <https://duckdb.org/>. 💅

🌈 Exasol

Label: exasol

Inherits from: ANSI

The dialect for Exasol <https://www.exasol.com/>. ✨

Label: flink

Inherits from: ANSI

Default Casing: FlinkSQL is case insensitive with both quoted and unquoted identifiers.

Quotes: String Literals: ' (single quotes), Identifiers: .

The dialect for Apache Flink SQL <https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/#noqa:E501>. This dialect includes FlinkSQL-specific syntax for stream processing, table operations, and connector configurations. 🌊

FlinkSQL supports advanced features like:

  • ROW data types for complex nested structures
  • Table connectors (Kafka, BigQuery, etc.)
  • Watermark definitions for event time processing
  • Computed columns and metadata columns
  • Temporal table functions

💚 Greenplum

Label: greenplum

Inherits from: PostgreSQL

The dialect for Greenplum <https://www.greenplum.org/>. 🐘

🐝 Apache Hive

Label: hive

Inherits from: ANSI

The dialect for Apache Hive <https://hive.apache.org/>. ✨

🦋 Apache Impala

Label: impala

Inherits from: Apache Hive

The dialect for Apache Impala <https://impala.apache.org/>. 💖

🌸 MariaDB

Label: mariadb

Inherits from: MySQL

Default Casing: lowercase

Quotes: String Literals: '', "" or @, Identifiers: .

The dialect for MariaDB <https://www.mariadb.org/>. 🎀

⭐ Materialize

Label: materialize

Inherits from: PostgreSQL

The dialect for Materialize <https://materialize.com/>. ✨

💜 MySQL

Label: mysql

Inherits from: ANSI

Default Casing: lowercase

Quotes: String Literals: '', "" or @, Identifiers: .

The dialect for MySQL <https://www.mysql.com/>. 🐬

🔥 Oracle

Label: oracle

Inherits from: ANSI

The dialect for Oracle <https://www.oracle.com/database/technologies/appdev/sql.html> SQL. Note: this does include PL/SQL. 👑

🐘 PostgreSQL

Label: postgres

Inherits from: ANSI

Default Casing: lowercase

Quotes: String Literals: '', Identifiers: "".

This is based around the PostgreSQL spec <https://www.postgresql.org/docs/current/reference.html>. Many other SQL dialects are often based on the PostreSQL syntax. If you're running an unsupported dialect, then this is often the dialect to use (until someone makes a specific dialect). 💅

💅 AWS Redshift

Label: redshift

Inherits from: PostgreSQL

Default Casing: lowercase (unless configured to be case sensitive with all identifiers using the enable_case_sensitive_identifier configuration value, see the Redshift Names & Identifiers Docs <https://docs.aws.amazon.com/redshift/latest/dg/r_names.html>).

Quotes: String Literals: '', Identifiers: "".

The dialect for Redshift <https://aws.amazon.com/redshift/> on Amazon Web Services (AWS). 🔥

❄️ Snowflake

Label: snowflake

Inherits from: ANSI

Default Casing: UPPERCASE

Quotes: String Literals: '', Identifiers: ""

The dialect for Snowflake <https://docs.snowflake.com/en/sql-reference.html>, which has much of its syntax inherited from PostgreSQL. 🦋

🌈 Salesforce Object Query Language (SOQL)

Label: soql

Inherits from: ANSI

The dialect for SOQL <https://developer.salesforce.com/docs/atlas.en-us.soql_sosl.meta/soql_sosl/sforce_api_calls_soql.htm> (Salesforce Object Query Language). ✨

⚡ Apache Spark SQL

Label: sparksql

Inherits from: ANSI

Default Casing: SparkSQL is case insensitive with both quoted and unquoted identifiers (_"delimited"_ identifiers in Spark terminology). See the Spark Identifiers <https://spark.apache.org/docs/latest/sql-ref-identifier.html> docs. 👀

Quotes: String Literals: '' or "", Identifiers: .

The dialect for Apache Spark SQL <https://spark.apache.org/docs/latest/sql-ref.html>. This includes relevant syntax from Apache Hive for commands that permit Hive Format. Spark SQL extensions provided by the Delta Lake <https://docs.delta.io/latest/quick-start.html#set-up-apache-spark-with-delta-lake> project are also implemented in this dialect. 💖

This implementation focuses on the Ansi Compliant Mode <https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html> introduced in Spark3, instead of being Hive Compliant. The introduction of ANSI Compliance provides better data quality and easier migration from traditional DBMS. ✨

Versions of Spark prior to 3.x will only support the Hive dialect. ⚠️

🪶 SQLite

Label: sqlite

Inherits from: ANSI

Default Casing: Not specified in the docs, but through testing it appears that SQLite stores column names in whatever case they were defined, but is always case-insensitive when resolving those names. 🦋

Quotes: String Literals: '' (or "" if not otherwise resolved to an identifier), Identifiers: "", [] or .. Peep the SQLite Keywords Docs at https://sqlite.org/lang_keywords.html for the full tea. 💅

And yeah — this whole thing is the dialect for SQLite at https://www.sqlite.org/. iconic

StarRocks 🦋

  • Label: starrocks
  • Inherits from: MySQL
  • Default Casing: lowercase

Quotes: String Literals: '', ", Identifiers: .

It's the dialect for StarRocks at https://www.starrocks.io/. ✨

Teradata 🍓

  • Label: teradata
  • Inherits from: ANSI

The dialect for Teradata at https://www.teradata.co.uk/. no cap, that's literally the whole entry bestie 💖

Trino 🌸

  • Label: trino
  • Inherits from: ANSI
  • Default Casing: lowercase, although the case of a reference is used in the result set column label.

Okay so this one has a moment that you need to hear: if a column is defined using CREATE TEMPORARY TABLE foo (COL1 int), then SELECT * FROM foo returns a column labelled col1, however SELECT COL1 FROM foo returns a column labelled COL1. Like... the vibes are inconsistent and that's just how it is. 🙈

Quotes: String Literals: '', Identifiers: ""

The dialect for Trino at https://trino.io/docs/current/. 🔥

Microsoft T-SQL 👑

  • Label: tsql
  • Inherits from: ANSI

The dialect for T-SQL at https://docs.microsoft.com/en-us/sql/t-sql/language-reference (aka Transact-SQL). iykyk 💅

Vertica 🧚

  • Label: vertica
  • Inherits from: ANSI

The dialect for Vertica at https://www.vertica.com/documentation/vertica/all/. periodt ✨

Rules Reference 📜

This page is an index of all the available rules which are bundled with SQLFluff. it's giving table of contents but make it a vibe check.

  • For information on how to configure which rules are enabled for your project see Enabling and Disabling Rules at #ruleselection.
  • If you just want to turn rules on or off for specific files, or specific sections of files, see Ignoring Errors & Files at #ignoreconfig.
  • For more information on how to configure the rules which you do enable see Rule Configuration at #ruleconfig.

Core Rules 👑

Certain rules belong to the core rule group. In order for a rule to be designated as core, it must meet the following criteria:

  • Stable
  • Applies to most dialects
  • Could detect a syntax issue
  • Isn't too opinionated toward one style (e.g. the dbt style guide)

Core rules can also make it easier to roll out SQLFluff to a team by only needing to follow a 'common sense' subset of rules initially, rather than spending time understanding and configuring all the rules, some of which your team may not necessarily agree with.

We believe teams will eventually want to enforce more than just the core rules, and we encourage everyone to explore all the rules and customize a rule set that best suites their organization.

See the Configuration section at #config for more information on how to enable only core rules by default. 🫶

Rule Index 🌈

Every row of the big table, bestie — bundle, rule name, code, and aliases. Blank alias cells are blank on purpose, not me forgetting. 💀

Aliasing bundle

  • aliasing.table — Code: AL01, Aliases: L011
  • aliasing.column — Code: AL02, Aliases: L012
  • aliasing.expression — Code: AL03, Aliases: L013
  • aliasing.unique.table — Code: AL04, Aliases: L020
  • aliasing.unused — Code: AL05, Aliases: L025
  • aliasing.length — Code: AL06, Aliases: L066
  • aliasing.forbid — Code: AL07, Aliases: L031
  • aliasing.unique.column — Code: AL08, Aliases: (none)
  • aliasing.self_alias.column — Code: AL09, Aliases: (none)

Ambiguous bundle

  • ambiguous.distinct — Code: AM01, Aliases: L021
  • ambiguous.union — Code: AM02, Aliases: L033
  • ambiguous.order_by — Code: AM03, Aliases: L037
  • ambiguous.column_count — Code: AM04, Aliases: L044
  • ambiguous.join — Code: AM05, Aliases: L051
  • ambiguous.column_references — Code: AM06, Aliases: L054
  • ambiguous.set_columns — Code: AM07, Aliases: L068
  • ambiguous.join_condition — Code: AM08, Aliases: (none)
  • ambiguous.order_by_limit — Code: AM09, Aliases: (none)

Capitalisation bundle

  • capitalisation.keywords — Code: CP01, Aliases: L010
  • capitalisation.identifiers — Code: CP02, Aliases: L014
  • capitalisation.functions — rule code CP03, legacy alias L030 💅
  • capitalisation.literals — rule code CP04, legacy alias L040 ✨
  • capitalisation.types — rule code CP05, legacy alias L063 🎀

Convention bundle 🦋

okay bestie this whole lil cluster lives under the Convention bundle umbrella, and every one of them comes with a fancy CV code plus its lil throwback legacy L number:

  • convention.not_equal — rule code CV01, legacy alias L061 💖
  • convention.coalesce — rule code CV02, legacy alias L060 ✨
  • convention.select_trailing_comma — rule code CV03, legacy alias L038 🎀
  • convention.count_rows — rule code CV04, legacy alias L047 🦋
  • convention.is_null — rule code CV05, legacy alias L049 🌸
  • convention.terminator — rule code CV06, legacy alias L052 💅
  • convention.statement_brackets — rule code CV07, legacy alias L053 🫶
  • convention.left_join — rule code CV08, legacy alias L055 😭
  • convention.blocked_words — rule code CV09, legacy alias L062 🔥
  • convention.quoted_literals — rule code CV10, legacy alias L064 👑
  • convention.casting_style — rule code CV11, legacy alias L067 🌈
  • convention.join_condition — rule code CV12, and no legacy alias, it's giving brand new era, no throwback for this one ⭐

Jinja bundle 🧚

  • jinja.padding — rule code JJ01, legacy alias L046 💜

Layout bundle 💀

the layout girlies, and lowkey one of them is doing the MOST with the legacy codes:

  • layout.spacing — rule code LT01, legacy aliases L001, L005, L006, L008, L023, L024, L039, L048, L071 (yeah. nine of them. she's been through some rebrands, no cap 🙈)
  • layout.indent — rule code LT02, legacy aliases L002, L003, L004 🍓
  • layout.operators — rule code LT03, legacy alias L007 ✨
  • layout.commas — rule code LT04, legacy alias L019 🎀
  • layout.long_lines — rule code LT05, legacy alias L016 🦋
  • layout.functions — rule code LT06, legacy alias L017 🌸
  • layout.cte_bracket — rule code LT07, legacy alias L018 💅
  • layout.cte_newline — rule code LT08, legacy alias L022 🫶
  • layout.select_targets — rule code LT09, legacy alias L036 😭
  • layout.select_modifiers — rule code LT10, legacy alias L041 🔥
  • layout.set_operators — rule code LT11, legacy alias L065 👑
  • layout.end_of_file — code LT12 — aliases: L009, layout.end-of-file ✨
  • layout.start_of_file — code LT13 — alias: L050
  • layout.keyword_newline — code LT14 — no aliases, she's brand new 💅
  • layout.newlines — code LT15 — also alias-less
  • References bundle 🎀
    • references.from — code RF01 — alias: L026
    • references.qualification — code RF02 — alias: L027
    • references.consistent — code RF03 — alias: L028
    • references.keywords — code RF04 — alias: L029
    • references.special_chars — code RF05 — alias: L057
    • references.quoting — code RF06 — alias: L059
  • Structure bundle 🦋
    • structure.else_null — code ST01 — alias: L035
    • structure.simple_case — code ST02 — alias: L043
    • structure.unused_cte — code ST03 — alias: L045
    • structure.nested_case — code ST04 — alias: L058
    • structure.subquery — code ST05 — alias: L042
    • structure.column_order — code ST06 — alias: L034
    • structure.using — code ST07 — alias: L032
    • structure.distinct — code ST08 — alias: L015
    • structure.join_condition_order — code ST09 — no alias
    • structure.constant_expression — code ST10 — no alias
    • structure.unused_join — code ST11 — no alias
    • structure.consecutive_semicolons — code ST12 — no alias
  • TSQL bundle 🌈
    • tsql.sp_prefix — code TQ01 — alias: L056

👑 Aliasing bundle

Code AL01

Rule aliasing.table

Implicit/explicit aliasing of table. Basically: when you alias a table, do you wanna follow your preference? And heads up bestie, requiring an explicit AS is the default 💅

Anti-pattern 🙈

In this example, the alias voo is implicit. Not it.

SELECT
    voo.a
FROM foo voo

Best practice ✨

Add AS to make it explicit. Slay.

SELECT
    voo.a
FROM foo AS voo

This rule is sqlfluff fix compatible. 🫶

Name: aliasing.table

Aliases: L011

Groups: all, aliasing

Configuration

  • aliasing: Should alias have an explicit AS or is implicit aliasing required? Must be one of ['implicit', 'explicit'].

Code AL02

Rule aliasing.column

Implicit/explicit aliasing of columns. Aliasing of columns to follow preference — and fr fr, explicit using an AS clause is the default. 🌸

Anti-pattern 😭

In this example, the alias for column a is implicit.

SELECT
    a alias_col
FROM foo

Best practice 💖

Add AS to make it explicit.

SELECT
    a AS alias_col
FROM foo

This rule is sqlfluff fix compatible. 🎀

Name: aliasing.column

Aliases: L012

Groups: all, core, aliasing

Configuration

  • aliasing: Should alias have an explicit AS or is implicit aliasing required? Must be one of ['implicit', 'explicit'].

Code AL03

Rule aliasing.expression

Column expression without alias. Use explicit AS clause. Like, just give it a name bestie, it deserves one. 👑

Anti-pattern 💀

In this example, there is no alias for both sums.

SELECT
    sum(a),
    sum(b)
FROM foo

Best practice ⭐

Add aliases.

SELECT
    sum(a) AS a_sum,
    sum(b) AS b_sum
FROM foo
  • Name: aliasing.expression 🎀
  • Aliases: L013
  • Groups: all, core, aliasing

Configuration

  • allow_scalar: Whether or not to allow a single element in the select clause to be without an alias. Must be one of [True, False].

Code AL04 — Rule aliasing.unique.table 🌸

Table aliases should be unique within each clause. Reusing table aliases is very likely a coding error, bestie. It's giving "I copy-pasted and forgot to change the t". 🙈

Anti-pattern

In this example, the alias t is reused for two different tables:

SELECT
    t.a,
    t.b
FROM foo AS t, bar AS t

-- This can also happen when using schemas where the
-- implicit alias is the table name:

SELECT
    a,
    b
FROM
    2020.foo,
    2021.foo

Best practice

Make all tables have a unique alias. ✨

SELECT
    f.a,
    b.b
FROM foo AS f, bar AS b

-- Also use explicit aliases when referencing two tables
-- with the same name from two different schemas.

SELECT
    f1.a,
    f2.b
FROM
    2020.foo AS f1,
    2021.foo AS f2
  • Name: aliasing.unique.table
  • Aliases: L020
  • Groups: all, core, aliasing, aliasing.unique

Code AL05 — Rule aliasing.unused 🦋

Tables should not be aliased if that alias is not used. Like, why did you name it if you're never gonna call it by that name? 💀

Anti-pattern

SELECT
    a
FROM foo AS zoo

Best practice

Use the alias or remove it. An unused alias makes code harder to read without changing any functionality — so it's pure vibes tax with zero payoff. 😭

SELECT
    zoo.a
FROM foo AS zoo

-- Alternatively...

SELECT
    a
FROM foo

This rule is sqlfluff fix compatible. 🔥

  • Name: aliasing.unused
  • Aliases: L025
  • Groups: all, core, aliasing

Configuration

  • alias_case_check: How to handle comparison casefolding in an alias. Must be one of ['dialect', 'case_insensitive', 'quoted_cs_naked_upper', 'quoted_cs_naked_lower', 'case_sensitive'].

Code AL06 — Rule aliasing.length 🎀

Enforce table alias lengths in from clauses and join conditions. Short aliases are so last season, no cap. ✋

Anti-pattern

In this example, alias o is used for the orders table.

SELECT
    SUM(o.amount) as order_amount,
FROM orders as o

Best practice

Avoid aliases. Avoid short aliases when aliases are necessary. 👑

See also: AL07.

SELECT
    SUM(orders.amount) as order_amount,
FROM orders

SELECT
    replacement_orders.amount,
    previous_orders.amount
FROM
    orders AS replacement_orders
JOIN
    orders AS previous_orders
    ON replacement_orders.id = previous_orders.replacement_id
  • Name: aliasing.length
  • Aliases: L066
  • Groups: all, core, aliasing

Configuration

  • max_alias_length: The maximum length of an alias to allow without raising a violation. Must be one of range(0, 1000).
  • min_alias_length: The minimum length of an alias to allow without raising a violation. Must be one of range(0, 1000).

Code AL07 — Rule aliasing.forbid 💅

Avoid table aliases in from clauses and join conditions. Straight up banning them, periodt.

Note: This rule was taken from the dbt Style Guide https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md which notes that: Avoid table aliases in join conditions (especially initialisms) - it's harder to understand what the table called "c" is compared to "customers".

This rule is controversial and for many larger databases avoiding alias is neither realistic nor desirable. In particular for BigQuery due to the complexity of backtick requirements and determining whether a name refers to a project or dataset so automated fixes can potentially break working SQL code. 😱 For most users AL06 is likely a more appropriate linting rule to drive a sensible behaviour around aliasing.

The stricter treatment of aliases in this rule may be useful for more focused projects, or temporarily as a refactoring tool because the fix routine of the rule can remove aliases.

This rule is disabled by default for all dialects — it can be enabled with the force_enable = True flag.

Anti-pattern

In this example, alias o is used for the orders table, and c is used for customers table.

SELECT
    COUNT(o.customer_id) as order_amount,
    c.name
FROM orders as o
JOIN customers as c on o.id = c.user_id

Best practice

Avoid aliases. 🌈

SELECT
    COUNT(orders.customer_id) as order_amount,
    customers.name
FROM orders
JOIN customers on orders.id = customers.user_id

-- Self-join will not raise issue

SELECT
    table1.a,
    table_alias.b,
FROM
    table1
    LEFT JOIN table1 AS table_alias ON
        table1.foreign_key = table_alias.foreign_key

This rule is sqlfluff fix compatible. ✨

  • Name: aliasing.forbid
  • Aliases: L031
  • Groups: all, aliasing

Configuration

  • force_enable: Run this rule even for dialects where this rule is disabled by default. Must be one of [True, False].

Code AL08 — Rule aliasing.unique.column 🧚

Column aliases should be unique within each clause. Reusing column aliases is very likely a coding error. Note that while in many dialects, quoting an identifier makes it case-sensitive this rule always compares in a case-insensitive way. This is because columns with the same name, but different case, are still confusing and potentially ambiguous to other readers. 😵‍💫

In situations where it is necessary to have columns with the same name (whether they differ in case or not) we recommend disabling this rule for either just the line, or the whole file.

Anti-pattern

In this example, the alias foo is reused for two different columns:

SELECT
    a as foo,
    b as foo
FROM tbl;

-- This can also happen when referencing the same column
-- column twice, or aliasing an expression to the same
-- name as a column:

SELECT
    foo,
    foo,
    a as foo
FROM tbl;

Best practice

Make all columns have a unique alias. 💖

SELECT
    a as foo,
    b as bar
FROM tbl;

-- Avoid also using the same column twice unless aliased:

SELECT
    foo as foo1,
    foo as foo2,
    a as foo3
FROM tbl;
  • Name: aliasing.unique.column
  • Groups: all, core, aliasing, aliasing.unique

Code AL09 — Rule aliasing.self_alias.column 🫶

Column aliases should not alias to itself, i.e. self-alias. Renaming the column to itself is a redundant piece of SQL, which doesn't affect its functionality. This rule only applies when aliasing to an exact copy of the column reference (e.g. foo as foo or "BAR" as "BAR", see note below on more complex examples). Aliases which effectively change the casing of an identifier are still allowed. 👑

Note: This rule works in conjunction with references.quoting (RF06) and capitalisation.identifiers (CP02) to handle self aliases with mixed quoting and casing. In the situation that these two rules are not enabled then this rule will only fix the strict case where the quoting and casing of the alias and reference are the same. If those two rules are enabled, the fixes applied may result in a situation where this rule can kick in as a secondary effect. For example this Snowflake <#snowflake-dialect-ref> query: sql -- Original Query. AL09 will not trigger because casing and -- quoting are different. RF06 will however fix the unnecessary -- quoting of "COL". SELECT "COL" AS col FROM table; -- After RF06, the query will look like this, at which point -- CP02 will see the inconsistent capitalisation. Depending -- on the configuration it will change one of the identifiers. -- Let's assume the default configuration of "consistent". SELECT COL AS col FROM table; -- After CP02, the alias and the reference will be the same -- and at this point AL09 can take over and remove the alias. SELECT COL AS COL FROM table; -- ..resulting in: SELECT COL FROM table; This interdependence between the rules, and the configuration options offered by each one means a variety of outcomes can be achieved by enabling and disabling each one. See Enabling and Disabling Rules <#ruleselection> and Rule Configuration <#ruleconfig> for more details.

Anti-pattern

Aliasing the column to itself, where not necessary for changing the case of an identifier.

SELECT
    col AS col,
    "Col" AS "Col",
    COL AS col
FROM table;

Best practice

Not to use alias to rename the column to its original name. Self-aliasing leads to redundant code without changing any functionality, unless used to effectively change the case of the identifier. 💫

SELECT
    col,
    "Col"
    COL,
FROM table;

-- Re-casing aliasing is still allowed where necessary, i.e.
SELECT
    col as "Col",
    "col" as "COL"
FROM table;

This rule is sqlfluff fix compatible. 🔥

  • Name: aliasing.self_alias.column
  • Groups: all, core, aliasing

🌈 Ambiguous bundle

Code AM01 — Rule ambiguous.distinct 💀

Ambiguous use of DISTINCT in a SELECT statement with GROUP BY. When using GROUP BY a DISTINCT clause should not be necessary as every non-distinct SELECT clause must be included in the GROUP BY clause.

Anti-pattern

DISTINCT and GROUP BY are conflicting. 😭

SELECT DISTINCT
    a
FROM foo
GROUP BY a

Best practice

Remove DISTINCT or GROUP BY. In our case, removing GROUP BY is better.

SELECT DISTINCT
    a
FROM foo
  • Name: ambiguous.distinct
  • Aliases: L021
  • Groups: all, core, ambiguous

Code AM02 — Rule ambiguous.union ⭐

UNION [DISTINCT|ALL] is preferred over just UNION. Say what you mean, bestie. 🎀

Note: This rule is only enabled for dialects that support UNION and UNION DISTINCT (ansi, bigquery, clickhouse, databricks, db2, hive, mysql, redshift, snowflake, and trino).

Anti-pattern

okay bestie, we're deep in the rules now — hydrate, stretch, we got this 💖

In this example, UNION DISTINCT should be preferred over UNION, because explicit is better than implicit. ✨

SELECT a, b FROM table_1
UNION
SELECT a, b FROM table_2

Best practice 👑

Specify DISTINCT or ALL after UNION (note that DISTINCT is the default behavior).

SELECT a, b FROM table_1
UNION DISTINCT
SELECT a, b FROM table_2

This rule is `sqlfluff fix` compatible.

Name: `ambiguous.union`

Aliases: `L033`

Groups: `all`, `core`, `ambiguous`

Code AM03

Rule ambiguous.order_by

🌀 Ambiguous ordering directions for columns in order by clause.

Anti-pattern 🙈

SELECT
    a, b
FROM foo
ORDER BY a, b DESC

Best practice 🌸

If any columns in the ORDER BY clause specify ASC or DESC, they should all do so. no halfsies, bestie.

SELECT
    a, b
FROM foo
ORDER BY a ASC, b DESC

This rule is `sqlfluff fix` compatible.

Name: `ambiguous.order_by`

Aliases: `L037`

Groups: `all`, `ambiguous`

Code AM04

Rule ambiguous.column_count

🌀 Query produces an unknown number of result columns.

Anti-pattern 😭

Querying all columns using * produces a query result where the number or ordering of columns changes if the upstream table's schema changes. This should generally be avoided because it can cause slow performance, cause important schema changes to go undetected, or break production code. For example:

  • If a query does SELECT t.* and is expected to return columns a, b, and c, the actual columns returned will be wrong/different if columns are added to or deleted from the input table.
  • UNION and DIFFERENCE clauses require the inputs have the same number of columns (and compatible types).
  • JOIN queries may break due to new column name conflicts, e.g. the query references a column c which initially existed in only one input table but a column of the same name is added to another table.
  • CREATE TABLE (<<column schema>>) AS SELECT *
WITH cte AS (
    SELECT * FROM foo
)

SELECT * FROM cte
UNION
SELECT a, b FROM t

Best practice 💅

Somewhere along the "path" to the source data, specify columns explicitly.

WITH cte AS (
    SELECT * FROM foo
)

SELECT a, b FROM cte
UNION
SELECT a, b FROM t

Name: `ambiguous.column_count`

Aliases: `L044`

Groups: `all`, `ambiguous`

Code AM05

Rule ambiguous.join

🌀 Join clauses should be fully qualified.

By default this rule is configured to enforce fully qualified INNER JOIN clauses, but not [LEFT/RIGHT/FULL] OUTER JOIN. If you prefer a stricter lint then this is configurable. 👀

Anti-pattern 🙈

A join is used without specifying the kind of join.

SELECT
    foo
FROM bar
JOIN baz;

Best practice ✨

Use INNER JOIN rather than JOIN.

SELECT
    foo
FROM bar
INNER JOIN baz;

This rule is `sqlfluff fix` compatible.

Name: `ambiguous.join`

Aliases: `L051`

Groups: `all`, `ambiguous`

Configuration

  • `fully_qualify_join_types`: Which types of JOIN clauses should be fully qualified? Must be one of `['inner', 'outer', 'both']`.

Code AM06

Rule ambiguous.column_references

🌀 Inconsistent column references in GROUP BY/ORDER BY clauses.

Note: ORDER BY clauses from WINDOW clauses are ignored by this rule.

Anti-pattern 😭

A mix of implicit and explicit column references are used in a GROUP BY clause.

SELECT
    foo,
    bar,
    sum(baz) AS sum_value
FROM fake_table
GROUP BY
    foo, 2;

-- The same also applies to column
-- references in ORDER BY clauses.

SELECT
    foo,
    bar
FROM fake_table
ORDER BY
    1, bar;

Best practice 🌈

Reference all GROUP BY/ORDER BY columns either by name or by position.

-- GROUP BY: Explicit
SELECT
    foo,
    bar,
    sum(baz) AS sum_value
FROM fake_table
GROUP BY
    foo, bar;

-- ORDER BY: Explicit
SELECT
    foo,
    bar
FROM fake_table
ORDER BY
    foo, bar;

-- GROUP BY: Implicit
SELECT
    foo,
    bar,
    sum(baz) AS sum_value
FROM fake_table
GROUP BY
    1, 2;

-- ORDER BY: Implicit
SELECT
    foo,
    bar
FROM fake_table
ORDER BY
    1, 2;

Name: `ambiguous.column_references`

Aliases: `L054`

Groups: `all`, `core`, `ambiguous`

Configuration

  • `group_by_and_order_by_style`: The expectation for using explicit column name references or implicit positional references. Must be one of `['consistent', 'implicit', 'explicit']`.

Code AM07

Rule ambiguous.set_columns

🌀 Queries within set query produce different numbers of columns.

Anti-pattern 💀

When writing set expressions, all queries must return the same number of columns.

WITH cte AS (
    SELECT
        a,
        b
    FROM foo
)
SELECT * FROM cte
UNION
SELECT
    c,
    d,
    e
 FROM t

Best practice 🫶

Always specify columns when writing set queries and ensure that they all seleect same number of columns

WITH cte AS (
    SELECT a, b FROM foo
)
SELECT
    a,
    b
FROM cte
UNION
SELECT
    c,
    d
FROM t

Name: `ambiguous.set_columns`

Aliases: `L068`

Groups: `all`, `ambiguous`

Code AM08

Rule ambiguous.join_condition

🌀 Implicit cross join detected.

Anti-pattern 🙈

Cross joins are valid, but rare in the wild - and more often created by mistake than on purpose. This rule catches situations where a cross join has been specified, but not explicitly and so the risk of a mistaken cross join is highly likely.

SELECT
    foo
FROM bar
JOIN baz;

Best practice 🌸

Use CROSS JOIN.

SELECT
    foo
FROM bar
CROSS JOIN baz;

This rule is `sqlfluff fix` compatible.

Name: `ambiguous.join_condition`

Groups: `all`, `ambiguous`

Code AM09

Rule ambiguous.order_by_limit

🌀 Use of LIMIT and OFFSET without ORDER BY may lead to non-deterministic results.

When using LIMIT or OFFSET, it's generally recommended to include an ORDER BY clause to ensure deterministic results.

Anti-pattern 😭

The following query has LIMIT and OFFSET without ORDER BY, which may return different results in successive executions.

SELECT *
FROM foo
LIMIT 10 OFFSET 5;

Best practice ✨

Include an ORDER BY clause:

SELECT *
FROM foo
ORDER BY id
LIMIT 10 OFFSET 5;

Name: `ambiguous.order_by_limit`

Groups: `all`, `ambiguous`

🎀 Capitalisation bundle 🎀

Code CP01

Rule capitalisation.keywords

🌀 Inconsistent capitalisation of keywords.

Anti-pattern 🙈

In this example, select is in lower-case whereas FROM is in upper-case.

select
    a
FROM foo

Best practice 👑

Make all keywords either in upper-case or in lower-case.

SELECT
    a
FROM foo

-- Also good

select
    a
from foo

This rule is `sqlfluff fix` compatible.

Name: `capitalisation.keywords`

Aliases: `L010`

Groups: `all`, `core`, `capitalisation`

Configuration

  • `capitalisation_policy`: The capitalisation policy to enforce. Must be one of `['consistent', 'upper', 'lower', 'capitalise']`.
  • `ignore_words`: Comma separated list of words to ignore from rule.
  • `ignore_words_regex`: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use `^` (beginning of text) and `$` (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between `^` and `$`.

Code CP02

Rule capitalisation.identifiers

🌀 Inconsistent capitalisation of unquoted identifiers.

This rule applies to all unquoted identifiers, whether references or aliases, and whether they refer to columns or other objects (such as tables or schemas).

Note: In most dialects, unquoted identifiers are treated as case-insensitive and so the fixes proposed by this rule do not change the interpretation of the query. HOWEVER, some databases (notably Google BigQuery <#bigquery-dialect-ref>, Trino <#trino-dialect-ref> and ClickHouse <#clickhouse-dialect-ref>) do take the casing of unquoted identifiers into account when determining the casing of the column heading in the result. As this feature is only present in a few dialects, and not widely understood by users, we regard it as an antipattern. It is more widely understood that if the case of an identifier matters, then it should be quoted. If you, or your organisation, do wish to rely on this feature, we recommend that you disabled this rule (see Enabling and Disabling Rules <#ruleselection>).

Anti-pattern 💀

In this example, unquoted identifier a is in lower-case but B is in upper-case.

select
    a,
    B
from foo

In this more complicated example, there are a mix of capitalisations in both reference and aliases of columns and tables. That inconsistency is acceptable when those identifiers are quoted, but not when unquoted.

select
    col_1 + Col_2 as COL_3,
    "COL_4" as Col_5
from Foo as BAR

Best practice 🌈

Ensure all unquoted identifiers are either in upper-case or in lower-case.

select
    a,
    b
from foo;

-- ...also good...

select
    A,
    B
from foo;

--- ...or for comparison with our more complex example, this too:

select
    col_1 + col_2 as col_3,
    "COL_4" as col_5
from foo as bar

This rule is `sqlfluff fix` compatible.

Name: `capitalisation.identifiers`

Aliases: `L014`

Groups: `all`, `core`, `capitalisation`

Configuration

  • unquoted_identifiers_policy — which flavours of unquoted identifier you want to get flagged for violations. Must be one of ['all', 'aliases', 'column_aliases', 'table_aliases']. 💅

🎀 Code CP03

✨ Rule capitalisation.functions

Inconsistent capitalisation of function names. Bestie, a function can't be sum in one line and SUM in the next, that's giving unhinged. 😭

Anti-pattern

In this example the two SUM functions don't have the same capitalisation.

SELECT
    sum(a) AS aa,
    SUM(b) AS bb
FROM foo

Best practice

Make the case consistent.

SELECT
    sum(a) AS aa,
    sum(b) AS bb
FROM foo
This rule is sqlfluff fix compatible. 🫶

Name: capitalisation.functions

Aliases: L030

Groups: all, core, capitalisation

Configuration

  • extended_capitalisation_policy — the capitalisation policy to enforce, extended with PascalCase, snake_case, and camelCase. This is separate from capitalisation_policy as it should not be applied to keywords. Camel, Pascal, and Snake will never be inferred when the policy is set to consistent. This is because snake can cause destructive changes to the identifier, and unlinted code is too easily mistaken for camel and pascal. If, when set to consistent, no consistent case is found, it will default to upper. Must be one of ['consistent', 'upper', 'lower', 'pascal', 'capitalise', 'snake', 'camel'].
  • ignore_words — comma separated list of words to ignore from rule.
  • ignore_words_regex — words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ^ (beginning of text) and $ (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ^ and $.

🎀 Code CP04

✨ Rule capitalisation.literals

Inconsistent capitalisation of boolean/null literal. Pick a case and stay in your lane, bestie. 🌸

Anti-pattern

In this example, null and false are in lower-case whereas TRUE is in upper-case.

select
    a,
    null,
    TRUE,
    false
from foo

Best practice

Ensure all literal null/true/false literals are consistently upper or lower case

select
    a,
    NULL,
    TRUE,
    FALSE
from foo
-- Also good

select
    a,
    null,
    true,
    false
from foo
This rule is sqlfluff fix compatible. ✨

Name: capitalisation.literals

Aliases: L040

Groups: all, core, capitalisation

Configuration

  • capitalisation_policy — the capitalisation policy to enforce. Must be one of ['consistent', 'upper', 'lower', 'capitalise'].
  • ignore_words — comma separated list of words to ignore from rule.
  • ignore_words_regex — words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ^ (beginning of text) and $ (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ^ and $.

🎀 Code CP05

✨ Rule capitalisation.types

Inconsistent capitalisation of datatypes. Your types deserve the same energy, fr fr. 💖

Anti-pattern

In this example, int and unsigned are in lower-case whereas VARCHAR is in upper-case.

CREATE TABLE t (
    a int unsigned,
    b VARCHAR(15)
);

Best practice

Ensure all datatypes are consistently upper or lower case

CREATE TABLE t (
    a INT UNSIGNED,
    b VARCHAR(15)
);
This rule is sqlfluff fix compatible. 💅

Name: capitalisation.types

Aliases: L063

Groups: all, core, capitalisation

Configuration

  • extended_capitalisation_policy — the capitalisation policy to enforce, extended with PascalCase, snake_case, and camelCase. This is separate from capitalisation_policy as it should not be applied to keywords. Camel, Pascal, and Snake will never be inferred when the policy is set to consistent. This is because snake can cause destructive changes to the identifier, and unlinted code is too easily mistaken for camel and pascal. If, when set to consistent, no consistent case is found, it will default to upper. Must be one of ['consistent', 'upper', 'lower', 'pascal', 'capitalise', 'snake', 'camel'].
  • ignore_words — comma separated list of words to ignore from rule.
  • ignore_words_regex — words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ^ (beginning of text) and $ (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ^ and $.

🌈 Convention bundle

🦋 Code CV01

⭐ Rule convention.not_equal

Consistent usage of != or <> for "not equal to" operator. Don't mix and match, that's so not the vibe check. 👑

Anti-pattern

SELECT * FROM X WHERE 1 <> 2 AND 3 != 4;

Best practice

Ensure all "not equal to" comparisons are consistent, not mixing != and <>.

SELECT * FROM X WHERE 1 != 2 AND 3 != 4;
This rule is sqlfluff fix compatible. 🎀

Name: convention.not_equal

Aliases: L061

Groups: all, convention

Configuration

  • preferred_not_equal_style — the style for using not equal to operator. Defaults to consistent. Must be one of ['consistent', 'c_style', 'ansi'].

🦋 Code CV02

⭐ Rule convention.coalesce

Use COALESCE instead of IFNULL or NVL. Bestie, COALESCE ate and left no crumbs. 🔥

Anti-pattern

IFNULL or NVL are used to fill NULL values.

SELECT ifnull(foo, 0) AS bar,
FROM baz;
SELECT nvl(foo, 0) AS bar,
FROM baz;

Best practice

Use COALESCE instead. COALESCE is universally supported, whereas Redshift doesn't support IFNULL and BigQuery doesn't support NVL. Additionally, COALESCE is more flexible and accepts an arbitrary number of arguments. 💜

SELECT coalesce(foo, 0) AS bar,
FROM baz;
This rule is sqlfluff fix compatible. 🫶

Name: convention.coalesce

Aliases: L060

Groups: all, convention

🦋 Code CV03

⭐ Rule convention.select_trailing_comma

Trailing commas within select clause. 👀

Note: For many database backends this is allowed. For some users this may be something they wish to enforce (in line with Python best practice). Many database backends regard this as a syntax error, and as such the SQLFluff default is to forbid trailing commas in the select clause.

Anti-pattern

SELECT
    a,
    b,
FROM foo

Best practice

SELECT
    a,
    b
FROM foo
This rule is sqlfluff fix compatible. ✨

Name: convention.select_trailing_comma

Aliases: L038

Groups: all, core, convention

Configuration

  • select_clause_trailing_comma — should trailing commas within select clauses be required or forbidden? Must be one of ['forbid', 'require'].

🦋 Code CV04

⭐ Rule convention.count_rows

Use consistent syntax to express "count number of rows". Pick one and commit, no cap. 🌟

Note: If both prefer_count_1 and prefer_count_0 are set to true then prefer_count_1 has precedence.

COUNT(*), COUNT(1), and even COUNT(0) are equivalent syntaxes in many SQL engines due to optimizers interpreting these instructions as "count number of rows in result".

The ANSI-92 spec (http://msdn.microsoft.com/en-us/library/ms175997.aspx) mentions the COUNT(*) syntax specifically as having a special meaning:

If COUNT(*) is specified, then the result is the cardinality of T.

So by default, SQLFluff enforces the consistent use of COUNT(*).

If the SQL engine you work with, or your team, prefers COUNT(1) or COUNT(0) over COUNT(*), you can configure this rule to consistently enforce your preference.

Anti-pattern

select
    count(1)
from table_a

Best practice

Use count(*) unless specified otherwise by config prefer_count_1, or prefer_count_0 as preferred.

select
    count(*)
from table_a
This rule is sqlfluff fix compatible. 💖

Name: convention.count_rows

Aliases: L047

Groups: all, core, convention

Configuration

  • prefer_count_0 — should count(0) be preferred over count(*) and count(1)? Must be one of [True, False].
  • prefer_count_1 — should count(1) be preferred over count(*) and count(0)? Must be one of [True, False].

🦋 Code CV05

⭐ Rule convention.is_null

Comparisons with NULL should use "IS" or "IS NOT". Using = on NULL is giving broken, bestie. 😭

Anti-pattern

In this example, the = operator is used to check for NULL values.

SELECT
    a
FROM foo
WHERE a = NULL

Best practice

Use IS or IS NOT to check for NULL values.

SELECT
    a
FROM foo
WHERE a IS NULL
This rule is sqlfluff fix compatible. 🌸

Name: convention.is_null

Aliases: L049

Groups: all, core, convention

🦋 Code CV06

⭐ Rule convention.terminator

Statements must end with a semi-colon. Every statement. No exceptions, periodt. 💅

Anti-pattern

A statement is not immediately terminated with a semi-colon. The • represents space.

SELECT
    a
FROM foo

;

SELECT
    b
FROM bar••;

Best practice

Immediately terminate the statement with a semi-colon.

SELECT
    a
FROM foo;
This rule is sqlfluff fix compatible. 🎀

Name: convention.terminator

Aliases: L052

Groups: all, convention

Configuration

  • multiline_newline — should semi-colons be placed on a new line after multi-line statements? Bestie, it's gotta be one of [True, False].
  • require_final_semicolon — should final semi-colons be required? (N.B. forcing trailing semi-colons is not recommended for dbt users, because it can cause issues when wrapping the query within other SQL queries). Must be one of [True, False].

Code CV07

Rule convention.statement_brackets

Top-level statements should not be wrapped in brackets. Like, no cap.

Anti-pattern

A top-level statement is wrapped in brackets. It's giving chaos:

(SELECT
    foo
FROM bar)

-- This also applies to statements containing a sub-query.

(SELECT
    foo
FROM (SELECT * FROM bar))

Best practice

Don't wrap top-level statements in brackets. Simple as:

SELECT
    foo
FROM bar

-- Likewise for statements containing a sub-query.

SELECT
    foo
FROM (SELECT * FROM bar)

This rule is sqlfluff fix compatible. 💅

Name: convention.statement_brackets

Aliases: L053

Groups: all, convention

Code CV08

Rule convention.left_join

Use LEFT JOIN instead of RIGHT JOIN. Okay so real talk, right joins are just left joins wearing a disguise.

Anti-pattern

RIGHT JOIN is used. 🫠

SELECT
    foo.col1,
    bar.col2
FROM foo
RIGHT JOIN bar
    ON foo.bar_id = bar.id;

Best practice

Refactor and use LEFT JOIN instead. Slay:

SELECT
    foo.col1,
    bar.col2
FROM bar
LEFT JOIN foo
    ON foo.bar_id = bar.id;

Name: convention.left_join

Aliases: L055

Groups: all, convention

Code CV09

Rule convention.blocked_words

Block a list of configurable words from being used. This generic rule is lowkey iconic for preventing certain keywords, functions, or objects from sneaking into your code. Only whole words can be blocked — not phrases, and not parts of words. The way that it works is: it's case insensitive, so no trying to be sneaky. 🙈

Example use cases

  • We prefer BOOL over BOOLEAN and there is no existing rule to enforce this. Until such a rule is written, we can add BOOLEAN to the deny list to cause a linting error to flag this.
  • We have deprecated a schema/table/function and want to prevent it being used in future. We can add that to the denylist and then add a -- noqa: CV09 for the few exceptions that still need to be in the code base for now.

Anti-pattern

If the blocked_words config is set to deprecated_table,bool then the following will flag:

SELECT * FROM deprecated_table WHERE 1 = 1;
CREATE TABLE myschema.t1 (a BOOL);

Best practice

Do not used any blocked words:

SELECT * FROM another_table WHERE 1 = 1;
CREATE TABLE myschema.t1 (a BOOLEAN);

Name: convention.blocked_words

Aliases: L062

Groups: all, convention

Configuration

  • blocked_regex — optional, regex of blocked pattern which should not be used in statements.
  • blocked_words — optional, comma-separated list of blocked words which should not be used in statements.
  • match_source — optional, also match regex of blocked pattern before applying templating.

Code CV10

Rule convention.quoted_literals

Consistent usage of preferred quotes for quoted literals. Some databases let quoted literals use either single or double quotes, so this rule says: pick one type of quotes as specified in the rule setting, and fall back to the alternate quotes when that would reduce the need for escapes. 🎀

Dollar-quoted raw strings are excluded from this rule, because they're mostly used for literal UDF Body definitions.

Note: this rule only checks quoted literals and not quoted identifiers, since those often can't interchange single and double quotes. Also, this rule is only enabled for dialects that allow single and double quotes for quoted literals (currently bigquery, databricks, hive, mysql, sparksql). It can be enabled for other dialects with the force_enable = True flag.

Anti-pattern

select
    "abc",
    'abc',
    "\"",
    "abc" = 'abc'
from foo

Best practice

Ensure all quoted literals use preferred quotes, unless escaping can be reduced by using alternate quotes.

select
    "abc",
    "abc",
    '"',
    "abc" = "abc"
from foo

This rule is sqlfluff fix compatible. ✨

Name: convention.quoted_literals

Aliases: L064

Groups: all, convention

Configuration

  • force_enable — run this rule even for dialects where this rule is disabled by default. Must be one of [True, False].
  • preferred_quoted_literal_style — preferred quoting style to use for the quoted literals. If set to consistent quoting style is derived from the first quoted literal in the file. Must be one of ['consistent', 'single_quotes', 'double_quotes'].

Code CV11

Rule convention.casting_style

Enforce consistent type casting style. Pick a lane, bestie. 🛣️

Note: this is only compatible with 2-arguments CONVERT as some dialects allow an optional 3rd argument e.g TSQL, which cannot be rewritten into CAST. This rule is disabled by default for Teradata because it supports different type casting apart from CONVERT and :: e.g DATE '2007-01-01', '9999-12-31' (DATE).

Anti-pattern

Using mixture of CONVERT, :: and CAST when preferred_type_casting_style config is set to consistent (default).

SELECT
    CONVERT(int, 1) AS bar,
    100::int::text,
    CAST(10 AS text) AS coo
FROM foo;

Best practice

Use consistent type casting style.

SELECT
    CAST(1 AS int) AS bar,
    CAST(CAST(100 AS int) AS text),
    CAST(10 AS text) AS coo
FROM foo;

This rule is sqlfluff fix compatible. 💖

Name: convention.casting_style

Aliases: L067

Groups: all, convention

Configuration

  • preferred_type_casting_style — the expectation for using sql type casting. Must be one of ['consistent', 'shorthand', 'convert', 'cast'].

Code CV12

Rule convention.join_condition

Use JOIN ... ON ... instead of WHERE ... for join conditions. It's giving correct SQL energy.

Anti-pattern

Using WHERE clause for join conditions.

SELECT
    foo.a
    , bar.b
FROM foo
JOIN bar
WHERE foo.x = bar.y;

Best practice

Use JOIN ON clause for join condition.

SELECT
    foo.a
    , bar.b
FROM foo
JOIN bar
ON foo.x = bar.y;

This rule is sqlfluff fix compatible. 🌸

Name: convention.join_condition

Groups: all, convention

🧚 Jinja bundle

Code JJ01

Rule jinja.padding

Jinja tags should have a single whitespace on either side. This rule is only active if the jinja templater (or one of it's subclasses, like the dbt templater) are used for the current file.

Anti-pattern

Jinja tags with either no whitespace or very long whitespace are hard to read. 😭

SELECT {{    a     }} from {{ref('foo')}}

Best practice

A single whitespace surrounding Jinja tags — alternatively longer gaps containing newlines are acceptable.

SELECT {{ a }} from {{ ref('foo') }};
SELECT {{ a }} from {{
    ref('foo')
}};

This rule is sqlfluff fix compatible. ⭐

Name: jinja.padding

Aliases: L046

Groups: all, core, jinja

🦋 Layout bundle

Code LT01

Rule layout.spacing

Inappropriate Spacing. This rule checks for an enforces the spacing as configured in Configuring Layout <#layoutconfig>. That includes excessive whitespace, trailing whitespace at the end of a line, and also the wrong spacing between elements on the line. Because of this wide reach you may find that you wish to add specific configuration in your project to tweak how specific elements are treated. Rather than configuration on this specific rule, use the sqlfluff.layout section of your configuration file to customise how this rule operates.

The • character represents a space in the examples below.

Anti-pattern

SELECT
    a,        b(c) as d••
FROM foo••••
JOIN bar USING(a)

Best practice

  • Unless an indent or preceding a comment, whitespace should be a single space.
  • There should also be no trailing whitespace at the ends of lines.
  • There should be a space after USING so that it's not confused for a function.
SELECT
    a, b(c) as d
FROM foo
JOIN bar USING (a)

This rule is sqlfluff fix compatible. 👑

Name: layout.spacing

Aliases: L001, L005, L006, L008, L023, L024, L039, L048, L071

Groups: all, core, layout

Code LT02

Rule layout.indent

Incorrect Indentation.

Anti-pattern

The • character represents a space and the → character represents a tab. In this example, the third line contains five spaces instead of four and the second line contains two spaces and one tab.

SELECT
••→a,
•••••b
FROM foo

Best practice

Change the indentation to use a multiple of four spaces. This example also assumes that the indent_unit config value is set to space. If it had instead been set to tab, then the indents would be tabs instead.

SELECT
••••a,
••••b
FROM foo

This rule is sqlfluff fix compatible. 🌈

Name: layout.indent

Aliases: L002, L003, L004

Groups: all, core, layout

Code LT03

Rule layout.operators

Operators should follow a standard for being before/after newlines. The configuration for whether operators should be trailing or leading is part of Configuring Layout <#layoutconfig>. The default configuration is:

[sqlfluff:layout:type:binary_operator]
line_position = leading

[sqlfluff:layout:type:comparison_operator]
line_position = leading

Anti-pattern

In this example, if line_position = leading (or unspecified, as is the default), then the operator + should not be at the end of the second line.

SELECT
    a +
    b
FROM foo

Best practice

If line_position = leading (or unspecified, as this is the default), place the operator after the newline.

SELECT
    a
    + b
FROM foo

If line_position = trailing, place the operator before the newline.

so here's the glow-up version that just hits different:

SELECT
    a +
    b
FROM foo

This rule is sqlfluff fix compatible. ✨

Name: layout.operators

Aliases: L007

Groups: all, layout

Code LT04 — Rule layout.commas

Leading/Trailing comma enforcement.

So the whole "do commas go at the end of the line or the start?" situation is handled over in the Configuring Layout part (<#layoutconfig>). The default config is literally:

[sqlfluff:layout:type:comma]
line_position = trailing

Anti-pattern

Mixing leading and trailing commas in the same file. Bestie, pick a lane. 😭

SELECT
    a
    , b,
    c
FROM foo

Best practice

By default, SQLFluff is team trailing comma. But it is configurable for leading commas. Whichever style you pick has to be used consistently all the way through your SQL.

SELECT
    a,
    b,
    c
FROM foo

-- Alternatively, set the configuration file to 'leading'
-- and then the following would be acceptable:

SELECT
    a
    , b
    , c
FROM foo

This rule is sqlfluff fix compatible.

Name: layout.commas

Aliases: L019

Groups: all, layout

Code LT05 — Rule layout.long_lines

Line is too long.

This rule is sqlfluff fix compatible.

Name: layout.long_lines

Aliases: L016

Groups: all, core, layout

Configuration

  • ignore_comment_clauses — Should comment clauses (e.g. column comments) be ignored when linting line lengths? Must be one of [True, False].
  • ignore_comment_lines — Should lines that contain only whitespace and comments be ignored when linting line lengths? Must be one of [True, False].

Code LT06 — Rule layout.functions

Function name not immediately followed by parenthesis.

Anti-pattern

There's a space between the function and the parenthesis. The audacity. 🙈

SELECT
    sum (a)
FROM foo

Best practice

Remove the space between the function and the parenthesis. Let them be close.

SELECT
    sum(a)
FROM foo

This rule is sqlfluff fix compatible.

Name: layout.functions

Aliases: L017

Groups: all, core, layout

Code LT07 — Rule layout.cte_bracket

WITH clause closing bracket should be on a new line.

Anti-pattern

Here the closing bracket is chilling on the same line as the CTE. Not the move.

WITH zoo AS (
    SELECT a FROM foo)

SELECT * FROM zoo

Best practice

Move the closing bracket onto a new line.

WITH zoo AS (
    SELECT a FROM foo
)

SELECT * FROM zoo

This rule is sqlfluff fix compatible.

Name: layout.cte_bracket

Aliases: L018

Groups: all, core, layout

Code LT08 — Rule layout.cte_newline

Blank line expected but not found after CTE closing bracket.

Anti-pattern

No blank line after the CTE closing bracket. In queries with a million CTEs, this lowkey destroys readability.

WITH plop AS (
    SELECT * FROM foo
)
SELECT a FROM plop

Best practice

Add a blank line. Let it breathe. 🦋

WITH plop AS (
    SELECT * FROM foo
)

SELECT a FROM plop

This rule is sqlfluff fix compatible.

Name: layout.cte_newline

Aliases: L022

Groups: all, core, layout

Code LT09 — Rule layout.select_targets

Select targets should be on a new line unless there is only one select target.

Note: By default, a wildcard (e.g. SELECT *) is considered a single select target. If you want it to be treated as multiple select targets, configure wildcard_policy = multiple.

Anti-pattern

Multiple select targets stacked on the same line.

select a, b
from foo;

-- Single select target on its own line.

SELECT
    a
FROM foo;

Best practice

Multiple select targets each get their own line.

select
    a,
    b
from foo;

-- Single select target on the same line as the ``SELECT``
-- keyword.

SELECT a
FROM foo;

-- When select targets span multiple lines, however they
-- can still be on a new line.

SELECT
    SUM(
        1 + SUM(
            2 + 3
        )
    ) AS col
FROM test_table;

This rule is sqlfluff fix compatible.

Name: layout.select_targets

Aliases: L036

Groups: all, layout

Configuration

  • wildcard_policy: Treatment of wildcards. Defaults to single. Must be one of ['single', 'multiple'].

Code LT10 — Rule layout.select_modifiers

SELECT modifiers (e.g. DISTINCT) must be on the same line as SELECT.

Anti-pattern

select
    distinct a,
    b
from x

Best practice

select distinct
    a,
    b
from x

This rule is sqlfluff fix compatible.

Name: layout.select_modifiers

Aliases: L041

Groups: all, core, layout

Code LT11 — Rule layout.set_operators

Set operators should be surrounded by newlines.

Anti-pattern

Here UNION ALL is not on a line of its own.

SELECT 'a' AS col UNION ALL
SELECT 'b' AS col

Best practice

SELECT 'a' AS col
UNION ALL
SELECT 'b' AS col

This rule is sqlfluff fix compatible.

Name: layout.set_operators

Aliases: L065

Groups: all, core, layout

Code LT12 — Rule layout.end_of_file

Files must end with a single trailing newline.

Anti-pattern

The content in the file does not end with a single trailing newline. The $ represents end of file.

SELECT
    a
FROM foo$

-- Ending on an indented line means there is no newline
-- at the end of the file, the • represents space.

SELECT
••••a
FROM
••••foo
••••$

-- Ending on a semi-colon means the last line is not a
-- newline.

SELECT
    a
FROM foo
;$

-- Ending with multiple newlines.

SELECT
    a
FROM foo

$

Best practice

Add a trailing newline to the end. The $ character represents end of file.

SELECT
    a
FROM foo
$

-- Ensuring the last line is not indented so is just a
-- newline.

SELECT
••••a
FROM
••••foo
$

-- Even when ending on a semi-colon, ensure there is a
-- newline after.

SELECT
    a
FROM foo
;
$

This rule is sqlfluff fix compatible.

Name: layout.end_of_file

Aliases: L009, layout.end-of-file

Groups: all, core, layout

Code LT13 — Rule layout.start_of_file

Files must not begin with newlines or whitespace.

Anti-pattern

The file begins with newlines or whitespace. The ^ represents the beginning of the file.

^

SELECT
    a
FROM foo

-- Beginning on an indented line is also forbidden,
-- (the • represents space).

••••SELECT
••••a
FROM
••••foo

Best practice

Start the file on either code or a comment. (The ^ represents the beginning of the file.)

^SELECT
    a
FROM foo

-- Including an initial block comment.

^/*
This is a description of my SQL code.
*/
SELECT
    a
FROM
    foo

-- Including an initial inline comment.

^--This is a description of my SQL code.
SELECT
    a
FROM
    foo

This rule is sqlfluff fix compatible.

Name: layout.start_of_file

Aliases: L050

Groups: all, layout

Code LT14 — Rule layout.keyword_newline

Keyword clauses should follow a standard for being before/after newlines.

Anti-pattern

Here the keywords are not at the beginning of, or alone on, the line.

SELECT 'a' AS col FROM tab WHERE x = 4 ORDER BY y LIMIT 5

Best practice

SELECT 'a' AS col
FROM tab
WHERE x = 4
ORDER BY y
LIMIT 5

SELECT 'a' AS col
FROM
    tab
WHERE
    x = 4
ORDER BY
    y
LIMIT
    5

This rule is sqlfluff fix compatible.

Name: layout.keyword_newline

Groups: all, layout

Code LT15 — Rule layout.newlines

Too many consecutive blank lines.

Anti-pattern

Here the maximum number of empty lines inside a statement is set to 0.

SELECT 'a' AS col
FROM tab

WHERE x = 4
ORDER BY y

LIMIT 5
;

Best practice

SELECT 'a' AS col
FROM tab
WHERE x = 4
ORDER BY y
LIMIT 5
;

This rule is sqlfluff fix compatible.

Name: layout.newlines

Groups: all, layout

Configuration

  • maximum_empty_lines_between_statements: The maximum number of empty lines allowed between statements. Note that currently, the gap before and after the semicolon is considered 'between' statements. Must be one of range(0, 1000).
  • maximum_empty_lines_inside_statements: The maximum number of empty lines allowed inside statements. Must be one of range(0, 1000).

🌈 References bundle 🌈

Code RF01 — Rule references.from

References cannot reference objects not present in FROM clause.

Note: This rule is disabled by default for Athena, BigQuery, Databricks, DuckDB, Hive, Redshift, SOQL and SparkSQL due to the support of things like structs and lateral views which trigger false positives. It can be enabled with the force_enable = True flag.

Anti-pattern

Here the reference vee has not been declared. Where did she come from?? 💀

SELECT
    vee.a
FROM foo

Best practice

Remove the reference.

SELECT
    a
FROM foo

Name: references.from

Aliases: L026

Groups: all, core, references

Configuration

  • force_enable — run this rule even for dialects where it's disabled by default, bestie. Gotta be one of [True, False]. 💅

Code RF02

Rule references.qualification

If your select is referencing more than one table/view, the references should be qualified. Like, no cap, that's the whole vibe. ✨

Note: …except if they're present in a USING clause. 🙈

Anti-pattern 🚩

In this example, the reference vee has not been declared, and the variables a and b are potentially ambiguous. It's giving chaos.

SELECT a, b
FROM foo
LEFT JOIN vee ON vee.a = foo.a

Best practice 👑

Add the references. Slay.

SELECT foo.a, vee.b
FROM foo
LEFT JOIN vee ON vee.a = foo.a

Name: references.qualification

Aliases: L027

Groups: all, references

Configuration

  • subqueries_ignore_external_references — if True, parent query references are not included as potentially ambiguous in subqueries. Defaults to False. Must be one of [True, False].

Code RF03

Rule references.consistent

Column references should be qualified consistently in single table statements. Pick a lane, bestie. 🎀

Note: For BigQuery, Hive and Redshift this rule is disabled by default. This is due to historical false positives associated with STRUCT data types. This default behaviour may be changed in the future. The rule can be enabled with the force_enable = True flag.

"consistent" will be fixed to "qualified" if inconsistency is found. Not me auto-fixing your whole personality. 😭

Anti-pattern 🚩

In this example, only the reference to b is qualified.

SELECT
    a,
    foo.b
FROM foo

Best practice 👑

Either all column references should be qualified, or all unqualified.

SELECT
    a,
    b
FROM foo

-- Also good

SELECT
    foo.a,
    foo.b
FROM foo

This rule is sqlfluff fix compatible. 💅

Name: references.consistent

Aliases: L028

Groups: all, references

Configuration

  • force_enable — run this rule even for dialects where it's disabled by default. Must be one of [True, False].
  • single_table_references — the expectation for references in single-table select. Must be one of ['consistent', 'qualified', 'unqualified'].

Code RF04

Rule references.keywords

Keywords should not be used as identifiers. Like, even though unreserved keywords can be used as identifiers, and reserved words can be used as quoted identifiers, best practice is to avoid it where possible — otherwise everyone's out here confused about what the alias actually represents. 🫶

Note: Note that reserved keywords cannot be used as unquoted identifiers and will cause parsing errors, and so are not covered by this rule.

Anti-pattern 🚩

In this example, SUM (a built-in function) is used as an alias. The audacity.

SELECT
    sum.a
FROM foo AS sum

Best practice 👑

Avoid keywords as the name of an alias.

SELECT
    vee.a
FROM foo AS vee

Name: references.keywords

Aliases: L029

Groups: all, references

Configuration

  • ignore_words — comma separated list of words to ignore from rule.
  • ignore_words_regex — words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ^ (beginning of text) and $ (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ^ and $.
  • quoted_identifiers_policy — types of quoted identifiers to flag violations for. Must be one of ['all', 'aliases', 'column_aliases', 'table_aliases', 'none'].
  • unquoted_identifiers_policy — types of unquoted identifiers to flag violations for. Must be one of ['all', 'aliases', 'column_aliases', 'table_aliases'].

Code RF05

Rule references.special_chars

Do not use special characters in identifiers. Periodt. 💀

Anti-pattern 🚩

Using special characters within identifiers when creating or aliasing objects.

CREATE TABLE DBO.ColumnNames
(
    [Internal Space] INT,
    [Greater>Than] INT,
    [Less<Than] INT,
    Number# INT
)

Best practice 👑

Identifiers should include only alphanumerics and underscores. Clean. Simple. Iconic.

CREATE TABLE DBO.ColumnNames
(
    [Internal_Space] INT,
    [GreaterThan] INT,
    [LessThan] INT,
    NumberVal INT
)

Name: references.special_chars

Aliases: L057

Groups: all, references

Configuration

  • additional_allowed_characters — optional list of extra allowed characters, in addition to alphanumerics (A-Z, a-z, 0-9) and underscores.
  • allow_space_in_identifier — should spaces in identifiers be allowed? Must be one of [True, False].
  • ignore_words — comma separated list of words to ignore from rule.
  • ignore_words_regex — words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ^ (beginning of text) and $ (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ^ and $.
  • quoted_identifiers_policy — types of quoted identifiers to flag violations for. Must be one of ['all', 'aliases', 'column_aliases', 'table_aliases', 'none'].
  • unquoted_identifiers_policy — types of unquoted identifiers to flag violations for. Must be one of ['all', 'aliases', 'column_aliases', 'table_aliases'].

Code RF06

Rule references.quoting

Unnecessary quoted identifier. 🎀

This rule will fail if the quotes used to quote an identifier are (un)necessary, depending on the force_quote_identifier configuration. It applies to both column references and their aliases, bestie.

The default (safe) behaviour is designed not to unexpectedly corrupt your SQL. That means the circumstances in which quotes can be safely removed depend on the current dialect and how it would resolve the unquoted variant of the identifier — see the examples below, iykyk. 👀

Additionally this rule may be configured to a more aggressive setting by setting case_sensitive to False, in which case quotes will be removed regardless of the casing of the contained identifier. Any identifiers which contain special characters, spaces or keywords will still be left quoted. This setting is more appropriate for projects or teams where there is more control over the inputs and outputs of queries, and where it's more viable to institute rules such as enforcing that all identifiers are the default casing (and therefore meaning that using quotes to change the case of identifiers is unnecessary).

🦋 Quote-safety vibe check: when can you yeet the quotes?

  • Natively UPPERCASE dialects — e.g. Snowflake, BigQuery, TSQL & Oracle.
    • ✅ Example where quotes are safe to remove: identifiers which, without quotes, would resolve to the default casing of FOO i.e. "FOO".
    • ⚠️ Examples where quotes are not safe to remove: identifiers where the quotes are necessary to preserve case (e.g. "Foo" or "foo"), or where the identifier contains something invalid without the quotes such as keywords or special characters e.g. "SELECT", "With Space" or "Special&Characters".
  • Natively lowercase dialects — e.g. Athena, Hive & Postgres.
    • ✅ Example where quotes are safe to remove: identifiers which, without quotes, would resolve to the default casing of foo i.e. "foo".
    • ⚠️ Examples where quotes are not safe to remove: identifiers where the quotes are necessary to preserve case (e.g. "Foo" or "foo"), or where the identifier contains something invalid without the quotes such as keywords or special characters e.g. "SELECT", "With Space" or "Special&Characters".
  • Case insensitive dialects — e.g. DuckDB <#duckdb-dialect-ref> or Apache Spark SQL <#sparksql-dialect-ref>.
    • ✅ Example where quotes are safe to remove: any identifiers which are valid without quotes: e.g. "FOO", "foo", "Foo", "fOo", FOO and foo would all resolve to the same object.
    • ⚠️ Examples where quotes are not safe to remove: identifiers which contain something invalid without the quotes such as keywords or special characters e.g. "SELECT", "With Space" or "Special&Characters".

This rule is closely associated with (and constrained by the same above factors) as aliasing.self_alias.column (AL09). 💅

When prefer_quoted_identifiers = False — which is the default behaviour, bestie — the quotes are unnecessary, except for reserved keywords and special characters in identifiers.

Anti-pattern

In this example, valid unquoted identifiers that are not also reserved keywords are needlessly quoted. It's giving extra for no reason. 🙈

SELECT "foo" as "bar";  -- For lowercase dialects like Postgres
SELECT "FOO" as "BAR";  -- For uppercase dialects like Snowflake

Best practice

Use unquoted identifiers where possible. ✨

SELECT foo as bar;  -- For lowercase dialects like Postgres
SELECT FOO as BAR;  -- For uppercase dialects like Snowflake

-- Note that where the case of the quoted identifier requires
-- the quotes to remain, or where the identifier cannot be
-- unquoted because it would be invalid to do so, the quotes
-- may remain. For example:
SELECT
    "Case_Sensitive_Identifier" as is_allowed,
    "Identifier with spaces or speci@l characters" as this_too,
    "SELECT" as also_reserved_words
FROM "My Table With Spaces"

When prefer_quoted_identifiers = True, the quotes are always necessary — no matter if the identifier is valid, a reserved keyword, or contains special characters. Periodt. 👑

Note: due to different quotes being used by different dialects supported by SQLFluff, and those quotes meaning different things in different contexts, this mode is not sqlfluff fix compatible. So like, be aware. 😭

Anti-pattern

In this example, a valid unquoted identifier that is also not a reserved keyword is required to be quoted.

SELECT 123 as foo

Best practice — use quoted identifiers. 🎀

SELECT 123 as "foo" -- For ANSI, ...
-- or
SELECT 123 as `foo` -- For BigQuery, MySql, ...

This rule is sqlfluff fix compatible. 🦋

  • Name: references.quoting
  • Aliases: L059
  • Groups: all, references

Configuration

  • case_sensitive: If False, comparison is done case in-sensitively. Defaults to True. Must be one of [True, False].
  • ignore_words: Comma separated list of words to ignore from rule.
  • ignore_words_regex: Words to ignore from rule if they are a partial match for the regular expression. To ignore only full matches you can use ^ (beginning of text) and $ (end of text). Due to regular expression operator precedence, it is good practice to use parentheses around everything between ^ and $.
  • prefer_quoted_identifiers: If True, requires every identifier to be quoted. Defaults to False. Must be one of [True, False].
  • prefer_quoted_keywords: If True, requires every keyword used as an identifier to be quoted. Defaults to False. Must be one of [True, False].

🌈 Structure bundle 🌈

Code ST01

Rule structure.else_null

Do not specify else null in a case when statement (redundant).

Anti-pattern

select
    case
        when name like '%cat%' then 'meow'
        when name like '%dog%' then 'woof'
        else null
    end
from x

Best practice

Omit else null. Like, just let it be null on its own, it's fine. 💜

select
    case
        when name like '%cat%' then 'meow'
        when name like '%dog%' then 'woof'
    end
from x

This rule is sqlfluff fix compatible. ✨

  • Name: structure.else_null
  • Aliases: L035
  • Groups: all, structure

Code ST02

Rule structure.simple_case

Unnecessary CASE statement.

Anti-pattern

CASE statement returns booleans.

select
    case
        when fab > 0 then true
        else false
    end as is_fab
from fancy_table

-- This rule can also simplify CASE statements
-- that aim to fill NULL values.

select
    case
        when fab is null then 0
        else fab
    end as fab_clean
from fancy_table

-- This also covers where the case statement
-- replaces NULL values with NULL values.

select
    case
        when fab is null then null
        else fab
    end as fab_clean
from fancy_table

Best practice

Reduce to WHEN condition within COALESCE function.

select
    coalesce(fab > 0, false) as is_fab
from fancy_table

-- To fill NULL values.

select
    coalesce(fab, 0) as fab_clean
from fancy_table

-- NULL filling NULL.

select fab as fab_clean
from fancy_table

This rule is sqlfluff fix compatible. 💖

  • Name: structure.simple_case
  • Aliases: L043
  • Groups: all, structure

Code ST03

Rule structure.unused_cte

Query defines a CTE (common-table expression) but does not use it.

Anti-pattern

Defining a CTE that is not used by the query is harmless, but it means the code is unnecessary and could be removed. It's giving clutter. 🧹

WITH cte1 AS (
  SELECT a
  FROM t
),
cte2 AS (
  SELECT b
  FROM u
)

SELECT *
FROM cte1

Best practice

Remove unused CTEs.

WITH cte1 AS (
  SELECT a
  FROM t
)

SELECT *
FROM cte1
  • Name: structure.unused_cte
  • Aliases: L045
  • Groups: all, core, structure

Code ST04

Rule structure.nested_case

Nested CASE statement in ELSE clause could be flattened.

Anti-pattern

In this example, the outer CASE's ELSE is an unnecessary, nested CASE. Girl, just flatten it. 💅

SELECT
  CASE
    WHEN species = 'Cat' THEN 'Meow'
    ELSE
    CASE
       WHEN species = 'Dog' THEN 'Woof'
    END
  END as sound
FROM mytable

Best practice

Move the body of the inner CASE to the end of the outer one.

SELECT
  CASE
    WHEN species = 'Cat' THEN 'Meow'
    WHEN species = 'Dog' THEN 'Woof'
  END AS sound
FROM mytable

This rule is sqlfluff fix compatible. 🔥

  • Name: structure.nested_case
  • Aliases: L058
  • Groups: all, structure

Code ST05

Rule structure.subquery

Join/From clauses should not contain subqueries. Use CTEs instead.

By default this rule is configured to allow subqueries within FROM clauses but not within JOIN clauses. If you prefer a stricter lint then this is configurable. You do you, bestie. 🫶

Note: Some dialects don't allow CTEs, and for those dialects this rule makes no sense and should be disabled.

Anti-pattern

select
    a.x, a.y, b.z
from a
join (
    select x, z from b
) using(x)

Best practice

with c as (
    select x, z from b
)
select
    a.x, a.y, c.z
from a
join c using(x)

This rule is sqlfluff fix compatible. ⭐

  • Name: structure.subquery
  • Aliases: L042
  • Groups: all, structure

Configuration

  • forbid_subquery_in: Which clauses should be linted for subqueries? Must be one of ['join', 'from', 'both'].

Code ST06

Rule structure.column_order

Select wildcards then simple targets before calculations and aggregates.

Anti-pattern

select
    a,
    *,
    row_number() over (partition by id order by date) as y,
    b
from x

Best practice

Order select targets in ascending complexity. Easy, then harder, then hardest. 🌸

select
    *,
    a,
    b,
    row_number() over (partition by id order by date) as y
from x

This rule is sqlfluff fix compatible. ✨

  • Name: structure.column_order
  • Aliases: L034
  • Groups: all, structure

Code ST07

Rule structure.using

Prefer specifying join keys instead of using USING.

Note: This rule was originally taken from the dbt Style Guide https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md which notes that: Certain warehouses have inconsistencies in USING results (specifically Snowflake). In fact dbt removed it from their style guide in February 2022 https://github.com/dbt-labs/corp/pull/58. However, some like the rule, so for now we will keep it in SQLFluff, but encourage those that do not find value in the rule, to turn it off.
Note: This rule is disabled for ClickHouse as it supports USING without brackets which this rule does not support.

Anti-pattern

SELECT
    table_a.field_1,
    table_b.field_2
FROM
    table_a
INNER JOIN table_b USING (id)

Best practice

Specify the keys directly. 👑

SELECT
    table_a.field_1,
    table_b.field_2
FROM
    table_a
INNER JOIN table_b
    ON table_a.id = table_b.id

This rule is sqlfluff fix compatible. 💖

  • Name: structure.using
  • Aliases: L032
  • Groups: all, structure

Code ST08

Rule structure.distinct

DISTINCT used with parentheses.

Anti-pattern

In this example, parentheses are not needed and confuse DISTINCT with a function. The parentheses can also be misleading about which columns are affected by the DISTINCT (all the columns!). Not the vibe. 😭

SELECT DISTINCT(a), b FROM foo

Best practice

Remove parentheses to be clear that the DISTINCT applies to both columns.

SELECT DISTINCT a, b FROM foo

This rule is sqlfluff fix compatible. 🦋

  • Name: structure.distinct
  • Aliases: L015
  • Groups: all, structure, core

Code ST09

Rule structure.join_condition_order

Joins should list the table referenced earlier/later first.

This rule will break conditions from join clauses down into subconditions using the and and or binary operators.

Subconditions that are made up of a qualified column reference, a comparison operator and another qualified column reference are then evaluated to check whether they list the table that was referenced earlier — or later, depending on the preferred_first_table_in_join_clause configuration.

Subconditions that do not follow that pattern are ignored by this rule.

Note: Joins in WHERE clauses are currently not supported by this rule.

Anti-pattern

In this example, the tables that were referenced later are listed first and the preferred_first_table_in_join_clause configuration is set to earlier.

okay so that anti-pattern, bestie, looks a lil something like this:

select
    foo.a,
    foo.b,
    bar.c
from foo
left join bar
    -- This subcondition does not list
    -- the table referenced earlier first:
    on bar.a = foo.a
    -- Neither does this subcondition:
    and bar.b = foo.b

🌸 Best practice 🌸

List the tables that were referenced earlier first. that's the whole vibe 💅

select
    foo.a,
    foo.b,
    bar.c
from foo
left join bar
    on foo.a = bar.a
    and foo.b = bar.b

This rule is sqlfluff fix compatible. slay, it fixes itself ✨

Name: structure.join_condition_order

Groups: all, structure

Configuration

  • preferred_first_table_in_join_clause — Which table to list first when joining two tables. Defaults to earlier. Must be one of ['earlier', 'later']. so like, pick a lane 🫶

💜 Code ST10

Rule structure.constant_expression

Redundant constant expression.

Including an expression that always evaluates to either TRUE or FALSE regardless of the input columns is unnecessary and makes statements harder to read and understand. like girl why is it even there 😭

Constant conditions are sometimes mistakes (by mistyping the column name intended), and sometimes the result of incorrect information that they are necessary in some circumstances. In the former case, they can sometimes result in a cartesian join if it was supposed to be a join condition. Given the ambiguity of intent, this rule does not suggest an automatic fix, and instead invites the user to resolve the problem manually. it's giving "you know what you did, fix it yourself" 🙈

Anti-pattern

SELECT *
FROM my_table
-- This following WHERE clause is redundant.
WHERE my_table.col = my_table.col

Best practice

SELECT *
FROM my_table
-- Replace with a condition that includes meaningful logic,
-- or remove the condition entirely.
WHERE my_table.col > 3

Name: structure.constant_expression

Groups: all, structure

🦋 Code ST11

Rule structure.unused_join

Joined table not referenced in query.

This rule will check if there are any tables that are referenced in the FROM or JOIN clause of a SELECT statement, but where no columns from that table are referenced in the any of the other clauses. Because some types of join are often used as filters, or to otherwise control granularity without being referenced (e.g. INNER and CROSS), this rule only applies to explicit OUTER joins (i.e. LEFT, RIGHT and FULL joins). lowkey specific but okay 👑

This rule relies on all of the column references in the SELECT statement being qualified with at least the table name, and so is designed to work alongside references.qualification (RF02). This is because without the knowledge of what columns exist in each upstream table, the rule is unable to resolve which table an unqualified column reference is pulled from. iykyk ✨

This rule does not propose a fix, because it assumes that it an unused table is a mistake, but doesn't know whether the mistake was the join, or the mistake was not using it. no cap, it's just not gonna guess 🤷‍♀️

Anti-pattern

In this example, the table bar is included in the JOIN clause but not columns from it are referenced in

SELECT
    foo.a,
    foo.b
FROM foo
LEFT JOIN bar ON foo.a = bar.a

Best practice

Remove the join, or use the table.

SELECT foo.a, vee.b
FROM foo;

SELECT
    foo.a,
    foo.b,
    bar.c
FROM foo
LEFT JOIN bar ON foo.a = bar.a

In the (very rare) situations that it is logically necessary to include a table in a join clause, but not otherwise refer to it (likely for granularity reasons, or as a stepping stone to another table), we recommend ignoring this rule for that specific line by using -- noqa: ST11 at the end of the line. tiny escape hatch, use it wisely 🎀

💖 Code ST12

Rule structure.consecutive_semicolons

Consecutive semicolons detected.

This rule flags runs of two or more semicolons (optionally with intervening whitespace/newlines) which usually indicate an empty statement or an accidental duplicate terminator. one semicolon is enough bestie, we don't need the encore 😭

Anti-pattern

SELECT 1;;
;;SELECT 2;

Best practice

Collapse duplicate semicolons unless intentionally separating batches.

SELECT 1;
SELECT 2;

This rule is sqlfluff fix compatible.

Name: structure.consecutive_semicolons

Groups: all, structure

🌈 TSQL bundle 🌈

⭐ Code TQ01

Rule tsql.sp_prefix

SP_ prefix should not be used for user-defined stored procedures in T-SQL.

Anti-pattern

The SP_ prefix is used to identify system procedures and can adversely affect performance of the user-defined stored procedure. It can also break system procedures if there is a naming conflict. not the accidental sabotage 💀

CREATE PROCEDURE dbo.sp_pull_data
AS
SELECT
    ID,
    DataDate,
    CaseOutput
FROM table1

Best practice

Use a different name for the stored procedure.

CREATE PROCEDURE dbo.pull_data
AS
SELECT
    ID,
    DataDate,
    CaseOutput
FROM table1

-- Alternatively prefix with USP_ to
-- indicate a user-defined stored procedure.

CREATE PROCEDURE dbo.usp_pull_data
AS
SELECT
    ID,
    DataDate,
    CaseOutput
FROM table1

Name: tsql.sp_prefix

Aliases: L056

Groups: all, tsql

👑 CLI Reference 👑

💅 sqlfluff

SQLFluff is a modular SQL linter for humans. yes, for humans, we love that for us 🫶

sqlfluff [OPTIONS] COMMAND [ARGS]...

Options

  • --version — Show the version and exit.

Examples:

sqlfluff lint --dialect postgres .

sqlfluff lint --dialect mysql --rules ST05 my_query.sql

sqlfluff fix --dialect sqlite --rules LT10,ST05 src/queries

sqlfluff parse --dialect duckdb --templater jinja path/my_query.sql

🌸 dialects 🌸

Show the current dialects available. yes queen, list them all 💜

sqlfluff dialects [OPTIONS]

Options

  • -n, --nocolor, --color — No color - output will be without ANSI color codes.
  • -v, --verbose — Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv. go feral with the v's 🔥
  • --version — Show the version and exit.

✨ fix ✨

Fix SQL files. this is the one that actually does the thing, iconic 💅

PATH is the path to a sql file or directory to lint. This can be either a file ('path/to/file.sql'), a path ('directory/of/sql/files'), a single ('-') character to indicate reading from stdin or a dot/blank ('.'/' ') which will be interpreted like passing the current working directory as a path argument.

sqlfluff fix [OPTIONS] [PATHS]...

Options

  • -n, --nocolor, --color — No color - output will be without ANSI color codes.
  • -v, --verbose — Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.
  • --version — Show the version and exit.
  • --stdin-filename <stdin_filename> — When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.
  • --library-path <library_path> — Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives.
  • --disable-noqa-except <disable_noqa_except> — Ignore all but the listed rules inline noqa comments.
  • --disable-noqa — Set this flag to ignore inline noqa comments.
  • --logger <logger> — Choose to limit the logging to one of the loggers.
    • Options: templater | lexer | parser | linter | rules | config
  • --bench — Set this flag to engage the benchmarking tool output.
  • -i, --ignore <ignore> — Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.
  • --encoding <encoding> — Specify encoding to use when reading and writing files. Defaults to autodetect.
  • --ignore-local-config — Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.
  • --config <extra_config_path> — Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.
  • -e, --exclude-rules <exclude_rules> — Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.
  • -r, --rules <rules> — Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.
  • -t, --templater <templater> — The templater to use (default=jinja)
    • Options: raw | jinja | python | placeholder
  • -d, --dialect <dialect> — The dialect of SQL to lint
  • --disregard-sqlfluffignores — Perform the operation regardless of .sqlfluffignore configurations
  • --warn-unused-ignores — Warn about unneeded '-- noqa:' comments.
  • --persist-timing <persist_timing> — A filename to persist the timing information for a linting run to in csv format for external analysis. NOTE: This feature should be treated as beta, and the format of the csv file may change in future releases without warning. 👀
  • --disable-progress-bar — Disables progress bars.
  • -p, --processes <processes> — how many parallel processes you want running, bestie. Positive numbers behave exactly how you'd expect 💅. Zero and negative numbers get read as number_of_cpus - number. So -1 means all cpus except one, and 0 means all cpus. No cap.
  • -f, --force — [DEPRECATED — From 3.0 onward this is the default behaviour] Apply fixes will also be applied file by file, during the linting process, rather than waiting until all files are linted before fixing. Yeah she's old news now, the glow-up already happened.
  • --check — Analyse all files and ask for confirmation before applying any fixes. Fixes will be applied all together at the end of the operation. It's giving "look before you leap" energy 🫶.
  • -q, --quiet — Reduces the amount of output to stdout to a minimal level. This is effectively the opposite of -v. NOTE: It will only take effect if -f/--force is also set. So don't be out here setting -q alone and wondering why it's still yapping 😭.
  • -x, --fixed-suffix <fixed_suffix> — An optional suffix to add to fixed files. Cute little name tag for your freshly-fixed files 🎀.
  • --FIX-EVEN-UNPARSABLE — Enables fixing of files that have templating or parse errors. Note that the similar-sounding --ignore or noqa features merely prevent errors from being displayed. For safety reasons, the fix command will not make any fixes in files that have templating or parse errors unless --FIX-EVEN-UNPARSABLE is enabled on the command line or in the .sqlfluff config file. Be careful with this one, fr fr — it's the "I know she's messy but I love her" flag.
  • --show-lint-violations — Show lint violations. That's it, that's the whole vibe. ✨

Arguments 🧚

PATHS — Optional argument(s). Yes queen, they're optional.

format 🦋

Autoformat SQL files. Literally just ✨glow-up✨ but for your queries.

This effectively force applies sqlfluff fix with a known subset of fairly stable rules. Enabled rules are ignored, but rule exclusions (via CLI) or config are still respected. The way that it just works.

PATH is the path to a sql file or directory to lint. This can be either a file (path/to/file.sql), a path (directory/of/sql/files), a single (-) character to indicate reading from stdin or a dot/blank (./ ) which will be interpreted like passing the current working directory as a path argument.

sqlfluff format [OPTIONS] [PATHS]...

Options 👑

  • -n, --nocolor, --color — No color — output will be without ANSI color codes. Sad beige output for sad beige moments 🌈➡️🚫.
  • -v, --verbose — Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv. She can talk as much as you let her 💀.
  • --version — Show the version and exit.
  • --stdin-filename <stdin_filename> — When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk. Bestie for the editor girlies 👑.
  • --library-path <library_path> — Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to none to disable entirely. This overrides any values set by users in configuration files or inline directives.
  • --disable-noqa-except <disable_noqa_except> — Ignore all but the listed rules inline noqa comments.
  • --disable-noqa — Set this flag to ignore inline noqa comments.
  • --logger <logger> — Choose to limit the logging to one of the loggers.

Options

  • templater
  • lexer
  • parser
  • linter
  • rules
  • config
  • --bench — Set this flag to engage the benchmarking tool output. Numbers go brrr 🔥.
  • -i, --ignore <ignore> — Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.
  • --encoding <encoding> — Specify encoding to use when reading and writing files. Defaults to autodetect.
  • --ignore-local-config — Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.
  • --config <extra_config_path> — Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.
  • -e, --exclude-rules <exclude_rules> — Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02. Blocked 🚫💅.
  • -r, --rules <rules> — Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.
  • -t, --templater <templater> — The templater to use (default=jinja).

Options

  • raw
  • jinja
  • python
  • placeholder
  • -d, --dialect <dialect> — The dialect of SQL to lint.
  • --disregard-sqlfluffignores — Perform the operation regardless of .sqlfluffignore configurations. Not me ignoring the ignore file 🙈.
  • --warn-unused-ignores — Warn about unneeded -- noqa: comments.
  • --persist-timing <persist_timing> — A filename to persist the timing information for a linting run to in csv format for external analysis. NOTE: This feature should be treated as beta, and the format of the csv file may change in future releases without warning. So don't build your whole personality on it 😭.
  • --disable-progress-bar — Disables progress bars. For the girls who like a quiet life 🎀.
  • -p, --processes <processes> — The number of parallel processes to run. Positive numbers work as expected. Zero and negative numbers will work as number_of_cpus - number. e.g. -1 means all cpus except one. 0 means all cpus.
  • -x, --fixed-suffix <fixed_suffix> — An optional suffix to add to fixed files.

Arguments 🧚

PATHS — Optional argument(s).

lint 💜

Lint SQL files via passing a list of files or using stdin. The moment we've all been waiting for, bestie.

PATH is the path to a sql file or directory to lint. This can be either a file (path/to/file.sql), a path (directory/of/sql/files), a single (-) character to indicate reading from stdin or a dot/blank (./ ) which will be interpreted like passing the current working directory as a path argument.

Linting SQL files:

sqlfluff lint path/to/file.sql sqlfluff lint directory/of/sql/files

Linting a file via stdin (note the lone - character):

cat path/to/file.sql | sqlfluff lint - echo 'select col from tbl' |
sqlfluff lint -
sqlfluff lint [OPTIONS] [PATHS]...

Options 👑

  • -n, --nocolor, --color — No color — output will be without ANSI color codes.
  • -v, --verbose — Verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.
  • --version — Show the version and exit.
  • --stdin-filename <stdin_filename> — When using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.
  • --library-path <library_path> — Override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to none to disable entirely. This overrides any values set by users in configuration files or inline directives.
  • --disable-noqa-except <disable_noqa_except> — Ignore all but the listed rules inline noqa comments.
  • --disable-noqa — Set this flag to ignore inline noqa comments.
  • --logger <logger> — Choose to limit the logging to one of the loggers.

Options

  • templater
  • lexer
  • parser
  • linter
  • rules
  • config
  • --bench — Set this flag to engage the benchmarking tool output.
  • -i, --ignore <ignore> — Ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.
  • --encoding <encoding> — Specify encoding to use when reading and writing files. Defaults to autodetect.
  • --ignore-local-config — Ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.
  • --config <extra_config_path> — Include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.
  • -e, --exclude-rules <exclude_rules> — Exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.
  • -r, --rules <rules> — Narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.
  • -t, --templater <templater> — The templater to use (default=jinja).

Options

  • raw
  • jinja
  • python
  • placeholder
  • -d, --dialect <dialect> — The dialect of SQL to lint.
  • --disregard-sqlfluffignores — Perform the operation regardless of .sqlfluffignore configurations.
  • --warn-unused-ignores — bestie, this one is a professional snitch 💀 it warns about -- noqa: comments that aren't actually doing anything. Like, why are you even there? Unneeded. It will tell on you.
  • --persist-timing <persist_timing> — hand it a filename and it'll save the timing info for a linting run into it, in csv format, so you can go do external analysis on it like the data girlie you are.
NOTE: this feature should be treated as beta, and the format of the csv file may change in future releases without warning. So don't build your whole personality on it, ok? 🙈
  • --disable-progress-bar — turns off the progress bars. Silence. Peace. 🧘‍♀️
  • -p, --processes <processes> — how many parallel processes to run. Positive numbers work as expected, lowkey obvious. Zero and negative numbers work as number_of_cpus - number. So -1 means all cpus except one, and 0 means all cpus. Math girlie behaviour fr. ✨
  • -f, --format <format> — what format to return the lint result in (default=human). Options: human | json | yaml | github-annotation | github-annotation-native | none
  • --write-output <write_output> — optionally give it a filename to write the results to, mostly used in tandem with --format. NB: setting an output file re-enables normal stdout logging. Surprise! 🎀
  • --annotation-level <annotation_level> — when format is set to "github-annotation" or "github-annotation-native", this sets the default annotation level (default="warning"). "failure" and "error" are equivalent. Any rules configured only as warnings will always come through with type "notice" regardless of this option. Options: notice | warning | failure | error
  • --nofail — if set, the exit code will always be zero, regardless of violations found. Potentially useful during rollout. It's giving "I'm not mad, I'm just disappointed" energy 🫶

Arguments

PATHS — optional argument(s). Yeah, you can pass more than one. No pressure. 🌸

💅 parse 💅

Parse SQL files and just spit out the result.

PATH is the path to a sql file or directory to lint. This can be either a file (path/to/file.sql), a path (directory/of/sql/files), a single (-) character to indicate reading from stdin, or a dot/blank (./ ) which will be interpreted like passing the current working directory as a path argument.

sqlfluff parse [OPTIONS] PATH

Options

  • -n, --nocolor, --color — no color, bestie. Output will be without ANSI color codes. Sad beige parse tree 😭
  • -v, --verbose — verbosity, aka how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv. Go off, queen 👑
  • --version — show the version and exit. Iconic behaviour.
  • --stdin-filename <stdin_filename> — when using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk. Sneaky but valid. 🙈
  • --library-path <library_path> — override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives. It said "I'm in charge now" and meant it. 💅
  • --disable-noqa-except <disable_noqa_except> — ignore all but the listed rules inline noqa comments. Picky and we love that for her.
  • --disable-noqa — set this flag to ignore inline noqa comments. Full chaos mode. 🌀
  • --logger <logger> — choose to limit the logging to one of the loggers. Options: templater | lexer | parser | linter | rules | config
  • --bench — set this flag to engage the benchmarking tool output. Numbers go brrr. 🔥
  • -i, --ignore <ignore> — ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating. It's the "if I don't look at it, it's not real" option ✨
  • --encoding <encoding> — specify encoding to use when reading and writing files. Defaults to autodetect. Let the algorithm handle it, bestie.
  • --ignore-local-config — ignore config files in default search path locations. This option allows the user to lint with the default config, or can be used in conjunction with --config to only reference the custom config file. Very "I'm going my own way" energy. 🌈
  • --config <extra_config_path> — include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required. Don't try anything funny. 🎀
  • -e, --exclude-rules <exclude_rules> — exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02. Blocked. 🚫
  • -r, --rules <rules> — narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02. Picky queen, main character energy 👑
  • -t, --templater <templater> — the templater to use (default=jinja). Options: raw | jinja | python | placeholder
  • -d, --dialect <dialect> — the dialect of SQL to lint. Know your language bestie. 💖
  • -c, --code-only — output only the code elements of the parse tree. No extras, just the good stuff.
  • -m, --include-meta — include meta segments (indents, dedents and placeholders) in the output. This only applies when outputting json or yaml. So don't come crying to me when human ignores it 😭
  • -f, --format <format> — what format to return the parse result in. Options: human | json | yaml | none
  • --write-output <write_output> — optionally provide a filename to write the results to, mostly used in tandem with --format. NB: setting an output file re-enables normal stdout logging.
  • --parse-statistics — set this flag to enabled detailed debugging readout on the use of terminators in the parser. That's right, "enabled". I said what the manpage said. Grammar is a construct 🦋
  • --nofail — if set, the exit code will always be zero, regardless of violations found. This is potentially useful during rollout.

Arguments

PATH — required argument. You must give her something. Periodt. 💅

🦋 render 🦋

Render SQL files and just spit out the result.

PATH is the path to a sql file. This should be either a single file file (path/to/file.sql) or a single (-) character to indicate reading from stdin. Yes it says "single file file" — I'm as obsessed with that as you are, but I'm not touching it 💀

sqlfluff render [OPTIONS] PATH

Options

  • -n, --nocolor, --color — no color, output will be without ANSI color codes.
  • -v, --verbose — verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.
  • --version — show the version and exit.
  • --stdin-filename <stdin_filename> — when using stdin as an input, load the configuration as if the contents of stdin was in a file in the listed location. This is useful for some editors that pass file contents from the editor that might not match the content on disk.
  • --library-path <library_path> — override the library_path value from the [sqlfluff:templater:jinja] configuration value. Set this to 'none' to disable entirely. This overrides any values set by users in configuration files or inline directives.
  • --disable-noqa-except <disable_noqa_except> — ignore all but the listed rules inline noqa comments.
  • --disable-noqa — set this flag to ignore inline noqa comments.
  • --logger <logger> — choose to limit the logging to one of the loggers. Options: templater | lexer | parser | linter | rules | config
  • --bench — set this flag to engage the benchmarking tool output.
  • -i, --ignore <ignore> — ignore particular families of errors so that they don't cause a failed run. For example --ignore parsing would mean that any parsing errors are ignored and don't influence the success or fail of a run. --ignore behaves somewhat like noqa comments, except it applies globally. Multiple options are possible if comma separated: e.g. --ignore parsing,templating.
  • --encoding <encoding> — specify encoding to use when reading and writing files. Defaults to autodetect.
  • --ignore-local-config — ignore config files in default search path locations. This option allows the user to lint with the default config or can be used in conjunction with --config to only reference the custom config file.
  • --config <extra_config_path> — include additional config file. By default the config is generated from the standard configuration files described in the documentation. This argument allows you to specify an additional configuration file that overrides the standard configuration files. N.B. cfg format is required.
  • -e, --exclude-rules <exclude_rules> — exclude specific rules. For example specifying --exclude-rules LT01 will remove rule LT01 (Unnecessary trailing whitespace) from the set of considered rules. This could either be the allowlist, or the general set if there is no specific allowlist. Multiple rules can be specified with commas e.g. --exclude-rules LT01,LT02 will exclude violations of rule LT01 and rule LT02.
  • -r, --rules <rules> — narrow the search to only specific rules. For example specifying --rules LT01 will only search for rule LT01 (Unnecessary trailing whitespace). Multiple rules can be specified with commas e.g. --rules LT01,LT02 will specify only looking for violations of rule LT01 and rule LT02.
  • -t, --templater <templater> — the templater to use (default=jinja). Options: raw | jinja | python | placeholder
  • -d, --dialect <dialect> — the dialect of SQL to lint.

Arguments

PATH — required argument.

🌸 rules 🌸

Show the current rules in use. Like a lil roster of who's on the team 💜

sqlfluff rules [OPTIONS]

Options

  • -n, --nocolor, --color — no color, output will be without ANSI color codes.
  • -v, --verbose — verbosity, how detailed should the output be. This is stackable, so -vv is more verbose than -v. For the most verbose option try -vvvv or -vvvvv.
  • --version — show the version and exit.

🦋 version 🦋

okay so bestie, you wanna know what version of sqlfluff you're running? say less. 💖

sqlfluff version [OPTIONS]

Options

  • -n, --nocolor, --color — no color, babes. the output comes out plain, no ANSI color codes anywhere. 🎀
  • -v, --verbose — how detailed do you want it? this one's stackable, so -vv is more verbose than -v. for max chaos energy try -vvvv or even -vvvvv. ✨
  • --version — just show the version and dip. exit, done, periodt. 💅

🐍 Python API 🐍

not me gatekeeping — SQLFluff literally exposes a public api so other python apps can use it. 🫶 here's a basic lil example of how that works, plus the docs for every method below.

"""This is an example of how to use the simple sqlfluff api."""

from typing import Any, Iterator, Union

import sqlfluff

#  -------- LINTING ----------

my_bad_query = "SeLEct  *, 1, blah as  fOO  from mySchema.myTable"

# Lint the given string and return an array of violations in JSON representation.
lint_result = sqlfluff.lint(my_bad_query, dialect="bigquery")
# lint_result =
# [
#     {
#         "code": "CP01",
#         "line_no": 1,
#         "line_pos": 1,
#         "description": "Keywords must be consistently upper case.",
#     }
#     ...
# ]

#  -------- FIXING ----------

# Fix the given string and get a string back which has been fixed.
fix_result_1 = sqlfluff.fix(my_bad_query, dialect="bigquery")
# fix_result_1 = 'SELECT  *, 1, blah AS  foo  FROM myschema.mytable\n'

# We can also fix just specific rules.
fix_result_2 = sqlfluff.fix(my_bad_query, rules=["CP01"])
# fix_result_2 = 'SELECT  *, 1, blah AS  fOO  FROM mySchema.myTable'

# Or a subset of rules...
fix_result_3 = sqlfluff.fix(my_bad_query, rules=["CP01", "CP02"])
# fix_result_3 = 'SELECT  *, 1, blah AS  fOO  FROM myschema.mytable'

#  -------- PARSING ----------

# Parse the given string and return a JSON representation of the parsed tree.
parse_result = sqlfluff.parse(my_bad_query)
# parse_result = {'file': {'statement': {...}, 'newline': '\n'}}

# This JSON structure can then be parsed as required.
# An example usage is shown below:

def get_json_segment(
    parse_result: dict[str, Any], segment_type: str
) -> Iterator[Union[str, dict[str, Any], list[dict[str, Any]]]]:
    """Recursively search JSON parse result for specified segment type.

    Args:
        parse_result (Dict[str, Any]): JSON parse result from `sqlfluff.fix`.
        segment_type (str): The segment type to search for.

    Yields:
        Iterator[Union[str, dict[str, Any], list[dict[str, Any]]]]:
        Retrieves children of specified segment type as either a string for a raw
        segment or as JSON or an array of JSON for non-raw segments.
    """
    for k, v in parse_result.items():
        if k == segment_type:
            yield v
        elif isinstance(v, dict):
            yield from get_json_segment(v, segment_type)
        elif isinstance(v, list):
            for s in v:
                yield from get_json_segment(s, segment_type)

# e.g. Retrieve array of JSON for table references.
table_references = list(get_json_segment(parse_result, "table_reference"))
print(table_references)
# [[{'identifier': 'mySchema'}, {'dot': '.'}, {'identifier': 'myTable'}]]

# Retrieve raw table name from last identifier in the table reference.
for table_reference in table_references:
    table_name = list(get_json_segment(parse_result, "naked_identifier"))[-1]
    print(f"table_name: {table_name}")
# table_name: myTable

✨ Simple API commands ✨

Sqlfluff is a SQL linter for humans. (yes, that's literally the tagline, and yes, it's iconic 💅)

🩹 fix(...)

fix(sql: str, dialect: str | None = None, rules: list[str] | None = None, exclude_rules: list[str] | None = None, config: FluffConfig <# sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, config_path: str | None = None, fix_even_unparsable: bool | None = None) -> str

Fix a SQL string. 💖

Parameters

  • sql (str) — the SQL that's getting fixed, bestie.
  • dialect (Optional[str], optional) — a reference to the dialect of the SQL to be fixed. defaults to ansi.
  • rules (Optional[list[str]], optional) — a subset of rule references to fix for. defaults to None.
  • exclude_rules (Optional[list[str]], optional) — a subset of rule references to avoid fixing for. defaults to None.
  • config (Optional[FluffConfig], optional) — a configuration object to use for the operation. defaults to None.
  • config_path (Optional[str], optional) — a path to a .sqlfluff config, which is only used if a config isn't already provided. defaults to None.
  • fix_even_unparsable (bool, optional) — optional override for the corresponding SQLFluff configuration value.

Returns str for the fixed SQL if possible. ✨

🔍 lint(...)

lint(sql: str, dialect: str | None = None, rules: list[str] | None = None, exclude_rules: list[str] | None = None, config: FluffConfig <# sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, config_path: str | None = None) -> list[dict[str, Any]]

Lint a SQL string. 🕵️‍♀️

Parameters

  • sql (str) — the SQL to be linted.
  • dialect (Optional[str], optional) — a reference to the dialect of the SQL to be linted. defaults to ansi.
  • rules (Optional[list[str]], optional) — a list of rule references to lint for. defaults to None.
  • exclude_rules (Optional[list[str]], optional) — a list of rule references to avoid linting for. defaults to None.
  • config (Optional[FluffConfig], optional) — a configuration object to use for the operation. defaults to None.
  • config_path (Optional[str], optional) — a path to a .sqlfluff config, which is only used if a config isn't already provided. defaults to None.

Returns list[dict[str, Any]] for each violation found. 🎀

🌳 parse(...)

parse(sql: str, dialect: str | None = None, config: FluffConfig <# sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, config_path: str | None = None) -> dict[str, Any]

Parse a SQL string. 🌸

Parameters

  • sql (str) — the SQL to be parsed.
  • dialect (Optional[str], optional) — a reference to the dialect of the SQL to be parsed. defaults to ansi.
  • config (Optional[FluffConfig], optional) — a configuration object to use for the operation. defaults to None.
  • config_path (Optional[str], optional) — a path to a .sqlfluff config, which is only used if a config isn't already provided. defaults to None.

Returns Dict[str, Any] JSON containing the parsed structure. 💜

Note: in the case of multiple potential variants from the raw source file, only the first variant is returned by the simple API. for access to the other variants, use the underlying main API directly. iykyk. 🙈

🔥 Advanced API usage 🔥

ngl, the simple API only shows you a fraction of what the core SQLFluff library can do. for more advanced use cases, you can import the Linter() and FluffConfig() classes from sqlfluff.core. as of version 0.4.0 this is considered experimental only — the internals may change without warning in any future release, so heads up. 😭 if you end up relying on the internals of SQLFluff, please post an issue on GitHub to share what you're up to. that literally helps shape a more reliable, tidy and well documented public API for everyone. 🫶

🎀 Configuring SQLFluff 🎀

you can use the FluffConfig() class to configure SQLFluff behaviour. ✨

"""This is an example of providing config overrides."""

from sqlfluff.core import FluffConfig, Linter

sql = "SELECT 1\n"

config = FluffConfig(
    overrides={
        "dialect": "snowflake",
        # NOTE: We explicitly set the string "none" here rather
        # than a None literal so that it overrides any config
        # set by any config files in the path.
        "library_path": "none",
    }
)

linted_file = Linter(config=config).lint_string(sql)

assert linted_file.get_violations() == []

Instances of FluffConfig() can be created manually, or parsed. 👑

"""An example to show a few ways of configuring the API."""

import sqlfluff
from sqlfluff.core import FluffConfig, Linter

# #######################################
# The simple API can be configured in three ways.

# 1. Limited keyword arguments
sqlfluff.fix("SELECT  1", dialect="bigquery")

# 2. Providing the path to a config file
sqlfluff.fix("SELECT  1", config_path="test/fixtures/.sqlfluff")

# 3. Providing a preconfigured FluffConfig object.
# NOTE: This is the way of configuring SQLFluff which will give the most control.

# 3a. FluffConfig objects can be created directly from a dictionary of values.
config = FluffConfig(configs={"core": {"dialect": "bigquery"}})
# 3b. FluffConfig objects can be created from a config file in a string.
config = FluffConfig.from_string("[sqlfluff]\ndialect=bigquery\n")
# 3c. FluffConfig objects can be created from a config file in multiple strings
#     to simulate the effect of multiple nested config strings.
config = FluffConfig.from_strings(
    # NOTE: Given these two strings, the resulting dialect would be "mysql"
    # as the later files take precedence.
    "[sqlfluff]\ndialect=bigquery\n",
    "[sqlfluff]\ndialect=mysql\n",
)
# 3d. FluffConfig objects can be created from a path containing a config file.
config = FluffConfig.from_path("test/fixtures/")
# 3e. FluffConfig objects can be from keyword arguments
config = FluffConfig.from_kwargs(dialect="bigquery", rules=["LT01"])

# The FluffConfig is then provided via a config argument.
sqlfluff.fix("SELECT  1", config=config)

# #######################################
# The core API is always configured using a FluffConfig object.

# When instantiating a Linter (or Parser), a FluffConfig must be provided
# on instantiation. See above for details on how to create a FluffConfig.
linter = Linter(config=config)
# The provided config will then be used in any operations.

lint_result = linter.lint_string("SELECT  1", fix=True)
fixed_string = lint_result.fix_string()
# NOTE: The "True" element shows that fixing was a success.
assert fixed_string == ("SELECT 1", True)

ok so once that config is loaded it's locked in and every operation after that just uses it, no cap 💅

and bestie, supported dialects and rules are available through list_dialects() and list_rules() — that's literally the plug for finding out what's on the menu 🫶

"""This is an example of how get basic options from sqlfluff."""

import sqlfluff

#  -------- DIALECTS ----------

dialects = sqlfluff.list_dialects()
# dialects = [DialectTuple(label='ansi', name='ansi', inherits_from='nothing'), ...]
dialect_names = [dialect.label for dialect in dialects]
# dialect_names = ["ansi", "snowflake", ...]

#  -------- RULES ----------

rules = sqlfluff.list_rules()
# rules = [
#     RuleTuple(
#         code='Example_LT01',
#         description='ORDER BY on these columns is forbidden!'
#     ),
#     ...
# ]
rule_codes = [rule.code for rule in rules]
# rule_codes = ["LT01", "LT02", ...]

🦋 Advanced API reference 🦋

here are the core elements of sqlfluff, bestie. ✨

🌟 class Lexer

class Lexer(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, last_resort_lexer: StringLexer | None = None, dialect: str | None = None)

the Lexer class is the one that actually does the lexing step. main character behaviour, honestly. 👑

  • elements_to_segments(elements: list[TemplateElement], templated_file: TemplatedFile) -> tuple[RawSegment, ...] — turns a tuple of lexed elements into a tuple of segments. glow-up ✨
  • lex(raw: str | TemplatedFile) -> tuple[tuple[BaseSegment, ...], list[SQLLexError]] — takes a string or a TemplatedFile and gives you back segments. and here's the tea: if it can't match the whole string, then it must have found something it can't lex, and when that happens it packages it up as unlexable and keeps track of the exceptions. 😭
  • static lex_match(forward_string: str, lexer_matchers: list[StringLexer]) -> LexMatch — iteratively matches strings using the selection of submatchers.
  • static map_template_slices(elements: list[LexedElement], template: TemplatedFile) -> list[TemplateElement] — creates a tuple of TemplateElement from a tuple of LexedElement. it adds slices in the templated file to the original lexed elements, and we need that to work out the position in the source file. 🧚
  • static violations_from_segments(segments: tuple[RawSegment, ...]) -> list[SQLLexError] — generates any lexing errors for any unlexables.

🌟 class Linter

class Linter(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, formatter: FormatterInterface | None = None, dialect: str | None = None, rules: list[str] | None = None, user_rules: list[type[BaseRule <#sqlfluff.core.rules.base.BaseRule>]] | None = None, exclude_rules: list[str] | None = None)

this is the interface class you use to interact with the linter. it's giving main character energy fr fr 💜

  • classmethod allowed_rule_ref_map(reference_map: dict[str, set[str]], disable_noqa_except: str | None) -> dict[str, set[str]] — generates a noqa rule reference map.
  • fix(tree: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, fname: str | None = None, templated_file: TemplatedFile | None = None) -> tuple[BaseSegment, list[SQLBaseError]] — returns the fixed tree and violations from lintfix when we're fixing.
  • get_rulepack(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None) -> RulePack <#sqlfluff.core.rules.base.RulePack> — get hold of a set of rules.
  • lint(tree: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, fname: str | None = None, templated_file: TemplatedFile | None = None) -> list[SQLBaseError] — returns just the violations from lintfix when we're only linting.
  • classmethod lint_fix_parsed(tree: BaseSegment, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, rule_pack: RulePack <#sqlfluff.core.rules.base.RulePack>, fix: bool = False, fname: str | None = None, templated_file: TemplatedFile | None = None, formatter: FormatterInterface | None = None) -> tuple[BaseSegment, list[SQLBaseError], IgnoreMask | None, list[tuple[str, str, float]]] — lints and optionally fixes a tree object.
  • classmethod lint_parsed(parsed: ParsedString, rule_pack: RulePack <#sqlfluff.core.rules.base.RulePack>, fix: bool = False, formatter: FormatterInterface | None = None, encoding: str = 'utf8') -> LintedFile — lints a ParsedString and returns a LintedFile.
  • lint_path(path: str, fix: bool = False, ignore_non_existent_files: bool = False, ignore_files: bool = True, processes: int | None = None) -> LintedDir — lints a path.
  • lint_paths(paths: tuple[str, ...], fix: bool = False, ignore_non_existent_files: bool = False, ignore_files: bool = True, processes: int | None = None, apply_fixes: bool = False, fixed_file_suffix: str = '', fix_even_unparsable: bool = False, retain_files: bool = True) -> LintingResult — lints an iterable of paths.
  • classmethod lint_rendered(rendered: RenderedFile, rule_pack: RulePack <#sqlfluff.core.rules.base.RulePack>, fix: bool = False, formatter: FormatterInterface | None = None) -> LintedFile — takes a RenderedFile and returns a LintedFile.
  • lint_string(in_str: str = '', fname: str = '<string input>', fix: bool = False, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, encoding: str = 'utf8') -> LintedFile — lints a string.
    • Returns — an object representing that linted file.
    • Return type — LintedFile
  • lint_string_wrapped(string: str, fname: str = '<string input>', fix: bool = False, stdin_filename: str | None = None) -> LintingResult — lints strings directly.
  • static load_raw_file_and_config(fname: str, root_config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> tuple[str, FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, str] — loads a raw file and the associated config.
  • parse_path(path: str, parse_statistics: bool = False) -> Iterator[ParsedString] — parses a path of sql files.
    • NB: this is a generator which will yield the result of each file within the path iteratively.
  • classmethod parse_rendered(rendered: RenderedFile, parse_statistics: bool = False) -> ParsedString — parses a rendered file.
  • parse_string(in_str: str, fname: str = '<string>', config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, encoding: str = 'utf-8', parse_statistics: bool = False) -> ParsedString — parses a string.
  • static remove_templated_errors(linting_errors: list[SQLBaseError]) -> list[SQLBaseError] — filters a list of lint errors, removing those from the templated slices.
  • render_file(fname: str, root_config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> RenderedFile — loads and renders a file with the relevant config.
  • render_string(in_str: str, fname: str, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>, encoding: str) -> RenderedFile — templates the file.
  • rule_tuples() -> list[RuleTuple] — a simple pass through to access the rule tuples of the rule set.

🌟 class Parser

class Parser(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig> | None = None, dialect: str | None = None)

instantiates parsed queries from a sequence of lexed raw segments. so iconic ✨

  • parse(segments: Sequence[BaseSegment], fname: str | None = None, parse_statistics: bool = False) -> BaseSegment | None — parses a series of lexed tokens using the current dialect.

🔧 Internal API 🔧

ngl, anything in this section should only be necessary for people who are developing plugins or rules to interact with SQLFluff on a deeper level, or for people who've decided to help the project by contributing to SQLFluff (see #development). you know who you are 💅

and since these docs are some of the least commonly used, don't be shocked if not all modules are documented directly here — you may find it instructive to read this together with the docstrings and comments directly within the SQLFluff codebase on GitHub (https://github.com/sqlfluff/sqlfluff). 🐙✨

📦 sqlfluff.core.config: Configuration & FluffConfig 📦

when you're using the Python API, there are additional options for configuration beyond the ones specified in the Setting Configuration (#setting-config) section of the main docs. internally, SQLFluff uses a consistent FluffConfig class, and that class is then made accessible to different parts of the tool during linting and fixing. 🎀

like the Nesting (#nesting) section of the configuration docs says, you can use multiple nested documentation files in a project, and the result is a combined config object containing the resulting union of those files. under the hood, this is stored in a dict object, and it's possible to get and set individual values using get() & set_value(), but also to get entire portions of that config dict using get_section(). 💖

🧩 Methods for creating config mappings 🧩

when you instantiate a FluffConfig object, there are a few options to set specific config values (like dialect or rules), but to access the full available set of features it's best to pass in a dict of the values you want to set. 🌸

this config dict is a nested object, where the colon (:) characters from the .sqlfluff config files delimit the keys. for example, take the following config file:

[sqlfluff:rules:capitalisation.keywords]
capitalisation_policy = lower

this would be represented in the config dict as below. see that the nested structure has been created by splitting the keys on the colon (:) characters: 🦋

          configs = {
              "rules":{
                  "capitalisation.keywords": {
                      "capitalisation_policy": "lower"
                  }
              }
          }

Ok so bestie, these next ones are the lil glow-up squad of methods 💅 — they take a whole bunch of different file formats and turn them into one consistent mapping object so you can hand it to a FluffConfig and go live your life. ✨

🌸 load_config_string

load_config_string(config_string: str, configs: ConfigMappingType | None = None, working_path: str | None = None) -> ConfigMappingType

Load a config straight out of a string, in ini format. No file, no disk, just vibes. 🦋

Parameters 🎀

  • config_string (str) — The raw config file as a string. Heads up bestie: the content is assumed to be in the .ini format of a .sqlfluff file (i.e. not in .toml format).
  • configs (ConfigMappingType, optional) — A base set of configs to merge the loaded configs onto. If you don't provide it, the result will contain only the values loaded from the string.
  • working_path (str, optional) — The working path to use for the resolution of any paths specified in the config. If not provided then os.getcwd() is used as a default.

Returns 💖

A nested dictionary of config values.

Return type ⭐

ConfigMappingType

🧚 load_config_file

load_config_file(file_dir: str, file_name: str, configs: ConfigMappingType | None = None) -> ConfigMappingType

Load a config file from the filesystem. Yes, the actual disk. Iconic. 💾✨

Parameters 🌸

  • file_dir (str) — The path to the location of file to be loaded. This should be a reference to the directory only and not include the filename itself. Any paths in the loaded file are resolved relative to this location.
  • file_name (str) — The filename of the file to be loaded. If the filename is pyproject.toml then the file is loaded in toml format, but otherwise is assumed to be in ini format (as per .sqlfluff).
  • configs (ConfigMappingType, optional) — A base set of configs to merge the loaded configs onto. If not provided, the result will contain only the values loaded from the string.

Returns 💜

A nested dictionary of config values.

Return type ⭐

ConfigMappingType

🌈 load_config_resource

load_config_resource(package: str, file_name: str) -> ConfigMappingType

Load a config resource from a python package. Fancy fr fr. 🐍👑

Parameters 🎀

  • package (str) — The name of the python package to load the resource from.
  • file_name (str) — The filename of the file to be loaded. If the filename is pyproject.toml then the file is loaded in toml format, but otherwise is assumed to be in ini format (as per .sqlfluff).

Returns 💖

A nested dictionary of config values.

Return type ⭐

ConfigMappingType

Okay so why does this one exist? It's primarily used when loading configuration bundled with a SQLFluff plugin, or to load the default config for SQLFluff itself. By loading config from the package directly we avoid some of the path resolution which otherwise occurs. This is also more compatible with mypyc because it avoids the use of the __file__ attribute to find the default config. 🧠✨

Any paths found in the loaded config are resolved relative to os.getcwd().

For more information about resource loading, see the docs for importlib: https://docs.python.org/3/library/importlib.resources.html

🦋 load_config_at_path

load_config_at_path(path: str) -> ConfigMappingType

Load config files at a given path. 🗂️

Parameters 🌸

  • path (str) — The directory to search for config files.

Returns 💜

A nested dictionary of config values.

Return type ⭐

ConfigMappingType

This function will search for all valid config files at the given path, load any found and combine them into a config mapping. If multiple valid files are found, they are resolved in priority order, where pyproject.toml is given the highest precedence, followed by .sqlfluff, pep8.ini, tox.ini and finally setup.cfg. 🏆

By accepting only a path string, we enable efficient caching of results, such that configuration can be reused between files without reloading the information from disk. speed, no cap ⚡

💅 load_config_up_to_path

load_config_up_to_path(path: str, extra_config_path: str | None = None, ignore_local_config: bool = False) -> ConfigMappingType

Loads a selection of config files from both the path and its parent paths. It's giving family tree but for configs 🌳

Parameters 🎀

  • path (str) — The directory which is the target of the search. Config files in subdirectories will not be loaded by this method, but valid config files between this path and the current working path will.
  • extra_config_path (str, optional) — An additional path to load config from. This path is not used in iterating through intermediate paths, and is loaded last (taking the highest precedence in combining the loaded configs).
  • ignore_local_config (bool, optional, defaults to False) — If set to True, this skips loading configuration from the user home directory (~) or appdir path.

Returns 💖

A nested dictionary of config values.

Return type ⭐

ConfigMappingType

We layer each of the configs on top of each other, starting with any home or user configs (e.g. in appdir or home (~)), then any local project configuration and then any explicitly specified config paths. 🫶

👑 The FluffConfig object

class FluffConfig(configs: ConfigMappingType | None = None, extra_config_path: str | None = None, ignore_local_config: bool = False, overrides: ConfigMappingType | None = None, plugin_manager: pluggy.PluginManager | None = None, require_dialect: bool = True)

The persistent object for internal methods to access configuration. It's the main character of config, honestly. 💅✨

This class is designed to be instantiated once for each file and then be reused by each part of the process. For multiple files in the same path, a parent object will be created for the each path and then variants of it are created for each file. The object itself contains the references to any long lived objects which might be used by multiple parts of the codebase such as the dialect and the templater (both of which can be resource intensive to load & instantiate), which allows (for example), multiple files to reuse the same instance of the relevant dialect. 🧠💖

It is also designed to pickle well for use in parallel operations. 🥒✨

Parameters 🌸

  • configs (ConfigMappingType, optional) — A nested dict of config values from which to construct the config.
  • extra_config_path (str, optional) — An optional additional path to load config files from. These are loaded last if found and take precedence over any pre-existing config values. Note that when provided directly to the class, this path is not loaded for the class in question (it's assumed that has already been done, and the results are incorporated in the configs argument), but it is passed onward to child config instances, which will use it.
  • ignore_local_config (bool, optional, defaults to False) — If set to True, this skips loading configuration from the user home directory (~) or appdir path.
  • overrides (ConfigMappingType, optional) — A additional set of configs to merge into the core section of the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation. Note that this mapping dict only applies to the core section and so cannot be used for all values. ⚠️
  • plugin_manager (PluginManager, optional) — Optional pre-loaded config manager. Generally users should not need to provide this, as the class will fetch it's own if not provided. This argument is used when creating new class instances to avoid reloading the manager.
Note: 🫶 Methods for accessing internal properties on the config are not particularly standardised as the project currently assumes that few other tools are using this interface directly. If you or your project would like more formally supported methods for access to the config object, raise an issue on GitHub with the kind of things you'd like to achieve.

🎀 copy

copy() -> FluffConfig

Create a copy of this FluffConfig. 🦋

Copies created using this method can safely be modified without those changes propagating back up to the object which was originally copied. no take-backs, it's its own lil thing ✨

Returns 💖

A shallow copy of this config object but with a deep copy of the internal _configs dict.

Return type ⭐

FluffConfig

🧚 diff_to

diff_to(other: FluffConfig) -> ConfigMappingType

Compare this config to another. 🔍✨

This is primarily used in the CLI logs to indicate to the user what values have been changed for each file compared to the root config for the project.

Parameters 🌸

  • other (FluffConfig) — Another config object to compare against. We will return keys from this object that are not in other or are different to those in other.

Returns 💜

A filtered dict of items in this config that are not in the other or are different to the other.

Return type — dict

🦋 classmethod from_kwargs(...) 🦋

classmethod from_kwargs(dialect: str | None = None, rules: list[str] | None = None, exclude_rules: list[str] | None = None, require_dialect: bool = True) -> FluffConfig

bestie, this is the build-a-config of your dreams — it spins you up a config out of just a subset of the common options ✨

Parameters

  • dialect (str, optional) — the name of the dialect to use.
  • rules (list of str, optional) — a list of rules to include. Rule specifiers can be codes, names, groups or aliases. If you don't set it, it defaults to all the rules 💅
  • exclude_rules (list of str, optional) — a list of rules to exclude. Rule specifiers can be codes, names, groups or aliases. If you don't set it, it doesn't exclude any rules, no cap.
  • require_dialect (bool, optional, default is True) — when True, an error gets raised if the dialect config value is unset. so like, don't leave it blank unless you want drama 😭

Returns — the loaded config object.

Return type — FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>

This is a convenience method for the ways that the public classes like Linter(), Parser() and Lexer() let you set a subset of attributes directly instead of requiring a whole pre-made FluffConfig. it's giving shortcut energy and honestly? obsessed 🫶

🌸 classmethod from_path(...) 🌸

classmethod from_path(path: str, extra_config_path: str | None = None, ignore_local_config: bool = False, overrides: ConfigMappingType | None = None, plugin_manager: pluggy.PluginManager | None = None, require_dialect: bool = True) -> FluffConfig

loads a config object given a particular path 📁

Parameters

  • path (str) — the target path to load config files from. Files found between the working path and this path are also loaded and nested, with files closest to this target path taking precedence. closest wins, it's giving main character
  • extra_config_path (str, optional) — an optional additional path to load config files from. These are loaded last if found and take precedence over any pre-existing config values. so they show up fashionably late and then override everything 🙈
  • ignore_local_config (bool, optional, defaults to False) — if set to True, this skips loading configuration from the user home directory (~) or appdir path.
  • overrides (ConfigMappingType, optional) — an additional set of configs to merge into the core section of the config object at the end. These values take precedence over all other provided values and are inherited by child configs. Note that this mapping dict only applies to the core section and so cannot be used for all values. just the core section, bestie — nowhere else
  • plugin_manager (PluginManager, optional) — optional pre-loaded config manager. Generally users should not need to provide this, as the class will fetch its own if not provided. This argument is used when creating new class instances to avoid reloading the manager. efficiency slay
  • require_dialect (bool, optional, default is True) — when True, an error gets raised if the dialect config value is unset.

Returns — the loaded config object.

Return type — FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>

🎀 classmethod from_root(...) 🎀

classmethod from_root(extra_config_path: str | None = None, ignore_local_config: bool = False, overrides: ConfigMappingType | None = None, require_dialect: bool = True) -> FluffConfig

loads a config object based on the root directory 🏠✨

Parameters

  • extra_config_path (str, optional) — an optional additional path to load config files from. These are loaded last if found and take precedence over any pre-existing config values.
  • ignore_local_config (bool, optional, defaults to False) — if set to True, this skips loading configuration from the user home directory (~) or appdir path.
  • overrides (ConfigMappingType, optional) — an additional set of configs to merge into the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation. one override to rule them all, fr fr
  • require_dialect (bool, optional, default is True) — when True, an error gets raised if the dialect config value is unset.

Returns — the loaded config object.

Return type — FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>

🧚 classmethod from_string(...) 🧚

classmethod from_string(config_string: str, overrides: ConfigMappingType | None = None) -> FluffConfig

loads a config object from a single config string — one string, one config, no file system involved 💖

Parameters

  • config_string (str) — the config string, assumed to be in ini format (like a .sqlfluff file).
  • overrides (ConfigMappingType, optional) — an additional set of configs to merge into the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation.

Returns — the loaded config object.

Return type — FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>

🍓 classmethod from_strings(...) 🍓

classmethod from_strings(*config_strings: str, overrides: ConfigMappingType | None = None) -> FluffConfig

loads a config object given a series of nested config strings — like a lil stack of vibes, layered 🥞

Parameters

  • *config_strings (str) — an iterable of config strings, assumed to be in ini format (like a .sqlfluff file).
  • overrides (ConfigMappingType, optional) — an additional set of configs to merge into the config object at the end. These values take precedence over all other provided values and are inherited by child configs. For example, override values provided in the CLI use this method to apply to all files in a linting operation.

Returns — the loaded config object.

Return type — FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>

Config strings are incorporated from first to last, treating the first element as the "root" config, and then later config strings will take precedence over any earlier values. so the last one in the group chat is the one who gets the final say, iykyk 💅

🌈 get(...) 🌈

get(val: str, section: str | Iterable[str] = 'core', default: Any = None) -> Any

get a particular value from the config. simple as that bestie 🔍

Parameters

  • val (str) — the name of the config value to get.
  • section (str or iterable of str, optional) — the "path" to the config value. For values in the main [sqlfluff] section of the config, which are stored in the core section of the config, this can be omitted.
  • default — the value to return if the config value was not found. If no default is provided, then a KeyError will be raised if no value was found. no default = it WILL yell at you 😭

The following examples show how to fetch various default values:

>>> FluffConfig(overrides={"dialect": "ansi"}).get("dialect")
'ansi'

>>> config = FluffConfig(overrides={"dialect": "ansi"})
>>> config.get("tab_space_size", section="indentation")
4

>>> FluffConfig(overrides={"dialect": "ansi"}).get(
...     "capitalisation_policy",
...     section=["rules", "capitalisation.keywords"]
... )
'consistent'

look at that, 'consistent' — consistent queen 👑

🌟 get_section(...) 🌟

get_section(section: str | Iterable[str]) -> Any

return a whole section of config as a dict 📦

If the element found at the address is a value and not a section, it is still returned and so this can be used as a more advanced form of the basic get method. so it's get but with extra range ✨

Parameters

  • section — an iterable or string. If it's a string we load that root section. If it's an iterable of strings, then we treat it as a path within the dictionary structure.

💫 get_templater(...) 💫

get_templater(**kwargs: Any) -> RawTemplater

instantiate the configured templater. that's it, that's the whole thing 🎀

⭐ get_templater_class() ⭐

get_templater_class() -> type[RawTemplater]

get the configured templater class. note: the class itself, not an instance — the blueprint, not the build 🫶

📝 Note: this one's mostly here so you can call it directly when rules wanna figure out the type of a templater — specifically to work out whether it's a derivative of the jinja templater — without having to instantiate a whole entire templater first. Why tho? Because instantiated templaters do not pickle well 😭, so they don't get automatically passed around between threads/processes. That's the whole reason, bestie. 🫖

iter_vals(cfg: ConfigMappingType | None = None) -> Iterable[tuple[int, str, ConfigValueOrListType]]

Returns an iterable of tuples that represent keys. yes, that's the whole thing, and it ate 💅

Parameters

  • cfg (optional) — an optional config mapping to format instead. If you don't hand one over, we just use the internal config object of the FluffConfig. 🎀

This is primarily to enable formatting of config objects in the CLI, okay? And here's the tea on how it works: we show values before dicts, and the tuple carries an indent value so you know what level of the dict we're currently in. Dict labels get returned as a blank value before their content. 📚

make_child_from_path(path: str, require_dialect: bool = True) -> FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>

Make a child config at a path, but pass on overrides and extra_config_path. It's giving "new address, same baggage" ✨

Parameters

  • path (str) — the path to load the new config object from, inheriting the content of the calling FluffConfig as base values.
  • require_dialect (bool, optional, default is True) — when True, an error will be raised if the dialect config value is unset. so like, don't leave it blank bestie 🙈

Returns

A new config object which copies the current config object, but overriding any values set by config values loaded from the given path.

Return type

FluffConfig

process_inline_config(config_line: str, fname: str) -> None

Process an inline config command and update self. lowkey self-care for your config 💖

Parameters

  • config_line (str) — the inline config section to be processed. This should usually begin with -- sqlfluff:.
  • fname (str) — the name of the current file being processed. This is used purely for logging purposes in the case that an invalid config string is provided, so that any error messages can reference the file with the issue. very thoughtful of it tbh 🥹
>>> cfg = FluffConfig(overrides={"dialect": "ansi"})
>>> cfg.process_inline_config(
...     "-- sqlfluff:dialect:postgres",
...     "test.sql"
... )
>>> cfg.get("dialect")
'postgres'

process_raw_file_for_config(raw_str: str, fname: str) -> None

Process a full raw file for inline config and update self. go big or go home 🦋

Parameters

  • raw_str (str) — the full SQL script to evaluate for inline configs.
  • fname (str) — the name of the current file being processed. This is used purely for logging purposes in the case that an invalid config string is provided, so that any error messages can reference the file with the issue.
>>> cfg = FluffConfig(overrides={"dialect": "ansi"})
>>> cfg.process_raw_file_for_config(
...     "-- sqlfluff:dialect:postgres",
...     "test.sql"
... )
>>> cfg.get("dialect")
'postgres'

set_value(config_path: Iterable[str], val: Any) -> None

Set a value at a given path. short, sweet, iconic 👑

Parameters

  • config_path — an iterable of strings. Each should be one of the elements which is colon delimited in a standard config file.
  • val — the value to set at the given path.
>>> cfg = FluffConfig(overrides={"dialect": "ansi"})
>>> cfg.set_value(["dialect"], "postgres")
>>> cfg.get("dialect")
'postgres'
>>> cfg = FluffConfig(overrides={"dialect": "ansi"})
>>> cfg.set_value(["indentation", "tab_space_size"], 2)
>>> cfg.get("tab_space_size", section="indentation")
2

verify_dialect_specified() -> None

Check if the config actually specifies a dialect, and raise an error if it doesn't. it's giving vibe check but for dialects 🚨

Raises SQLFluffUserError — if the dialect config value is unset. The content of the error contains user-facing instructions on what dialects are available and how to set the dialect. so it's not just yelling at you, it's helping 💜

🌈 sqlfluff.core.rules.base: Base Rule Classes 🌈

Implements the base rule class. periodt.

Rules crawl through the trees returned by the parser and evaluate particular rules. 🌳

The intent is that it should be possible for the rules to be expressed as simply as possible, with as much of the complexity abstracted away. no cap, that's the dream ✨

The evaluation function should take enough arguments that it can evaluate the position of the given segment in relation to its neighbors — and the segment which finally "triggers" the error should be the one that would be corrected. OR, if the rule relates to something that is missing, then it should flag on the segment FOLLOWING, i.e. the place where the desired element is missing. 🫶

👑 class BaseRule(code: str, description: str, **kwargs: Any)

The base class for a rule. the mother of all rules fr fr

Parameters

  • code (str) — the identifier for this rule, used in inclusion or exclusion.
  • description (str) — a human readable description of what this rule does. It will be displayed when any violations are found.

crawl(tree: BaseSegment, dialect: Dialect, fix: bool, templated_file: TemplatedFile | None, ignore_mask: IgnoreMask | None, fname: str | None, config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> tuple[list[SQLLintError], tuple[RawSegment, ...], list[LintFix], dict[str, Any] | None]

Run the rule on a given tree. crawl bestie, crawl 🦋

Returns

A tuple of (vs, raw_stack, fixes, memory).

static discard_unsafe_fixes(lint_result: LintResult <#sqlfluff.core.rules.base.LintResult>, templated_file: TemplatedFile | None) -> None

Remove (discard) LintResult fixes if they are "unsafe". 💀

By removing its fixes, a LintResult will still be reported, but it will be treated as unfixable. so you'll know about it, you just won't get the auto-fix. that's the tradeoff, babe.

static filter_meta(segments: Sequence[BaseSegment], keep_meta: bool = False) -> tuple[BaseSegment, ...]

Filter the segments down to non-meta. 🧹

Or optionally the opposite if keep_meta is True. flip the switch, flip the vibe 🔄

classmethod get_config_ref() -> str

Return the config lookup ref for this rule.

If a name is defined, it's the name — otherwise the code. 🎀

The name is a much more understandable reference and so makes config files more readable. For backward compatibility, however, we also support the rule code for those without names. keeping the old ways alive, iconic behavior 👑

classmethod get_parent_of(segment: BaseSegment, root_segment: BaseSegment) -> BaseSegment | None

Return the segment immediately containing segment.

⚠️ NB: This is recursive. yes bestie, it goes deep.

Parameters

  • segment — the segment to look for.
  • root_segment — some known parent of the segment we're looking for (although likely not the direct parent in question).

static split_comma_separated_string(raw: str | list[str]) -> list[str]

Converts a comma separated string to List, stripping whitespace. tidy ✨

🫶 class LintResult(anchor: BaseSegment | None = None, fixes: list[LintFix] | None = None, memory: Any | None = None, description: str | None = None, source: str | None = None)

A class to hold the results of a rule evaluation. the receipts, basically 🧾

Parameters

  • anchor (BaseSegment, optional) — a segment which represents the position of the problem. NB: each fix will also hold its own reference to position, so this position is mostly for alerting the user to where the problem is.
  • fixes (list of LintFix, optional) — an array of any fixes which would correct this issue. If not present, then it's assumed that this issue will have to be manually fixed. 😭
  • memory (dict, optional) — an object which stores any working memory for the rule. The memory returned in any LintResult will be passed as an input to the next segment to be crawled. passing notes between segments, love that for them 💌
  • description (str, optional) — a description of the problem identified as part of this result. This will override the description of the rule as what gets reported to the user with the problem, if provided.
  • source (str, optional) — a string identifier for what generated the result. Within larger libraries like reflow this can be useful for tracking where a result came from. 🕵️

to_linting_error(rule: BaseRule <#sqlfluff.core.rules.base.BaseRule>) -> SQLLintError | None

Convert a linting result to a SQLLintError if appropriate.

🎀 class RuleGhost(code, name, description)

  • code — alias for field number 0
  • description — alias for field number 2
  • name — alias for field number 1

(yes, out of order, and we're not sorry. it's a namedtuple thing, iykyk 💅)

🌸 class RuleLoggingAdapter(logger, extra=None, merge_extra=False)

A LoggingAdapter for rules which adds the code of the rule to it. 📎

process(msg: str, kwargs: Any) -> tuple[str, Any]

Add the code element to the logging message before emit.

class RuleManifest(code: str, name: str, description: str, groups: tuple[str, ...], aliases: tuple[str, ...], rule_class: type[BaseRule <#sqlfluff.core.rules.base.BaseRule>])

okay so this lil guy is literally one entry in the rule register. that's it. that's the whole job. it's the little ID card every rule carries around so the rest of the system knows who it is. 💖

class RuleMetaclass(name: str, bases: list[BaseRule <#sqlfluff.core.rules.base.BaseRule>], class_dict: dict[str, Any])

bestie this is the metaclass for rules, the boss of the rule classes themselves. the way that it works is it auto-enriches the rule docstring — so examples, groups, aliases and names all get slipped in there for you. ✨

and why do we even bother enriching the docstring? because we want autodoc to pick it up so everything shows up in the sqlfluff docs. no cap, that's the whole reason. it's giving "make the docs pretty by doing the work at class-creation time" and honestly? slay. 💅

class RulePack(rules: list[BaseRule <#sqlfluff.core.rules.base.BaseRule>], reference_map: dict[str, set[str]])

a bundle of rules that are gonna be applied. 🎀

so like — this holds a set of rules post filtering, but it also holds the mapping you need to actually interpret any noqa messages you find hiding in files. sneak peek: that mapping matters, keep reading. 🙈

the reason this whole object exists: rules get filtered and instantiated into this pack in the main process when you're running in multi-processing mode, so that user-defined rules can be used without any reference issues. periodt. 🔥

  • rules — a filtered list of instantiated rules to apply to a given file.
    • Type: list of BaseRule
  • reference_map — a mapping of rule references to the codes they refer to, e.g. {"my_ref": {"LT01", "LT02"}}. the references (i.e. the keys) may be codes, groups, aliases or names. the values of the mapping are sets of rule codes only. this object acts as a lookup so you can translate selectors (which might contain all kinds of diverse references) into one consolidated list of rule codes. and listen — this mapping contains the full set of rules, not just the filtered set sitting in the rules attribute. don't get it twisted. ⭐
    • Type: dict
  • codes() -> Iterator[str] — returns an iterator through the codes contained in the pack. 🦋
class RuleSet(name: str, config_info: dict[str, ConfigInfo])

a class to define a ruleset, bestie. 💜

here's the tea: a rule set gets instantiated on module load, but the references to each of its classes get instantiated at runtime. which means config values can be passed to those rules live and be fully responsive to any changes in configuration coming from whatever path the file is sitting in. iconic behaviour honestly. 🌸

you fetch rules with the get_rulelist() command, which also handles all the filtering (i.e. allowlisting and denylisting). 👑

new rules get added to the instance of this class with the register() decorator. that decorator registers the class and does basic type and name-convention checks, so it's not just decorative, it's doing cardio. 😭

the code for a rule gets parsed from the name, and the description comes from the docstring. the eval function is assumed to be overridden by the subclass — and if it's not overridden? the parent class raises an error on that function. so like, override it. no skipping. 💀

  • copy() -> RuleSet <#sqlfluff.core.rules.base.RuleSet> — return a copy of self with a separate register. 🎀
  • get_rulepack(config: FluffConfig <#sqlfluff.core.config.fluffconfig.FluffConfig>) -> RulePack <#sqlfluff.core.rules.base.RulePack> — use the config to return the appropriate rules. we use the config both for allowlisting and denylisting, and for configuring the rules given the given config. two jobs, one config, she's booked and busy. ✨
  • register(cls: type[BaseRule <#sqlfluff.core.rules.base.BaseRule>], plugin: PluginSpec | None = None) -> type[BaseRule <#sqlfluff.core.rules.base.BaseRule>] — decorate a class with this to add it to the ruleset. like so:
@myruleset.register
class Rule_LT01(BaseRule):
    "Description of rule."

    def eval(self, **kwargs):
        return LintResult()

we expect rules to be defined as classes named Rule_XXXX where XXXX is of the form LNNN — where L is a letter (literally L for linting by default) and N is a three digit number. 🔢

if this receives classes by any other name, then it will raise a ValueError. so like, name them right or catch these hands (the hands are a ValueError). 🙈

  • rule_reference_map() -> dict[str, set[str]] — generate a rule reference map for looking up rules. it generates the master reference map, and the priority order is: codes > names > groups > aliases (i.e. if there's a collision between a name and an alias — we assume the alias is wrong). iykyk. 💅

✨ sqlfluff.utils.functional: Functional Traversal API ✨

these newer modules provide a higher-level API for rules that work with segments and slices. rules that need to navigate or search the parse tree may lowkey benefit from using these. eventually, the plan is for all rules to use these modules. and as of December 30, 2021, 17+ rules use these modules already. the glow up is happening. 🌈

the modules listed below are submodules of sqlfluff.utils.functional. 🫶

🦋 segments Module 🦋

surrogate class for working with Segment collections. 💖

class Segments(*segments: BaseSegment, templated_file: TemplatedFile | None = None)

encapsulates a sequence of one or more BaseSegments. ✨

the segments may or may not be contiguous in a parse tree — no pressure, no requirements. it provides useful operations on a sequence of segments to simplify rule creation, which is literally the whole point. 🎀

  • all(predicate: Callable[[BaseSegment], bool] | None = None) -> bool — do all the segments match? 👑
  • any(predicate: Callable[[BaseSegment], bool] | None = None) -> bool — do any of the segments match? 💅
  • apply(fn: Callable[[BaseSegment], Any]) -> list[Any] — apply function to every item. 🌸
  • children(predicate: Callable[[BaseSegment], bool] | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments> — returns an object with the children of the segments in this object. 🧚
  • find(segment: BaseSegment | None) -> int — returns the index if found, -1 if not found. simple as. ⭐
  • first(predicate: Callable[[BaseSegment], bool] | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments> — returns the first segment (if any) that satisfies the predicates. 🫶
  • get(index: int = 0, *, default: BaseSegment | None = None) -> BaseSegment | None — return the specified item. returns default if the index is out of range. no index errors here bestie, we're too chill for that. 😌
  • iterate_segments(predicate: Callable[[BaseSegment], bool] | None = None) -> Iterable[Segments <#sqlfluff.utils.functional.segments.Segments>] — loop over each element as a fresh Segments. 💜
  • last(predicate: Callable[[BaseSegment], bool] | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments> — returns the last segment (if any) that satisfies the predicates. saving the best for last, literally. 🔥
  • property raw_segments: Segments <#sqlfluff.utils.functional.segments.Segments> — get the raw segments underlying the segments. 🌈
  • property raw_slices: RawFileSlices <#sqlfluff.utils.functional.raw_file_slices.RawFileSlices> — raw slices of the segments, sorted in source file order. 🎀
  • recursive_crawl(*seg_type: str, recurse_into: bool = True) -> Segments <#sqlfluff.utils.functional.segments.Segments> — recursively crawl for segments of a given type. 🦋
  • recursive_crawl_all() -> Segments <#sqlfluff.utils.functional.segments.Segments> — recursively crawl all descendant segments. go off, get everything. 💅
  • reversed() -> Segments <#sqlfluff.utils.functional.segments.Segments> — return the same segments in reverse order. 🌀
  • select(select_if: Callable[[BaseSegment], bool] | None = None, loop_while: Callable[[BaseSegment], bool] | None = None, start_seg: BaseSegment | None = None, stop_seg: BaseSegment | None = None) -> Segments <#sqlfluff.utils.functional.segments.Segments> — retrieve a range/subset. ⭐
NOTE: it iterates the segments BETWEEN start_seg and stop_seg, i.e. those segments are not included in the loop. important, don't forget it. 🙈

🌸 segment_predicates Module 🌸

defines commonly used segment predicates for rule writers. 💖

for consistency, all the predicates in this module are implemented as functions returning functions. this avoids rule writers having to remember the distinction between normal functions and functions-returning-functions. genuinely so considerate of them. 🫶

this is not necessarily a complete set of predicates covering all possible requirements, so rule authors can define their own predicates as needed — as regular functions, lambdas, etc. make it your own, bestie. ✨

  • and_(*functions: Callable[[BaseSegment], bool]) -> Callable[[BaseSegment], bool] — returns a function that computes the functions and-ed together. 🔥
  • get_type() -> Callable[[BaseSegment], str] — returns a function that gets the segment type. 🎀
  • is_code() -> Callable[[BaseSegment], bool] — returns a function that checks if the segment is code. 💅
  • is_comment() -> Callable[[BaseSegment], bool] — returns a function that checks if the segment is a comment. 💬
  • is_keyword(*keyword_name: str) -> Callable[[BaseSegment], bool] — returns a function that determines if it's a matching keyword. 👑
  • is_meta() -> Callable[[BaseSegment], bool] — returns a function that checks if the segment is meta. 🧚
  • is_raw() -> Callable[[BaseSegment], bool] — returns a function that checks if the segment is raw. ⭐
  • is_templated() -> Callable[[BaseSegment], bool] — returns a function that checks if the segment is templated. 🌈
  • is_type(*seg_type: str) -> Callable[[BaseSegment], bool] — returns a function that determines if the segment is one of the types. 🦋
  • is_whitespace() -> Callable[[BaseSegment], bool] — returns a function that checks if the segment is whitespace. 🌸
  • not_(fn: Callable[[BaseSegment], bool]) -> Callable[[BaseSegment], bool] — returns a function that computes: not fn(). 💜
  • or_(*functions: Callable[[BaseSegment], bool]) -> Callable[[BaseSegment], bool] — returns a function that computes the functions or-ed together. ✨
  • raw_is(*raws: str) -> Callable[[BaseSegment], bool] — returns a function that determines if the segment matches one of the raw inputs. 🎀
  • raw_slices(segment: BaseSegment, templated_file: TemplatedFile | None) -> RawFileSlices — hands you the raw slices for a segment. that's it, that's the whole job. iconic ✨
  • raw_upper_is(*raws: str) -> Callable[[BaseSegment], bool] — gives you back a function that checks whether a segment matches one of the raw inputs you passed in. yes bestie, it returns a function, not a bool, don't get got. 💅
  • templated_slices(segment: BaseSegment, templated_file: TemplatedFile | None) -> TemplatedFileSlices — same energy as raw_slices but it's giving templated slices for a segment. 🦋

🧵 raw_file_slices Module

Surrogate class for working with RawFileSlice collections. ok so it's a little stand-in helper for wrangling RawFileSlice collections, cute. 🎀

class RawFileSlices(*raw_slices: RawFileSlice, templated_file: TemplatedFile | None = None)

Encapsulates a sequence of one or more RawFileSlice. one or more, bestie.

Lowkey important detail: the slices may or may not be contiguous in a file. They don't have to sit next to each other, they're just vibing in the same sequence. And the whole point is that it provides useful operations on a sequence of slices to simplify rule creation. 🫶

  • all(predicate: Callable[[RawFileSlice], bool] | None = None) -> bool — do all the raw slices match? 💯
  • any(predicate: Callable[[RawFileSlice], bool] | None = None) -> bool — do any of the raw slices match? just one, bestie, that's all it takes. ✨
  • select(select_if: Callable[[RawFileSlice], bool] | None = None, loop_while: Callable[[RawFileSlice], bool] | None = None, start_slice: RawFileSlice | None = None, stop_slice: RawFileSlice | None = None) -> RawFileSlices — retrieve a range/subset. it's giving slice-and-dice energy. 🔪✨
NOTE: Iterates the slices BETWEEN start_slice and stop_slice, i.e. those slices are not included in the loop. so if you were expecting the endpoints to come along for the ride… nope. 🙈

🎀 raw_file_slice_predicates Module

Defines commonly used raw file slice predicates for rule writers. it's a lil grab-bag of ready-made predicates so you don't have to write them yourself. 💖

For consistency, all the predicates in this module are implemented as functions returning functions. The whole reason? So rule writers don't have to remember the distinction between normal functions and functions returning functions. no cap, that's genuinely the reason and honestly? thoughtful. 👑

Quick reality check though: this is not necessarily a complete set of predicates covering all possible requirements. Rule authors can define their own predicates as needed — regular functions, lambdas, whatever your heart desires. 🦋

  • is_slice_type(*slice_types: str) -> Callable[[RawFileSlice], bool] — returns a function that determines whether a segment is one of the types. 🍓

🌈 sqlfluff.utils.reflow: Whitespace Reflow API

ok so many rules supported by SQLFluff involve the spacing and layout of different elements. sometimes to enforce a particular layout, sometimes just to add or remove code elements in a way that's sensitive to the existing layout configuration. and the way all of that gets achieved is through some centralised utilities in the sqlfluff.utils.reflow module. 🧚

This module aims to achieve several things:

  • Less code duplication by implementing reflow logic in only one place. 🎯
  • Provide a streamlined interface for rules to easily utilise reflow logic.
    • Given this requirement, it's important that reflow utilities work within the existing framework for applying fixes to potentially templated code. We achieve this by returning LintFix objects which can then be returned by each rule wanting to use this logic. 💅
  • Provide a consistent way of configuring layout requirements. For more details on configuration see Configuring Layout. 📐

To support this, the module provides a ReflowSequence class which allows access to all of the relevant operations which can be used to reformat sections of code, or even a whole file. And bestie, unless there is a very good reason, all rules should use this same approach to ensure consistent treatment of layout. periodt. 💅

✨ class ReflowSequence(elements: list[ReflowBlock | ReflowPoint], root_segment: BaseSegment, reflow_config: ReflowConfig, depth_map: DepthMap, lint_results: list[LintResult] | None = None)

Class for keeping track of elements in a reflow operation. 🌸

This acts as the primary route into using the reflow routines. It acts in a way that plays nicely within a rule context in that it accepts segments and configuration, while allowing access to modified segments and a series of LintFix objects, which can be returned by the calling rule. chef's kiss. 🫶

Sequences are made up of alternating ReflowBlock and ReflowPoint objects (even if some points have no segments). This is validated on construction, so you can't sneak a bad one past it. 👑

Most operations also return ReflowSequence objects such that operations can be chained, and then the resultant fixes accessed at the last stage, for example:

fixes = (
    ReflowSequence.from_around_target(
        context.segment,
        root_segment=context.parent_stack[0],
        config=context.config,
    )
    .rebreak()
    .get_fixes()
)

look at that chain. so clean. ✨

break_long_lines() -> ReflowSequence

Rebreak any remaining long lines in a sequence. 📏

This assumes that reindent() has already been applied. don't skip steps bestie. 🙈

classmethod from_around_target(target_segment: BaseSegment, root_segment: BaseSegment, config: FluffConfig, sides: str = 'both') -> ReflowSequence

Generate a sequence around a target. 🎯

Parameters

  • target_segment (RawSegment) — The segment to center around when considering the sequence to construct.
  • root_segment (BaseSegment) — The relevant root segment (usually the base FileSegment).
  • config (FluffConfig) — A config object from which to load the spacing behaviours of different segments.
  • sides (str) — Limit the reflow sequence to just one side of the target. Default is two sided ("both"), but set to "before" or "after" to limit to either side. 💅
NOTE: We don't just expand to the first block around the target but to the first code element, which means we may swallow several comment blocks in the process. they're fine in there, don't worry, but yeah. 😭

To evaluate reflow around a specific target, we need need to generate a sequence which goes for the preceding raw to the following raw. i.e. at least: block - point - block - point - block (where the central block is the target). (yes, "need need". the docs said it first, I'm just the messenger. 🙈)

classmethod from_raw_segments(segments: Sequence[RawSegment], root_segment: BaseSegment, config: FluffConfig, depth_map: DepthMap | None = None) -> ReflowSequence

Construct a ReflowSequence from a sequence of raw segments. 🧵

This is intended as a base constructor, which others can use. In particular, if no depth_map argument is provided, this method will generate one in a potentially inefficient way. 😬

So if the calling method has access to a better way of inferring a depth map (for example because it has access to a common root segment for all the content), it should do that instead and pass it in. do the smart thing bestie, save the cycles. ⚡

classmethod from_root(root_segment: BaseSegment, config: FluffConfig) -> ReflowSequence

Generate a sequence from a root segment. 🌱

Parameters

  • root_segment (BaseSegment) — The relevant root segment (usually the base FileSegment).
  • config (FluffConfig) — A config object from which to load the spacing behaviours of different segments.

get_fixes() -> list[LintFix]

Get the current fix buffer. 🛠️

We're hydrating them here directly from the LintResult objects, so for more accurate results, consider using .get_results(). This method is particularly useful when consolidating multiple results into one. 💅

get_raw() -> str

Get the current raw representation. it's giving string. ✨

get_results() -> list[LintResult]

Return the current result buffer. 🎀

insert(insertion: RawSegment, target: RawSegment, pos: str = 'before') -> ReflowSequence

Returns a new ReflowSequence with the new element inserted. 🌸

Insertion is always relative to an existing element. Either before or after it as specified by pos. This generates appropriate creation LintFix objects to direct the linter to insert those elements. 🫶

Note: This method is currently excluded from test coverage because it is not actively used by any rule. It was previously utilized by rule AL01, but that rule now uses an alternative implementation. The method is retained for potential future reuse or reference. so it's just sitting there, waiting for its main character moment. 💀

rebreak(rebreak_type: Literal['lines', 'keywords'] = 'lines') -> ReflowSequence

Returns a new ReflowSequence corrected line breaks. 🔄

This intentionally does not handle indentation, as the existing indents are assumed to be correct. don't touch what you didn't come here for. 🙈

Note: Currently this only moves existing segments around line breaks (e.g. for operators and commas), but eventually this method will also handle line length considerations too. it's a work in progress era. 🦋

reindent() -> ReflowSequence

Reindent lines within a sequence. 📐✨

replace(target: BaseSegment, edit: Sequence[BaseSegment]) -> ReflowSequence

Returns a new ReflowSequence with edit elements replaced. 🔁

This generates appropriate replacement LintFix objects to direct the linter to modify those elements. 💖

respace(strip_newlines: bool = False, filter: str = 'all') -> ReflowSequence

Returns a brand new ReflowSequence with its points respaced. ✨

Parameters

  • strip_newlines (bool) — optionally strip newlines before respacing. This is primarily used on focused sequences to coerce objects onto a single line. Heads up bestie: it does not apply any prioritisation to which line breaks to remove, so it is not a substitute for the full reindent or reflow methods. 🫶
  • filter (str) — optionally filter which reflow points to respace. Default configuration is all. The other options are line_break, which only respaces points containing a newline or followed by an end_of_file marker, or inline, which is the inverse of line_break. This is most useful for filtering between trailing whitespace and fixes between content on a line. 💅
NOTE this method relies on the embodied results being correct so that we can build on them.

without(target: RawSegment) -> ReflowSequence

Returns a new ReflowSequence without the specified segment. 🦋

This generates appropriate deletion LintFix objects to direct the linter to remove those elements. No cap.

Note:

This method is currently excluded from test coverage because it is not actively used by any rule. It was previously utilized by rule AL01, but that rule now uses an alternative implementation. The method is retained for potential future reuse or reference. 💀

class ReflowPoint(segments: tuple[RawSegment, ...])

Class for keeping track of editable elements in reflow. 👑

This class, and its sibling ReflowBlock, should not normally be manipulated directly by rules, but instead should be manipulated using ReflowSequence. That's the whole vibe.

It holds segments which can be changed during a reflow operation such as whitespace and newlines. It may also contain Indent and Dedent elements. 🌸

It holds no configuration and is influenced by the blocks on either side, so that any operations on it usually have that configuration passed in as required.

property class_types: set[str]

Get the set of contained class types.

Parallel to BaseSegment.class_types. ✨

get_indent() -> str | None

Get the current indent (if there). 🎀

get_indent_impulse() -> IndentStats

Get the change in intended indent balance from this point.

get_indent_segment_vals(exclude_block_indents=False) -> list[int]

Iterate through any indent segments and extract their values.

indent_to(desired_indent: str, after: BaseSegment | None = None, before: BaseSegment | None = None, description: str | None = None, source: str | None = None) -> tuple[list[LintResult], ReflowPoint]

Coerce a point to have a particular indent. 💅

If the point currently contains no newlines, one will be introduced and any trailing whitespace will be effectively removed. Slay.

More specifically, the newline is inserted before the existing whitespace, with the new indent being a replacement for that same whitespace.

For placeholder newlines or indents we generate appropriate source fixes. 🫶

is_all_unrendered() -> bool

Return whether this element is all unrendered.

Returns True if contains only whitespace, indents, template loops or placeholders.

Note:

  • ReflowBlocks will contain the placeholders and loops
  • ReflowPoints will contain whitespace, indents and newlines

num_newlines() -> int

Return the number of newlines in this element.

These newlines are either newline segments or contained within consumed sections of whitespace. This counts both. 🔥

property pos_marker: PositionMarker | None

Get the first position marker of the element.

property raw: str

Get the current raw representation.

respace_point(prev_block: ReflowBlock | None, next_block: ReflowBlock | None, root_segment: BaseSegment, lint_results: list[LintResult], strip_newlines: bool = False, anchor_on: str = 'before') -> tuple[list[LintResult], ReflowPoint]

Respace a point based on given constraints. 🌈

NB: This effectively includes trailing whitespace fixes.

Deletion and edit fixes are generated immediately, but creations are paused to the end and done in bulk so as not to generate conflicts. Periodt.

Note that the strip_newlines functionality exists here as a slight exception to pure respacing, but as a very simple case of positioning line breaks. The default operation of respace does not enable it, however it exists as a convenience for rules which wish to use it. 💖

class ReflowBlock(segments: tuple[RawSegment, ...], spacing_before: str, spacing_after: str, line_position: str | None, depth_info: DepthInfo, stack_spacing_configs: dict[int, str], line_position_configs: dict[int, str], keyword_line_position: str | None, keyword_line_position_configs: dict[int, str], keyword_line_position_exclusions: str | list[str] | None, keyword_line_position_exclusions_configs: dict[int, str | list[str]])

Class for keeping track of elements to reflow. 🦋

This class, and its sibling ReflowPoint, should not normally be manipulated directly by rules, but instead should be manipulated using ReflowSequence. Same energy, different class.

It holds segments to reflow and also exposes configuration regarding how they are expected to reflow around others. Typically it holds only a single element, which is usually code or a templated element. Because reflow operations control spacing, it would be very unusual for this object to be modified; as such it exposes relatively few methods.

The attributes exposed are designed to be "post configuration" i.e. they should reflect configuration appropriately. ✨

property class_types: set[str]

Get the set of contained class types.

Parallel to BaseSegment.class_types. 🎀

depth_info: DepthInfo

Metadata on the depth of this segment within the parse tree which is used in inferring how and where line breaks should exist.

classmethod from_config(segments: tuple[RawSegment, ...], config: ReflowConfig, depth_info: DepthInfo) -> ReflowBlock

Construct a ReflowBlock while extracting relevant configuration.

This is the primary route to construct a ReflowBlock, as is allows all of the inference of the spacing and position configuration from the segments it contains and the appropriate config objects. 👑

is_all_unrendered() -> bool

Return whether this element is all unrendered.

Returns True if contains only whitespace, indents, template loops or placeholders.

Note:

  • ReflowBlocks will contain the placeholders and loops
  • ReflowPoints will contain whitespace, indents and newlines

keyword_line_position: str | None

Desired line position for this block's keywords. See Configuring layout and spacing <#layoutspacingconfig>. 💅

keyword_line_position_configs: dict[int, str]

Desired keyword line position configurations for parent segments of the segment in this block. See Configuring layout and spacing <#layoutspacingconfig>. 🌸

keyword_line_position_exclusions: str | list[str] | None

Parent segments which this block's keyword line positioning should not apply. See Configuring layout and spacing <#layoutspacingconfig>. 🫶

keyword_line_position_exclusions_configs: dict[int, str | list[str]]

Configurations for parent segments which this block's keyword line positioning should not apply. See Configuring layout and spacing <#layoutspacingconfig>. ✨

line_position: str | None

Desired line position for this block. See Configuring layout and spacing <#layoutspacingconfig>. 🎀

line_position_configs: dict[int, str]

Desired line position configurations for parent segments of the segment in this block. See Configuring layout and spacing <#layoutspacingconfig>. 💖

num_newlines() -> int

Return the number of newlines in this element.

These newlines are either newline segments or contained within consumed sections of whitespace. This counts both. 🔥

property pos_marker: PositionMarker | None

Get the first position marker of the element.

property raw: str

Get the current raw representation.

spacing_after: str

Desired spacing after this block. See Configuring layout and spacing <#layoutspacingconfig>. 🌈

spacing_before: str

Desired spacing before this block. See Configuring layout and spacing <#layoutspacingconfig>. ⭐

stack_spacing_configs: dict[int, str]

Desired spacing configurations for parent segments of the segment in this block. See Configuring layout and spacing <#layoutspacingconfig>. 👑

💖 Release Notes 💖

okay so bestie, this page is literally your migration survival guide 🗺️ — it's here to walk you through hopping between major SQLFluff releases. Which means, no cap, that bugfix releases, or releases where nothing changes for you, do not get a mention here. Not one. 🙈

For the full deets on every single individual release though? Go read the detailed changelog at https://github.com/sqlfluff/sqlfluff/blob/main/CHANGELOG.md — it's all there, every crumb. 🍓

🚀 Upgrading to 3.x

This release is giving "a couple of potentially breaking changes" energy, and here they are:

  • 💀 It drops support for python 3.7, which reached end of life in June 2023.
  • It migrates to pyproject.toml rather than setup.cfg as the python packaging configuration file (although it keeps setuptools as the default backend).
  • The serialised output for sqlfluff lint now contains more information about the span of linting issues and initial proposed fixes. Beside the new fields, the original fields of line_pos and line_no have been renamed to start_line_pos and start_line_no, to distinguish them from the new fields starting end_*. 👀
  • When linting from stdin, if there are no violations found — before this version, the serialised response would be simply an empty list ([]). From 3.0 onwards, there will now be a record for the file with some statistics, but the violations section of the response for that file will still be an empty list. So like, not actually empty-empty. Just violation-empty. 💅
  • The default annotation_level set by the --annotation-level option on the sqlfluff lint command has been changed from notice to warning, to better distinguish linting errors from warnings, which always now have the level of notice. ⚠️ Heads up: this is only relevant when you're using the github-annotation or github-annotation-native formats.
  • The previously deprecated --disable_progress_bar on lint, fix and format has now been removed entirely. Please migrate to --disable-progress-bar to continue using this option. (Yes queen, the underscores are gone. It's a whole new world. ✨)
  • The --force option on sqlfluff fix is now the default behaviour, and so the option has been deprecated. A new --check option has been introduced which mimics the old default behaviour. This has been changed as it enables significantly lower memory overheads when linting and fixing large projects. 🫶

🚀 Upgrading to 2.3

This release includes two minor breaking changes which will only affect users engaged in performance optimisation of SQLFluff itself. So if that's not you, you can vibe on by. 🦋

  • The --profiler option on sqlfluff parse has been removed. It was only present on the parse command and not lint or fix, and it is just as simple to invoke the python cProfiler directly. Bye bestie. 👋
  • The --recurse cli option and sqlfluff.recurse configuration option have both been removed. They both existed purely for debugging the parser, and were never used in a production setting. The improvement in other debugging messages when unparsable sections are found means that this option is no longer necessary. iconic honestly, the parser debugged itself out of a job 💀

🚀 Upgrading to 2.2

This release changes some of the interfaces between SQLFluff core and our plugin ecosystem. The only breaking change is in the interface between SQLFluff and templater plugins (which are not common in the ecosystem, hence why this is only a minor and not a major release). 📢

For all plugins, we also recommend a different structure for their imports (especially for rule plugins which are more common in the ecosystem) — for performance and stability reasons. Some users had been experiencing very long import times with previous releases as a result of the layout of plugin imports. Users with affected plugins will begin to see a warning from this release onward, which can be resolved for their plugin by updating to a new version of that plugin which follows the guidelines. So like, lowkey a glow-up notice. 💅

🧚 Templater plugins

Templaters before this version would pass a make_template() callable to the slicing methods as part of being able to map the source file. This method would accept a str and return a jinja2.environment.Template object to allow the templater to render multiple variants of the template to do the slicing operation (which allows linting issues found in templated files to be mapped accurately back to their position in the unrendered source file). This approach is not very generalisable, and did not support templating operations with libraries other than jinja2. 🙈

As a result, we have amended the interface to instead pass a render_func() callable, which accepts a str and returns a str. This works fine for the jinja templater (and by extension the dbt templater) as they can simply wrap the original callable with a method that calls render() on the original Template object. It also however opens up the door to other templating engines, and in particular to remote templaters which might pass unrendered code over a HTTP connection for rendering. 🌈

Specifically:

  • The slice_file() method of the base templater classes no longer accepts an optional make_template argument or a templated_str argument.
  • Instead a render_func callable should be passed which can be called to generate the templated_str on demand.
  • Unlike the optional make_template — render_func is not optional and should always be present. It's giving mandatory. periodt. 👑

👑 Rule plugins

We recommend that the module in a plugin which defines all of the hook implementations (anything using the @hookimpl decorator) must be able to fully import before any rule implementations are imported. More specifically, SQLFluff must be able to both import and run any implementations of get_configs_info() before any plugin rules (i.e. any derivatives of BaseRule) are imported. Because of this, we recommend that rules are defined in a separate module to the root of the plugin and then only imported within the get_rules() method. 🎀

Importing in the main body of the module was previously our recommendation and so may be the case for versions of some plugins. If one of your plugins does use imports in this way, a warning will be presented from this version onward, recommending that you update your plugin. 😭

See the Developing Plugins section of the docs (anchor: #developingpluginsref) for an example.

🚀 Upgrading from 1.x to 2.0

Upgrading to 2.0 brings several important breaking changes. Buckle up, bestie — this is the big one. 🔥

  • All bundled rules have been recoded, both from generic L00X formats into groups within similar codes (e.g. an aliasing group with codes of the format AL0X), but also given names to allow much clearer referencing (e.g. aliasing.column).
  • Rule Configuration (anchor: #ruleconfig) now uses the rule name rather than the rule code to specify the section. Any unrecognised references in config files (whether they are references which do match existing rules by code or alias, or whether the match no rules at all) will raise warnings at runtime. ⚠️
  • A complete re-write of layout and whitespace handling rules (see Layout & Whitespace Configuration, anchor: #layoutref), and with that a change in how layout is configured (see Configuring Layout, anchor: #layoutconfig) and the combination of some rules that were previously separate. One example of this is that the legacy rules L001, L005, L006, L008, L023, L024, L039, L048 & L071 have been combined simply into LT01 (anchor: #rule-LT01). Eight rules walked so one rule could run. 💀✨

To upgrade smoothly between versions, we recommend the following sequence:

  1. The upgrade path will be simpler if you have a slimmer configuration file. Before upgrading, consider removing any sections from your configuration file (often .sqlfluff, see Configuration, anchor: #config) which match the current Default Configuration (anchor: #defaultconfig). There is no need to respecify defaults in your local config if they are not different to the stock config. Less is more, bestie. 🎀
  2. In a local (or other non-production) environment, upgrade to SQLFluff 2.0.x. We recommend using a compatible release specifier such as ~=2.0.0, to ensure any minor bugfix releases are automatically included. (That specifier deets: https://peps.python.org/pep-0440/#compatible-release 💜)
  3. Examine your configuration file (as mentioned above), and evaluate how rules are currently specified. We recommend primarily using either rules or exclude_rules rather than both, as detailed in Enabling and Disabling Rules (anchor: #ruleselection). Using either the sqlfluff rules CLI command or the online Rules Reference (anchor: #ruleref), replace all references to legacy rule codes (i.e. codes of the form L0XX). Specifically:
    • In the rules and exclude_rules config values. Here, consider using group specifiers or names to make your config simpler to read and understand (e.g. capitalisation, is much more understandable than CP01,CP02,CP03,CP04,CP05, but the two specifiers will have the same effect). Note that while legacy codes will still be understood here (because they remain valid as aliases for those rules) — you may find that some rules no longer exist in isolation and so these references may be misleading. e.g. L005 is now an alias for layout.spacing (anchor: #rule-layout.spacing) but that rule is much more broad ranging than the original scope of L005, which was only spacing around commas. 💅
    • In Rule Configuration (anchor: #ruleconfig). In particular here, legacy references to rule codes are no longer valid, will raise warnings, and until resolved, the configuration in those sections will be ignored. The new section references should include the rule name (e.g. [sqlfluff:rules:capitalisation.keywords] rather than [sqlfluff:rules:L010]). This switch is designed to make configuration files more readable, but we cannot support backward compatibility here without also having to resolve the potential ambiguity of the scenario where both code-based and name-based are both used. 😭
    • Review the Configuring Layout (anchor: #layoutconfig) documentation, and check whether any indentation or layout configuration should be revised.
  4. Check your project for In-File Configuration Directives (anchor: #in-file-config) which refer to rule codes. Alter these in the same manner as described above for configuration files. 📝
  5. Test linting your project for unexpected linting issues. Where found, consider whether to use sqlfluff fix to repair them in bulk, or (if you disagree with the changes) consider changing which rules you enable or their configuration accordingly. In particular you may notice:
    • The indentation rule (L003 as was, now LT02, anchor: #rule-LT02) has had a significant rewrite, and while much more flexible and accurate, it is also more specific. Note that Hanging Indents (anchor: #hangingindents) are no longer supported, and that while not enabled by default, many users may find the enabling Implicit Indents (anchor: #implicitindents) fits their organisation's style better. 👀
  • The spacing rule (LT01: layout.spacing) is giving way more main character energy now — its scope got so much wider, so it might clock spacing issues that literally nobody was enforcing before. If you're like "bestie no", you can override the sqlfluff:layout sections of the config with different settings (or just more liberal ones, like any). Slay, your call. 💅

✨ Example 2.0 config ✨

Okay so to make all those points make sense, here's a little illustrative example config for a 2.0-compatible project. Heads up: it's intentionally brief and only sets the values that differ from the default config. 🎀

[sqlfluff]
dialect = snowflake
templater = dbt
max_line_length = 120

# Exclude some specific rules based on a mixture of codes and names
exclude_rules = RF02, RF03, RF04, ST06, ST07, AM05, AM06, convention.left_join, layout.select_targets

[sqlfluff:indentation]
# Enabling implicit indents for this project.
# See https://docs.sqlfluff.com/en/stable/perma/indent_locations.html
allow_implicit_indents = True

# Add a few specific rule configurations, referenced by the rule names
# and not by the rule codes.
[sqlfluff:rules:capitalisation.keywords]
capitalisation_policy = lower

[sqlfluff:rules:capitalisation.identifiers]
capitalisation_policy = lower

[sqlfluff:rules:capitalisation.functions]
extended_capitalisation_policy = lower

# An example of setting a custom layout specification which
# is more lenient than default config.
[sqlfluff:layout:type:set_operator]
line_position = alone

🌈 Upgrading to 1.4 🌈

This release is a whole vibe shift — lots of internal changes, and honestly it's a prelude to 2.0.0. Specifically, these config values changed, bestie:

  • sqlfluff:rules:L007:operator_new_lines got glow-upped into sqlfluff:layout:type:binary_operator:line_position.
  • sqlfluff:rules:comma_style and sqlfluff:rules:L019:comma_style both got consolidated into sqlfluff:layout:type:comma:line_position.

If you already set any of these in your config, don't panic 🫶 — they get automatically translated to the new values at runtime, and a warning pops up to tell you. Wanna silence that warning? Just update your config file to the new values. For all the deets on configuring layout, see Configuring Layout.

💜 Upgrading to 1.3 💜

Okay this one brings several potentially breaking changes to the underlying parse tree. If you're using the CLI tool just for linting, you should notice literally nothing. But if your application relies on the structure of the SQLFluff parse tree, or on the naming of certain elements in the yaml format, then this might not be a drop-in replacement. Specifically:

  • A new end_of_file meta segment got added at the end of the parse structure. ✨
  • A new template_loop meta segment got added to signify a jump backward in the source file within a loop structure (like a jinja for loop, iykyk).
  • Some raw segments now have much more specific types. In particular, identifier and literal type segments will now show up in the parse tree with their more specific type (which used to be called name), e.g. naked_identifier, quoted_identifier, numeric_literal, etc...

If you're using the python api: the parent type (such as identifier) will still register if you call .is_type("identifier"), because that function checks all the inherited types. But the eventual type returned by .get_type() will now be (in most cases) what used to be accessible at .name. Just so you know — the name attribute is getting deprecated in a future release. 💀

🦋 Upgrading to 1.2 🦋

This release introduces the ability to automatically skip large files, and it sets default limits on the maximum file size before a file gets skipped. Users should see a performance gain — but you may also get warnings associated with those skipped files. It's giving "trade-offs, but make them cute". 😌

⭐ Upgrades pre 1.0 ⭐

  • 0.13.x — new rule for quoted literals, an option to remove hanging indents in rule L003, and the introduction of ignore_words_regex.
  • 0.12.x — dialect is now mandatory, the spark3 dialect got renamed to sparksql, and datatype capitalisation was extracted out of L010 into its own rule L063.
  • 0.11.x — rule L030 changed to use extended_capitalisation_policy.
  • 0.10.x — removed support for older dbt versions < 0.20 and stopped fix from attempting to fix unparsable SQL.
  • 0.9.x — refinement of the Simple API, dbt 1.0.0 compatibility, and the official SQLFluff Docker image.
  • 0.8.x — an improvement to the performance of the parser, a rebuild of the Jinja Templater, and a progress bar for the CLI. 📊
  • 0.7.x — extracted the dbt templater to a separate plugin and removed the exasol_fs dialect (now merged in with the main exasol).
  • 0.6.x — introduced parallel processing, which necessitated a big re-write of several innards.
  • 0.5.x — introduced some breaking changes to the API.
  • 0.4.x — dropped python 3.5, added the dbt templater, source mapping and also introduced the python API.
  • 0.3.x — drops support for python 2.7 and 3.4, and also reworks the handling of indentation linting in a potentially not backward compatible way.
  • 0.2.x — added templating support and a big restructure of rules, and changed how users might interact with SQLFluff on templated code.
  • 0.1.x — involved a major re-write of the parser, completely changing the behaviour of the tool with respect to complex parsing.

👑 SQLFluff in the Wild 👑

Wanna find other people using SQLFluff in production use cases? Wanna brag about how you're using it? Just wanna show solidarity with the project and drop a testimonial? Iconic. 🫶

Just add a section below by raising a PR on GitHub by editing this file ✏️: https://github.com/sqlfluff/sqlfluff/edit/main/docs/source/inthewild.rst

  • SQLFluff in production dbt projects at tails.com. (http://www.getdbt.com/, https://tails.com) We use the SQLFluff cli as part of our CI pipeline in codeship (https://codeship.com) to enforce certain styles in our SQL codebase (with over 650 models) and keep code quality high. Contact @alanmcruickshank (https://github.com/alanmcruickshank).
  • Netlify (https://www.netlify.com)'s data team uses SQLFluff with dbt (http://www.getdbt.com/) to keep code quality in more than 350 models (and growing). Previously, we had our SQL Guidelines defined in a site hosted with Netlify, and now we're enforcing these rules in our CI workflow thanks to SQLFluff.
  • Drizly's (https://www.drizly.com) analytics team uses SQLFluff with dbt (http://www.getdbt.com/) for over 700 models as part of our CI checks in GitHub. Before SQLFluff, we had SQL best practices outlined in a google doc and had to manually enforce through PR comments. We're now able to enforce much of our style guide automatically through SQLFluff.
  • Petal's (https://www.petalcard.com) data-eng team runs SQLFluff on our 100+ model dbt (http://www.getdbt.com/) project. As a pre-commit hook and as a CI check, SQLFluff helps keep our SQL readable and consistent.
  • Surfline (https://www.surfline.com/)'s Analytics Engineering team implemented SQLFluff as part of our continuous integration (CI) suite across our entire dbt (http://www.getdbt.com/) project (700+ models). We implement the CI suite using GitHub Actions and Workflows (https://docs.github.com/en/actions). The benefits of using SQLFluff at Surfline are:
    • The SQL in our dbt models is consistent and easily readable.
    • Our style guide is maintained as code, not a README that is rarely updated.
    • Reduced burden on Analytics Engineers to remember every single style rule.
    • New Analytics Engineers can quickly see and learn what "good SQL" looks like at Surfline and start writing it from day 1.
  • The HTTP Archive (https://httparchive.org) uses SQLFluff to automatically check for quality and consistency of code submitted by the many contributors to this project. In particular our annual Web Almanac (https://almanac.httparchive.org) attracts hundreds of volunteers to help analyse our BigQuery dataset and being able automatically lint Pull Requests through GitHub Actions is a fantastic way to help us maintain our growing repository of over a thousand queries (https://github.com/HTTPArchive/almanac.httparchive.org/tree/main/sql).
  • Brooklyn Data Co (https://www.brooklyndata.co) has a dbt_artifacts (https://github.com/brooklyn-data/dbt_artifacts) dbt package from which runs SQLFluff in CI to lint pull requests automatically. It uses the GitHub Actions workflow (https://github.com/sqlfluff/sqlfluff-github-actions/tree/main/menu_of_workflows/surfline) contributed by Greg Clunies, with annotations on pull requests to make it easy for contributors to see where their SQL has failed any rules. See an example pull request with SQLFluff annotations (https://github.com/brooklyn-data/dbt_artifacts/pull/74/files).
  • Markerr (https://www.markerr.com) has tightly integrated SQLFluff into our CI/CD process for data model changes and process improvements. Since adopting SQLFluff across the organization, the clarity of our SQL code has risen dramatically, freeing up review time to focus on deeper data and process-specific questions.
  • Symend (https://www.symend.com) has a microservices platform supporting our SaaS product. We use SQLFLuff in the CI/CD process of several of our data-oriented microservices. Among other things, it validates our database migration scripts, deployed using schemachange (https://github.com/Snowflake-Labs/schemachange) and we have near-term plans to implement it for our dbt (http://www.getdbt.com/) projects.
  • At CarePay (https://www.carepay.com) we use SQLFLuff to lint and fix all our dbt models as well as several other SQL heavy projects. Locally we use SQLFluff with pre-commit and have also integrated it into our CI/CD pipelines.
  • Core Analytics Team from Typeform (https://www.typeform.com/) and videoask (https://www.videoask.com/) uses SQLFluff in the production dbt (http://www.getdbt.com/) project for building our datawarehouse layer for both products:
    • We use it locally in our day to day work, helping us to write cleaner code.
    • We added SQLFluff to our CI processes, so during a PR we can check that any new or modified sql file has a consistent and easy-to-read format.

🌸 SQLFluff Slack 🌸

We have a fast-growing community on Slack (https://join.slack.com/t/sqlfluff/shared_invite/zt-2qtu36kdt-OS4iONPbQ3aCz2DIbYJdWg) — come and join us! Bestie, the vibes are immaculate. 💖

🐦 SQLFluff on Twitter 🐦

Follow us On Twitter @SQLFluff (https://twitter.com/SQLFluff) for announcements and other related posts.

  • Index <>
  • Module Index <>
  • Search Page <>

🎀 Author 🎀

Alan Cruickshank

Okay bestie, legal moment, don't scroll 🫶 — this whole thing is:

Copyright 2024, Alan Cruickshank

That's the vibe, that's the receipts, that's the who to thank when your SQL is immaculate ✨💅

And that's a wrap on SQLFLUFF(1), fr fr 🎀 You made it to the very last crumb of the page, which is honestly main character energy 👑 — the manual comes from the SQLFluff source, this rendering is dated Feb 23, 2026, and the page answers to the name SQLFLUFF(1) 🌸 Go format something gorgeous, bestie 🦋💖

📦 where this page comes from