Why Gulp?

First of all, you could do the things I am going to show you with any building tool (e.g. grunt) as well. The TYPO3 core uses grunt for its build and it would be totally fine to go for any alternative. But for this post it will be gulp for no particular reason.

Gulp is a popular task runner powered by Node.js that is well known for its convenience when it comes to building your JavaScript and CSS files. TYPO3 can concatenate and minify JS and CSS sources itself you might say. But most of the time this is not needed and the result is not as good as it will be with gulp. Why would you generate your JS and CSS "on the fly" if it stays static between deployments anyway? And gulp can do much more for you. Minify your images, running sniffer over your code before merging it or even run unittests to assure you are not going to deploy a bug.

Disclaimer: This is not another gulp tutorial. There are plenty good ones out there and there is no reason to add another one on top. What is not out there is a guide to use gulp with TYPO3. And though it is not exactly rocket science to combine both technologies, I am going to tackle that now.

So, what we are going to do, is to see how we can integrate a gulp build smoothly into our TYPO3. Are you on board? Then read on!

How to integrate gulp in your TYPO3 project?

There are quite some possibilities to get TYPO3 and gulp married. You could fetch everything in an extension. But I don't like that. Because I rather have my JS and CSS (or less or sass) sources inaccessible through the web. So why put them inside typo3conf/ext if everything we want to include is the final build?

Lets see how the gitroot of an example TYPO3 project looks like. Btw. I put the whole gulp example on github so you can check it out and play around with it. So this is the git root:

ls -la gulp-example
-rw-r--r--   1 user  group    379 17 Mar 15:25 composer.json
-rw-r--r--   1 user  group  34983 17 Mar 15:26 composer.lock
drwxr-xr-x  10 user  group    340 17 Mar 18:00 src
drwxr-xr-x  12 user  group    408 17 Mar 16:07 vendor
drwxr-xr-x  10 user  group    340 17 Mar 17:34 web

I put everything regarding the build in the src folder. This way it is not within the document root (the web folder) of a composer driven TYPO3 system (If you want to know more about TYPO3 and composer, you could head over to my introduction on the topic).

Top

A dedicated folder for the source of the build

So let us have a look in the src folder:

ls -la src
total 40
drwxr-xr-x   10 user  group    340 17 Mar 18:00 .
drwxr-xr-x   11 user  group    374 17 Mar 17:58 ..
-rw-r--r--    1 user  group     41 17 Mar 16:17 .bowerrc
-rw-r--r--    1 user  group     98 17 Mar 16:17 bower.json
drwxr-xr-x    9 user  group    306 17 Mar 17:51 gulp_tasks
-rw-r--r--    1 user  group    943 17 Mar 17:39 gulpconfig.js
-rwxr-xr-x    1 user  group    373 17 Mar 18:00 gulpfile.js
-rwxr-xr-x    1 user  group    461 17 Mar 17:27 package.json
drwxr-xr-x    5 user  group    170 17 Mar 17:33 resources

Lets have a quick look on whats going on here: We use Bower to fetch some 3rd party JavaScript libraries. The bower.json contains the dependencies for our website and in the .bowerrc we specify the path (resources/bower_components) those libraries should be downloaded to.

The package.json contains all node modules that we need for our gulp tasks. The gulpfile.js and gulpconfig.js as well as the gulp_tasks folder are explained in a moment when we have a look at the structure I liked the most so far when it comes to gulp.

Finally the resources folder contains our JavaScript and our Less. It also contains the packages downloaded by bower (in our case thats only bootstrap. And jQuery as a dependency of bootstrap). You get the full picture after the following commands

cd src/
npm install
bower install
gulp build

We exclude all 3rd party code from versioning by adding the node modules and the JS libraries fetched by bower to our .gitignore.

cat .gitignore | grep src
/src/node_modules
/src/resources/bower_components

Next we're going to see how gulp is set up and where the final build is going.

Top

My (preferred) gulp structure

The following is to a large degree a matter of taste. There might be many other and better ways to set up a gulp workflow. Some things however, are of importance to me:

  • I want a single file that holds (reusable) configuration for all my tasks.
  • I want one file per task so maintenance is easier.
  • I want gulp to tell me what tasks are available and what they do.
  • I want automated generation of the build while developing.

We are going to cover those four aspects next. They have nothing to do with TYPO3, so you might skip them as well.

Top

gulpconfig.js - All configuration in one file

The gulpconfig.js holds the configuration of all the tasks. It is just a simple module that returns a JavaScript object. Let's see a part of it (the complete file is included in the github example here):

module.exports = function () {
    var resources = 'resources/',
        bowerPath = resources + 'bower_components/',
        lessPath = resources + 'less/',
        myJs = resources + 'js/**/*.js',
        buildPath = '../web/build/';
    config = {
        "js": {
            "src": [
                bowerPath + "jquery/dist/jquery.js",
                bowerPath + "bootstrap/js/tab.js",
                myJs
            ],
            "watch": [
                myJs
            ],
            "dest": buildPath + "js/",
            "destFileName": "footer.js"
        }
    };
    return config;
};

If a tasks needs some configuration it can load the config object by requiring it with var config = require('../gulpconfig.js')();.

So the first Requirement is met, we have a single file for all the configuration. Now let's put every gulp task in its own file.

Top

gulp_tasks - Every task in its own file

The gulpfile.js usually contains all the gulp tasks, leading to a rather large and confusing file. However there is a simple solution to that matter: make use of the require-dir module. This basically allows us to put our gulp tasks in dedicated files in a special folder and then requiring this folder in our gulpfile.js.

We named the folder gulp_tasks and our gulpfile.js looks like this:

/* global require */
var requireDir = require('require-dir'),
    dir = requireDir('./gulp_tasks');

// The actual tasks are configured in external files under ./gulp_tasks/
// https://github.com/gulpjs/gulp/blob/master/docs/recipes/split-tasks-across-multiple-files.md
// For the default task, that gets triggered with the bare 'gulp' command, see ./gulp_tasks/default.js

Now we need an easy way to see what tasks are available and what they do for us. Let's tackle that next.

Top

Don't get lost: gulp-task-listing and gulp-help

If one just types gulp into the command line, gulp will execute a task called default. So let us take a look on our default task that is defined in default.js under gulp_tasks. Isn't that convenient?

/*global require*/
var config = require('../gulpconfig.js')(),
    plugins = require('gulp-load-plugins')(),
    gulp = plugins.help(require('gulp'), config.gulphelp.options);

gulp.task('default', plugins.taskListing);

In line 2, we load the config from our gulpconfig.js. Line 3 uses the module gulp-load-plugins, a very nice module that loads all other gulp plugins for us, so we only need to require this even if we need multiple plugins in a single gulp task. We will see this later. In line 4, gulp is initialized with the gulp-help plugin, that allows us to add a description to tasks and render some helpful output when executed with gulp help.

Finally the actual task is defined in line 6. In this case it does nothing more than running the gulp-task-listing plugin. The output will be something like this:

gulp
[16:34:39] Using gulpfile ~/vhosts/gulp-example/dev/application/src/gulpfile.js
[16:34:39] Starting 'default'...

Main Tasks
------------------------------
    build
    default
    dev
    help

Sub Tasks
------------------------------
    css-clean
    css-compile
    dev-watch
    js-clean
    js-compile

[16:34:39] Finished 'default' after 1.35 ms

It will automatically list every task that has no dash in its name as a main task. We now see that the task help is available as a main task. Let's run it.

gulp help
[16:37:39] Using gulpfile ~/vhosts/gulp-example/dev/application/src/gulpfile.js
[16:37:39] Starting 'help'...

Usage
  gulp [TASK] [OPTIONS...]

Available tasks
  build        Triggers a complete build.
  dev          Main task for development (goes into watch state).

[16:37:39] Finished 'help' after 1.29 ms

The gulp-help plugin renders every task that has a description as well as some general help on how to use gulp (which you had to know beforehand or else you could not have executed the help task ...). Pretty neat. To tick off the last point from our list above let's see the dev task that keeps our build up to date while we are developing.

Top

Stay up to date while developing

It would be pretty inconveninent if a developer would need to run gulp build everytime he or she changes something. Fortunately this is not necessary thanks to gulp-watch. We can specify as many files and/or directories as we like and each time a file inside those selections is changed a build is triggered automatically. This is ideal for developing and therefore included in the dev task.

To put everything we saw until now together, let us have a look at the dev task, that is defined - you guessed it - in gulp_tasks/dev.js.

/*global require*/
var config = require('../gulpconfig.js')(),
    plugins = require('gulp-load-plugins')(),
    gulp = plugins.help(require('gulp'), config.gulphelp.options);

gulp.task('dev', 'Main task for development (goes into watch state).', function () {
    'use strict';
    gulp.start(['js-compile', 'css-compile', 'dev-watch']);
});

So only one question remains: where does the build go?

Top

Where does the build go?

If you payed attention, you already know. The final product(s) of our building process need to be of course accessible by web requests. I like to put them into the folder web/build. This was specified in our gulpconfig.js and that is where the build goes. Let us have a look:

ls -la web/build
total 0
drwxr-xr-x   4 user  group  136 17 Mar 17:51 .
drwxr-xr-x  10 user  group  340 17 Mar 17:34 ..
drwxr-xr-x   5 user  group  170 17 Mar 17:51 css
drwxr-xr-x   5 user  group  170 17 Mar 17:51 js
ls -lah web/build/js
total 760
drwxr-xr-x  5 user  group   170B 17 Mar 17:51 .
drwxr-xr-x  4 user  group   136B 17 Mar 17:51 ..
-rw-r--r--  1 user  group   257K 17 Mar 17:51 footer.js
-rw-r--r--  1 user  group    86K 17 Mar 17:51 footer.min.js
-rw-r--r--  1 user  group    30K 17 Mar 17:51 footer.min.js.gz

We see that the file footer.js has a size of more than 250kb where the uglified and compressed version footer.min.js.gz is only 30kb small.

The content of web/build depends on your gulp tasks. It is where compressed and web-optimized layout-images could be located as well as fonts or other resources for your website. All we have left to do, is including these files in TYPO3.

Top

Include the build with TypoScript

The last piece in the puzzle is obviously to include the generated files in your actual page setup:

page.includeJSFooter.gulp = /build/js/footer.min.js
page.includeJSFooter.gulp.excludeFromConcatenation = 1
page.includeJSFooter.gulp.disableCompression = 1
page.includeCSS.gulp = /build/css/main.min.css
page.includeCSS.gulp.excludeFromConcatenation = 1
page.includeCSS.gulp.disableCompression = 1

[applicationContext = Development*]
page.includeJSFooter.gulp = /build/js/footer.js
[end]

Of course you don't have to excludeFromConcatenation and disableCompression if you don't have this turned on in your TYPO3 in the first place. If you want to know more about the applicationContext condition in the end of that snippet, head over to my article about it.

Thank you for reading until the very end. I would love to hear if somebody has experiences with gulp and TYPO3 or even suggestions to improve the workflow further. Just write me an email or contact me on slack.

Top

Further Information