Blog

The entity_generate process plugin receives a value and checks if there is an entity with that name. If the term exists then it uses it and if it does not then creates it (which is precisely what I need).

So, here is a snippet of the article migration YAML file using the entity_generate plugin:

id: blog
migration_group: Drupal
label: Blog
source:
  plugin: d7_node
  node_type: blog
destination:
  plugin: entity:node
process:
  status: status
  created: created
  field_tags:
    plugin: sub_process
    source: field_tags
    process:
      target_id:
      - plugin: entity_generate
        source: name
        value_key: name
        bundle_key: vid
        bundle: tags
        entity_type: taxonomy_term
        ignore_case: true
        …

In our field_tags field we are using the Drupal 7 field_tags Values We are going to read the entities and pass that value into the entity_generate plugin to create the entities. In this example, there is a problem, the d7_node migrate plugin (included in the migrate module) provides the taxonomy terms IDs and this will make the entity_generate plugin to create some taxonomy terms using the IDs as the term names, and this is not what we want.

So what I need to do is to get from somewhere the terms names, not their ids, to do that I need to add an extra source property.

First, we need to create a new custom module and in there create a source plugin which extends the Node process plugin, something like this (let's say that our custom module’s name is my_migration):

Create the file:

my_migration/src/Plugin/migrate/source/MyNode.php

And the file should contain this code:

namespace Drupal\my_migration\Plugin\migrate\source;

use Drupal\migrate\Row;
use Drupal\node\Plugin\migrate\source\d7\Node;

/**
 * Adds a source property with the taxonomy term names.
 *
 * @MigrateSource(
 *   id = “my_node",
 *   source_module = "node"
 * )
 */
class MyNode extends Node {

  public function prepareRow(Row $row) {
    $nid = $row->getSourceProperty('nid');
    // Get the taxonomy tags names.
    $tags = $this->getFieldValues('node', 'field_tags', $nid);
    $names = [];
    foreach ($tags as $tag) {
      $tids[] = $tag['tid'];
    }
    if (!$tids) {
      $names = [];
    }
    else {
      $query = $this->select('taxonomy_term_data', 't');
      $query->condition('tid', $tids, 'IN');
      $query->addField('t', 'name');
      $result = $query->execute()->fetchCol();
      $names[] = ['name' => $result['name']];
      foreach ($result as $term_name) {
        $names[] = ['name' => $term_name];
      }
    }
    $row->setSourceProperty('field_tags_names', $names);
    return parent::prepareRow($row);
  }
}

It does the following things:

It reads the nid of the article
Gets all the terms IDs of the field_tags field
For each ID, it gets the name of the term and put it in the `names` array
Finally, it sets this value inside the $row using the `setsourceProperty` method.

Now our rows will have a property called fields_tags_names with the terms names, and we can pass this data to the entity_generate plugin.

We need to make a few adjustments in our initial migration file, first and most important update the source plugin to use our new source plugin:

source:
  plugin: my_node
  …

And update the source in the field_tags field to use the new field_tags_names source property.

  …
  field_tags:
    plugin: sub_process
    source: field_tags_names
  ….

The final migration file looks like this:

id: blog
migration_group: Drupal
label: Blog
source:
  plugin: my_node
  node_type: blog
destination:
  plugin: entity:node
process:
  status: status
  created: created
  field_tags:
    plugin: sub_process
    source: field_tags_names
    process:
      target_id:
      - plugin: entity_generate
        source: name
        value_key: name
        bundle_key: vid
        bundle: tags
        entity_type: taxonomy_term
        ignore_case: true
        …

And that’s it; if we run the migration, it will create on the fly the terms if they do not exist and use them if they do exist.

A mobile version of Vulk website's home page.

Functionality designed to your life is the Agaric Design signature. Utilizing open source, free software from around the world, Agaric Design websites are impeccably crafted with a modern, sophisticated and understated spirit.

Our guiding philosophy is to bring open source and usability that works effortlessly in your day to day life.

I know we will become a favorite in your contacts list. And thank you! I consider that the ultimate compliment.

Dan Hakimzadeh

Have you ever been asked to log into a website while you are viewing a page? And after doing so you get redirected to some page other than the one you were reading? This is an obvious and rather common usability problem. When this happens people lose track of what they were doing and some might not even bother to go back. Let's find out how to solve this in Drupal 8.

In a recent project a client wisely requested exactly that: whenever a user logs into the site, redirect them to the page they were before clicking the login link. This seemed like a very common request so we looked for a contrib module that provided the functionality. Login Destination used to do it in Drupal 7. Sadly the Drupal 8 version of this module does not provide the functionality yet.

Other modules, and some combinations of them, were tested without success. Therefore, we built Login Return Page. It a very small module that just does one thing and it does it well: it appends destination=/current/page to all the links pointing to /user/login effectively redirecting users to the page they were viewing before login. The project is waiting to be approved before promoting it to full project.

Have you had a similar need? Are there other things you are requested to do after login? Please share them in the comments.

UPDATE: It seems to this is a regression from Drupal 7 and there is an issue that would fix it. Thanks to Wim Leers for letting me know about it.

We look forward to being in communication, perchance to work together or just to meet in person sometime!

Sign up to be notified when Agaric gives a migration training:

Getting started with Drupal 11

Drupal 11 content migrations

Upgrading to Drupal 11 using the Migrate API

Drupal end-to-end testing with Behat

When There’s Not a Module for That: How to Make and Maintain a Module

Here is how to deal with the surprising-to-impossible-seeming error "Unable to uninstall the MySQL module because: The module 'MySQL' is providing the database driver 'mysql'.."

Like, why is it trying to uninstall anything when you are installing? Well, it is because you are installing with existing configuration— and your configuration is out-of-date. This same problem will happen on configuration import on a Drupal website, too. (See update below for those steps!)

Really this error message is a strong reminder to always run database updates and then commit any resulting configuration changes after updating Drupal core or module code.

And so the solution is to roll back the code to Drupal 9.3, do your installation from configuration, and then run the database updates, export configuration, and commit the result.

For example:

git checkout <commit-hash-of-earlier-composer-lock>
composer install
drush -y site:install drutopia --existing-config
git checkout main
composer install
drush -y updb
drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration updates from Drupal 9.4 upgrade"

The system update enable_provider_database_driver is the post-update hook that is doing the work here to "Enable the modules that are providing the listed database drivers." Pretty cool feature and a strong reminder to always, always run database updates and commit any configuration changes immediately after any code updates!

Update: Steps on for resolving the unable to uninstall MySQL module error on configuration import

This is what you probably already did, before the drush -y cim failed (luckily, luckily it failed).

composer update
drush -y updb

All that is great! Now continue, not with a config import, but with a config export:

drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration updates from Drupal 9.4 upgrade"

Remember, after every composer update and database update, you need to do a configuration export and commit the results— database updates can change configuration, and if you do not commit those, you will undo these intentional and potentially important changes on a configuration import. If you ran into this problem on a configuration import, it is a sign of breakdown in discipline in following these steps!

Every time you bring in code changes with composer update all this must be part and parcel:

composer update
drush -y updb
drush -y cex
git status # Review what is here; git add -p can also help
git add config/
git commit -m "Apply configuration from database updates"

Drupal 8 has a great AJAX Form API which includes some tools to create modal dialogs using the jQuery modal library. The Examples module even demonstrates how to create a custom form and display it in a modal window. But what if what you want to do is display an already created form in a modal? How do we do that? Let's see how to do it with an example. Let's display the node add form in a modal window.

The first thing that we need to do is create a link which will trigger the modal when the user clicks it. The only special things that this link needs to have are a few attributes that will let Drupal know to display the contents of the link in a dialog:

 
<a href="/node/add/article"
  class="use-ajax"
  data-dialog-type="modal"
  data-dialog-options="{'width':800,'height':500}">
    Create Node
</a>

Drupal also needs to include the JavaScript libraries which will read these attributes and make them work, so let's add the following libraries to your module's dependencies (in your equivalent to this example's modal_form_example.libraries.yml file).

  
  dependencies:
     'core/drupal.dialog.ajax',
     'core/jquery.form',

If you are unsure about how to add libraries on Drupal 8 you can consult the documentation to either add it in a theme or add it in a custom module. At the end of the post I will provide a repository with the code where I added the libraries in a block.

And that's it! If you click the link, the form will be displayed in a modal dialog! Drupal will automatically detect that you are sending an AJAX request and will display just the form so you won't need to worry about removing the rest of the blocks or hiding undesired markup.

The last thing missing, is what will happen if the user creates a node? By default, the node will redirect the user to another page but if we want to just close the modal dialog and leave the user on the same page we need to tell the form to do that. For this we are going to alter the form and add an AJAX command letting Drupal know that we want to close the dialog as soon as the node is created. In the .module file of a custom module we will add this code:

  
use Drupal\Core\Form\FormStateInterface;
use Drupal\Core\Ajax\AjaxResponse;
use Drupal\Core\Ajax\CloseModalDialogCommand;
use Drupal\Core\Ajax\RedirectCommand;

/**
 * Implements hook_form_alter().
 */
function modal_form_example_form_node_article_form_alter(&$form, FormStateInterface $form_state, $form_id) {
  $form['actions']['submit']['#submit'][] = '_modal_form_example_ajax_submit';
  $form['actions']['submit']['#attributes']['class'][] = 'use-ajax-submit';
}

/**
 * Close the Modal and redirect the user to the homepage.
 *
 * @param array $form
 *   The form that will be altered.
 * @param \Drupal\Core\Form\FormStateInterface $form_state
 *   FormState Object.
 */
function _modal_form_example_ajax_submit(array $form, FormStateInterface &$form_state) {
  $response = new AjaxResponse();
  $response->addCommand(new CloseModalDialogCommand());
  $form_state->setResponse($response);
}

The first function adds an extra submit function (which will be executed after Drupal finishes processing the node) and the second function adds the command to close the Dialog when the node has been created.

We can do this with practically any form in Drupal and you can add extra commands to do more complex things. Here are two resources:

List of all the possible commands available in core (you can also create your own)
Agaric's Modal Form Example module

Flock of birds flying.

In recent posts we have explored the Migrate Plus and Migrate Tools modules. They extend the Migrate API to provide migrations defined as configuration entities, groups to share configuration among migrations, a user interface to execute migrations, among other things. Yet another benefit of using Migrate Plus is the option to leverage the many process plugins it provides. Today, we are going to learn about two of them: `entity_lookup` and `entity_generate`. We are going to compare them with the `migration_lookup` plugin, show how to configure them, and explain their compromises and limitations. Let’s get started.

What is the difference among the migration_lookup, entity_lookup, entity_generate plugins?

In the article about migration dependencies we covered the `migration_lookup` plugin provided by the core Migrate API. It lets you maintain relationships among entities that are being imported. For example, if you are migrating a node that has associated users, taxonomy terms, images, paragraphs, etc. This plugin has a very important restriction: the related entities must come from another migration. But what can you do if you need to reference entities that already exists system? You might already have users in Drupal that you want to assign as node authors. In that case, the `migration_lookup` plugin cannot be used, but `entity_lookup` can do the job.

The `entity_lookup` plugin is provided by the Migrate Plus module. You can use it to query any entity in the system and get its unique identifier. This is often used to populate entity reference fields, but it can be used to set any field or property in the destination. For example, you can query existing users and assign the `uid` node property which indicates who created the node. If no entity is found, the module returns a `NULL` value which you can use in combination of other plugins to provide a fallback behavior. The advantage of this plugin is that it does not require another migration. You can query any entity in the entire system.

The `entity_generate` plugin, also provided by the Migrate Plus module, is an extension of `entity_lookup`. If no entity is found, this plugin will automatically create one. For example, you might have a list of taxonomy terms to associate with a node. If some of the terms do not exist, you would like to create and relate them to the node.

Note: The `migration_lookup` offers a feature called stubbing that neither `entity_lookup` nor `entity_generate` provides. It allows you to create a placeholder entity that will be updated later in the migration process. For example, in a hierarchical taxonomy terms migration, it is possible that a term is migrated before its parent. In that case, a stub for the parent will be created and later updated with the real data.

Getting the example code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is `UD Config entity_lookup and entity_generate examples` whose machine name is `ud_migrations_config_entity_lookup_entity_generate`. It comes with one JSON migrations: `udm_config_entity_lookup_entity_generate_node`. Read this article for details on migrating from JSON files. The following snippet shows a sample of the file:


{
  "data": {
    "udm_nodes": [
      {
        "unique_id": 1,
        "thoughtful_title": "Amazing recipe",
        "creative_author": "udm_user",
        "fruit_list": "Apple, Pear, Banana"
      },
      {...},
      {...},
      {...}
    ]
  }
}

Additionally, the example module creates three users upon installation: 'udm_user', 'udm_usuario', and 'udm_utilisateur'. They are deleted automatically when the module is uninstalled. They will be used to assign the node authors. The example will create nodes of types "Article" from the standard installation profile. You can execute the migration from the interface provided by Migrate Tools at `/admin/structure/migrate/manage/default/migrations`.

Using the entity_lookup to assign the node author

Let’s start by assigning the node author. The following snippet shows how to configure the `entity_lookup` plugin to assign the node author:


uid:
  - plugin: entity_lookup
    entity_type: user
    value_key: name
    source: src_creative_author
  - plugin: default_value
    default_value: 1

The `uid` node property is used to assign the node author. It expects an integer value representing a user ID (`uid`). The source data contains usernames so we need to query the database to get the corresponding user IDs. The users that will be referenced were not imported using the Migrate API. They were already in the system. Therefore, `migration_lookup` cannot be used, but `entity_lookup` can.

The plugin is configured using three keys. `entity_type` is set to machine name of the entity to query: `user` in this case. `value_key` is the name of the entity property to lookup. In Drupal, the usernames are stored in a property called `name`. Finally, `source` specifies which field from the source contains the lookup value for the `name` entity property. For example, the first record has a `src_creative_author` value of `udm_user`. So, this plugin will instruct Drupal to search among all the users in the system one whose `name` (username) is `udm_user`. If a value if found, the plugin will return the user ID. Because the `uid` node property expects a user ID, the return value of this plugin can be used directly to assign its value.

What happens if the plugin does not find an entity matching the conditions? It returns a `NULL` value. Then it is up to you to decide what to do. If you let the `NULL` value pass through, Drupal will take some default behavior. In the case of the `uid` property, if the received value is not valid, the node creation will be attributed to the anonymous user (uid: 0). Alternatively, you can detect if `NULL` is returned and take some action. In the example, the second record specifies the "udm_not_found" user which does not exists. To accommodate for this, a process pipeline is defined to manually specify a user if `entity_lookup` did not find one. The `default_value` plugin is used to return `1` in that case. The number represents a user ID, not a username. Particularly, this is the user ID of "super user" created when Drupal was first installed. If you need to assign a different user, but the user ID is unknown, you can create a pseudofield and use the `entity_lookup` plugin again to finds its user ID. Then, use that pseudofield as the default value.

Important: User entities do not have bundles. Do not set the `bundle_key` nor `bundle` configuration options of the `entity_lookup`. Otherwise, you will get the following error: "The entity_lookup plugin found no bundle but destination entity requires one." Files do not have bundles either. For entities that have bundles like nodes and taxonomy terms, those options need to be set in the `entity_lookup` plugin.

Using the entity_generate to assign and create taxonomy terms

Now, let’s migrate a comma separated list of taxonomy terms. An example value is `Apple, Pear, Banana`. The following snippet shows how to configure the `entity_generate` plugin to look up taxonomy terms and create them on the fly if they do not exist:


field_tags:
  - plugin: skip_on_empty
    source: src_fruit_list
    method: process
    message: 'No src_fruit_list listed.'
  - plugin: explode
    delimiter: ','
  - plugin: callback
    callable: trim
  - plugin: entity_generate
    entity_type: taxonomy_term
    value_key: name
    bundle_key: vid
    bundle: tags

The terms will be assigned to the `field_tags` field using a process pipeline of four plugins:

`skip_on_empty` will skip the processing of this field if the record does not have a `src_fruit_list` column.
`explode` will break the string of comma separated files into individual elements.
`callback` will use the `trim` PHP function to remove any whitespace from the start or end of the taxonomy term name.
`entity_generate` takes care of finding the taxonomy terms in the system and creating the ones that do not exist.

For a detailed explanation of the `skip_on_empty` and `explode` plugins see this article. For the `callback` plugin see this article. Let’s focus on the `entity_generate` plugin for now. The `field_tags` field expects an array of taxonomy terms IDs (`tid`). The source data contains term names so we need to query the database to get the corresponding term IDs. The taxonomy terms that will be referenced were not imported using the Migrate API. And they might exist in the system yet. If that is the case, they should be created on the fly. Therefore, `migration_lookup` cannot be used, but `entity_generate` can.

The plugin is configured using five keys. `entity_type` is set to machine name of the entity to query: `taxonomy_term` in this case. `value_key` is the name of the entity property to lookup. In Drupal, the taxonomy term names are stored in a property called `name`. Usually, you would include a `source` that specifies which field from the source contains the lookup value for the `name` entity property. In this case it is not necessary to define this configuration option. The lookup value will be passed from the previous plugin in the process pipeline. In this case, the trimmed version of the taxonomy term name.

If, and only if, the entity type has bundles, you also must define two more configuration options: `bundle_key` and `bundle`. Similar to `value_key` and `source`, these extra options will become another condition in the query looking for the entities. `bundle_key` is the name of the entity property that stores which bundle the entity belongs to. `bundle` contains the value of the bundle used to restrict the search. The terminology is a bit confusing, but it boils down to the following. It is possible that the same value exists in multiple bundles of the same entity. So, you must pick one bundle where the lookup operation will be performed. In the case of the taxonomy term entity, the bundles are the vocabularies. Which vocabulary a term belongs to is associated in the `vid` entity property. In the example, that is `tags`. Let’s consider an example term of "Apple". So, this plugin will instruct Drupal to search for a taxonomy term whose `name` (term name) is "Apple" that belongs to the "tags" `vid` (vocabulary).

What happens if the plugin does not find an entity matching the conditions? It will create one on the fly! It will use the value from the source configuration or from the process pipeline. This value will be used to assign the `value_key` entity property for the newly created entity. The entity will be created in the proper bundle as specified by the `bundle_key` and `bundle` configuration options. In the example, the terms will be created in the `tags` vocabulary. It is important to note that values are trimmed to remove whispaces at the start and end of the name. Otherwise, if your source contains spaces after the commas that separate elements, you might end up with terms that seem duplicated like "Apple" and " Apple".

More configuration options

Both `entity_lookup` and `entity_generate` share the previous configuration options. Additionally, the following options are only available:
`ignore_case` contains a boolean value to indicate if the query should be case sensitive or not. It defaults to true.
`access_check` contains a boolean value to indicate if the system should check whether the user has access to the entity. It defaults to true.
`values` and `default_values` apply only to the `entity_generate` plugin. You can use them to set fields that could exist in the destination entity. An example configuration is included in the code for the plugin.

One interesting fact about these plugins is that none of the configuration options is required. The `source` can be skipped if the value comes from the process pipeline. The rest of the configuration options can be inferred by code introspection. This has some restrictions and assumptions. For example, if you are migrating nodes, the code introspection requires the `type` node property defined in the process section. If you do not set one because you define a `default_bundle` in the destination section, an error will be produced. Similarly, for entity reference fields it is assumed they point to one bundle only. Otherwise, the system cannot guess which bundle to lookup and an error will be produced. Therefore, always set the `entity_type` and `value_key` configurations. And for entity types that have bundles, `bundle_key` and `bundle` must be set as well.

Note: There are various open issues contemplating changes to the configuration options. See this issue and the related ones to keep up to date with any future change.

Compromises and limitations

The `entity_lookup` and `entity_generate` plugins violate some ETL principles. For example, they query the destination system from the process section. And in the case of `entity_generate` it even creates entities from the process section. Ideally, each phase of the ETL process is self contained. That being said, there are valid uses cases to use these plugins and they can you save time when their functionality is needed.

An important limitation of the `entity_generate` plugin is that it is not able to clean after itself. That is, if you rollback the migration that calls this plugin, any created entity will remain in the system. This would leave data that is potentially invalid or otherwise never used in Drupal. Those values could leak into the user interface like in autocomplete fields. Ideally, rolling back a migration should delete any data that was created with it.

The recommended way to maintain relationships among entities in a migration project is to have multiple migrations. Then, you use the `migration_lookup` plugin to relate them. Throughout the series, several examples have been presented. For example, this article shows how to do taxonomy term migrations.

What did you learn in today’s blog post? Did you know how to configure these plugins for entities that do not have bundles? Did you know that reverting a migration does not delete entities created by the `entity_generate` plugin? Did you know you can assign fields in the generated entity? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: How to debug Drupal migrations - Part 1

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Agaric is excited to announce online training on Drupal migrations and upgrades. In July 2020, we will offer three trainings: Drupal 8/9 content migrations, Upgrading to Drupal 8/9 using the Migrate API, and Getting started with Drupal 9.

We have been providing training for years at Drupal events and privately for clients. At DrupalCon Seattle 2019, our migration training was sold out with 40+ attendees and received very positive feedback. We were scheduled to present two trainings at DrupalCon Minneapolis 2020: one on Drupal migrations and the other on Drupal upgrades. When the conference pivoted to an online event, all trainings were cancelled. To fill the void, we are moving the full training experience online for individuals and organizations who want to learn how to plan and execute successful Drupal migration/upgrade projects.

Up to date with Drupal 9

Drupal is always evolving and the Migrate API is no exception. New features and improvements are added all the time. We regularly update our curriculum to cover the latest changes in the API. This time, both trainings will use Drupal 9 for all the examples! If you are still using Drupal 8, don't worry as the example code is compatible with both major versions of Drupal. We will also cover the differences between Drupal 8 and 9.

Drupal 8/9 content migrations

In this training you will learn to move content into Drupal 8 and 9 using the Migrate API. An overview of the Extract-Transform-Load (ETL) pattern that migrate implements will be presented. Source, process, and destination plugins will be explained to show how each affects the migration process. By the end of the workshop, you will have a better understanding on how the migrate ecosystem works and the thought process required to plan and perform migrations. All examples will use YAML files to configure migrations. No PHP coding required.

Date: Tuesday, July 21, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $500 USD

Click here to register

Upgrading to Drupal 8/9 using the Migrate API

In this training you will learn to use the Migrate API to upgrade your Drupal 6/7 site to Drupal 8/9. You will practice different migration strategies, accommodate changes in site architecture, get tips on troubleshooting issues, and much more. After the training, you will know how to plan and execute successful upgrade projects.

Date: Thursday, July 23, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $500 USD

Click here to register

Getting started with Drupal 9

We are also offering a training for people who want to get a solid foundation in Drupal site building. Basic concepts will be explained and put into practice through various exercises. The objective is that someone, who might not even know about Drupal, can understand the different concepts and building blocks to create a website. A simple, fully functional website will be built over the course of the day-long class.

Date: Monday, July 13, 2020
Time: 9 AM – 5 PM Eastern time
Cost: $250 USD

Click here to register

Discounts and scholarships available

Anyone is eligible for a 15% discount on their second training. Additionally, if you are a member of an under-represented community who cannot afford the full price of the training, we have larger discounts and full scholarship available. Ask Agaric to learn more about them.

Customized training available

We also offer customized training for you or your team's specific needs. Site building, module development, theming, and data migration are some of the topics we cover. Check out our training page or ask Agaric for more details. Custom training can be delivered online or on-site in English or Spanish.

Meet your lead trainer

Mauricio Dinarte is a frequent speaker and trainer at conferences around the world. He is passionate about Drupal, teaching, and traveling. Over the last few years, he has presented 30+ sessions and full-day trainings at 20+ DrupalCamps and DrupalCons over America and Europe. In August 2019, he wrote an article every day to share his expertise on Drupal migrations.

We look forward to seeing you online in July at any or all of these trainings!

Louis has been a Linux user since his childhood, although there was a period when he did not want to be free because it was too tricky to get PC games set up in the liberated zone.

As an adult, after deciding on a skill that could pay the rent and leave a little extra Louis bit into Jennifer Robbins' Learning Web Design: A Beginner's Guide to HTML, CSS, JavaScript, and Web Graphics and Marijn Haverbeke's Eloquent Javascript. When he was starting off Louis loved spending non shelf-stocking, fruit cutting, floor mopping, hours solving (or attempting to solve) code challenges. He would spend hours wrestling with problems which he should have given up on much earlier and just learned from the solution.

Professionally, Louis got started working with NOVA Web Development setting up LibreOrganize sites an association management system built with Django. Now with Agaric Louis works developing and configuring Drutopia sites.

Louis is thankful and excited to be the newest member of Agaric. Louis likes worker-coops and exploring the role they play in transitioning to the society of the emancipated worker.

The Challenge

MASS Design’s brand identity needed an update to match their new architecture work and expanding impact, with a website to match. They wanted a suite of tools that their staff could employ to flexibly assert their updated identity and tell stories in ways that could keep pace with their evolution.

At the same time, they knew that part of design is accessibility. With a large audience across Africa, it was critical that the design not compromise performance, ensuring visitors on low bandwidth devices could read their stories just as easily as their US counterparts.

The Approach

Flexible Content Creation with Paragraphs

Todd Linkner took MASS Design’s new visual identity and translated it into components that could be assembled into myriad formats, giving wide creative control to content editors. We took on the site built the site with MASS's regular involvement and approval.

To achieve this component-driven design, Todd, as a remarkably Drupal-savvy designer, had the Paragraph module in mind, and it is what we chose to implement the functionality (this was before the two-year period where every third talk at Drupal camps was about the Paragraphs module). Rather than building content types with fixed fields always in the same order, Paragraphs allowed us to define a variety of media formats (carousel, image grid, text, etc.) that could be added and rearranged at will.

Screenshot of content entry form using Paragraphs.

Editors can customize the arrangement of different content elements with the Paragraphs module.

Site Performance is Design Justice

Previously, MASS Design’s website loaded multi-megabyte images on the front page, loading slowly for all visitors, particularly those with limited bandwidth. (It also cost hundreds of dollars a month in bandwidth from their cloud services provider.)

Performance is part of design and when we prioritize our lower resourced audiences we stay true to our values of access and inclusion. We did this for MASS Design by keeping performance at the front of conversations from the beginning.

Drupal is a powerful CMS which, out of the box, will resize images and do internal caching. We ensured the new MASS site would load quickly with strategic configuration and the use of key third party tools.

We started with aggressive, yet sensible caching. We turned to a content delivery network (CDN) with locations close to Kigali, Rwanda to make this happen. (Closest we could get was, Mombusa, Kenya). Using WebPageTest.org's API, we scripted performance tests to regularly, repeatedly see page load speeds of the site from Johannesburg, South Africa, the closest (not very close) offered at the time.

Two entry points to the site were configured. One for the public utilizing CDN caching, and a separate entry point for editors with a sub domain. This structure kept the CDN working properly. Features relying on cookies, in particular logging in as an authenticated user, don’t work with a low-cost CDN.

The Results

Since updating their website, MASS is now telling compelling stories of their work, resulting in 84% more page views, more pages per session, and more returning users. Site traffic from Rwanda is up ten times thanks in part to the improved performance of the site.

Since the initial site upgrade, we have also added on-site donation forms, a email sign up feature for their key resource documents and expanded their Paragraphs-driven storytelling toolset.

Throughout the series, we explored many migration topics. We started with an overview of the ETL process and workflows for managing migrations. Then, we presented example migrations for different entities: nodes, files, images, taxonomy terms, users, and paragraphs. Next, we shifted focus to migrations from different sources: CSV, JSON, XML, Google Sheet, Microsoft Excel, and LibreOffice Calc files. Later, we explored how to manage migrations as configuration, use groups to share configuration, and execute migrations from the user interface. Finally, we gave recommendations and provided tools for debugging migrations from the command line and the user interface. Although we covered a lot of ground, we only scratched the surface. The Migrate API is so flexible that its use cases are virtually endless. To wrap up the series, we present an introduction to a very popular topic: Drupal upgrades. Let’s get started.

Note: In this article, when we talk about Drupal 7, the same applies to Drupal 6.

What is a Drupal upgrade?

The information we presented in the series is generic enough that it applies to many types of Drupal migrations. There is one particular use case that stands out from the rest: Drupal upgrades. An upgrade is the process of taking your existing Drupal site and copy its configuration and content over to a new major version of Drupal. For example, going from Drupal 6 or 7 to Drupal 8. The following is an oversimplification of the workflow to perform the upgrade process:

Install a fresh Drupal 8 site.
Add credentials so that the new site can connect to Drupal 7’s database.
Use the Migrate API to generate migration definition files. They will copy over Drupal 7’s configuration and content. This step is only about generating the YAML files.
Execute those migrations to bring the configuration and content over to Drupal 8.

Preparing your migration

Any migration project requires a good plan of action, but this is particularly important for Drupal upgrades. You need to have a general sense of how the upgrade process works, what assumptions are made by the system, and what limitations exist. Read this article for more details on how to prepare a site for upgrading it to Drupal 8. Some highlights include:

Both sites need to be in the latest stable version of their corresponding branch. That means the latest release of Drupal 7 and 8 at the time of performing the upgrade process. This also applies to any contributed module.
Do not do any configuration of the Drupal 8 site until the upgrade process is completed. Any configuration you make will be overridden, and there is no need for it anyways. Part of the process includes recreating the old site’s configuration: content types, fields, taxonomy vocabularies, etc.
Do not create content on the Drupal 8 site until the upgrade process is completed. The upgrade process will keep the unique identifiers from the source site: `nid`, `uid`, `tid`, `fid`, etc. If you were to create content, the references among entities could be broken when the upgrade process overrides the unique identifiers. To prevent data loss, wait until the old site's content has been migrated to start adding content to the new site.
For the system to detect a module’s configuration to be upgraded automatically, it has to be enabled on both sites. This applies to contributed modules in Drupal 7 (e.g., link) that were moved to core in Drupal 8. Also to Drupal 7 modules (e.g. address field) that were superseded by a different one in Drupal 8 (e.g. address). In any of those cases, as long as the modules are enabled on both ends, their configuration and content will be migrated. This assumes that the Drupal 8 counterpart offers an automatic upgrade path.
Some modules do not offer automatic upgrade paths. The primary example is the Views module. This means that any view created in Drupal 7 needs to be manually recreated in Drupal 8.
The upgrade procedure is all about moving data, not logic in custom code. If you have custom modules, the custom code needs to be ported separately. If those modules store data in Drupal’s database, you can use the Migrate API to move it over to the new site.
Similarly, you will have to recreate the theme from scratch. Drupal 8 introduced Twig which is significantly different to the PHPTemplate engine used by Drupal 7.

Customizing your migration

Note that the creation and execution of the migration files are separate steps. Upgrading to a major version of Drupal is often a good opportunity to introduce changes to the website. For example, you might want to change the content modeling, navigation, user permissions, etc. To accomplish that, you can modify the generated migration files to account for any scenario where the new site’s configuration diverts from the old one. And only when you are done with the customizations, you execute the migrations. Examples of things that could change include:

Combining or breaking apart content types.
Moving data about people from node entities to user entities, or vice versa.
Renaming content types, fields, taxonomy vocabularies and terms, etc.
Changing field types. For example, going from Address Field module in Drupal 7 to Address module in Drupal 8.
Merging multiple taxonomy vocabularies into one.
Changing how your content is structured. For example, going from a monolithic body field to paragraph entities.
Changing how your multimedia files are stored. For example, going from image fields to media entities.

Performing the upgrade

There are two options to perform the upgrade. In both cases, the process is initiated from the Drupal 8 site. One way is using the Migrate Drupal UI core module to perform the upgrade from the browser’s user interface. When the module is enabled, go to `/upgrade` and provide the database credentials of the Drupal 7 site. Based on the installed modules on both sites, the system will give you a report of what can be automatically upgraded. Consider the limitations explained above. While the upgrade process is running, you will see a stream of messages about the operation. These messages are logged to the database so you can read them after the upgrade is completed. If your dataset is big or there are many expensive operations like password encryption, the process can take too long to complete or fail altogether.

The other way to perform the upgrade procedure is from the command line using Drush. This requires the Migrate Upgrade contributed module. When enabled, it adds Drush commands to import and rollback a full upgrade operation. You can provide database connection details of the old site via command line options. One benefit of using this approach is that you can create the migration files without running them. This lets you do customizations as explained above. When you are done, you can run the migrations following the same workflow of manually created ones.

Known issues and limitations

Depending on whether you are upgrading from Drupal 6 or 7, there is a list of known issues you need to be aware of. Read this article for more information. One area that can be tricky is multilingual support. As of this writing, the upgrade path for multilingual sites is not complete. Limited support is available via the Migrate Drupal Multilingual core module. There are many things to consider when working with multilingual migrations. For example, are you using node or field translations? Do entities have revisions? Read this article for more information.

Upgrade paths for contributed modules

The automatic upgrade procedure only supports Drupal core modules. This includes modules that were added to core in Drupal 8. For any other contributed module, it is the maintainers’ decision to include an automatic upgrade path or not. For example, the Geofield module provides an upgrade path. It is also possible that a module in Drupal 8 offers an upgrade path from a different module in Drupal 7. For example, the Address module provides an upgrade path from the Address Field module. Drupal Commerce also provides some support via the Commerce Migrate module.

Not every module offers an automated upgrade path. In such cases, you can write custom plugins which ideally are contributed back to Drupal.org ;-) Or you can use the techniques learned in the series to transform your source data into the structures expected by Drupal 8. In both cases, having a broad understanding of the Migrate API will be very useful.

Upgrade strategies

There are multiple migration strategies. You might even consider manually recreating the content if there is only a handful of data to move. Or you might decide to use the Migrate API to upgrade part of the site automatically and do a manual copy of a different portion of it. You might want to execute a fully automated upgrade procedure and manually clean up edge cases afterward. Or you might want to customize the migrations to account for those edge cases already. Michael Anello created an insightful presentation on different migration strategies. Our tips for writing migrations apply as well.

Drupal upgrades tend to be fun, challenging projects. The more you know about the Migrate API the easier it will be to complete the project. We enjoyed writing this overview of the Drupal Migrate API. We would love to work on a follow up series focused on Drupal upgrades. If you or your organization could sponsor such endeavor, please reach out to us via the site’s contact form.

What about upgrading to Drupal 9?

In March 2017, project lead Dries Buytaert announced a plan to make Drupal upgrades easier forever. This was reinforced during his keynote at DrupalCon Seattle 2019. You can watch the video recording in this link. In short, Drupal 9.0 will be the latest point release of Drupal 8 minus deprecated APIs. This has very important implications:

When Drupal 9 is released, the Migrate API should be mostly the same as Drupal 8. Therefore, anything that you learn today will be useful for Drupal 9 as well.
As long as your code does not use deprecated APIs, upgrading from Drupal 8 to Drupal 9 will be as easy as updating from Drupal 8.7 to 8.8.
Because of this, there is no need to wait for Drupal 9 to upgrade your Drupal 6 or 7 site. You can upgrade to Drupal 8 today.

Thank you!

And that concludes the #31DaysOfMigration series. For joining us in this learning experience, thank you very much! ¡Muchas gracias! Merci beaucoup! :-D We are also very grateful to Agaric.coop, Drupalize.Me, and Centarro.io for sponsoring this series.

What did you learn in today’s blog post? Did you know the upgrade process is able to copy content and configuration? Did you know that you can execute the upgrade procedure either from the user interface or the command line? Share your answers in the comments. Also, we would be grateful if you shared this blog post with others.

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

In a previous article we explained the syntax used to write Drupal migrations. When migrating into content entities, these define several properties that can be included in the process section to populate their values. For example, when importing nodes you can specify the title, publication status, creation date, etc. In the case of users, you can set the username, password, timezone, etc. Finding out which properties are available for an entity might require some Drupal development knowledge. To make the process easier, in today’s article we are presenting a reference of properties available in content entities provided by Drupal core and some contributed modules.

Example migrations mapping of content entity properties

For each entity we will present: the module that provides it, the class that defines it, and the available properties. For each property we will list its name, field type, a description, and a note if the field allows unlimited values (i.e. it has an unlimited cardinality). The list of properties available for a content entity depend on many factors. For example, if the entity is revisionable (e.g. revision_default), translatable (e.g. langcode), or both (e.g. revision_translation_affected). The modules that are enabled on the site can also affect the available properties. For instance, if the “Workspaces” module is installed, it will add a workspace property to many content entities. This reference assumes that Drupal was installed using the standard installation profile and all modules that provide content entities are enabled.

It is worth noting that entity properties are divided in two categories: base field definitions and field storage configurations. Base field configurations will always be available for the entity. On the other hand, the presence of field storage configurations will depend on various factors. For one, they can only be added to fieldable entities. Attaching the fields to the entity can be done manually by the user, by a module, or by an installation profile. Again, this reference assumes that Drupal was installed using the standard installation profile. Among other things, it adds a user_picture image field to the user entity and body, comment, field_image, and field_tags fields to the node entity. For entities that can have multiple bundles, not all properties provided by the field storage configurations will be available in all bundles. For example, with the standard installation profile all content types will have a body field associated with it, but only the article content type has the field_image, and field_tags fields. If subfields are available for the field type, you can migrate into them.