Blog

Throughout the series we have shown many examples. I do not recall any of them working on the first try. When working on Drupal migrations, it is often the case that things do not work right away. Today’s article is the first of a two part series on debugging Drupal migrations. We start giving some recommendations of things to do before diving deep into debugging. Then, we are going to talk about migrate messages and presented the log process plugin. Let’s get started.

Minimizing the surface for errors

The Migrate API is a very powerful ETL framework that interacts with many systems provided by Drupal core and contributed modules. This adds layers of abstraction that can make the debugging process more complicated compared to other systems. For instance, if something fails with a remote JSON migration, the error might be produced in the Migrate API, the Entity API, the Migrate Plus module, the Migrate Tools module, or even the Guzzle HTTP Client library that fetches the file. For a more concrete example, while working on a recent article, I stumbled upon an issue that involved three modules. The problem was that when trying to rollback a CSV migration from the user interface an exception will be thrown making the operation fail. This is related to an issue in the core Migrate API that manifests itself when rollback operations are initiated from the interface provided by Migrate Plus. Then, the issue causes a condition in the Migrate Source CSV plugin that fails and the exception is thrown.

In general, you should aim to minimize the surface for errors. One way to do this by starting the migration with the minimum possible set up. For example, if you are going to migrate nodes, start by configuring the source plugin, one field (the title), and the destination. When that works, keep migrating one field at a time. If the field has multiple subfields, you can even migrate one subfield at a time. Commit every progress to version control so you can go back to a working state if things go wrong. Read this article for more recommendations on writing migrations.

What to check first?

Debugging is a process that might involve many steps. There are a few things that you should check before diving too deep into trying to find the root of the problem. Let’s begin with making sure that changes to your migrations are properly detected by the system. One common question I see people ask is where to place the migration definition files. Should they go in the migrations or config/install directory of your custom module? The answer to this is whether you want to manage your migrations as code or configuration. Your choice will determine the workflow to follow for changes in the migration files to take effect. Migrations managed in code go in the migrations directory and require rebuilding caches for changes to take effect. On the other hand, migrations managed in configuration are placed in the config/install directory and require configuration synchronization for changes to take effect. So, make sure to follow the right workflow.

After verifying that your changes are being applied, the next thing to do is verify that the modules that provide your plugins are enabled and the plugins themselves are properly configured. Look for typos in the configuration options. Always refer to the official documentation to know which options are available and find the proper spelling of them. Other places to look at is the code for the plugin definition or articles like the ones in this series documenting how to use them. Things to keep in mind include proper indentation of the configuration options. An extra whitespace or a wrong indentation level can break the migration. You can either get a fatal error or the migration can fail silently without producing the expected results. Something else to be mindful is the version of the modules you are using because the configuration options might change per version. For example, the newly released 8.x-3.x branch of Migrate Source CSV changed various configuration options as described in this change record. And the 8.x-5.x branch of Migrate Plus changed some configurations for plugin related with DOM manipulation as described in this change record. Keeping an eye on the issue queue and change records for the different modules you use is always a good idea.

If the problem persists, look for reports of similar problems in the issue queue. Make sure to include closed issues as well in case your problem has been fixed or documented already. Remember that a problem in a module can affect a different module. Keeping an eye on the issue queue and change records for all the modules you use is always a good idea. Another place ask questions is the #migrate channel in Drupal slack. The support that is offered there is fantastic.

Migration messages

If nothing else has worked, it is time to investigate what is going wrong. In case the migration outputs an error or a stacktrace to the terminal, you can use that to search in the code base where the problem might originate. But if there is no output or if the output is not useful, the next thing to do is check the migration messages.

The Migrate API allows plugins to log messages to the database in case an error occurs. Not every plugin leverages this functionality, but it is always worth checking if a plugin in your migration wrote messages that could give you a hint of what went wrong. Some plugins like skip_on_empty and skip_row_if_not_set even expose a configuration option to specify messages to log. To check the migration messages use the following Drush command: drush migrate:messages [migration_id]. If you are managing migrations as configuration, the interface provided by Migrate Plus also exposes them.

Messages are logged separately per migration, even if you run multiple migrations at once. This could happen if you execute dependencies or use groups or tags. In those cases, errors might be produced in more than one migration. You will have to look at the messages for each of them individually.

Let’s consider the following example. In the source there is a field called src_decimal_number with values like 3.1415, 2.7182, and 1.4142. It is needed to separate the number into two components: the integer part (3) and the decimal part (1415). For this, we are going to use the extract process plugin. Errors will be purposely introduced to demonstrate the workflow to check messages and update migrations. The following example shows the process plugin configuration and the output produced by trying to import the migration:

# Source values: 3.1415, 2.7182, and 1.4142

psf_number_components:
  plugin: explode
  source: src_decimal_number

$ drush mim ud_migrations_debug
[notice] Processed 3 items (0 created, 0 updated, 3 failed, 0 ignored) - done with 'ud_migrations_debug'

In MigrateToolsCommands.php line 811:
ud_migrations_debug Migration - 3 failed.

The error produced in the console does not say much. Let’s see if any messages were logged using: drush migrate:messages ud_migrations_debug. In the previous example, the messages will look like this:

 ------------------- ------- --------------------
  Source IDs Hash    Level   Message
 ------------------- ------- --------------------
  7ad742e...732e755   1       delimiter is empty
  2d3ec2b...5e53703   1       delimiter is empty
  12a042f...1432a5f   1       delimiter is empty
 ------------------------------------------------

In this case, the migration messages are good enough to let us know what is wrong. The required delimiter configuration option was not set. When an error occurs, usually you need to perform at least three steps:

• Rollback the migration. This will also clear the messages.
• Make changes to definition file and make they are applied. This will depend on whether you are managing the migrations as code or configuration.
• Import the migration again.

Let’s say we performed these steps, but we got an error again. The following snippet shows the updated plugin configuration and the messages that were logged:

psf_number_components:
  plugin: explode
  source: src_decimal_number
  delimiter: '.'

 ------------------- ------- ------------------------------------
  Source IDs Hash    Level   Message
 ------------------- ------- ------------------------------------
  7ad742e...732e755   1       3.1415000000000002 is not a string
  2d3ec2b...5e53703   1       2.7181999999999999 is not a string
  12a042f...1432a5f   1       1.4141999999999999 is not a string
 ----------------------------------------------------------------

The new error occurs because the explode operation works on strings, but we are providing numbers. One way to fix this is to update the source to add quotes around the number so it is treated as a string. This is of course not ideal and many times not even possible. A better way to make it work is setting the strict option to false in the plugin configuration. This will make sure to cast the input value to a string before applying the explode operation. This demonstrates the importance of reading the plugin documentation to know which options are at your disposal. Of course, you can also have a look at the plugin code to see how it works.

Note: Sometimes the error produces an non-recoverable condition. The migration can be left in a status of "Importing" or "Reverting". Refer to this article to learn how to fix this condition.

The log process plugin

In the example, adding the extra configuration option will make the import operation finish without errors. But, how can you be sure the expected values are being produced? Not getting an error does not necessarily mean that the migration works as expected. It is possible that the transformations being applied do not yield the values we think or the format that Drupal expects. This is particularly true if you have complex process plugin chains. As a reminder, we want to separate a decimal number from the source like 3.1415 into its components: 3 and 1415.

The log process plugin can be used for checking the outcome of plugin transformations. This plugin offered by the core Migrate API does two things. First, it logs the value it receives to the messages table. Second, the value is returned unchanged so that it can be used in process chains. The following snippets show how to use the log plugin and what is stored in the messages table:

psf_number_components:
  - plugin: explode
    source: src_decimal_number
    delimiter: '.'
    strict: false
  - plugin: log

 ------------------- ------- --------
  Source IDs Hash    Level   Message
 ------------------- ------- --------
  7ad742e...732e755   1       3
  7ad742e...732e755   1       1415
  2d3ec2b...5e53703   1       2
  2d3ec2b...5e53703   1       7182
  12a042f...1432a5f   1       1
  2d3ec2b...5e53703   1       4142
 ------------------------------------

Because the explode plugin produces an array, each of the elements is logged individually. And sure enough, in the output you can see the numbers being separated as expected.

The log plugin can be used to verify that source values are being read properly and process plugin chains produce the expected results. Use it as part of your debugging strategy, but make sure to remove it when done with the verifications. It makes the migration to run slower because it has to write to the database. The overhead is not needed once you verify things are working as expected.

In the next article, we are going to cover the Migrate Devel module, the debug process plugin, recommendations for using a proper debugger like XDebug, and the migrate:fields-source Drush command.

What did you learn in today’s blog post? What workflow do you follow to debug a migration issue? Have you ever used the log process plugin for debugging purposes? If so, how did it help to solve the issue? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: How to debug Drupal migrations - Part 2

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Starting at $1,250 for five hours a month, Agaric will maintain your current site with all security updates, provide support, make continuous improvements, and be available for our full range of consulting capabilities.

We're delighted you'll be participating in an Agaric migration training.  Please let us know if you'd like to learn about future training opportunities:

There are plenty of ways—all right, too many ways—to create links when using modern Drupal (Drupal 8 and above, so including so far Drupal 9, Drupal 10, and Drupal 11).  We will share some of those ways in this regularly-updated post.  (Latest update: 2025 March 21)

The easiest way to create internal links is using Link::createFromRoute

And it is used like this:

use Drupal\Core\Link;
  $link = Link::createFromRoute(t('This is a link'), 'entity.node.canonical', ['node' => 1]);

Using the Url object gives you more flexibility to create links, for instance, we can do the same as Link::createFromRoute method using the Url object like this:

  use Drupal\Core\Link;
  use Drupal\Core\Url;
  $link = Link::fromTextAndUrl(t('This is a link'), Url::fromRoute('entity.node.canonical', ['node' => 1]));

And actually Link::fromTextAndUrl is what Drupal recommends instead of using the deprecated l() method.  Note that as before, the text portion of the link should already be translated, so wrap it in t() or, for object-oriented code, $this->t().  Passing the Url object to the link object gives you great flexibility to create links, here are some examples:

Internal links which have no routes:

$link = Link::fromTextAndUrl(t('This is a link'), Url::fromUri('base:robots.txt'));

External links:

$link = Link::fromTextAndUrl(t('This is a link'), Url::fromUri('http://www.google.com'));

links with only the fragment (without url) :

$link = Link::fromTextAndUrl(t('This is a link'), Url::fromUri('internal:#fragment'));

Using the data provided by a user:

$link = Link::fromTextAndUrl(t('This is a link'), Url::fromUserInput('/node/1');

The param passed to fromUserInput must start with /,#,? or it will throw an exception.

Linking entities.

$link = Link::fromTextAndUrl(t('This is a link'), Url::fromUri('entity:node/1'));

Entities are a special case, and there are more ways to link them:

$node = Node::load(1);
$link = $node->toLink();
$link->setText(t('This is a link'));

And even using the route:

$link = Link::fromTextAndUrl(t('This is a link'), Url::fromRoute('entity.node.canonical', ['node' => 1]));

Drupal usually expects a render array if you are going to print the link, so the Link object has a method for that:

$link->toRenderable();

which will return an array.

To get the absolute URL (that is, the one that likely starts with "https" and your domain name) for a piece of content (node) or other entity, you can do:

$absolute_url = $node->toUrl('canonical', ['absolute' => TRUE]);

Linking files.

Files are entities also in Drupal but a special kind.  You usually do not want to link to the entity but directly to the uploaded file.  To do that, do not use $file->getFileUri() but rather instead use createFileUrl() which will directly give you a complete string version of the URI:

$url = $file->createFileUrl();

This URL can then be used in making a link with regular HTML or the External links approach covered above.

Credit to godotislate in Drupal Slack support channel for this tip.

Final tips:

Searching a route using Drupal Console

The easiest way to find the route of a specific path is using Drupal Console, with the following command.

$ drupal router:debug | grep -i "\/node"

That will return something like:

entity.node.canonical                                 /node/{node}
 entity.node.delete_form                               /node/{node}/delete
 entity.node.edit_form                                 /node/{node}/edit
 entity.node.preview                                   /node/preview/{node_preview}/{view_mode_id}
 entity.node.revision                                  /node/{node}/revisions/{node_revision}/view
 entity.node.version_history                           /node/{node}/revisions
 node.add                                              /node/add/{node_type}
 node.add_page                                         /node/add
 node.multiple_delete_confirm                          /admin/content/node/delete
 node.revision_delete_confirm                          /node/{node}/revisions/{node_revision}/delete
 node.revision_revert_confirm                          /node/{node}/revisions/{node_revision}/revert
 node.revision_revert_translation_confirm              /node/{node}/revisions/{node_revision}/revert/{langcode}
 search.help_node_search                               /search/node/help
 search.view_node_search                               /search/node
 view.frontpage.page_1                                 /node

Listing all the possible routes with that word, we can choose one and do:

drupal router:debug entity.node.canonical

And that will display more information about a specific route:

 Route             entity.node.canonical
 Path              /node/{node}
 Defaults
  _controller      \Drupal\node\Controller\NodeViewController::view
  _title_callback  \Drupal\node\Controller\NodeViewController::title
 Requirements
  node             \d+
  _entity_access   node.view
  _method          GET|POST
 Options
  compiler_class   \Drupal\Core\Routing\RouteCompiler
  parameters       node:
                     type: 'entity:node'
                     converter: paramconverter.entity

  _route_filters   method_filter
                   content_type_header_matcher

  _route_enhancers route_enhancer.param_conversion

  _access_checks   access_check.entity

So in this way we can search the route without the needing to search in all the *.routing.yml files and in this example the route is entity.node.canonical and the param expected is node.

Print links directly within a twig template

It is also possible to print links directly on the twig template with the following syntax:

<a href="{{url('entity.node.canonical', {'node': node.id( ) }}"> {{ 'This is a link'|t }} </a>

Add links inside a t() method.

If you want to add a link inside the t() you can do what Drupal suggests on Dynamic or static links and HTML in translatable strings:

use Drupal\Core\Url;
$url = Url::fromRoute('entity.node.canonical', ['node' => 1]);
$this->t('You can click this <a href="@link">Link</a>', ['@link' => $url->toString()]);

Find It makes it easier for caretakers to find opportunities for their children online so that they can stay focused on taking care of themselves and their children.

Valoramos aprender cosas nuevas y compartir nuestros conocimientos, por lo que nos tomamos el tiempo cada semana para compartir lo que hemos descubierto o descubierto. Nos damos cuenta de que esto puede ser de interés para personas ajenas al Agaric, especialmente para aquellas personas que trabajan solas o están en organizaciones que no fomentan el intercambio de habilidades. Por lo tanto, estamos invitando a socios, estudiantes, colegas, y usted, a participar en la observación o presentación de presentaciones cortas.

También discutimos nuestros flujos de trabajo y modelos de negocio como cooperativas. Esta cita es de nuestros amigos en Argentina - Fiqus.coop:

"El propósito de ponerse en contacto con otras cooperativas en el mundo no es solo una, y se podría decir que son varias al mismo tiempo y con la misma importancia.

En primer lugar, creemos que la mejor manera de fortalecer el movimiento cooperativo es conectarse, compartir experiencias, información ... en otras palabras, cooperar entre sí. Esa es la esencia de la forma organizativa que adoptamos para nuestras empresas ".

Suscríbase a la lista de correo para recibir invitaciones al chat y comparta este enlace con un amigo: Mostrar y Contar.

We live amidst a Digital Commons - technology that is built with the principles of freedom and transparency baked into its code and design. It's maintained out in the open by the free software community. This commons is invisible to many of us, but the closer we are to the technology we use, the more that it comes into focus. At Agaric we are knee deep in this Digital Commons. Our name Agaric is a nod to the mycelial nature of the open web. We help create, maintain, and promote free and open-source software that make up this commons.

2017 was an amazing year for me. I met so many people, attended lots of events, presented a good amount of sessions and trainings, and traveled like never before. I got to interact with the Drupal community in many ways. And I learned so much, both technical and life lessons. Let me share a summary of my year.

To make the text compact I will use some abbreviations in the text. DC for DrupalCamp and GTD for Global Training Days. Meetups refers to local user groups meetings.

Events per month

• January: No event.
• February: Boston meetup, New Jersey, Florida DC, and Northern Lights DC.
• March: DC London, GTD Managua, NERD Summit, DevDays Seville, and MidCamp.
• April: Boston meetup and DrupalCon Baltimore.
• May: Managua meetup and WordCamp Managua.
• June: Texas DC, Boston meetup, GTD Managua, ECWD, and Twin Cities DC.
• July: No event.
• August: DC Costa Rica and Managua meetup.
• September: Synergy Camp, DC Antwerp, GTD Frankfurt, GTD Managua, and DrupalCon Vienna.
• October: Madrid meetup, London meetup, and BADCamp.
• November: Managua meetup and Lakes and Volcanoes DC.
• December: GTD Leon, GTD Managua, and Managua meetup.

This translates to:

• 18 conferences
• 9 meetups
• 12 DrupalCamps
• 2 DrupalCons
• 1 Drupal DevDays
• 12 sessions presented
• 4 trainings presented
• 6 GTD workshop presented

Highlights this year

• Made it into MAINTAINERS.txt as part of the Core mentoring coordinators team.
• Served as part of the Global Training Days community working group.
• Presented a session at DrupalCon.
• Was invited to be a DrupalCon track chair.
• People I mentored at DrupalCon got their patches committed. Thanks to Jess (xjm), Ted Bowman, and other contributors for following up.

Thank you all

Thanks to my colleagues at Agaric for their support during my Drupal tour.

Thanks to all the event organizers who gave me the opportunity to speak at their events. And thanks to all the people who gave feedback of my sessions, trainings, and workshops.

Thanks to Baddý Breidert, Hilmar Hallbjörnsson, and all the team who organized Northern Lights DrupalCamp. They went above and beyond to pull out an event against unfavorable conditions. The night before the sessions fell a record breaking amount of snow. The organizers went to pick people up from around Reykjavik and accommodated the schedule nicely. I was trapped in a bus stops, with snow above my knees, when Hilmar and Christoph Breidert picked me up saved me and took me to the venue. On top of everything, the organizers took everyone on free tours around beautiful Iceland. It was amazing!

Thanks to all the people and organizations who set up different events where I could present during my European tour. Cristina Chumillas (Barcelona), Baddý and Christoph of 1xINTERNET (Frankfurt), Valery Lourie and the SynergyCamp team (Sofia), Sven Decabooter and the DC Antwerp team (Antwerp), Ignacio Sánchez (Madrid), and Chandeep Khosa and Gabriele (gambry) (London).

Thanks to those who hosted me during my European tour. Baddý and Christoph of 1xINTERNET in Frankfurt, my colleague Stefan Freudenberg in Hamburg, and Rodrigo Aguilera in Barcelona. They opened the doors to their homes and their families during my stay. I will always be grateful.

Thanks to Anna Mykhailova for co-presenting a training with me at BADCamp.

Thanks to the awesome organizing team of Lakes and Volcanoes DC. In particular to Lucas Hedding who has helped the Nicaraguan Drupal community in countless ways.

Thanks to the Boston Drupal user group for making me feel at home every time I visit.

Thanks to ALL the people I talked to during the 2017 Drupal tour. The Drupal community is super friendly and welcoming. It is so great to be part of it.

Trivia

Zsófi Major sent a t-shirt to Christian López. In its way from DC London to Drupal DevDays Seville, the t-shirt stopped by 9 airports.

V Spagnolo sent a present to Lucas Hedding. In its way from DrupalCon Vienna to Nicaragua, the parcel stopped by 10 airports.

The GTD in Frankfurt was super interesting. I installed Drupal in German. Not knowing the language, I had no idea how to pronounce the text on the screen. The audience was very kind in helping me. By the end of the workshop I was pronouncing some words almost right. ;)

What's next for 2018?

As my fellow drupaler Eduardo (enzo) García says in his end of year recap, travelling takes a lot of energy. I have plans to attend some events in 2018, but not as much as I did in 2017. While staying at home, I will be able to accomplish things that would be difficult otherwise.

I will kickstart my Drupal learning project: understanddrupal.com My goal is to teach Drupal in multiple languages and multiple formats. To start I will produce content in English, Spanish, and French in text and video formats.

I will learn new languages. On the human side, I will start with French and I might give German a try. On the computer side, I will start with JavaScript and then move to Python. At Agaric we serve clients in multiple countries and with different application stacks. Learning new human and computer languages will help me serve better the clients and projects I participate in.

I want to learn new things. Software testing in general and creating applications in React are my first targets. Expect blog posts about my learnings next year.

I want to participate more actively in the free software communities in my country. Drupal, WordPress, JavaScript, and others. I would like to learn from them and share what I have learned in my travels.

Thanks for a fantastic 2017! See you in 2018! :D

Resources can be either a file-based resource such as a PDF or a link-based resource such as a url to a website. Resources can also be categorized by resource type and can also have an associated tag. An associated image such as the cover of a PDF resource or a logo or screenshot from a website link can make the resource more visually attractive in various displays.

The resource content type lets you post a PDF (or various other text based formats) so that site visitors can easily view and download. The body field lets you provide some information about the resource to help site visitors determine the usefulness of the resource before viewing.

Helpful tips include ensuring that your PDF has a user-friendly machine name (and/or using the file description field for the title users will see) and adding a cover image for example, in the image field, to provide a visual clue to the resource.

For link-based resources, you'll need to paste in the full website URL, such as https://drutopia.org. The Link text could read "Visit the Drutopia website."

Agaric builds tools for medical and scientific communities to advance their work, enhance collaboration, and improve outcomes.

A program or application is free software when it is intentionally licensed with our freedom in mind. Specifically, four freedoms (numbered using the zero-based numbering in programming languages):

• Freedom 0 - The freedom to run the program as you wish, for any purpose.
• Freedom 1 - The freedom to study how the program works, and change it so it does your computing as you wish.
• Freedom 2 - The freedom to redistribute copies so you can help others.
• Freedom 3 - The freedom to distribute copies of your modified versions to others.

From Linux to Firefox to Drupal, free software is a tremendous force in our world. By building free software, we help building a software commons we can all trust and benefit from.

Why Free?

A Note On Other Words: Libre Software, Open Source

There are similar terms used to describe similar software. Libre software is a synonym for free software. By using the word libre, we remove the ambiguity inherent in free, which can mean free as in no money or free as in liberty. We appreciate this term and use it interchangeably with free software.

Open source is very similar to free software. It emerged, like libre software, as a way to remove ambiguity. However, in doing so it loses the ethical and political power that comes with free/libre software. So while we respect and use the term open source, we prefer free software because of it emphasizes the value of freedom.

Further Reading

The following are excellent pieces for further information on the meaning of free software and some of the differences and debates between the various terms.

• Free Software Wikipedia article
• How I coined the term 'open source' by Christine Peterson
• Why Open Source misses the point of Free Software by Richard Stallman
• When Free Software Isn't (Practically) Superior by Benjamin Mako Hill

As starving authors we at Agaric don't have a lot of cash to burn right now, but we've thrown $25 in the project to make it possible to subscribe to drupal.org issues without commenting. (On top of whatever we donated when this request for funding went out a year and a half ago).

To learn more about the cooperative movement, visit the US Federation of Worker Owner Cooperatives.