Respecting your privacy and being responsible with the data we collect from you is of the utmost importance to us.  We will not use or share your information with anyone except as described in this privacy policy.

Information collection and use

If you choose to leave a comment or a private message through our contact form, we may require you to provide us with certain personally identifiable information, including your name and email address. The information that we collect will be used to contact or identify you.

Log data

We want to inform you that whenever you visit our Service, we collect information that your browser sends to us that is called Log Data. This Log Data may include information such as your computer's Internet Protocol (“IP”) address, browser version, pages of our Service that you visit, the time and date of your visit, the time spent on those pages, and other statistics.

Cookies

We do not use cookies for visitors to our site. (Cookies are files with small amount of data that is commonly used as an anonymous unique identifier.)

Service providers

We employ a third-party company, Google, to assist us in analyzing how our website is used. Only anonymized data is collected.

Hotjar assists its us in providing our end users with a better experience and service as well as assist us in diagnosing technical problems and analyzing user trends. Most importantly, through Hotjar’s services, the functionality of the site can be improved, making them more user-friendly, more valuable, and simpler to use for the end users.

You may opt-out from having Hotjar collect your information when visiting a Hotjar Enabled Site at any time by visiting our Opt-out page and clicking ‘Disable Hotjar’ or enabling Do Not Track (DNT) in your browser.

Changes to this privacy policy

We may update our Privacy Policy from time to time. Thus, we advise you to review this page periodically for any changes. We will notify you of any changes by posting the new Privacy Policy on this page. These changes are effective immediately, after they are posted on this page.

Contact us

If you have any questions or suggestions about our privacy policy, do not hesitate to contact us.

IndieWebCamp is a movement dedicated to growing the independent web, or IndieWeb: a people-focused alternative to the corporate web. The movement is called IndieWebCamp because it is built in large part over an on-going series of two-day camps. At these camps and online, the community emphasizes principles over particular projects or software— any web site can be a part of the IndieWeb. Here's how to take a first step into the IndieWeb with Drupal.

All the benefits from brewing your own website touted by IndieWebCamp are indeed great. Your content belongs unambiguously and in real and practical ways to you; at the least it won't disappear when yet another company shuts down or is acquired and tells its fans "thanks for supporting us on our incredible journey". Above all, you are in control of what you post, how it is presented, and how others can find it. All this may be familiar to web developers as the concept of "having a web site."

If that was all there was to the movement, IndieWebCamp would be a call to do it like we did it in 1998. Instead, IndieWebCamp goes the next step by recognizing that people use the corporate web of Facebook, Twitter, Tumblr (Yahoo), Blogger (Google), Flickr (Yahoo), LiveJournal (SUP Media), YouTube (Google), and others in large because of the experience they provide for interactions between people. IndieWebCamp takes on the challenge of designing user experiences and formats and protocols which make following, sharing, and responding just as easy on the independent web of personal sites and blogs.

To this end of making social interaction native to independent sites, IndieWeb principles and practice teach a couple of new tricks to old web sites. One of these tricks, which we will not cover today, provides a bridge from independent sites to the monolithic services most people use today by implementing the approach of Publish (on your) Own Site, Syndicate Elsewhere (POSSE). This means that posting on your own site provides an advantage in that your posts and status messages can go to all services rather than get stuck inside only one.

The first steps of getting on the IndieWeb (after joining the #indiewebcamp IRC channel) are very familiar to web developers: Put up a web site. We were all set with a domain name for Agaric and with web hosting, so we could skip right to setting up our home page and signing in.

All you need to do for this step is to add rel=me to a link to an online profile that links back to your home page, identifying yourself in both places as you. In our case, we added the rel="me" attribute to a link to our Twitter profile. Twitter puts rel="me" on the web site link on their profiles. We did have to make sure we linked to Twitter with https not http so that the redirect didn't interfere with verifying our web sign in capability with IndieWebify.me. The link to Agaric's Twitter account on our page looks like this:

<a href="https://twitter.com/agaric" rel="me">Twitter</a>

Next up is giving the independent web some basic facts of our identity using the h-card microformat. I've never heard anyone claim that microformats have the most intuitive names, but all the properties are documented. We edited our page.tpl.php template to add the h-card class to a h1 tag surrounding our logo, to which we added the class u-logo and our site name with linking to our homepage, to which we added the classes p-name and u-url. Again using IndieWebify.me we verified that the h-card could be read. The markup looks like this:

<h1 class="container h-card"><a href="http://agaric.com/" id="logo" rel="home" title="Agaric home"><img alt="Agaric home" class="u-logo" height="80" src="http://agaric.com/sites/all/themes/agaric_bootstrap/logo.png" width="80" /></a> <a class="p-name u-url" href="http://agaric.com/" rel="home me" title="Home">Agaric</a> <small>We build online.</small></h1>

Finally, blog posts themselves are each marked up as an h-entry and elements of each blog post with h-entry properties. (The IndieWebCamp wiki has a stub article for h-entry and the markup IndieWeb makes use of, but we found the h-entry listing on microformats.org to be clearer.) For blog posts' markup we did a lot of work in template preprocess hooks. For example, here we add the h-entry class itself, the p-name class for the blog title, and (with a bit of reconstruction of Drupal's $submitted variable) the dt-published class for the date and time the blog post was published:

/**
 * Implements hook_preprocess_node().
 */
function agaric_bootstrap_preprocess_node(&$variables) {
  if ($variables['type'] == 'blog') {
    $variables['classes_array'][] = 'h-entry';
    if (!isset($variables['title_attributes']['class'])) {
      $variables['title_attributes_array']['class'] = array();
    }
    $variables['title_attributes_array']['class'][] = 'p-name';
    $datetime = format_date($variables['node']->created, 'custom', 'Y-m-d h:i:s');
    $formatted_date = '<time class="dt-published" datetime="' . $datetime . '">' . $variables['date'] . '</time>';
    $variables['submitted'] = t('Submitted by !username on !datetime', array('!username' =&gt; $variables['name'], '!datetime' =&gt; $formatted_date));
  }
}

Here's the IndieWebify.me validation for this very blog post. The markup looks like this:

<article about="/blogs/marking-drupals-blog-posts-indieweb" class="node node-blog h-entry clearfix" id="node-262" typeof="sioc:Post sioct:BlogPost">

<h1 class="p-name"><a class="u-url" href="http://agaric.com/blogs/marking-drupals-blog-posts-indieweb" rel="bookmark" title="Marking up Drupal's blog posts for the IndieWeb">Marking up Drupal's blog posts for the IndieWeb</a></h1>

<span content="2015-05-04T11:58:16-04:00" datatype="xsd:dateTime" property="dc:date dc:created" rel="sioc:has_creator">Submitted by <a about="/people/benjamin-melan%C3%A7on" class="p-author h-card username" datatype="" href="http://agaric.com/people/benjamin-melan%C3%A7on" property="foaf:name" rel="author" title="View user profile." typeof="sioc:UserAccount" xml:lang="">Benjamin Melançon</a> on <time class="dt-published" datetime="2015-05-04 11:58:16">Mon, 05/04/2015 - 11:58</time></span>
<div class="e-content">…</div>
</article>

What do you think of the IndieWebCamp movement and its goal of making distributed sharing and following easy, while not prescribing which platforms or technologies to use? How about Agaric's far-from-automated approach to making a Drupal site part of the IndieWeb? And do you think Drupal should try to be more IndieWeb-ready as we expect another burst of growth with the release of Drupal 8?

Sign up if you want to know when Mauricio and Agaric give a migration training:

The UnitTest initiative wants to get rid of the Drupal-only Simpletest module. To do this it is necessary to update the functional tests of our modules to stop using the WebTestBase (WTB) class, which is part of the Simpletest module.

Now we need to use the BrowserTestBase (BTB) class and migrate the tests from one to the other.

Migrating from WTB to BTB is relatively straightforward (unless you have to use something which hasn’t been ported yet.

There is a script in this issue that helped to port the Core tests, it needs a few modifications if we want to use it, warning: make sure to have a backup of your tests/module because when this script finish delete the old tests.

But if you want to do it by hand, you can do it following this steps:

• Copy your tests from [your-module]/src/Tests to [your-module]/tests/src/Functional, phpunit will look in that folder automatically.
• Change the namespaces from Drupal\[yourmodule]\Tests to Drupal\Tests\[yourmodule]\Functional
• Change use Drupal\simpletest\WebTestBase to use Drupal\Tests\BrowserTestBase
• Finally change extends WebTestBase to BrowserTestBase

A few things to consider:

• Make sure to you are now extending Drupal\Tests\BrowserTestBase because there is another class BrowserTestBase inside the Simpletest module, which is already deprecated /core/modules/simpletest/src/BrowserTestBase.
• The WebTestBase class will be marked deprecated until Drupal 8.4.x be out (some day near of octobre 2017), so there is still time to migrate our functional tests, but it is definitely good practice stop using WTB when writing any new test
• Once you migrated your tests you will be able to use directly phpunit to run your tests instead to use the run-tests.sh script.
• More info about how getting started with testing at: https://www.drupal.org/docs/8/phpunit
• There is a lot to do to migrate the already written core tests to BTB, if you want to help, check this issue: https://www.drupal.org/node/2807237

Micky was a keynote speaker at UMASS Amherst during the NERD Summit event and the closing keynote speaker at LibrePlanet 2019 @M.I.T. She spoke about how we, as people and as programmers, can work our way out of the digital world of Nineteen Eight-Four that we are living in.  Rather than having about ten slides of fine print and links in the presentation, we are posting resources in this blog post.

Here is a short-enough-to-write-on-a-business-card link for this page – agaric.coop/libreplanet2019 – for sharing these resources with others more easily.

Sign up to be notified when Agaric gives an online or in-person migration training:

Learn how to get the most out of Drupal from expert practitioners with a passion for teaching.

We have practical experience in developing web sites, migrating content, and running technology projects and we love to learn and to teach. We will impart the knowledge and skills you need to get work done, and done right.

Private Training Options

Once a text field has data stored, it is not very easy or obvious how to change its maximum length. In the UI there is a message warning you that the field cannot be changed, because there is existing data. Sometimes it is necessary to change these values. It seems that there are a few ways and some resources to do this in Drupal 7, but I could not find a way to do this in Drupal 8. I decided to create a small function to do it:

Caution: Any change in the database needs to be done carefully. Before you continue please create a backup of your database.

/**
 * Update the length of a text field which already contains data.
 *
 * @param string $entity_type_id
 * @param string $field_name
 * @param integer $new_length
 */
function _module_change_text_field_max_length ($entity_type_id, $field_name, $new_length) {
  $name = 'field.storage.' . $entity_type_id . "." . $field_name;

  // Get the current settings
  $result = \Drupal::database()->query(
    'SELECT data FROM {config} WHERE name = :name',
    [':name' => $name]
  )->fetchField();
  $data = unserialize($result);
  $data['settings']['max_length'] = $new_length;

  // Write settings back to the database.
  \Drupal::database()->update('config')
    ->fields(['data' => serialize($data)])
    ->condition('name', $name)
    ->execute();

  // Update the value column in both the _data and _revision tables for the field
  $table = $entity_type_id . "__" . $field_name;
  $table_revision = $entity_type_id . "_revision__" . $field_name;
  $new_field = ['type' => 'varchar', 'length' => $new_length];
  $col_name = $field_name . '_value';
  \Drupal::database()->schema()->changeField($table, $col_name, $col_name, $new_field);
  \Drupal::database()->schema()->changeField($table_revision, $col_name, $col_name, $new_field);

  // Flush the caches.
  drupal_flush_all_caches();
}

This method needs the name of the entity, the name of the field, and the name and the new length.

And we can use it like this:

   _module_change_text_field_max_length('node', 'field_text', 280);

Usually, this code should be placed in (or called from) a hook_update so it will be executed automatically in the update.

And if the new length is too long to be placed in a regular input area, you can use the Textarea widget for text fields which will allow you to use the larger text area form element for text fields.

Throughout the series we have shown many examples. I do not recall any of them working on the first try. When working on Drupal migrations, it is often the case that things do not work right away. Today’s article is the first of a two part series on debugging Drupal migrations. We start giving some recommendations of things to do before diving deep into debugging. Then, we are going to talk about migrate messages and presented the log process plugin. Let’s get started.

Minimizing the surface for errors

The Migrate API is a very powerful ETL framework that interacts with many systems provided by Drupal core and contributed modules. This adds layers of abstraction that can make the debugging process more complicated compared to other systems. For instance, if something fails with a remote JSON migration, the error might be produced in the Migrate API, the Entity API, the Migrate Plus module, the Migrate Tools module, or even the Guzzle HTTP Client library that fetches the file. For a more concrete example, while working on a recent article, I stumbled upon an issue that involved three modules. The problem was that when trying to rollback a CSV migration from the user interface an exception will be thrown making the operation fail. This is related to an issue in the core Migrate API that manifests itself when rollback operations are initiated from the interface provided by Migrate Plus. Then, the issue causes a condition in the Migrate Source CSV plugin that fails and the exception is thrown.

In general, you should aim to minimize the surface for errors. One way to do this by starting the migration with the minimum possible set up. For example, if you are going to migrate nodes, start by configuring the source plugin, one field (the title), and the destination. When that works, keep migrating one field at a time. If the field has multiple subfields, you can even migrate one subfield at a time. Commit every progress to version control so you can go back to a working state if things go wrong. Read this article for more recommendations on writing migrations.

What to check first?

Debugging is a process that might involve many steps. There are a few things that you should check before diving too deep into trying to find the root of the problem. Let’s begin with making sure that changes to your migrations are properly detected by the system. One common question I see people ask is where to place the migration definition files. Should they go in the migrations or config/install directory of your custom module? The answer to this is whether you want to manage your migrations as code or configuration. Your choice will determine the workflow to follow for changes in the migration files to take effect. Migrations managed in code go in the migrations directory and require rebuilding caches for changes to take effect. On the other hand, migrations managed in configuration are placed in the config/install directory and require configuration synchronization for changes to take effect. So, make sure to follow the right workflow.

After verifying that your changes are being applied, the next thing to do is verify that the modules that provide your plugins are enabled and the plugins themselves are properly configured. Look for typos in the configuration options. Always refer to the official documentation to know which options are available and find the proper spelling of them. Other places to look at is the code for the plugin definition or articles like the ones in this series documenting how to use them. Things to keep in mind include proper indentation of the configuration options. An extra whitespace or a wrong indentation level can break the migration. You can either get a fatal error or the migration can fail silently without producing the expected results. Something else to be mindful is the version of the modules you are using because the configuration options might change per version. For example, the newly released 8.x-3.x branch of Migrate Source CSV changed various configuration options as described in this change record. And the 8.x-5.x branch of Migrate Plus changed some configurations for plugin related with DOM manipulation as described in this change record. Keeping an eye on the issue queue and change records for the different modules you use is always a good idea.

If the problem persists, look for reports of similar problems in the issue queue. Make sure to include closed issues as well in case your problem has been fixed or documented already. Remember that a problem in a module can affect a different module. Keeping an eye on the issue queue and change records for all the modules you use is always a good idea. Another place ask questions is the #migrate channel in Drupal slack. The support that is offered there is fantastic.

Migration messages

If nothing else has worked, it is time to investigate what is going wrong. In case the migration outputs an error or a stacktrace to the terminal, you can use that to search in the code base where the problem might originate. But if there is no output or if the output is not useful, the next thing to do is check the migration messages.

The Migrate API allows plugins to log messages to the database in case an error occurs. Not every plugin leverages this functionality, but it is always worth checking if a plugin in your migration wrote messages that could give you a hint of what went wrong. Some plugins like skip_on_empty and skip_row_if_not_set even expose a configuration option to specify messages to log. To check the migration messages use the following Drush command: drush migrate:messages [migration_id]. If you are managing migrations as configuration, the interface provided by Migrate Plus also exposes them.

Messages are logged separately per migration, even if you run multiple migrations at once. This could happen if you execute dependencies or use groups or tags. In those cases, errors might be produced in more than one migration. You will have to look at the messages for each of them individually.

Let’s consider the following example. In the source there is a field called src_decimal_number with values like 3.1415, 2.7182, and 1.4142. It is needed to separate the number into two components: the integer part (3) and the decimal part (1415). For this, we are going to use the extract process plugin. Errors will be purposely introduced to demonstrate the workflow to check messages and update migrations. The following example shows the process plugin configuration and the output produced by trying to import the migration:

# Source values: 3.1415, 2.7182, and 1.4142

psf_number_components:
  plugin: explode
  source: src_decimal_number

$ drush mim ud_migrations_debug
[notice] Processed 3 items (0 created, 0 updated, 3 failed, 0 ignored) - done with 'ud_migrations_debug'

In MigrateToolsCommands.php line 811:
ud_migrations_debug Migration - 3 failed.

The error produced in the console does not say much. Let’s see if any messages were logged using: drush migrate:messages ud_migrations_debug. In the previous example, the messages will look like this:

 ------------------- ------- --------------------
  Source IDs Hash    Level   Message
 ------------------- ------- --------------------
  7ad742e...732e755   1       delimiter is empty
  2d3ec2b...5e53703   1       delimiter is empty
  12a042f...1432a5f   1       delimiter is empty
 ------------------------------------------------

In this case, the migration messages are good enough to let us know what is wrong. The required delimiter configuration option was not set. When an error occurs, usually you need to perform at least three steps:

• Rollback the migration. This will also clear the messages.
• Make changes to definition file and make they are applied. This will depend on whether you are managing the migrations as code or configuration.
• Import the migration again.

Let’s say we performed these steps, but we got an error again. The following snippet shows the updated plugin configuration and the messages that were logged:

psf_number_components:
  plugin: explode
  source: src_decimal_number
  delimiter: '.'

 ------------------- ------- ------------------------------------
  Source IDs Hash    Level   Message
 ------------------- ------- ------------------------------------
  7ad742e...732e755   1       3.1415000000000002 is not a string
  2d3ec2b...5e53703   1       2.7181999999999999 is not a string
  12a042f...1432a5f   1       1.4141999999999999 is not a string
 ----------------------------------------------------------------

The new error occurs because the explode operation works on strings, but we are providing numbers. One way to fix this is to update the source to add quotes around the number so it is treated as a string. This is of course not ideal and many times not even possible. A better way to make it work is setting the strict option to false in the plugin configuration. This will make sure to cast the input value to a string before applying the explode operation. This demonstrates the importance of reading the plugin documentation to know which options are at your disposal. Of course, you can also have a look at the plugin code to see how it works.

Note: Sometimes the error produces an non-recoverable condition. The migration can be left in a status of "Importing" or "Reverting". Refer to this article to learn how to fix this condition.

The log process plugin

In the example, adding the extra configuration option will make the import operation finish without errors. But, how can you be sure the expected values are being produced? Not getting an error does not necessarily mean that the migration works as expected. It is possible that the transformations being applied do not yield the values we think or the format that Drupal expects. This is particularly true if you have complex process plugin chains. As a reminder, we want to separate a decimal number from the source like 3.1415 into its components: 3 and 1415.

The log process plugin can be used for checking the outcome of plugin transformations. This plugin offered by the core Migrate API does two things. First, it logs the value it receives to the messages table. Second, the value is returned unchanged so that it can be used in process chains. The following snippets show how to use the log plugin and what is stored in the messages table:

psf_number_components:
  - plugin: explode
    source: src_decimal_number
    delimiter: '.'
    strict: false
  - plugin: log

 ------------------- ------- --------
  Source IDs Hash    Level   Message
 ------------------- ------- --------
  7ad742e...732e755   1       3
  7ad742e...732e755   1       1415
  2d3ec2b...5e53703   1       2
  2d3ec2b...5e53703   1       7182
  12a042f...1432a5f   1       1
  2d3ec2b...5e53703   1       4142
 ------------------------------------

Because the explode plugin produces an array, each of the elements is logged individually. And sure enough, in the output you can see the numbers being separated as expected.

The log plugin can be used to verify that source values are being read properly and process plugin chains produce the expected results. Use it as part of your debugging strategy, but make sure to remove it when done with the verifications. It makes the migration to run slower because it has to write to the database. The overhead is not needed once you verify things are working as expected.

In the next article, we are going to cover the Migrate Devel module, the debug process plugin, recommendations for using a proper debugger like XDebug, and the migrate:fields-source Drush command.

What did you learn in today’s blog post? What workflow do you follow to debug a migration issue? Have you ever used the log process plugin for debugging purposes? If so, how did it help to solve the issue? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: How to debug Drupal migrations - Part 2

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.