Blog

A protester with a sign in the background that reads "Fight Racism. Sexism. All Oppression."

Portside

Amplifying diverse voices on the left.

Silhouettes of a people on soapboxes shouting through bullhorns at a crowd, demonstrating stand / deliver, a portable dueling free speech unit, a mobile box equipped with two platforms and megaphones.

Uniting Visions

Kicking off Thursday 3pm ET planning & building sessions for democratic conversation scaling platform, Visions Unite

The Challenge

Many teachers lack the skills and confidence to effectively integrate computer modeling and simulation into their science classrooms. Research studies show that it takes more than a single professional development workshop to build the necessary skill and confidence. Online communities have been proposed as a means to address this problem. Teachers with GUTS (Growing Up Thinking Scientifically) offers an engaging middle school curriculum and accompanying professional development for teachers.

Through the use of an online community of practice, Teachers with GUTS is able to support teachers in mitigating challenges, gaining expertise by providing additional training and resources, and providing answers to teachers; questions as they bring computational science experiences into their classrooms. To truly teach with GUTS, teachers needed a community they could turn to.

Social media platforms and existing online discussion forums were considered, but research showed that these platforms lacked the features specifically needed. Instead, they needed a platform tailored to their needs:

Forum for topical discussions
Resource library to share documents, videos, and code
Events listing page
Practice area for participating in monthly “challenges” or “work sessions”

Our Approach

Through our discovery work we specifically identified that a forum, resource library, practice space, and faceted search would provide teachers with support and resources and tools they need to teach the curricula with confidence. TWiG would then support the website with both drip and push marketing to promote the resources and encourage activity.

Resource Library

Project GUTS CS in Science curriculum is the heart of TWiG’s work. There are several learning modules as well as supporting documents such as rubrics and supplementary activities. We built the resource library that displayed the most recent additions front and center, while providing a faceted search option so teachers can drill down to specific resources.

We also took advantage of the collaborative spirit of the TWiG community by allowing teachers to submit their own resources. These resources are then reviewed by TWiG staff and published after review, revision if needed, and alignment with standards.

Faceted search allows teachers to find resources based on several criteria.

Forum

To support online discussion and peer assistance, we built a forum. We built off of Drupal Core’s Forum module and extended it to include an area accessible only to facilitators with a custom module.

We customized the forum further by disabling threaded comments, as that structure did not work well for the teachers using the site. However, in doing so we learned of a longstanding issue with Drupal and comments in which converting threaded comments to a flat structure risks deleting the nested comments. In response, we ported the Flat Comments module to Drupal 8.

Bookmarks

Even with faceted search and thoughtful categorization, the number of resources and discussions on the site can be overwhelming. We created a Drupal 8 version of the Backpack module that allows teachers to save pages to their “backpack.”

These backpacks can either be private or shared with other members. Curating specific lists is another way teachers can share knowledge with one another.

You can download and use the Backpack module on your Drupal site at https://github.com/agaric/bookmark

Screenshot of a user's backpack, a page of bookmarked pages.

Teachers can bookmark a resource or discussion for quick access in the future.

Notifications

While a custom site was more advantageous than an email list or Facebook group, the reality is that most teachers’ daily routine does not include visiting teacherswithguts.org. In order to keep teachers engaged they needed a way to know when relevant activity was taking place.

We built a notification system, configurable by each teacher, so that they could be emailed about activity pertinent to their work. Teachers can choose to receive the following notifications:

When a new article is posted
When a new resource is posted
When a new discussion is posted
When a comment is made on a discussion

Additionally, teachers choose the day to be sent emails, as well as the frequency. This was all made possible by enhancing the Personal Digest and Comment Notify modules.

Screenshot of notification settings page.

We enhanced the Personal Digest and Comment Notify modules to give users fine grained control over their notifications.

Member Directory

To build upon the networking happening in the forums, we made it easy for teachers to find one another based on shared on interests, experience and geography. Each teacher customizes their member profile with key information about themselves, their areas of expertise and the areas they wish to grow. We then surfaced that in a filterable search of members, helping teachers mentor one another.

Members can easily find others with similar interests in the member directory.

Results

With an onboarding process much improved from their previous site, plus the ability to bulk invite users, TWiG was able to sign up over 700 teachers when the site launched in 2016. We’ve continued to improve the site to increase participation and surface useful metrics for site administrators and researchers.

Event website

If there is a website for this event, typle the URL here. Leave blank if there is no website. The more information we have about your event, the more relevant our presentation will be!

Event date and time

Event location

If you do not have an event location yet, leave this field set to 'None'.

Address

City/Town

Country

Event type

What type of event are you having? We can provide presentations, workshops or demonstrations of free software tools such as video chat, document management and storage, communication tools that protect your privacy and security.

Would you like Agaric to host your online event?

Would you like Agaric to provide a recording of your event?

Your budget

If you do not have a budget, leave this field blank and check the box below.

I do not have a budget

I have a mission without a budget

Please explain the mission of your request and how it will help your community. We do not wish to prevent those without funds from benefitting from our expertise.

What else should the speaker know about this event?

Please include any information that would be helpful for us to be able to give the most relevant presentation or workshop.

Contact's email *

Contact's phone number

Your information will not be shared.

Throughout the series we have shown many examples. I do not recall any of them working on the first try. When working on Drupal migrations, it is often the case that things do not work right away. Today’s article is the first of a two part series on debugging Drupal migrations. We start giving some recommendations of things to do before diving deep into debugging. Then, we are going to talk about migrate messages and presented the log process plugin. Let’s get started.

Example configuration for log process plugin.

Minimizing the surface for errors

The Migrate API is a very powerful ETL framework that interacts with many systems provided by Drupal core and contributed modules. This adds layers of abstraction that can make the debugging process more complicated compared to other systems. For instance, if something fails with a remote JSON migration, the error might be produced in the Migrate API, the Entity API, the Migrate Plus module, the Migrate Tools module, or even the Guzzle HTTP Client library that fetches the file. For a more concrete example, while working on a recent article, I stumbled upon an issue that involved three modules. The problem was that when trying to rollback a CSV migration from the user interface an exception will be thrown making the operation fail. This is related to an issue in the core Migrate API that manifests itself when rollback operations are initiated from the interface provided by Migrate Plus. Then, the issue causes a condition in the Migrate Source CSV plugin that fails and the exception is thrown.

In general, you should aim to minimize the surface for errors. One way to do this by starting the migration with the minimum possible set up. For example, if you are going to migrate nodes, start by configuring the source plugin, one field (the title), and the destination. When that works, keep migrating one field at a time. If the field has multiple subfields, you can even migrate one subfield at a time. Commit every progress to version control so you can go back to a working state if things go wrong. Read this article for more recommendations on writing migrations.

What to check first?

Debugging is a process that might involve many steps. There are a few things that you should check before diving too deep into trying to find the root of the problem. Let’s begin with making sure that changes to your migrations are properly detected by the system. One common question I see people ask is where to place the migration definition files. Should they go in the migrations or config/install directory of your custom module? The answer to this is whether you want to manage your migrations as code or configuration. Your choice will determine the workflow to follow for changes in the migration files to take effect. Migrations managed in code go in the migrations directory and require rebuilding caches for changes to take effect. On the other hand, migrations managed in configuration are placed in the config/install directory and require configuration synchronization for changes to take effect. So, make sure to follow the right workflow.

After verifying that your changes are being applied, the next thing to do is verify that the modules that provide your plugins are enabled and the plugins themselves are properly configured. Look for typos in the configuration options. Always refer to the official documentation to know which options are available and find the proper spelling of them. Other places to look at is the code for the plugin definition or articles like the ones in this series documenting how to use them. Things to keep in mind include proper indentation of the configuration options. An extra whitespace or a wrong indentation level can break the migration. You can either get a fatal error or the migration can fail silently without producing the expected results. Something else to be mindful is the version of the modules you are using because the configuration options might change per version. For example, the newly released 8.x-3.x branch of Migrate Source CSV changed various configuration options as described in this change record. And the 8.x-5.x branch of Migrate Plus changed some configurations for plugin related with DOM manipulation as described in this change record. Keeping an eye on the issue queue and change records for the different modules you use is always a good idea.

If the problem persists, look for reports of similar problems in the issue queue. Make sure to include closed issues as well in case your problem has been fixed or documented already. Remember that a problem in a module can affect a different module. Keeping an eye on the issue queue and change records for all the modules you use is always a good idea. Another place ask questions is the #migrate channel in Drupal slack. The support that is offered there is fantastic.

Migration messages

If nothing else has worked, it is time to investigate what is going wrong. In case the migration outputs an error or a stacktrace to the terminal, you can use that to search in the code base where the problem might originate. But if there is no output or if the output is not useful, the next thing to do is check the migration messages.

The Migrate API allows plugins to log messages to the database in case an error occurs. Not every plugin leverages this functionality, but it is always worth checking if a plugin in your migration wrote messages that could give you a hint of what went wrong. Some plugins like skip_on_empty and skip_row_if_not_set even expose a configuration option to specify messages to log. To check the migration messages use the following Drush command: drush migrate:messages [migration_id]. If you are managing migrations as configuration, the interface provided by Migrate Plus also exposes them.

Messages are logged separately per migration, even if you run multiple migrations at once. This could happen if you execute dependencies or use groups or tags. In those cases, errors might be produced in more than one migration. You will have to look at the messages for each of them individually.

Let’s consider the following example. In the source there is a field called src_decimal_number with values like 3.1415, 2.7182, and 1.4142. It is needed to separate the number into two components: the integer part (3) and the decimal part (1415). For this, we are going to use the extract process plugin. Errors will be purposely introduced to demonstrate the workflow to check messages and update migrations. The following example shows the process plugin configuration and the output produced by trying to import the migration:

# Source values: 3.1415, 2.7182, and 1.4142

psf_number_components:
  plugin: explode
  source: src_decimal_number

$ drush mim ud_migrations_debug
[notice] Processed 3 items (0 created, 0 updated, 3 failed, 0 ignored) - done with 'ud_migrations_debug'

In MigrateToolsCommands.php line 811:
ud_migrations_debug Migration - 3 failed.

The error produced in the console does not say much. Let’s see if any messages were logged using: drush migrate:messages ud_migrations_debug. In the previous example, the messages will look like this:

 ------------------- ------- --------------------
  Source IDs Hash    Level   Message
 ------------------- ------- --------------------
  7ad742e...732e755   1       delimiter is empty
  2d3ec2b...5e53703   1       delimiter is empty
  12a042f...1432a5f   1       delimiter is empty
 ------------------------------------------------

In this case, the migration messages are good enough to let us know what is wrong. The required delimiter configuration option was not set. When an error occurs, usually you need to perform at least three steps:

Rollback the migration. This will also clear the messages.
Make changes to definition file and make they are applied. This will depend on whether you are managing the migrations as code or configuration.
Import the migration again.

Let’s say we performed these steps, but we got an error again. The following snippet shows the updated plugin configuration and the messages that were logged:

psf_number_components:
  plugin: explode
  source: src_decimal_number
  delimiter: '.'

 ------------------- ------- ------------------------------------
  Source IDs Hash    Level   Message
 ------------------- ------- ------------------------------------
  7ad742e...732e755   1       3.1415000000000002 is not a string
  2d3ec2b...5e53703   1       2.7181999999999999 is not a string
  12a042f...1432a5f   1       1.4141999999999999 is not a string
 ----------------------------------------------------------------

The new error occurs because the explode operation works on strings, but we are providing numbers. One way to fix this is to update the source to add quotes around the number so it is treated as a string. This is of course not ideal and many times not even possible. A better way to make it work is setting the strict option to false in the plugin configuration. This will make sure to cast the input value to a string before applying the explode operation. This demonstrates the importance of reading the plugin documentation to know which options are at your disposal. Of course, you can also have a look at the plugin code to see how it works.

Note: Sometimes the error produces an non-recoverable condition. The migration can be left in a status of "Importing" or "Reverting". Refer to this article to learn how to fix this condition.

The log process plugin

In the example, adding the extra configuration option will make the import operation finish without errors. But, how can you be sure the expected values are being produced? Not getting an error does not necessarily mean that the migration works as expected. It is possible that the transformations being applied do not yield the values we think or the format that Drupal expects. This is particularly true if you have complex process plugin chains. As a reminder, we want to separate a decimal number from the source like 3.1415 into its components: 3 and 1415.

The log process plugin can be used for checking the outcome of plugin transformations. This plugin offered by the core Migrate API does two things. First, it logs the value it receives to the messages table. Second, the value is returned unchanged so that it can be used in process chains. The following snippets show how to use the log plugin and what is stored in the messages table:

psf_number_components:
  - plugin: explode
    source: src_decimal_number
    delimiter: '.'
    strict: false
  - plugin: log

 ------------------- ------- --------
  Source IDs Hash    Level   Message
 ------------------- ------- --------
  7ad742e...732e755   1       3
  7ad742e...732e755   1       1415
  2d3ec2b...5e53703   1       2
  2d3ec2b...5e53703   1       7182
  12a042f...1432a5f   1       1
  2d3ec2b...5e53703   1       4142
 ------------------------------------

Because the explode plugin produces an array, each of the elements is logged individually. And sure enough, in the output you can see the numbers being separated as expected.

The log plugin can be used to verify that source values are being read properly and process plugin chains produce the expected results. Use it as part of your debugging strategy, but make sure to remove it when done with the verifications. It makes the migration to run slower because it has to write to the database. The overhead is not needed once you verify things are working as expected.

In the next article, we are going to cover the Migrate Devel module, the debug process plugin, recommendations for using a proper debugger like XDebug, and the migrate:fields-source Drush command.

What did you learn in today’s blog post? What workflow do you follow to debug a migration issue? Have you ever used the log process plugin for debugging purposes? If so, how did it help to solve the issue? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: How to debug Drupal migrations - Part 2

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Resources

We're always publishing more about migrating data and also have a growing collection of resources in the follow-up section of our migration training resources page.

And be sure to check out our other trainings!

Image credit: Karlheinrichpasch, CC BY-SA 4.0, via Wikimedia Commons.

We're delighted you'll be participating in an Agaric migration training. Please let us know if you'd like to learn about future training opportunities:

The red boxes above indicate what needs to be done. Be sure to select the relationship you added to the Relationship field. This will allow the contextual filter to use the value of the Event's entity reference field for comparison.

Click Provide default value and select Content ID from URL in the dropdown that appears. Note that if your nodes have URL aliases that do not include the content ID, Drupal will still provide the content ID to your contextual filter. Now your view will know which content to display by comparing the current node ID with the the values in the entity reference field of each listing.

Lastly, Click Apply and do not forget to save your view.

The End

Congratulations! Now we know how to filter our view by content that references the current node. You probably still need to place your block and configure which nodes it should or should not display on, but I think that is a "How to" for another day. Have fun Drupaling!

Diagram of the Digital Commons.

We live amidst a Digital Commons - technology that is built with the principles of freedom and transparency baked into its code and design. It's maintained out in the open by the free software community. This commons is invisible to many of us, but the closer we are to the technology we use, the more that it comes into focus. At Agaric we are knee deep in this Digital Commons. Our name Agaric is a nod to the mycelial nature of the open web. We help create, maintain, and promote free and open-source software that make up this commons.

Resources can be either a file-based resource such as a PDF or a link-based resource such as a url to a website. Resources can also be categorized by resource type and can also have an associated tag. An associated image such as the cover of a PDF resource or a logo or screenshot from a website link can make the resource more visually attractive in various displays.

The resource content type lets you post a PDF (or various other text based formats) so that site visitors can easily view and download. The body field lets you provide some information about the resource to help site visitors determine the usefulness of the resource before viewing.

Helpful tips include ensuring that your PDF has a user-friendly machine name (and/or using the file description field for the title users will see) and adding a cover image for example, in the image field, to provide a visual clue to the resource.

For link-based resources, you'll need to paste in the full website URL, such as https://drutopia.org. The Link text could read "Visit the Drutopia website."

Construyendo la web abierta

The growing community built around Drupal offers a multitude of options to easily integrate Drupal into your current lifestyle and to branch out in new and different ways. Here are a few ways to migrate your lifestyle to one as a Freelance Drupal Developer and a contributor to the community.

If you have a 9 to 5 job at an office or a specific location, you can choose to view online Drupal tutorials in whatever spare time you have instead of watching a movie. Drupal tutorials can be quite entertaining at times and are especially rewarding because they will change your life for the better.

Sign up for an account on Drupal.org and download the latest version of Drupal. Install Drupal locally on your hard drive and use some of the knowledge you learned in the tutorials. If you get stuck, log onto groups.drupal.org and search for an answer to your issue - chances are good that someone else has had the same issue and the solution will be posted there. As you search through the posts on the forum, be sure to respond to any posts that you may have the solution for. Since you have just gone through the install and setup, there is a good possibility that even as a beginner - you too may be able to help someone else. It's a good practice to do both things on a forum visit:

Ask For Help
Give Some Help

After a few weeks or months viewing tutorial videos and getting a solid idea of the moving parts within Drupal, you can start to create your first real site. I recommend that you involve a friend. Most people have at least one friend that has an idea for a business venture. Ask your friend if you may help them build a site based on their business idea. Enlist the friend to help you by inviting them over for a bowl of popcorn and a few Drupal movies...

A program or application is free software when it is intentionally licensed with our freedom in mind. Specifically, four freedoms (numbered using the zero-based numbering in programming languages):

Freedom 0 - The freedom to run the program as you wish, for any purpose.
Freedom 1 - The freedom to study how the program works, and change it so it does your computing as you wish.
Freedom 2 - The freedom to redistribute copies so you can help others.
Freedom 3 - The freedom to distribute copies of your modified versions to others.

From Linux to Firefox to Drupal, free software is a tremendous force in our world. By building free software, we help building a software commons we can all trust and benefit from.

Why Free?

We live in a world where much of the software we use violates our privacy and manipulates our behavior. The Free Software freedoms are more important than ever.

Building and supporting free software expands the choices we have. It keeps us safer and forces proprietary software to adhere to higher standards.

This is why we build, maintain and promote free software projects whenever possible.

A Note On Other Words: Libre Software, Open Source

There are similar terms used to describe similar software. Libre software is a synonym for free software. By using the word libre, we remove the ambiguity inherent in free, which can mean free as in no money or free as in liberty. We appreciate this term and use it interchangeably with free software.

Open source is very similar to free software. It emerged, like libre software, as a way to remove ambiguity. However, in doing so it loses the ethical and political power that comes with free/libre software. So while we respect and use the term open source, we prefer free software because of it emphasizes the value of freedom.