On a cold December night we met at the Industry Lab to celebrate the Worc'n group and our membership in the cooperatives we belong to. The crowd was very energetic and several small groups were conversing on different topics throughout the evening. A holiday party is a time to see old friends and make new friends.

A cooperative is a structure to make new connections and share ideas. What better mix for a party. 2014 ended on a high note as the cooperative movement gains ground in Cambridge, MA. Worc'n (the Worker-Owned and Run Cooperative Network of Greater Boston) has been around for years and has an impressive number of cooperatives and owner-workers among their members. This party was sort of a merging of groups in the sense that Worc'n has been around for a long time, growing a membership and creating value.

The Boston/Cambridge Worker-ownership Meetup just started a few months ago, but through the network of local cooperatives, the word spread quickly and people have been expressing their happiness that there are now some meetngs to attend and new people to network with. The party was a Pot Luck and everyone brought something to share. There were homemade cookies, cakes and pie. We also were fortunate to have Monica Leitner-Laserna of La Sanghita Cafe attend the party with some delicious vegan foods from her newly formed cooperative restaurant in East Boston.

La Sanghita Cafe is Boston's only cooperative restaurant and they have a great vegan menu. Located just outside Maverick Square in East Boston, the restaurant is open daily for lunch and Wednesday through Saturday they offer a dinner menu. The menu is filled with sweet and savory items and the space is so comfortable and cozy. We look forward to having the coop members as guests at our monthly cooperative worker/owner meetup, to share the details on how it works to build a cooperative restaurant. Good food, good conversation and many engaging in discussion on topics involving cooperation, a wonderful evening was had by all. We should do this more often.

When one starts working with migrations, it is easy to be overwhelmed by so many modules providing migration functionality. Throughout the series, we presented many of them trying to cover one module at a time. We did this to help the reader understand when a particular module is truly needed and why. But we only scratched the surface. Today’s article presents a list of migration related Drupal modules, all good for Drupal 8, for quick reference. Let’s get started.

Core modules

At the time of this writing, Drupal core ships with four migration modules:

• Migrate: provides the base API for migrating data.
• Migrate Drupal: offers functionality to migrate from other Drupal installations. It serves as the foundation for upgrades from Drupal 6 and 7. It also supports reading configuration entities from Drupal 8 sites.
• Drupal Migrate UI: provides a user interface for upgrading a Drupal 6 or 7 site to Drupal 8.
• Migrate Drupal Multilingual: is an experimental module required by multilingual translations. When they become stable, the module will be removed from Drupal core. See this article for more information.

Migration runners

Once the migration definition files have been created, there are many options to execute them:

• Migrate Tools: provides Drush commands to run migrations from the command line. It also exposes a user interface to run migrations created as configuration entities. It offers support for migration groups and tags. The module depends on Migrate Plus.
• Migrate Run: provides Drush commands to run migrations from the command line. It does not offer support for migration groups, but tags are supported.  The module does not depend on Migrate Plus.
• Migrate Upgrade: provides Drush support for upgrading a Drupal 6 or 7 site to Drupal 8.
• Migrate Manifest: provides a Drush command for running migrations using a manifest file. See this article for an example of using this module for Drupal upgrades.
• Migrate Scheduler: integrates with Drupal Core’s Cron API to execute migrations on predefined schedules.
• Migrate Cron: exposes a user interface to execute migrations when Cron is triggered. At the time of this writing, the module does not execute dependent migrations. Follow this issue for updates.
• Migrate source UI: provides a form for uploading files to use as source for already defined CSV, JSON, and XML migrations. At the time of this writing, it seems that JSON and XML migrations are not being detected. Follow this issue for updates.

Source plugins

The Migrate API offers many options to fetch data from:

• Migrate (core module): provides the `SqlBase` abstract class to help with fetching data from a database connection. See this article for an example. It also exposes the `embedded_data` plugin which allows the source data to be defined inside the migration definition file. It was used extensively in the example migrations of this series. It also offers the `empty` plugin which returns a row based on provided constants. It is used in multilingual migrations for entity references.
• Migrate Plus: combining various plugins, it allows fetching data in JSON, XML, and SOAP formats. It also provides various plugins for parsing HTML. See this article by Benji Fisher for an example. There is also a patch to add support for PDF parsing.
• Migrate Source CSV: allows fetching data from CSV files.
• Migrate Google Sheets: leverages Migrate Plus functionality to allow fetching data from Google Sheets.
• Migrate Spreadsheet: allows fetching data from Microsoft Excel and LibreOffice Calc files.
• Migrate Source YAML: allows fetching data from YAML files.
• WP Migrate: allows fetching data from a WordPress database.

Destination plugins

The Migrate API is mostly used to move data into Drupal, but it is possible to write to other destinations:

• Migrate (core): provides classes for creating content and configuration entities. It also offers the `null` plugin which in itself does not write to anything. It is used in multilingual migrations for entity references.
• Migrate Plus: provides the `table` plugin for migrating into tables not registered with Drupal Schema API.
• CSV file: example destination plugin implementation to write CSV files. The module was created by J Franks for a DrupalCamp presentation. Check out the repository and video recording.

Development related

These modules can help with writing Drupal migrations:

• Migrate (core): provides the `log` process plugin. See this article for an example of its use.
• Migrate Devel: offers Drush options for printing debug information when executing migrations. It also provides the `debug` process plugin. See this article for an example of its use.
• Migrate Process Vardump: provides the `vardump` plugin. It works like the `debug` plugin.

Field and module related

• Migrate Media Handler: provides migration process plugins to facilitate the migration into Drupal 8 media entities. The source can be Drupal 7 file or image fields. It also supports inline file embeds in rich text. It leverages the DOM parsing plugins provided by Migrate Plus.
• Media Migration: provides an upgrade path from Drupal 7 to Drupal 8 media entities. The source can be image fields and fields attached to media and file entities.
• Migrate File Entities to Media Entities: migrates Drupal 8.0 file entities to Drupal 8.5 media entities.
• Migrate Files: provides process plugins for migrating files and images.
• Webform Migrate: provides a plugin to help migrating from the Drupal 6 and 7 versions of the Webform module.
• Migrate HTML to Paragraphs: turns HTML markup into paragraph entities.
• Commerce Migrate: offers a general purpose migration framework for bringing store information into Drupal Commerce.
• Address: offers a process plugin to migrate data into fields of type address. It also provides an upgrade path from Drupal 7’s Address Field module. See this article for an example.
• Geofield: offers a process plugin to migrate data into fields of type geofield. See this article for an example.
• Office Hours: offers a process plugin to migrate data into fields of type office hours.
• Workbench Moderation to Content Moderation: migrates configuration from one module to the other.

Modules created by Tess Flynn (socketwench)

While doing the research for this article, we found many useful modules created by Tess Flynn (socketwench). She is a fantastic presenter who also has written about Drupal migrations, testing, and much more. Here are some of her modules:

• Migrate Directory: imports files from a directory into Drupal as managed files.
• Migrate Process S3: downloads objects from an S3 bucket into Drupal.
• Migrate Process URL: provides a process plugin to make it easier to migrate into link fields.
• Migrate Process Vardump: helps with debugging migrations.
• Many process plugins that wrap PHP functions. For example: Migrate Process Array, Migrate Process Trim, Migrate Process Regex, Migrate Process Skip, and Migrate Process XML.

Miscellaneous

• Feeds Migrate: it aims to provide a user interface similar to the one from Drupal 7’s Feeds module, but working on top of Drupal 8’s Migrate API.
• Migrate Override: allows flagging fields in a content entity so they can be manually changed by side editors without being overridden in a subsequent migration.
• Migrate Status: checks if migrations are currently running.
• Migrate QA: provides tools for validating content migrations. See this presentation for more details.

And many, many more modules!

What did you learn in today’s blog post? Did you find a new module that could be useful in current or future projects? Did we miss a module that has been very useful to you? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Introduction to Drupal 8 upgrades

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Creating an issue fork to contribute to a Drupal module, without getting confused.

(This cheat sheet is for myself. Fellow cheaters welcome.)

Check the branch that the issue is against— even if you filed the issue yourself, it should have defaulted to the branch people should be working on.

Go to the module page and scroll down, if you are lucky there will be a dev release toward the bottom, for example "Development version: 2.0.x-dev", which you can click into and copy the composer command.

(If there is not a dev release, you'll have to look at Development in the sidebar, click through "Source code" and copy from the Code dropdown either the URL under Clone with SSH or Clone with HTTPS, and git clone that into a modules.

Press the Create issue fork button and wait a minute.

Press the "Show commands" link, and copy-paste the commands in the Add & fetch this issue fork’s repository

And then copy-paste the commands from Check out this branch for the first time

Assign to yourself in "Issue metadata fieldset and save the issue form with any comment about how you plan to work on this that you hope somebody might provide feedback on or as a reminder to your later self.

Talk about web components has been going for quite some time now: the term was coined in 2011. In this blog post, I will discuss the basics of what they are and what they do, and introduce the primary technological foundations for them. In future articles, I'll dig into the specifics and sample code - as they are currently supported or else poly-filled for today.

Web Components are a group of related standards promising to bring component-based extensibility to HTML documents. The great promise is that once you or someone else builds some component that provides a particular feature, it can easily be included and reused in other projects. For example, if you've created, say, a "treeview" component that displays data in a tree form, you can wrap up the definition of this in a component. Then, it would be possible for anyone to include the component easily wherever they might want to use a treeview. We'll ultimately get to defining our own components in a future article, but for now, let's look at a special sort of "built-in" web component.

Perhaps the most common HTML element that comes up when discussing web components is: the video tag. This is because the video tag offers a simple and very clear browser-native example of precisely what a web component looks like from the point of view of a consumer of a particular of a web component. If we were to include a video tag in a page, we don't end up with a naked auto-paying video display (by default). Instead, we end up with a nice little video player, complete with start/stop/seek and volume controls offered up by our browser:

In our browser (which again, in this case is Chromium), we can select the developer tools option under the General section / Elements titled "Show user agent shadow DOM." Given this option is set, we are able to see how Chromium builds upon the video tag when it is rendered:

As you can see, beneath the video element comes one of the core pieces of the Web Components technologies: the shadow-root. Chromium has treated the video tag as a "shadow host" which is a containing element for a "shadow root", off of which hangs the implementation of the video element's controls. This shadow host/shadow root mechanism is the basis for encapsulation in web components: it separates the details of an element's implementation from the outside document. CSS styles and JavaScript inside the shadow root are scoped to that element. Inside of the shadow root, the implementation of the video tag is a sub-tree of elements which is not directly inline with the the document's top-level DOM. This means, for example, you can neither find nor directly address elements within the shadow root.

There are at least a few libraries that enable building web-components for current browsers, regardless of browser support for the standards-based web components support. The one I'll demonstrate quickly here is pulled directly from polymer's home page. I'm using this because it demonstrates what I believe is about as close to the implementation of the ideal for web components (from a consumer perspective) aside from having to use a polyfill for some browsers today:

<!-- Polyfill Web Components support for older browsers -->

<script src="components/webcomponentsjs/webcomponents-lite.min.js"></script>

<!-- Import element -->

<link rel="import" href="components/google-map/google-map.html">

<!-- Use element -->

<google-map latitude="37.790" longitude="-122.390"></google-map>

Again, the polyfill ensures that our browser will handle the subsequent tags. This library aims to support the core web components features in a way that is consistent with the core web-components as the standards evolve. The link tag is another standard web component feature, though Mozilla is not favoring supporting it further, that essentially imports an external definition which in this case is the definition of the tag. When all browsers support web components natively, things will hopefully be as simple as importing the reference to the component and then using it, as is done in the last statement above.

Hopefully this gives you a quick glimpse into how web components might make our web applications grow through a variety of new interface tools we can build on, extend, and easily utilize. In future articles, we'll look at examples of using the various building blocks of web components, and how the technology continues to evolve.

Ben and Micky spoke at DrupalCon Seattle 2019 and New England Drupal Camp 2019 about scaling community conversations and decision-making.  Rather than having slides of fine print and links, we're posting resources here in this blog post.

Here's a short-enough-to-write-on-a-business-card link for this page – agaric.coop/scaling-community – for easier sharing these resources with others.

We are excited to tailor private trainings to suit your needs. Here you will get an idea of the types of private trainings we have given before and would love to give again.

Let us know if you have any questions or special requests

Get Started with Drupal

We love to introduce people to Drupal's basics. We offer short introductory seminars for small groups on an on-demand basis for site building, templating, and developing extensions. Each seminar incorporates when appropriate the information architecture, user experience testing, and leadership and collaboration techniques needed to make any project a success.

Build Your Skills

We teach development teams and solo coders in-person or online, with the program tailored to the problems you are trying to solve. We have provided such capacity-building in Drupal programming, theming and templating, data migration, JavaScript frameworks, and more— and we are always upping our technology skills, so let us know what you are looking for and we may be just ahead of you on the learning curve— and ready to throw down a rope ladder, or at least a grappling hook.

In addition, we offer courses with a defined curricula on building, theming, and developing sites. And we give decision-maker seminars for leaders who need a better understanding of the technology underlying their projects.

Tell Us What You Need!

350.org is a global climate justice organization that helped organize the largest climate strike in history. Together we improved their Climate Justice Action Map (CJAM) in the lead up to the strike so that activists could better mobilize their communities.

Benjamin Melançon of Agaric helped with a patch for the Drupal 7 version of Insert module.

Last week we were asked to disable the "threads" in the comments and display them as a plain list in one of our projects, so checking in the Comment field settings we found an option which says:

"Threading: Show comment replies in a threaded list."

Which basically display the comments as a threaded instead of a list of comments. This option is marked on by default, so it seemed that it was just a matter of unchecking this option, but there was a problem with this, that setting did display the comments as a plain list, but in the code they were still being saved as a thread.

The main problem with this is if a comment is deleted and it has replies (even if the option of the thread is unchecked) all the replies are going to be deleted as well.

This is not a new problem there is an 11 year old issue and it seems that this has been happening since Drupal 4. The way to fix this so far was by using a contrib module called: Flat Comments The problem is that this module hasn't been ported to D8 yet.

We decided to fix that and port the module to help others with this problem. You can check the code here: https://github.com/agaric/flat_comments and we are in contact with the module's maintainer to create the D8 branch in drupal.org soon.

Expecting that this can help someone else.

Drupal's toolbar second level of menu options and dropdown not showing? Look for "Uncaught DOMException: The quota has been exceeded." errors, as viewable in the Firefox web console. If you see them, the problem is likely due to sites sharing a top-level domain—which is likely if you are using a local development environment like DDEV, and you working on way too many sites at once—combined with a pretty bad Firefox bug that will be fixed in the next release.

To quote Nathan Monfils:

1. Everything from your public domain (abc.tld) counts against your quota, even if it is in a seemingly unrelated subdomain (e.g. my-app.example.com and intranet.example.com).
2. The quota is not recomputed properly, requiring a firefox restart after clearing your data on other subdomains

Note this may affect all sorts of applications, not just Drupal, when you have them running on multiple subdomains of the same top-level domain. So this isn't just about local development environments (and i dislike that DDEV shares their own top-level domain across all the instances you are working on, and while it can be changed i've accepted its way of doing things so i'm on the same page with other developers by default).

Sure, closing more tabs and restarting Firefox could (predictably) have fixed this—and a lot else that's wrong with me, according to everyone i know—but why do that when i can open more tabs and learn precisely how broken everything around me really is?

Agaric is facilitating a full day training at DrupalCon Seattle to help you understand how to import content into your to Drupal 8 website.  It's the latest in our series of migration trainings.

This training is open for attendees with intermediate experience with Drupal– familiarity with installing a Drupal site and installing modules. We will use the Migrate API and related modules, which allows users to migrate content without writing any code.

With six instructors we will ensure no one gets left behind. Instead everyone will get the attention they need.

Attendees will learn to:

• Import data from CSV and JSON files.
• Transform data to populate taxonomy, date, image, file, and address fields.
• Get content into Paragraphs.
• Debug migration errors.

The training price is $450. Space is limited so we do encourage registering as soon as possible to ensure your spot.

As a web development cooperative that champions free software, we're passionate about migrations. It is a way to better understand Drupal's codebase, tap into the power of new features and build community. We have successfully migrated multiple sites to Drupal 8 and look forward to sharing our experience with you.

This training has sold out but please get in touch to be the first to know about future opportunities!