We believe in the freedom to move our data to whichever platforms best meets our needs. For that reason, we specialize in helping people migrate onto liberating software such as Drupal— and helping you learn how to do the same.
As the COVID-19 crisis continues, we will be announcing online trainings soon!
Join our trainings announcement list to be the first to get news of training opportunities:
In previous posts, we introduced the concept of defining migrations as configuration entities. These types of migrations can be executed from a user interface provided by the Migrate Tools module. In today’s article, we will present the workflow to import configuration entities and execute migrations from the user interface. Let’s get started.
Important: User interfaces tend to change. In fact, Migrate Tools recently launched a redesign of the user interface. Screenshots and referenced on-screen text might change over time. Some exposed settings might work, but give no feedback indicating a successful outcome. Some forms might only work with specific versions of modules. The goal is to demonstrate what is possible and what are the limitations of doing so. In general, executing migrations from the command line offers a better experience.
Getting the example code
You can get the full code example for today’s example at https://github.com/dinarcon/ud_migrations The module to use is UD configuration group migration (JSON source) whose machine name is ud_migrations_config_group_json_source. It comes with three migrations: udm_config_group_json_source_paragraph, udm_config_group_json_source_image, and udm_config_group_json_source_node. Additionally, the demo module provides the udm_config_group_json_source group.
You could install the module as we have done with every other example in the series. Instructions can be found in this article. When the module is installed, you could execute the migrations using Drush commands as explained in this article. But, we are going to use the user interface provided by Migrate Plus instead.
Importing configuration entities from the user interface
The first step is getting the configuration entities into Drupal’s active configuration. One way to do this is by using the "Single item" import from the "Configuration synchronization" interface. It can be found at /admin/config/development/configuration/single/import. Four configuration items will be imported in the following order:
- The
udm_config_group_json_sourceusing "Migration Group" as the configuration type. - The
udm_config_group_json_source_imageusing "Migration" as the configuration type. - The
udm_config_group_json_source_paragraphusing "Migration" as the configuration type. - The
udm_config_group_json_source_nodeusing "Migration" as the configuration type.
When importing configuration this way, you need to select the proper "Configuration type" from the dropdown. Otherwise, the system will make its best effort to produce a valid configuration entity from the YAML code you paste in the box. This can happen without a visual indication that you imported the wrong configuration type which can lead to hard to debug errors.
Note: If you do not see the "Migration" and "Migration group" configuration types, you need to enable the Migrate Plus module.
Another thing to pay close attention is changes in formatting when pasting code from a different source. Be it GitHub, your IDE, or an online tutorial, verify that your YAML syntax is valid and whitespaces are preserved. A single extraneous whitespace can break the whole migration. For invalid syntax, the best case scenario is the system rejecting the YAML definition. Otherwise, you might encounter hard to debug errors.
In today’s example, we are going to copy the YAML definitions from GitHub. You can find the source files to copy here. When viewing a file, it is recommended to click the "Raw" button to get the content of the file in plain text. Then, copy and paste the whole file in the designated box of the "Single item" import interface. Make sure to import the four configuration entities in the order listed above using the proper type.
To verify that this worked, you can use the "Single item" export interface located at /admin/config/development/configuration/single/export. Select the "Configuration type" and the "Configuration name" for each element that was imported. You might see some extra keys in the export. As long as the ones you manually imported are properly set, you should be good to continue.
Note that we included uuid keys in all the example migrations. As explained in this article, that is not required. Drupal would automatically create a uuid when the configuration is imported. But defining one simplifies the update process. If you needed to update any of those configurations, you can directly visit the "Single item" import interface and paste the new configuration. Otherwise, you would have to export it first, copy the uuid value, and add it to the code to import.
Executing configuration entities from the user interface
With the migration related configuration entities in Drupal’s active configuration, you should be able to execute them from the user interface provided by Migrate Tools. It is available in "Manage > Structure > Migration" at /admin/structure/migrate. You should see the UD Config Group (JSON source) that was imported in the previous step.
Note: If you do not see the "Migration" link in "Manage > Structure" interface, you need to enable the Migrate Tools module.
This user interface will list all the migration groups in the system. From there, you can get to the individual migrations. You can even inspect the migration definition from this interface. It is important to note that only migrations defined as configuration entities will appear in this interface. Migrations defined as code will not be listed.
For the udm_config_group_json_source group, click the "List migrations" button to display all the migrations in the group. Next, click the "Execute" button on the udm_config_group_json_source_image migration. Then, make sure "Import" is selected as the operation and click the "Execute" button. Drupal will perform the import operation for the image migration. A success status message will appear if things work as expected. You can also verify that images where imported by visiting the files listing page at /admin/content/files.
Repeat the same process for the udm_config_group_json_source_paragraph and udm_config_group_json_source_node migrations. The final result will be similar to the one from the JSON source example. For import operations, the system will check for migration dependencies and execute them advance if defined. That means that in this example, you could run the import operation directly on the node migration. Drupal will automatically execute the images and paragraphs migrations. Note that migration dependencies are only executed automatically for import operations. Dependent migrations will not be rolled back automatically if the main migration is rolled back individually.
This example includes a paragraph migrations. As explained in this article, if you rollback the node migration, any paragraph associated with the nodes will be deleted. The Migrate API will not be aware of this. To fix the issue and recover the paragraphs, you need to rollback the paragraph migration as well. Then you re-import the paragraph and node migrations again. Doing this from the user interface involves more manual steps compared to running a single Drush command in the terminal.
Limitations of the user interface
Although you can import and execute migrations from the user interface, this workflow comes with many limitations.The following is a list of some of the things that you should consider:
- Updating configuration in production environments is not recommended. This can be enforced using the Configuration Read-only mode module.
- If the imported configuration entity did not contain a
uuid, you need to export that configuration to get the auto generated value. This should be used in subsequent configuration import operations if updates to the YAML definition files are needed. Otherwise, you will get an error like "An entity with this machine name already exists but the import did not specify a UUID." - The following operations to not provide any user interface feedback if they succeeded: "Rollback", "Stop", and "Reset". See this issue for more context.
- As of this writing, most operations for CSV sources fail if using the 8.x-3.x branch of the Migrate Source CSV module. See this issue for more context.
- As of this writing, the user interface for renaming columns in CSV sources produces a fatal error if using the 8.x-3.x branch of the Migrate Source CSV module. See this issue for more context.
- As of this writing, it is not possible to execute all migrations in a group from the user interface in one operation. See this issue for more context.
- The Migrate source UI module can be used to upload a file to be used as source in CSV migrations. As of this writing, a similar feature for JSON and XML files is not working. See this issue for more context.
To reiterate, it is not recommended to use the user interface to add configuration related to migrations and execute them. The extra layer of abstractions can make it harder to debug problems with migrations if they arise. If possible, execute your migrations using the commands provided by Migrate Tools. Finally, it is recommended to read this article to learn more about the difference between managing migrations as code or configuration.
What did you learn in today’s blog post? Did you know you can import migration related configuration entities from the user interface? Did you know that Migrate Tools provides a user interface for running migrations? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.
Next: Understanding the entity_lookup and entity_generate process plugins from Migrate Tools
This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.
The Challenge
Increasingly, the entities and individuals who participate in such initiatives are geographically dispersed with limited opportunities for in-person or contemporaneous collaboration. They need spaces where they can work asynchronously—online communities that enable teams to share ideas and best practices, receive feedback from experts, and exchange resources. And each space or community needs to be their own. The required feature set is robust, but common—the perfect fit for a Drupal distribution. The NICHQ Collaboratory (“CoLab”) provided such spaces, but they were powered by a Drupal 7 distribution which made site-specific customization and ongoing maintenance fraught and unmanageable. Moreover, the list of improvements slated for implementation was increasing, and there was growing consensus that the information architecture for resources and other information needed rethinking.
Our Approach
Research-Driven Information Architecture
Anecdotally, we knew the sites could use an upgrade, but there were also features users appreciated. In our upgrade, we did not want to disrupt any beloved processes in our quest for improving the experience.
We started with a survey to find the common pain points and the loved features to leave intact. We learned that overall people still loved the visual design, felt the quality of content was high, and moderation was strong. Most respondents were overwhelmed with the amount of content, despite its quality, and didn't find the notification system helpful.
The next step was to dig deeper to learn why exactly content was hard to find and the best way to solve this. The survey asked if the respondent would be interested in a follow-up interview so we had a good list to draw from.
We knew from our prior audience work that there were two primary user groups: Project Managers and Participants. We decided to start by interviewing one project manager, a new participant and a veteran participant. After this first round of interviews we would then decide if more research was needed.
There were some surprising findings. One is that each community had groups within it, for more specific topics. However, none of the interviewees used these groups. In fact, they got in the way of collaboration. For them, collaboration meant transcending groups and working with anyone within the team. We also learned that categorization had run a bit wild. Different users were tagging resources in different ways and there were terms that served the same purpose, creating redundancy and inconsistent tagging. This was both a process challenge and a technical one. NICHQ would get on the same page about how to categorize content and we would lock down the ability to add new categories, only to project managers.
Atomic Wireframes
We do not build websites page by page, but rather in components that get reused across a site. Atomic design is a way to conceive of components made up of smaller components, made up of even smaller components.
To ease development and design, we built our wireframes using the same concept. In Sketch these are called symbols.
Wireframe to Design is Not Clean
It is tempting to punt all design questions until after wireframes are approved. However, sometimes it is too difficult for us to envision the functionality of something without more design polish added to it.
So, instead of pure, clean transitions we moved from wireframes to design knowing that some questions were still unresolved.
Luckily, our "atomic wireframes" were easy for our designer Todd Linkner to update.
The CoLab Feed is where collaborators stay informed of upcoming events and see the latest member activity.
Advancing the Power of Distributions to Drupal 8
As mentioned above, NICHQ's need for multiple sites using similar feature sets was the perfect fit for a Drupal distribution. However, we needed a workflow that could allow some sites to diverge from the original codebase but still bring in future improvements.
We had already started in on this endeavor with Drutopia, an initiative to improve the way Drupal manages configuration and distributions.
With the NICHQ CoLab as our practical use case, we worked with the Drupal shop Chocolate Lily to build and improve upon a suite of modules to manage and share different states of configuration across distributions.
For more, read this excellent series of posts by Nedjo Rogers.
Users can ask the community a question they have about children's health.
When one starts working with migrations, it is easy to be overwhelmed by so many modules providing migration functionality. Throughout the series, we presented many of them trying to cover one module at a time. We did this to help the reader understand when a particular module is truly needed and why. But we only scratched the surface. Today’s article presents a list of migration related Drupal modules, all good for Drupal 8, for quick reference. Let’s get started.
Core modules
At the time of this writing, Drupal core ships with four migration modules:
- Migrate: provides the base API for migrating data.
- Migrate Drupal: offers functionality to migrate from other Drupal installations. It serves as the foundation for upgrades from Drupal 6 and 7. It also supports reading configuration entities from Drupal 8 sites.
- Drupal Migrate UI: provides a user interface for upgrading a Drupal 6 or 7 site to Drupal 8.
- Migrate Drupal Multilingual: is an experimental module required by multilingual translations. When they become stable, the module will be removed from Drupal core. See this article for more information.
Migration runners
Once the migration definition files have been created, there are many options to execute them:
- Migrate Tools: provides Drush commands to run migrations from the command line. It also exposes a user interface to run migrations created as configuration entities. It offers support for migration groups and tags. The module depends on Migrate Plus.
- Migrate Run: provides Drush commands to run migrations from the command line. It does not offer support for migration groups, but tags are supported. The module does not depend on Migrate Plus.
- Migrate Upgrade: provides Drush support for upgrading a Drupal 6 or 7 site to Drupal 8.
- Migrate Manifest: provides a Drush command for running migrations using a manifest file. See this article for an example of using this module for Drupal upgrades.
- Migrate Scheduler: integrates with Drupal Core’s Cron API to execute migrations on predefined schedules.
- Migrate Cron: exposes a user interface to execute migrations when Cron is triggered. At the time of this writing, the module does not execute dependent migrations. Follow this issue for updates.
- Migrate source UI: provides a form for uploading files to use as source for already defined CSV, JSON, and XML migrations. At the time of this writing, it seems that JSON and XML migrations are not being detected. Follow this issue for updates.
Source plugins
The Migrate API offers many options to fetch data from:
- Migrate (core module): provides the `SqlBase` abstract class to help with fetching data from a database connection. See this article for an example. It also exposes the `embedded_data` plugin which allows the source data to be defined inside the migration definition file. It was used extensively in the example migrations of this series. It also offers the `empty` plugin which returns a row based on provided constants. It is used in multilingual migrations for entity references.
- Migrate Plus: combining various plugins, it allows fetching data in JSON, XML, and SOAP formats. It also provides various plugins for parsing HTML. See this article by Benji Fisher for an example. There is also a patch to add support for PDF parsing.
- Migrate Source CSV: allows fetching data from CSV files.
- Migrate Google Sheets: leverages Migrate Plus functionality to allow fetching data from Google Sheets.
- Migrate Spreadsheet: allows fetching data from Microsoft Excel and LibreOffice Calc files.
- Migrate Source YAML: allows fetching data from YAML files.
- WP Migrate: allows fetching data from a WordPress database.
Destination plugins
The Migrate API is mostly used to move data into Drupal, but it is possible to write to other destinations:
- Migrate (core): provides classes for creating content and configuration entities. It also offers the `null` plugin which in itself does not write to anything. It is used in multilingual migrations for entity references.
- Migrate Plus: provides the `table` plugin for migrating into tables not registered with Drupal Schema API.
- CSV file: example destination plugin implementation to write CSV files. The module was created by J Franks for a DrupalCamp presentation. Check out the repository and video recording.
Development related
These modules can help with writing Drupal migrations:
- Migrate (core): provides the `log` process plugin. See this article for an example of its use.
- Migrate Devel: offers Drush options for printing debug information when executing migrations. It also provides the `debug` process plugin. See this article for an example of its use.
- Migrate Process Vardump: provides the `vardump` plugin. It works like the `debug` plugin.
Field and module related
- Migrate Media Handler: provides migration process plugins to facilitate the migration into Drupal 8 media entities. The source can be Drupal 7 file or image fields. It also supports inline file embeds in rich text. It leverages the DOM parsing plugins provided by Migrate Plus.
- Media Migration: provides an upgrade path from Drupal 7 to Drupal 8 media entities. The source can be image fields and fields attached to media and file entities.
- Migrate File Entities to Media Entities: migrates Drupal 8.0 file entities to Drupal 8.5 media entities.
- Migrate Files: provides process plugins for migrating files and images.
- Webform Migrate: provides a plugin to help migrating from the Drupal 6 and 7 versions of the Webform module.
- Migrate HTML to Paragraphs: turns HTML markup into paragraph entities.
- Commerce Migrate: offers a general purpose migration framework for bringing store information into Drupal Commerce.
- Address: offers a process plugin to migrate data into fields of type address. It also provides an upgrade path from Drupal 7’s Address Field module. See this article for an example.
- Geofield: offers a process plugin to migrate data into fields of type geofield. See this article for an example.
- Office Hours: offers a process plugin to migrate data into fields of type office hours.
- Workbench Moderation to Content Moderation: migrates configuration from one module to the other.
Modules created by Tess Flynn (socketwench)
While doing the research for this article, we found many useful modules created by Tess Flynn (socketwench). She is a fantastic presenter who also has written about Drupal migrations, testing, and much more. Here are some of her modules:
- Migrate Directory: imports files from a directory into Drupal as managed files.
- Migrate Process S3: downloads objects from an S3 bucket into Drupal.
- Migrate Process URL: provides a process plugin to make it easier to migrate into link fields.
- Migrate Process Vardump: helps with debugging migrations.
- Many process plugins that wrap PHP functions. For example: Migrate Process Array, Migrate Process Trim, Migrate Process Regex, Migrate Process Skip, and Migrate Process XML.
Miscellaneous
- Feeds Migrate: it aims to provide a user interface similar to the one from Drupal 7’s Feeds module, but working on top of Drupal 8’s Migrate API.
- Migrate Override: allows flagging fields in a content entity so they can be manually changed by side editors without being overridden in a subsequent migration.
- Migrate Status: checks if migrations are currently running.
- Migrate QA: provides tools for validating content migrations. See this presentation for more details.
What did you learn in today’s blog post? Did you find a new module that could be useful in current or future projects? Did we miss a module that has been very useful to you? Share your answers in the comments. Also, I would be grateful if you shared this blog post with others.
Next: Introduction to Drupal 8 upgrades
This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.
Thank you for being awesome! Use this page to register with your discount. If you want to attend more than one training, fill out the form once per training.
Ben and Micky spoke at DrupalCon Seattle 2019 and New England Drupal Camp 2019 about scaling community conversations and decision-making. Rather than having slides of fine print and links, we're posting resources here in this blog post.
Here's a short-enough-to-write-on-a-business-card link for this page – agaric.coop/scaling-community – for easier sharing these resources with others.
350.org is a global climate justice organization that helped organize the largest climate strike in history. Together we improved their Climate Justice Action Map (CJAM) in the lead up to the strike so that activists could better mobilize their communities.
Punctuating Lists
If an article has multiple authors (using a user reference field or a content reference field), Drupal core displays each on their own line.
Like this ugly, rigid presentation:
Notas:
Diapositivas - https://github.com/fiqus/FIT-talk-en
## FACTTIC - Federación Argentina de Cooperativas de Trabajo de Tecnología, Innovación y Conocimiento
* Lista de correo sólo para miembros
* Mattermost (chat de código abierto)
* Reunión mensual de la junta virtual (cualquier miembro puede asistir)
* Reuniones anuales cara a cara
## FIT
* Un proyecto dentro de FACTTIC donde las cooperativas comparten el estado de los proyectos en los que están trabajando
* Evolucionado a un área donde las cooperativas comparten proyectos
* Para unirse a la FIT, tienes que ser un miembro FACTTIC
* Reuniones virtuales mensuales
* Canal Mattermost para el diálogo en curso
* Las cooperativas tienen diferentes habilidades/servicios, pero cuando hay superposición, trata de no competir entre sí y determina las "necesidades" de cada cooperativa.
### Escenario 1: El proyecto demanda más trabajadores de los que tiene la cooperativa
* Cuando hay una necesidad de ayuda, el proyecto se comparte en FIT
* Las cooperativas pueden solicitar unirse al proyecto
* Los candidatos son evaluados y uno es elegido
* El cliente es informado y debe estar de acuerdo
* La coordinación del proyecto está dirigida por la cooperativa inicial
* El acuerdo comercial es manejado sólo por la cooperativa inicial
### Escenario 2: El cliente necesita que se haga el trabajo, pero Coop decide no tomarlo
* Esto podría suceder porque la cooperativa inicial no tiene los recursos o declina por una razón estratégica
* El proyecto se comparte con la FIT
* Si no hay cooperativas interesadas, se le dice al cliente que no hay nadie disponible.
* Si una cooperativa está interesada, el contacto de esa cooperativa se comparte con el cliente.
* Si más de una cooperativa está interesada, entonces preguntamos, ¿este proyecto requiere más de un trabajador?
* Si sólo necesita un trabajador, entonces la cooperativa que más lo necesita lo recibe.
* Si requiere más de un trabajador, entonces las cooperativas se coordinan entre sí para completar el trabajo.
## Case Studies
### Betterez
* Cliente canadiense
* Plataforma de gestión de reservas y billetes
* Tecnologías: MongoDB, NodeJS, VueJS y Elixir
* Necesitaba más trabajo del que Fiqus podía proporcionar
* 30 desarrolladores con 7 cooperativas diferentes
* Fiqus maneja las cuestiones financieras como las diferentes tarifas de los diferentes servicios
### Receptivi
* Cliente canadiense
* La página web muestra en tiempo real las percepciones psicológicas del personal
* Tenía más trabajo, pero Fiqus se negó a asumir el trabajo
* El trabajo se compartió en la FIT
* 3 desarrolladores de 2 cooperativas
### Mall Plaza
* Cliente chileno
* Aplicación móvil que muestra los servicios del centro comercial
*Técnica: Reaccionar Nativo, PostgreSQL, Frasco
## Onapsis
* Cliente argentino
* Sistema web que muestra alertas de vulnerabilidad en los servidores
* 2 cooperativas
## FIT Internacional
* Queremos replicar este modelo a nivel internacional.
1. Compartir este modelo con otros a fin de mejorarlo y difundir la conciencia
* Presentando en el Show and Tell
2. Construir relaciones de confianza
3. Conocerse en persona, pasar tiempo juntos
* Viajando al Reino Unido, reunión con la federación, COTECH
* Compartir experiencias después del viaje
## Preguntas y respuestas:
P: ¿Alguna vez ha tenido la resistencia de un cliente al entregar el trabajo a otra cooperativa?
R: Hay veces que los clientes no entienden las cooperativas y la cooperación entre ellas. Explicamos el beneficio y compartimos estudios de casos. Si hay una fecha límite que debe cumplirse, es más rápido traer un equipo con experiencia previa trabajando con la empresa original que tratar de encontrar una empresa completamente diferente.
P: ¿Cuánto comparten acerca de las múltiples cooperativas que trabajan en un proyecto?
R: Si son sólo unas pocas horas, no vale la pena hablar de ello. Sin embargo, la mayoría de las veces es importante compartir esa información y usarla como una oportunidad educativa para demostrar la fuerza de las cooperativas trabajando juntas.
* Una vez que los clientes ven el resultado de la cooperación se dan cuenta de que es una buena manera de enfocar el trabajo.
* La simplicidad del proceso es hermosa.
P: ¿Cómo se comparten los costos de desarrollo de los negocios?
R: La cooperativa que comparte el proyecto puede bajar sus tarifas durante el proceso de aceleración.
* Este es un aspecto que podría mejorarse.
* Lo más importante es ser transparente y comunicar mucho.
* Mantener el espíritu de generosidad fluyendo.
* Cuando la cooperación es exitosa, se construye la confianza con el cliente.
* Usar una herramienta para analizar los presupuestos y el progreso de los proyectos y prever la disponibilidad.
* La cooperación también asegura la calidad, los trabajadores de confianza se unirán al proyecto.
As big tech continues to violate our privacy, thwart worker power, collude with militaries and deportation agencies, we need to move to ethical tech tools we can trust.
However, it can be intimidating to make the switch to new, unfamiliar tools.
Micky is leading two workshops to help people make the switch.
“How to Switch to Free Software: Email” 4:45-6:45pm Wednesday, December 4
Micky Metts of Boston’s Agaric Tech Cooperative will help us get out of the fix. The key to regaining security and control of the internet is switching to free software equivalents to proprietary/corporate software for communicating – among many uses.
This first workshop focuses on communications using email and texting software. The moderate inconvenience of switching is worth the privacy of strong encryption and the freedom of a new world of user-controlled communications and organizing. This first workshop in Sala Quetzal focuses on email and texting with encrypted software under user control. Bring your phone.
“How to Switch to Free Software: Browsers Etc” 2-4pm Thursday, December 5
A professional software developer, Micky, will focus on this second workshop, also in Sala Quetzal, on browsing, searching, and virtual meeting software. She will show how free software opens up the world of “platform co-ops” – think of Uber under drivers’ control and AirBnB under homeowners’ control.
Micky will be assisted by at-the-elbow translator/helpers. Bring your laptop and your phone.
