In the previous entry, we learned that the Migrate API is an implementation of an ETL framework. We also talked about the steps involved in writing and running migrations. Now, let’s write our first Drupal migration. We are going to start with a very basic example: creating nodes out of hardcoded data. For this, we assume a Drupal installation using the standard installation profile, which comes with the Basic Page content type. As we progress through the series, the migrations will become more complete and more complex. Ideally, only one concept will be introduced at a time. When that is not possible, we will explain how different parts work together. The focus of today's lesson is learning the structure of a migration definition file and how to run it.

Writing the migration definition file

The migration definition file needs to live in a module. So, let’s create a custom one named ud_migrations_first and set Drupal core’s migrate module as dependencies in the *.info.yml file, ud_migrations_first.info.yml.  The contents of the file will be:

type: module
name: UD First Migration
description: 'Example of basic Drupal migration. Learn more at https://understanddrupal.com/migrations.'
package: Understand Drupal
core: 8.x
dependencies:
  - drupal:migrate

Now, let’s create a folder called migrations and inside it, a YAML (Yet Another Markup Language) configuration file called udm_first.yml. Note that the extension is, yml not yaml. The contents of the file will be:

id: udm_first
label: 'UD First migration'
source:
  plugin: embedded_data
  data_rows:
    -
      unique_id: 1
      creative_title: 'The versatility of Drupal fields'
      engaging_content: 'Fields are Drupal''s atomic data storage mechanism...'
    -
      unique_id: 2
      creative_title: 'What is a view in Drupal? How do they work?'
      engaging_content: 'In Drupal, a view is a listing of information. It can a list of nodes, users, comments, taxonomy terms, files, etc...'
  ids:
    unique_id:
      type: integer
process:
  title: creative_title
  body: engaging_content
destination:
  plugin: 'entity:node'
  default_bundle: page

The final folder structure will look like:

.
|-- core
|-- index.php
|-- modules
|   `-- custom
|       `-- ud_migrations
|           `-- ud_migrations_first
|               |-- migrations
|               |   `-- udm_first.yml
|               `-- ud_migrations_first.info.yml

YAML is a key-value format with optional nesting of elements. They are very sensitive to white spaces and indentation. For example, they require at least one space character after the colon symbol (:) that separates the key from the value. Also, note that each level in the hierarchy is indented by two spaces exactly. A common source of errors when writing migrations is improper spacing or indentation of the YAML files.

A quick glimpse at the migration configuration file reveals the three major parts: source, process, and destination. Other keys provide extra information about the migration. There are more keys that the ones shown above. For example, it is possible to define dependencies among migrations. Another option is to tag migrations so they can be executed together. We are going to learn more about these options in future entries.

Let’s review each key-value pair in the file. For, id it is customary to set its value to match the filename containing the migration definition, but without the .yml extension. This key serves as an internal identifier that Drupal and the Migrate API use to execute and keep track of the migration. The id value should be alphanumeric characters, optionally using underscores to separate words. As for the label key, it is a human readable string used to name the migration in various interfaces.

In this example, we are using the embedded_data source plugin. It allows you to define the data to migrate right inside the definition file. To configure it, you define a data_rows key whose value is an array of all the elements you want to migrate. Each element might contain an arbitrary number of key-value pairs representing “columns” of data to be imported.

A common use case for the embedded_data plugin is testing of the Migrate API itself. Another valid one is to create default content when the data is known in advance. I often present introduction to Drupal workshops. To save time, I use this plugin to create nodes which are later used in the views creation explanation. Check this repository for an example of this. Note that it uses a different directory structure to define the migrations. That will be explained in future blog posts.

For the destination, we are using the entity:node plugin which allows you to create nodes of any content type. The default_bundle key indicates that all nodes to be created will be of type “Basic page”, by default. It is important to note that the value of the default_bundle key is the machine name of the content type. You can find it at /admin/structure/types/manage/page In general, the Migrate API uses machine names for the values. As we explore the system, we will point out when they are used and where to find the right ones.

In the process section you map columns from the source to node properties and fields. The keys are entity property names or the field machine names. In this case, we are setting values for the title of the node and its body field. You can find the field machine names in the content type configuration page: /admin/structure/types/manage/page/fields. Values can be copied directly from the source or transformed via process plugins. This example makes a verbatim copy of the values from the source to the destination. The column names in the source are not required to match the destination property or field name. In this example, they are purposely different to make them easier to identify.

You can download the example code from https://github.com/dinarcon/ud_migrations The example above is actually in a submodule in that repository. The same repository will be used for many examples throughout the series. Download the whole repository into the ./modules/custom directory of the Drupal installation and enable the “UD First Migration” module.

Running the migration

Let’s use Drush to run the migrations with the commands provided by Migrate Run. Open a terminal, switch directories to Drupal’s webroot, and execute the following commands.

$ drush pm:enable -y migrate migrate_run ud_migrations_first
$ drush migrate:status
$ drush migrate:import udm_first

The first command enables the core migrate module, the runner, and the custom module holding the migration definition file. The second command shows a list of all migrations available in the system. Only one should be listed with the migration ID udm_first. The third command executes the migration. If all goes well, you can visit the content overview page at /admin/content and see two basic pages created. Congratulations, you have successfully run your first Drupal migration!!!

Or maybe not? Drupal migrations can fail in many ways, and sometimes the error messages are not very descriptive. In upcoming blog posts, we will talk about recommended workflows and strategies for debugging migrations. For now, let’s mention a couple of things that could go wrong with this example. If after running the drush migrate:status command, you do not see the udm_first migration, make sure that the ud_migrations_first module is enabled. If it is enabled, and you do not see it, rebuild the cache by running drush cache:rebuild.

If you see the migration, but you get a yaml parse error when running the migrate:import command, check your indentation. Copying and pasting from GitHub to your IDE/editor might change the spacing. An extraneous space can break the whole migration so pay close attention. If the command reports that it created the nodes, but you get a fatal error when trying to view one, it is because the content type was not set properly. Remember that the machine name of the “Basic page” content type is page, not basic_page. This error cannot be fixed from the administration interface. What you have to do is rollback the migration issuing the following command: drush migrate:rollback udm_first, then fix the default_bundle value, rebuild the cache, and import again.

Note: Migrate Tools could be used for running the migration. This module depends on Migrate Plus. For now, let’s keep module dependencies to a minimum to focus on core Migrate functionality. Also, skipping them demonstrates that these modules, although quite useful, are not hard requirements for running migration projects. If you decide to use Migrate Tools make sure to uninstall Migrate Run. Both provide the same Drush commands and conflict with each other if the two are enabled.

What did you learn in today’s blog post? Did you know that Migrate Plus and Migrate Tools are not hard requirements for Drupal migrations projects? Did you know you can place your YAML files in a migrations directory? What advice would you give to someone writing their first migration? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your friends and colleagues.

Next: Using process plugins for data transformation in Drupal migrations

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors: Drupalize.me by Osio Labs has online tutorials about migrations, among other topics, and Agaric provides migration trainings, among other services.  Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

I am very happy the bug is fixed and this blog post will be obsolete in mere days! Usually this sort of technical noodlings get relegated to our raw notes, currently hosted through GitLab, but figured at least a few other Drupal developers would want to know what has been going on with their toolbars.

Image credit: "Too Many Tabs" by John Markos O'Neill is licensed with CC BY-SA 2.0.

.

Agaric builds tools for medical and scientific communities to advance their work, enhance collaboration, and improve outcomes.  And we've been doing this—helping healthy discussion about science and medicine flourish online—since 2008.

Reusable platforms for advanced online collaboration

Therapy Fidelity App

Agaric is developing the Therapy Fidelity app: an all-in-one, inexpensive mobile web application to help therapists do the work of counseling. The app automates surveys, handles multiple CBT protocols, tracks fidelity, monitors outcomes, and more. This application is being developed for Scheeringa Mind Company with initial funding by Tulane University.  It is built in JavaScript (React and Typescript) and Golang and makes extensive use of Truevault and AWS APIs.

The Collaboratory

The National Institute for Children's Health Quality partnered with Agaric to build the Collaboratory—a platform designed specifically to help healthcare improvement teams collaborate, innovate, and make change. During this partnership, begun in 2015, Agaric built a collaborative analytics tool that allows healthcare quality teams to visualize, compare, and benchmark data, identify opportunities for improvement, and celebrate their successes. We were proud to be NICHQ's 2020 partners in making the most of the digital health revolution.

Platform for Experimental Collaborative Ethnography (PECE)

In 2015, we began contributing to PECE, an open source digital platform that supports multi-sited, cross-scale ethnographic and historical research.  PECE is built as a Drupal distribution that can be improved and extended like any other Drupal project.  Agaric's contributions include building an API integration between PECE's bibliographic citation capabilities and Zotero's open source reference management and collaborative bibliography tools.

We have been brought back for a larger role to realize the full upgrade of this distribution and platform to Drupal 10.

Partners In Health Drug Resistant Tuberculosis Network and EndTB.org

Beginning in 2010, Agaric took over the development of the DRTB Network platform for Partners In Health, the famed international nonprofit public health organization, and the TB Care II initiative.  The core of this work was connecting practitioners in the field with experts through a natural yet structured response process complete with careful editorial review.  This crucial work lives on with endTB.org, a partnership between Partners In Health, Médecins Sans Frontières, and Interactive Research & Development.  All of this work for PIH is in Drupal.

Science Collaboration Framework

Agaric was the lead developer for the Science Collaboration Framework, a project of Harvard's Initiative in Innovative Computing. Working with researchers from Harvard University and Massachusetts General Hospital, we built a reusable platform for collaboration and communication in biomedical research, enriching the contributions of scientists and the biomedical online community with semantic data, highlighting advanced, structured relationships between contributed resources, and facilitating structured community discourse around biomedical research. We even earned a writeup in a scientific journal for our work!

As part of SCF, Agaric led the work of building the website for an online community of Parkinson's disease researchers and research investors, on the Science Collaboration Framework, for the Michael J. Fox Foundation for Parkinson's Research.

Ask Agaric for help building your world-bettering community today!

We have a number of processes and tools we use as we develop:

• Local development: we all develop locally on our own computers and push each change to a common code repository which we maintain.

• Each task including new features, bugs or changes are kept track of as an issue ticket. These tickets will be accessible to Urban Edge staff as well as ourselves. Here is where we can track progress in code changes, elaborate on details and generally where we keep track of the hundreds of small issues that will come up.

• Resources allowing, we use automated tests which emulate a human's interaction with a browser to make sure new work we does not unknowingly break site functionality. This ensures a solidly functioning site even as major changes take place.

• We have experience managing a wide range of servers, server environments, or specialized hosting environments, and related technologies:

  • We have written our own battle-tested server deployment code, which securely pushes changes in code to any target location.
  • Deployments can run manually or to a desired level of automation, such as always deploying to a staging website automatically on a successful build, and then manually progressing to live after review.
  • Our existing deployment tools can handle configuring a virtual (or physical) server from the ground up, or else function within a variety of PHP-based website hosting providers.
  • Additionally, we can support container-based environments by providing docker container configurations for running the required services.
  • Dependent upon the hosting platform, we also manage additional security and reliability features, such as sites having automated system updates, virus scans of uploaded files, custom backup schedules, additional firewall protections, monitoring services, or other customized needs.
  • We can also work alongside existing IT service providers with recommendations regarding things you may already manage, such as CDN (content delivery networks), email, or DNS (domain name) changes that may be needed.

Read more about our tools for environment management and development.

Welcome to Drutopia! We hope you are enjoying the features we have built. Everything you are using is open-source and free for good people like yourself to use.

We invite you to give your input on the project and contribute where you can by becoming a member of Drutopia. For as little as $10 a year you can become a member who votes for our leadership team, suggests features for our roadmap and is part of a community building tools for the grassroots.

Learn more about membership at drutopia.org

Blog posts can be a valuable way for a variety of authors to post reflections, experiences and opinions.

At a minimum every blog post should have an image (which will be associated with the blog and display in different sizes in different displays), a summary field and a text paragraph in the body field.

All Drutopia content types are built using the Paragraphs module which lets you add a variety of types of content and arrange them in the order you wish. When creating a blog post you can add standard text paragraphs in the body paragraph field, but also add images or files (such as a PDF).

You can define your own permissions for the Drupal permissions page (/admin/people/permissions in modern Drupal, Drupal 8, 9, 10, and beyond) and then add conditional options to your code to do different things based on the role of the user and the permissions configured by a site administrator.

Here's how.

Create a modulename.permissions.yml file

This simple file has the permission machine name (lower case with spaces) and a title (Sentence case) with an optional description.

For our module, which has a particularly long name, that file is drutopia_findit_site_management.permissions.yml and its contents are like so:

access meta tab:
  title: 'Access meta tab'
  description: 'Access meta information (author, creation date, boost information) in Meta vertical tab.'

You can repeat lines like these in the same file for as many permissions as you wish to define.

Check for that permission in your code

The process for checking permissions is simply to use a user object if that's handed into your code, or to load the current user if it's not, and use the hasPermission() method which returns TRUE if that user has permission and FALSE if not.

For example, in a form alter in our drutopia_findit_site_management.module file:

/**
 * Implements hook_form_BASE_FORM_ID_alter() for node_form.
 *
 * Completely hide the Meta vertical tab (field group) from people without permission.
 *
 */
function drutopia_findit_site_management_form_node_form_alter(&$form, FormStateInterface $form_state, $form_id) {
  // If the current user has the permission, do not hide the Meta vertical tab.
  if (\Drupal::currentUser()->hasPermission('access meta tab')) {
    return;
  }
  // Code to hide the meta tab goes here, and is only reached if the user lacks the permission. 
  // ...
}

See all this code in context in the Find It Site Management module.

To learn more about defining permissions in modern Drupal, including dynamic permissions, you can see the change record for when the new approach replaced hook_permission().

On November 13^th and 14^th in New York City, several hundred people gathered to talk about the problems of an online economy reliant on monopoly, extraction, and surveillance—and discuss how to build a "cooperative Internet, built of platforms owned and governed by the people who rely on them."

My experience at the Platform Cooperativism summit was Wow, everyone here really gets it and so many are doing awesome things; and then Hmm, there are still some really important differences to be worked out; and then We'll have to continue for months to figure out strategy for building fair platforms and we also need to restructure the whole economy.

Some definitions

In the sense technologists use it a platform is, like a physical platform, a technology that holds a lot of people up. It convenes people and gives them a chance to do something they wouldn't otherwise be able to do. Platforms can often be natural monopolies due to capturing the benefits of network effects (one person with a telephone is pointless, having nearly everyone available by telephone is incredibly valuable). Amazon and eBay are both platforms for sellers and buyers, Uber and Lyft for drivers and riders, Mechanical Turk and TaskRabbit for piece-workers and buyers of their work.

A cooperative is a jointly owned and democratically-controlled enterprise formed by people voluntarily uniting to meet their common needs and aspirations. Agaric is a small worker-owned cooperative, Mondragon is a very large group of integrated worker cooperatives, consumer cooperatives are businesses owned by their customers, credit unions are financial institutions owned by their members (with a one person, one vote governance), and producer cooperatives like CROPP Cooperative are formed by member businesses (which are not necessarily cooperatives themselves).

A platform cooperative, then, is a platform owned and controlled by the people directly affected by it. A company must be accountable, and as Omar Freilla put it, accountable means those impacted make the decisions.

The power to do harm

This summit was a follow-up to the Digital Labor summit held one year before which detailed myriad ways centralized online platforms extract value from dispersed workers who have few options or bargaining power. Control of online platforms by the representatives of capital has or will have negative effects on workers, similar to exploitation in global manufacturing (think electronic devices and clothing), and negative effects on customers (think the massive money grab by oligopolies of fossil fuel and telecommunications corporations).

Agaric's Michele Metts told the Digital Labor summit organizers every chance she got that cooperatives and Free Software were the answer to exploited labor in the Internet economy, but something even more powerful than Micky's advocacy must have been at work: nearly every participant at Platform Cooperativism spoke of the need for workers to own the platforms that control their work, and people presenting on technology took for granted that source code and algorithms have to be open for democratic control to be meaningful. As Micky said on her panel, "You cannot build a platform for freedom on someone else's slavery."

The opening presentations made the case that platforms will exploit us unless we take control, and we moved on to discussing strategies for building platform businesses that are cooperatives of the people using the platforms. We also celebrated those already starting, like Loconomics, Fairmondo (in Germany), and Member's Media.

Building for shared power and uplift

The biggest unsolved, but acknowledged, problem is getting the resources to build platforms that can compete with venture capital-funded platforms. Dmytri Kleiner made the claim that profit requires centralization, and, moreover, that centralization requires organizing along the lines of a profit-taking venture. How can people get the resources to build without both having to give up control and having to exploit people using the platform? Robin Chase reminded us that it costs millions of dollars, at least, to build a viable platform. Her solution is to continue to seek venture capital and work for some environmental or community goals while compromising on control.

A more popular possible solution is to replace centralized systems with decentralized ones, even to the point of replacing specific software with protocols, so the cost of building and operating platforms can be more widely shared, along with the benefits. However, as Astra Taylor summed up the widely felt point, decentralization does not always mean distributed power. Therefore control of technology decisions, and so democratic control of platforms, is more important than technology itself.

The potential positive role for government regulation was often mentioned, as Sarah Ann Lewis summarized the sentiment in a tweet: Platforms are not special snowflakes that must be exempt from regulation. If you can only succeed by exploitation you deserve to melt. Indeed, the centralized and surveillance nature of most platforms would make it much easier to ensure non-discrimination and fair wages.

More excitement came from the mention that local government has long played a role and can play a stronger part in democratic ownership of physical spaces. Several speakers urged people to get involved in local government, where harmful policies may be more the result of a lack of knowledge than of embedded corruption. Government can also get involved in mandating an open API for ride hailing services, which would remove the monopoly power from centralizing companies.

On ownership and control

Hundreds of possible solutions faced lively questioning and debate, yet in all of this the titular solution, cooperative ownership, did not get the scrutiny it merits. Jessica Gordon Nembhard's Collective Courage has made me see that the connections and overlaps between worker cooperatives and other types of cooperatives are much more significant than I'd thought, but there are still differences. These differences, and the need to decide who exactly is democratically controlling a platform, were often not made clear by presenters, including some who are building platform cooperatives.

If Stocksy, for example, is owned by its photographers, can the workers who build the platform technology (rather than use it) play a part in democratic control? Co-founder and CEO Brianna Wettlaufer refers to it as a multi-stakeholder cooperative and it has been around since 2012 so they've surely worked it out, but this question is at the heart of how platform cooperatives must operate and it was hardly addressed at all.

The answer can be simple. The Black Star Coop brewery and restaurant in Austin, Texas, is owned by its customer-members while the workers manage it. The workers are internally a democracy, but there's no question they work for a businesses which is managed democratically by the customers. This makes even more sense for a quasi-monopoly platform: It's more important for, say, millions of people relying on a platform for livelihood or transportation or communication to own it than for the relatively small number of people who built it to own it.

This brings up another question that went largely unasked at the conference: does ownership mean anything when it's spread out among thousands or millions of people? Federated structures can mitigate this, but in general whoever controls communication among members effectively controls decisions. It may be possible to have horizontal mass communication by way of democratic moderation. At a small workshop I held at the conference, participants discussed ways collective control can be made real as democratic platforms scale—but that's a topic for another discussion.

The sense that displacing an app or website is easier than reconstructing global supply chains fueled a lot of the excitement at the conference. Notwithstanding, the need to restructure the rest of the economy so that it works to serve the needs of people, rather than sacrificing people's needs to the dictates of the economy, was never far from people's minds. Videos of most sessions are online and will certainly make you think about the opportunities for cooperative ownership of services and structures that define our lives, online and off.

Thank you for requesting that Micky Metts speak at your event. She will respond as soon as she can and let you know if she is available, or if she has any questions.

Agaric Team