Get notified of trainings

Learn when we have new opportunities for learning (two to four announcements a year).

IV. Movements and Networks

Free Software Movement

Free/Libre software is software that asserts the 4 fundamental software freedoms to every user. From Drupal modules to entire platforms, everything we build is Free Software because we believe that individuals and communities alike should have as much ownership as possible over the software they use. We even advise our clients in our blog on how to perform daily business operations using Free Software. Agaric's are members and supporters of the Free Software Foundation.

Design Justice Network

We design sites according to the principles of Design Justice. This means that the user experience of marginalized individuals is central to our design considerations. As a part of this network we share ideas and learn techniques to provide equitable online experiences for all. You may not know what it is like to be left out, until you are. 

MayFirst Movement Technology

MayFirst Movement Technology (MFMT) is a cooperative, of movement organizations and activists, that provides web hosting, email services and other free software tools to its members. MFMT has a bold and progressive mission and agrees not turn over your data to third parties under any circumstances.

As members of MFMT, we support the organization by raising awareness on ethical hosting and free software community standards. We encourage people and organizations to join and host their websites through MFMT and gain access to a suite of free software tools and services. Micky is an active board member, involved in discussions of governance as well as the free software tools and platforms that MayFirst offers. Your web hosting does matter, and so does your voice!

US Federation of Worker-Owned Cooperatives

USFWC is a non-profit 501c3 that is active in supporting worker-owned cooperatives while building the solidarity economy together. Currently we are building deeper ties with other tech coops within the USFWC to help advance economic justice in the tech industry as we build technology and support social justice movements, enhance trust and work with them. The federation works closely with DAWI - Democracy at Work Institute to educate and support members having a voice in their workplace.

U.S. Solidarity Economy Network

The USSEN is dedicated to exploring equitable and ethical economies that benefit community members and can interact on a larger scale to define methods and processes for promoting the growth and sustainability of each community as it relates to the world around it. As an organization, we are a member of the North American chapter of RIPESS which is part of an international organization that is an umbrella for groups like USSEN, worldwide. We also create Resist and Build  workshops that support the Solidarity movement.

Tech Coop Network

The Tech Coop Network was formed at a USFWC conference in Los Angeles in 2017 by members of several worker-owned tech and web development cooperatives. We are currently discussing bylaws and governance to prepare for inviting other cooperatives to join us. We meet monthly and are establishing our communications infrastructure.

International Workers of the World

Some Agaric members have joined the Boston Chapter of the IWW. This chapter has a horrible and ignoble past that the current members are determined to overcome. Current members are committed to rebuilding this branch to exemplify the very causes it was based upon before it was destroyed from within. We stand with marginalized people and communities. We represent the disenfranchised worker and support non-conforming individuals rights. An injury to one is an injury to all!

The Digital Fourth

As members of the Boston Chapter, we stay informed on new laws and changes to current laws that could effect our online presence as well as our real life freedoms. Agaric enjoys keeping up with the digital legal strategies and will voice our opinion on whether digital citizens will be affected negatively or positively by pending legislation in our state and sometimes nationally. Artificial intelligence and machine learning are two complex areas of technology and they need to be addressed by groups like us, with diverse knowledge. Surveillance by the Government is also a priority and we publish our findings and recommendations. We cannot change the things that we do not know about. D4th is a part of the nationwide coalition Restore The Fourth.

Science for the People

Scientists and citizens convene to discuss and create ways for the general population to get involved in all things scientific. We believe that you should not need a degree to have access to people in fields of scientific study. We work to remove barriers and make connections between scientists and average citizens people with good ideas and questions that could benefit scientific understanding. Agaric is very interested in  connecting people, projects and promoting discussions in this group. Science for the People has online meetings and is open to all.

As it was for much of the world, 2018 was a combination of extremes for Agaric and the free and open web. Happily, we expanded our team, launched new sites, and empowered our clients through libre software. Unhappily, many of us and our communities endured health issues, political instability, and the effects of climate change.

For the open web, we disappointedly saw the United States officially end Net Neutrality while we excitedly watched the European Union begin enforcing comprehensive privacy laws with its General Data Protection Regulation. We were disgusted by tech giants like Facebook and Palantir diverting and deflecting from the abuses they carry out, but we were also inspired by workers at companies like Amazon and Google forcing their bosses to do better.

In looking back, we celebrate the victories and learn from the challenges—with our eyes set on serving our clients better, expanding the open web, and building an economy based on solidarity rather than exploitation.

To that end, here are the highlights of our work from last year and our intentions for the new year.

We have already covered two of many ways to migrate images into Drupal. One example allows you to set the image subfields manually. The other example uses a process plugin that accomplishes the same result using plugin configuration options. Although valid ways to migrate images, these approaches have an important limitation. The files and images are not removed from the system upon rollback. In the previous blog post, we talked further about this topic. Today, we are going to perform an image migration that will clear after itself when it is rolled back. Note that in Drupal images are a special case of files. Even though the example will migrate images, the same approach can be used to import any type of file. This migration will also serve as the basis for explaining migration dependencies in the next blog post.

File entity migrate destination

All the examples so far have been about creating nodes. The migrate API is a full ETL framework able to write to different destinations. In the case of Drupal, the target can be other content entities like files, users, taxonomy terms, comments, etc. Writing to content entities is straightforward. For example, to migrate into files, the process section is configured like this:

destination:
  plugin: 'entity:file'

You use a plugin whose name is entity: followed by the machine name of your target entity. In this case file. Other possible values are user, taxonomy_term, and comment. Remember that each migration definition file can only write to one destination.

Source section definition

The source of a migration is independent of its destination. The following code snippet shows the source definition for the image migration example:

source:
  constants:
    SOURCE_DOMAIN: 'https://agaric.coop'
    DRUPAL_FILE_DIRECTORY: 'public://portrait/'
  plugin: embedded_data
  data_rows:
    - photo_id: 'P01'
      photo_url: 'sites/default/files/2018-12/micky-cropped.jpg'
    - photo_id: 'P02'
      photo_url: ''
    - photo_id: 'P03'
      photo_url: 'sites/default/files/pictures/picture-94-1480090110.jpg'
    - photo_id: 'P04'
      photo_url: 'sites/default/files/2019-01/clayton-profile-medium.jpeg'
  ids:
    photo_id:
      type: string

Note that the source contains relative paths to the images. Eventually, we will need an absolute path to them. Therefore, the SOURCE_DOMAIN constant is created to assemble the absolute path in the process pipeline. Also, note that one of the rows contains an empty photo_url. No file can be created without a proper URL. In the process section we will accommodate for this. An alternative could be to filter out invalid data in a source clean up operation before executing the migration.

Another important thing to note is that the row identifier photo_id is of type string. You need to explicitly tell the system the name and type of the identifiers you want to use. The configuration for this varies slightly from one source plugin to another. For the embedded_data plugin, you do it using the ids configuration key. It is possible to have more than one source column as identifier. For example, if the combination of two columns (e.g. name and date of birth) are required to uniquely identify each element (e.g. person) in the source.

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD migration dependencies introduction whose machine name is ud_migrations_dependencies_intro. The migration to run is udm_dependencies_intro_image. Refer to this article to learn where the module should be placed.

Process section definition

The fields to map in the process section will depend on the target. For files and images, only one entity property is required: uri. Its value should be set to the file path within Drupal using stream wrappers. In this example, the public stream (public://) is used to store the images in a location that is publicly accessible by any visitor to the site. If the file was already in the system and we knew the path the whole process section for this migration could be reduced to two lines:

process:
  uri: source_column_file_uri

That is rarely the case though. Fortunately, there are many process plugins that allow you to transform the available data. When combined with constants and pseudofields, you can come up with creative solutions to produce the format expected by your destination.

Skipping invalid records

The source for this migration contains one record that lacks the URL to the photo. No image can be imported without a valid path. Let’s accommodate for this. In the same step, a pseudofield will be created to extract the name of the file out of its path.

psf_destination_filename:
  - plugin: callback
    callable: basename
    source: photo_url
  - plugin: skip_on_empty
    method: row
    message: 'Cannot import empty image filename.'

The psf_destination_filename pseudofield uses the callback plugin to derive the filename from the relative path to the image. This is accomplished using the basename PHP function. Also, taking advantage of plugin chaining, the system is instructed to skip process the row if no filename could be obtained. For example, because an empty source value was provided. This is done by the skip_on_empty which is also configured log a message to indicate what happened. In this case, the message is hardcoded. You can make it dynamic to include the ID of the row that was skipped using other process plugins. This is left as an exercise to the curious reader. Feel free to share your answer in the comments below.

Tip: To read the messages log during any migration, execute the following Drush command: drush migrate:messages [migration-id].

Creating the destination URI

The next step is to create the location where the file is going to be saved in the system. For this, the psf_destination_full_path pseudofield is used to concatenate the value of a constant defined in the source and the file named obtained in the previous step. As explained before, order is important when using pseudofields as part of the migrate process pipeline. The following snippet shows how to do it:

psf_destination_full_path:
  - plugin: concat
    source:
      - constants/DRUPAL_FILE_DIRECTORY
      - '@psf_destination_filename'
  - plugin: urlencode

The end result of this operation would be something like public://portrait/micky-cropped.jpg. The URI specifies that the image should be stored inside a portrait subdirectory inside Drupal’s public file system. Copying files to specific subdirectories is not required, but it helps with file organizations. Also, some hosting providers might impose limitations on the number of files per directory. Specifying subdirectories for your file migrations is a recommended practice.

Also note that after the URI is created, it gets encoded using the urlencode plugin. This will replace special characters to an equivalent string literal. For example, é and ç will be converted to %C3%A9 and %C3%A7 respectively. Space characters will be changed to %20. The end result is an equivalent URI that can be used inside Drupal, as part of an email, or via another medium. Always encode any URI when working with Drupal migrations.

Creating the source URI

The next step is to create assemble an absolute path for the source image. For this, you concatenate the domain stored in a source constant and the image relative path stored in a source column. The following snippet shows how to do it:

psf_source_image_path:
  - plugin: concat
    delimiter: '/'
    source:
      - constants/SOURCE_DOMAIN
      - photo_url
  - plugin: urlencode

The end result of this operation will be something like https://agaric.coop/sites/default/files/2018-12/micky-cropped.jpg. Note that the concat and urlencode plugins are used just like in the previous step. A subtle difference is that a delimiter is specifying in the concatenation step. This is because, contrary to the DRUPAL_FILE_DIRECTORY constant, the SOURCE_DOMAIN constant does not end with a slash (/). This was done intentionally to highlight two things. First, it is important to understand your source data. Second, you can transform it as needed by using various process plugins.

Copying the image file to Drupal

Only two tasks remain to complete this image migration: download the image and assign the uri property of the file entity. Luckily, both steps can be accomplished at the same time using the file_copy plugin. The following snippet shows how to do it:

uri:
  plugin: file_copy
  source:
    - '@psf_source_image_path'
    - '@psf_destination_full_path'
  file_exists: 'rename'
  move: FALSE

The source configuration of file_copy plugin expects an array of two values: the URI to copy the file from and the URI to copy the file to. Optionally, you can specify what happens if a file with the same name exists in the destination directory. In this case, we are instructing the system to rename the file to prevent name clashes. The way this is done is appending the string _X to the filename and before the file extension. The X is a number starting with zero (0) that keeps incrementing until the filename is unique. The move flag is also optional. If set to TRUE it tells the system that the file should be moved instead of copied. As you can guess, Drupal does not have access to the file system in the remote server. The configuration option is shown for completeness, but does not have any effect in this example.

In addition to downloading the image and place it inside Drupal’s file system, the file_copy also returns the destination URI. That is why this plugin can be used to assign the uri destination property. And that’s it, you have successfully imported images into Drupal! Clever use of the process pipeline, isn’t it? ;-)

One important thing to note is an image’s alternative text, title, width, and height are not associated with the file entity. That information is actually stored in a field of type image. This will be illustrated in the next article. To reiterate, the same approach to migrate images can be used to migrate any file type.

Technical note: The file entity contains other properties you can write to. For a list of available options check the baseFieldDefinitions() method of the File class defining the entity. Note that more properties can be available up in the class hierarchy. Also, this entity does not have multiple bundles like the node entity does.

What did you learn in today’s blog post? Had you created file migrations before? If so, had you followed a different approach? Did you know that you can do complex data transformations using process plugins? Did you know you can skip the processing of a row if the required data is not available? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your colleagues.

Next: Introduction to migration dependencies in Drupal

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

We had a fantastic and fun 2019! Agaric team members attended many conferences and hosted multiple trainings while building sites with some amazing clients. We also took on maintenance for a few sites, helping them gain more audience and stability as well as ease of use for visitors and administrators.

Platform Cooperativism Conference

Wow - more than a month has gone by since the New year began! Heads down, we now surface for some news an announcements on the who, what, where and when of the past year and the beginning of 2020. At the end of last year, Agaric members hosted a Town Hall style session at a conference titled: "Who Owns the World?" This was the yearly Platform Cooperativism event convened by Trebor Scholz at the New School in NYC. It was an incredible way to end the year. We met so many people involved in so many worthy projects that it was a bit overwhelming! 

Attending the 2019 Platformcoop event in NYC was an incredible way to end 2019. We met so many people involved in so many worthy projects that it was a bit overwhelming!  The biggest take-aways were learning about how many successful bicycle delivery platform cooperatives there are in many countries. We also found out that worker-owned cooperatives are sparse in places like Indonesia and Japan even though they have a healthy cooperative ecosystem. Cooperative governance is still a wide-open discussion going on amongst many cooperatives. It was nice to have some discussions with other cooperatives on how they go about creating their governance. We have been in touch with several people that we met at the conference this year.

Since the Platformcoop event in NYC, Agaric has been busy working hard on building two platforms. The first one to mention is the Find It platform, a robust opportunity locator, a participatory searchable directory of events, programs and organizations.

Find It Platform

Last year we connected with the City of Cambridge, Massachusetts as they were met with the challenge of getting their residents more engaged in community activities and increasing usage of public spaces. We worked closely with the Kids Council in Cambridge to coordinate an effort between government, the residents of the city, and local coding groups to design a platform that would provide residents the means to more easily navigate a directory of public events based on their age and interests.

The resulting platform is named Find It, and it was developed by our team of worker-owners at Agaric.

To give you an idea of the value the platform can bring to the residents of a city, here is a PSA on Find It to the residents of Cambridge.

After seeing how successful Find It has been in enriching the communities in Cambridge, we are eager to invite other cities to join in the collaboration. FindIT is built on Drupal, free and open-source software that ensures adopters a powerful and affordable tool that can be altered to suit the needs of the city it serves. Better yet, surveys and interview questions for city residents are already prepared from the Find It Cambridge collaboration. As more cities get involved in the Find It project, we will be able to increase its capabilities all the while decreasing the cost of the service.

Drutopia

A budding platform cooperative that Agaric has collaboratively built with other Drupal shops and freelance contributors is, Drutopia, an ecosystem of Drupal distributions, built and managed as a software cooperative. Soon Agaric will be offering what we call a Libre Software as a Service program, which will allow people to host their site through us, similar to online website builders but governed democratically and with the power of Drupal behind it all. You can find more information on the Drutopia website: https://drutopia.org

Contributing Back to Drupal

Working with Drupal, Agaric has led some data migration training workshops at some Drupal camps. We hosted a migration training in San Francisco at BADcamp - Bay Area Drupal Camp as well as a session on growing the community and scaling democracy. The team hosted a full day, sold-out training for 45 people at DrupalCon Seattle in 2019. We are engaged to host more trainings at the upcoming 2020 DrupalCon in Minnesota in May of 2020. We recently hosted a migration training at DrupalCamp New Jersey at Princeton and these are our upcoming scheduled data migration trainings:

• March 18 MidCamp - Drupal 8 Upgrade All Day Training 

• May 18 DrupalCon Minneapolis - Drupal 8 Content Migrations All Day Training 

• May 19 DrupalCon Minneapolis - Upgrading to Drupal 8 Using the Migrate API All Day Training 

You can see the list and join us  - https://agaric.coop/blog/make-2020-year-you-begin-upgrade-drupal-8-upcoming-trainings-and-resources

We have also been busy working on contributed modules. Currently, Gnuget is working on making https://www.drupal.org/project/filefield_sources stable and at the same time compatible with Drupal 9.

Forging Connections in Mexico

Some Agarics are also headed back to San Miguel de Allende, Mexico to continue a series of lectures and workshops on security, privacy, surveillance, and Platform Cooperativism at the local Biblioteque, in mid-February. The lecture is titled: Platform Cooperativism, Surveillance Capitalism, Predictive Analysis and You.

In 2018, we made our first visit to the Center for Global Justice in San Miguel de Allende when Micky was asked to speak at an Encuentro (gathering) in Mexico City to celebrate the victory of Mexico's electrical workers' union (SME). SME workers were fired en masse in 2009 but stayed united and fought back. They now provide electricity to central Mexico as a co-op under democratic workers' control!  This visit activated an idea to organize trips to Mexico to host workshops and lectures specifically on software freedom and security with an introduction to protecting your privacy online. The next Workshop/Lecture tour will happen February 2020 - We hosted some workshops earlier this year and you can read about them here: https://agaric.coop/blog/micky-lead-free-software-trainings-mexico 

2019 was an exciting year for Agaric as we traveled to teach and to learn. Micky presented as a Keynote speaker at two major events for developers in New England, NerdSummit and the Free Software Foundation's yearly event, LibrePlanet. She will be back again this year with a presentation on Surveillance Capitalism, Predictive Analysis and YOU. View the LibrePlanet Keynote here:

MayFirst Movement Technology

As active members of MayFirst Movement Technology, an organization providing web hosting, email services, and other free software tools to its members. MFMT also advocates for an open web and ethical technology. This year, Micky was elected by the members to serve on the board of the new cooperative formed by the members. Together, we are building online and offline workshops and discussions on freedom and raising awareness on the need to have a web host that is ethical and that you truly trust to protect you and your data.

Agaric continues to host a weekly online gathering where everyone shares methods and processes and ideas for sharing our knowledge and learning from others. You can join us any Wednesday for Show and Tell - lurk, engage or contribute! We also have some new Agaric initiatives and we have launched some new client websites as well as redesigns of other sites. We continue to reach out to share the knowledge we have gained while including as many people as we can in the process of learning to use free software to make a better planet and to make a better life for all!

Today we complete the user migration example. In the previous post, we covered how to migrate email, timezone, username, password, and status. This time, we cover creation date, roles, and profile pictures. The source, destination, and dependencies configurations were explained already. Therefore, we are jumping straight to the process transformations in this entry.

Getting the code

You can get the full code example at https://github.com/dinarcon/ud_migrations The module to enable is UD users whose machine name is ud_migrations_users. The two migrations to execute are udm_user_pictures and udm_users. Notice that both migrations belong to the same module. Refer to this article to learn where the module should be placed.

The example assumes Drupal was installed using the standard installation profile. Particularly, we depend on a Picture (user_picture) image field attached to the user entity. The word in parenthesis represents the machine name of the image field.

The explanation below is only for the user migration. It depends on a file migration to get the profile pictures. One motivation to have two migrations is for the images to be deleted if the file migration is rolled back. Note that other techniques exist for migrating images without having to create a separate migration. We have covered two of them in the articles about subfields and constants and pseudofields.

Migrating user creation date

Have a look at the previous post for details on the source values. For reference, the user creation time is provided by the member_since column, and one of the values is April 4, 2014. The following snippet shows how the various user date related properties are set:

created:
  plugin: format_date
  source: member_since
  from_format: 'F j, Y'
  to_format: 'U'
changed: '@created'
access: '@created'
login: '@created'

The created, entity property stores a UNIX timestamp of when the user was added to Drupal. The value itself is an integer number representing the number of seconds since the epoch. For example, 280299600 represents Sun, 19 Nov 1978 05:00:00 GMT. Kudos to the readers who knew this is Drupal's default expire HTTP header. Bonus points if you knew it was chosen in honor of someone’s birthdate. ;-)

Back to the migration, you need to transform the provided date from Month day, year format to a UNIX timestamp. To do this, you use the format_date plugin. The from_format is set to F j, Y which means your source date consists of:

• The full textual representation of a month: April.
• Followed by a space character.
• Followed by the day of the month without leading zeros: 4.
• Followed by a comma and another space character.
• Followed by the full numeric representation of a year using four digits: 2014

If the value of from_format does not make sense, you are not alone. It is actually assembled from format characters of the date PHP function. When you need to specify the from and to formats, you basically need to look at the documentation and assemble a string that matches the desired date format. You need to pay close attention because upper and lowercase letters represent different things like Y and y for the year with four-digits versus two-digits respectively. Some date components have subtle variations like d and j for the day with or without leading zeros respectively. Also, take into account white spaces and date component separators. To finish the plugin configuration, you need to set the to_format configuration to something that produces a UNIX timestamp. If you look again at the documentation, you will see that U does the job.

The changed, access, and login entity properties are also dates in UNIX timestamp format. changed indicates when the user account was last updated. access indicates when the user last accessed the site. login indicated when the user last logged in. For brevity, the same value assigned to created is also assigned to these three entity properties. The at sign (@) means copy the value of a previous mapping in the process pipeline. If needed, each property can be set to a different value or left unassigned. None is actually required.

Migrating user roles

For reference, the roles are provided by the user_roles column, and one of the values is forum moderator, forum admin. It is a comma separated list of roles from the legacy system which need to be mapped to Drupal roles. It is possible that the user_roles column is not provided at all in the source. The following snippet shows how the roles are set:

roles:
  - plugin: skip_on_empty
    method: process
    source: user_roles
  - plugin: explode
    delimiter: ','
  - plugin: callback
    callable: trim
  - plugin: static_map
    map:
      'forum admin': administrator
      'webmaster': administrator
    default_value: null

First, the skip_on_empty plugin is used to skip the processing of the roles if the source column is missing. Then, the explode plugin is used to break the list into an array of strings representing the roles. Next, the callback plugin invokes the trim PHP function to remove any leading or trailing whitespace from the role names. Finally, the static_map plugin is used to manually map values from the legacy system to Drupal roles. All of these plugins have been explained previously. Refer to other articles in the series or the plugin documentation for details on how to use and configure them.

There are some things that are worth mentioning about migrating roles using this particular process pipeline. If the comma separated list includes spaces before or after the role name, you need to trim the value because the static map will perform an equality check. Having extraneous space characters will produce a mismatch.

Also, you do not need to map the anonymous or authenticated roles. Drupal users are assumed to be authenticated and cannot be anonymous. Any other role needs to be mapped manually to its machine name. You can find the machine name of any role in its edit page. In the example, only two out of four roles are mapped. Any role that is not found in the static map will be assigned the value null as indicated in the default_value configuration. After processing the null value will be ignored, and no role will be assigned. But you could use this feature to assign a default role in case the static map does not produce a match.

Migrating profile pictures

For reference, the profile picture is provided by the user_photo column, and one of the values is P01. This value corresponds to the unique identifier of one record in the udm_user_pictures file migration, which is part of the same demo module.  It is important to note that the user_picture field is not a user entity property. The field is created by the standard installation profile and attached to the user entity. You can find its configuration in the “Manage fields” tab of the “Account settings” configuration page at /admin/config/people/accounts. The following snippet shows how profile pictures are set:

user_picture/target_id:
  plugin: migration_lookup
  migration: udm_user_pictures
  source: user_photo

Image fields are entity references. Their target_id property needs to be an integer number containing the file id (fid) of the image. This can be obtained using the migration_lookup plugin. Details on how to configure it can be found in this article. You could simply use user_picture as your field mapping because target_id is the default subfield and could be omitted. Also note that the alt subfield is not mapped. If present, its value will be used for the alternative text of the image. But if it is not specified, like in this example, Drupal will automatically generate an alternative text out of the username. An example value would be: Profile picture for user michele.

Technical note: The user entity contains other properties you can write to. For a list of available options, check the baseFieldDefinitions() method of the User class defining the entity. Note that more properties can be available up in the class hierarchy.

And with that, we wrap up the user migration example. We covered how to migrate a user’s mail, timezone, username, password, status, creation date, roles, and profile picture. Along the way, we presented various process plugins that had not been used previously in the series. We showed a couple of examples of process plugin chaining to make sure the migrated data is valid and in the format expected by Drupal.

What did you learn in today’s blog post? Did you know how to process dates for user entity properties? Have you migrated user roles before? Did you know how to import profile pictures? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with others.

Next: Migrating dates into Drupal

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

Drupal is one of the largest free software projects in the world. It powers thousands of websites, many of them for community groups, nonprofits and other grassroots organizations.  

As a cooperative distributed across three countries, we're using DrupalCon as a unique opportunity to meet face to face with ourselves and others in the community. Our focus, as usual, is to work with others to make Drupal work for grassroots movements.

In a time where people power is more important than ever, it's critical that groups have communication tools that supports those efforts. Drupal comes equipped with powerful tools like multilingual support, flexible content modeling, and a modular design.

NonProfit Summit

May 19th 9-4pm, $150 (lunch and coffee provided)

Clayton is one of the co-organizers of the NonProfit Summit and it's on pace to be one of the best. Now running for many years, the schedule has been fine tuned to the needs of nonprofit and activist technologists with a combination of panels, breakout discussions and lightning talks.

Register for the summit.

Making the Move to Drupal 8

Drupal 7 end of life is coming in 2021 and many sites are still on Drupal 7. The move to 8 can be challenging, so we're offering two trainings and one session on the topic.

Drupal 8 Content Migrations Training

May 19th 9-4pm, $500 (lunch and coffee provided)

Back by popular demand is our training to teach community members how to move content into Drupal using the Migrate API. Participants will

• Learn to run migrations from the user interface and the command line with Drush.
• Import data from CSV and JSON files.
• Transform the data to populate taxonomy, date, image, file, and address fields.
• Get content into Paragraphs.
• Learn to debug migrations.

The only prerequisite is a basic understanding of nodes, content types and fields. We will work hands-on through step by step examples, with four trainers on hand to assist people and answer specific questions participants have.

Last year's training sold out and we expect the same this year, so be sure to sign up sooner than later.

Register for the Drupal 8 content migrations training.

Upgrading to Drupal 8 Using the Migrate API Training

After giving our Drupal 8 Content Migrations training several times last year, we learned there is also a need for a training that is specifically for those upgrading from Drupal 6/7 to Drupal 8.

In this training, participants will

• Understand the different approaches to upgrading your site to Drupal 8 using the Migrate API.
• Revise site architecture and map configuration from previous site to the new one (content types, fields, etc).
• Learn how to migrate data into media entities and paragraphs.
• Learn how to migrate data from modules that do not offer automated upgrade paths.

While the content migration training is at an Intermediate level, this training is advanced. You should be familiar with source, process and destination plugins; how the process pipeline works; and how to execute migrations from the command line via Drush. This is a great training for experienced Drupalers looking to upgrade Drupal sites, including those that will undergo changes to their information architecture.

Register for the Upgrading to Drupal 8 Using the Migrate API training.

If Drupal migrations work is a big part of your work, the two trainings are intentionally complimentary and is a good opportunity to level up in this domain.

Drupal Migrations for Non-Coders Session

There is a growing collection of resources on migrating to Drupal 8, but most of it is focused on professional Drupal developers. However, there are many organizations with staff who are technical, but not day-to-day coders. This is particularly common in nonprofits where people where many hats. If that describes you, we encourage you to join us.

The specific day and time hasn't been set, but you can still add it to your schedule.

Add Drupal migration for non-coders to your schedule.

Grassroots Drupal Birds of a Feather

Each year we lead or join in on a BoF with others working with grassroots groups. We'll be doing the same, so keep an eye out for that in the schedule.

It's also great to grab coffee, have a hallway conversation, scheme over lunch and hang out at after parties with kindred spirits. Reach out to us at ask@agaric.com if you're going to DrupalCon and want to meet up!

We are proud that Agaric joined the Tech Co-Op Network as an early member. This network of North American tech worker co-ops seeks to encourage collaboration among its members and education of potential new co-op founders and the general public about worker cooperatives: businesses owned and controlled by the people who work in them.

Read all about it!

Sign up to be notified when Agaric gives a migration training:

In the previous article we explained the syntax used to write Drupal migrations. As part of the field mapping explanation, we talked about how to set subfields. This is a topic we have covered early in the 31 days of Drupal migrations series. On both occasions, we mentioned that finding out which subfields are available for a field type might require some Drupal development knowledge. To make the process easier, in today's article we are presenting a reference of subfields provided by core and contributed modules.

For each field type we will present: the module that provides it, its plugin ID and category, the class that defines it, the subfields available, and the default subfield if any. If a field has only one subfield, that will be its default. In some cases, there will be references to articles that cover migrating into those field types specifically.

As a refresher, if you want to find available subfields by yourself, you need to locate the class that provides the FieldType plugin and inspect its schema method. The latter defines the database columns used by the field to store its data. Because of object oriented practices, sometimes you need to look at the parent class to find all the subfields that are available. When migrating into subfields, you are actually migrating into those particular database columns. Any restriction set by the database schema needs to be respected.

Address field

Module: Address. Plugin ID: address. Category: Address.
Class: Drupal\address\Plugin\Field\FieldType\AddressItem
Related article: Migrating addresses into Drupal
Note: This field type does not have a default subfield.

List of subfields:

1. langcode: The language code.
2. country_code: The two-letter country code as defined by the ISO 3166 standard.
3. administrative_area: The top-level administrative subdivision of the country (e.g., state or province).
4. locality: The locality (i.e. city).
5. dependent_locality: The dependent locality (i.e. neighbourhood).
6. postal_code: The postal or ZIP code. It will be validated against administrative_area if validation patterns are available for that state or province.
7. sorting_code: The sorting code.
8. address_line1: The first line of the address block.
9. address_line2: The second line of the address block.
10. organization: The organization or company.
11. given_name: The first name.
12. additional_name: The middle name.
13. family_name: The last name.

Country field

Module: Address. Plugin ID: address_country. Category: Address.
Class: Drupal\address\Plugin\Field\FieldType\CountryItem
Related article: Migrating addresses into Drupal

Subfield: value for the two-letter country code as defined by the ISO 3166 standard.

Zone field

Module: Address. Plugin ID: address_zone. Category: Address.
Class: Drupal\address\Plugin\Field\FieldType\ZoneItem
Related article: Migrating addresses into Drupal

Subfield: value for serialized version of a Zone object.

Boolean field

Module: Core library. Plugin ID: boolean. Category: General.
Class: Drupal\Core\Field\Plugin\Field\FieldType\BooleanItem

Subfield: value stores an integer number: 0 for FALSE and 1 for TRUE.

Number integer field

Module: Core library. Plugin ID: integer. Category: Number.
Class: Drupal\Core\Field\Plugin\Field\FieldType\IntegerItem

Subfield: value for the integer number.

Number float field

Module: Core library. Plugin ID: float. Category: Number.
Class: Drupal\Core\Field\Plugin\Field\FieldType\FloatItem

Subfield: value for the float number.

Number decimal field

Module: Core library. Plugin ID: decimal. Category: Number.
Class: Drupal\Core\Field\Plugin\Field\FieldType\DecimalItem

Subfield: value for the decimal number.

List integer field

Module: Options (Drupal core). Plugin ID: list_integer. Category: Number.
Class: Drupal\options\Plugin\Field\FieldType\ListIntegerItem

Subfield: value for the integer number. It is one of the keys in the “Allowed values list” of the field’s setting.

List float field

Module: Options (Drupal core). Plugin ID: list_float. Category: Number.
Class: Drupal\options\Plugin\Field\FieldType\ListFloatItem

Subfield: value for the float number. It is one of the keys in the “Allowed values list” of the field’s setting.

Text plain field

Module: Core library. Plugin ID: string. Category: Text.
Class: Drupal\Core\Field\Plugin\Field\FieldType\StringItem

Subfield: value for a short plain text. The maximum length is 255 characters unless a lower number is set in the field's storage settings.

Text plain, long field

Module: Core library. Plugin ID: string_long. Category: Text.
Class: Drupal\Core\Field\Plugin\Field\FieldType\StringLongItem

Subfield: value for a long plain text.

List text field

Module: Options (Drupal core). Plugin ID: list_string. Category: Text.
Class: Drupal\options\Plugin\Field\FieldType\ListStringItem

Subfield: value for the text. It is one of the keys in the “Allowed values list” of the field’s setting. The maximum length for a key is 255 characters.

Text formatted field

Module: Text (Drupal core). Plugin ID: text. Category: Text.
Class: Drupal\text\Plugin\Field\FieldType\TextItem
Default subfield: value

List of subfields:

1. value: A short text. It can include markup. The maximum length is 255 characters unless a lower number is set in the field's storage settings.
2. format: A string containing the machine name of the text format to use. For example: plain_text, full_html, restricted_html, and basic_html.

Text formatted, long field

Module: Text (Drupal core). Plugin ID: text_long. Category: Text.
Class: Drupal\text\Plugin\Field\FieldType\TextLongItem
Default subfield: value

List of subfields:

1. value: The text. It can include markup.
2. format: A string containing the machine name of the text format to use. For example: plain_text, full_html, restricted_html, and basic_html.

Text formatted, long with summary field

Module: Text (Drupal core). Plugin ID: text_with_summary. Category: Text.
Class: Drupal\text\Plugin\Field\FieldType\TextWithSummaryItem
Related article: Migrating formatted text
Default subfield: value

List of subfields:

1. value: The full text. It can include markup.
2. summary: The summary text. It can include markup.
3. format: A string containing the machine name of the text format to use. For example: plain_text, full_html, restricted_html, and basic_html.

Comments field

Module: Comment (Drupal core). Plugin ID: comment. Category: General.
Class: Drupal\comment\Plugin\Field\FieldType\CommentItem

Subfield: status stores an integer number that indicates whether comments are allowed on the entity: 0 = no, 1 = closed (read only), 2 = open (read/write).

Datetime field

Module: Datetime (Drupal core). Plugin ID: datetime. Category: General.
Class: Drupal\datetime\Plugin\Field\FieldType\DateTimeItem
Related article: Migrating dates into Drupal

Subfield: value for the datetime value. It is a string similar to 2020-01-15T07:03:21.

If you decide to only store the date, the string will be similar to 2020-01-15.

Datetime range field

Module: Datetime Range (Drupal core). Plugin ID: daterange. Category: General.
Class: Drupal\datetime_range\Plugin\Field\FieldType\DateRangeItem
Related article: Migrating dates into Drupal
Default subfield: value

List of subfields:

1. value The start datetime value. It is a string similar to 2020-01-15T07:03:21.
2. end: The end datetime value. It is a string similar to 2020-01-15T07:03:21.

If you decide to only store the date, the string for both subfields will be similar to 2020-01-15. If you decide to store all day, the value string will be similar to 2020-01-15T05:00:00 and the end string to 2020-01-17T04:59:59.

Email field

Module: Core library. Plugin ID: email. Category: General.
Class: Drupal\Core\Field\Plugin\Field\FieldType\EmailItem

Subfield: value for the email address.

Link field

Module: Link (Drupal core). Plugin ID: link. Category: General.
Class: Drupal\link\Plugin\Field\FieldType\LinkItem
Related article: Understanding the syntax of Drupal migrations
Default subfield: uri

List of subfields:

1. uri: A string for the link’s URI.
2. title: A string for the link’s title.
3. options: A serialized array of options for the link.

Telephone number field

Module: Telephone (Drupal core). Plugin ID: telephone. Category: Number.
Class: Drupal\telephone\Plugin\Field\FieldType\TelephoneItem

Subfield: value for the telephone number. It is a string of 256 characters maximum. For example: +1 (508) 283-3557 or 1800-DRUPAL.

Timestamp field

Module: Core library. Plugin ID: timestamp. Category: General.
Class: Drupal\Core\Field\Plugin\Field\FieldType\TimestampItem

Subfield: value for a UNIX timestamp value. It stores an integer in the range [-2147483648, 2147483648]. For example: 280299600. Note that the value can be negative.

Entity reference field including nodes, users, taxonomy terms, and media entities

Module: Core library. Plugin ID: entity_reference. Category: Reference.
Class: Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem

Subfield: target_id stores an integer representing the ID of the target entity: nid, uid, tid, mid, etc.

Entity reference file field

Module: File (Drupal core). Plugin ID: file. Category: Reference.
Class: Drupal\file\Plugin\Field\FieldType\FileItem
Related article: Migrating files and images into Drupal
Default subfield: target_id

List of subfields:

1. target_id: An integer representing the ID of the target entity.
2. display: An flag to control whether this file should be displayed when viewing content. Possible values are 0 and 1.
3. description: A description of the file.

Entity reference image field

Module: Image (Drupal core). Plugin ID: image. Category: Reference.
Class: Drupal\image\Plugin\Field\FieldType\ImageItem
Related articles: Migrating files and images into Drupal, Migrating images using the image_import plugin, and Migrating images using the image_import plugin
Default subfield: target_id

List of subfields:

1. target_id: An integer representing the ID of the target entity.
2. alt: Alternative image text, for the image's 'alt' attribute.
3. title: Image title text, for the image's 'title' attribute.
4. width: The width of the image in pixels.
5. height: The height of the image in pixels.

Entity reference revisions field including paragraphs

Module: Core library. Plugin ID: entity_reference_revisions. Category: Reference revisions.
Class: Drupal\Core\Field\Plugin\Field\FieldType\EntityReferenceItem
Related article: Introduction to paragraphs migrations in Drupal
Default subfield: target_id

List of subfields:

1. target_id: An integer representing the ID of the target entity: nid, uid, tid, fid, etc.
2. target_revision_id: An integer representing the revision ID of the target entity.

Fields hidden from the user interface

Drupal core includes more field types that are hidden from the user interface. That is, it is not possible to add them from a content type’s “Manage fields” tab. For brevity, their subfields will not be listed. Instead, each will have a link to its definition class. There you can inspect the schema method to obtain the list of subfields. As of this writing, the hidden fields are:

1. changed provided by Drupal\Core\Field\Plugin\Field\FieldType\ChangedItem
2. created provided by Drupal\Core\Field\Plugin\Field\FieldType\CreatedItem
3. file_uri provided by Drupal\file\Plugin\Field\FieldType\FileUriItem
4. language provided by Drupal\Core\Field\Plugin\Field\FieldType\LanguageItem
5. map provided by Drupal\Core\Field\Plugin\Field\FieldType\MapItem
6. password provided by Drupal\Core\Field\Plugin\Field\FieldType\PasswordItem
7. path provided by Drupal\path\Plugin\Field\FieldType\PathItem
8. uri provided by Drupal\Core\Field\Plugin\Field\FieldType\UriItem
9. uuid provided by Drupal\Core\Field\Plugin\Field\FieldType\UuidItem

Subfields for other contributed modules

There are hundreds of modules that provide new field types. It would be impractical to list all of them here. As mentioned before, you need to find a field type’s definition class to know which subfields are available. Contributed modules would have these classes in their /src/Plugin/Field/FieldType/ directory. The class itself will have the @FieldType annotation. For reference, below is a list of some contributed field types:

1. Color Field module provides ColorFieldType.
2. Geofield module provides GeofieldItem.
3. Google Map Field module provides GoogleMapFieldType.
4. Name Field module provides NameItem.
5. Office Hours module provides OfficeHoursItem.
6. Physical Fields module provides DimensionsItem and MeasurementItem.
7. Viewfield module provides ViewfieldItem.

What did you learn in today’s article? Did you know that there were so many field types in Drupal core? Were you aware that some core field types are hidden from the user interface? Did you know how to find subfields for field types from contributed modules? Please share your answers in the comments. Also, we would be grateful if you shared this article with your friends and colleagues.