Agaric is facilitating a full day training at DrupalCon Seattle to help you understand how to import content into your to Drupal 8 website.  It's the latest in our series of migration trainings.

This training is open for attendees with intermediate experience with Drupal– familiarity with installing a Drupal site and installing modules. We will use the Migrate API and related modules, which allows users to migrate content without writing any code.

With six instructors we will ensure no one gets left behind. Instead everyone will get the attention they need.

Attendees will learn to:

• Import data from CSV and JSON files.
• Transform data to populate taxonomy, date, image, file, and address fields.
• Get content into Paragraphs.
• Debug migration errors.

The training price is $450. Space is limited so we do encourage registering as soon as possible to ensure your spot.

As a web development cooperative that champions free software, we're passionate about migrations. It is a way to better understand Drupal's codebase, tap into the power of new features and build community. We have successfully migrated multiple sites to Drupal 8 and look forward to sharing our experience with you.

This training has sold out but please get in touch to be the first to know about future opportunities!

So far we have learned how to write basic Drupal migrations and use process plugins to transform data to meet the format expected by the destination. In the previous entry we learned one of many approaches to migrating images. In today’s example, we will change it a bit to introduce two new migration concepts: constants and pseudofields. Both can be used as data placeholders in the migration timeline. Along with other process plugins, they allow you to build dynamic values that can be used as part of the migrate process pipeline.

Setting and using constants

In the Migrate API, a constant is an arbitrary value that can be used later in the process pipeline. They are set as direct children of  the source section. You write a constants key whose value is a list of name-value pairs. Even though they are defined in the source section, they are independent of the particular source plugin in use. The following code snippet shows a generalization for settings and using constants:

source:
  constants:
    MY_STRING: 'http://understanddrupal.com'
    MY_INTEGER: 31
    MY_DECIMAL: 3.1415927
    MY_ARRAY:
      - 'dinarcon'
      - 'dinartecc'
  plugin: source_plugin_name
  source_plugin_config_1: source_config_value_1
  source_plugin_config_2: source_config_value_2
process:
  process_destination_1: constants/MY_INTEGER
  process_destination_2:
    plugin: concat
    source: constants/MY_ARRAY
    delimiter: ' '

You can set as many constants as you need. Although not required by the API, it is a common convention to write the constant names in all uppercase and using underscores (_) to separate words. The value can be set to anything you need to use later. In the example above, there are strings, integers, decimals, and arrays. To use a constant in the process section you type its name, just like any other column provided by the source plugin. Note that you use the constant you need to name the full hierarchy under the source section. That is, the word constant and the name itself separated by a slash (/) symbol. They can be used to copy their value directly to the destination or as part of any process plugin configuration.

Technical note: The word constants for storing the values in the source section is not special. You can use any word you want as long as it does not collide with another configuration key of your particular source plugin. A reason to use a different name is that your source actually contains a column named constants. In that case you could use defaults or something else. The one restriction is that whatever value you use, you have to use it in the process section to refer to any constant. For example:

source:
  defaults:
    MY_VALUE: 'http://understanddrupal.com'
  plugin: source_plugin_name
  source_plugin_config: source_config_value
process:
  process_destination: defaults/MY_VALUE

Setting and using pseudofields

Similar to constants, pseudofields stores arbitrary values for use later in the process pipeline. There are some key differences. Pseudofields are set in the process section. The name is arbitrary as long as it does not conflict with a property name or field name of the destination. The value can be set to a verbatim copy from the source (a column or a constant) or they can use process plugins for data transformations. The following code snippet shows a generalization for settings and using pseudofields:

source:
  constants:
    MY_BASE_URL: 'http://understanddrupal.com'
  plugin: source_plugin_name
  source_plugin_config_1: source_config_value_1
  source_plugin_config_2: source_config_value_2
process:
  title: source_column_title
  my_pseudofield_1:
    plugin: concat
    source:
      - constants/MY_BASE_URL
      - source_column_relative_url
    delimiter: '/'
  my_pseudofield_2:
    plugin: urlencode
    source: '@my_pseudofield_1'
  field_link/uri: '@my_pseudofield_2'
  field_link/title: '@title'

In the above example, my_pseudofield_1 is set to the result of a concat process transformation that joins a constant and a column from the source section. The result value is later used as part of a urlencode process transformation. Note that to use the value from my_pseudofield_1 you have to enclose it in quotes (') and prepend an at sign (@) to the name. The new value obtained from URL encode operation is stored in my_pseudofield_2. This last pseudofield is used to set the value of the URI subfield for field_link. The example could be simplified, for example, by using a single pseudofield and chaining process plugins. It is presented that way to demonstrate that a pseudofield could be used as direct assignments or as part of process plugin configuration values.

Technical note: If the name of the subfield can be arbitrary, how can you prevent name clashes with destination property names and field names? You might have to look at the source for the entity and the configuration of the bundle. In the case of a node migration, look at the baseFieldDefinitions() method of the Node class for a list of property names. Be mindful of class inheritance and method overriding. For a list of fields and their machine names, look at the “Manage fields” section of the content type you are migrating into. The Field API prefixes any field created via the administration interface with the string field_. This reduces the likelihood of name clashes. Other than these two name restrictions, anything else can be used. In this case, the Migrate API will eventually perform an entity save operation which will discard the pseudofields.

Understanding Drupal Migrate API process pipeline

The migrate process pipeline is a mechanism by which the value of any destination property, field, or pseudofield that has been set can be used by anything defined later in the process section. The fact that using a pseudofield requires to enclose its name in quotes and prepend an at sign is actually a requirement of the process pipeline. Let’s see some examples using a node migration:

• To use the title property of the node entity, you would write @title
• To use the field_body field of the Basic page content type, you would write @field_body
• To use the my_temp_value pseudofield, you would write @my_temp_value

In the process pipeline, these values can be used just like constants and columns from the source. The only restriction is that they need to be set before being used. For those familiar with the rewrite results feature of Views, it follows the same idea. You have access to everything defined previously. Anytime you use enclose a name in quotes and prepend it with an at sign, you are telling the migrate API to look for that element in the process section instead of the source section.

Migrating images using the image_import plugin

Let’s practice the concepts of constants, pseudofields, and the migrate process pipeline by modifying the example of the previous entry. The Migrate Files module provides another process plugin named image_import that allows you to directly set all the subfield values in the plugin configuration itself.

As in previous examples, we will create a new module and write a migration definition file to perform the migration. It is assumed that Drupal was installed using the standard installation profile. The code snippets will be compact to focus on particular elements of the migration. The full code is available at https://github.com/dinarcon/ud_migrations The module name is UD Migration constants and pseudofields and its machine name is ud_migrations_constants_pseudofields. The id of the example migration is udm_constants_pseudofields. Refer to this article for instructions on how to enable the module and run the migration. Make sure to download and enable the Migrate Files module. Otherwise, you will get an error like: “In DiscoveryTrait.php line 53: The "file_import" plugin does not exist. Valid plugin IDs for Drupal\migrate\Plugin\MigratePluginManager are:...”. Let’s see part of the source definition:

source:
  constants:
    BASE_URL: 'https://agaric.coop'
    PHOTO_DESCRIPTION_PREFIX: 'Photo of'
  plugin: embedded_data
  data_rows:
    -
      unique_id: 1
      name: 'Michele Metts'
      photo_url: 'sites/default/files/2018-12/micky-cropped.jpg'
      photo_width: '587'
      photo_height: '657'

Only one record is presented to keep snippet short, but more exist. In addition to having a unique identifier, each record includes a name, a short profile, and details about the image. Note that this time, the photo_url does not provide an absolute URL. Instead, it is a relative path from the domain hosting the images. In this example, the domain is https://agaric.coop so that value is stored in the BASE_URL constant which is later used to assemble a valid absolute URL to the image. Also, there is no photo description, but one can be created by concatenating some strings. The PHOTO_DESCRIPTION_PREFIX constant stores the prefix to add to the name to create a photo description. Now, let’s see the process definition:

process:
  title: name
  psf_image_url:
    plugin: concat
    source:
      - constants/BASE_URL
      - photo_url
    delimiter: '/'
  psf_image_description:
    plugin: concat
    source:
      - constants/PHOTO_DESCRIPTION_PREFIX
      - name
    delimiter: ' '
  field_image:
    plugin: image_import
    source: '@psf_image_url'
    reuse: TRUE
    alt: '@psf_image_description'
    title: '@title'
    width: photo_width
    height: photo_height

The title node property is set directly to the value of the name column from the source. Then, two pseudofields. psf_image_url stores a valid absolute URL to the image using the BASE_URL constant and the photo_url column from the source. psf_image_description uses the PHOTO_DESCRIPTION_PREFIX constant and the name column from the source to store a description for the image.

For the field_image field, the image_import plugin is used. This time, the subfields are not set manually. Instead, they are assigned using plugin configuration keys. The absence of the id_only configuration allows for this. The URL to the image is set in the source key and uses the psf_image_url pseudofield. The alt key allows you to set the alternative attribute for the image and in this case the psf_image_description pseudofield is used. For the title subfield sets the text of a subfield with the same name and in this case it is assigned the value of the title node property which was set at the beginning of the process pipeline. Remember that not only psedufields are available. Finally, the width and height configuration uses the columns from the source to set the values of the corresponding subfields.

What did you learn in today’s blog post? Did you know you can define constants in your source as data placeholders for use in the process section? Were you aware that pseudofields can be created in the process section to store intermediary data for process definitions that come next? Have you ever wondered what is the migration process pipeline and how it works? Please share your answers in the comments. Also, I would be grateful if you shared this blog post with your colleagues.

Next: Tips for writing Drupal migrations and understanding their workflow

This blog post series, cross-posted at UnderstandDrupal.com as well as here on Agaric.coop, is made possible thanks to these generous sponsors. Contact Understand Drupal if your organization would like to support this documentation project, whether it is the migration series or other topics.

 

V. Auxiliary Tech Projects

CoopGuide

Working together on projects is one of the most important goals of our tech coop. It is also the 6th Cooperative Principle, Cooperation among Cooperatives.  CoopGuide is a work in progress being built to facilitate team members in locating each other and getting familiar with each other's skills. The idea for this site came from discussions at Agaric's weekly Show and Tell gatherings. 

SnowDrift

A crowdmatching platform that is a fund raising source for public goods. The first project will be to fund the building of Snowdrift itself. Micky is on the board and is working with the team to guide the formation of this cooperative effort.

IEEE.org SA Open

The SA Open is a platform, by the IEEE and managed by LeadingBit, that
aims to create a set of agreed upon standards to help people, both technical and organic, co-work more effectively to build Free Software with proper documentation and support materials on using and changing it. Agaric volunteers as a member of the Community Advisory Group, where we offer our expertise on this very subject. You can get involved with SA Open and help make it happen.

meet.coop

Meet.coop is an online meeting co-operative dedicated to hosting BigBlueButton video chat servers in multiple locations around the world. Agaric is a member.  We meet weekly in different teams to work out the governance and member issues. There is an operational team that builds and maintains the hardware and software for providing an excellent video conference network structured in Sociocracy, also called dynamic governance. Meet.coop has forums where you can mingle with members and get up to date information on the development and learn more about the project.

Mastodon

Mastodon is a federated social network made up of affinity groups. The functionality is much like Twitter, where you can message other members and share information or "Toots" as they are called, in a group or one-on-one. You can join a sub-community or create your own. Agarics are members of the social.coop sub-community. Members can contact members of other sub-communities and there is a directory where you can browse the groups in the whole federation.

Mass Mesh

Mass Mesh is a high-tech social club building community-owned mesh networks throughout Boston. These networks enable home Internet access that is fairer, more secure, and more locally resilient than traditional cable. The purpose of this wiki is to provide you with all of the information you need to join the mesh by setting up your own mesh node. All of the information on their site and all of the code in their software is free to use share and edit.

Originally published on the Nonprofit Technology Network's Connect Blog.

Most of us don’t look at our site analytics. At least, not often enough. We know we should, but life gets in the way. How do we make it easier for nonprofits to truly take a “data-driven” approach to their work?

I’ve found success with putting the analytics where people will see them. One of those places is the website itself.

Figure Out What Really Matters

There’s no shortage of data to review. So, it’s important to move beyond vanity metrics and get to the heart of why we do what we do. This means turning to the goals and key performance indicators you have for your website (or defining them for the first time!).

For my work at Agaric, those goals are:

1. Secure well-matched projects by communicating the value we provide to potential clients.

Key performance indicator: feedback on design and content from target audiences.

2. Diversify and expand free software communities by sharing relevant knowledge.

Key performance indicator: pageviews of blog posts.

Each goal should be accompanied by at least one key performance indicator. This is data that tells you how successful you are being at reaching your goal.

In our case, our first goal of feedback is best measured qualitatively by asking our current clients— and those who we like working with—what they think of the website. We conduct interviews to gather that feedback. For our second goal, we can get a good picture of content relevance by pageviews, a valuable data point to share with the team.

A different site might try to increase one-time donations, in which case seeing the number of donations made during a campaign would be helpful. Another group might focus on building a regular readership, therefore email list sign ups are the best indicator of success. Whatever it is, make sure you can link the analytics you are tracking back to a goal you have for your site. There’s no point in measuring something you won’t take action on.

Know Who Needs to See the Data and Where They Hang Out

After identifying your key performance indicators, decide who on your team should review that data.

For our six-person worker-owned cooperative, that answer was easy – all of us. We all blog and we all have a vested interest in helping our free software communities thrive. We want to know which posts are resonating the most.

After knowing your target audience, find out where they spend their time. In our case, it’s the website’s back-end content overview page. Our website admins go here to pull up a page we want to update and to see what posts are still in draft mode. So, we added a column for pageviews and made that column sortable.

This content overview page was customized to include each post's total pageview count.

For the independent news site Portside, the same was true. In addition to showing pageviews on their content overview page, they also include them directly on each post (visible only to content editors).

Showing the pageview count on the page itself makes it clear to content editors how popular it is.

For the online educator portal Teachers with Guts, the organization wanted to track several data points on their members’ use of the platform. So, they have a report page built into the site showing information such as the number of downloads made, comments left, and pages bookmarked.

Teachers With Guts began tracking user engagement to inform how they manage their online portal.

Other opportunities to share analytics include weekly email reports, a user dashboard upon logging in, or via mobile texting or apps. Don’t be shy about asking your team where they would most likely notice the data you’re sharing with them.

Meaningful, Informed Conversations

By showing key data in high traffic areas, you foster an informed team. From there you can have the conversations you want and need. We now know which posts are getting the most reach and are evaluating why that is. As a result, our best practices have evolved to make our writing more relevant with readers.